This publication was produced at the request of the United States Department of Agriculture. It was prepared independently by Comercializadora Quetzalteca of Bienes and Servicios S.A. (EQSA) Foreign Agricultural Service, United States Department of Agriculture McGovern-Dole Guatemala EDUCAMOS Program – Educate Us: Quality Education and Improved Feeding Focused on Sustainability Mid-Term Evaluation September 2019 2 EDUCAMOS Program Mid-Term Evaluation The EDUCAMOS Program covers 294 schools in six municipalities of the department of Huehuetenango in Guatemala and implements comprehensive interventions to improve the literacy of school age children, as well as increase their use of child health and nutrition practices. To accomplish this, the program improves the quality of literacy teaching, improves student attention and attendance, increases knowledge of health and hygiene practices and of safe food preparation and storage, increases access to clean water and sanitation services, increases the capacities of governmental institutions, improves policies and the regulatory environment, and increases the commitment of local organizations and community groups. This document evaluates progress towards results in intervention schools, compares results of intervention schools to comparison schools on key outcomes of interest, and responds to key evaluation questions on design, implementation, relevance, sustainability, impact, and gender. It presents conclusions and recommendations that will be useful to the program to ensure the effective accomplishment of indicators and to achieve quality implementation. Agreement Number: FFE-520-2016/008-00 Project Duration: 2017-2021 Implemented by: Project Concern International – (PCI) Evaluation Authored by: Comercializadora Quetzalteca de Bienes y Servicios S.A. (EQSA) DISCLAIMER: The author’s views expressed in this publication do not necessarily reflect the views of the United States Department of Agriculture or the United States Government. 3 Table of Contents Table of Contents.............................................................................................................................. 3 Index of Tables.................................................................................................................................. 5 Index of Illustrations......................................................................................................................... 6 List of Acronyms............................................................................................................................... 7 1. Executive Summary .......................................................................................................... 8 1.1. Introduction....................................................................................................................... 8 1.2 Summary of key indicators ............................................................................................... 8 1.2.1 Improve literacy of school age children. ............................................................................. 8 1.2.2 Improve dietary and health practices................................................................................... 9 1.3 Response to key evaluation questions............................................................................... 9 1.4 Conclusions..................................................................................................................... 10 1.5 Recommendations........................................................................................................... 10 2. Information about the evaluation and study area........................................................ 11 2.1 Description of the beneficiary population....................................................................... 11 2.2 Geographic implementation area .................................................................................... 11 2.3 Summary of the results framework ................................................................................. 12 3. Purpose and Objectives of the Evaluation .................................................................... 12 3.1. Objectives of the evaluation............................................................................................ 12 3.2 Learning Questions......................................................................................................... 13 3.3 Audience for this report .................................................................................................. 13 4. Methodology .................................................................................................................... 13 4.1 Determination of the sample ........................................................................................... 13 4.2 Research techniques........................................................................................................ 14 4.2.1 Quantitative approach........................................................................................................ 14 4.2.2 Qualitative approach.......................................................................................................... 15 4.3 Data quality control......................................................................................................... 15 4.4 Ethical aspects................................................................................................................. 15 4.5 The study team................................................................................................................ 16 4.6 Study Limitations............................................................................................................ 16 5. Evaluation of the Results Framework........................................................................... 16 5.1. Improve literacy of school aged children (McGovern-Dole SO1)............................... 16 5.1.1 Improvement in the quality of literacy instruction (McGovern-Dole 1.1)...................... 20 5.1.2 Improved student attention (McGovern-Dole 1.2).......................................................... 23 5.1.2.1 Short-term reduction of hunger....................................................................................... 24 5.1.4 Improvement of student attendance (McGovern-Dole 1.3) ............................................ 26 5.1.4.1 Reduce health-related absences....................................................................................... 27 5.1.4.2 Increase school enrollment.............................................................................................. 28 5.1.4.3 Increase community understanding of the benefits of education .................................... 30 5.2. Increased use of dietary and health practices (McGovern-Dole SO2) ....................... 31 5.2.1 Improve knowledge of health and hygiene practices (McGovern-Dole 2.1).................. 33 5.2.2 Increase knowledge of safe food preparation and storage practices (McGovern-Dole 2.2)34 5.2.3. Increase access to clean water and sanitation services (McGovern-Dole 2.4)................ 36 5.2.4. Increased access to requisite food prep and storage tools and equipment (McGovern-Dole 2.6) ......................................................................................................................................... 38 5.3. Foundational Results ...................................................................................................... 39 5.4. Responses to key evaluation questions.......................................................................... 39 5.4.1 Design, implementation (strategies), and relevance........................................................ 39 5.4.2 Sustainability................................................................................................................... 40 5.4.3 Outcomes and impact...................................................................................................... 41 5.4.4 Gender............................................................................................................................. 44 5.4.5. Women Empowered........................................................................................................ 47 4 5.4.6. School gardens................................................................................................................ 49 5.4.7. Infrastructure ................................................................................................................... 51 6. Conclusions...................................................................................................................... 51 7. Recommendations........................................................................................................... 54 8. Annexes............................................................................................................................ 56 8.1 School Selection Process................................................................................................. 56 8.1.1 Intervention schools........................................................................................................... 56 8.1.2 Control schools.................................................................................................................. 56 8.1.3 Sample ............................................................................................................................... 57 8.2 Training schedules for staff and ethics considerations.................................................... 57 8.2.1 Ethics considerations......................................................................................................... 58 8.3 Constraints....................................................................................................................... 58 8.3.1 Time and communication................................................................................................ 58 8.3.2 With the schools.............................................................................................................. 58 8.3.3 Knowledge gaps.............................................................................................................. 59 8.4 Additional Analysis of Difference in Differences.......................................................... 59 8.4.1 Reading test..................................................................................................................... 59 8.4.2 School aged children enrolled in schools........................................................................ 60 8.4.3 Health related absences................................................................................................... 60 8.4.4 Parents who can name at least three benefits of primary education................................ 60 8.4.5 School aged children who receive a minimum acceptable diet....................................... 60 8.4.6 Percentage of parents who can identify at least three important health / hygiene practices61 8.4.7 Percentage of mothers (or service providers) who can identify at least three key practices for safe food preparation and storage........................................................................................... 61 8.4.8 Number of schools using an improved water source ...................................................... 61 8.5. Additional Tables and Graphs....................................................................................... 61 8.5.1 Table of literacy achievement by municipality ............................................................... 62 8.5.2 Table of other causes for student absences ..................................................................... 62 8.5.4 Table on Training Received by Teachers in Intervention Schools ................................. 62 8.5.5 Early literacy learning..................................................................................................... 62 8.5.6 Students who are distracted in class, by intervention municipality ............................... 64 8.5.7 Students who indicate that they are hungry, by municipality ......................................... 64 8.5.8 Water by type of sanitary service.................................................................................... 64 8.6 Education Statistics by Intervention Municipality....................................................... 65 8.6.1 Santa Cruz Barillas.......................................................................................................... 65 8.6.2 Cuilco.............................................................................................................................. 65 8.6.3 San Gaspar Ixchil............................................................................................................ 66 8.6.4 San Pedro Necta .............................................................................................................. 66 8.6.5 Santa Eulalia ................................................................................................................... 66 8.6.6 San Pedro Soloma ........................................................................................................... 67 8.6.7 National........................................................................................................................... 67 8.6.8 Huehuetenango................................................................................................................ 67 8.7 DIGEDUCA Reading comprehension tests .................................................................. 67 8.8. Maps related to water supply......................................................................................... 68 8.9 Lessons learned................................................................................................................ 70 1. Regarding organization ................................................................................................... 70 2. Regarding the process..................................................................................................... 71 3. Regarding the community ............................................................................................... 72 8.10 Success Stories................................................................................................................. 72 1. School Gardens............................................................................................................... 73 2. Reading Corner ............................................................................................................... 76 3. IEP (Institutional Educational Project) Model................................................................ 77 5 4. Sanitation services........................................................................................................... 78 5. Virtual Classroom ........................................................................................................... 79 6. Child with a special case ................................................................................................. 80 7. Dining hall for students................................................................................................... 81 9. Related Annexes.............................................................................................................. 82 9.1 Tables ............................................................................................................................. 82 9.1.1 Table of summary of indicators................................................................................... 82 9.1.2 Comparative table of variables for students............................................................... 83 9.1.3 Characterization of the schools.................................................................................... 84 9.2 Summary of results for analysis of work strategies ................................................... 86 9.3 Results of the analysis of digital classrooms ............................................................... 91 9.4 Tests for Statistical Significance .................................................................................. 95 9.5 Field Instruments............................................................................................................ 97 9.6 Evaluation Terms of Reference.................................................................................... 123 9.7 EQSA staff for this consultancy....................................................................................... 2 Index of Tables Table 1. Description of the categories............................................................................................................. 17 Table 2. Most used techniques by teachers..................................................................................................... 21 Table 3. Main changes observed in students................................................................................................... 21 Table 4. Difficulties for implementation of reading strategies ....................................................................... 22 Table 5. Enrolled in intervention schools........................................................................................................ 28 6 Index of Illustrations Illustration 1. Performance of literacy between not achieved and achieved .................................................. 17 Illustration 2. Reading comprehension performance by ethnic group ............................................................ 17 Illustration 3. Achieved performance in intervention schools by municipality............................................... 18 Illustration 4. Achieved performance by school location................................................................................ 19 Illustration 5. Reading comprehension performance/sex Illustration 6. Reading comprehension performance/repeated grade.............................................................. 19 Illustration 7. Principals who comply with school administrative processes................................................. 22 Illustration 8. Curriculum management in intervention and control schools.................................................. 23 Illustration 9. Evaluation of teachers by principals......................................................................................... 23 Illustration 10. Percentage of students who are distracted in class................................................................. 24 Illustration 11. Students who indicate that they are hungry or very hungry ................................................... 24 Illustration 12. Hunger situation in the intervention schools.......................................................................... 25 Illustration 13. Students with regular school attendance................................................................................. 26 Illustration 14. Students with regular school attendance in intervention schools........................................... 26 Illustration 15. Students who state illness as main cause of absence .............................................................. 27 Illustration 16. Types of illness for school absences....................................................................................... 27 Illustration 17. Percentage of school aged children enrolled in school........................................................... 28 Illustration 18. School aged children who leave because of migration........................................................... 29 Illustration 19. Percent of parents who can name at least three benefits of education.................................... 30 Illustration 20. Benefits of primary education mentioned by parents............................................................. 31 Illustration 21. Number of daily meals Illustration 22. Meal times 32 Illustration 23. Students with minimum acceptable diet................................................................................. 32 Illustration 24. Parents who can identify at least three health practices.......................................................... 33 Illustration 25. Cleanliness practices mentioned by parents (%) .................................................................... 34 Illustration 26. Mothers who identify at least three key.................................................................................. 34 Illustration 27. Cleanliness practices mentioned by parents ........................................................................... 35 Illustration 28. Cleanliness practices carried out by collaborating mothers (%)............................................. 35 Illustration 29. Cleanliness practices that collaborating mothers know and carry out in intervention schools ......................................................................................................................................................................... 36 Illustration 30. Schools using an improved water source................................................................................ 36 Illustration 31. Sources of improved water..................................................................................................... 37 Illustration 32. Schools with improved sanitation services............................................................................. 37 Illustration 33. Separate facilities for boys and girls....................................................................................... 38 Illustration 34. Schools with better food preparation and storage equipment................................................. 38 Illustration 35. Satisfaction with Program Interventions ................................................................................ 44 Illustration 36. Kitchen observations .............................................................................................................. 51 7 List of Acronyms ADD Acute Diarrheal Disease ANACAFE National Coffee Association ARIs Acute Respiratory Illness BL Baseline CNB National Base Curriculum COCODE Community Development Council COMUDE Municipal Development Council COMUSAN Municipal Food and Nutrition Security Commission CREA Centers of Education Resources for Learning CRS Catholic Relief Services DASH Huehuetenango Area Health Directorate DD Difference in Differences DIGEDUCA General Direction of Educational Evaluation and Research EDUCAMOS Educate Us: Quality Education and Improved Feeding Focused on Sustainability EGRA Early Grade Reading Assessment ELGI Reading Evaluation for Early Grades EQSA Empresa Quetzalteca de Bienes and Servicios S.A. HDI Human Development Index IARNA Agriculture, Natural Resources, and Environment Institute INSIVUMEH National Institute of Seismology, Volcanology, Meteorology and Hydrology J&A Juárez y Asociados McGovern-Dole McGovern-Dole MOA Ministry of Agriculture MOE Ministry of Education MTE Mid-Term Evaluation PCI Project Concern International SESAN Secretariat for Food and Nutrition Security SIRE Educational Records System TARGETPLAN Name of the technique used for qualitative data collection URL Rafael Landívar University USDA United States Department of Agriculture UVG Universidad del Valle of Guatemala WE Women Empowered 8 1. Executive Summary 1.1. Introduction The Food For Education (FFE) McGovern-Dole (McGovern-Dole) program supported by the United States Department of Agriculture (USDA) is carried out by Project Concern International (PCI). The program is in its third phase (2017 to 2021) and is called EDUCAMOS (Educate Us: Quality Education and Improved Feeding Focused on Sustainability). The program focuses on 294 schools in six municipalities of the department of Huehuetenango: Cuilco, San Gaspar Ixchil, San Pedro Necta, Soloma, Santa Eulalia, and Barillas. The two strategic objectives are: a) Improve the literacy of school age children, and b) Increase the use of health and nutrition practices. This report presents the EDUCAMOS Program mid-term evaluation (MTE), which aimed to evaluate the design, implementation, relevance, sustainability, efficacy, impact, and strategies developed to reach the established targets and to measure progress since the baseline evaluation (BL). For this report, the evaluation team selected a random sample of 30 intervention schools and 30 control schools to administer quantitative and qualitative data collection with a total of 72 teachers, 56 principals, 2,773 students, 184 parents, 171 collaborating mothers, 30 Parent Organizations (OPFs) and 20 Women Empowered (WE) groups. 1.2 Summary of key indicators 1.2.1 Improve literacy of school age children. Reading test. The evaluation tested literacy levels of third grade students to assess the ability to read at a second grade level. Intervention schools achieved 30.8% and control schools achieved 33.3% of students reading at a second grade level, as compared to 32.6% and 27.4%, respectively, found during the baseline. The indigenous populations had the lowest scores (29.6%), while Ladino (mestizo) intervention communities had an achievement rate of 45.8%. The program’s literacy strategy is expected to show better results when the students who are currently in first grade are exposed to the project’s literacy training improvements and are evaluated at endline. Teachers. The principals indicated that 75% of the teachers had improved in their literacy instructions as a result of PCI training. After observing the application of the literacy techniques the teachers learned, the evaluators saw that 95.56% of the teachers were using at least one new technique. In intervention schools, 83.3% of the teachers were present for 90% of the school days (out of 120 possible days) between January and June 2019, a decrease from 100% during the BL. Principals. In intervention schools, 78.57% of the principals demonstrated the application of at least one new technique from PCI trainings. Improvement in school administrative processes showed the highest rate of application (44.1%), followed by improvement in literacy instruction (20.6%), better school hygiene (14.7%) and better organization with the OPF (11.8%). Students. A total of 19.5% of the intervention students and 21.83% of students in the control schools were distracted in the classroom (BL= 27.45%). The analysis between the BL and the MTE showed a decrease of 8% in intervention schools and 8.25% in control schoolsin distracted students. A total of 14.4% of intervention students (BL=8.5%) and 20.10% of control students were hungry during the school day, with a difference in differences (DD) value of -10.7%, which is favorable for the control schools and is interpreted as a result of the implementation of the School Feeding Law (LAE – for its name in Spanish), which came into effect in 2018. In the intervention schools, only 2.3% of the students stated they were “very” hungry. Of the intervention students, 91.69% regularly attended class (BL=97.70%), and of the control students, 87.82% did (BL=95.30%). The most frequent causes were illness and family travel. In the intervention group, 8.70% of students were absent due to illness (BL=7.70%) during the school year, and in the control group, 9 10.20% were, which might be the result of the high incidence of diarrhea and acute respiratory illnesses (ARIs) reported. In the intervention schools, there was an increase of 4.5% in student enrollments compared to the BL, while at the national level, the Ministry of Education (MOE) reported a 3% increase. Parents. In the intervention schools, 92% of parents were able to mention at least three benefits of primary education (BL=84.6%), and 79.7% were in the control schools. Learning to read and write and learning to speak Spanish well were the main benefits of primary education that were mentioned. 1.2.2 Improve dietary and health practices Students. Of the intervention students, 84.2% received a minimum acceptable diet (BL=45.1%), as did 82.90% of those in control schools (BL=29.9%). A minimum acceptable diet consists of consuming at least three meals and four basic food groups a day. This significant increase in the intervention and control groups is a product of the increased diversity of foods promoted by the LAE and the greater access and regularity in school feedings of fruits, vegetables, eggs, dairy products, and other foods. Parents. A total of 90.6% of parents in intervention schools had basic health and hygiene knowledge (BL=93.2%), along with 85.5% (BL=59.20%) of parents in control schools. The significant increase in the control schools was a result of the MOE staff trainings that PCI promoted to transfer knowledge and practices to schools outside the program coverage area. Collaborating mothers. In the intervention schools, 89.40% of mothers were able to identify at least three key practices for safe food preparation (BL= 99.1%), and in the control schools, 86.30% were (BL=79.4%). PCI trained MOE staff and members of OPFs outside the coverage area on safe food preparation and shared the tools used in their trainings. Schools. There was a positive impact in both school groups in terms of improved water sources, which increased to 100% from a BL figure of 86.70% in intervention schools and to 96% from 90% in control schools. Regarding improved sanitation services, the intervention schools did not show significant differences between the BL and MTE, with 90% and 87.5%, respectively. However, control schools deteriorated from 96.7% at the BL to 84% at MTE. Regarding food preparation and storage infrastructure, intervention schools improved from 61.5% at BL to 75% at MTE, while control schools deteriorated from 43.8% to 37.5%. 1.3 Response to key evaluation questions The interviews with MOE staff, Municipal Councils, PCI staff, principals, teachers, students, OPFs, WE groups, and others provided the following results for three of the five learning objectives1 : Design, implementation and relevance. The PCI model for bilingual literacy education takes the local context into account with the use of materials in three languages (Spanish, Mam, and Q´anjob´al) and the training of teachers from key early grades in literacy instruction and for school feedings, the preparation of culturally relevant menus, and the strengthening of lead mothers’ skills. The most effective strategies for achieving program results in line with the results framework are trainings for the MOE, principals, and OPFs, within and outside of the coverage area, the distribution of materials in Mayan languages, support for school infrastructure, food safety practices, the use of the Cleanliness Traffic Light2 1 Learning objectives around outcomes, impact and literacy are addressed in the quantitative results sections of this report. 2 An instrument to monitor daily cleaning and hygiene in different areas throughout the school, which is applied by lead mothers and students. It is located in a visible place throughout the school. 10 and the Information Corner3 , women’s empowerment, strengthening of the OPFs, the educational and productive use of gardens, schools’ reading fairs and camps, and coordination with McGovern-Dole project implementing agencies and other national entities. The results where Educamos was least effective in achieving desired results included the promotion of teacher attendance, the cascading training of lead mothers to collaborating mothers, the involvement of a greater number of parents in the topics of hygiene, health, and food preparation, follow-up for better use of reading spaces, maintenance of school infrastructure, incorporation into Municipal Councils, and the dissemination of successes, achievements, and good practices to the educational community and outside the program. Sustainability. To ensure the continuity of program activities, PCI maintains close collaboration with: a) members of OPFs to strengthen their skills; b) government agencies, including the MOE with which materials are shared and trainings are provided to extend the coverage area; and c) decision making opportunities at the municipal level. The transfer of the school feeding program creates various challenges because: a) the OPFs are not able to take ownership because of low literacy rates, a lack of time, and a fear of making mistakes; b) government institutions are in the process of incorporation to promote the LAE; and c) the principals lack information on family agriculture. Gender. PCI promotes activities to achieve gender equity. These include support to girls so that they don’t drop out of school, participatory techniques with teachers to engage both girls and boys (e.g. school governments), and the empowerment of women to increase participation in school and community. Nevertheless, women face constraints to their participation in the groups, such as: little support for women’s organizations, a lack of experience in financial topics, fear of failure and of complying with rules, lack of time, machismo, lack of motivation, and low self-esteem. 1.4 Conclusions 1. PCI, in coordination with the MOE, MDG implementing partners, and other organizations, promotes the operationalization of the LAE, which has facilitated the improvement of students’ diets, promoted healthy environments, created contextualized menus for each region, improved the skills of the OPFs, and increased health, hygiene, and food preparation knowledge and practices. PCI trained MOE staff and shared tools so that their impact transcended the intervention area, even though they did not disseminate their successes and results with key actors, including Municipal Councils. 2. The results for basic hygiene and food preparation knowledge showed high standards and adoption at school and at home, even in the control schools, due to the support PCI provided, although it is important to reach more participants given the high turnover rate. 3. The literacy strategy did not show impact with the third-grade students who were evaluated. Nevertheless, there was improvement in five of the six intervention municipalities when compared to the BL. 4 1.5 Recommendations 1. Generate a training of trainers plan to where community members trained by PCI can cascade traing to other community members, to continue to strengthen local skills in nutrition, hygiene, organization, food preparation, and in general, the topics supported by PCI. 3 An instrument that allows PCI staff, principals and OPF members in each school to post project information on key indicators, in order to allow all stakeholders to measure and analyze progress towards achieving project targets. 4 Because sampling was done at the project level, results disaggregated at the municipality level are not statistically significant and may not be representative of all schools in those municipalities. These results, and all results related to municipalities in this report, are illustrative. 11 2. Strengthen the strategy for the implementation of bilingual literacy through the use of the techniques that were disseminated, an increase in pedagogical accompaniment, the use of reading corners and school libraries, and the preparation of strategies for multi-grade or single teacher schools, taking into account that the results will be seen over time. To measure progress, an internal mechanism can be used to measure reading comprehension beginning in first grade. 3. Communicate successes to key actorsto promote appreciation, the adoption of good practices, investments in program activities (infrastructure), and sustainability. 1.6 Study Limitations Among several study limitations noted in the methodology section of this report, it is most important to point out that many of the control schools have been exposed to EDUCAMOS interventions as part of the project’s sustainability strategy. This has led to the contamination of interventions, particularly around health and hygiene training, and food preparation and storage training and limits the ability of these schools to serve as a proper comparison group to identify causality. It is noted throughout the narrative where the control schools were contaminated with the intervention. 2. Information about the evaluation and study area 2.1 Description of the beneficiary population EDUCAMOS is implemented in six municipalities in the department of Huehuetenango, including: San Pedro Necta, San Gaspar Ixchil and Cuilco in the Mam area; and Soloma, Santa Eulalia and Barillas in the Q'anjob'al area, located in the northwestern region of Guatemala. Target municipalities have not changed throughout the duration of project implementation. In San Gaspar Ixchil, the project covers 100% of the schools in the municipality (n = 12), while in other municipalities the project is prioritizing schools with poor educational indicators and where stakeholders are willing to engage in project activities. The first phase of the project was implemented from 2010 to 2013 in 275 schools, benefiting 30,266 students. The second phase was from 2013 to 2016, in 294 schools with 38,020 students, and the current third phase of the project started in 2017 and is scheduled to end in 2021, targeting 294 schools and 38,387 students (51.56% boys and 48.44% girls). Currently the project is working with a total of 1,451 teachers (46.52% men and 53.48% women), 1,196 collaborating mothers and 1,673 members of Women Empowered (WE) groups. A total of 331 lead mothers support hygiene and nutrition activities and 59 volunteers support educational activities. In each school, the program works closely with Parents’ Organization (OPF), which includes five members. The beneficiary population is centered on the 294 schools in the department of Huehuetenango, where the population is mainly Mayan from the Mam (San Pedro Necta, San Gaspar Ixchil, Cuilco) and Q´anjob´al (Soloma, Santa Eulalia, Barillas) ethnic groups. A total of 38,387 students (51.56% boys, 48.44% girls), 1,451 teachers (46.52% men, 53.48% women), 1,196 collaborating mothers, and 1,673 members of WE groups compose the population. A total of 331 lead mothers support hygiene and nutrition activities, and 59 volunteers support educational activities. In each school, the program works with an OPF, which is composed of five members in the roles of president, secretary, treasurer, and two members at large. They each serve a four-year term. 2.2 Geographic implementation area The EDUCAMOS Program covers six municipalities in the department of Huehuetenango. Huehuetenango is the fifth largest department of Guatemala, and the departmental capital is located 269 km from Guatemala City. According to the National Statistics Institute (INE), the population for 2019 was 12 estimated at 1,381,969 people. The department of Huehuetenango receives the second largest amount of remittances5 in the country: in 2018, it received USD $810.8 million from the United States. In San Andrés Cuilco municipality, the majority of the population (82.54%) are farmers, and the remainder work in service provision or community activities, construction, and formal and informal commercial activities. In San Gaspar Ixchil, the economically active population is mainly composed of men dedicated to farming activities, especially for home consumption, given the lack of technology and low yields due to the fact that the soil is not apt for agriculture. Some areas produce coffee, which is marketed via intermediaries. In the municipality of San Pedro Necta, corn, beans, and coffee are the major crops, and parchment coffee is sold through the National Coffee Association (ANACAFE). Soloma has corn, beans, coffee, and avocado crops, and parents often migrate from September to December to work on coffee plantations in Mexico. The main crops in Santa Cruz Barillas are coffee, cardamom, sugar cane, oranges, bananas, plantain, and pineapples. Coffee and cardamom are the most stable and most marketable crops. Livestock activities include cattle, pigs, sheep, and goats. In Santa Eulalia, employment revolves around coffee and cardamom, with some livestock activities (sheep, cattle, pigs, goats, and poultry). They also weave bags, typical blouses and shirts, and work in carpentry, iron works, furniture making, and other woodworking. 2.3 Summary of the results framework The EDUCAMOS Program has two strategic objectives; the first seeks to improve literacy in school aged children by means of: a) Improved quality of literacy instruction, improved access to school and didactic materials, and increased skills of teachers and principals; b) Improved students’ attention through hunger reduction and increased access to foods; and c) Improved student attendance through increased economic incentives for attending school, reduction of health-related absences, improvedschool infrastructure, increased school enrollment, and increased community understanding of the benefits of education. The second objective seeks to increase the use of dietary and health practices through: a) Improved knowledge of these practices; b) Increased knowledge of safe food preparation and storage practices; and c) Increased access to clean water and sanitation services. To achieve its strategic objectives, the program’s foundational results are to: a) Improve the capacity of government institutions through capacity building and coordination, b) Improve policy and regulatory framework, and c) Increase engagement of local organizations and community groups focused on training for the parent councils. 3. Purpose and Objectives of the Evaluation 3.1. Objectives of the evaluation a. Evaluate the relevance, efficacy, sustainability, and focus of EDUCAMOS program strategies in the intervention areas. b. Determine progress since the BL to achieve the established targets, including an assessment of the effectiveness, efficiency, and timeliness of the interventions. 5 Source of data on remittances: Guatemalan Development Foundation. 13 c. Measure program impact against key impact indicators around literacy, student attention and attendance, nutrition, teacher attendance, parent knowledge of health, hygiene and food preparation practices and school infrastructure. d. Provide recommendations to strengthen the performance, efficiency, sustainability, and other aspects for planning the final evaluation. e. Document lessons learned. 3.2 Learning Questions The main prioritized topics are listed below, and the full list of learning priorities are listed in the evaluation Terms of Reference in Annex 9.6 a. Design, implementation, and relevance: Relationship between the results framework, strategies, and results achieved, based on the local context and needs of the participants, in conjunction with Juárez & Asociados (J&A) and the Universidad del Valle of Guatemala (UVG). b. Sustainability: Relationships among government agencies, partners, and local actors for the transfer of the school feeding programs, current efforts, and planned measures during the remainder of the program, to ensure the continuity of program activities. c. Outcomes and impact: Analysis of the measurement of literacy and child nutrition, as well as other intermediate results presented in the results framework and in the indicator table. d. Gender: Evaluation of gender equity and dynamics, strategies for equitable male/female participation, the approach for addressing gender constraints, and increased decision-making power by members of WE groups in the home and community. The evaluators focused on using qualitative data to respond to the learning questions in this evaluation. To develop effective qualitative tools that would respond to the learning questions, the evaluators carried out a detailed review of the BL results and held meetings with the PCI management team to determine the qualitative topics for evaluation. 3.3 Audience for this report The report was prepared for high level staff in PCI, including at the International Office (IO) and especially for the EDUCAMOS program staff and the decision-makers responsible for achieving the expected impact. USDA is a key audience, as a cooperating institution that is interested in understanding the current, mid-term status of this program with an impartial report from a perspective external to PCI. Additionally, key audience for sharing the MTE results include the Departmental School Feeding Working Group, the Ministry of Education at the national and departmental level, including DIGEDUCA (General Directorate of Educational Evaluation and Research) with whom PCI will share datasets. At the municipal level, audiences will include Municipal Councils, School Feeding Working Groups, Parents’ Organizations and school principals. 4. Methodology The MTE for the EDUCAMOS program used a quasi-experimental, mixed methods impact evaluation. Details of this methodology are described below. 4.1 Determination of the sample The project’s Mid-Term Evaluation (MTE), just like the baseline study conducted at the beginning of the project, includes schools where the EDUCAMOS is conducting direct actions (known as “intervention schools”). In addition, the evaluation included a sample of schools not receiving direct project activities, titled 14 control schools, in order to provide a counterfactual for the evaluation. The number of schools in the sample represents 10% of the intervention schools (n= 30), which is the same sample size used during the baseline study. The number of control schools was also 30. Intervention schools were randomly selected. For comparison purposes, PCI worked closely with MINEDUC to select a sample of control schools located in the same municipalities6 as the intervention schools, whenever possible, in order to ensure similar characteristics between the two groups in terms of geographic location, literacy levels, chronic malnutrition, Human Development Index (HDI) and language. A two-stage cluster sampling procedure was used to select the sample. To identify the number of schools per municipality in the intervention area, a percentage method was used based on the number of students per municipality. Distribution of the 30 intervention schools in the sample was as follows: Barillas (7), Cuilco (7), Santa Eulalia (6), Soloma (5) and San Gaspar Ixchil (1). Intervention schools were then randomly selected. The control schools were selected jointly by PCI and the departmental MOE office in the city of Huehuetenango. They were chosen based on the similarity of characteristics in terms of language, geographic location, illiteracy rate, malnutrition indices, development index, and other elements considered relevant to ensure their comparability. The selection process used for each type of school to obtain the list of 30 intervention schools is described in Annex 8.1 and the sample table in Annex 8.1.3. The literacy test was applied to all third grade students in the selected schools. Fourth-Sixth grade students as well as parents were randomly selected to respondent to other survey instruments, with the sample size based on a 95% confidence level and 5% confidence interval. Evaluators interviewed one or two teachers per school, who are members of the school feeding commission, and the school principal. The school collaborating mother who was at the school the day of data collection to prepare school meals was also surveyed. 4.2 Research techniques 4.2.1 Quantitative approach Structured surveys were used to measure 17 results indicators. The Mobenzi mobile data collection platform was used to collect quantitative survey data. The same design, methodology, and instruments from the BL were used. The target population was parents, collaborating mothers, and students from fourth to sixth grade for the minimum acceptable diet, students who had graduated from second grade for the reading test, and first grade students for the observations of attention in class. The study used the SPSS version 25 statistical program to process and analyze data, using the following statistical techniques as established by PCI and Empresa Quetzalteca de Bienes and Servicios S.A. (EQSA) in the coordination meetings: i. Simple frequency calculation: Counting the number of times a variable is repeated in data distribution. ii. Preparation of cross frequency tables: Tables that show the repetition of one variable in relation to another study variable. iii. T tests (means/proportions) were carried out to measure differences of significance between variables (using 0.05, which means an accuracy of 95%). By default, the SPSS software conducts the “Bonferroni correction method” which consists of dividing the level of significance α by the number of two-by-two comparisons conducted. This correction ensures that the probability of obtaining at least one false positive among all comparisons (family-wise error rate) is ≤α iv. Analysis of DD: This econometric method for quasi-experimental impact evaluations was used to measure the effect of the PCI intervention compared to the non-intervention group over time. 6 Except for San Gaspar Ixchil, where EDUCAMOS is covering 100% of schools, so the control school was selected from San Sebastián H., a nearby municipality which shares similar characteristics with San Gaspar Ixchil. 15 Reading test. For comparison purposes, the literacy test used during the baseline was also used during this evaluation. It was designed by Juarez & Associates, the literacy and education partner on Educamos, in collaboration with MINEDUC. The standardized MOE literacy test designed for second grade students was used and is only available in Spanish. The test has not been implemented by MINEDUC in other parts of the country, thus results are not comparable with other government-led literacy tests. The test was administered to third grade students to comply with the literacy indicator that requires students to complete two primary school grades. Per the official protocol, the test consisted of three different booklets to prevent students from copying from each other, along with an explanation with examples on how to fill out the test, and the students sat in alphabetical order with the booklets alternating within the classroom and among schools. It was forbidden to take pictures of the test. J&A was responsible for the training and supervision of the test administrations, and the results were shared with the MOE. 4.2.2 Qualitative approach Data was collected from WE groups and OPFs in focus groups using a guide with questions about relevance, performance, efficacy, and sustainability. The TARGETPLAN qualitative methodology was used, which involves discussing a central topic and allowing participants to analyze and propose ideas and reflections that are written down on flashcards and classified by the moderator to encourage further discussion and reaching consensus and conclusions. The flashcards are structured on a flipchart attached to the wall or to the board for further analysis and discussion. . Following this methodology, the main tendencies were transformed into frequencies, grouping the most common opinions to avoid bias when presenting only representative information. In-depth interviews were carried out to collect individual opinions from principals, members of Municipal Councils, Technical Administrative Coordinators (CTA for their name in Spanish), and J&A staff, using a questionnaire. The team conducted observations to record variables for food preparation and consumption, food storage in storage rooms, improved sanitation installations, and water sources. Students in first grade were also observed to determine how many were distracted and for which reasons, both before and after receiving a school meal. 4.3 Data quality control Staff training. Quality begins with trained staff who are fully equipped to use the evaluation instruments and who have a clear understanding of them. It is also useful to have interpreters for the Q´anjob´al and Mam languages to ensure effective communication, avoid interpretation biases or erroneous interpretation, and ensure effective communication of concepts. Efficient electronic data management. Maximum and minimum ranges were defined for numeric values in the data collection forms in Mobenzi (e.g., from zero to seven for food groups in the minimum acceptable diet indicator), which reduced the likelihood of entering incorrect data. Data integrity was analyzed and validated in SPPS. Double review in tabulation of qualitative data. The results from focus groups with the OPFs and WE groups were tabulated, and a second review was carried out to ensure correct entry. 4.4 Ethical aspects To guarantee an ethical data collection process and to ensure the confidentiality of data management, the data collectors read an informed consent statement to the respondents, giving them the right to decide whether or not to participate, fulfilling their right to safety and avoiding physical or psychological damage. They had the right to be informed about the study objectives and the right to privacy related to the information that they 16 shared. All participants gave informed consent before beginning the study, and they did not receive any benefits or payment for their participation in the evaluation. 4.5 The study team Four teams worked simultaneously in the intervention and control schools. Each team was made up of three people who carried out all the activities in each school, and there were a total of 13 people in the field. To get a clear understanding of the study objectives and the skills needed to compile the data, the evaluation staff were trained in theoretical and ethical aspects, and they carried out exercises to introduce the study to the principal, practice the observation process, focus groups, and surveys, administer the informed consent form, and carry out interviews. The exercise ended with the validation of instruments prior to starting field work. 4.6 Study Limitations 1. In the process of obtaining MOE approval for the administration of the reading test, setbacks made it clear that it was important to allow sufficient lead time before beginning the evaluation. 2. Poor communication between the MOE and principals meant principals were not consistently informed in a timely manner about the visit, so visits had to be rescheduled in seven schools. 3. The teams faced resistance from the Guatemalan Education Worker’s Union (STEG) around providing information about their schools, especially in the control group, because they assumed there might be negative consequences if the results were poor. 4. The geographic location of the schools influenced the results obtained. Generally, the more rural the school, 7 the poorer the literacy learning results. For that reason, the urban John F. Kennedy control school in Soloma was excluded from the analysis because it would have distorted the literacy results, given that all the intervention schools in the MTE were rural. 5. Some teachers in totally bilingual areas spoke only Spanish, which made it difficult for them to initiate strong communication with their students. 6. Many of the control schools have been exposed to EDUCAMOS interventions as part of the project’s sustainability strategy. This has led to the contamination of interventions, particularly around health and hygiene training, and food preparation and storage training and limits the ability of these schools to serve as a proper comparison group to identify causality. It is noted throughout the narrative where the control schools were contaminated with the intervention. 7. Implementation of LAE has improved food availability and variety in both intervention and control schools, making it difficult to compare schools with and without a school feeding program, and likely impacted results around nutrition, and possibly other indicators. 8. The literacy test used for this evaluation is only available in Spanish, whereas EDUCAMOS promotes a bilingual literacy training approach. This could lead to lower literacy levels where Spanish is not the language of instruction. 5. Evaluation of the Results Framework 5.1. Improve literacy of school aged children (McGovern-Dole SO1) Percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text The reading comprehension test was designed for students who have completed second grade, so it was applied to the third grade students. Because the test was provided by the MOE and was only available in Spanish, it was administered only in Spanish, even though the EDUCAMOS literacy strategy focuses on bilingual 7 According to the 2002 Census, urban is defined as the population that resides in cities, towns, and department or municipal seats. Rural areas are limited in economic growth, access roads, and receive fewer services and influences from the urban area. 17 education. The team completed 860 tests: 509 with students in intervention schools (59.2%) and 351 with students in control schools (40.8%). Of these, 45.2% were boys and 54.8% girls. A total of 85.2% stated that they were Mayan, 12.3% mestizos (Ladinos), 1.8% Garífunas or Xincas, and 0.63%, did not respond. A total of 89.3% were in third grade for the first time, and 10.8% were repeating the grade. The average age was 10.1 years. See Annex 8.7. 8 Student performance was measured according to the following parameters: Table 1. Description of the categories Degree of Achievement Performance Interpretation ACHIEVED Excellent In this level, students performed above the “Satisfactory” level. They showed adequate and superior proficiency in the skills, abilities, and knowledge expected for the grade. Satisfactory In this level, students were proficient in the skills, abilities, and knowledge required for the grade. It also included skills from the previous level. NOT ACHIEVED Should Improve In this level, the students performed at a level close to, but inferior to “Satisfactory.” Although they demonstrated abilities, skills, and knowledge, their proficiency is less than what is expected for the grade. Unsatisfactory In this level, the students performed at a level close to, and inferior to that of “Should Improve”. They lack proficiency in the skills, abilities, and knowledge that they should develop or exercise in the grade. Source: MOE testing parameters. The comparison of results between the two evaluations showed that the intervention schools performed at a similar level to what was seen in the BL, while the control schools showed an increase of 3.8 percentage points for students who achieved an “Excellent performance" and 2.2 percentage points in students who achieved a “Satisfactory” performance. There were no significant differences between the results of the BL and the MTE. See the graph in Annex 8.5.1. Illustration 1. Performance of literacy between not achieved and achieved Illustration 2. Reading comprehension performance by ethnic group 8 For comparison purposes, the MTE used the same test used during baseline, which was designed by Juárez & Associates and MINEDUC for third grade students. MINEDUC has not applied this test in other areas of the country, thus there are no comparable literacy results with other parts of Guatemala. 32.6% 27.4% 30.8% 33.3% 0% 10% 20% 30% 40% Baseline Midterm Intervention Control 18 The study showed that 30.8% of the students in the intervention schools reached the “Achieved” level for their grade level, while in the control schools, 33.3% reached “Achieved.” These results did not present statistically significant differences between the two groups (p= 0.442). Regarding the two main ethnic groups, the Mayan students had a lower percentage of achievement in both groups of schools, which was not surprising given that their native language is not Spanish and that they had difficulties understanding the test. The Ladino students had a higher percentage of achievement in both intervention and control schools,9 though the analysis did not show significant differences between the two ethnic groups in the two evaluations. An association was found between the percentage of achievement and the municipality of the school: larger percentages of children who spoke Spanish well and who were closer to urban centers were associated with higher achievement rates. 10 The municipalities of San Pedro Soloma and Cuilco had the highest percentage of achievement. Cuilco had the highest proportion of Ladino population, and as was mentioned before, had the highest achievement levels. The principals interviewed felt that students living in urban areas with access to television, internet, and other communication resources learned to speak Spanish faster, while students in the hard to access areas without these “connectivity” benefits were slower to learn Spanish, impairing their ability to learn to read in Spanish (see the table in Annex 8.5.2.). There was no difference between boys and girls in the tests, and both reached similar levels of achievement. Illustration 3. Achieved performance in intervention schools by municipality The percentage of achievement increased from the BL in the intervention schools in the municipalities of San Pedro Necta (+3.8 percentage points), San Gaspar Ixchil (+4.9 percentage points), Cuilco (+7.3 percentage points), Santa Eulalia (+22.7 percentage points), and Santa Cruz Barillas (+4.5 percentage points). San Pedro Soloma’s achievement rate decreased by -26.4 percentage points, likely because in the BL, the randomly selected school in Cotzán was located in the municipal seat, an urban area, while no urban schools were selected in the MTE. It should be noted that while the evaluator feels this data provides important context, sampling was not done at the municipal level, and the comparisons between municipalities are not statistically significant. Three categories were used to describe the location of each school and community where the test was administered: a) Urban: located in municipal seats, b) Semirural: located in a rural area but with access to 9 p=0.001 10 Sampling not stratified at the municipality level, these results are not statistically significant. 70.4% 54.2% 69.2% 51.1% 29.6% 45.8% 30.8% 48.9% Maya Latin Maya Latin Intervention Control Did not achieve Achieved 3.8% 21.4% 38.5% 68.7% 6.6% 19.6% 7.7% 26.3% 45.8% 42.3% 29.2% 24.0% 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% San Pedro Necta San Gaspar Ixchil Cuilco Soloma Santa Eulalia Barillas Baseline Midterm 19 communication services such as cell phones, telephones, cable, internet, and year round access to the municipal seat, and c) Rural: located in a rural area, with difficult access due to road conditions, and with limited access to communication services. Illustration 4. Achieved performance by school location A statistically significant difference was found between the percent of achievement and the location of the school11. Reading comprehension results were higher in the less rural areas. The principals confirmed this result when they stated that students who live in urban areas and have access to television, internet, and other communication resources learn to speak and read Spanish faster, while in the rural areas, the situation is more precarious, and reading challenges are greater. There was no significant difference between boys and girls in the tests. The number of times students repeated a grade was analyzed in relation to the achievement percentage. Two categories were established for this analysis: Students who had never repeated the grade and those who had repeated the grade. The achievement percentage was lowest for repeat students in the control schools. Illustration 5. Reading comprehension performance/sex Illustration 6. Reading comprehension performance/repeated grade The DD analysis showed a positive increase in the control schools, while the overall results in the intervention schools were stable, given that the MTE and BL had the same results confidence intervals (26.9% - 35%). Significant differences between the two evaluations were only found in intervention schools in the municipality of Santa Eulalia, and no other significant differences were found between the two groups in the MTE. See the Significance Tests table in Annex 9.4. While significant differences in achievement rates in this short term analysis were not seen between intervention schools and control schools, the team noted improvements in the schools with PCI interventions. The literacy program is relatively new, having begun its first phase in 2018 with the preparation and adaptation of educational materials and team training. The UVG diploma program was carried out in only one cohort in the 2019 fiscal year, which covered about 10% of the teachers. 11 p=0.000 69.8% 68.6% 67.9% 65.6% 30.2% 31.4% 32.1% 34.4% Male Female Male Female Intervention Control Did not achieve Achieved 70.6% 57.1% 65.3% 77.5% 29.4% 42.9% 34.7% 22.5% No Repeat Repeat No Repeat Repeat Intervention Control Did not Achieve Achieved 27.7% 39.0% 25.0% 44.9% 67.4% 72.3% 61.0% 75.0% 55.1% 32.6% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Rural Semirural Urban Rural Semirural Urban Intervention Control Achieved Did Not Achieve 20 Given that literacy is a process that requires time and effort to demonstrate impact, it is possible that the program will achieve better results when the students who are now in first grade reach third grade and will have immersed in the literacy materials and teaching. Based on the data reported by PCI, 157 pre-school and first through third grade teachers are in the process of graduating from the Literacy in Bilingual and Intercultural Environments Diploma program in the UVG, and the final evaluation will measure the results. 5.1.1 Improvement in the quality of literacy instruction (McGovern-Dole 1.1) Percent of teachers who demonstrate improved literacy instruction as identified by supervisors, mentors, or coaches The team consulted with principals (the teachers’ direct supervisors) about the training received from PCI and its application by first through third grade teachers in terms of the use of the linguistic profile (in bilingual schools), the use of materials and resources provided by PCI, and the use of literacy strategies (phonemic awareness, alphabetic principle, fluidity, vocabulary, reading comprehension, and creative writing) to determine this indicator. 12 According to the school principals, 75% of the teachers demonstrated an improvement in literacy instruction as a result the training received from PCI, while the remaining 25% were still in the process of improvement. Since this indicator was not measured at BL, it is difficult to determine performance of this indicator in relation to the project interventions. It is worth mentioning that the project surprassed the midterm target of 40%. 5.1.1.1. More Consistent Teacher Attendance (McGovern-Dole 1.1.1) Percentage of schools in which at least 80% of the teachers are present 90% of the school year The teams could not verify teacher attendance records due to orders from the STEG in the control schools, so this analysis was only completed in the intervention schools. Teams verified attendance from January to June, which represented 120 class days from a total of 204 class days from January to October, according to the 2019 MOE school calendar. The result is that 83.33% of the teachers were present from 90% to 100% of the school days, while the remaining 16.67% were present from 80% to 89% of the school days. Teachers reported that 87% of the time, their absences from school were due to commitments with STEG or other administrative situations, paperwork and citations from the CTA, or other activities in the education supervision offices, which are excused absences according to article 17.3 of their collective agreement. Another 13% of the absences were for personal reasons such as illness or they were given permissions, in which case they notified the principal. 5.1.1.2. Increased skills and knowledge of teachers (McGovern-Dole 1.1.4) Number of teachers/educators/teaching assistants in target schools who demonstrate use of new and quality teaching techniques or tools To determine this indicator, the team observed 45 randomly selected teachers in all 30 sampled intervention schools during a “master class.”13 The project attempted to observe two teachers per school, but in some schools was only able to observe one teacher. The evaluator used the list of techniques provided by the project in order mark those techniques that were used by the teacher during the master class. 12 The evaluator was only able to interview 28 of the 30 principles in intervention schools. 13 A master class is when the teacher verbally transmits knowledge to the group of students in question. It does not use any kind of dynamics or student-teacher interrelation. The strategy of the master class involves an expository technique that occupies the entire class time, in this case the entire observation time. 21 The evaluation found that 95.5% of the teachers used new, quality literacy instruction techniques or tools. This percent was used to calculate the numeric indicator result. Based on 1,030 teachers trained it is estimated that 984 demonstrated the use of new and quality teaching techniques or tools. No differences were found between the responses of the teachers who received the diploma course (6.7% of the teachers interviewed) and those who only received pedagogical accompaniment and training in the use of the resources provided by PCI. The most common new techniques used are Big Books and the flipcharts. Big Books are large storybooks which allow group reading in the classroom. Educamos distributed two Big Books series targeting schoolchildren in the first levels of primary school. Big Books series included: "Leo y me divierto" (I read and have fun) with 8 titles in Spanish, Mam and Q'anjob'al; and "Leo y me cuido" (I read and take care of myself) with 36 titles in Spanish, Mam and Q'anjob'al. As shown in the table below, the rest of the techniques were used only by a minority of teachers. Table 2. Most used techniques by teachers Techniques used Number of Responses Percentage Use of Big Books 32 71.10% Use of flip chart 27 60.00% Use of educational games 13 28.90% Improvement in reading variation (MOE) 9 20.00% Use of the CREA14 educational resources 8 17.80% Use of virtual classroom 5 11.10% Use of reading corners 4 8.90% Use of guidelines for reading spaces 2 4.40% Use of linguistic profile 1 2.20% Source: Teacher Interviews The evaluators asked the principals to describe the changes that they noticed in students as a result of the application of new techniques, and as shown below increased participation, reading comprehension and increased learning are the most common areas of improvement. Table 3. Main changes observed in students Area of improvement Number of responses Percentage of teachers Associated technique Increased participation 22 51.2% Reading comprehension Reading comprehension 19 44.2% Reading comprehension Increased learning 16 37.2% Phonemic awareness, alphabetic principle, fluidity, vocabulary, reading comprehension, and creative writing Improved vocabulary 8 18.6% Phonemic awareness Improved writing 8 18.6% Phonemic awareness, alphabetic principle Interest in class/reading 7 16.3% Reading comprehension Development of creativity 5 11.6% Creative writing Fluidity in reading 4 9.3% Fluidity Total responses 89 Source: Field work See table of training received by teachers in intervention schools in Annex 8.5.4 14 The Educational Learning Resource Centers (CREA) are a technique promoted by PCI and it refers to the place or environment within the school where teachers use these resources or materials with their students. CREA being a technique promoted by PCI, it is not being implemented in control schools. 22 The teachers mentioned the following difficulties in implementing the reading strategy and the application of new techniques: Table 4. Difficulties for implementation of reading strategies Main difficulties Percentage Lack of understanding by the children (especially in the most rural areas with predominant native language) 41.67% Lack of retention by children, they are shy, little participation 14.58% Don’t always have support materials 12.50% Lack of books for reading at home, don’t read 30 minutes at home 8.33% Poor attendance in class 8.33% Repeating students whose age is greater than that of the other students, who don’t make progress and hold back the learning process for the rest of the class 6.25% Having responsibility for too many grades or being the only teacher in the school 4.17% Teacher not bilingual in a bilingual community 4.17% Source: Field work. Based on 48 teachers. 5.1.1.3. Increased skills and knowledge of principals (McGovern-Dole 1.1.5) Number of principals who demonstrate the use of new techniques or tools The team interviewed 56 principals, 28 from control schools and 28 from intervention schools, to evaluate the use of new or improved techniques or tools. The evaluators verified documents and records such as files, books used, and minutes in the following areas: a) School management and leadership, b) School administrative processes, c) School organization, d) Curriculum management, and d) Evaluation of teachers to demonstrate the use of these techniques. In the intervention schools, 22 principals (78.57%) demonstrated the application of at least one new technique in any of the areas evaluated, as a result of the training received from PCI. Tranlsating this result to the entire project estimates that 231 of the 294 trained principles demonstrated new techniques or tools. In control schools, only 46.4% of principles demonstrated new or improved techniques or tools. When broken down by area of improvement for intervention schools, school administrative processes showed the greatest improvement (44.1%), which included annual planning, personnel files, use of an account book for purchases, inventory of furnishings, use of the Educational Records System (SIRE, for its name in Spanish), and transparency mural. School management and leadership: To evaluate the leadership of the principals and their capacity for management and organization, the team determined whether parents attended school meetings. In 100% of the intervention schools, parents attended school meetings, and in the control schools, a high percent of parents did (96.4%). Approximately nine of every ten intervention schools had an improvement plan, compared to 82.1% of the control schools. Illustration 7. Principals who comply with school administrative processes The principals of the intervention schools conducted the administrative procedures required by the MOE, and this was verified via school documents. This was less common in the control schools, especially for the SIRE, the system the MOE implements to record the main educational 89% 93% 96% 93% 68% 89% 100% 100% 100% 100% 100% 96% 50% 60% 70% 80% 90% 100% Annual Planning Maintain personnel files Purchase Records Supply Inventory Utilize SIRE Transparency Mural Control Intervención 23 indicators (for example, effective class days). Illustration 8. Curriculum management in intervention and control schools Curriculum management, as stipulated by the MOE, was carried out in both groups of schools, with only a slightly higher degree of the application of the reading model in bilingual environments by the intervention schools. PCI’s training for principals enabled them to work with their teachers to improve student literacy. The table below presents a summary of the improvement areas evaluated by the principals. Illustration 9. Evaluation of teachers by principals The principals expressed satisfaction with the training received from PCI, especially as they confirmed that they can now better optimize resources, apply techniques for conflict resolution, and call more frequent meetings with staff and parents, and that they have greater administrative management and control abilities, including filling out the school feeding documents. The MOE indicated that PCI’s support is relevant for the participation of principals in administration, school management, leadership for quality education, school administrative processes, curriculum management, and pedagogical accompaniment. 5.1.2 Improved student attention (McGovern-Dole 1.2) Percentage of students who are off task The methodology used two classroom observations were carried out at specific intervals: first at the beginning of class, and second when students returned from recess, after they received food. The two measurements were averaged so that the effect of the time of measurement was minimized. Additionally, the team only carried out observations in lecture classes to create equal conditions for each observation and because lecture classes are where students need to pay close attention to the teacher. 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0% Linguistic Profile Phonological Awareness Alfabet Teaching Flow Vocabulary Reading comprehension Creative Writing Intervention Control 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% CNB Planning Utilize CREA Report Cards Bilingual learning environment Control Intervención 24 Disagregated by municipality, the data show that San Pedro Soloma had the highest percentage of students off task (38.24%), and Santa Eulalia had the lowest number of distracted students (9.13%). The intervention schools generally had a lower percentage of distracted students. Althought not statistically significant, comparisons by municipality are useful for project planning purposes. See the graph in Annex 8.5.6. The percent of distracted students dropped from 27.45% in the BL to 19.46% in the MTE, an 8.25 percentage point decrease. The control schools dropped eight percentage points in that same period. Students looking around without paying attention in class were observed the most frequently, followed by students talking to a classmate, which meant that both were distracted from their school tasks. Increased availability of food in intervention and control schools due to LAE implementation could have been a factor in the reduction of students’ distraction. Illustration 10. Percentage of students who are distracted in class A moderate, direct correlation was found between the percentage of distracted students and the percentage of students who said they were hungry. The correlation coefficient was higher in the control schools (0.6205) than in the intervention schools(0.4897), reflecting that the implementation of school feeding in control schools decreased the percentage of distracted students. The correlation coefficient increased significantly at the municipal level (0.7885) when the intervention and control schools were analyzed together (since they both received school feedings), showing a high correlation between the percentage of distracted students and the percentage who stated that they were hungry. Thus, when the percentage of students who are hungry decreases, students’ attention for school work improves. 5.1.2.1 Short-term reduction of hunger Percentage of students in the selected schools who indicate that they are “hungry” or “very hungry” during the school day. The team interviewed boys and girls from fourth, fifth, and sixth grades at the beginning of school, before the daily school meal was served. The team interviewed boys and girls from fourth, fifth, and sixth grades at the beginning of school, before the daily school meal was served. Illustration 11. Students who indicate that they are hungry or very hungry A total of 85.6% of students in the intervention schools were not hungry, and 79.9% were not hungry in control schools. 98.4% of the intervention students said they had breakfast that morning, as did 95.9% in the control schools. The focus groups with parents showed that in general, children had a small or no breakfast at home, which meant that the food served at school was their first formal meal of the day. The DD analysis showed a negative value of -10.7 percentage points, indicating a higher rate of improvement in the control schools compared to the intervention schools. Since control schools did not have a formal school 24.90% 8.50% 20.10% 14.40% 0% 5% 10% 15% 20% 25% 30% Control Intervention Control Intervention Baseline Midterm 27.45% 19.46% 30.08% 21.83% 15% 17% 19% 21% 23% 25% 27% 29% 31% LB EMT Intervention Control 25 feeding program during the baseline they had a higher percentage of school children hungry during the school day and the implementation of the LAE had a larger marginal impact on hunger than in the intervention schools. Even with a higher rate of improvement in control schools, the intervention schools still show 6.7 percentage point fewer schools children hungry during the midterm than did the control schools. These results between control and intervention schools and baseline and endline are statistically significant.15 No significant difference was found in the results between boys and girls. In the control and intervention schools together, 16.9% of boys and 17.6% of girls were hungry. The intervention schools saw a 5.9 percentage point difference from the BL to the MTE. No significant difference was found in the results between boys and girls. In the control and intervention schools together, 16.9% of boys and 17.6% of girls were hungry. The intervention schools saw a 5.9 percentage point difference from the BL to the MTE. No significant difference was found in the results between boys and girls. In the control and intervention schools together, 16.9% of boys and 17.6% of girls were hungry. The intervention schools saw a 5.9 percentage point difference from the BL to the MTE. Illustration 12. Hunger situation in the intervention schools Regarding the intensity of hunger in the intervention schools, 12.1% of students claimed they were a little bit hungry, and 2.3% said they were very hungry. In the control schools, 17.0% of students said they were a little hungry and 3.1%, very hungry. Depending on the municipality, the percentage of students who indicated they were hungry varied, although the average across municipalities stayed consistent. 16 The Municipality of Cuilco had a slightly lower percentage of students who expressed hunger. According to the economic potential study carried out by the Economic and Social Research Institute (IDIES) of the Universidad Rafael Landívar (URL), this municipality has one of the highest migration rates for men in Huehuetenango, leading to a higher percentage of female headed households in the municipality and a population that is 54.7% female, which in turn leads to additional income sources from women and improved diets in the households. In contrast, the control school San Sebastián H. in the municipality of San Gaspar Ixchil had a higher percentage of hungry students because delays in funding from the government meant there was no school feeding on the day of the evaluation. See table in annex 8.5.7. It is worth noting that sampling is not statistically significant at the municipality level and these results are not indicative of program impact. However, PCI believes there are important contextual differences among municipalities that can lead to differing outcomes, as explained in the paragraph. The LAE increased access to and the diversity of foods in both intervention and control schools, and a favorable change was found in the control schools, based on the decrease in students who indicated that they were hungry or very hungry during the school day. Similarly, PCI’s participation was key in the intervention schools, where 14,362 quintals of foods were distributed for student consumption. In 2018, a total of 5,557,403 school breakfasts were distributed (MOE data). 15 Significant at p<0.05 16 Comparisons between municipalities are not statistically significant. 85.6% 12.1% 2.3% None A Little A Lot 26 The LAE increased access to and the diversity of foods in both intervention and control schools, and a favorable change was found in the control schools, based on the decrease in students who indicated that they were hungry or very hungry during the school day. Similarly, PCI’s participation was key in the intervention schools, where 14,362 quintals of foods were distributed for student consumption. In 2018, a total of 5,557,403 school breakfasts were distributed (MOE data). 5.1.4 Improvement of student attendance (McGovern-Dole 1.3) Number of students (males / females) regularly attending (80%) USDA supported classrooms/ schools Illustration 13. Students with regular school attendance A decrease in the percentage of students regularly attending class was seen from the BL to the MTE. In intervention schools, 91.7% of students attended school regularly, compared to 87.8% of students in control schools, representing a six percentage point and 7.5 percentage point decrease from baseline, respectively. This difference was not statistically significant in the intervention schools. Illustration 14. Students with regular school attendance in intervention schools In the intervention schools, 8.3% of students were not regularly attending school, with boys contributing to a higher proportion (5.5%) of these absences than girls (2.8%). The same pattern was observed in the control schools: boys made up 7.9% of absent students and girls 4.3%. The DD analysis found a positive impact of 1.47% for intervention schools, as the rate of students with regular attendance decreased more in control schools from the BL to the MTE than it did in intervention schools. The correlation coefficient in intervention schools between students who received a minimum acceptable diet and students who regularly attended class was 0.780, indicating a high correlation and demonstrating the impact of the EDUCAMOS program on improving regular attendance. 91.69% 5.54% 2.77% 8.31% Regular Attendance Boys Girls 95.30% 97.70% 87.80% 91.70% 75% 80% 85% 90% 95% 100% Control Intervention Control Intervention Baseline Midterm 27 5.1.4.1 Reduce health-related absences Percent of students who state that illness is the primary reason they are absent from school Illustration 15. Students who state illness as main cause of absence School absenteeism is a key barrier to education the program addresses. Illness was the cause of 10.2% of absences in the control schools and 8.7% of absences in the intervention schools. No significant difference was found between the control and intervention groups at midline. 17 Fevers, colds, and coughs, which are symptoms of ARIs, were the most frequently reported illnesses (48%), followed by intestinal illnesses, stomach pain, and diarrhea (19%). The DD analysis found a difference of -1.70% between the intervention and control schools, showing fewer absences for health reasons and a positive impact in schools where PCI is present. See graph in Annex 8.4.3. No significant differences were found based on the sex of the students. In the intervention schools, girls were absent 8.9% of days, and boys 8.5%, while in the control schools, girls were absent in 10.7% of days, and boys 9.8%. Illustration 16. Types of illness for school absences Fevers made up 25% of the illnesses reported, which can come from multiple illness sources and becomes a generic element. Intestinal illness with diarrhea and stomach acheswere the second most cited illness. These were especially common in the winter because the increased humidity from the rainy season facilitates the propagation of viruses, bacteria, and parasites. A 62% correlation between the lack of healthy food preparation and appropriate storage equipment and the percentage of students in intervention schools who said they missed school due to illness was 17 P=0.11, not statistically significant 7.50% 7.70% 10.20% 8.70% 0% 2% 4% 6% 8% 10% 12% Control Intervention Control Intervention Baseline Midterm Stomach 19% Fever 25% Cold 14% Cough 9% Teeth pain 6% Don't Know 2% Other 22% Ear pain 3% 28 found, showing that PCI’s support for healthy food preparation and storage practices contributed to reducing health-related absences. According to the Huehuetenango Area Health Directorate (DASH), the incidence of ARIs was particularly high in the department during the field work stage of this evaluation (22 July to 15 August), and the endemic corridor analysis characterized the spike in infections as an epidemic. Diarrhea monitoring rates also showed an alarming spike. A 65% correlation was found between students who did not receive the minimum acceptable diet and those who reported illness as a cause of school absences. The other causes of absences cited were: work at home, work for someone else, family trips, and “I didn’t feel like it.” See the table of percentages in Annex 8.5.3. In intervention schools, girls cited work at home as the second most common reason for their absences after illness, and boys cited family travel as the second most common reason. These reasons are generally associated with accompanying their parents to market days for business and does not imply total school abandonment. In control schools, family travel was the most commonly cited reason for absences for boys, at 9%. 5.1.4.2 Increase school enrollment Number of students enrolled in schools receiving USDA assistance Based on official MOE statistics, 38,387 students attended school in the 2019 school year - 19,794 boys and 18,593 girls. Enrollment increased 4.5% from the 2017 BL, and acording to MOE statistics, enrollment at the national level was 3% higher in 2019 than the previous year. The following table presents the comparison of student enrollment. Table 5. Enrolled in intervention schools Municipality Boys 2017 Boys 2019 Girls 2017 Girls 2019 Total 2017 Total 2019 Percentage + - San Gaspar Ixchil 946 982 865 931 1811 1,913 5.33% San Pedro Necta 2,394 2,359 2,376 2,330 4,770 4,689 -1.73% Cuilco 4,440 4,636 4,115 4,326 8,555 8,962 4.54% San Pedro Soloma 2,906 2,990 2,805 2,929 5,711 5,919 3.51% Santa Eulalia 3,695 3,898 3,367 3,469 7,062 7,367 4.14% Santa Cruz Barillas 4,610 4,929 4,141 4,608 8,751 9,537 8.24% Total students 18,991 19,794 17,669 18,593 36,660 38,387 4.50% Prepared by study team. Source: MOE San Pedro Necta was the only municipality that saw a reduction in enrollment rates compared to 2017. Santa Cruz Barillas saw the largest enrollment increase. Percent of school aged children enrolled in USDA assisted schools Illustration 17. Percentage of school aged children enrolled in school The enrollment rates for school-aged children in intervention schools did not statistically significantly change from the BL, but the control schools saw a 6.4 percentage point statistically significant decrease. In both groups of schools, migration was the most frequently cited reason for non-enrollment, due to the parents’ need to generate economic resources. Migration has increasingly affected children in Guatemala in recent years. Parents who migrate generally take a child with them 92.80% 95% 86.39% 93.53% 0% 20% 40% 60% 80% 100% Control Intervention Control Intervention Baseline Midterm 29 because they are convinced that will help them gain entry to the United States. Huehuetenango has the second highest migration rate in the country and receives a large quantity of remittances. Enrollment rates in communities with PCI interventions were 4.95 percentage points higher than in the communities without PCI interventions. This translates to 3,584 more students in intervention schools, and 1,441 more students in control schools. See the DD table in Annex 8.4.2. Illustration 18. School aged children who leave because of migration Poverty is another reason boys and girls drop out of school, as they have to work or go to the fields to help support their family. Learning challengesin children, either because of their abilities or because of rebellious conduct, were the third most common cause of drop outs.Poverty is another reason boys and girls drop out of school, as they have to work or go to the fields to help support their family. Learning challenges in children, either because of their abilities or because of rebellious conduct, were the third most common cause of drop outs.Poverty is another reason boys and girls drop out of school, as they have to work or go to the fields to help support their family. Learning challenges in children, either because of their abilities or because of rebellious conduct, were the third most common cause of drop outs.Poverty is another reason boys and girls drop out of school, as they have to work or go to the fields to help support their family. Learning challenges in children, either because of their abilities or because of rebellious conduct, were the third most common cause of drop outs.Poverty is another reason boys and girls drop out of school, as they have to work or go to the fields to help support their family. Learning challenges in children, either because of their abilities or because of rebellious conduct, were the third most common cause of drop outs. Poverty is another reason boys and girls drop out of school, as they have to work or go to the fields to help support their family. Learning challenges in children, either because of their abilities or because of rebellious conduct, were the third most common cause of drop outs. Poverty is another reason boys and girls drop out of school, as they have to work or go to the fields to help support their family. Learning challenges in children, either because of their abilities or because of rebellious conduct, were the third most common cause of drop outs. 9.85% 4.32% 0% 2% 4% 6% 8% 10% 12% Control Intervention 30 5.1.4.3 Increase community understanding of the benefits of education Percent of parents in target communities who can name at least three benefits of primary education Illustration 19. Percent of parents who can name at least three benefits of education The percentage of parents who could identify at least three benefits of edu cation increased more in control schools than in intervention schools. Nevertheless, a higher percentage of parents in intervention schools (92%) could name at least three benefits compared to those in control schools (79.7%),.18 According to parents from intervention schools, the trainings from PCI technicians gave them greater ownership and awareness of these benefits. A favorable change was observed in the control schools as a result of the trainings for OPF that the MOE carries out in the framework of implementation of the LAE. The program’s impact was also seen in the proportion of parents who mentioned not just three, but from four to 13 education benefits. In the intervention schools, 63.8% of parents mentioned four or more benefits, while 50% of parents from control schools did the same, which was a statistically significant difference. Learning to read and write as well as how to speak Spanish are the main benefits that parents value in primary education. The benefit of primary education being the foundation for further studies was only mentioned by 57.4% of parents, which could explain the low percentage of students who continue after primary school. This is substantiated in the data from the 2018 MOE Annual Statistical Report19 that shows that for the department of Huehuetenango the net rate of primary school coverage is 80.35% while for middle school the rate drops to 23.29%. This is in addition to parents’ lack of economic resources to cover the expenses of middle school enrollment given that while there are 1937 primary schools, there are only 575 middle schools in the whole department, which then involves expenses for transportation, uniforms, school supplies, food, etc. 18 P=0.015, statistically significant 19 http://estadistica.MOE.gob.gt/Anuario/home.html# 50% 84.60% 79.7% 92% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Control Intervention Control Intervention Midterm Midterm 31 Illustration 20. Benefits of primary education mentioned by parents When the team observed students in the different intervention schools, they saw that the community had absorbed the knowledge from trainings on the topics of hygiene, cleanliness, self-care and appearance. The children come to school clean, generally with neat hair. They wear appropriate clothes (although simple) and everyone has shoes (some with inappropriate rubber boots). While the environment is dusty and dirty and students have to walk some distance to school, in general, the intervention students look more presentable than those of the control schools. In both control and intervention schools the students were friendly and respectful. 5.2. Increased use of dietary and health practices (McGovern-Dole SO2) The School Feeding Law (LAE) under decree 16-2017 was passed by Congress in Guatemala in order to improve school feeding for students, increasing government cash transfers for school feeding from Q1.54 per day/student (without the LAE) to Q4.00 student/day and favoring local procurement of perishable food. Implementation of the LAE has substantially improved the availability and variety of foods purchased in schools, which in turn has had a direct effect for improving the number of children receiving a minimum acceptable diet, both in intervention and control schools. Percentage of school aged children receiving a minimum acceptable diet To determine whether students were receiving a minimum acceptable diet the team first asked how many times the child ate during the day; 83.4% of the intervention students ate between four and five times a day and of control students, 81.3%. According to the interviews with control school principals, the LAE20 allowed them to now offer a formal school feeding because before they only had enough to offer a hot drink (atol) as a snack and the students didn’t eat breakfast at home for lack of resources. They said that the subsidy has been coming regularly since 2018. 20 Prior to the LAE, the average government contribution per student/day was Q.1.58 and the disbursements were irregular and insufficient for the 180 official days of class. In 2018 and 2019, as a result of the LAE, the contribution was increased to Q.4.00 and disbursements have been on time. This facilitates food access and diversity, because the MOE promotes nutritious official menus that include fruits, vegetables, dairy products, etc. 99.3 71.6 64.5 60.3 57.4 43.3 37.6 36.2 27.7 23.4 23.4 22.0 0.0 20.0 40.0 60.0 80.0 100.0 Read and Write Improve Spanish Speaking Prepare to find work Respect for others Improve study skills Hygiene practices Be a good citizen Math Feed myself well To know and value my body To have good health Avoid malnutrition 32 Illustration 21. Number of daily meals Illustration 22. Meal times The control schools which previously had little access to a morning meal, now show an increase from 78.8% in the BL to 88.2% in the MTE as a reflection of the LAE resources. In the intervention schools the OPF say that the children come home without being too hungry and don’t eat much lunch but in the middle of the afternoon they ask for a snack, which was evident when observing that this mealtime increased in both groups of students. In control schools this increased from 36.2% in the BL to 47.8% in the MTE, while for intervention students the increase was from 36.7 % to 48.8% in the MTE. The second factor that was evaluated as part of minimum acceptable diet was daily intake of at least four of seven basic food groups: a) Grains, roots and tubers, b) Legumes and dried fruits, c) Dairy products, d) Meat products, e) Eggs, f) Foods with Vitamin A (vegetable oil, fruits and vegetables), and f) Other fruits and vegetables. In the control schools, 83.4% consume a diverse diet while in the intervention schools 85.1% do. Illustration 23. Students with minimum acceptable diet21 A main finding about the minimum acceptable diet indicator is that there were positive changes in students of both groups: in control schools there was a 53 percentage point increase between the two evaluations and in intervention schools there was an increase of 39.1 percentage points. Interestingly, 58% of students who report that they receive a minimum diet and have parents that participated in PCI trainings, indicating the importance of efforts to educate families on nutrition. The DD analysis showed a larger increase in the percentage of students who received a minimum acceptable diet in the control schools than the intervention schools between the BL and the MTE, likely due to the implementation of the LAE, which increased food access and diversity in the schools. However, there was still a significant difference between intervention and control schools observed during the BL (45.1% vs. 29.9%, respectively), highlighting the importance of the complementary foods donated by 21 p=0.582, not statistically significant between control and intervention at midterm. P=0.545for males vs. females at midterm, not statistically significant. 2.0% 16.7% 46.7% 34.6% 0.9% 15.7% 44.3% 39.1% 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% 35.0% 40.0% 45.0% 50.0% 2 meals 3 meals 4 meals 5 meals Control Intervention 95.9% 88.2% 86.5% 47.8% 95.7% 98.4% 85.3% 93.8% 48.8% 95.6% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Breakfast Morning Snak Lunch Afternoon Snack Dinner Intervention Control 29.90% 45.10% 82.90% 84.20% 0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% Control Intervention Control Intervention Baseline Midterm 33 USDA. These foods, which provide two of the seven food groups required by the minimum acceptable diet, are essential where the LAE isn’t implemented or where the disbursements aren’t timely. LAE implementation led to a dramatic increase in the variety of food groups that students consumed in the intervention and control schools. At the BL, only 0.5% of students in control schools and 2.2% in intervention schools had access to six or more food groups on a daily basis. At the MTE, 25.8% of students in the control schools and 28.1% in intervention schools had daily access to six or more food groups. This can be observed in the comparative table in Annex 9.1.2. Principals and focus group participants confirmed that the school menus are culturally relevant. In some schools, the OPF and principal tested the menus that were given to them by the MOE to get approval from the parents and to ensure they would be liked by the children. For these tests, they used foods purchased at the municipal level. Respondents from the control schools mentioned that they need more training to prepare the MOE menus, but the intervention schools had sufficient support from PCI. 5.2.1 Improve knowledge of health and hygiene practices (McGovern-Dole 2.1) Percent of parents who can identify at least three important health / hygiene practices Illustration 24. Parents who can identify at least three health practices. PCI led food, hygiene, and health practices trainings in both intervention and control schools in coordination with staff from the MOE, the Ministry of Agriculture (MOA), the Ministry of Health (MOH), and the Secretariat for Food and Nutrition Security (SESAN). While this limits the ability of the control schools to serve as a comparison, the results do indicate the effectiveness of EDUCAMOS led trainings in control schools. As graph 25 shows, knowledge among parents in control schools increased by 26.3 percentage points, likely as a result of trainings supported by EDUCAMOS. No significant difference was found between the results of the BL (93.20%) and the MTE (90.60%) in intervention schools. 22 However, the program’s impact is evident from the proportion of parents who could identify not just three, but from four to seven practices. In the intervention schools, 51.6% of parents mentioned four or more practices, while only 38.7% did so in the control schools. See the DD table in Annex 8.4.6. 22 p=0.582 for midterm comparison of control and intervention schools, not statistically significant. 59.20% 93.20% 85.5% 90.60% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Control intervention Control intervention baseline midterm 34 Illustration 25. Cleanliness practices mentioned by parents (%) Students primarily mentioned hand washing, changing clothes, and daily baths as the main cleanliness and hygiene practices. The participants of the OPF focus groups stated that the hygiene trainings that PCI provided were replicated in the home in the intervention schools (though it was not possible to determine to what degree), while a more generational transmission of knowledge happened in the control schools, where the children talk about the topic in the home sometimes. 5.2.2 Increase knowledge of safe food preparation and storage practices (McGovern-Dole 2.2) Percent of mothers (or care providers) who can identify at least three key practices aimed at safe food preparation and storage practices Illustration 26. Mothers who identify at least three key practices for food preparation and storage23 Collaborating mothers from the intervention schools demonstrated an equal amount of knowledge of at least three key practices for safe food preparation compared o mothers from the control schools in the MTE. A likely cause is that PCI trained department and municipal level MOE staff (principals) and OPF members from schools outside the PCI coverage area on food, hygiene, and health practices, and they have in turn taken this information back to their schools. They also shared “La Taza Escolar” (“The School Cup” in English) tool, which helped to increase control school parents’ knowledge. While this limited the ability of controls schools to serve as a comparison, the results do indicate the effectiveness of the EDUCAMOS training strategy. However, the statistically significant decrease in the percentage of mothers who could identify three food preparation and storage practices from the BL to MTE in intervention schools indicates that Educamos should implement a refresher training for mothers who have not received training recently. Observation evidence also indicates that mothers at intervention schools are implementing practices taught by PCI. The study team observed collaborating mothers in the kitchen while they prepared the school meal and found that the mothers in intervention schools adopted the practices to a stronger extent than the mothers in the control schools. For example, In several control schools, animals were observed roaming around in the food preparation area, but less frequently in intervention schools. No correlation was found between schooling levels and the ability to mention three or more food preparation practices, implying that training on food preparation can be generalized to all parents regardless of education level. The high percentage of mothers who could mention three practices showed that they took ownership of 23 P=0.565 for comparison of control and intervention midterm results, not statistically significant 85.5 83.9 48.9 68.3 74.2 30.6 23.7 79.40% 99.10% 86.30% 89.40% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Control Intervention Control Intervention Baseline Midterm 35 the practices, regardless of their schooling level. The following cleanliness practices were mentioned by both school groups. Illustration 27. Cleanliness practices mentioned by parents Illustration 28. Cleanliness practices carried out by collaborating mothers (%) The relationship between knowledge and practice is evident, given that the practices that were most frequently mentioned were also the ones most practiced in the intervention schools, confirming the relevance and efficacy of PCI’s actions. 58.8 47.1 64.7 47.1 45.1 92.2 54.9 17.6 47.1 66.4 50.4 64.6 43.4 40.7 91.2 69.0 34.5 46.9 0.0 20.0 40.0 60.0 80.0 100.0 Cover food Water stored in containers where hands do… prepare food on separate tables use a covered trash can no garbage on the floor Cooks clean hands before cooking Cooks don't have rings or painted nails Test water No animals in the kitchen Intervention Control 75 57.7 63 18.5 25.9 72.2 7.4 19 11.1 88.5 86.2 100 53.3 6.7 100 13.3 70 3.3 0 20 40 60 80 100 Cover food Water stored in containers where hands… prepare food on separate tables use a covered trash can no garbage on the floor Cooks clean hands before cooking Cooks don't have rings or painted nails Test water No animals in the kitchen Intervention Control 36 Illustration 29. Cleanliness practices that collaborating mothers know and carry out in intervention schools 5.2.3. Increase access to clean water and sanitation services (McGovern-Dole 2.4) Number of schools using an improved water source Illustration 30. Schools using an improved water source The percentage of schools using an improved water source increased by 13.3 percentage points from the BL to the MTE in intervention schools, compared to an increase of six percentage points in control schools. The DD analysis showed a 7.3% positive impact in intervention schools, See the table in Annex 8.4.8. Although 100% of intervention schools and 96% of control schools have an improved water source, guaranteeing a quality water supply throughout the entire year remains a challenge. Water availability varies across Huehuetenango. Some regions pipe water from springs into homes and schools but don’t have taps because the high water pressure means the pipes will burst if the taps are closed. In these situations, the water overflows into cleaning basins (pilas), toilets, and onto the ground. In other, nearby schools, water is more scarce and has to be collected and carried into the home or school. 66.4 50.4 64.6 43.4 40.7 91.2 69.0 34.5 46.9 88.5 86.2 100 53.3 6.7 100 13.3 70 3.3 0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0 100.0 Cover food Water stored in containers where hands do… prepare food on separate tables use a covered trash can no garbage on the floor Cooks clean hands before cooking Cooks don't have rings or painted nails Test water No animals in the kitchen Implement Know 90% 86.70% 96% 100% 80% 85% 90% 95% 100% 105% Control Intervention Control Intervention Baseline Midterm 37 Illustration 31. Sources of improved water Similar to the BL, piped water continues to be the main source of improved water in the schools i n the MTE. Good infrastructure was found in 100% of the PCI supported schools the evaluation team visited; many had rainwater catchment systems and stored the water in tanks. The structures were generally in poorer condition and were more deteriorated in control schools, and the education community had no maintenance plans. Although all intervention schools had good infrastructure or a means of collecting water, water was only available 75% of the time in control schools and 72.4% of the time in intervention schools, according to direct observations and consultations with principals. Some zones of Huehuetenango, particularly the northern transverse corridor, have abundant water, but others have poor hydrological potential (the supply of water and the recharge rate in the rainy season). When no water was available in either intervention or control schools, the school asked students to bring a container of water so that the students could still wash their hands. Number of schools with improved sanitation services The percentage of schools with improved sanitary installations slightly decreased from the BL to the MTE in intervention schools, but the difference was not significant (2.5%). The percentage of control schools also decreased, but the difference was significant (12.7%). Water was available for use with the improved sanitation services in 72.4% of the intervention schools. The DD analysis showed a 10.20% impact in the intervention schools compared to the control schools between the BL and MTE. Maintenance of the sanitation services was a challenge for both groups of schools, either in terms of odor reduction via permanent cleaning or because of the lack of a drainage system (where lavatory facilities had flush toilets), which was the case for a large proportion of schools with latrines or a septic system. Illustration 32. Schools with improved sanitation services Sewer systems and septic systems (dry pit) were the main types of facilities were for both groups of schools. The septic system was the most common system in the intervention schools (31.3%), compared to only 8% in the control schools. A total of 75.9% of the intervention schools had separate facilities for boys and girls, compared to 60.7% of the control schools. 4.0% 7.1% 64.3% 3.6% 10.7% 10.7% 0.0% 3.4% 58.6% 0.0% 27.6% 10.3% 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% None Public Tap Piped Water Well Protected headwater Rain water collection Intervention Control 96.70% 90% 84% 87.50% 75.00% 80.00% 85.00% 90.00% 95.00% 100.00% Control Intervention Control Intervention Baseline Midterm 38 Illustration 33. Separate facilities for boys and girls The evaluation team observed the use of cleanliness practices before eating and after using the lavatories. Parents told the evaluation team in the focus groups that their children had begun bathing, washing their hands, brushing their teeth, combing their hair, and cleaning themselves. Handwashing stations are critical in implementing these new hygiene practices and preventing many diseases; of the intervention schools, 96.6% had handwashing stations, while only 67.9% of control schools did. A higher water availability was observed in more intervention schools with sewer systems than in control schools. A total of 90% of the intervention schools and 81.3% of the control schools had sewer systems. See graph for crossing the water availability and type of sanitary facility variables in Annex 8.5.8. 5.2.4. Increased access to requisite food prep and storage tools and equipment (McGovern-Dole 2.6) Percentage of schools with improved food prep and storage equipment The results of the measurements of good food preparation practices and adequate food storage areas were both used to calculate this indicator. For a school to have good food preparation practices, it had to meet at least three of four criteria: it must be clean, it must have dry utensils, and the food must be covered and stored separately from contamination sources. The storage area requirements were: the area had to store only food items, it had to have a scale, sufficient physical space, appropriate walls and a roof, shelves or platform, and ventilation, and it had to be clean, orderly, and free of pests, and the products had to be separated. Illustration 34. Schools with better food preparation and storage equipment The intervention schools improved their compliance with this indicator by 13.50 percentage points between the BL and MTE, while control schools decreased theirs by 6.3 percentage points. The DD analysis showed that the program had a 19.80% impact, an important accomplishment for LAE food storage and preparation safety compliance. The difference that was found between the intervention and control schools was significant, and it was likely a result of the food preparation practices training that the collaborating mothers received in the intervention schools as well as the use of checklists in the school storage areas to ensure compliance with the standards. The training and use of a checklist did not happen in the control schools, where only an orientation for teachers was provided. 60.7% 75.9% Control Intervention 43.80% 61.50% 37.50% 75% 0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% Control Intervention Control Intervention Baseline Midterm 39 Annex 9.4 presents the results of this indicator from the MTE and BL and shows the details of the statistical significance tests. 24 5.3. Foundational Results Foundational results are measured through regular monitoring of project activities and thus are not a focus of this evaluation. For details and accomplishments related to the foundational results, please refer to the semi￾annual progress reports submitted to USDA. More emphasis on foundational results will be made in the final evaluation of Educamos. 5.4. Responses to key evaluation questions 5.4.1 Design, implementation (strategies), and relevance a. Design. The literacy in bilingual and intercultural environments model that the program developed is valid for the local context, as it incorporated: a) a measurement of the degree of bilingualism of pre￾school and first grade students; b) the use of Mayan and Spanish language reading materials and other educational resources; c) efforts to make tools available to teachers for instruction and learning in bilingual contexts with educational technicians who speak the local language (including a university diploma program); and d) the promotion of reading spaces with bilingual materials. Similarly, the health and nutrition models were adapted to the local context through the inclusion of key topics such as the consumption of safe water, safe food preparation, dietary diversity, and the incorporation of lead mothers. b. Implementation. The following strategies were the most effective25: i. Training MOE staff, OPF members, and school principals outside the coverage area in health, nutrition, hygiene, and food preparation topics, because it increased their knowledge and practices. The impact extended beyond the coverage area. ii. The distribution of materials in Mayan languages, because the teachers and CTA recognized that these were the key inputs to address students’ bilingualism as well as to promote literacy techniques. The majority of teachers applied at least one new technique, demonstrating that the collaboration between J&A and UVG was effective. iii. Support for school infrastructure has been received with overhelming acceptance by communities, as this intervention is solving key challenges that no other entity had addressed before, either due to a lack of interest or a lack of resources. The MOE is on the verge of beginning a basic refurbishment program. iv. The impact of improved food storage practices in store rooms and food management through final consumption was evident through observation and comparison with the control schools. v. The traffic light tool for cleanliness, because in general, the areas were clean, and there was a notable difference in comparison to the control schools. vi. The school information corner tool, because based on the indicators, it sensitized users to consider the numbers in red and was used in meetings with parents. Users knew how to interpret it and will continue to use it to improve their performance. vii. The WE group initiative, because it helped women take leadership, create health (lead mothers and their actions) and education (support for continuing studies) improvements, learn about their rights, and begin productive activities. 24 p=0.005 for midterm comparison between control and intervention schools, statistically significant. 25 Annex 9.2 Summary of results from analysis of work strategies, presents details on the strategies. Annexes 9.6, 9.7, 9.8 and 9.9 present the results of interviews with the MOE, J&A, principals and teachers, respectively. 40 viii. OPF formation, because they planned food purchases, creating an organization that drove program activities. ix. The introduction of vegetables in gardens and macro tunnels, which supplied fresh foods to the school. It also provided a learning opportunity for the community and promoted sustainability. x. The school fairs and camps, because they were distinctive and successful activities that can be replicated in other municipal and departmental environments. xi. Integration with Save the Children International (SCI) and Catholic Relief Services (CRS), because they promoted joint actions in their FFE programs and in the development of the Nuestra Cosecha project with USDA funds. xii. Coordination with government institutions (MOE, MOH, MOA, and others), because implementation adhered to national guidelines and policies. The least effective strategies were: i. Promotion of teacher attendance, as the results decreased between the BL and MTE. ii. Replication of lead mothers, because the percentage of mothers who could mention food preparation practices decreased. A significant rotation of mothers meant not all mothers were covered. iii. The frequency of health and hygiene trainings for staff, as well as the involvement of a greater number of parents. The training didn’t reach all the parents. iv. Follow up for the effective use of reading spaces (corners, libraries), because even though reading spaces had a real impact on students, some schools didn’t use them. v. Despite the acceptance of new school infrastructure, sustainability and maintenance continues to be an issue because the project/community leaders did not designate a particular member/stakeholder to be directly responsible for conducting maintenance activities, As a result, existing infrastructure showed deterioration in intervention schools. vi. The incorporation of Municipal Councils, because their contributions were minimal and no mechanism was defined for program information feedback. vii. External dissemination of success, accomplishments, and good practices, because learnings were not well publicized or shared. c. Relevance. The program is active in the area of Guatemala with the lowest performing education and health population indicators. It is implemented with respect for the Mayan languages and culture and integrates local foods into the official menus, which is coordinated with the MOE for their approval, and according to the results of the hedonic scale assessment, the majority of students are satisfied with the school meals. 5.4.2 Sustainability a. Collaborative relationships. At the local level, OPF members demonstrated commitment to the program, managed meals, and organized the community for infrastructure and school garden activities. Constraints for the OPFs in direct management of the school feeding subsidy (which is their responsibility by law) included illiteracy, a lack of time, and a fear of making mistakes. Because of those constraints, the school principals tended to take charge of the entire process, which took time away from their educational activities. Additionally, the reporting instruments for the MOE were complex even for the teachers, so OPF leadership is not expected in the short term future. The OPF needs a close relationship with the COCODE, as the legal representative of the community, to carry out joint efforts to obtain institutional government or municipal support to cover education, health, feeding, infrastructure, and other priorities to promote the sustainability of the program. 41 The relationship with government agencies is appropriate, as a close relationship with the MOE is needed to gather support for trainings for teachers and principals. PCI supervisors and technical staff indicated that principals are notified of municipal events by the MOE before they occur. Deworming campaigns are coordinated with the health districts at the municipal level, and water quality tests are conducted in schools. The intervention schools prepared recipes with PCI foods and with MOE approval, demonstrating the degree of coordination that exists. Joint implementation of the Nuestra Cosecha project with CRS and SCI demonstrated alignment with the national school feeding policy because it strengthens local producers to sell foods to the schools. PCI supervisors represented the program in other municipal venues, such as Municipal Food and Nutrition Security Commission (COMUSAN) and Municipal Development Council (COMUDE) meetings. Given that each entity seeks to accomplish its own goals, that financial challenges remain in the public sector, and the lack of sufficient staff for the Municipal Councils, who by law are responsible for the economic development of their population, the main challenge is to achieve a greater integration of the organizations to agree on a joint agenda for school feeding or education. b. Transition of school feeding. In spite of the higher subsidies than previous years, it is difficult to increase OPF ownership at the local level in their role in analyzing what to purchase, determining where to purchase it, and maintaining financial records with the MOE. Additionally, the government institutions were not able to agree upon their convergence points for LAE implementation. For example, the MOA is slow to register producers, the mothers who prepare food do not have experience in preparing diverse recipes (at least 10 per month), so a teacher needs to be involved in food preparation, the principals don’t receive enough information to promote family agriculture with local producers, and it is easier to only have one provider for all products, thus missing the opportunity to support local development. PCI should analyze these issues in conjunction with government entities and develop a strategy to accomplish the expected transition. c. Actions for sustainability. Current program efforts include: i. Develop the Nuestra Cosecha project because it promotes local food purchases. ii. Coordinate with other USDA projects to operationalize the LAE. iii. Incorporate community staff to generate local knowledge. iv. Empower women to become actively involved in the education of their children. v. Integrate bilingual education through the training of teachers and provision of materials. vi. Incorporate tools such as the cleanliness traffic light and the check list for storage rooms. vii. Incorporate parents in activities such as camps and fairs. viii. Work with and replicate demonstration projects (gardens, macro tunnels). ix. Train sustainability commissions, as they have not continued to meet. x. Propitiate feeding policies at the municipal level (this hasn’t happened). 5.4.3 Outcomes and impact Quantitative Results. The team used the DD analysis to compare progress from the BL to the MTE in both school groups to measure the efficacy and impact in the program indicators. Positive results were found in: a) the application of new techniques by teachers and principals, b) student attention in class, c) increased enrollment, and d) the use of improved water sources. Except where water access was limited, evaluators noted the constant use of improved sanitation services and improved food preparation and storage equipment, particularly in comparison with control schools. Outcomes was not evident in the following areas due to the lack of a significant difference between intervention and control schools: a) teachers’ attendance, b) reduction of student hunger, c) parents’ knowledge about the importance of education and about health and hygiene practices; d) the minimum diet for students, e) mothers’ knowledge about safe food preparation and storage, and f) literacy. 42 Qualitative Results. The following qualitative differences were found between the intervention and control schools. The program’s impact was established in the areas listed below, stemming from the results of the focus groups, interviews, and observations: a. Involvement of WE group members: The WE groups facilitated leadership building, as the participants became lead mothers or obtained a role in the school or community, overcoming many of the barriers women face in these communities. Their participation led to their increased value to and recognition in the community. Additionally, they worked in productive activities that generated investment capital. Some groups of women began to sell foods to the schools, which is in line with the goals of the LAE and ensures that a part of the foods purchased in the feeding programs come from family agriculture. b. Reinforcement of teachers’ capacities, because they had opportunities to update and improve their literacy instruction techniques with support from educational materials in Mayan languages. Another highlight was the distribution of materials in the Q´anjob´al language, which was not available through the MOE, meaning the schools in that language area could only teach with Spanish materials. c. Strengthening of the capacities of OPF members, because the MOE did not have enough staff to provide personalized attention the way EDUCAMOS did. Stonger OPF involvement, mainly in the purchase and management of foods, is critical for LAE implementation. Additionally, the program involved them in other activities such as school hygiene, food management by volunteers, etc. d. Improvement of school infrastructure via the refurbishment of kitchens, lavatories, and water access projects. These activities were critical because the community and the central and municipal governments were unable to address them. e. The production and marketing of foods produced in the macro tunnels, which contributed to the increased availability of foods for the school. This was also a learning opportunity for the community and promoted sustainability. f. School fairs and camps, which were successful in motivating the educational community and have the potential to be replicated by other schools and municipalities. g. The incorporation of community volunteers to support school hygiene, because it built local capacity in education, health, hygiene, and agricultural activities. h. The integration of technological resources into digital classrooms in rural areas, an initiative in which the students were very interested. This promoted self-learning and provided educational content that was only previously accessible in urban areas (use of Wikipedia, National Base Curriculum [CNB] materials, etc.). However, it will not likely have an impact in reaching the program’s objectives because only 12 classrooms were integrated. Behavior change and adoption of practices. The following activities were adopted in intervention schools, resulting in notable outcomes differences when compared to the control schools: a. School hygiene: The use of the “cleanliness traffic light” was mentioned as a hygiene-promoting tool. Better results were noted when follow up was provided by the school governments and lead mothers. Schools adopted the traffic light in the classrooms, and bins were placed for the reuse and recycling of materials. b. The establishment of the Reading Corner methodology in classrooms helped teachers motivate students with a series of reading materials that encouraged them to read. Students see the Reading Corner as something novel that allows them to have fun while practicing their reading skills. c. The reading fairs motivated students to practice reading so they could win their peers’ and other schools’ recognition. d. Efficient food management in store rooms was facilitated with the use of a check list, follow up by PCI staff, oversight by the lead mother, and the involvement of the OPF, which led to safe and optimal food management, including record keeping for food receival and use. e. Knowledge about and the practice of school gardens and macro tunnels extended beyond the schools. Parents and principals mentioned that some parents who participated directly in school implementations 43 and those who were curious about how to grow foods they hadn’t seen in the community before started to use the techniques learned and applied in the school. f. The health and nutrition topics PCI trained schools on were put into practice to some degree (without defining the dimension). The topics applied were mentioned in the following order: water disinfection methods, personal and household hygiene, handwashing, and food recipes. g. The WE group participants showed notable behavior change from the program, beginning with “losing their fear of speaking,” which seems simple but is actually the beginning of many changes to come from exercising leadership and speaking in public, to entering the productive environment and generating income, things that men traditionally did. There was no evidence that the WE groups encouraged school enrollment or attendance. Barriers to effectiveness. Focus groups and interviews identified the following: a. Veteran teachers who felt that they had enough experience teaching reading and writing with traditional methods and didn’t need “young ones” to tell them how to do their job resisted some of the literacy activities. Some principals questioned the fact that technicians who hadn’t been principals tried to teach them new literacy methods as well. b. The program cannot demand teachers’ attendance because they are protected by the collective agreement between the teacher’s union and the MOE and because it created conflict with the schools. c. Parents’ illiteracy especially limited the effectiveness of trainings. Posters or banners written in Mayan languages did not contribute to the trainings because the parents generally couldn’t read their language; they only spoke it. The trainings were more effective when the staff used dynamic techniques to involve parents in the topic. The program’s dissemination method also created a barrier because the training focused on a specific group of parents and was not replicated with all the parents in the school. d. Training on school menus did not happen as frequently as expected because the technical staff rarely visited the school. When they did, they could only work with a small group of mothers, covering only two or three recipes, even though the MOE required at least 10 different menus. This problem, plus the fact that different mothers helped on different days, meant that a teacher had to take time from the classroom to manage the recipe of day. e. The prolonged drought and the poverty of the parents meant that many students came to school without eating breakfast, which resulted in hunger in the morning if the school breakfast wasn’t served until 10 a.m. Some schools served a hot drink (atol) first thing in the morning. f. Hygiene practices could not be effectively implemented in schools where water was scarce, meaning the lavatories and school environments could not be kept clean and the students could not wash their hands before the school feeding and after using the toilet. Although the program supported the installation of rain water collection systems, these are only functional in the rainy season, and the rest of the year students had to bring water from home. This also affected the school garden and macro tunnel initiatives. g. Women faced multiple problems when joining the WE groups. These included: a lack of experience managing their own finances, little organization of women at the local level, a fear of “wasting time” and savings, time constraints (days and nights were full), machismo that prevented women’s organizations because they might learn different, non-traditional behaviors, problems with other women (aggression), low motivation, mistrust because they didn’t see the tangible benefits (for example, generating family income), low opinion of their own abilities to do something new and make decisions, and a lack of experience in taking on risks. This limited the formation of new groups in the communities and increases in memberships where the groups already existed. Level of satisfaction. Satisfaction was measured qualitatively through focus group discussions. Participants were asked what activities they participated in, and their level of satisfaction with each activity. Due to low literacy levels of the participants, the consultant used a sheet of paper with facial expressions to represent levels of satisfaction (using a five-point likert scale from very satisfied to very unsatisfied) with various components of the program and asked the participants which expression most closely explains their level of 44 satisfaction. In all 30 intervention schools, the participants indicated that they are satisfied or very satisfied with the different program interventions, and there were no participants who mentioned dissatisfaction. Illustration 35. Satisfaction with Program Interventions Although participants said in the BL that they hoped PCI would continue to support them permanently, they said at the MTE that they plan to prepare themselves to continue their work in activities such as store room management, food preparation, fairs, the cleanliness traffic light, and others. Another group indicated that they hope PCI will not stop supporting them. 5.4.4 Gender Because gender is a cross cutting element of the program, the study evaluated gender parity by measuring men’s and women’s participation in leadership roles, especially their participation as teachers and OPF and WE group members, and particularly by examining the program activities aimed at the equitable involvement of men and women in strategic actions and decision making opportunities at the family, social, economic, and political levels. a. Women’s participation and decision-making power As of September 30, 2019 (report submission date), EDUCAMOS had 108 WE groups. The majority of WE groups that were organized across the intervention communities stated that they received technical assistance from PCI and that they are more knowledgeable about organization, savings and loans, and leadership building. Although males participate in some groups, women’s leadership is promoted. One group said they were organized by the OPF. WE group members stated that because of the knowledge they acquired in the trainings, they now know their rights, how they can obtain support and benefits, and that they have changed the way they participate in their community. Since these trainings, they feel empowered and value their own point of view as equal to their husband’s, and they make decisions about education, diet, money management, and opportunities for participation in their community and family. Women now have confidence in their opinions, and they speak up in meetings at the school, church, and community and in institutional visits and with political parties. Sometimes, when money is needed, the wife respects the husband’s decisions for the family budget. Where women manage their own money, the couple makes decisions and support each other for their children’s needs. Women who are part of a WE group now have savings and a productive cycle, so decisions in these homes are made together. There are no WE groups in control schools, and the evaluators observed during focus 6.7 46.7 36.7 60.7 15 89.5 66.7 93.3 53.3 63.3 39.3 85 10.5 33.3 0 20 40 60 80 100 School Feeding (n=30) Education, training, bilingual materials (n=30) Health and Nutrition training (n=30) Infrastructures (n=28) Women Empowered (n=20) School Gardens (n=19) PTA trainings (n=18) % Satisfaction Satisfied Very Satisfied 45 groups that women there were timid in sharing their opinions and generally expressed that they do not participate in household or community decision-making. Women’s participation and decision-making power changed recently due to the increased migration. In municipalities where the men who held OPF and leadership positions in the community migrated to the United States, women necessarily assumed these positions in the school, COCODE, and church, and because of their participation in a WE group, they were willing to take on these new responsibilities (with some trepidation but also with confidence due to the trainings they received). Women’s families and communities now support their participation. Their neighbors also saw changes in the women who were trained. b. Equitable scope of program activities Because women face more challenges in leading school and community activities, the program focuses on supporting female leadership and increasing their representation in decision making positions via WE group formation and the involvement of lead mothers, encouraging women to overcome their fear of participation so the community will value them more. Participation in decision making roles that were traditionally led by men, such as the OPFs, now provide an opportunity for women because PCI motivated them to become involved and because male migration impacted gender dynamics throughout the department of Huehuetenango. Many empty positions were filled, with minimal resistance or complications, by female leaders in training with the motivation for change. c. Activities and results Program activities were created using strategies to develop and improve conditions for men and boys as well as women and girls so that each will benefit equitably. The goal is for men/boys and women/girls to share opportunities without discrimination, work together, and participate equally in decision making. The schools now have similar numbers of male and female students. The WE groups accomplished their goal of equipping and empowering women to balance decision making at the family and community levels. The following activities helped to achieve these goals: i. Promotion of savings and loans so that mothers can contribute to the family’s finances. ii. Support for behavior change so that girls do not drop out of school. iii. Training and follow up on techniques for teachers to involve girls and boys in the education process. iv. Support for women to occupy leadership roles at the school and community levels. v. Support for WE group members to join the OPF. vi. Strengthening of women to take over the leadership openings that arose from male migration. vii. Encouragment of boys and girls to participate equally in classroom commission roles. viii. The name of WE groups could be changed to perhaps People Empowered so that men are not reluctant to join. ix. Introduction of investment ideas for productive projects that appeal to men. While some of the topics were appropriate for both, the majority focused on women. x. Introduce the importance of savings for men to support the household from the beginning. xi. Conduct meetings at times that would better facilitate male participation. d. Addressing gender barriers Gender equity in schools is still barred by historical aspects. The devaluation of women, which has consistently been a part of cultural traditions, still exists. While programs like WE and the needs of single mothers have influenced existing paradigms, change will continue to take time. Poverty directly affects girls, who often drop out due to the need to help at home. Based on information collected by the MOE, girls’ lack of rights is not due to ancestral customs or machismo, but is generally a result of poverty and violence and often goes unrecorded. 46 Through the formation of WE groups, women became involved in activities where their opinions were valued, and they can now make decisions that are respected. Women who were valued in their communities received the same education and nutrition opportunities for their daughters as the boys in their homes. Both parents make decisions about their children’s educations. The program also facilitated the savings and loan methodology in the WE groups to partially address families’ financial problems to enable them to pay lower interest rates than they would get from the loan sharks in the community. e. Importance of education for men and women Given the changes seen in families over time, men allowed women to make the decisions about education for the family (for boys or for girls), and the tasks at home are divided so that both can attend school. Decisions about school attendance for girls and boys are made by both parents. Women talked about the importance of their children going to school so that they learn to speak Spanish well, were able to keep accounts, and knew how to read. Similar numbers of boys and girls are now enrolled in school. f. Gender practices at the school level The team analyzed gender practices in activities conducted by the community and by members of the educational community. In general, no significant differences were found in the treatment of either gender for primary school students either by the parents or by teachers and principals. Both genders went to school in the same conditions and had the same use of and access to the benefits offered by the program. Both boys and girls used all the classroom resources: flip charts, big books, and educational games. Each gender participated equally in the reading fairs and camps. According to teachers and principals, many parents think girls only need to attend school until third grade so that they obtain just enough knowledge to function in the community: so they will learn to read and write, speak Spanish, and do basic math. However, departamental level school data from 2018 indicate that boys and girls maintain about the same ratio in first and sixth grade: out of the 44,703 students in first grade, 48.5% were girls and 51.5% boys, and out of the 21,057 students in sixth grade, 48.9% were girls and 51.1% boys. g. Safety and entry into the classroom Eight of every 10 students went to school by themselves or accompanied by other boys and girls, such as siblings, friends, or neighbors that they met along the way. In 10% of the cases observed, children (boys or girls) were accompanied by an adult woman, while an adult male was observed taking a child to school in only one of the 56 schools that the team observed. Once the children arrived at the school, the line-up to enter the classroom was mixed in 96% of the schools. In one school, the children made two lines and entered the classroom in pairs of boys with girls, while in another school, the girls lined up ahead of the boys. Boys and girls had mixed seating inside the classroom in 96% of the schools. The team observed separated seating for boys and girls in only two schools. h. Teaching approach The team found that cleaning tasks were distributed equitably among boys and girls in all the schools (for example, in the cleaning and hygiene work using the cleanliness traffic light in intervention schools). Most (80%) of the teachers observed used inclusive language by addressing the students by their names or as “boys and girls,” while the other 20% used “boys” (in Spanish the masculine word is also the generic word for children). Additionally, 94% of the teachers did not treat girls and boys differently, while the other 6% treated girls more positively, that is with more sensitivity and gentleness than for boys. 47 The MOE runs a specific strategic program that provides scholarships for girls and a training program for the prevention of premature pregnancies. The MOE is in the process of creating interinstitutional coordination efforts to protect girls from violence and other actions in the equity curriculum. i. School feeding The team observed that girls and boys received their food at the same time in all of the schools, though if they ate together or separately depended on whether the school had a dining hall. When there was no dining hall, the children sat together, but when there was a dining hall, boys and girls were seated together by grade. When collaborating mothers served the children in a dining hall, boys and girls received the same rations, but when there was no dining hall, the children took their own plates and asked for the amount of food that they wanted. j. Health and Hygiene The lavatories were separate and duly marked for boys and girls in 75.9% of the intervention schools, but only 60.7% of the control schools had adequate signage. Both boys and girls washed their hands before the school feeding. In intervention schools, 65.38% washed their hands adequately, but only 50% in the control schools did the same. 5.4.5. Women Empowered The focus groups results showed a high differential value of the groups in the improvement of family finances and involvement in children’s education and nutrition. Through the trainings they received, women in the WE groups improved their self-esteem and increased their participation in household decision making. They said that their husbands take them into account now, first because they are more extroverted, and also because some of them contribute financially to the household. The interactions among the women helped them open up socially, and their comradery is useful. Migration also played a “distortional” role because it accelerated their participation and turned them into heads of their households, administrators of their resources, and decision makers. Within the WE groups, those who knew how to read and write were important because they kept the financial records for the group. Some expressed regret that their parents did not send them to school to learn as a child, and now many want to send their daughters to school so they will have better development opportunities within and outside the community. WE group members said that they now make decisions about their sons’ and daughters’ schooling jointly with their spouse. They share their opinions and attend meetings at the school. Constraints for women who participate in WE groups The WE model is adept in developing women’s knowledge and skills by encouraging them to take action in their daily lives so that they learn by doing, and these practices motivate and encourage them, promoting behavior change and empowerment in the long term. In this way, the women were able to take ownership of their activities in the community and improve their quality of life at home. They learned to seek financial independence and created a ready source of small but immediate, accessible, and reliable funding. While these advantages were evident from the comparison of women who participated in WE groups and those who didn’t, some women still did not participate for several reasons. The key reasons that were mentioned in the focus groups and interviews with principals and school leaders are listed below: i. Lack of experience managing their own financial matters, since the only loan experience they have is with loan sharks, who charge high interest rates and who could take the lendees’ land and money. ii. Few women’s organizations in the community focused on savings and loans. iii. Fear that it won’t work well and that they will quickly lose their savings. 48 iv. Fear of paying fines based on the group’s internal participation rules. v. Lack of time due to their many other responsibilities, mainly in the home and church. vi. Machismo, which doesn’t allow women to organize themselves, since they might learn and behave differently at home and with their husbands. vii. Problems sometimes arise in the existing groups, so some women prefer to not participate. viii. They don’t see any activity in the group other than savings. ix. The presence of some violent, aggressive women in groups scares others away from participation. x. Lack of motivation to learn new skills in innovative, productive activities to generate income for the family. xi. Some women wait to see how the group goes, but afterwards, some don’t investigate to see if things are going well. xii. Low opinion of their own abilities to do something new and to make decisions. xiii. Little experience in assuming risks and making decisions. The control schools: The difference in organizational and community progress between control and intervention schools was notable. While WE groups did not exist in every school, the absence of any similar type of organization was more evident in the control schools. The women were allowed to sell foods to the schools even where there were no WE or other similar organizations, but the savings, loan, training, and empowerement attitudes were not found where this methodology did not exist. While organization models within the control schools were similar to those in intervention schools, they were much less organized, and they will take additional time to reach maturity. These included the OPF model, as required by the LAE, and the collaborating mothers in charge of food preparation and other school support. The team did not hold focus groups with the OPFs in the control schools, but they did interview the collaborating mothers and talk to the parents who were at the school at the time of the study. In general, only women were present, but they were timid and not very communicative. When a man was present, he was the one who spoke and commented about the day-to-day aspects of the school feeding process and the existing structures. Because the control schools did not have WE groups, the team used the collaborating mothers as the basis for comparison. The team found that the women faced challenges in the performance and improvement of their responsibilities in the school. The interviewers obtained this information in informal talks with the collaborating mothers who worked in the kitchen and store rooms, principals, teachers, and those in charge of supporting the mothers. In organizations: i. The women generally participated in church groups. ii. In some large schools, parents agreed to pay two or three mothers to be in charge of preparing the meals, since the others worked in their homes or in commercial activities such as making food and selling other products. iii. The teacher was responsible for providing the food preparation instructions when a mother was new or was substituting for another mother. iv. Some mothers who were OPF members were also paid to prepare the meals. v. The women generally didn’t have the opportunity to join organizations or to learn about empowerment and development, which would enable them to obtain additional leadership roles in the school and their community. This would also help them become more independent, while gaining the knowledge and skills that would allow them to become more active in their homes and to contribute to the family’s and the community’s well-being and development. 49 In the control schools, the collaborating mothers didn’t have the ability to organize themselves. In the majority of cases, a teacher was responsible for organizing the collaborating mothers’ shifts and would send them a notice when it was their turn. This affected other activities that the mothers were involved in, as they had to change their plans from one day to the next. Because of the short notice, some mothers paid someone the regular day labor rate to take their place, which meant that one of the teachers had to tell them step by step how to prepare the food. The collaborating mothers generally had their children in the kitchen with them while they prepared the foods, as they didn’t trust anyone enough to take care of them. Differences between schools with and without WE groups The women in schools with WE groups had more skills, participated more often, and shared their opinions freely. This was generally due to the training they received on strengthening their will, their potential for involvement in decision making, and behavior change for community development. In the schools without a WE group, the women were involved in some type of organization with the specific knowledge and skills that are traditional for women in home management. They didn’t have a vision for organization, participation skills, awareness of the need for savings, how to manage money, productive activities, the importance of active participation, and especially, convictions about the equality of men’s and women’s rights. WE groups are necessary in all communities: This type of organization is extremely important because it provides women with the opportunity to gain knowledge and because they can maximize their involvement in activities that promote the development of their community through the strengthening their family and their finances. Women who are trained have better opportunities for progress for themselves and for their whole family. Their children have better diets and are more encouraged and even obligated to go to school and grow intellectually. These groups are necessary and should be strengthened, with a vision for group organization and personal development. Advantages of communities with WE groups compared to control communities: 1. Women were organized. 2. Women were trained for decision-making and took care of themselves, especially when their spouses migrated and they were left in charge of the household. 3. Their knowledge about the possibility of conducting productive projects to obtain necessary resources was strengthened. 4. Women were more able to manage their savings and obtain loans through the group for investments or emergencies. 5. Women participated more and were able to share their opinions. 6. Women were able to become involved in the OPF or other organizations such as the church, the COCODE, etc. 5.4.6. School gardens The goal of school gardens is to help schools produce foods that can be consumed in the school feeding program, therefore saving the schools the cost of purchasing those foods and freeing those resources to be used to purchase other products. Additionally, it was used as a practical teaching resource for various classes, especially from fourth to sixth grades. Through the life of the project, Educamos has helped establish 134 gardens in 117 schools. Of the schools visited during the MTE, 20 had school gardens, two had gardens in the process of implementation, and eight did not have gardens. 50 The school gardens are especially important because they offer educational opportunities and produce for the school kitchens. They also generated synergies among the parents as they took turns tending them. Nevertheless, water, space, and the lack of resources were challenges for their growth. Two schools implemented gardens, meaning they made the effort to keep it functioning in the face of problems with water, pests, and the need for more training. The team found macro tunnels in only two locations. While this strategy was initially used in the schools, farmers in the community began to replicate it after seeing positive results. PCI coordinated its work with the MOA, which facilitates and coordinates technical assistance in the intervention schools. The existence of a garden tended to depend on the attitude of the principal, the teachers, and the OPF, as they were the ones who sought to maintain the garden without institutional support. The team found various abandoned gardens, where the schools produced one harvest of vegetables but stopped due to a lack of seedse, since PCI no longer distributed seeds. Those gardens looked abandoned, and the MOA no longer provided follow up. The difficulty in renting land also caused some gardens to be abandoned; for example, in Santa Clara Chejoj, the garden was abandoned after two years of production because the owners stopped renting the land to the school. In Gelná, the school said it did not get enough support from the parents to maintain the garden. Nevertheless, the methodology is important and can be copied, as happened in Yatziquin, where the WE group planted their own garden, even though the school didn’t have one. The gardens were generally small, although some covered up to one cuerda (441 square meters). Schools would need a larger plot of land, designated diversification in production and financial capital (to rent land, to invest in seeds, to invest in maintenance), and to learn to harvest water with local products (in-ground pits, plastic sheeting, etc.) where there was no water to yield a surplus from the gardens. According to a MOA official, their targets for the communities were unachievable. Community residents tended to produce a monoculture and were resistant to change. Remittances made it possible for them to buy a higher amount of low quality products rather than to invest in food security. In some cases, the population prefered to spend Q 1.00 on junk food rather than pay the same price for an egg. The official stated that a strength of PCI was that new promoters were regularly sought and trained, leading to more trained resources in the community. Coordination existed for some activities such as producers’ fairs and production exchanges with teachers for joint, local purchases. They saw that PCI participated in the COMUSAN, which presented the MOA, SESAN, and the majority of organizations present in the different municipalities the opportunity to coordinate. The gardens had an educational role and supplied the kitchens, but they did not focus on producing a surplus to generate income and become sustainable. However, the gardens were generally replicated in some households as family gardens. The following needs were mentioned in the focus groups that said they had gardens: i. Greenhouses are needed in cold climates ii. Pests sometimes ruined the crops iii. The gardens needed to be fenced so that chickens and other animals can’t get in to eat and damage the produce iv. Training for the management of garden issues (pests and disease) is needed v. Seeds are needed, as PCI no longer distributes them School gardens were very important to Huehuetenango MOE officials because they saw the learning and progress that students received through the learning by doing methodology. The students participated in the 51 harvest and saw the benefits and were then able to apply what they learned in their homes (when they had land to plant on). While the MOE has no specific school garden policy, based on the LAE, schools should promote gardens when they have the space, water, and teachers’ time. 5.4.7. Infrastructure PCI invested heavily in the schools to improve kitchens, stoves, sanitation services, water cisterns, and handwashing stations. Almost all the intervention schools the team visited had at least one infrastructure project. The principals and OPFs valued this assistance because they did not receive support from the government. With PCI’s help, they will apply to a MOE program for school refurbishment. The team observed that the infrastructure improved in quality when compared to previous investments. One particular area of improvement was the design of sanitation infrastructure with separate spaces for boys and girls. Illustration 36. Kitchen observations Smoke management particularly needs improvement because it affects the health of the collaborating mothers as well as the students when it reaches the classrooms. According to MOE data, support was provided for 20 construction and rehabilitation projects for school kitchens in 2018. Although water storage cisterns only work during the rainy season, it is important that they are provided in schools where water is scarce. More smoke was noted in control school kitchens than in intervention school kitchens, as well as poorer infrastructure either due to the type of construction (adobe walls, dirt floors, roofing sheets in poor condition) or the lack of furnishings for adequate kitchen utensil storage, which were then exposed to dust and dirt. In terms of hygiene, dogs were present in intervention and control communities, and this was more common when the school didn’t have a fence. The team observed dogs playing with the children during recess in at least five control schools. Some control schools were very orderly and clean, and this was generally related to the presence of a fence enclosure around the school. Significant differences in hygiene were seen between intervention and control schools, including the use of water storage recipients and the use of closed trash bins. 6. Conclusions a. General 1. From support for school feedings to investments in teacher trainings, community organizations, and behavior change, PCI changed the intervention communities’ realities in a way that improved the quality of life and created a better future. 2. The LAE’s (decree number 16-2017) impact involves an improvement in the quality of school feeding, with an increase in the number of students eating a minimum acceptable diet, both in intervention and control schools.. Constructive participation by the MOE together with PCI, other McGovern-Dole projects, and national and international organizations benefited LAE implementation. 3. The program’s accompaniment helped communities adopt good practices in health, nutrition, food preparation, and gardening. 4. The degree of organization in the EDUCAMOS intervention schools was greater than in the control schools, where the OPF and the school operated less efficiently. 40.7% 7.4% 3.7% 19.2% 33.3% 13.3% 16.7% 6.7% Humo en cocina Uso de fogones Espacio Reducido Infraestructura deficiente Control Intervención 52 5. The program received recognition from authorities and officials in the educational system and the STEG (teachers union) in the intervention area, which was positive for the implementation of related public policies. 6. The program’s successes and positive results should be shared, but the program lacks a communications strategy with key actors in the Huehuetenango educational environment, especially in the intervention area. 7. The intervention and control areas both have problems with primary school completion, as evidenced by the significant difference between the number of students enrolled in first grade and the number who graduated from sixth grade. See Annex 8.6 for the 2018 annual MOE statistics for intervention municipalities at the department and the national level. 8. PCI’s investments in school infrastructure were relevant and necessary, given the need that existed and the fact that no government entity took responsibility for these areas. Local governments intend to provide support, though the requests from PCI might come at an inopportune time. b. By Indicator 1. The results that measured the literacy strategy didn’t show significant progress, but the true impact of the program will be visible when the final evaluation measures the progress of students who are currently in first grade (which is a key grade) as they reach third grade. 2. Positive changes in the percentage of students who successfully completed the reading test between the BL and MTE were evident in the municipalities of San Pedro Necta, San Gaspar Ixchil, San Andrés Cuilco, Santa Eulalia, and Santa Cruz Barillas. 26 3. The percentage of teachers who showed improvements in literacy instruction, as identified by the principals, rose to 75%, well above the life of project target of 50%. 4. Given the challenges of managing a multi-grade classroom, multi-grade and one teacher schools found it more difficult to apply tools and new techniques for improved literacy instructions in first grade classes. 5. The turn over of teachers who were trained in literacy instruction to other grades negatively impacted the sustainability of the literacy improvement initiative in the first three grades and requires the program to conduct additional training for the new teachers. 6. The main reasons teachers were not present at least 90% of school days were because of their activities with supervisors, with CTAs about the school feedings, or with the STEG, which is an activity justified by article 17.3 of the teachers’ collective agreement with the MOE (the program cannot influence meetings with the STEG). 7. The fact that 984 of the 1,030 pre-school through third grade teachers who were trained applied new techniques or quality tools for literacy instruction in their classroom, exceeding the expected target of 700, demonstrated the efficacy, efficiency, and relevance of the teacher trainings. An estimated total of 231 principals also demonstrated that they used new techniques or tools. 8. The number of students who didn’t stay on task (who where distracted in class) decreased by 7.99 percentage points between the BL and MTE, showing the positive impacts of the program. 9. The percent of students who reported being hungry or very hungry increased from baseline to midterm. The likely explanation is that parents are saving money and not feeding their children breakfast at home since they will receive a mid-morning meal at school, possibly an indication of a deepening food security crisis for the most vulnerable households. 10. LAE implementation had a positive impact on the hunger levels found in the EDUCAMOS intervention schools. The Q4.00/student/day, together with the program’s contributions of rice, beans, oil, and corn soy blend, decreased the percentage of students who indicated that they were hungry or very hungry during the school day. The project’s target was for no more than 10% of students to express hunger, though that percentage was 14.40% at the time of the MTE. 26 However these results are illustrative only, not statistically significant. 53 11. The percentage of students who claimed that illness was the main reason for their absence increased from 7.7% at BL to 10.20% at the MTE. However, the communities were affected by a high incidence of ARIs and diarrhea during the data collection period, according to the Huehuetenango Health Area Directorate. 12. Student enrollment increased by 4.5% from the BL, while at the national level, the MOE reported a 3% increase. 13. The percentage of parents who could name at least three benefits of primary education was 92%, well above the 60% target, showing the efficacy and efficiency of the program’s parent training component. 14. A total of 84.20% of school aged children in the intervention schools received a minimum acceptable diet, compared to 45.1% in the BL, demonstrating that the feeding, health, and nutrition component of the program, along with LAE implementation, had positive effects for students. 15. The percentage of parents who could identify at least three important health/hygiene practices exceeded the target by 5.6 percentage points, demonstrating the effectiveness of the parent training component of the program. 16. The percentage of mothers who could identify at least three key safe food preparation and storage practices was 89.4%, above the project target of 85%. 17. All 294 intervention schools were found to be using an improved water source, 7.30% more than in control schools. 18. A total of 257 schools had improved sanitation services, exceeding the original project target of 149 schools. However, these sanitation services have a short lifespan, and they deteriorate quickly. 19. Program schools had a 19.8% impact compared to the control schools regarding the presence of better food preparation and storage equipment. However, the MTE found that 75% of schools have better equipment, still below the project target of 90%. 20. Despite the presence of an installed stove, some of the schools continued to cook with a fire on the ground for a variety of reasons, including deterioration, accidents, disuse, or insufficient space on the stoves, meaning that the food was prepared with some degree of risk. Animals were consistently present in the schools for a variety of reasons. c. Specific Topics 1. Some participants in the WE focus groups mentioned that they were not clear about the process they were participating in or why they were asked to join and that their group focused only on savings. 2. The WE program was very useful for training participants, not only because they discussed women’s rights, but because it is a comprehensive program that helps their personal growth as well as trains them in topics such as food preparation, care of children, income generation, and active participation in the home and community. 3. According to the OPF focus groups, the EDUCAMOS program was useful for the school in general because it helped parents, teachers, children, and the infrastructure. The parents were clearly happy and emphatic when they requested that the program continue, given its obvious benefits. 4. Men’s participation within the WE process was an oxymoron. On one hand, men’s involvement established a new dynamic and generated a benefit, but on the other hand, it could create discomfort among spouses and decrease women’s participation. 5. The school gardens faced serious production constraints, such as the absence of improved seeds, water scarcity, the invasion of animals that ruined the produce, limited space, and the effects of climate change (too much rain, drought). 6. PCI’s support for schools’ infrastructure was key in improving the learning environment for students and for the mothers who prepared the meals. In almost all schools, the team observed that some infrastructure was supported by PCI. The most important challenge facing the schools now is the maintenance to increase the lifespan of the projects. 7. In addition to serving as a means for key indicator data collection, the information corners boosted interactions between the school and the community and helped them resolve the problems they faced. 54 8. The virtual classrooms generated high expectations among teachers and students, and they are an effective tool for improving educational quality. 7. Recommendations a. General 1. It would be important to create a plan for training of trainers so that the schools have trained people to continue the accompaniment that PCI has been carrying out when the program ends. This will be a contribution to sustainability in the areas of health, organization, cleanliness, and seeking support similar to what they have had with EDUCAMOS. 2. It is important to maintain a degree of socialization with key members of the educational and productive community in the project intervention area so that they are informed and receptive to the objectives of the EDUCAMOS Program as well as others that might be strategically necessary. 3. To ensure support from local governments, close communication should be maintained with them and making sure to present project requests in a manner that coincides with budget preparation schedules. It is important to plan from one year to the next so that the projects can be programmed and budgeted as needed. b. By Indicators 1. An internal mechanism to measure students’ reading comprehension from first grade onward should be established to enable PCI to identify needed changes or adjust the program’s strategy as necessary. 2. The application of new literacy techniques should be strengthened so that students achieve positive reading results in the first two years. 3. The project should continue the literacy topics trainings, but teachers need to take ownership of it. 4. The literacy instruction techniques accompaniment from the education technicians should be reinforced. 5. Given that the multi-grade or single teacher school structures will not change in the near future, it is important to find a teaching method that will enable the teacher to introduce tools and new literacy techniques to students. 6. The reading corner strategy and school library should be strengthened, equipped, and supported, as they motivate students and could become standard methodology in classrooms and schools. 7. The meal service schedules should be reviewed to further reduce students’ hunger. Some schools serve a hot drink (atol) first thing in the morning and the formal breakfast at 10:00 a.m., which reduced hunger and could be replicated where it not currently the practice. 8. The OPFs need constant and repeated trainings so that they feel empowered in the school feedings to positively influence purchases so the schools purchase the greatest quantity of foods possible and so they can keep better accounts. 9. A constant and uniform water analysis methodology should be established within the intervention schools, especially during the rainy season, when health risks are greater. This should also be done to determine if the water supply has improved. 10. The sensitization of parents about the benefits of education should continue so that enrollment and school attendance is maintained and increased and to comply with parents’ expectations, 99.30% of whom consider learning to read and write as the the main benefit. 11. Continued training for collaborating mothers in key safe food preparation practices is needed for them to meet the high food preparation standards, especially because of the rotation of mothers on different days. 12. The cascade training strategy should be reviewed with the lead mothers because the trainings didn’t happen at the expected frequency and didn’t cover a large number of parents, which could affect the food preparation practices indicator for health and hygiene. 55 13. A contingency and maintenance plan for the sanitation services should be established due to their short lifespan and the speed at which they deteriorate. 14. A contingency plan to discontinue the practice of cooking student meals over open fires on the ground should be established for the schools that continue to do so. 15. Schools need to be fenced to avoid student contact with dogs, chickens, ducks, cats, and other animals. 16. The OPFs should generate internal regulations for the schools to prohibit students from bringing their dogs to school, for hygiene reasons and to stop the students from feeding the dog their portion of food. In one school in the Aldea Majchá of Soloma, the team noted a sign that said, “It is forbidden to bring your dog to school.” 17. Because some of the collaborating mothers cannot leave their children under three years of age at home or with a family member when they work in the kitchen, it might be possible to arrange a contained area nearby to keep them safe 18. A system for sharing information and outreach efforts with key actors from institutions that would add value to the EDUCAMOS Program should be created to help generate synergies and establish specific bilateral agreements. The STEG should be included so that they become a key strategic ally. c. By specific topic 1. At the beginning of each WE meeting, the philosophy of the groups should be reinforced and repeated so that it reaches all the participants. Additionally, they should continue to conduct social and political empowerment activities. 2. Given the positive impact that was observed in WE participants, invitations to join WE groups should be strengthened, and mechanisms overcome the obstacles that women cited should be sought. 3. A close link should be established between family gardens and the MOA so those with a family garden can address and guide the school gardens, as they have an important educational role, and to help it become a small food supplier for the school kitchens. 4. PCI should coordinate with the MOH to promote community campaigns to share information on hygienic practices at all levels, including in the school and at home, to prevent ARIs and diarrhea. 5. Schools should be trained on preventive maintenance of the virtual classroom equipment and negotiating opportunities to be able to replace damaged equipment or to increase the number of tablets available should be facilitated. 6. The school information corner should be shared with the MOE so it can be institutionalized in other schools, given the benefits of its use and the interactions it facilitated with the educational community. 56 8. Annexes 8.1 School Selection Process 8.1.1 Intervention schools The program works directly with the schools listed in the following table. A percentage weight was given to each municipality based on the current number of students, and from that number, the evaluators determined how many schools would be selected per municipality to ensure that they were proportional to the size of the municipality. These schools were selected using the program ALAZAR.INFO, which uses random numbers. Number of intervention schools per municipality Municipality No. Students % Weight Nr. Schools Sample San Andrés Cuilco 8,962 23.35% 90 7 San Gaspar Ixchil 1,913 4.98% 12 1 San Pedro Necta 4,689 12.22% 38 4 San Pedro Soloma 5,919 15.42% 34 5 Santa Eulalia 7,367 19.19% 45 6 Santa Cruz Barillas 9,537 24.84 75 7 Total 38,387 100 294 30 Source: Prepared for this study 8.1.2 Control schools The following table lists the comparison schools each of which had similar attributes to the intervention schools in terms of geographic location, language, number of students, Human Development Index (HDI), health, education, and income. The team explained the “pairing” process to the MOE and emphasized the two basic criteria: homogeneity with the intervention schools and that no other support organizations were present. The MOE gave official approval for the study and requested support from the CTAs to inform the schools’ principals in their municipalities about the study. The following table presents the characteristics of the intervention and control municipalities. 57 Table of Characterization of Intervention and Control Municipalities School Number Type Municipality No. Inhabitants IDH Health Education Income Language 7 Intervention San Andrés Cuilco 46,407 0.562 0.7 0.488 0.497 Mam 7 Control 1 Intervention San Gaspar Ixchil 7,485 0.516 0.694 0.399 0.454 Mam 0 Control 4 Intervention San Pedro Necta 26,025 0.562 0.666 0.507 0.512 Mam 4 Control 5 Intervention San Pedro Soloma 59,209 0.627 0.912 0.450 0.518 Q´anjob´al 5 Control 6 Intervention Santa Eulalia 30,102 0.482 0.555 0.406 0.485 Q´anjob´al 6 Control 7 Intervention Santa Cruz Barillas 127,170 0.562 0.751 0.444 0.491 Q´anjob´al 7 Control 1 Control San Sebastian Huehuetenango 21,198 0.508 0.588 0.427 0.509 Mam The urban control school John F. Kennedy in Soloma was excluded from the literacy results because it distorted the findings due to the fact that all the intervention schools in the MTE were rural. This school was selected as the control for Najab, which had a large number of students but a lower literacy performance due to its rural location. No significant differences were found in the other MTE results, so John F. Kennedy was only excluded from the reading test. 8.1.3 Sample The following table lists the intervention and control schools’ samples: People reached during the research Sample Population Per Intervention Sschool Control Schools Total Achieved BL MT BL MT BL MT Students grades 4-6 401 586 401 563 802 1,146 First grade students 488 496 615 300 1,103 796 Students graduated from 2nd grade 509 509 456 351 965 860 Parents 177 120 76 64 253 184 Teachers 54 45 49 27 103 72 Directors 30 28 30 28 60 56 Collaborating Mothers 75 119 34 52 119 171 PTA FGDs 30 30 WE FGDs 20 20 CTA 3 3 3 2 6 5 MAdvisors 3 3 Source: Prepared for this study 8.2 Training schedules for staff and ethics considerations The specific training schedule for the team implementing the study is listed below: i. July 5: Training of evaluation staff in EQSA Quetzaltenango offices 58 ii. July 10: Training with J&A in PCI Guatemala City offices iii. July 11-12, 18: Training of evaluation staff in Huehuetenango iv. July 19: Validation of instruments in school in San Pedro Necta v. July 22 – August 15: Field work in 60 schools in seven municipalities 8.2.1 Ethics considerations To guarantee an ethical data collection approach and to ensure confidentiality in data management, the following rights were adapted27 and used in this study: i. Right to Choose: Everyone who was asked to participate in the study was given the opportunity to decide whether they wanted to participate or not and was allowed to end their participation at any moment (even after giving consent) or to refuse to respond to any uncomfortable question. If they decided not to participate, they were told it was okay and that there would be no negative consequences. ii. Right to safety: Participants were protected from physical and psychological harm, and although no sensitive questions were included on the test, no one was pressured to provide a response. They were also not stigmatized based on any of their responses. iii. Right to be informed: The participants were informed about all of the study objectives, the time required for the survey, and the use of data, which allowed the participant to make an informed decision about whether to participate or not in the study. iv. Right to privacy: Everyone’s personal information and responses remained protected and private. Individual information was not revealed in any way, as the evaluation only analyzed aggregate results. v. Benefits: The people interviewed did not receive any benefit or payment for their participation in the study, neither in cash or in kind. vi. Informed consent. All participants provided their informed consent before beginning the interviews, and all were informed about: the objectives of the study, procedures, voluntary participation, and confidentiality. The consent process was carried out individually, and approval was obtained verbally. The investigators ensured that the participants understood the objective prior to initiating the activity. 8.3 Constraints 8.3.1 Time and communication a. The team suffered setbacks in their request for MOE approval to administer the reading test, and in the future, requests should be initiated with sufficient lead time. b. Poor communication among the MOE’s CTAs meant their respective control school principals were not informed of the reading tests or non-programmed fairs and invitations in a timely manner. Due to the poor communication, the research had to be reprogrammed in two schools of Santa Cruz Barillas, three in Santa Eulalia, one in San Gaspar Ixchil, and four in Cuilco. c. The teachers seemed to fear evaluation. The researchers’ presence in control schools concerned the union, and because it was not previously informed of the study, it ordered the schools to prohibit entry. After conversations with PCI, the evaluation team was allowed entry, though no written guarantee was provided, and the team was forbidden to administer the reading tests. The team also faced resistance from the teachers in providing information that might be shared with the MOE. After some time, these challenges were overcome so that the data could be collected. 8.3.2 With the schools a. The time allotted for field work was affected by official holidays, participation in school activities, the mobilization of principals and teachers for official internal processes, and STEG activities. None 27 Adapted from the rights of participants promoted by the Marketing Research Association, USA. 59 of these activities were considered absences because they were authorized by the MOE, based on the collective agreement with the teachers’ union. b. On some community access roads, the population charged a “road maintenance” fee, which was as high as Q200.00 at times. This is illegal but used by some community leaders, particularly in the municipality of Barillas. 8.3.3 Knowledge gaps a. A school’s location significantly influenced the results. The more rural28 a school was, the poorer the literacy learning results. In some schools, third grade students had serious reading comprehension difficulties. b. Some teachers only spoke Spanish in totally bilingual areas, which inhibits good communication with the students. 8.4 Additional Analysis of Difference in Differences This section presents DD tables for the indicators for which it was applicable. DD is an econometric method for quasi-experimental impact evaluations that measures the effect that an activity had in the intervention schools compared to the same activity in the control schools, which serves as the comparison parameter over time. The term difference in differences refers to the fact that this method calculates the effect of the intervention as a subtraction between two subtractions. The first subtraction is the treatment group’s result after receiving the intervention minus the treatment group’s result before treatment. The second subtraction is the result of the comparison group after treatment (though they weren’t treated) minus the comparison group’s result prior to “treatment.” The first subtraction is then subtracted from the second to calculate the difference in differences. Thus, DD calculates the degree of change over time of the treatment group compared to the degree of change in the comparison group. 8.4.1 Reading test Percent of students at the end of grade two who demonstrate that they can read and understand text at their grade level Intervention Control Baseline 32.60% 24.70% Midterm 30.80% 33.30% Difference -1.80% 8.60% Difference in Difference -10.40% The urban control school in Soloma was excluded from the analysis because it distorted the literacy results due to the fact that all the intervention schools in the MTE were rural. This school was selected initially as the comparison for Najab, which had a large number of students but lower literacy results due to its rural location. No significant differences were found in other MTE results. 28 The definition of urban according to the 2002 Census is the population that lives in cities, towns and departmental or municipal seats. Rural is defined as areas which are limited in economic growth, with precarious access routes, and that receive minimal services and influence from urban areas. 60 8.4.2 School aged children enrolled in schools Percent of school-aged children enrolled in primary school as a result of USDA assistance Intervention Control Baseline 95.00% 92.80% Midterm 93.53% 86.39% Difference -1.47% -6.41% Difference in Difference 4.94% Source: Prepared for this study 8.4.3 Health related absences For health-related absences, the -1.70% score indicates a positive impact in intervention schools because it is a lower comparative proportion of students who reported illness as the main cause of school absences in control schools. Percent of students who report that illness is the primary reason for school absence Control Intervention Baseline 7.5% 7.7% Midterm 10.2% 8.7% Difference 2.7% 1% Difference in Difference -1.7% 8.4.4 Parents who can name at least three benefits of primary education Percent of parents who can name at least three benefits of primary education Intervention Control Baseline 84.60% 50.00% Midterm 92.00% 79.70% Difference 7.40% 29.70% Difference in Difference -22.60% Source: Prepared for this study 8.4.5 School aged children who receive a minimum acceptable diet Percent of school age children who receive a mínimum aceptable diet Intervention Control Baseline 45.10% 29.90% Midterm 84.20% 82.90% Difference 39.10% 53.00% Difference in Difference -13.90% Source: Prepared for this study 61 8.4.6 Percentage of parents who can identify at least three important health / hygiene practices The DD analysis presented a negative number as program impact due to the significant increase in parents in the control schools. Percent of parents who can identify at least three important health and nutrition practices Intervention Control Baseline 93.20% 59.20% Midterm 90.60% 85.50% Difference -2.60% 26.30% Difference in Difference -28.90% Source: Prepared for this study 8.4.7 Percentage of mothers (or service providers) who can identify at least three key practices for safe food preparation and storage Percentage of mothers (or service providers) who can identify at least three key practices for safe food preparation and storage Intervention Control Baseline 99.10% 79.40% Midterm 89.40% 86.30% Difference -9.70% 6.90% Difference in Difference -16.60% Source: Prepared for this study The DD analysis established that the mothers in control schools improved in their ability to name three safe food preparation and storage practices, while the percentage of mothers in the intervention schools decreased. Because of this, it will be important to reinforce the topics with the current collaborating mothers. 8.4.8 Number of schools using an improved water source Number of schools using an improved water source Intervención Control Baseline 86.70% 90.00% Midterm 100.00% 96.00% Difference 13.30% 6.00% Difference in Difference 7.30% Source: Prepared for this study 8.5. Additional Tables and Graphs 62 8.5.1 Table of literacy achievement by municipality Barillas Santa Eulalia Soloma Cuilco San Gaspar Ixchil San P. Necta Int. Ctrl. Int. Ctrl. Int. Ctrl. Int. Ctrl. Int. Ctrl.1 Int. Ctrl. Achieved 76.0% 73.8% 70.8% 97.2% 57.7% 52.7% 54.2% 52.0% 73.7% 81.6% 92.3% 60.0% Did No Achieve 24.0% 26.2% 29.2% 2.8% 42.3% 47.3% 45.8% 48.0% 26.3% 18.4% 7.7% 40.0% 1: Se incluyen los resultados de San Sebastián H, como escuela de control. Elaboración propia. Fuente trabajo de campo 8.5.2 Table of other causes for student absences Sex Work in home (%) Work for someone else (%) Don’t have money (%) Family travel (%) Intervention Male 4 0.4 1.5 3.3 Female 5.1 0.3 1.4 2.7 Control Male 1.7 1 0.7 4.9 Female 1.5 0 1.1 3 Elaboración propia. Fuente trabajo de camp 8.5.4 Table on Training Received by Teachers in Intervention Schools When asked what kind of support they received from PCI to improve their teaching knowledge and abilities, the teachers could respond spontaneously with various options. The below table groups a multiple response question so the sum of all responses can be greated than 100%. Responses Number Percent of Teachers Pedagogical assessment 37 82.20% Use of Big Books 7 15.60% CREA Training 5 11.10% Training in use of rota folio 5 11.10% Literacy diploma 5 11.10% Did not receive anything 4 8.90% Training in school gardens 3 6.70% School feeding and hygiene 3 6.70% Administration 2 4.40% Reading corners 2 4.40% Leadership 1 2.20% 8.5.5 Early literacy learning All the concepts described in this section, from subsection “a” to subsection “e,” were taken from the “Evaluation of Initial Reading in Second Elementary Students” document published by the General Direction of Educational Evaluation and Research (DIGEDUCA) branch of the MOE in 2017, which presents students’ reading results at the end of second grade according to the ELGI test. a. Oral Language: It is the ability to understand and have a command of the oral language, including the phonological component, vocabulary, morphology, grammar and pragmatic skills. 63 b. Alphabetical Principle: It consists in knowing that graphemes have one or more phonemes that correspond to them and vice versa, that is, that the letters have sounds and that these can be written. It is the codification of the sounds of the language in concrete symbols, therefore it is unique for each language. c. Decoding: Use the relationship between grapheme and phoneme to read unknown words in their written form, but which generally exist in the student's oral lexicon. To decode it is necessary for the student to apply the alphabetic principle through a series of phonological skills that facilitate linguistic processing through sensitization to complex phonological or acoustic information. d. Phonological Awareness: It refers to the sensitivity you have to the internal phonological structure of words and that, unlike simply listening to spoken language, is more explicit. Students should realize that words are made up of smaller and simpler units that are phonemes. Phonological awareness manifests itself through the ability of students to identify and manipulate the sounds of words presented orally. Some reasons have been identified that explain how phonological awareness affects reading fluency, helps to understand the alphabetic principle and helps students notice the regular ways in which letters represent the sounds that form words. e. Fluency: It is the ability to read quickly accurately and expressively. The ability to read is the result of the automation of the lower processes, word recognition and decoding since it frees attention resources to devote to the higher processes, the understanding. For them, fluency is considered an indicator of word recognition skills in text comprehension. The following concepts were obtained from the document “Teaching Reading Comprehension,” as published by the MINEDUC in 2017, with USAID funding for the Mam language. a. Reading Comprehension: It is a process that involves a set of activities that when practiced repeatedly and progressively starting with the easy and advancing to the difficult ones will make the reader demonstrate their understanding. It is the main purpose of reading and is the result of the application of strategies to learn, remember and find the meaning of what has been read. b. Linguistic profile: This is an evaluation that allows you to determine the proficiency of one or more languages that the student has. The information obtained serves the teacher to determine in which language the reading will teach. 64 8.5.6 Students who are distracted in class, by intervention municipality 8.5.7 Students who indicate that they are hungry, by municipality 8.5.8 Water by type of sanitary service Greater water availability was found in intervention schools that had a sewage system when compared to control schools with a sewage system (90% and 81.3% respectively), but control schools with a septic system had more water than intervention schools with a septic system. 10.48 33.33 26.67 38.24 20.26 9.13 13 54 29.6 32.1 23.6 8.3 Cuilco San Gaspar Ixchil San Pedro Necta Soloma Barillas Santa Eulalia Intervención Control 16.1% 26.0% 19.0% 13.3% 29.7% 35.0% 15.1% 10.4% 12.5% 10.7% 23.5% 19.0% Barillas Santa Eulalia Soloma Cuilco San Pedro Necta San Gaspar Ixchil Control Intervención 65 8.6 Education Statistics by Intervention Municipality 8.6.1 Santa Cruz Barillas MINEDUC Data 2018 Enrolled Repeated Did not repeat Final Enrollment Promoted Not Promoted Retired Total 26.136 2.181 23.955 23.627 20.917 2.710 2.509 Pre-primary 2.160 0 2.160 2.002 2.002 0 158 Primary 18.785 2.128 16.657 17.183 14.849 2.334 1.602 Adult Primary 34 0 34 25 25 0 9 Basic 3.743 42 3.701 3.158 2.890 268 585 Diversified 1.414 11 1.403 1.259 1.151 108 155 Fuente: Anuario Estadístico de la Educación 8.6.2 Cuilco Enrolled Repeated Did not repeat Final Enrollment Promoted Not Promoted Retired Total 12.386 956 11.430 11.012 9.669 1.343 1.374 Pre-primary 1.200 0 1.200 1.085 1.085 0 115 Primary 9.940 931 9.009 8.820 7.672 1.148 1.120 Adult Primary 989 21 968 878 704 174 111 Basic 257 4 253 229 208 21 28 Fuente: Anuario Estadístico de la Educación 81.3% 90.0% 75.0% 62.5% Control Intervención Sistema de alcantarillado Sistema septico 66 8.6.3 San Gaspar Ixchil Enrolled Repeated Did not repeat Final Enrollment Promoted Not Promoted Retired Total 2.003 211 1.792 1.790 1.553 237 213 Pre-primary 257 0 257 219 219 0 38 Primary 1.652 211 1.441 1.486 1.255 231 166 Basic 94 0 94 85 79 6 9 Fuente: Anuario Estadístico de la Educación 8.6.4 San Pedro Necta Enrolled Repeated Did not repeat Final Enrollment Promoted Not Promoted Retired Total 8.920 687 8.233 8.372 7.440 932 548 Pre-primary 1.219 0 1.219 1.133 1.133 0 86 Primary 6.385 669 5.716 5.992 5.308 684 393 Basic 893 14 879 839 670 169 54 Diversified 423 4 419 408 329 79 15 Fuente: Anuario Estadístico de la Educación 8.6.5 Santa Eulalia Enrolled Repeated Did not repeat Final Enrollment Promoted Not Promoted Retired Total 10.929 1.138 9.791 10.232 8.804 1.428 697 Pre-primary 1.242 0 1.242 1.162 1.162 0 80 Primary 7.957 1.095 6.862 7.420 6.220 1.200 537 Basic 1.396 36 1.360 1.319 1.156 163 77 Diversified 334 7 327 331 266 65 3 Fuente: Anuario Estadístico de la Educación 67 8.6.6 San Pedro Soloma Enrolled Repeated Did not repeat Final Enrollment Promoted Not Promoted Retired Total 12.990 1.231 11.759 11.667 9.853 1.814 1.323 Preprimaria 1.549 0 1.549 1.408 1.408 0 141 Primaria 9.317 1.160 8.157 8.369 6.968 1.401 948 Básico 1.557 70 1.487 1.369 1.033 336 188 Diversificado 567 1 566 521 444 77 46 Fuente: Anuario Estadístico de la Educación 8.6.7 National Enrolled Repeated Did not repeat Final Enrollment Promoted Not Promoted Retired Total 4.239.233 264.499 3.974.734 4.012.646 3.509.052 503.594 226.587 Pre-primary 626.638 1 626.637 596.377 596.377 0 30.261 Primary 2.362.116 212.915 2.149.201 2.251.553 1.995.774 255.779 110.563 Adult primary 23.268 1.227 22.041 18.606 15.568 3.038 4.662 Basic 805.077 41.422 763.655 749.121 566.458 182.663 55.956 Diversified 422.134 8.934 413.200 396.989 334.875 62.114 25.145 Fuente: Anuario Estadístico de la Educación 8.6.8 Huehuetenango Enrolled Repeated Did not repeat Final Enrollment Promoted Not Promoted Retired Total 294.221 23.133 271.088 270.736 238.199 32.537 23.485 Pre-primary 37.401 0 37.401 34.205 34.205 0 3.196 Primary 204.345 21.608 182.737 188.931 164.115 24.816 15.414 Adult primary 514 13 501 406 386 20 108 Basic 35.857 1.319 34.538 32.293 26.432 5.861 3.564 Diversified 16.104 193 15.911 14.901 13.061 1.840 1.203 Fuente: Anuario Estadístico de la Educación 8.7 DIGEDUCA Reading comprehension tests The main test used by the DIGEDUCA to measure reading comprehension was created for third and sixth graders. However, the MOE administered the test to students at the end of second grade in 2017, and DIGEDUCA published the results in the document entitled “Evaluation of Initial Reading in Second Grade Students.” The team used the ELGI test for the MTE, which is an adaptation of the EGRA, a test created in 68 the United States to evaluate the degree of fluidity and reading comprehension in early grades. The ELGI was designed to evaluate reading performance, regardless of the instruction method used to teach the students. The tasks involved in the test are based on the five essential reading components identified by the National Reading Panel of the United States: phonemes, phonological awareness, vocabulary, fluidity, and comprehension. The EGRA test itself was not translated into Spanish, but the Spanish language’s structural aspects were incorporated, according to the document published by DIGEDUCA. Based on that document’s conclusions, the best literacy results in Guatemala were found in Sacatepéquez, Quetzaltenango, Guatemala City, Chimaltenango and Santa Rosa. Jalapa, Chiquimula, Huehuetenango, Quiché and Peten had the lowest performance results. The national literacy results according to the data published by DIGEDUCA in May 2018 in the document, “Model for explaining and predicting literacy learning for Guatemala” are presented below: Source: DIGEDUCA 2015 The data produced by DIGEDUCA are not comparable to the MTE data because each followed a different methodology. DIGEDUCA measured the five separate stages of learning: a) Emerging, where students have not learned formal reading, b) Initial, where the have begun to learn letters, c) Decoding, where they can join letters to form words, d) Automated, where they develop fluidity through constant practice, and e) Fluid, where the children can read with fluidity. The annexes of the document called, “Model for explaining and predicting literacy learning for Guatemala” presents the percent of children in each literacy stage by department and municipality. The full document is attached as independent Annex No. 2. The data for each control and intervention school municipality are presented in the table below: Municipality Emerging Decoding Automated Development of Fluidity Fluidity Cuilco 11% 50% 33% 6% 0% San Gaspar Ixchil 11% 83% 0% 0% 6% San Pedro Necta 6% 38% 6% 6% 44% San Pedro Soloma 22% 44% 11% 6% 17% Santa Cruz Barillas 39% 47% 11% 3% 0% Santa Eulalia 56% 17% 11% 17% 0% San Sebastián Huehuetenango 11% 61% 6% 17% 6% Source DIGEDUCA 2018 8.8. Maps related to water supply After evaluating Guatemala’s existing water supply and basic management in light of climate change considerations, the Agriculture, Natural Resources and Environment Institute (IARNA) of the URL published the document, “Hydric balance of the sub-watersheds of the Republic of Guatemala” in July 2015, which found that “the watersheds that show a medium decrease in hydrological supply by 2020 are mainly located in the dry corridor around the volcanic chain (eastern sector), in the department of Huehuetenango and in the southern part of Petén.” 69 The watersheds with the greatest hydrological supply had two million m3 / km2 / year or were in the range of 1.5 to 2 million m3 . Watersheds with a medium supply had one to 1.5 million, those with a low water supply had 500,000 to one million, and the watersheds with the most alarming supply had less than 500 thousand. A high number of days without rain leads to harvest losses throughout the country. According to data from the National Institute of of Seismology, Volcanology, Meteorology, and Hydrology (INSIVUMEH), the <500,000 500,000 – 1 millón 1 – 1.5 millones 1.5 – 2 millones >2 millones 70 MOA, and the United Nations Office for Humanitarian Affairs, Huehuetenango was particularly affected by poorer rainfall in 2018, specifically from June 1 to August 31. 8.9 Lessons learned Lessons learned: 1. Regarding organization a. The research team should contact the STEG at the highest level in each region where data collection is planned (the key person varies), to ensure collaboration and obtain approvals whenever an evaluation using control schools is planned. This should be done well before field work is scheduled to start, and written approval should be obtained. b. Each participating institution’s schedule should be considered to ensure that each procedure has sufficient time to be completed. Each procedure is necessary to complete the field work, and a project will be delayed as long as approvals are in process. This was especially a problem with requests presented to the MOE. c. Technicians should map out possible conflicts in the schools, based on their knowledge of their work area or the sector under their responsibility, and prepare an alternate plan in case they encounter a Days without Rain Source: INSIVUMEH, MOA, OCHA 71 negative reaction or resistance to the field work, especially in control schools. While they may not know the internal workings of the control schools, they could obtain information that would anticipate such a conflict. d. Given that conditions might change during field work, close communication should be maintained with the respective CTAs working in the intervention and control school areas to prevent delays due to rescheduling needs. If routes are rescheduled in a timely manner, the study team can avoid delays in the finalization of the project. e. Strong communication between PCI supervisors and CTAs allows work to continue without setbacks, even when adverse conditions arise. For example, in Soloma, good communication facilitated the completion of the field work as planned. 2. Regarding the process a. The team enlisted a series of guides who knew the location of the schools to ensure timely arrivals to carry out the work in each study area. This should be maintained and improved so that the guides assigned to the research team can check the route ahead of time to learn about the road conditions and avoid delays. b. Despite explanations from the team about the objective of the literacy test for third grade students and the emphasis that it was not an evaluation of the teachers, the teachers resisted the literacy tests, especially in the control schools. A written statement about the tests, including the objectives and the fact that there neither the students nor the teachers will be graded, should be given to the staff to further ease their minds. c. Particularly in bilingual communities where Spanish is the second language, the third-grade students’ colloquial language and skills did not meet the level of accomplishment required by the test. The outcomes were further diminished by the inability to use an interpreter when providing the instructions (per the MOE protocol) and the nervousness related to an unexpected test administered by strangers rather than the teacher. d. The literacy tests included some words that were easily known to those with access to communication media and who spoke Spanish as their first language, but they were not known to those who learned Spanish as a second language; for example, “corchetes,” “enfadado,” “melindrosa,” “aislada,” and “sanción.” e. Option “c” was the correct answer for the two examples in the literacy test, which might influence students with low Spanish language skills to interpret that the correct answer will always be the same. f. The team had to always turn off the social networks when collecting information in remote areas using Mobenzi because the program transmitted information automatically, and when the signal was not stable, it generated a lot of junk that had to be cleaned up later. g. The school menus approved by the MOE made it possible for students to eat a variety of meals without repetition over at least 10 days. This was successful mainly in the intervention schools. h. It is important to use information recorders who speak the local language when conducting interviews with students from the fourth, fifth, and sixth grades, especially for girls, who tended to be very shy and needed encouragement to speak during the interviews. 72 i. Some students had some type of disability, and the teachers didn’t have the knowledge to be able to help them or improve their situation. j. Although corn soy blend is a complete food, the students sometimes don’t enjoy the hot drink it is often made into, so trainings on different ways to prepare corn soy blend would help increase its acceptance in the schools. k. Most of schools provided the research team a mid-morning snack. However, the team observed better food preparation practices in the intervention schools than in the control schools, where some of the team members experienced intestinal problems. l. Internal (or external) staff could conduct another evaluation in one year in the same schools to see progress and reversals. This evaluation could use a sample of the sample to reduce the workload and should obtain specific permissions from the MOE to administer the literacy test, which would serve as another parameter prior to the final evaluation. 3. Regarding the community a. Direct contact with the president of the OPF in each intervention school to be visited should be established, given that the principal or assistant principal did not communicate the invitation in some places, causing the team to send students to fetch the ones who were not at the school that day. The consultant should contact the OPF at least two days prior to the visit to confirm their availability. b. The study led some people in the control schools, such as collaborating mothers, principals, and teachers, to expect future support from PCI. Although the team clearly explained that the visit was merely for statistical research, they still expected that they might receive support in the future. c. The principal or the school feeding commission generally makes the purchasing decisions in schools, making the OPF laborers, so their role in the process as defined by the LAE should be strengthened. d. Migration in the communities was directly related to employment opportunities and income generation and did not depend on a good education or a good diet for children. The children became an instrument to facilitate acceptance at the US border. e. The closer a school was to urban areas, the better their literacy test results, due to the influence of different communication media, greater access to reading materials, and because Spanish is generally spoken more. The literacy test was only available in Spanish. f. Hygiene practices improved when the schools were fenced and when the school government intervened to monitor cleanliness in the different school environments. In intervention schools, the school government used the “Cleanliness Traffic Light” tool as a checklist for maintaining school hygiene. The difference in hygiene levels was very obvious between the intervention and control schools. 8.10 Success Stories In conjunction with PCI Monitoring and Evaluation staff, the team designed the following matrix to document the good practices or success stories that were identified during the field research. 73 1. School Gardens Name of the good practice or success story What is it? School gardens when there isn’t much space, in Palestina la Unión School How was it done? By using recycled materials and hanging containers to have more space What problem was solved or addressed? Lack of space for school garden How did PCI intervene? Provided training for teachers and promoted its use in the municipality of Santa Cruz Barillas Who participated? PCI technician, teachers and students Why is it considered different/innova tive/practical? In many schools they say there is no space or water to create a garden, so this modality provides an opportunity to have a school garden and motivates students to replicate this action at home. Photographs Hanging gardens Ease of care and use of few resources 74 Name of the good practice or success story What is it? Use of macro tunnels in Temux Grande school, Santa Eulalia How was it done? By using macro tunnels in the schools, the community was motivated to replicate it to generate income What problem was solved or addressed? Family finances, community empowerment How did PCI intervene? Creating macro tunnels in the schools and training parents Who participated? PCI agronomy technician, teachers, students, OPF and community in general Why is it considered different/inno vative/practic al? The parents thought it was innovative because it could be done in a small space and they can do it in their homes without having large plots of land for crops. Photographs Macro tunnel outside the school Replica of macro tunnel in the community Name of the good practice or success story What is it? Negotiation to borrow land for the garden in Buena Vista el Boquerón School of San Pedro Necta How was it done? The OPF and school administration negotiated with a neighbor so that he would loan them an adjacent piece of land to use for the school garden 75 What problem was solved or addressed ? Lack of land for the school garden How did PCI intervene? Promoting the creation of a garden Who participated? Principal, OPF Why is it considered different/inn ovative/prac tical? It solved a constant problem in the schools that is the lack of land for planting to use its production for the school feeding program as well as the educational aspect of the garden Photographs View of garden signage The principal and some children in the garden 76 2. Reading Corner Name of the good practice or success story What is it? Educational corners in El Zapotillo School, Cuilco How was it done? Replicating the reading corners promoted by PCI for other subjects What problem was solved or addressed? Motivates students by being more participatory, reinforces the topics that are learned How did PCI intervene? Providing resources in the CREA Who participates? Teachers and students Why is it considered different/inn ovative/pract ical? Because they replicated the reading corners model and extended it to other subjects Photographs Recycling materials Math Corner 77 3. IEP (Institutional Educational Project) Model Name of the good practice or success story What is it? IEP in Cuá School, Cuilco How was it done? By means of a community assessment, involved different actors from the educational community to carry out different improvement areas What problem was solved or addressed? Educational, administrative and infrastructure short, medium and long term problems How did PCI intervene? This practice was found in a control school. Who participated? Teachers and parents Why is it considered different/inn ovative/pract ical? It involves all the actors and good results were observed in all the areas of the school. Photographs Magazine with news about the IEP implementation Computer lab obtained by means of the IEP 78 4. Sanitation services Name of the good practice or success story What is it? Use of trash can and air freshener for bathrooms in Yulacheque el Quetzal School How was it done? By promoting the use of covered trash cans in bathrooms and the use of air fresheners on the walls of the bathrooms What problem was solved or addressed? Cleanliness and odor in the bathrooms How did PCI intervene? Promoting cleanliness practices and improving bathroom infrastructure Who participated? PCI, teachers, OPF and students Why is it considered different/inn ovative/pract ical? Because there were no trash cans observed in the bathrooms of most control and intervention schools and it also dealt with odors in the bathrooms. Photographs Outside of bathrooms Use of trash can and air freshener 79 5. Virtual Classroom Name of the good practice or success story What is it? Involvement of the OPF to improve and bring internet signal to the Buena Vista el Boquerón School of San Pedro Necta How was it done? Parents organized to support having internet in the school What problem was solved or addressed? Connectivity for the equipment that was provided How did PCI intervene? Contributing a virtual classroom to the school Who participated? PCI, teachers, OPF Why is it considered different/inn ovative/pract ical? Because of the involvement of the parents with the computer classroom where they were motivated to support the school to receive an internet signal at the school and also learn how to use the electronic equipment along with their children. Photographs View of the virtual classroom OPF involved in the project 80 6. Child with a special case Name of the good practice or success story What is it? Managing to control and help a child with mental problems to progress. Snenajab School, Soloma. How was it done? By means of the dedication of the teacher in a one-teacher school What problem was solved or addressed? Individual development of a child with mental problems How did PCI intervene? Should promote a course for cases of children with mental problems Who participated? Principal / student Why is it considered different/inn ovative/pract ical? Because without this support from the teacher the child would not make progress and be left without aid Photographs Child with mental problems Child looks happy to be able to interact with his friends 81 7. Dining hall for students Name of the good practice or success story What is it? Construction of a dining hall that improves conditions for children eating at Peñas Blancas School of the municipality of Barillas How was it done? Given the limited space that belongs to the school, the students do not have an adequate place for school feeding, so by means of agreements, they managed to build an elevated dining hall. What problem was solved or addressed? An adequate and hygienic place for students to eat the school meals How did PCI intervene? Directly supporting construction of the dining hall Who participated? OPF, Principal and PCI Why is it considered different/inn ovative/pract ical? Because the construction adds space to the school Photographs The dining hall under construction The dining hall under construction 82 9. Related Annexes 9.1 Tables 9.1.1 Table of summary of indicators No. Indicator ID Indicator Baseline Midterm 1 McGovern￾Dole #26 Percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text 32.6% 30.8% 2 Custom Percent of teachers in target schools who demonstrate improved literacy instruction as identified by supervisors, mentors, or coaches 0% 75% 3 Custom Percent of schools in which at least 80 percent of teachers were present on 90 percent of school days 100% 83.33% 4 McGovern￾Dole #5 Number of teachers/educators/teaching assistants in target schools who demonstrate use of new and quality teaching techniques or tools as a result of USDA assistance 0 984 5 McGovern￾Dole #3 Number of school administrators in targeted schools who demonstrate use of new techniques or tools as a result of USDA assistance 0 231 6 Custom Percent of students that do not do their homework 27.45% 19.46% 7 Custom Percent of students who indicate that they are “hungry” or “very hungry” during the school day 8.5% 14.40% 8 Custom Percent of schools in which at least 80 percent of teachers were present on 90 percent of school days 97.7% 91.69% 9 Custom Percent of students who state that illness is the primary reason they are absent from school 7.7% 8.70% 10 Custom Percent of primary school aged children enrolled in USDA assisted schools 95% 93.53% 11 Custom Percent of parents in target communities who can name at least three benefits of primary education 84.6% 92.0% 12 Custom Percent of school-age children who receive a mínimum aceptable diet 45.1% 84.20% 13 Custom Percent of parents in targeted communities who can identify at least three important health/ hygiene practices 93.2% 90.60% 14 Personalizado Percent of mothers (or care takers) who can identify at least three key food preparation and storage practices 99.1% 89.40% 83 No. Indicator ID Indicator Baseline Midterm 15 Standard #19 Number of schools using an improved wáter source 86.70% 100% 16 Standard #20 Number of schools with improved sanitation facilities 90% 87.50% 17 Custom Percent of schools with improved food storage and preparation equipment 61.5% 75.0% 9.1.2 Comparative table of variables for students Variable Baseline Midterm DD Analysis Control Intervention Control Intervention Control Intervention DD Number of Meals 2 2.2 1 2 0.9 -0.2 -0.1 0.1 3 22.4 7.2 16.7 15.7 -5.7 8.5 14.2 4 52.1 59.4 46.7 44.3 -5.4 -15.1 -9.7 5 22.7 32.7 34.6 39.1 11.9 6.4 -5.5 Meal times Breakfast 96.3 97.5 95.9 98.4 -0.4 0.9 1.3 Morning snack 78.8 93.8 88.2 85.3 9.4 -8.5 -17.9 Lunch 97.5 95 86.5 93.8 -11 -1.2 9.8 Afternoon snack 36.2 36.7 47.8 48.8 11.6 12.1 0.5 Dinner 94.3 94.3 95.7 95.6 1.4 1.3 -0.1 Food Groups 1 4.2 5.5 0 0 -4.2 -5.5 -1.3 2 24.5 20 2.7 2.1 -21.8 -17.9 3.9 3 41.4 29.3 12.9 12.8 -28.5 -16.5 12 4 25.2 33.2 25 26.9 -0.2 -6.3 -6.1 5 4.2 9.8 33.6 30.1 29.4 20.3 -9.1 6 0.5 2.2 21.5 22.1 21 19.9 -1.1 7 0 0 4.3 6 4.3 6 1.7 84 9.1.3 Characterization of the schools Characterization of schools by location Type of school Municipality Location Total Rural Semirural Urban Intervention Cuilco 7 0 0 7 San Gaspar Ixchil 0 1 0 1 San Pedro Necta 4 0 0 4 San Pedro Soloma 4 1 0 5 Santa Cruz Barillas 6 1 0 7 Santa Eulalia 4 2 0 6 Total 25 5 0 30 Control Cuilco 6 1 0 7 San Pedro Necta 3 1 0 4 San Pedro Soloma 1 3 1 5 San Sebastián H 0 1 0 1 Santa Cruz Barillas 7 0 0 7 Santa Eulalia 6 0 0 6 Total 23 6 1 30 Total Cuilco 13 1 0 14 San Gaspar Ixchil 0 1 0 1 San Pedro Necta 7 1 0 8 San Pedro Soloma 5 4 1 10 San Sebastián H 0 1 0 1 Santa Cruz Barillas 13 1 0 14 Santa Eulalia 10 2 0 12 Total 48 11 1 60 Source: Prepared for this study 85 The characterization of the sample by number of teachers is as follows: Characterization of schools by number of teachers Type of school Municipality Teachers Single Multi-grade Grade Total Intervention Cuilco 0 6 1 7 San Gaspar Ixchil 0 1 0 1 San Pedro Necta 0 2 2 4 San Pedro Soloma 1 3 1 5 Santa Cruz Barillas 1 3 3 7 Santa Eulalia 3 1 2 6 Total 5 16 10 30 Control Cuilco 2 5 0 7 San Pedro Necta 0 2 2 4 San Pedro Soloma 0 1 4 5 San Sebastián H 0 0 1 1 Santa Cruz Barillas 1 3 3 7 Santa Eulalia 2 4 0 6 Total 3 14 11 30 Total Cuilco 2 11 1 14 San Gaspar Ixchil 0 1 0 1 San Pedro Necta 0 4 4 8 San Pedro Soloma 1 4 5 10 San Sebastián H 0 0 1 1 Santa Cruz Barillas 2 6 6 14 Santa Eulalia 5 5 2 12 Total 8 30 21 60 Source: Prepared for this study. 86 9.2 Summary of results for analysis of work strategies Education Aspects that work What needs improvement Development and strengthening of reading spaces in the classroom • The principals’ organization for giving follow up to the reading spaces which can be used by their teachers and students. • The design and practicality of the reading games makes them attractive for the students (they use dice) • The use of the Big Books to promote reading (they look worn). • The development and use of bilingual materials, including in Q´anjob´al (there are few available) • The use of the reading spaces guide provided by PCI. • The quantity of materials for the Centers of Education Resources for Learning (CREA, for name in Spanish), since those provided by PCI and those given in the educational bag are insufficient. • Sharing among teachers in the same or other schools about the use of the spaces. • An analysis of the practice of delivering materials each year, because it is unsustainable. • Reading techniques for multi-grade or single teacher schools. • Incorporating parents to support reading at home, currently the students don’t read much because they are doing household chores or are tired. Improving access to, and use of literacy and education materials • The use of the flip chart because it allows teachers to teach using non-traditional forms. • The materials are useful and do not duplicate the ones provided by the MOE. • The strengthening of education technicians because they bring personalized attention with the first through third grade teachers. • Greater sensitivity towards principals and teachers, so that they don’t see the accompaniment of the technicians as an evaluation of their classes. • More consistency in technical accompaniment because two or three visits a year are not sufficient for meeting with the teachers and effectively resolving their doubts. • The use of the linguistic profile because not all the teachers use it at the beginning of the school year. Teachers’ knowledge and skills, increased • The training for first through third grade teachers is valued by the majority of the teachers. • The most utilized tools are the flip charts and the reading corners. • PCI’s manner of addressing bilingual education because it aligns with the MOE’s national policy and is a need at the local level. • The university diploma program because it allows teachers to vary the manner of teaching, in spite of teachers’ time limitations. • Greater teacher skills for using guides and books provided by PCI since they do not seem to be using them. • Have work strategies for teachers of single teacher and multi-grade schools. • Greater sensitization of teachers so that they buy in completely to the teacher training and understand that they “don’t know it all”. 87 Aspects that work What needs improvement Knowledge and skills of principals, increased The topics in the leadership and management program cover the needs of principals because no one teaches them how to carry out their tasks (they learn as they go). • The main improvement is in with regard to school administrative procedures for annual planning, staff files, purchase account books, inventories, SIRE and the transparency mural. • More follow up by field staff who work with teachers but not with the principals. • Give follow up for use of the book provided by PCI “School Leadership and Management Program”. • Formation of the learning communities planned by PCI. Integration of Gender and Cultural Relevance • Integration by PCI of boys and girls, male and female teachers with no distinction. • The incorporation of inclusive language in training materials and sessions. • Respect for Mayan cultures by incorporating bilingual materials, signage in reading spaces and Mayan speaking PCI staff. • Definition of a gender policy specifically for EDUCAMOS, with clear guidelines that can be measurable. Currently it is based on observation, responses from supervisors and technical staff, and there is no mention of a procedural guide on the topic. 88 Gardens and macro tunnels Aspects that work What needs improvement Socialization of the new methodology of the agriculture component • The work that is carried out in each school by the PCI agronomist with OPF involvement to determine responsibilities in the gardens or macro tunnels being implemented. • Interest on the part of principals and OPF for establishing gardens since this helps students have access to fresh vegetables and they can grow new crops. • The lack of space is a recurring theme and thus they should involve the OPF to seek solutions that don’t require rent payments. • Access to water in the dry season. Alternative sources should be sought to have this resource year-round. First phase of implementation of the initiatives • The prioritization that PCI carries out at the beginning of the year because it covers a large number of schools. • The replication that is taking place in homes for family gardens or macro tunnels. • Linkage with health and nutrition, to agree upon crops that provide more nutritional elements for school feeding. • Coordination with the WE initiative to train groups of women who are already selling foods to their schools. • The staggering of crops because it facilitates having more foods available during the year. • The diversification of crops because it reduces the danger of pests and disease. • Follow up so that the gardens can continue without PCI support (sustainability). • The involvement of more agricultural volunteers (there are few and weren’t seen). • Increase crop staggering in more schools and increase yields. • Use interactive material to teach the work with the productive initiatives such as is stated in the strategy. • Negotiation by the OPF to obtain technical accompaniment from the MOA and inputs to continue the work with gardens. Second phase of implementation of the initiatives • The incorporation of native species in the gardens which facilitates greater food availability and appreciation by the students. • Incorporate artisanal technologies to improve productive capacity in the schools. • Manage water in the schools where it is not available, especially in the dry season. Results-oriented Technical Accompaniment • The technical accompaniment because it ensure quality in implementation of productive activities • Increase the frequency of visits to the productive units based on the level of school ownership, and for timely attention to technical problems or pests and diseases. Integration of Gender and Cultural Relevance • The introduction of native species because they are already consumed at the local level. • Boys and girls were not assigned different garden tasks based on gender. • Increase native species in other gardens. 89 Infrastructure Aspects that work What needs improvement Assessment visits to schools • The prioritization of projects because this results in working where it is most needed. • Adaptation of the size of the job according to the number of students. If resources are insufficient, the OPF can negotiate to extend to the best size. Planning • Effective negotiation with municipalities so that they provide resources for the job. • The designs are adjusted for each specific project, calculating costs and then making designs. • Training of masons who are generally from the same community. Greater involvement of the municipalities in supervision of their projects. Execution • Community negotiation for labor and in some cases, they supply materials to expand the project based on needs. • Compliance with commitment of delivering projects and the community accepts and signs off on them. • Following the established design and making adjustments when necessary. • Better planning of time for execution and for distribution of supervisory staff to ensure the quality of the jobs. • Agree on maintenance tasks for the projects. • Treat odors in the sanitation services. Integration of Gender and Cultural Relevance • The sanitary services have adequate spaces for boys and girls. 90 Health and nutrition Aspects that work What needs improvement Increased use of dietary and health practices in the educational community • The incorporation of lead mothers into direct activities in the school to promote practices. • The cleanliness traffic light because it monitors hygiene in different areas of the school. • The promotion of safe water consumption for students by provision of eco filters. • Cascade replication by the lead mothers. • Hygiene outside the school and community environment. Strengthen skills for food management and plans for food purchases • Application of checklists for food safety. • Integration of PCI and MOE foods in the same physical space. • Preparation of MOE official recipes. • Training on purchase plans to ensure good nutrition. • Increase management of non-perishable foods to ensure safety. • Incorporate the OPF in the whole process for food purchases, requesting price quotations from various suppliers, and filling out MOE records. Integration of Gender and Cultural Relevance • The identification of women leaders so that mothers feel more comfortable receiving trainings. 91 Women Empowered Aspects that work What needs improvement Organization of, and attention to WE groups • Experience exchanges to motivate participation and formation of new groups. • The instruments for recording savings in a system that consolidates the information. • Acting on problems identified in the discussion of social topics. • The recognition the groups are receiving in the community. They are beginning to be valued. • The women have become lead mothers and are holding positions in the OPF. There are few cases of incorporation into the COCODE (key positions). • The training that helps them to know their rights, to take on decision making roles (with less fear) and become involved in community activities (e.g., church) • Confront the barriers the women face for joining the groups (identified and presented in the section on WE key questions). • Involve principals and school and community authorities to work on actions together with the groups. Promotion of economic empowerment • The use of loans to cover household and personal needs as well as for their children’s education. • Productive activity implementation by two WE groups and their linkage with schools to supply them with vegetables, which is in line with the LAE. • Incorporate and link more WE groups for selling foods to the schools. Integration of Gender and Cultural Relevance • The initiative seeks balance in decision making power between men and women. • The women who were sensitized support education equally for boys and girls (previously this wasn’t true) 9.3 Results of the analysis of digital classrooms The digital classroom is a room that was arranged for the installation of the tablets PCI bought for the school. These tablets are meant to act as computers for the classroom, and they don’t require an internet connection because a server with multiple types of educational material is directly connected to them. While the donated equipment is helpful, the schools need more so that they can be used for longer periods of time and by more students. Parents, especially the OPF, generally supported this initiative and encouraged the technician to visit more frequently. The virtual classrooms are used by students from all grades either as part of the regular curriculum or based on specific work requests. In the multi-grade schools, this coordination is easier because there are only two teachers. Only one grade can use the digital classroom at a time due to the limited amount of equipment available, and if the class is large, students work in pairs and take turns so that everyone can use the tablets. 92 The upper grade students use the equipment to complete their homework or to conduct the research that the teachers assigned. The equipment opens a window of opportunity for the students who have never been outside their home town to observe the world and see how people live in other places. The lower grade students use the tablets to learn how to write their letters. However, the digital classrooms were limited in several ways. The teachers themselves weren’t always comfortable using tablets, and the students were not accustomed to using electronic devices. Some students have used telephones but not keyboards and screens. Often, they were timid when they first learned how to use the tablets, until they became familiar with them and started to play games. The students did not take long to become completely familiar with the tablets and learn how to use the different educational applications. In general, the schools did not have a contingency plan for when the devices no longer function. They expected the program to maintain its presence and to renew or provide more tablets so that all the students can learn how to use the technology. If this doesn’t happen, they might call the CTA or ask for support from the municipality or the COCODE, but they had no formal plans. The schools would benefit from more equipment and a printer, but a teacher or someone from the community with the necessary knowledge would need to be trained on repair and maintenance. Municipalities with Digital Classrooms The digital classroom strategy implemented by PCI aims to create an environment within the school where the students can use a digital system, which currently consists of tablets, to help them become familiar with this technology and give them the potential to access the internet. This strategy should help students gain new information and be able to communicate within a technological environment. The team interviewed 32 students in the following municipalities: Number students interviewed Municipality Number of Students Barillas 5 Santa Eulalia 11 Soloma 3 Cuilco 7 San Pedro Necta 6 Total 32 Prepared for this study Barillas 15.6% Santa Eulalia 34.4% Soloma 9.4% Cuilco 21.9% San Pedro Necta 18.8% Source: Field work 93 A total of 18 boys and 14 girls were interviewed. Their ages varied and generally depended on their grade in school. The number of students at each age is below: Age of students interviewed Age Number of students 8 years 6 9 years 4 10 years 4 11 years 5 12 years 9 13 years 2 15 years 2 Total 32 Prepared for this study. Source: Field work While students from all grades used the digital classroom for purposes appropriate for their age and characteristics, students in the upper grades (fourth to sixth) used it the most, since they understood it the best. The children learned to use the tablets with exercises and games. Percentages of most used applications The Latin American library application was used by the most students (46.9%). The students said they use this application the most because they learn a lot with it (50.0%), it’s fun (15.6%), and it’s easy to use (9.40%). Others said that it’s colorful or that it had sound effects, and some students did not respond at all. Although the students didn’t mention Wikipedia, it was obviously widely used, particularly when the teacher assigned a research topic. 6.3 3.1 46.9 12.5 6.3 12.5 12.5 0.0 10.0 20.0 30.0 40.0 50.0 Wiki how Ka lite math Latin American Library Guatemala Education Collective Educa lab Other Don't Know/ Didn't Respond 94 What students learned in the digital classroom The students said they mostly learned reading, followed by the human body and math in the digital classrooms. Their answers were based on their opinions and what they saw when they arrived in the virtual classroom. Within the classroom, 43.8% responded that they most frequently used the Spanish language course on the tablets, followed by math and natural sciences, with 18.8% each, and social studies, with 9.4%. Some did not answer the question. The students had their own preferences about what they would like to do in the virtual classroom and how it should be installed. What students would most like in the virtual classroom 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% 35.0% 40.0% Learn to Read Human Body Math About Animals Drawing Speak Spanish Mayan Numbers Computer Typing Key Strokes Evolution How the earth was formed Continents Computer parts Playing games History and Environment Didn't Learn Anything 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% Computers Printers Decotrations Videos Mobile Apps Games Books Computers and phones to fix Systems Interesting things Desks Notebooks Everything is fine 95 The students wished they had more tablets so they wouldn’t have to take turns and so that each one would have its own free space. They liked to print, so they wanted one printer per tablet, without any printing limits. They also said they would like chairs so they can sit down and see what others are working on, a consistent internet connection, more tables, and notebooks to write down the teacher’s instructions. This learning strategy is still evolving and will over time help systematize teaching methods so that both students and teachers will be familiar with the new instruction methods. 9.4 Tests for Statistical Significance For all tables, * = statistically significant at p<0.05, ** = statistically significant at p<0,10. Literacy Indicator variables N Sig. (bilateral) Read at 2nd grade level Male 391 0.6 Female 469 Read at 2nd grade level Maya 731 0.001* Latino 106 Read at 2nd grade level Rural 572 0* Semirural 288 Read at 2nd grade level Teacher in charge of all grade levels 32 Teacher in charge of 0.006* more than one grade level 290 Read at 2nd grade level Teacher in charge of all grade levels 32 0.001* Single Grade Teacher 538 Read at 2nd grade level Teacher in charge of more than one grade level 290 0.081** Single Grade Teacher 538 Read at 2nd grade level Intervention 509 0.442 Control 351 Nutrtition Indicator variables N Sig. (bilateral) Hunger during the School Day Intervention 483 0.011* Control 445 Hunger during the School Day Male 464 0.074** Female 464 Hunger during the School Day San Pedro Necta 107 0.002* San Sebastian H 13 96 Cuilco 172 Indicator variables N Sig. (bilateral) Illness as a cause of school absence Intervention 53 0.11 Control 39 Illness as a cause of school absence Male 49 0.47 Female 43 Indicator variables N Sig. (bilateral) Minimum Acceptable Diet Intervention 562 0.582 Control 557 Minimum Acceptable Diet Male 558 0.545 Female 561 Minimum Acceptable Diet Missed Class 197 Did not miss 0.768 class 922 Minimum Acceptable Diet Barillas 309 0.017** Santa Eulalia 169 0.038* Soloma 262 0* Parents Indicador Variables N Sig. (bilateral) 3 Cleaning Practices or More Intervention 117 0.289 Control 69 3 Cleaning Practices or More Barillas 31 0.03 Cuilco 46 0.024 3 Cleaning Practices or More Male 36 0.583 Female 150 Indicador variables N Sig. (bilateral) Parents know 3 or more benefits of education Intervention 113 0.015* Control 69 Parents know 3 or more benefits of education Barillas 31 0* Parents know 3 or more benefits of education Male 34 0.687 Female 148 97 Parents know 3 or more benefits of education Trained by PCI 106 Not trained by PCI 76 0.047* Collaborating Mothers Indicator variables N Sig. (bilateral) Collaborating mothers know more than 3 cleaning practices Intervention 113 0.565 Control 51 Collaborating mothers know more than 3 cleaning practices Barillas 51 0* Collaborating mothers know more than 3 cleaning practices Trained by PCI 102 0.927 Not trained by PCI 62 Observations Indicator variables N Sig. (bilateral) Meal Preparation Intervention 30 0.005* Control 24 Indicator variables N Sig. (bilateral) Improved Water Source Intervention 30 0.254 Control 24 Indicator variables N Sig. (bilateral) Improved Sanitary Facilities Intervention 30 0.706 Control 24 9.5 Field Instruments Evidencia fotográfica a obtener (Todos) • Llegada de niños y niñas a la escuela ¿Los niños y niñas llegan a la escuela acompañados de una persona adulta o llegan solos? Niñas – Adulto Hombre ___ Niñas – Adulto Mujer____ solas______ otros niños Niños - Adulto Hombre ___ Niños- Adulto Mujer_____ solos______ otros niños • Ingreso al aula ¿Cómo es la formación de filas de los estudiantes para ingresar al aula? Niñas primero____________ 98 Niños primero____________ Filas Mixtas ______________ ¿Dentro del aula, como están ubicados (sentados) los niños y las niñas? ¿Separados___ o combinados? • Abordaje del Docente ¿Las personas docentes asignan tareas de limpieza u orden el aula de forma equitativa a los niños que a las niñas? (ver semáforo de la limpieza) Sí _________ No_________ ¿El lenguaje utilizado por las personas docente es inclusivo? Sí _________ No_________ ¿Hay diferencia en el trato y forma de hablar de las personas docentes con niños y niñas?__________________________________________________________________________________ _____________________________________________________ 99 OPF Focus Groups: Alimentación Escolar ¿Cómo se realiza el proceso de asistencia técnica para un adecuado manejo y almacenamiento de los alimentos en bodegas escolares? ¿Cómo se realiza el manejo de alimentos USDA y MINEDUC? ¿Cómo se elaboran los planes de compras, quienes intervienen? ¿Cuántos menús tienen y como los seleccionaron? ¿Cuántos proveedores tienen? ¿Porque no hay proveedores en la comunidad? ¿Tiene ustedes reuniones con los padres de familia? ¿Los apoya PCI en capacitación sobre la LAE? ¿El técnico del MINEDUC les da asistencia? ¿Se reúne la comisión de alimentación? Agricultura ¿Hay Huertos escolares? ¿Qué acciones se realizan para lograr la réplica en el hogar? ¿Qué papel desempeñan los voluntarios en este proceso? ¿Cómo se afronta el tamaño reducido de las áreas en las escuelas para el establecimiento de huertos escolares? ¿Qué potencialidades tienen los macro túneles para fomentar la agricultura familiar (ley de alimentación escolar)? ¿Qué experiencias exitosas se tienen sobre la réplica de huertos y macro túneles en los hogares? ¿Es solo para fines de enseñanza? ¿Utilizan los productos para abastecer la cocina? ¿Qué productos son los que siembran? ¿Tienen capacidad de crecer? 100 Referencia del Programa ¿Cuál es su opinión sobre el programa EDUCAMOS? ¿Qué cambios positivos ha observado en sus hijos? ¿En qué le ha beneficiado a usted directamente? ¿Hay algo que no le gustó y que desearía cambiar de EDUCAMOS? ¿Qué sugeriría para mejorar el programa? Varias ¿Qué participación tiene PCI en las jornadas de desparasitación? ¿Qué participación tiene PCI en las jornadas de vacunación? Infraestructura: ¿Hasta qué punto los padres y los estudiantes perciben que la escuela es un lugar seguro para niñas y niños? 101 Questionnaire for School Directors Municipio: ________________________Escuela_______________________________ Intervención: _____________________ Control: ____________________________ 1. Asistencia alumnos: Indicador Situación a verificar / días faltados Total alumnos inscritos Ene Feb Mar Abr May Jun Masc. Fem. Asistencia estudiantes Primer Grado Segundo Grado Tercer Grado Cuarto Grado Quinto Grado Sexto Grado 2. Asistencia de maestros Indicador Situación a verificar Ene Feb Mar Abr May Jun Asistencia maestros Primer Grado Segundo Grado Tercer Grado Cuarto Grado Quinto Grado Sexto Grado 3. Técnicas de calidad en la docencia A. Dirección escolar y liderazgo 1. ¿Asisten los padres y madres de familia en reuniones de la escuela? Si No 2. ¿Qué porcentaje de padres/madres asisten a las reuniones? 3. ¿Quiénes asisten a las reuniones de padres? Padres Madres Ambos Otros 4. ¿La participación de los padres y madres es activa en las reuniones? 102 Si no 4.1 ¿Por qué? 5. ¿Se cuenta con un plan de mejora de la escuela? Si No 6. ¿Qué aspectos son los prioritarios en el plan de mejora de la escuela? 7. ¿Quiénes participaron en la elaboración del plan de mejora de la escuela? 8. ¿Cuáles son las principales dificultades para cumplir las mejoras del plan? B. Procesos administrativos escolares 9. ¿cuenta con los siguientes documentos? documento Si No Planificación anual Expedientes de personal Libros de compras Inventario de mobiliario utiliza el SIRE Mural de transparencia Planificación anual 10. ¿Quiénes participaron en la elaboración de la planificación anual de la escuela? Expedientes de personal 11. ¿Se encuentran actualizados los expedientes de personal? Si 103 No SIRE 12.¿Está actualizado el SIRE? Si No 12.1 ¿Por qué no? Mural de transparencia 13.¿Está actualizado el Mural de transparencia? Si No 13.1 ¿Porqué no? 14. ¿Hasta qué fecha esta actualizado? 15. ¿quiénes son los encargados de elaborarlo? 17. ¿Qué temas reciben como directores para incrementar sus conocimientos y habilidades, son de interés y utilidad? 18. ¿Qué acciones de mejora y diferentes realiza como producto de la formación recibida de PCI? 19. ¿Qué procesos administrativos nuevos / mejorados está aplicando como director como resultado de la asistencia de PCI (Diplomado)? 20. ¿Cómo se utiliza el Manual Pedagógico y Administrativo proporcionado por PCI? C. Organización escolar 21. ¿Cuenta con lo siguiente? Documento Si No 104 Actas de comisiones escolares Plan de riesgos Mural de transparencia Comisiones escolares 22. ¿Con que comisiones se cuenta en la escuela? 23. ¿Ha sido efectivo el trabajo de las comisiones? Si No 23.1 ¿Por qué? Plan de riesgos 24. ¿Quiénes elaboraron el plan de riesgos? 25. Fecha de elaboración del plan de riesgos 26. ¿Han recibido apoyo para el cumplimiento del plan de riegos, de quien(es)? D. Gestión curricular 27. ¿Sus maestros aplican lo siguiente? documento Si No Planificación acorde CNB Uso de CREA Libreta calificaciones Modelo lectura ambientes bilingües 28. ¿Cómo se realiza el proceso de establecimiento y organización de Bibliotecas escolares? ¿qué favorece, que limitantes se tienen? Observar si es biblioteca, rincón de lectura, CREA. Ver si tiene características de uso, si están las cajas de libros del ministerio, la dotación de EDUCAMOS. Se debe observar el uso de las mismas, si prestan libros 105 29. ¿Cómo se realiza el proceso de establecimiento y organización de rincones de lectura? ¿qué favorece, que limitantes se tienen? 30. ¿Cómo se realiza el proceso de establecimiento y organización de Centros de Recursos Educativos para el Aprendizaje - CREA-? ¿qué favorece, que limitantes se tienen? 31. A nivel de la escuela, ¿Cómo se utiliza la Guía escolar para el manejo de las bibliotecas, rincones y CREA? ¿se siguen los lineamientos, sugerencias pedagógicas y recomendaciones para su uso? 32. ¿Cómo los docentes hacen uso de los recursos que reciben del MINEDUC en el CREA? ¿qué materiales están disponibles, qué recursos nuevos e innovadores se observan? E. Acompañamiento pedagógico 33. ¿Realiza acompañamiento pedagógico a sus maestros ? Si No ¿Por qué? 34. ¿Se cuenta con registros / hojas de acompañamiento por parte del personal de PCI/UVG? Si No 106 34.1 ¿Cuándo fue el último acompañamiento? Evaluación de docentes 35. Se utiliza la determinación del perfil lingüístico de los estudiantes de preprimaria y primer grado? Si No 35.1 ¿Qué perfil utilizan? PCI MINEDUC Otro 36. ¿Qué materiales y recursos utilizan sus maestros para la enseñanza de lectoescritura? Material didáctico Si No Rota folio Mega libro Juegos de lectura Libro de aprendizaje 37. ¿Qué tan valioso han sido estos recursos para los maestros? Nada valioso Algo valioso Indiferente Valioso Muy valioso 38. ¿Cuáles de las siguientes estrategias de lectoescritura utilizan sus maestros? Estrategia Si No Grado Conciencia fonológica Principio alfabético Fluidez Vocabulario Comprensión lectora Escritura creativa De acuerdo a la respuesta que se obtenga, se procede a observar si por ejemplo el rotafolio existe en el aula y está usado, si dice que utiliza el mega libro cual es el que ha usado, si contesta “Las Manzanas” observarlo para determinar el uso que se le ha dado si es de primero primaria. Observar las otras técnicas Se debe tener claro cada una de las etapas anteriores y como se aplica en 1ro, 2do y 3ro primaria. Luego se debe de hacer una observación en cualquiera de los grados para verificar si alguna de las técnicas se aplica dentro de la escuela, ya que la aplicación no es esporádica, es un modelo constante. 107 Salud y Nutrición 39. ¿Cómo se integra la pertinencia cultural en la preparación de menús escolares y otras actividades de salud y nutrición? Infraestructura 40. ¿Qué acciones y compromisos de mantenimiento posterior se tiene por tipo de infraestructura? Donaciones de materiales para infraestructura 41. ¿Cómo se realiza el registro de materiales de donación? 108 Cuestionario para docentes Municipio: ________________________Escuela_______________________________ Intervención: _____________________ Control: ____________________________ Grado__________________________ sexo: ____________________________ 1. ¿Cuáles de las siguientes estrategias de lectoescritura utiliza en su aula? Estrategia Si No Grado Conciencia fonológica Principio alfabético Fluidez Vocabulario Comprensión lectora Escritura creativa 2. ¿Cuál ha sido la principal dificultad para la implementación de estrategias de lectura? 3. ¿Cuáles han sido los principales cambios observados en los alumnos como resultado de la implementación de nuevas estrategias? Materiales educativos 4. ¿Qué materiales están disponible y en uso en la escuela, como parte del apoyo directo de PCI? 5. ¿Cómo se organiza la escuela para hacer uso de estos materiales (evidencia de uso)? 6. ¿Cómo se aprovecha el material bilingüe que se proporciona por parte de docentes y estudiantes (evidencia de uso)? 7. ¿Cómo complementan estos materiales a los proporcionados por el MINEDUC, hay duplicidad, que hace falta? 109 8. ¿Cómo se realiza el fortalecimiento de los Técnicos de Educación en el uso de los materiales, para que los docentes pueden aprovecharlos? 9. ¿Cómo usan los alumnos los materiales en los espacios de lectura? 10. ¿Cómo utilizan los juegos de lectura, que diferenciación tienen con otras herramientas escolares? 11. ¿Cómo se promueve el espíritu colaborativo de los docentes? Capacitación docente 12. ¿Qué tipo de formación reciben de PCI para mejorar sus conocimientos y habilidades como docentes, cómo les ha ayudado? 13. ¿Qué herramientas metodológicas se implementan con los docentes para mejorar la enseñanza y aprendizaje de la lectoescritura en L1 y L2? 14. ¿El apoyo de PCI responde a las características lingüísticas y socioculturales de la región, cómo? 15. ¿Qué utilidad tiene como docente la evaluación del perfil lingüístico de los estudiantes? 16. ¿Cómo la formación recibida contribuye al uso del CNB? 17. ¿Cómo utilizan los libros de lectoescritura proporcionados por PCI? 110 18. El Diplomado Universitario de Lectoescritura en Ambientes Bilingües e Interculturales responde a los intereses y necesidades de los docentes de preprimaria a tercero primaria, ¿cómo? (donde aplica) 19. ¿Qué favorece y qué limita recibir este tipo de diplomado virtual y presencial? 111 Evidencia fotográfica a obtener (Todos) • Llegada de niños y niñas a la escuela ¿Los niños y niñas llegan a la escuela acompañados de una persona adulta o llegan solos? Niñas – Adulto Hombre ___ Niñas – Adulto Mujer____ solas______ otros niños Niños - Adulto Hombre ___ Niños- Adulto Mujer_____ solos______ otros niños • Ingreso al aula ¿Cómo es la formación de filas de los estudiantes para ingresar al aula? Niñas primero____________ Niños primero____________ Filas Mixtas ______________ ¿Dentro del aula, como están ubicados (sentados) los niños y las niñas? ¿Separados___ o combinados? __ • Abordaje del Docente ¿Las personas docentes asignan tareas de limpieza u orden el aula de forma equitativa a los niños que a las niñas? (ver semáforo de la limpieza) Sí _________ No_________ ¿El lenguaje utilizado por las personas docente es inclusivo? Sí _________ No_________ ¿Hay diferencia en el trato y forma de hablar de las personas docentes con niños y niñas?__________________________________________________________________________________ _____________________________________________________ 2019EMT: Guía de observación de cocina, preparación de alimentos e instalaciones sanitarias Section 1. Identificación 1.1 Tipo de escuela Tipo de escuela Intervención [1] Control [0] 1.2 Municipio Municipio Barillas [1] Santa Eulalia [2] 112 Soloma [3] Cuilco [4] San Gaspar Ixchil [5] San Pedro Necta [6] San Sebastian H [7] Section 2. Instrucciones 2.1 Instrucciones Verificar por medio de observación previo a la preparación de alimentos los siguientes aspectos y luego verificar por medio de observación las instalaciones sanitarias (baños) Section 3. Observación de cocina 3.1 Utensilios limpios 1. Los utensilios se encuentran limpios? Si [1] No [0] 3.2 Utensilios secos 2. Los utensilios se encuentran secos? Si [1] No [0] 3.3 Utensilios cubiertos o tapados 3. Los utensilios se encuentran cubiertos o tapados? Si [1] No [0] 3.4 Utensilios colocados en un lugar especial 4. Los utensilios se encuentran colocados en un lugar especial? Si [1] No [0] 3.5 cumple con 3 de 4 criterios 5. cumple con 3 de 4 criterios? Si [1] No [0] 3.6 Observaciones cocina 7. Hay alguna observación con la cocina? (tiene humo, cocinan afuera de la cocina, etc) 113 Section 4. Preparación de alimentos 4.1 Comida tapada 1. La comida estaba tapada antes de prepararse? No [0] Si [1] No se pudo observar [999] 4.2 Uso de agua 2. si no hay agua entubada, el agua está almacena en recipientes donde no quepan las manos, para servirla usan algún chorrito o tipi tap. No [0] Si [1] No se pudo observar [999] 4.3 Preparacion de alimentos 3. La comida se procesa en mesas o tablas separadas según sea carne, verdura o fruta No [0] Si [1] No se pudo observar [999] 4.4 Basurero 4. había un basurero tapado? No [0] Si [1] No se pudo observar [999] 4.5 Limpieza de pisos 5. había residuos de comida o basura en el piso? No [0] Si [1] No se pudo observar [999] 4.6 lavado de manos 6. Los cocineros tienen que lavarse las manos antes de cocinar, y hacerlo bien (lavado con jabón y agua corriendo, y por lo menos 20 segundos, y secarse bien con toallas limpias no con gabacha) No [0] Si [1] No se pudo observar [999] 114 4.7 manos cocineras 7. las cocineras usan anillos o tienen pintadas las uñas alcocinar? No [0] Si [1] No se pudo observar [999] Analisis biologicos 8. Sabe si se ha hecho análisis micro biológico (laboratorios) al agua? No [0] Si [1] No se pudo observar [999] 4.8 Animales 9. Habían animales sueltos en la cocina (perros, gatos, gallinas, etc)? No [0] Si [1] No se pudo observar [999] Section 5. Fuentes de agua 5.1 Fuentes de agua 1. Qué fuente de agua abastece a la escuela? Grifo público [1] Sistema entubado [2] Pozo [3] Pozo protegido [4] Nacimiento protegido [5] Recolección de agua de lluvia [6] Ninguno [0] 5.2 disponibilidad de agua 2. el agua esta normalmente disponible de la fuente identificada? Si [1] No [0] 5.3 pago de agua 3. Pagan por el agua que recibe la escuela? No [0] Si, cada alumno aporta una cuota [1] Si, el MINEDUC asigna fondos [2] 115 Section 6. Instalaciones Sanitarias 6.1 Sanitarios 1. Habían baños separados para niños y niñas? No [0] Si [1] No se pudo observar [999] 6.2 tipo de sanitario 2. Qué tipo de sanitario tienen los baños? Sistema de alcantarillado [1] Sistema septico [2] Letrina de pozo (tiene un revestimiento que evita la contaminación del suelo) [3] Letrina mejorada ventilada [4] Inodoro de compostaje [5] Pozo ciego [6] Otro no mejorado [7] 6.3 limpieza sanitarios 3. Estaban limpios los sanitarios? No [0] Si [1] No se pudo observar [999] 6.4 lavamanos 4. Existe un lugar para lavado de manos? No [0] Si [1] No se pudo observar [999] 6.5 uso de lavamanos 5. Cumple con criterios de agua + jabón + uso No [0] Si [1] No se pudo observar [999] 6.6 veces limpiezas sanitarios 116 6. Cuantas veces al día se limpian los baños? 1 vez diaria [1] 2 veces diarias [2] 3 veces diarias [3] 4 veces diarias [4] 5 o más veces diarias [5] Section 7. Semaforo de la limpieza 7.1 uso semaforo 1. Están utilizando en la escuela el semáforo de la limpieza? Si [1] No [0] 7.2 porque no lo usan Porque no lo usan? 7.3 participacion limpieza Cuentan todos las aulas con semáforo de la limpieza 1 [Si] 0 [no] 117 2019EMT: Guía de observación de bodega Section 1. Identificación 1.1 Tipo de escuela Tipo de escuela Intervención [1] Control [0] 1.2 Municipio Municipio Barillas [1] Santa Eulalia [2] Soloma [3] Cuilco [4] San Gaspar Ixchil [5] San Pedro Necta [6] San Sebastian H [7] Section 2. Instrucciones 2.1 Instrucciones Verificar por medio de observación el estado de la bodega de alimentos Section 3. Bodega de almacenamiento de alimentos 3.1 Bodega de alimentos 1. Existe una bodega para resguardar los alimentos en la escuela? No [0] Si [1] Si, pero no es bodega [3] 3.2 Tipo alimentos almacena 2. Qué tipo de alimentos guarda en esta bodega? Perecederos (frutas, verduras, carne) [1] Aborrotes (leche, cereal, pastas, otros) [2] Ambos (perecederos y no perecederos) [Ambos (perecederos y no perecederos)] 3.3 almacenaje perecederos 2.1 Dónde almacena los productos perecederos? 1. Bodega especial en escuela [1] Refrigerador/ congelador de escuela [2] Refrigerador / congelador de familia de estudiante [3] 118 Tienda cercana [4] No almacenan perecederos [5] No sabe / no responde [777] 3.4 Utiliza Kardex 2. Utiliza Kardex para control de los alimentos? Si [1] N 3.5 Otra forma de registro 3. qué otra forma utiliza para control de los alimentos? 3.7 espacio fisico suficiente 5. Es suficiente el espacio físico de este lugar con relación a los productos que se tienen que almacenar? Si [1] No [0] 3.8 calidad de paredes 6. Es buena la calidad de las paredes de este lugar? [De material impermeable, No presenta condiciones de humedad] Si [1] No [0] 3.9 Calidad del techo 7. Es buena la calidad del techo de este lugar? [completo y sin presencia de goteras] Si [1] No [0] 3.10 ventilacion 8. Es buena la ventilación de este lugar? [Entrada y cantidad de aire suficientes] Si [1] No [0] 3.11 Exclusividad bodega 9. Este lugar es exclusivo para el almacenamiento de alimentos? Si [1] No [0] 3.12 bodega limpia 10. Este lugar está limpio? Si [1] No [0] 119 3.13 bodega ordenada 11. Este lugar está ordenado? Si [1] No [0] 3.14 bodega con tarimas 12. Tiene este lugar tarimas para la colocación de los productos que allí se almacenan? (estibado) Si [1] No [0] 3.15 bodega libre de plagas 13. Hay ausencia de roedores u otros animales que pueden causar daños a los productos que se almacenan? (ver sí hay heces fecales) Si [1] No [0] separacion productos 14. Están separados los productos que se almacenan en este lugar o bodega? Deben estar separados 20 cms. de la pared y un metro de separación con alimentos de otros donantes. Expects a single option response (required) Si [1] No [0] 3.16 Finalizar Grabar y finalizar 120 Classroom observation guide Observaciones Intra-aula de lo siguiente: Durante 10 minutos, el observador selecciona un minuto en forma aleatoria, y durante el mismo se dedica a observar el comportamiento de los estudiantes con respecto a la clase que se estaba impartiendo, se hace solo en períodos de clase que implicaban al profesor dando una “clase normal”, en la que éste interactuaba con sus estudiantes, utilizando frecuentemente ayudas audiovisuales tales como el pizarrón. Las observaciones se hacen en primero primaria durante períodos de clase tales como matemáticas, estudios sociales, ciencias naturales y lenguaje; evitando la observación de clases donde se hizo un dictado o aquellas de corte artístico, o en las que los estudiantes necesitaban salir o entrar constantemente a las aulas. El observador, durante el minuto de evaluación, cuenta cuántos estudiantes encajan en la siguiente división a. Distraídos: mirando hacia las puertas, ventanas o techo, hablando en voz baja o murmurando, pensando o meditando, jugando con algo, comiendo, bebiendo y o masticando b. Con atención hacia algo diferente: mirando, leyendo o revisando algo insistentemente, escribiendo o dibujando algo sin levantar la cabeza, recostado completamente sobre el pupitre/mesa c. Haciendo actividad distractora: riendo a carcajadas o con sonidos audibles, golpeando algo (con pies o manos), caminando o corriendo dentro o hacia fuera del aula d. Sin motivación: con señales de sueño, bostezando, cabeceando, o durmiendo en el aula e. Con actividad distractora grupal: platicando con toro compañero, tocando a otro compañero/a, tirando algo a un compañero/a, pegando o golpeando a un compañero. A continuación se presenta el mecanismo de registro: Observación de atención de los estudiantes Municipio: ________________________Escuela_______________________________ Intervención: _____________________ Control: ____________________________ Número total de estudiantes en clase ____________________ Observación de lo que hace el docente: ________________________________________________ Para Grupos Focales de ME: Observación 1 2 Estudiantes distraídos viendo hacia otro lado Estudiantes distraídos en una actividad motriz Estudiantes distraídos en comunicación Estudiantes distraídos con señales de sueño o cansancio Estudiantes con una actividad distractora del grupo 121 Título de la Columna Preguntas de discusión Participación en ME Le ha beneficiado la participación en los grupos ME Que ha cambiado como resultado de ser miembro de un GRUPO ME Qué efecto están teniendo las actividades del proyecto en la carga de tiempo/carga de trabajo de las mujeres de la comunidad? Ej. ¿En términos de mantener los huertos escolares, preparar comidas, participar en los OPFs En qué medida los miembros de grupos ME aumentaron su poder de toma de decisiones dentro del hogar, particularmente con relación a las decisiones sobre la educación de sus hijos e hijas y la importancia de que las niñas asistan a la escuela? Tienen intercambio con otros grupos ME de localidades diferentes? ME Economía Que piensan del ahorro que se realiza en los ME Cuando ahorran aproximadamente y en qué tiempo (semana, mes, otros) Han realizado prestamos al grupo ME. En que han invertido su dinero Cuál es la actividad económica que creen más rentable para invertir en esta comunidad Cuantas de ustedes piensan que han crecido económicamente a partir de los grupos ME Tienen alguna propuesta o sugerencia para mejorar los resultados de la iniciativa ME. Como puede mejorar el trabajo del grupo de ME. ¿Hay o hubo Huerto Escolar? ¿Qué piensan del huerto, es posible que sea una actividad en donde ustedes puedan generar ingresos? ¿Conocen la Ley de Alimentación Escolar?.... Si dicen que si, como podrían aprovecharla para generar ingresos ME Social Qué podrían hacer las mujeres para mejorar la situación/ relación familiar en su hogar ¿Hay problemas que prevalecen en las comunidades? (por ejemplo, violencia familiar, alcoholismo, conflictos. Como lo solucionan 122 Si una familia de la comunidad tiene un problema o situaciones difíciles, que no puede resolver por sí misma o que necesite apoyo ¿estaría el grupo ME dispuesto a ayudarles? ¿Lo han hecho antes de participar en el grupo? Ha inscrito a sus hijas e hijos en edad escolar en la escuela. Sondear todo aspecto del género. Están ambos (niño y niña) en la escuela. Estar en el grupo ME, le favorece para apoyar más a sus hijas e hijos en la escuela ME Político Cuál es su papel en el hogar en la toma de decisiones. Ha mejorado la comunicación con el esposo a partir de su integración en el grupo?. Se siente algún cambio antes, durante y después del programa Cuantas de ustedes participan en un COCODE….. Alguna COOPERATIVA….. En el OPF….. Iglesia etc…… Han participado en actividades de política partidista Como grupo ME se relacionan con el gobierno local (municipalidad) 123 9.6 Evaluation Terms of Reference I. INTRODUCTION Project Concern International (PCI) is currently implementing a five-year (FY17 - FY21) USDA- funded McGovern-Dole Food for Education program titled EDUCAMOS (Educación con Calidad y Alimentación Mejorada Orientada a la Sostenibilidad). This program’s strategic objectives aim at improving literacy of school-aged children, and improving the use of health and nutrition practices, by achieving the following results: • Improved quality of literacy instruction • Improved school attendance • Improved school attendance • Improved knowledge of health and hygiene practices • Increased knowledge of safe food preparation and storage practices • Improved access to clean water and sanitation services • Strengthened capacity of government institutions • Improved policies and regulatory framework • Increased commitment from local organizations and community groups Currently, EDUCAMOS is working in 294 schools in the municipalities of San Gaspar Ixchil, San Pedro Necta, Cuilco, Soloma, Santa Eulalia and Barillas, in the department of Huehuetenango, Guatemala. Project participants include 38,387 students (19,794 male, 18,593 female), 1,451 teachers, 1,196 volunteer mothers, 331 mother community leaders, 59 education volunteers, 1,470 members of Parent-Teacher Associations (PTA) and 1,059 Women Empowered savings group members. PCI has two strategic partners for the implementation of EDUCAMOS: first, Juarez & Associates, an international NGO who specializes in education and whose role under this grant is to improve the quality of bilingual education through activities designed to improve materials and tools used by teachers in early grades, increase teachers’ and principals’ knowledge and capacity. The second partner is Universidad del Valle Guatemala (UVG) who is responsible for providing a teachers’ training program to local teachers of preschool through third grade. EDUCAMOS is midway through the five-year phase III implementation period; the baseline study was conducted in 2017, and the midterm evaluation (MTE) is planned for 2019. As such, PCI is requesting proposals for an external consultant to conduct the MTE to help PCI understand the quality and effectiveness of project implementation and to help design an appropriate corrective action plan to improve program activities and strategies as needed. II. PURPOSE AND SCOPE The MTE will evaluate the relevance and focus of program strategies, the validity of assumptions made in the design phase; measure the progress achieved as compared to the baseline, including the effectiveness, efficiency and timeliness of interventions to achieve the established objectives; document the lessons learned; assess effectiveness of efforts to achieve sustainability to date; provide recommendations to strengthen performance, efficiency and sustainability; and, to make recommendations for the final evaluation (for example, define areas of greater emphasis or focus). III. EVALUATION QUESTIONS AND LEARNING OBJECTIVES 124 The independent consultant team will review the tools and results of the baseline, with the direct involvement of program staff designated by PCI, to support the improvement and expansion of key evaluation questions. Overall, these questions will focus on the following areas: Program Design, Implementation and Relevance • To what extent is the results framework and program approach valid for the local context? • Which program strategies implemented have been more effective and relevant in contributing to program results? Which were the least effective and least relevant? • Did the program adhere to the agreed upon terms? If not, why? • Doesthe project design,structure, logic or management need to be modified to facilitate the achievement of desired outcomes and impact? • Is the partnership between PCI and Juarez and Associates effectively supporting improved bilingual education? • Is the partnership with Universidad del Valle leading to effective early grade teacher training and capacity improvement? • To what extent did PCI address the baseline evaluation recommendations and lessons learned? • To what degree have the needs of the beneficiary been included in the design of the program? Sustainability • How well have the projects developed and maintained collaborative relationships with appropriate government agencies, local partners, USDA and other key stakeholders? • What efforts has PCI made to transition school feeding activities to local actors (local government, private sector, federal government agencies, parents)? How successful have those efforts been? • To what extent have partners, governments, and local communitiesinternalized and implemented efforts to sustain school feeding activities, such as providing in-kind support for school feeding activities? (e.g. the emphasis on transferring the school feeding program to the local government, private partners and MINEDUC, MAGA; the promotion of program elements; ensuring that specific education and health outcomes are being achieved, etc.)? • What steps has PCI taken and/or plans to take to ensure the program’s sustainability? Which ones show the greatest or least potential to contribute to sustainability? Why? Outcomes and Impact • To what degree have program activities contributed to improved literacy, teaching, student attention and attendance, and nutrition? • To what degree have health, hygiene and nutrition behaviors and practices promoted by the project been adopted by the targeted beneficiaries? What evidence existsto demonstrate the relationship between project activities and outcomes and impact related to literacy, health and nutrition? • Has participation in WE groups led to increased school involvement and improved student outcomes (particularly for school attendance)? • Has the project met targets for key performance indicators? If targets are not being achieved, why not? • Are there any unplanned barriers to effectiveness? If yes, how has the project adapted? • What is the level of satisfaction of the direct beneficiaries or participants? Gender: cross-cutting topic to evaluate equality and gender dynamics, such as female participation and decision-making power. • To what extent are project activities reaching female and male beneficiaries equally? • What differential outcomes and impacts, if any, are project activities having on women and girls? Men and boys? 125 • To what extent is the project actively addressing gender-specific barriers and constraints to education and community engagement? • What specific activities and interventions have driven the program to promote gender equality, ensuring equal access to education and academic achievement for boys and girls? • Do men and women value education differently? Has household decision making changed among households with WE group members, particularly related to decisions around their children’s education, and importance of girls attending schools? Measurement of reading comprehension results: PCI has worked with Juarez & Associates on a series of activities to improve the reading comprehension of early grade students. To measure the level of progress achieved, MINEDUC will be asked to provide standardized national literacy tests to measure reading comprehension with third grade students, just as was done during the baseline. The evaluation must determine: • What changes in reading comprehension have happened since the baseline evaluation? • Is there a correlation between literacy outcomes and characteristics of the student, teacher and school? For example, what isthe relationship between the amount of literacy instruction in the classroom, quality of literacy instruction, class size, sex of the teacher, hunger level and nutrition of the student, and access to libraries and reading corners? • Have teachers improved the way to implement literacy instruction in the classroom as a result of participating in teacher training on literacy? For example, are there changes in classroom organization, literacy instruction content, teaching methods, or use of different teaching materials? IV. EVALUATION METHODOLOGY To improve the quality and usefulness of the midterm evaluation findings, a quasi-experimental impact evaluation design is proposed using a mixed methods approach, which is in line with the baseline study. All qualitative and quantitative methods used during the baseline study will be repeated during the MTE, only in beneficiary schools. Additional questions may be added to obtain key additional information. To ensure data comparability, quantitative data collection tools used during the baseline surveys will not be modified. While quantitative data can provide a level of standardization in terms of specific numbers, qualitative data on the other hand can be triangulated to confirm findings, improving the quality and validity of the study while providing a greater level of detail and "contextualization" of program results. The midterm evaluation should include a control group such as a sample of schools and respondents who are not involved in PCI’s EDUCAMOS program, to provide a counterfactual for the evaluation. Control group selection should be done in such a way as to ensure comparability between the control group and intervention schools and should be consistent with the reference sample selected. Sampling Stategy PCI will prepare the quantitative and qualitative sampling strategy in advance of implementing the evaluation and will provide a detailed sampling strategy to the consultant during the start-up phase. The scope of the evaluation includes all schools that participate in the EDUCAMOS program, including students, parents, teachers, and school administrators. The evaluation consultant is expected to review and understand the sampling strategy. See Annex 1 for more details on the sampling strategy. Additional qualitative data will be collected from key stakeholder groups to triangulate and contextualize the quantitative findings and help answer the key questions listed above, in addition to determining which program activities and results are considered the most relevant. Consider the use of the following data collection methods. 126 Proposed Data Collection Methods Beneficiary Group Proposed Data Collection Methodologies Data Collected Teachers Random sample of teachers who have been in school for a minimum of one year prior to data collection • Knowledge and practice of methodologies related to general education and literacy in particular • Knowledge, practice and existing interventions related to health, hygiene, nutrition, and water and sanitation • Potential and effective progress towards the sustainability of school feeding (example of areas that should be evaluated and taken into consideration for sustainability such as government support, etc.) 3rd grade students All 3rd grade students from schools selected in the sample • Literacy test 1st grade students One class per school selected in the sample • Observation of a class to determine the level of attention 4th-6th grade students Randomized sample • Knowledge and practices related to literacy, health, hygiene, nutrition, dietary consumption, and sanitation habits outside of school School Directors Intentional sampling; school directors from each of the schools selected in the sample will be interviewed. • School characteristics (program related activities, infrastructure, gardens, etc.). • Quantitative indicators, such as school enrollment, attendance levels, attention levels and academic achievement (which will be cross-referenced with MINEDUC data, whenever available). • Potential for progress towards the sustainability of school feeding programs and literacy. Parents and School Volunteers Random sample of those who have been members for at least one year • Participation in program activities and training. • Potential and effective progress towards the sustainability of school feeding • Availability to support family farming in line with the School Feeding Law 127 MINEDUC Administrative Technical Coordinators (CTAs) Intentional sampling; a representative from each of the program municipalities will be interviewed • Coordination and communication between different actors. For example, members from the communities, schools, municipal governments and organizations such as the Municipal Commission on Health and Food and Nutritional Security (COMUSSAN) • Potential and effective progress towards the sustainability of school feeding The consultant will be required to analyze and report on the following key performance indicators using primary data collected during this mid-term evaluation. Indicator IDs in the below table reference whether the indicator is custom or a standard indicator based on the 2016 FFP/ McGovern-Dole Indicator Guidance. Data for additional indicators collected by PCI through routine monitoring will be provided to the consultant for a complete picture of midterm project performance. Required Indicators Indicator ID Indicator Disaggregation McGovern-Dole #26 Percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text Sex Custom Percent of students who are off task (data collected in a single day, at specific intervals)” Sex Custom Percent of students who indicate that they are “hungry” or “very hungry” during the school day Sex Custom Percent of students who state that illness is the primary reason they are absent from school Sex Custom Percent of school-age children receiving a minimum acceptable diet Sex Custom Percent of mothers (or care providers) in target communities who can identify at least three key practices aimed at safe food preparation and storage practices Custom Percent of primary school aged children enrolled in USDA assisted schools Sex Standard #19 Number of schools using an improved water source Standard #20 Number of schools with improved sanitary facilities Standard #1 Number of students regularly attending (80%) USDA supported classrooms/schools Custom Percent of schools in which at least 80 percent of teachers were present on 90 percent of school days Custom Percent of parents in targeted communities who can identify at least three important health/ hygiene practices (e.g. use of latrines) Mothers/Fathers Custom Percent of parents in target communities who can name at least three benefits of primary education Mothers/Fathers McGovern-Dole #3 Number of school administrators in targeted schools who demonstrate use of new techniques or tools as a result of USDA assistance 128 McGovern-Dole #5 Number of teachers/educators/teaching assistants in target schools who demonstrate use of new and quality teaching techniques or tools as a result of USDA assistance Custom Percent of teachers in target schools who demonstrate improved literacy instruction as identified by supervisors, mentors, or coaches The consultant will conduct a separate analysis of the quantitative data for each group of schools (intervention and control). Midterm results should be calculated based on each indictator’s definition and presented in comparison to baseline results and to the control results. Appropriate tests of statistical significance should be used to compare baseline results to midterm results for the KPIs listed above. The consultant is also expected to analyze and present key results not related to the above KPIs, as relevant to answering the evaluation learning questions. Qualitative data should be analyzed using appropriate techniques, such as content analysis, and should go beyond simply pulling out key quotes. The consultant will present PCI with an analysis plan during the start-up process that explains the data analysis approach, methodologies and tools. The MTE must include a participatory approach. Structured interviews should be conducted with the following key stakeholders: • Representatives of municipal councils (mayors and council members) • Local authorities (Community Development Councils, Auxiliary Mayors) • Staff from MINEDUC and MAGA in Huehuetenango (i.e. DIGEDUCA) • Community volunteer staff: collaborating mothers, lead mothers, education volunteers • Municipal councils: COMUDE, COMUSSAN • Women Empowered Groups • PCI staff • USDA/FAS Guatemala staff • Others Once the MTE is completed, PCI will conduct a Quality Circle1 with program staff and other key stakeholders, in order to review the MTE’s findings, conclusions and recommendations. PCI will also invite USDA to participate in the workshop. If USDA staff are not able to participate in the workshop, PCI will share with them the initial report and request their feedback either before or after the workshop. The workshop will prioritize the recommendations and participants will develop action plans to take any immediate corrective action needed to address the findings. The EDUCAMOS program director will follow up with technical coordinators to ensure that action plans are implemented, and Monitoring & Evaluation staff will evaluate the effectiveness of corrective actions to address the identified challenges. 1 Participatory methodology adopted by PCI to conduct comprehensive analysis of project implementation, with the participation of 100% of PCI staff, key stakeholders from local schools, communities, etc. The MTE report and particularly the findings, conclusions and recommendations, will serve as the basis for the quality circle analysis and the following preparation of the 2020 work plan, which will be submitted to USDA. V. KEY TASKS OF THE CONSULTANCY Desk review of key project documents and secondary data sources o Review of project documents to understand the project design, local context, progress to date and baseline evaluation methodology and results o Meet with USDA as requested to review evaluation design Designing and developing a Work Plan that addresses the MTE questions described above and adjusts to a quasi-experimental evaluation design that can be replicated in the final evaluation. The plan must include, at a minimum: o A mixed-methods study design (framework, methodology, etc.) that adheres to the guidelines established below o A data collection plan that allows to report on the required indicators and disaggregations o A quality control plan o An analysis plan detailing the methodologies for analyzing data, presenting and visualizing results and performing tests of statistical significance o An updated timeline for completion of the consultancy Developing data collection tools and protocols for the MTE, in coordination with PCI staff o Review and refine tools and protocols as appropriate o Hire and train enumerators, team leaders/supervisors o Validate data collection tools Conducting data collection o Hire a team for the collection of quantitative and qualitative data o Train data collection team on the sampling strategy, data collection tools, use of cell phone apps for data collection, and data quality control protocol o Manage and secure quantitative and qualitative datasets o Monitor incoming data collected on mobile devices through an implementation management dashboard and automated email alerts. Communicate data quality issues to the data collection team o Review, code and clean quantitative datasets before analysis. Data coding and cleaning will be enabled by the use of mobile data collection platforms, but the consultant will be responsible for ensuring that all data meet PCI data standards. Conduct mid-term evaluation analysis o The consultant will be responsible for consolidating the results of the quantitative and qualitative analyses into a complete final report. Produce the comprehensive mid-term evaluation report o Use PCI guide for report writing and editing o Provide PCI and relevant stakeholders with sufficient time for reviewing and providing feedback on the draft reports o Incorporate feedback from PCI and other stakeholders and present the final report Provide a summary presentation o Conduct a presentation of key evaluation findings for program staff and other stakeholders VI. DELIVERABLES The Consultant must provide the following deliverables: 1 1. Inception report that includes evaluation design, methodology, analysis plan, tests of statistical significance, quality assurance plan, share out and knowledge management plan, and implementation timeline 2. Update or create data collection tools, as appropriate/needed, in consultation with PCI 3. Evaluation report outline 4. Presentation of initial results to PCI, prior to writing evaluation report 5. Draft midterm evaluation report 6. Final version of evaluation report approved by PCI. Reports will be in line with PCI guide for evaluation report writing and editing, as well as PCI evaluation report minimum standards 7. Complete, cleaned evaluation quantitative and qualitative data sets with corresponding documentation/data library. Quantitative datasets must adhere to PCI data standards, to be provided to consultant 8. Quantitative analysis outputs, in MS Word 9. Documentation of qualitative analysis including themes analyzed and frequencies VII. RESPONSIBILITIES Consultant Responsibilities • Submit technical and financial proposal with a clear description of methodologies being proposed • Conduct consultations with all evaluation informants • Hire, train and supervise enumerators for quantitative data collection • Conduct collection, entry and analysis of data from quantitative surveys • Collect qualitative data using facilitators with ample experience conducting these types of processes • Review relevant documents and literature • Analyze qualitative and quantitative data • Provide deliverables according to the agreed upon evaluation timeline • Cover all the logistics (transport, accommodation, etc.) related to data collection activities PCI’s Responsibilities • Facilitate consultations with relevant program stakeholders • Provide contacts in target areas • Develop tools in Mobenzi • Review the draft report and provide feedback to the consultant • Translate the final report from Spanish to English, or English to Spanish • Approve the deliverables • Make payment for services according to what is established in the contract • Participate in the training of enumerators on quantitative data collection • Organize the location for training enumerators VIII. TIMELINE The consultancy is expected to begin on June 8, 2019 and conclude on September 13, 2019. Activities Timeline Document review and kick-off June 8, 2019 2 Submission of inception report, including work plan, finalized evaluation design, and analysis plan June 21, 2019 Finalize data collection instruments, develop instruments in mobile data collection platform July 5, 2019 Enumerator training, finalize logistics July 8-12, 2019 Data collection July 15 - August 2, 2019 Data analysis, presentation of report outline and submission of first report draft August 23, 2019 PCI report review August 30, 2019 Submit final report September 13, 2019 IX. QUALIFICATIONS The candidate or consultant team must meet the following minimum requirements: • Academic background in social / education and/or health or related field; minimum of master’s degree • Experience in leading project evaluations using a range of qualitative and quantitative data collection and analysis methods for quasi-experimental designs • Experience leading evaluations in the areas of international development, specifically education programs and public health; experience in evaluating school feeding programs strongly preferred • Experience in the evaluation of U.S. government/USDA projects strongly preferred • Knowledge of Guatemala context, particularly around national legislation relating to education, health and nutrition preferred • Demonstrable capacity to deliver high quality outputs within the proposed timeline • Verbal and written fluency in English; one team member must speak Spanish to communicate with the Guatemala team 9.7 EQSA staff for this consultancy 1. Lic. Julio Antonio of la Roca Cacacho 2. Lic. José Gilberto Mena Alberto 3. Lic. Víctor Manuel Palacios Marroquín 4. Licda. Alma Patricia Galicia Gómez 5. Ana Carolina Samayoa Paniagua 6. Ericka Eugenia López 7. Roberto Enrique Maldonado Garavito 8. Carlos Humberto Pérez Enumerator team 9. Magnely Elodia Morales Ramírez 10. Yesenia Marili Ramirez Pérez 11. Juana Cristina Antonio Mateo 12. Eulalia Angela Raquel Toledo Mateo 13. Eugenia Diaz Antonio Office Staff 14. Licda. Andrea María de la Roca Alfonso 15. Licda. Silvia Beatriz Morales Cifuentes 16. Avelina García (data entry) 17. Carlos Santos Pérez (data entry) ;