Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) FINAL REPORT Office of Food for Peace (FFP) December 2018 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by ICF Macro, Inc. FINAL PERFORMANCE EVALUATION OF THE FOOD SECURITY PROGRAM FOCUSED ON THE FIRST 1,000 DAYS (SEGAMIL) FINAL REPORT OFFICE OF FOOD FOR PEACE (FFP) Submitted by: ICF Macro, Inc. 530 Gaither Rd., Suite 500 Rockville, MD 20850 Contract number: GS-00F-189CA/7200AA18M00002 Prepared by: Cristina de la Torre, Team Leader, Health and Nutrition Specialist Thoric Cederstrom, Food Security and Livelihoods Specialist Alma Olivet, Gender and Youth Specialist Chris Seremet, WASH Specialist Gheda Temsah, Data Analyst January 22, 2019 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) i CONTENTS Acronyms............................................................................................................................................................................ iv Executive Summary ........................................................................................................................................................... v 1. Introduction.................................................................................................................................................................1 1.1 Context...............................................................................................................................................................1 1.2 Program Background .......................................................................................................................................2 2. Evaluation Overview..................................................................................................................................................6 2.1 Evaluation Purpose ...........................................................................................................................................6 2.2 Evaluation Questions.......................................................................................................................................6 3. Evaluation Methods and Limitations......................................................................................................................8 3.1 Data Collection Methods................................................................................................................................8 3.2 Design for the Baseline and Endline PBS.....................................................................................................8 Sampling ..............................................................................................................................................................8 Data Analysis......................................................................................................................................................9 3.3 Design for Qualitative Data Collection.......................................................................................................9 Focus Group Discussions.............................................................................................................................10 Key Informant Interviews.............................................................................................................................10 In-depth Interviews.........................................................................................................................................11 Direct Observation ........................................................................................................................................11 Site Selection....................................................................................................................................................11 Evaluation Team..............................................................................................................................................12 3.4 Limitations of the Evaluation Methodology ..............................................................................................12 4. Evaluation Findings...................................................................................................................................................13 EQ1: To what extent has the project met its defined goals, purposes, and outcomes?.........................13 SO1: Food access of farmer households improved................................................................................13 SO2: Chronic malnutrition among vulnerable rural populations in targeted micro-watersheds reduced .............................................................................................................................................................18 SO3: Local and municipal resilience systems in food security improved..........................................26 Cross-cutting Factors Affecting Achievement of Project Objectives.................................................27 Changes in Food Security .............................................................................................................................28 EQ2: For each technical sector, what are the strengths of and challenges in the project’s design, including theories of change that influence the effectiveness of the interventions?.................................29 EQ3: In each technical sector, what are the strengths of and challenges of the effectiveness of the interventions’ implementation?.................................................................................................................32 Implementation Approaches and Methodologies That Contributed to Achieving Project Outputs.............................................................................................................................................................32 Implementation and Contextual Factors Associated with the Project’s Effectiveness in Producing Outputs of Higher or Lower Quality.....................................................................................36 EQ4: Which project outcomes are likely to be sustained?............................................................................39 Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) ii EQ5: What are the key lessons and best practices that should inform future projects in the country?..........................................................................................................................................................42 Best Practices...................................................................................................................................................42 Lessons Learned..............................................................................................................................................43 5. Recommendations....................................................................................................................................................46 Annexes Annex 1: Evaluation Statement of Work Annex 2: Population-based Survey Methods Annex 3: Evaluation Matrix Annex 4: SEGAMIL Impact Indicators Annex 5: PBS Tables Annex 6: Summary of Methodology and Results of Supplemental Multivariate Analyses Annex 7: Tables for Multivariate Analyses of Stunting and Underweight Annex 8: Tables for Multivariate Analyses of Per Capita Consumption Expenditures Annex 9: Data Collection Instruments Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) iii FIGURES Figure 1: Results Framework—CRS Guatemala Title II DFAP—SEGAMIL ........................................................3 Figure 2: Sustainable Crop Practices Used by Farmers..........................................................................................13 Figure 3: Sustainable Livestock Practices Used by Farmers..................................................................................15 Figure 4: Children’s Nutritional Status.......................................................................................................................19 Figure 5: Children’s Feeding Practices........................................................................................................................20 Figure 6: Components of MAD Indicator for Children 6–23 Months of Age...................................................20 Figure 7: Breastfeeding Status for Children 0–23 Months of Age, by Age in Months.....................................21 Figure 8: Food Groups Consumed by Women 15–49 Years of Age..................................................................22 Figure 9: Water and Sanitation Facilities and Prevalence of Diarrhea in Children..........................................23 Figure 10: Water Treatment Prior to Drinking .......................................................................................................24 Figure 11: Knowledge of Danger Signs in Pregnancy and Childhood .................................................................25 Figure 12: Women’s Health-seeking Behavior..........................................................................................................26 Figure 13: Prevalence of Poverty: Percentage of People Living on Less Than $1.25 per Day, and Mean Depth of Poverty .............................................................................................................................................................29 TABLES Table 1: Focus Group Discussions Conducted for Performance Evaluation ....................................................10 Table 2: Key Informant and In-depth Interviews for the Performance Evaluation ..........................................11 Table 3: Value Chain Revenue and Sales—April to June 2018 .............................................................................18 Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) iv ACRONYMS BCC behavior change communication COCODE Community Rural Development Committee COMUDE Municipal Development Committee (Comité de desarrollo municipal) CRS Catholic Relief Services DFAP Development Food Assistance Project DMM Municipal Department for Women ECADI Integrated Human Development Field School (Escuela de Campo para el Desarrollo Integral) EQ evaluation question FFP Office of Food for Peace FGD focus group discussion IDI in-depth interview KII key informant interview M&E monitoring and evaluation MAD minimum acceptable diet MAGA Ministry of Agriculture MOH Ministry of Health MTE mid-term evaluation NRM natural resource management OMAS Municipal Technical Office for Water and Sanitation (Oficina Municipal de Agua y Saneamiento) PBS population-based survey PEC Extension Coverage Program (Programa de Extensión de Cobertura) PHAST Participatory Hygiene and Sanitation Transformation SARAR Self-esteem, Associative strength (of a group), Resourcefulness, Action-planning and Responsibility SEGAMIL Food Security Program Focused on the First 1,000 days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días) SILC savings and internal lending community SO strategic objective USAID United States Agency for International Development UTAM Municipal Agricultural Technical Office WASH water, sanitation, and hygiene WMC water management committee Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) v EXECUTIVE SUMMARY The Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) is a six-year United States Agency for International Development Office of Food for Peace Title II Development Food Assistance Project that was awarded to Catholic Relief Services in 2012. The project was jointly implemented with Pastoral Social Caritas of San Marcos and the Association for Integral Development for the Western Highlands. The project aimed to improve food security in the Western Highlands of Guatemala through a multi-faceted approach that addresses households’ access to food, high rates of stunting, and local and municipal resilience systems. Coverage focused on eight municipalities in the departments of Totonicapán and San Marcos. SEGAMIL sought to improve food security for the most vulnerable households, namely those that are landless or small landholders. In those households, the project specifically targeted pregnant and lactating women and children under 2 years of age for improved health and nutrition. SEGAMIL aimed to reach 23,500 direct participant families over the course of the project. The SEGAMIL project design called for an integrated approach to combat food insecurity and extreme rates of chronic malnutrition in the targeted municipalities. Interventions were aimed to increase access to food through improved agricultural production of smallholder farmers, diversify income sources and establish savings and internal lending community (SILC) groups, and link farmer groups to markets. SEGAMIL’s approach to increase the nutritional status of children under 5 years of age was based on the first 1,000 days approach and the adoption of beneficial health, nutrition, and hygiene behaviors by mothers. Interventions included food transfers, behavior change communication (BCC), growth monitoring of children under 2 years of age, and case management of children with acute malnutrition. Educational and training activities at the community level were conducted through Integral Human Development Field Schools (Escuelas de Campo para el Desarrollo Integral [ECADIs]), which allowed for the integration of agriculture, nutrition, and health activities. Finally, the project worked to strengthen local and municipal organizations so they could better respond to community needs, particularly as those needs relate to food insecurity. This report presents the findings and conclusions of the final performance evaluation of the SEGAMIL project. The purpose of the evaluation is to measure the project’s development outcomes, understand which project outcomes are likely to be sustained, and document lessons learned that can inform the design of future projects. EVALUATION DESIGN The evaluation used a mixed methods approach that included the collection and analysis of qualitative and quantitative data, direct observation, and a document review. Quantitative data from two population-based surveys (PBSs) conducted in 2013 and 2018 were used to derive standard Food for Peace indicators and compare estimates between baseline and endline. The PBSs are statistically representative of the target areas where the SEGAMIL project was implemented and include households that may or may not have received SEGAMIL project interventions so that change for participants and non-participants in the target area can be evaluated for specific outcomes. The evaluation also collected qualitative data to get a deeper understanding of how the SEGAMIL project has affected the lives of project participants and to assess its achievements and challenges related to project implementation and sustainability. This included 41 focus group discussions, key informant interviews with 103 respondents, 10 in-depth interviews, and direct observation of homes, field schools, home gardens, livestock, and water, sanitation, and hygiene facilities. The evaluation team conducted visits and interviews in seven communities across five of the eight project municipalities, and interviewed government staff in three municipalities and at one departmental head. Project documents and monitoring data were also reviewed. Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) vi KEY FINDINGS Data from the baseline and endline PBS found significant reductions in poverty levels and improvements in childhood nutrition across the SEGAMIL project area. The percentage of people living on less than $1.25 per day decreased by more than half, from 43.8 percent in 2013 to 19.2 percent in 2018. The prevalence of stunting in children under 5 years of age decreased, from 79.6 percent in 2013 to 72.2 percent in 2018, and the percentage of children under 5 years of age who were underweight decreased, from 34.5 percent in 2013 to 25.3 percent in 2018. In comparison, national stunting levels had only decreased by 3 percentage points (from 50 percent to 47 percent) between 2008–2009 and 2014–2015.1 The data and study design, however, do not allow us to attribute these changes to SGAMIL specifically. Multivariate analyses indicate that increases in daily per capita consumption expenditures are correlated with households’ making savings and with the use of agricultural practices promoted by the project, including the use of sustainable crop practices, livestock practices, and value chain activities. Higher daily per capita consumption is also associated with lower odds of stunting and underweight. These findings support the project’s hypothesized pathways of how the uptake of sustainable agriculture practices contributes to increased consumption expenditures, which in turn translate into improved nutrition. SEGAMIL provided trainings to participants so they could adopt sustainable agricultural practices and improve household food production. The project contributed to the uptake of home gardens, herb production, fruit tree management, and animal rearing. Several participants indicated that the home gardens in particular had helped diversify their families’ diets, although water shortages and lack of land were impediments to their maintenance. Animal rearing was fairly ubiquitous and done primarily for resale and not for household consumption. Through the ECADIs and SILC groups, participants were able to access loans and invest them in productive activities. Most participants opted to invest in animal-rearing activities (mostly chicken and pigs) and were making small and much appreciated earnings from these enterprises. The SILC groups are likely to continue operating because they are self-sustaining, allow access to credit, and are an important safety net to families. The revolving funds made available through the ECADIs at the end of the project also show great promise for similar reasons and have been an important incentive for members to keep participating in ECADI activities even after food transfers were terminated. The project also supported a small number (approximately 500) of farmers to establish production groups and link them to markets. Some of these value chain activities were done in coordination with Feed the Future projects. These value chain activities, which included tomato, potato, honey, and coffee production, were very successful in generating impressive earnings for group members, who tended to reinvest part of the funds to further grow the business. The technical skills acquired by these groups were strong and evident in the quality of the products. Although these income-generating activities were successful, the project could have done more to diversify income beyond agriculture production, given that the poor households that SEGAMIL targeted have limited access to sufficient land to participate successfully in value chains. The project’s BCC intervention focused on improving infant and child feeding practices, antenatal care and nutrition for women, and health-seeking behaviors; in the second half of the project, it also focused on improving hygiene. The BCC approach developed by the project was strong, as evidenced by the high levels of knowledge seen in participants. The evaluation team found that many participants were able to easily describe the optimal hygiene and nutrition practices for children and were also able to explain why these practices were important. The combination of instruction at the ECADIs and home visits was seen as beneficial by promoters and participants alike. The promoters noted that home visits to 1 Ministry of Health and Social Assistance, National Statistics Institute, ICF International. 2017. Encuesta Nacional de Salud Materno Infantil 2014-2015. Informe Final. Guatemala. Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) vii participant households allowed them to assess practices, identify barriers impeding behavior change, and provide personalized counseling. Feeding practices for children 6–23 months of age improved significantly between 2013 and 2018. The proportion of children 6–23 months of age getting the minimum acceptable diet increased, from 18 percent in 2013 to 40 percent in 2018, and was significantly higher among participants compared to non-participants. Most of this improvement in minimum acceptable diet came from improved dietary diversity, although some can be attributed to increased feeding frequency. The food transfers and especially the voucher system allowed families to obtain a wide variety of foods, including eggs and Incaparina. Food demonstrations were highly valued by participants, and many said that these demonstrations were critical in getting them to incorporate new foods in their diets. The sustainability of the nutrition outcomes is questionable, and many women reported that they do not have the means to continue purchasing the same quantity and variety of foods that they had when getting food vouchers. The home gardens, fruit trees, and animal rearing can help diversify diets somewhat, but these are unlikely to produce enough food or income to sustain current levels of consumption. Another important achievement was the reduction of diarrhea in children under 5 years of age, from 42.6 percent in 2013 to 28.2 percent in 2018. This change was significantly associated with increased access to an improved drinking water source and to a lesser extent with the use of improved sanitation facilities. SEGAMIL worked with local water management committees and municipal water and sanitation offices to improve water systems, repair infrastructure, and establish water chlorination systems. Increased access to piped water, combined with widespread chlorination, is likely to account for much of the reduction in diarrheal diseases. The water management committees and municipal water and sanitation offices are well trained, but limited funding may hamper their ability to make future repairs to the system. Building the technical and managerial capacity of municipal offices was a key strategy of the SEGAMIL sustainability plan. The idea was to strengthen coordination between community organizations and government institutions to better respond to the food security needs of their constituents. To this end, SEGAMIL provided training to community organizations (community rural development committees and ECADIs) as well as a range of technical municipal offices that focus on women, agriculture, forestry, and water and sanitation. SEGAMIL surpassed the targeted number of organizations that it set out to train. The trainings were well received, and many of the organizational staff indicated that they had a much stronger understanding of the food security problem and that they felt better informed about the roles and responsibilities of their respective entities and the government processes for mobilizing resources for community interventions. On the other hand, they also felt constrained in their ability to respond, because of the lack of funds available. SEGAMIL was highly successful in empowering women in the communities it served. The trainings that women received on self-esteem and leadership have by their own account made a big impact on their lives and were greatly valued by many participants we interviewed. Women report the trainings as being an awakening that enabled them to recognize that they have a voice and a right to advocate for themselves and their families. The ECADIs provided a space for them to learn and practice technical and managerial skills and demonstrate their ability to lead an organization. Some women eventually managed to integrate into the community rural development committees, which are traditionally male organizations, although the process was not always welcome or easy. This is probably one of the most successful and sustainable impacts of the project. Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 1 1. INTRODUCTION 1.1 CONTEXT According to the World Bank, Guatemala has the largest economy in Central America and one of the strongest economies in Latin America, with an average gross domestic product growth rate of 3 percent in the past five years.2 However, it is also one of the few countries in the region where poverty has increased in recent years, from 51 percent in 2006 to 59 percent in 2014, and around two-thirds of the population live on less than USD$2 a day. In addition, Guatemala has high levels of inequality, with indigenous populations (which comprise about 40 percent of the total population) often having much worse rates of poverty, malnutrition, and mortality and reduced access to education and health services. In rural areas, the proportion of the population living in extreme poverty is 35 percent, and among indigenous populations it is 40 percent.3 Nearly half of Guatemalan children under 5 years of age are stunted due to chronic malnutrition.4 The proportion is higher among poor (66 percent), rural (53 percent), and indigenous populations (58 percent). In the regions covered by the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) project, 70 percent of the children in Totonicapán are stunted, and 55 percent of the children in San Marcos are stunted. Chronic undernutrition is attributed to high rates of poverty, food insecurity, inadequate hygienic environments, structural problems of inequality and exclusion, poor education, and insufficient child stimulus and care. Lack of financial resources among the rural poor is repeatedly cited as the primary barrier to a diversified diet, improved agricultural practices, and access to health care.5 In the Western Highlands, the rural poor rely primarily on subsistence farming and agricultural wage labor to get by. These families typically have small landholdings or none at all and are limited in the amount of food they can produce.6 The plots owned by poor families in these regions are often in mountainous, rocky areas, which further limits potential production. A 2011 Catholic Relief Services (CRS) study found that small landholders in the Western Highlands tended to plant monocrops, typically beans or maize, and were seeing diminishing returns due partly to low-quality seeds, poor land management practices, and ineffective pest and disease control.7 In recent years, Western Guatemala, including the departments of Totonicapán and San Marcos, have suffered from prolonged drought conditions, which have severely affected food production and food security. The situation was exacerbated by a rust plague that has affected coffee crops since 2012, impacting small farmers in particular and limiting employment opportunities for farm laborers.8 In 2015, the Famine Early Warning Systems Network declared several municipalities of the Western Highlands as 2 World Bank. 2017. “Guatemala Country Overview.” http://www.worldbank.org/en/country/guatemala/overview 3 Instituto Nacional de Estadística. 2015. Encuesta Nacional de Condiciones de Vida 2014 Principales Resultados. Guatemala. https://www.ine.gob.gt/sistema/uploads/2015/12/11/vjNVdb4IZswOj0ZtuivPIcaAXet8LZqZ.pdf 4 Ministry of Health and Social Assistance, National Statistics Institute, ICF International. 2017. Encuesta Nacional de Salud Materno Infantil 2014-2015. Informe Final. Guatemala. 5 ICF. 2013. Population-Based Survey. CRS. 2014. Informe Análysis de Barreras: Comportamientos de Nutricón y Agricultira. Guatemala. 6 CRS. 2012. SEGAMIL Technical Proposal submitted to FY12 Request for Applications by CRS to DCHA/FFP in Response to FY12 Request for Applications to Title II Development Food Aid Programs. 7 Ibid. 8 Famine Early Warning Systems Network. 2016. Households Affected by Drought and Coffee Rust Require Assistance. Key Message Update. http://fews.net/central-america-and-caribbean/guatemala/food-security-outlook-update/january-2016 Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 2 being in crisis.9 The current situation is still precarious, with lower than average rainfall already starting to affect crops.10 These areas are also at increased risk of experiencing climate shocks from hurricanes, floods, and landslides resulting from climate change. There is evidence that the impacts of El Niño and La Niña are becoming more frequent and more severe and will continue to adversely impact these communities.11 The communities in the Western Highlands have long been underserved by government programs. They lack infrastructure and access to basic services such as water, electricity, and health care. The SEGAMIL 2013 baseline study found that only 20 percent of households in the CRS project area were using an improved water source, and half were using a non-shared improved sanitation facility.12 Poor hygienic practices and water supplies that are often contaminated with fecal bacteria contribute to high levels of diarrheal diseases, further eroding the nutritional status of children. The Guatemalan health system is under-funded, and resources are often concentrated in urban areas. Access to health care in rural areas is limited, and the health facilities experience frequent stockouts of essential medicines and vaccines.13 In 2013, the Ministry of Health (MOH) cancelled its Extension Coverage Program (Programa de Extensión de Cobertura [PEC]). In 2012, this program provided primary care through a network of nongovernmental organizations to approximately 54 percent of the rural population. The cancellation left many rural and indigenous households without access to essential maternal and child health services, including ones that are critical for preventing and treating malnutrition.14 1.2 PROGRAM BACKGROUND SEGAMIL is a six-year United States Agency for International Development (USAID) Office of Food for Peace (FFP) Title II Development Food Assistance Project (DFAP) that was awarded to CRS and its partners15 in 2012. Field implementation ended in September 2018. The project aimed to improve food security in the Western Highlands of Guatemala through a multi-faceted approach that addresses households’ access to food, nutrition and health behaviors, and local and municipal resilience systems. Coverage focused on eight municipalities in Totonicapán and San Marcos, which were selected based on the levels of poverty and food insecurity as well as opportunities to collaborate with other USAID projects. SEGAMIL sought to improve food security for the most vulnerable households, namely those that are landless or small landholders who had limited access to markets. In those households, the project specifically targeted pregnant and lactating women and children under 2 years of age for improved health and nutrition. SEGAMIL aimed to reach 23,500 direct participant families over the course of the project. 9 Famine Early Warning Systems Network. 2015. Households Remain in Crisis, with Greatest Severity in the Western Highlands. Key Message Update. http://fews.net/central-america-and-caribbean/guatemala/food-security-outlook￾update/november-2015 10 Famine Early Warning Systems Network. 2018. http://fews.net/es/central-america-and-caribbean/guatemala 11 World Food Programme. 2016. To Reduce El Niño's Impact on Central America's Dry Corridor, Build Resilience and Invest in Sustainable Agriculture. https://www.wfp.org/node/649968 12 ICF. 2013. Population-Based Survey. 13 Avila, C., R. Bright, J. Gutierrez, K. Hoadley, C. Manuel, N. Romero, and M. P. Rodriguez. 2015. Guatemala Health System Assessment. Bethesda, MD: Health Finance & Governance Project, Abt Associates Inc. 14 Llanque, R. 2015. Integration of the Extension of Coverage Program (PEC): Effects on Project Outcomes and Lessons Learned. Guatemala: Cuaramericas Global, Inc. https://www.curamericas.org/wp-content/uploads/2016/01/Appendix-13-Effects￾of-the-Integration-of-the-Extension-of-Coverage-Program.pdf 15 Pastoral Social Caritas of San Marcos and the Association for Integral Development for the Western Highlands Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 3 The SEGAMIL project design was in response to the 2011 Country Specific Information for six-year Title II program in Guatemala (2012–2018) issued by FFP.16 The Country Specific Information highlighted the need for an integrated approach to combating food insecurity and the extreme rates of chronic malnutrition in the Western Highlands of Guatemala. Bidders were encouraged to consider a range of interventions aimed at improving livelihoods and income-generation opportunities for rural households; increasing agricultural production of smallholder farmers; encouraging dietary diversity through crop diversification, food transfers, and the adoption of optimal health and nutrition behaviors; and strengthening local organizations so they can better support the target population and facilitate linkages to Feed the Future projects for poor farmers. The Country Specific Information also emphasized prioritizing intensive work with households over reaching large numbers of participants. The SEGAMIL results framework reflects these priorities (Figure 1) and organized its interventions around three strategic objectives (SOs): SO1: Food access of farmer households improved SO2: Chronic malnutrition among vulnerable rural populations in targeted micro-watersheds reduced SO3: Local and municipal resilience systems in food security improved Figure 1: Results Framework—CRS Guatemala Title II DFAP—SEGAMIL Goal: Food security in targeted micro-watersheds in 8 municipalities in the Guatemalan Western Highlands improved SO1: Food access of farmer households improved SO2: Chronic malnutrition among vulnerable rural populations in targeted micro-watersheds reduced S03: Local and municipal resilience systems in food security improved IR1.1: Sustainable agricultural production practices among men and women improved. IR1.2: Sustainable natural resource management improved. IR1.3: Diversification of income sources among men and women increased. IR1.4: Linkages between farmers groups and market processes increased. IR2.1: Household health practices improved. IR2.2: Household nutrition and care practices improved. IR2.3: Use of health services increased. IR3.1: Community management of food insecurity and emergency situations improved. IR3.2: Municipal management of food insecurity and emergency situations improved (with their own resources). IRCC 1: Female decision-making either dependently or independently with spouse on use of income improved. IRCC 2: Female decision-making either independently or jointly with spouse to seek health services for herself and her children improved. IRCC 3: Female participation and leadership in the community improved. Cross-Cutting: Female empowerment to make decisions for the food security of their families improved. IR=intermediate result Under SO1, SEGAMIL aimed to improve access to foods through improved agricultural production, improved sustainable natural resource management (NRM), diversification of income sources, and strengthened linkages to markets. Interventions implemented by the project included the use of home gardens to produce micronutrient-rich foods, livestock management to increase animal rearing and animal food consumption, production and post-harvest storage of basic grains to maximize yields and minimize losses, production of organic fertilizer, and use of soil conservation techniques. The project 16 USAID, Office of Food for Peace. 2012. Country Specific Information: Guatemala. https://pdf.usaid.gov/pdf_docs/PDACU283.pdf Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 4 provided inputs such as seeds, animals, and building materials to support these efforts. Integrated Human Development Field Schools (Escuelas de Campo para el Desarrollo Integral [ECADIs]) were established to serve as training centers to provide education and hands-on training, demonstrate techniques, and engender buy-in from communities for the approaches being promoted. NRM activities initially focused on improving watershed management by households and community organizations using the Integrated Water Resources Management Approach. The project established reforestation nurseries and promoted the planting of trees, including fruit trees, to improve soil and watersheds. SEGAMIL sought to develop projects that could earn carbon credits to generate income for poor households. Based on recommendations from the mid-term evaluation (MTE), many of these NRM activities were discontinued. Acknowledging that current level of food production is insufficient to meet the needs of households and overcome the effects of climate shocks, SEGAMIL sought to diversify income sources. The project did this primarily through the creation of savings and internal lending community (SILC) groups that would allow households to access funds when needed, attain greater financial resilience, and invest in productive assets. SEGAMIL also worked to organize smallholder producers and link them to value chains. For SO2, SEGAMIL’s approach to increase the nutritional status of children under 5 years of age is based on the first 1,000 days approach and the adoption of beneficial health, nutrition, and hygiene behaviors by mothers. Interventions included food transfers, intensive behavior change communication (BCC), monthly growth monitoring of children under 2 years of age by health promoters, identification and referral of cases of acute malnutrition, and follow-up support and counseling to households with malnourished children. SEGAMIL promoted the use of health care and worked to teach mothers danger signs in pregnancy and in childhood illnesses. The project supported women to develop contingency plans so they would be prepared to go to the hospital when labor started. These health and nutrition activities were largely centered at ECADIs, where education and BCC could be integrated with agriculture activities. In addition, the project worked to increase access to safe water by working with local water management committees (WMCs) and municipal water and sanitation offices to repair and chlorinate water systems. The project also promoted household use of improved sanitation facilities. For SO3, the project worked to strengthen local and municipal organizations to monitor food security, assess community needs, develop plans and proposals, and advocate resources to implement the plans and proposals. SEGAMIL conducted a number of trainings to increase awareness of the problem of food security, improve understanding of the laws governing community rural development committees (COCODEs), build organizational capacity of local and municipal organizations, and increase coordination across community and municipal levels to address community needs and improve resilience. Initially, SEGAMIL also worked to establish sentinel site early warning systems, but this was discontinued after the MTE. The MTE identified areas in which the project was not functioning as intended and recommended adjustments to the project design. These included but were not limited to the following:  Improve targeting of the project interventions  Provide more targeted and context-specific livelihoods assistance  Develop more focused BCC  Improve linkages to municipal and government institutions  Increase emphasis on water, sanitation, and hygiene (WASH) activities  Begin to phase out food rations  Better position the project interventions for sustainability Following the MTE, the SEGAMIL project underwent a significant restructuring to address many of these recommendations. The project redesigned how it targeted interventions to households, introduced Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 5 WASH messaging in BCC, transitioned to a food voucher system, sought to involve men and youth more in project activities, developed customized training modules, and began engaging more actively with local and municipal governmental entities to improve sustainability of project activities. SEGAMIL also developed an exit and sustainability plan that delineated five pillars of sustainability on which the project would focus:  Institutionalization of key interventions  Demand driven by the market for project interventions  Accreditation of local human resources  Formation of alliances and local links  Social and institutional changes Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 6 2. EVALUATION OVERVIEW 2.1 EVALUATION PURPOSE FFP contracted ICF to conduct the final performance evaluation of the SEGAMIL project. The purpose of the evaluation is to measure the project’s development outcomes, understand which project outcomes are likely to be sustained, and document lessons learned that can inform the design of future projects. The evaluation provides evidence regarding the types of interventions that contribute to improved nutrition and food security of poor rural households, as well as implementation and contextual factors that affect their success. Key audiences for this report include USAID, implementing partners, and Guatemalan government institutions tasked with improving the health and welfare of rural populations in the Western Highlands of Guatemala. 2.2 EVALUATION QUESTIONS The evaluation questions (EQs) are as follows: EQ1: To what extent has the project met its defined goals, purposes, and outcomes? 1.1 Were the targets set by the project and key FFP indicators met? Did the key program indicators improve over time? 1.2 What factors promoted or inhibited the achievement of the project objectives, including, but not limited to, the effectiveness of sustainable agriculture, income diversification, NRM, and BCC interventions? 1.3 What were the targeting strategies, and how did they contribute to achieving project goals (especially with regard to gender and reaching the most vulnerable)? 1.4 What has been the contribution of SEGAMIL to USAID’s efforts to reduce food insecurity among households in the Western Highlands? EQ2: For each technical sector, what are the strengths of and challenges in the project’s design, including theories of change that influence the effectiveness of the interventions? 2.1 Were the technical interventions relevant to achieve project outcomes, especially to achieving food security and improved nutritional status for mothers and children under 2 years of age? 2.2 How plausible were the pathways and the determinants to achieving the key outcomes? Were there any, highlighting any unforeseen pathways leading to unintended positive and negative consequences of the project? EQ3: In each technical sector, what are the strengths of and challenges of the effectiveness of the interventions’ implementation? 3.1 How effective and relevant were the approach and methods of the technical interventions to achieve project outputs of greater or lesser quality? 3.2 What implementation and contextual factors were associated with greater or lesser effectiveness in producing outputs of higher or lower quality? 3.3 Which interventions and implementation processes were deemed more or less acceptable to members of the target communities? Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 7 EQ4: Which project outcomes are likely to be sustained? 4.1 What was the quality and functionality of systems, processes, and institutional arrangements established and strengthened independently by the project, as well as in collaboration with the private sector, the Government of Guatemala, community organizations, and research organizations to achieve project outcomes and sustainability? 4.2 Do the necessary resources and capacity strengthening exist to sustain service providers? What is the likelihood that service provision will continue after the project ends? 4.3 What were the communities’ perceptions of the quality, frequency, and effectiveness of the implementation processes and services provided? 4.4 Were the community and participants motivated to demand and pay (or invest time) for the services? What were the communities’ perceptions on sustainability of the services provided by the project? 4.5 To what extent did the project leverage other U.S. Government and non-U.S. Government investments to achieve sustained outcomes as identified in the results framework? 4.6 What evidence exists of enhanced linkages with other service providers? EQ5: What are the key lessons learned and best practices that should inform future projects in the country? 5.1 What are the lessons, best practices, and approaches that should be considered in designing and implementing future food and nutrition security projects? 5.2 What are the lessons learned with regard to achieving sustainable outcomes in food security programs? 5.3 Based on the lessons learned, how can potential unintended negative consequences be minimized, and positive consequences be captured systematically? Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 8 3. EVALUATION METHODS AND LIMITATIONS 3.1 DATA COLLECTION METHODS The overall design for this final evaluation of the SEGAMIL DFAP used a mixed methods approach that included the collection and analysis of qualitative and quantitative data, direct observation, and a document review. Quantitative data from two population-based surveys (PBSs) were used to estimate changes in key FFP indicators over time. Focus group discussions (FGDs), interviews, and observation allowed a deeper examination of how the SEGAMIL project has affected the lives of project participants and an assessment of the achievements and challenges related to project implementation and sustainability. The qualitative findings helped complement, contextualize, and better understand the survey results. The document review examined progress and challenges in project implementation over the course of the project and explored the contextual issues that affect the food security and livelihoods of families in the Western Highlands and that may have influenced the implementation and success of the SEGAMIL project. 3.2 DESIGN FOR THE BASELINE AND ENDLINE PBS The quantitative results for the performance evaluation use the data collected through baseline and endline PBSs that are statistically representative of the target areas where the SEGAMIL project was implemented. The baseline PBS was conducted from April to mid-June 2013, and the endline PBS was conducted from June to July 2018. The primary objective of the baseline PBS was to assess the status of FFP and custom indicators prior to project implementation; the primary objective of the endline PBS was to assess the change in these same indicators at the end of the six-year DFAP cycle by undertaking statistical tests of differences between baseline and endline measures. This pre-post design allows for the measurement of change in indicators over time; however, this design does not allow statements to be made about attribution or causation relating to project impact. The PBS design includes households that may or may not have received SEGAMIL project interventions so that change for participants and non-participants in the target area can be evaluated for specific outcomes. The indicators are related to food security, poverty, women’s and children’s health and nutritional status (including anthropometry), WASH, and agricultural practices. Custom indicators were added by the project to assess women’s health care awareness and decision-making for themselves and their children. The endline PBS used the same data collection instrument as the baseline PBS, with additional questions to calculate the Food Insecurity Experience Scale and the Poverty Probability Index, as well as questions on mobility, remittances, and exposure to project interventions. Sampling The household samples selected for the baseline and endline PBSs are based on a multi-stage clustered sample designed to power a test of differences between the baseline and endline estimates for the FFP child stunting indicator. The sample size is derived based on a formula to detect a difference in the prevalence of stunting from baseline to endline of 6 percentage points with 95 percent confidence and 80 percent power. Based on this calculation, the baseline PBS included 75 communities and 3,000 households, and the endline PBS included 35 communities and 1,400 households.17 These sample 17 Because the input parameters were somewhat different for the two time points, the sample size requirement for the endline was updated from a baseline value of 1,686 children per project to an endline value of 895 children per project, given that this Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 9 size calculations included a pre-inflation factor to take into account anticipated household-level non-response, as well as an adjustment factor to take into account the number of households that need to be contacted to achieve the required sample size of children under 5 years of age. A detailed description of the endline sampling plan is provided in the Guatemala Endline PBS Protocol, and a detailed description of the baseline sampling plan can be found in Annex 1 of the baseline study report.18 The characteristics of the sampled population are provided in Annex 5, Table A5.1 Data Analysis The baseline and endline PBS data were used to assess change over time in 20 FFP and 8 custom indicators. The data were also used, as appropriate to the EQ, to conduct additional descriptive analyses to help strengthen findings established from the quantitative and qualitative data. Multivariate analyses controlling for key socioeconomic and project-specific factors as covariates were used to explore the factors that were associated with the prevalence of stunting among children under 5 years of age, prevalence of obesity among non-pregnant women 15–49 years of age, and daily per capita consumption expenditures. These indicators were selected for additional analyses to help inform learning on key impact and outcome indicators that showed significant improvement over the project lifetime. The multivariate models pooled data from the baseline and endline surveys and controlled for a number of confounding factors, in addition to project participation status, to empirically explore hypothesized relationships between indicators as per the project theory of change and to determine whether project participation is associated with change over time net of other factors. The designation of participating versus non-participating project households was based on household respondents’ answers to a set of questions about exposure to the project interventions. One respondent per household was asked, “Have you or someone from your household participated in SEGAMIL project activities?” Respondents who answered “Yes” were considered to be project participants. Respondents who answered “No” were considered to be non-participants. Respondents who reported that they or someone in their household participated in the project were asked about the type of assistance received. Project assistance could include food rations, cash transfers, nutrition training or meetings, agriculture-related training or meetings, or other types of assistance. The responses were not validated by the project, so it was not possible to determine definitively that the respondents accurately reported the source and type of project assistance. Respondents or households were likely to be recipients of more than one type of assistance, so it was not possible to disentangle the effects of different types of project interventions. 3.3 DESIGN FOR QUALITATIVE DATA COLLECTION In September and October 2018, the evaluation team collected qualitative data through FGDs, key informant interviews (KIIs), and in-depth interviews (IDIs) with a range of stakeholders, including project participants and non-participants, SEGAMIL staff, community promoters, community organizations, combination of sample sizes for baseline (1,686) and endline (895) ensured an overall power of 80 percent for the two time points. In principle, the sample size of required children should be identical for the baseline and endline PBSs, because the computation is made at the time of the baseline to ensure that the pair of sample sizes provides an overall power of 80 percent. However, the actual realized sample size of children from the baseline PBS (2,778 children per project) greatly exceeded the target sample size for the baseline PBS (1,686 children per project). This implied that the overall power of the statistical test would likely be considerably greater than 80 percent had the same sample size used at baseline also been used at endline, because the power of the statistical test is determined by the actual realized sample size of children at the two time points. Therefore, it was possible to reduce the sample size at endline relative to what it was at baseline and still maintain 80 percent power overall. 18 A complete description of the baseline study is provided in the Baseline Study for the Title II Development Food Assistance Projects in Guatemala report. https://www.usaid.gov/sites/default/files/documents/1866/Guatemala%20Baseline%20Study%20Report%2C%20March%202014.pdf Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 10 municipal government offices, USAID FFP staff, and private sector partners. In addition, the evaluation team conducted site visits to participant homes, farms, ECADIs, and value chain sites to gauge how participants are applying the practices promoted by the SEGAMIL project. Focus Group Discussions The evaluation team facilitated 41 FGDs with a range of stakeholders, as listed in Table 1. The team used semi-structured FGD guides that were tailored for each respondent category. Given the breadth of interventions included in the project, the FGD guides covered a range of technical areas, and the subject matter experts on the team were given the flexibility to pursue lines of inquiry that developed. For project participants, FGD sessions were conducted separately for men and women. FGDs with project participants addressed their personal experiences with the project, the perceived benefits from participation and from specific interventions, and motivation and likelihood of continued practice. Project participants were purposively selected based on their participation in the ECADIs, savings and loan groups, or value chain activities. FGDs were also conducted with community promoters to assess the ease with which they were able to carry out their duties, key achievements and challenges faced, and intention to sustain their promotion activities in the future. Finally, FGDs were held with members and leaders of various community and municipal organizations to explore their perceptions of the project. Table 1: Focus Group Discussions Conducted for Performance Evaluation Type of Respondent Number of FGDs Project participants: members of ECADIs or SILC groups (women) 9 Project participants from value chain groups (mixed gender) 5 Community promoters (agriculture, health, mother monitors, educators, and nutritionists) (mostly women but occasionally mixed gender) 5 Male partners of project participants (men) 3 ECADI and SILC board members (women) 4 COCODE members (mixed but often all men) 6 Municipal organizations (COMUDE and COMUSAN) (mixed gender) 5 WMCs 4 TOTAL 41 COMUDE= Municipal Development Committee (Comité de desarrollo municipal), COMUSAN= Municipal Food and Nutrition Security Commission Key Informant Interviews KIIs were conducted at the community, municipal, and national levels with various types of respondents, as specified in Table 2. The KII guides were semi-structured, tailored to the type of respondent, and explored themes related to project achievements, factors that promoted or inhibited project activities and outcomes, perspectives on the effectiveness of project interventions, motivations and capacity to sustain activities, mechanisms that have been established to foster sustainability of project interventions and outcomes, and lessons learned. Key informants included SEGAMIL staff, community promoters, leaders of community organizations, municipal government staff, and USAID FFP staff. KIIs were also conducted with non-project participants in SEGAMIL target areas to assess their perceptions of the project interventions, and whether there were any spillover effects. Key informants were purposefully selected based on their roles in the project or their organizations and their familiarity with the SEGAMIL project. At times, the team interviewed two to four key informants at a time. Table 2 lists the total number of key informants interviewed. Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 11 Table 2: Key Informant and In-depth Interviews for the Performance Evaluation Type of Respondent Number of KIIs SEGAMIL staff (CRS and its partners, Association for Integral Development for the Western Highlands and Pastoral Social Caritas of San Marcos) 53 Staff from municipal or departmental government offices 30 Community promoters 7 Providers participating in the voucher system 4 Teacher 1 Non-participants in SEGAMIL communities 4 Staff from other USAID projects that collaborated with SEGAMIL 2 USAID staff 2 TOTAL 103 Number of IDIs Project participants 10 In-depth Interviews IDIs were conducted with 10 project participants to get a more detailed understanding of how project participation impacted their livelihoods and food security. These respondents were identified during site visits or through the FGDs. Direct Observation The evaluation team conducted site visits to observe homes, field schools, hygiene practices, demonstration plots, farms, home gardens, livestock, post-harvest processing, and WASH facilities, among others. Observation allowed the evaluation team to examine how practices promoted by the project are being applied. Site Selection The evaluation team collected data at the departmental, municipal, and community levels, covering both departments in which the SEGAMIL project operated: Totonicapán and San Marcos. The evaluation team conducted visits and interviews in seven communities across five of the eight project municipalities, and interviewed government staff in three municipalities and at one departmental head. Community selection was done purposefully, with the intention of selecting communities that were in different municipalities, were supported by both of SEGAMIL’s partners, were located in diverse geographic and climactic zones, included Quiché and Mam communities, and that taken together included the range of interventions supported by SEGAMIL. Community selection was done using a list of all communities served by SEGAMIL. The sampling frame was narrowed by excluding communities that were involved in the endline survey to avoid respondent fatigue and by excluding 25 communities that were not deemed safe due to ongoing community conflicts, drug trafficking activities, or reported robberies.19 Using maps, the evaluation team also excluded communities that were not easily accessible within a day’s travel (based on road infrastructure and distance) from central hubs where the evaluation team could be lodged. From the remaining communities, the evaluation team used random selection to obtain a shortlist of 30 communities. The evaluation team then selected six communities using the criteria described in the preceding paragraph and based on a plausible travel route for the evaluation team. A seventh community was selected even though it did not appear on our shortlist because of the unique set of interventions implemented there related to value chains, work with women’s groups, and work with coffee producers. 19 This was based on information provided to the endline survey team. Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 12 Evaluation Team The evaluation team comprised a Team Lead with expertise in health and nutrition, an Agricultural and Food Security Specialist, a WASH Specialist, and a Gender and Youth Specialist. The team shared responsibility for all data collection, analysis, and report preparation tasks. The team was supported in the field by Mayan language interpreters (Quiché and Mam). 3.4 LIMITATIONS OF THE EVALUATION METHODOLOGY The PBS sample was intended to be representative of the entire population residing in the SEGAMIL project area and not only the project participants. Only 37 percent of respondents in the endline survey reported having participated regularly in project activities. Estimates of change over time do not reflect the experience of direct project participants only and may show less improvements than if only project participants were sampled. The analysis did examine difference by participation status for a few key impact indicators (Annex 5, Table 5.3) and adjusted for this in multivariate analyses. However, most of the data reported throughout this report are representative of the full population of the SEGAMIL project area. These estimates may be capturing the effects of other food security projects that were simultaneously operating in the SEGAMIL project area even if they were working with other target groups. The survey did not capture participation in other projects. For the qualitative component, we spoke primarily to participants who were still engaged in project activities at the time of our visit because they were the ones who were most likely to participate in the FGDs. The evaluation team interviewed a few women who had never participated or were no longer active participants and one woman who had belonged to an ECADI group that had since disbanded. The majority of respondents, however, were active and likely represent the more motivated and better organized ECADIs and savings and loan groups. It should be noted that the community selection process included all communities, regardless of whether the ECADIs were still operational; all seven communities visited had operating ECADIs at the time of our visit. The evaluation has triangulated the qualitative findings with data from the PBS and the project’s performance monitoring data from the Indicator Performance Tracking Table. Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 13 4. EVALUATION FINDINGS EQ1: TO WHAT EXTENT HAS THE PROJECT MET ITS DEFINED GOALS, PURPOSES, AND OUTCOMES? Overall, the food security targets of SEGAMIL were met and in some cases surpassed. This section provides a detailed look at the project’s achievements across its various strategic objectives. SO1: Food access of farmer households improved Agricultural Practices According to the PBS, several of the agricultural practices promoted by SEGAMIL had significant uptake between 2013 and 2018. Overall, the proportion of farmers who used at least 3 sustainable agricultural practices in the previous 12 months increased significantly, from 50.1 percent in 2013 to 63.8 percent in 2018 (Annex 4). The PBS found significant increases in the proportion of farmers using household gardens, managing fruit trees, and producing native herbs (Figure 2 and Annex 5, Table A5.4), activities that were promoted by SEGAMIL. The use of organic fertilizer was not significantly different at endline. The use of soil conservation practices, however, decreased significantly between 2013 and 2018. Figure 2: Sustainable Crop Practices Used by Farmers 2.7 57.1 69.9 31.2 28.5 20.1 55.7 13.5 2.7 * 49.0 ns 75.7 *** 48.7 ** 41.0 *** 33.1 * 63.6 ns 13.2 0 20 40 60 80 100 Use of improved seeds for beans Use of soil conservation practices Use of organic fertilization Household garden production Management of fruit trees Production of native herbs Use of at least two sustainable crop practices No sustainable crop practices Percentage 2013 Baseline 2018 Endline Significance for test of differences: ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 Soil Conservation Per the annual reports, there was a concerted effort by SEGAMIL to reduce soil erosion with contour farming, earth barriers, hedgerows, forage grasses, and forage trees. Given the hilly nature of the target geographies, the SEGAMIL participants were quite cognizant of the effects of soil erosion and the need to mitigate it. Some practices, such as living barriers (grass planted at the edge of terraces), were readily evident in some communities, both among participants and non-participants, but it was not clear whether this was due to the project or whether perhaps it pre-dated it. The evaluation team observed few soil improvement practices, such as mulching, crop diversification, crop rotation, and reforestation. The exception was in farming activities in which higher-value crops, such as beans and tomatoes, are produced as part of the value chain activities. Soil conservation practices and the use of organic fertilizer were more likely to be adopted by participants in value chains, Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 14 who felt that the investment was warranted, given the likely return on investment. The number of participants participating in value chain activities was fairly low (only about 500 in total). For subsistence farmers, the labor required to implement some of the soil conservation efforts, such as trenching, may have been a factor inhibiting uptake on plots that grow maize only because the economic return is very low. A few respondents indicated that the use of organic fertilizer had improved crop output. They also noted a cost savings because they no longer had to purchase chemical fertilizers. Participants in the coffee and potato value chains were especially appreciative of the interventions in increasing productivity, reducing production costs, and generating greater income. The low participation rate of men in the agricultural trainings likely limited the uptake of these soil conservation practices. Many of the female participants noted that they would share what they learned in the trainings about land management and soil conservation with their husbands, who typically controlled the decisions about agricultural land, and, if the husbands agreed, they would implement the promoted practice. Home Gardens Many ECADI participants reported having established a home garden through SEGAMIL and listed a wide range of vegetables and herbs grown in them. The evaluation team heard from a number of women that the home gardens helped increase vegetable consumption for their families and led them to try new vegetables and herbs. Some noted that growing their own produce had reduced household expenditure, and in some cases it even generated some income through local sales to neighbors. At the time of the final evaluation visit, the number of FGD participants who reported having an active household garden was fairly low (only 1 or 2 participants out of 14 or more in some FGDs). For those who did not maintain home gardens, commonly cited reasons were lack of land and lack of water for irrigation. Of note, 2018 had a period of drought in the dry corridor that extended from June to August, during a period critical for growing crops.20 The lack of rain affected the planting and success of home gardens. Furthermore, the evaluation team visit (in late September/early October) coincided with the end of the growing season so few plants were seen growing. Other commonly cited reasons for not maintaining gardens were not wanting to purchase seeds (which had previously been provided by the project), poor soil conditions, and failed attempts to germinate seeds. A few participants mentioned time and effort as an additional constraint, whereas others stated that tending the garden was not onerous. Some promoters noted that setting up a garden was a requirement to receiving food assistance through their ECADI, and some participants were no longer motivated to continue. The evaluation team observed several household gardens in both SEGAMIL and non-SEGAMIL households. The household gardens that were observed varied greatly in terms of quality, how well tended they were, and the quantity diversification of cultivars. In addition to access to land, the evaluation team found that individual motivation, combined with competing demands on a woman’s time, were key in determining the presence and quality of a household garden. Some women were quite motivated, and some were actively seeking out seeds (some even from Mexico) to grow in their gardens. A few participants were trained in seed-saving techniques for open-pollenating varieties. Per the annual reports, SEGAMIL had promoted the use of grey waters for home gardens to address irrigation problems; however, the evaluation team did not see this being practiced. Participants were concerned about using soap and dirty water on their food. Rain catchment tanks, promoted by the project, were reported as being too expensive for participant households to adopt. Alternative, 20 Famine Early Warning Systems Network/ Incio Temprano de la epoca de escasez de alimentos en 2019. http://fews.net/es/central-america-and-caribbean/guatemala/food-security-outlook-update/september-2018 Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 15 lower-cost technologies for water management and dry land gardening did not seem to have been explored. Native Herbs Native herbs were often cited as plants grown in the home gardens. Native herbs were also observed growing in the traditional milpas. They were easy to obtain and grow. Some women mentioned learning to prepare them in different foods as a result of the food demonstrations. Fruit Tree Management SEGAMIL distributed fruit trees in some of the municipalities. Only a few examples of fruit trees were observed during the final evaluation field site visits. In one case, the trees did not look healthy. In the other, they were growing nicely and had started producing fruit. SEGAMIL staff indicated that the prolonged drought over the course of the project affected the health and survival of the fruit trees. Livestock Practices The proportion of farmers using at least two sustainable livestock practices increased significantly, from 10.2 percent to 23.5 percent over the course of the project (Figure 3 and Annex 5, Table A5.4). The increase was mostly driven by the uptake of improved livestock containment facilities (corrals, shelters, etc.), which were used by 32.9 percent of farmers at baseline and 57.2 percent of farmers at endline. The majority of project participants were able to describe the benefit of this practice for protecting their animals and for improving hygiene and preventing illness. In addition, having a livestock containment facility was a precondition for being able to receive animals distributed by the ECADI with its revolving funds. This was an added incentive for households to build them. Figure 3: Sustainable Livestock Practices Used by Farmers 32.9 27.1 10.2 50.3 *** 57.2 32.7 ns 23.5 *** 33.6 0 20 40 60 80 100 Improved livestock facilities Vaccination of poultry Use of at least two sustainable livestock practices No sustainable livestock practices Percentage 2013 Baseline 2018 Endline Significance for test of differences: ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 Vaccination of poultry in the previous 12 months improved slightly but not significantly (Figure 3). SEGAMIL annual reports indicate a reduction in the number of families reached through vaccinations in 2018 (2,448 families) compared to 2017 (7,088 families), possibly due to the phase out of the project.21 Paravets, usually the agriculture promoters, were trained to provide this service in their community for a nominal fee (around 2 Quetzales per chicken). Several promoters indicated that there was demand for 21 As reported in the SEGAMIL Fiscal Year 2017 Annual Report and Fiscal Year 2018 Annual Report Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 16 vaccination services, and the evaluation team heard from project participants that vaccination was valued because it safeguarded their investment in animals. In some cases, participants indicated that they did not vaccinate animals because the ones they obtained through the revolving funds came pre-vaccinated. The evaluation team was not able to verify whether that was the case. Some of the promoters mentioned that cultural beliefs (believing vaccines were poisonous) and a lack of familiarity with the practice acted as barriers to animal vaccination in some communities. The cost to households of the different practices affected their uptake. In the early years, SEGAMIL provided participants with the materials and equipment needed to adopt these practices (e.g., materials to build animal pens, fencing and seeds for home gardens, and silos for grain storage) but stopped doing this after a while for most participants (some exceptions were made for participants in the dry corridor who were highly vulnerable to food insecurity). Many participants reported that a lack of money was a key barrier to the adoption of these practices; this was especially true for rainwater catchment and grain storage (metal silos), but some mentioned that getting the materials for the construction of livestock containment facilities was a factor in their non-adoption. However, CRS mentioned that cost-share was never a problem and that participants had a variety of options from which they could choose. NRM Practices The PBS data indicate that there was no significant uptake of NRM practices, namely reforestation activities, in the SEGAMIL project area (Annex 5, Table A5.4). It should be noted, however, that many of the project’s reforestation activities were carried out at the municipal or community level and may not be reported by individual farmers. SEGAMIL established 102 communal nurseries and 6 municipal nurseries in the first few years of the project and subsequently worked with municipal forestry offices to improve technical skills and management of these nurseries. Farmers reforested 1,056 plots. SEGAMIL worked at the municipal and community levels to reforest and set up municipal nurseries to produce forest tree seedlings, some of which were still operational at the end of the project because they were generating income. This intervention was not prioritized after the MTE because the decision was made to reduce the number of interventions. The evaluation team observed some seedling nurseries and reforestation efforts in Comitancillo. In Tajumulco, the evaluation team visited a coffee seedling nursery that was growing rust-resistant varieties. The women were not only producing for their own coffee plantations but also for sale to neighbors, and this was a significant source of income for them. The women were also producing high-grade compost using the vermiculture technique promoted by SEGAMIL in which to grow the coffee seedlings. This reinforces our observation that many of the promoted agricultural practices will be more sustainable if there is a concrete and direct return on investment. Diversification of Income Sources SEGAMIL’s strategy to diversify income relied primarily on creating SILC groups, establishing revolving funds for the ECADIs with seed capital investment, and integrating a small number of famers into value chains. 22 SEGAMIL formed 387 SILC groups, which were composed primarily of women, and 359 of them were still active at project closeout. 23 At first the participants were hesitant to save, but eventually the benefits became apparent, and many other community members requested to be included in SILC groups. Demand for this intervention was very high. The PBS data, however, show a significant decrease in the number of farmers who report saving money (from 7.3 percent to 2.5 percent) (Annex 5, Table A5.5). Because the question about savings was asked 22 The project did not promote the use of other financial services captured in the survey (such as agricultural credit or farmer insurance ), and the proportion of households reporting their use did not change over time. 23 SEGAMIL Fiscal Year 2018 Annual Report Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 17 of the primary farmer in the household (typically the male head of household), this indicator was probably not capturing the experiences of the hundreds of women who participated in the SILC groups. During focus groups, various SILC members stated that they were using their savings and loans to invest in small productive assets. The SEGAMIL Fiscal Year 2018 Annual Report noted that 30 percent of loans were being invested in livestock production, and the evaluation team also heard that this was a preferred investment for SILC members. Some women also reported investing their borrowings in small enterprises, such as ice making, acquiring materials for existing businesses, and buying merchandise for resale. The loans, however, are relatively small due to the limited amount these women are able to save, and thus do not allow for a large capital investment in a business. Over time, as funds grow due to interest, the groups anticipate that the size and number of the loans can grow. They also noted that new members could add to an influx of cash. Other SILC members stated that they prefer to reserve their savings for emergencies or to cover extraordinary household expenses (such school fees or health care costs) and appreciated the safety net that the SILC provides. A few women related how these loans had saved their families from financial crises and greatly valued having an emergency reserve. They noted that it also let them avoid having to borrow from neighbors or others who often charge high interest rates. One agricultural promoter stated, “Thanks to SEGAMIL, I was able to pay off the medical expenses for my daughter that were drowning me. Now I can focus on growing my agricultural endeavors.” The SILC groups were observed to be well-trained in financial management and assessing the viability of the investment proposals by members. They also knew to ask for collateral in case the money could not be paid back. The Seed Capital Investment (Fondo Semilla) was started late in the project but is showing potential to help diversify incomes. These are funds distributed to 160 ECADIs for them to sustain themselves.24 Most recipient ECADIs are using this capital as a revolving fund for members to use as investments. Generally, the members choose to invest in animals (chickens or pigs) and pay back the funds when they sell the animals. Some households report netting approximately 200–600Q from the sale of animals. This finding reinforces the validity of activities in intermediate result 1: small livestock (pigs, goats, poultry, rabbits) can be a good source of income as well as contributing to increased meat consumption, mostly on special occasions (e.g., birthdays, graduation). This dual functionality of animals has motivated families to keep them. The infrequent, occasional consumption of animals may not be a significant net contributor to household nutrition, especially for women and children under 2 years of age. Some members report that they have been unable to access these funds because the ECADI requires them to The SILC methodology is an approach to community-managed finance specifically and purposefully tailored to poor and marginalized communities. It provides a safe place for members to save, obtain loans, and facilitate social insurance schemes. Groups are self-selected and generally comprise between 10 and 30 members. Members save and then lend to each other from the accumulated group savings. Loans are approved during group meetings by all members present, and loan terms are defined by the group in its SILC constitution. Members repay loans with interest, which allows the group’s investment fund to grow and provides a return on savings. At the end of a one-year cycle, the group performs a self-audit, paying out all accumulated savings and dividends to its members. In addition, members benefit from a social fund that acts as an insurance mechanism for emergency assistance. SILC groups are autonomous and self-managed. 24 These funds were generated from family contributions for a direct food transfer program implemented by SEGAMIL in the first four years of the project. When the project switched to a voucher system, SEGAMIL had to return the funds to the community through the ECADIs and opted to create seed capital for productive investments. A total of 160 ECADIs (about half) received these funds. The ECADIs were selected based on how well they were organized and managed and the potential sustainability of their membership. Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 18 have animal pens prior to accepting animals, and they say that they lack to resources to do so. In 2018, CRS noted a lack of financial management capacity among SILC and ECADI leaders as an inhibiting factor and therefore dedicated time and effort to train them in the proper practices to manage the capital seed funds. The value chain activities under SEGAMIL focused on organizing and providing technical and marketing support to associations of producers around four value chains—coffee, honey, tomatoes, and potatoes. SEGAMIL worked to strengthen 20 producers’ organizations, which included both men and women. The Quarterly Report for Q3 2018 notes some impressive earnings by the participants (Table 3). Several value chain participants shared that the income generated through this activity represented a sizeable portion of their total household income. The participants in these SEGAMIL-supported value chain activities were very motivated to continue working, and many were reinvesting their earnings to expand and diversify their production. Table 3: Value Chain Revenue and Sales—April to June 2018 Value Chain Product Market Type Total Earnings (USD) Net Profit (USD) Honey Formal (exports and direct sales to B-Waffles, Korean community, and others in Guatemala) $50,442 $17,420 Potatoes Formal (intermediaries and supermarkets); lower grades in local markets $9,897 $5,723 Green beans and peas Formal (export to the other Central American countries and the United States through Unispice) $22,703 $12,482 Tomatoes Informal—local markets $13,899 $8,713 Source: SEGAMIL Q3 Quarterly Report, 2018 The endline survey reveals that the percentage of farmers who practiced value chain activities promoted by the project in the previous12 months increased twofold, from 11.5 percent to 23.9 percent, surpassing SEGAMIL’s target of 15 percent (Annex 4). This indicator captures persons who report at least two activities that may include involvement in market-oriented production, calculation of cost of production, and development of production and sales plans (Annex 5, Table A5.5). Given the small number of SEGAMIL participants involved in value chains, it is likely that this indicator is affected by other projects, such as the Rural Value Chain Project that had substantial geographic overlap with SEGAMIL. The SEGAMIL Fiscal Year 2018 Annual Report indicated that the project provided some training to diversify the activities in which participants were investing, such as training in soap-making, baking, and tailoring. The evaluation team saw only a few examples of this type of diversification (e.g., the honey producers value chain was producing soap). Because the majority of the investments continue to focus on agricultural production, income sources were not diversified much by the project. Nevertheless, these activities have contributed to income generation of participant households. SO2: Chronic malnutrition among vulnerable rural populations in targeted micro-watersheds reduced Childhood Malnutrition and Stunting Between 2013 and 2018, there were significant decreases in the prevalence of underweight, stunted, and wasted children under 5 years of age in the SEGAMIL coverage area (Figure 4 and Annex 4). The percent of underweight children under 5 years of age decreased by 9 percentage points, from 34.5 percent to 25.3 percent, surpassing project targets, and the percentage of children who are stunted decreased, from 79.6 percent to 72.2 percent, coming close to the target (71.6 percent) set by the project. The prevalence of wasting, already very low in 2013, also decreased significantly, to 0.9 percent in 2018. (A breakdown of these indicators by age group is shown in Annex 5, Tables A5.6a–A5.6c). Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 19 Figure 4: Children’s Nutritional Status 34.5% 79.6% 2.0% 25.3% 72.2% 0.9% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Prevalence of underweight children under 5 years of age Prevalence of stunted children under 5 years of age Prevalence of wasted children under 5 years of age 2013 Baseline 2018 Endline ** ** ** Significance for test of differences: ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 There were no significant differences in stunting levels among children under 5 years of age between project participants and non-participants at endline (Annex 5, Tables A5.3 and A5.7). Multivariate analyses show that the factors inversely associated with odds of being stunted after controlling for maternal, child, and household characteristics include use of at least two sustainable crop practices and increased per capita consumption expenditures. More detailed examination of these multivariate analyses can be found in Annexes 6 and 7. The prevalence of underweight children under 5 years of age was lower among direct project participants (22.2 percent) than non-participants (29.6 percent) at endline (Annex 5, Table A5.3), and the difference was marginally significant. In multivariate analysis, project participation was significantly associated with reduced odds of being underweight among children under 5 years of age after controlling for maternal, child, and household characteristics and other factors (Annexes 6 and 7). Additional factors found to be significantly associated with reduced odds of being underweight in multivariate regression included having a handwashing station in the household, eating fish in the previous 24 hours, higher per capita consumption expenditures, and using at least two NRM practices. Infant and Young Children’s Diets Feeding practices for children 6–23 months of age improved significantly between 2013 and 2018. The proportion of children 6–23 months of age getting the minimum acceptable diet (MAD) increased, from 18 percent in 2013 to 40 percent in 2018 (Figure 5 and Annex 4). This far exceeded SEGAMIL’s target, which was set at 27.8 percent. Prevalence of MAD was significantly higher among direct project participants than among non-participants at endline (Annex 5, Table A5.3). The improvement in MAD was primarily driven by improvements in dietary diversity, and the frequency of feedings improved marginally (Figure 6 and Annex 5, Tables A5.8–A5.9). Significant increases in the consumption of all food groups were seen among children, with the exception of the consumption of grains, which were already consumed by 92.4 percent of children at baseline. Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 20 Figure 5: Children’s Feeding Practices 61.1% 17.9% 63.7% 40.1% 0.0% 20.0% 40.0% 60.0% 80.0% Prevalence of exclusive breastfeeding of children under 6 months of age Prevalence of children 6-23 months of age receiving a minimum acceptable diet 2013 Baseline 2018 Endline *** ns Significance for test of differences: ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 Food transfers, and the voucher system in particular, were important contributors to dietary diversity and improved nutritional status. The participants noted that through the voucher system they were able to obtain a wide variety of foods. The household funds they would have otherwise used toward basic food items could then be applied to purchasing fruits and vegetables. The inclusion of eggs in the voucher food basket gave women and children much-needed access to animal protein, which they were not consuming otherwise. There was high acceptance of eggs among the participants we spoke to and among their children. However, respondents noted that the eggs produced by their own chickens were often sold rather than consumed by the family. Figure 6: Components of MAD Indicator for Children 6–23 Months of Age 57.3 27.1 92.4 32.4 20.2 14.7 36.2 46.4 32.5 66.8 53.9 93 48.4 31.5 21 51 58.7 62 0 20 40 60 80 100 Percentage with minimum meal frequency Percentage with minimum dietary diversity Grains, roots, and tubers Legumes and nuts Dairy products (milk, yogurt, cheese) Flesh foods (meat, fish, poultry, and liver/organ meats) Eggs Vitamin A￾rich fruits and vegetables Other fruits and vegetables Percentage 2013 Baseline 2018 Endline ** ** ** * † *** ** ** Significance for test of differences: † p<0.1,* p<0.05, ** p<0.01, ***0.001 The voucher system also included Incaparina, a micronutrient-fortified blended food made from maize and soy flour with a high value amino acid profile commonly prepared as a liquid porridge for children. This was widely consumed and valued by the participants. Many participants reported sharing this product with the entire family, and the evaluation team observed women and children consuming this during the FGDs. When questioned, the women said that the children (and adults) liked it a lot as a “refresco,” especially between regular meals. “It keeps the children from crying,” some said. They noted that the voucher system allowed them to purchase larger quantities but said that they could not afford to buy it without the voucher system. They said that they would probably not continue to consume it after the distribution ends due to the cost and would revert back to atole de masa (corn gruel). One Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 21 respondent explained: “Incaparina is very tasty, but very expensive! On the other hand, maize is always available and much less costly.” The endline PBS occurred in June and July 2018 while the SEGAMIL project was still providing support to the majority of communities it served. Evidence from the qualitative interviews indicates that there was subsequently some decrease in participation in activities such as savings and loans or growth monitoring, and that dietary diversity had decreased following the cessation of the food transfers. Women reported not consuming as many fruits and vegetables because their budgets did not allow for those purchases. Among some participants, home gardens also contributed to the seasonal consumption of vegetables, although not all were maintaining them, as discussed earlier in this report. Several participants also reported having learned to dedicate more time and effort to feeding young children to make sure that they get the adequate quantities. Some mentioned that they now spent more time spoon-feeding their children instead of letting them eat on their own or continued offering foods even if the child becomes distracted by other things. Generally, the ECADI participants demonstrated a good understanding of the ages at which to introduce purees and more solid foods and noted that the food demonstration techniques had helped them learn new recipes that integrated different foods, especially herbs and vegetables, into their children’s diets. Breastfeeding Practices There was no significant change in the proportion of children under 6 months of age who were exclusively breastfed, which was 61.1 percent at baseline and 63.7 percent endline (Annex 4). From birth to 6 months of age, a subset of mothers continue to supplement breastmilk with water, milk, and other non-milk liquids, and 15 percent of children 4–5 months of age are being given complementary foods (Figure 7 and Annex 5, Table A5.10). Women in the FGDs knew that exclusive breastfeeding was promoted by SEGAMIL, and none wanted to admit that they had not done this. The health promoters and mother monitors stated that supplementing with water and other liquids is an ingrained cultural practice that has been very difficult to change. The promoters also noted that mothers sometimes give medicinal teas to babies when they are ill because medicines are not always available through the health system. Figure 7: Breastfeeding Status for Children 0–23 Months of Age, by Age in Months 0 20 40 60 80 100 <2 2-3 4-5 6-8 9-11 12-17 18-23 Percentage Age in Months Not breastfeeding Exclusively breastfed Breastfed and plain water only Breastfed and non-milk liquids Breastfed and other milk Breastfed and complementary foods Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 22 Maternal Diets The PBS data also indicated significant improvements in women’s dietary diversity score, which increased from 4.0 to 4.5 (Annex 4). Figure 8 shows the change in consumption of different food groups among women 15–49 years of age (Annex 5, Table 5.11). The SEGAMIL food transfer activity allowed for household rations to protect the mothers’ and children’s rations. Figure 8: Food Groups Consumed by Women 15–49 Years of Age 99.6 49.9 15.3 4.9 45.1 33.5 49.5 41.1 58.6 99.7 ** 59.5 19.0 4.1 *** 59.5 * 39.2 53.3 40.7 *** 79.6 0 10 20 30 40 50 60 70 80 90 100 Grains, roots, and tubers Legumes and nuts Dairy products (milk, yogurt, cheese) Organ meat Eggs Flesh foods and misc. small animal protein Vitamin A dark green leafy vegetables Other Vitamin A￾rich vegetables and fruits Other fruits and vegetables Percentage Significance for test of differences: † p<0.1,* p<0.05, ** p<0.01, ***0.001 2013 Baseline 2018 Endline The livestock-rearing activities were likely to increase dietary diversity because they generated income for the household. However, the evaluation team saw little evidence that livestock rearing directly contributed to increased consumption of meat. The majority of participants noted that they have few animals and prefer to monetize rather than consume them because they have other pressing and prioritized expenses (school fees, medicine and health care, clothing, etc.). When asked why the family does not eat more of the chickens from the project, one participant summed it up succinctly by saying, “We like chicken. It is good and tasty and leaves us feeling satisfied. But if we slaughter a chicken, we eat for one day. If we sell the chicken, we can buy enough maize to feed our family for a week!” They indicated that they consume their animals only for special occasions once or twice a year. Consumption of meat seems to be infrequent; many participants report eating it less than once a week. The data on foods consumed by children and women confirm that although meat consumption has risen, it was not consumed by the majority of respondents. Use of Improved Water and Sanitation Facilities and Diarrheal Diseases The percentage of households using an improved drinking water source and using an improved sanitation facility increased significantly over the course of the project (Figure 9 and Annex 4). More than 30 percent more people between the baseline survey (18.5 percent) and endline survey (49.8 percent) reported that they now use an improved water source. Most improvements were because of increased access to piped water (Annex 5, Table 5.12). It should be noted that SEGAMIL did not construct new community water supply systems. Rather, it strengthened community-based WMCs to better operate, manage, and repair their existing systems constructed prior to SEGAMIL. In a few communities, SEGAMIL provided materials for repairs. SEGAMIL did improve the quality of water provided by these systems by installing new water chlorination equipment at existing water storage tanks to disinfect the stored water prior to distribution to the communities served. SEGAMIL ensured the protection of the water sources (i.e., springs) that provide water to the systems so that the source Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 23 water does not get contaminated prior to entering the system. SEGAMIL also created new and strengthened the existing Municipal Technical Office for Water and Sanitation (Oficina Municipal de Agua y Saneamiento [OMAS]) to support the functionality of WMCs (see SO3 for more detail). Figure 9: Water and Sanitation Facilities and Prevalence of Diarrhea in Children 18.5% 51.4% 73.1% 42.6% *** 49.8% *** 66.7% ns 76.9% *** 28.2% 0% 20% 40% 60% 80% 100% Percentage of households using improved drinking water source Percentage of households using improved sanitation facilities Percentage of households with soap and water at a handwashing station commonly used by family Percentage of children under 5 years with diarrhea in the last two weeks 2013 Baseline 2018 Endline Significance for test of differences: ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 The percentage of participants using an improved sanitation facility increased by 15 percent (from 51.4 percent to 66.7 percent). SEGAMIL did not construct household latrines but did promote the use of latrines through the ECADIs and hygiene messaging campaigns. During home visits, the promoters emphasized the importance of latrines and provided guidance on their proper construction, care, and maintenance. The evaluation team found that many FGD respondents demonstrated understanding of why latrines help improve health and could spontaneously cite the importance of using of ash and keeping the latrines covered. Although the percentage of households with soap and water at a handwashing station did not increase significantly (from 73.1 percent to 76.9 percent), various FGD participants indicated washing their own and their children’s hands more frequently after learning the importance of this practice through SEGAMIL. Tippy taps were abundant in the communities visited, and respondents valued the fact that they used less water than washing hands in a pila (water basin). Several FGD participants also noted that washing their hands with tippy taps instead of in a pila helped avoid contamination of water stored in the pilas. The proportion of households treating their drinking water increased, with only 1.7 percent at endline reporting not treating their water at all, compared to 8.7 percent at baseline (Figure 10 and Annex 5, Table A5.12). The proportion of households boiling their drinking water increased, from 88.5 percent to 96.6 percent, and the proportion using chlorine increased, from 3.7 percent to 7.6 percent. Boiling water was the preferred method for treating water. Respondents noted that they take advantage of using the heat from cooking fires after they have prepared their meals. Various respondents still noted being uncomfortable using chlorine because of the smell and the belief that it could lead to unhealthy side effects. Filters were rarely used because of the cost associated with them. Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 24 Figure 10: Water Treatment Prior to Drinking An important achievement was the significant reduction in the percentage of children with diarrhea in the previous two weeks, from 42.6 percent in 2013 to 28.2 percent in 2018 (Figure 9 and Annex 4). Respondents in various communities, including community leaders, said that they had noticed a reduction in the number of children who are sick. They attribute this directly to the improved hygiene practices promoted through SEGAMIL, citing specifically the treatment of drinking water, washing children’s hands before they eat and after they defecate, keeping animals confined, and keeping the house clean. In bivariate analyses, the only factor significantly correlated with reduction in diarrheal disease is having an improved water source in the household, and having an improved sanitation facility was only marginally significant (Annex 5, Table A5.13). It is likely that the chlorination of the water systems through SEGAMIL assistance was a major factor in the decrease in diarrheal diseases. Having a handwashing station was not correlated with reduced diarrhea, but this indicator does not capture how often the family is using it to wash hands. It is possible that more frequent handwashing reported by participants in all the communities visited is a factor in reduced prevalence of diarrhea. Water treatment in the household was nearly ubiquitous at endline, and therefore differences between those who treated water and those who did not could not be detected; however, it is important to note that water treatment increased from baseline to endline and likely contributed to reduced diarrhea. Improved household hygiene and food storage and management were not measured by the survey but also are likely to have contributed, given the reported changes in behaviors. Health and Nutrition Knowledge An important achievement of the SEGAMIL project was the transfer of knowledge to project participants through its BCC approach. Figure 11 shows that women’s ability to recognize danger signs for pregnancy, neonates, and child illnesses increased significantly between 2013 and 2018. The evaluation team corroborated these findings through interviews and FGDs with women. In all communities visited, the participants demonstrated clear understanding of the desired practices and could explain the reasons why these practices were important. This was true across the range of nutritional, health, and hygiene practices promoted by the project. When asked what they had learned at the ECADIs, women enthusiastically explained the various practices in great detail and sometimes even demonstrated proper handwashing techniques or the proper portion sizes for children. 88.5% 3.1% 3.7% 8.7% 96.6% 3.6% 7.6% 1.7% 0% 20% 40% 60% 80% 100% Boil Filter Bleach/chlorine added No treatment 2013 Baseline 2018 Endline Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 25 Figure 11: Knowledge of Danger Signs in Pregnancy and Childhood 44.7% 32.0% 39.7% 73.2% 59.8% 58.9% 0.0% 20.0% 40.0% 60.0% 80.0% Percentage of women with a child under 5 years of age who recognize two or more pregnancy danger signs Percentage of women with a child under 5 years of age who recognize two or more neonatal danger signs Percentage of women with a child under 5 years of age who recognize two or more childhood illness signs 2013 Baseline 2018 Endline *** *** *** Significance for test of differences: ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 Health-seeking Behavior The PBS data did not show significant changes in the proportion of women who seek care when danger signs are present (Figure 12 and Annex 4). It should be noted that this question is not asked if the woman says she does not know any danger signs. It a measure of health-seeking behavior among women who are able to recognize danger signs. Because the proportion of women recognizing the signs has increased over time, this means that a larger number of women are seeking care for danger signs in 2018, despite the proportion staying stable. The uptake of health services was hindered by the cessation of the MOH PEC program and the lack of a substitute to fill the void in service delivery in rural areas. For many women, distance to health facilities was an important barrier. Participants also reported frequent stockouts as a reason for not using health services, noting that it was a waste of time if they did not receive the medicines they needed. Other barriers include language barriers, poor perceived quality of care, and fear of childhood vaccines based on misconceptions of side effects. The SEGAMIL project also made an effort to promote joint decision-making in health care decisions and to raise awareness among men and mothers-in-law of the importance of early treatment when danger signs are present. The evaluation team heard from a few women that their partners were better informed and supportive. Various women also noted that they found the birth planning process promoted by SEGAMIL very helpful. This process encouraged the woman and her partner to plan ahead for transportation, child care, and health care costs that will be needed at the time of delivery. One woman related that she had been hesitant to go to the hospital to deliver but eventually went on her husband’s insistence. Participants also noted the benefit of learning more traditional, herbal-based treatments for some childhood illnesses, especially given the common stockouts at health facilities. The evaluation team had some concerns that promoting herbal medicines might end up discouraging appropriate health-seeking behaviors when the child is really sick. Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 26 Figure 12: Women’s Health-seeking Behavior 66.9% 75.7% 84.2% 71.4% 71.0% 84.0% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Percentage of women with a child under 5 years of age seeking health care when pregnancy danger signs are present Percentage of women with a child under 5 years of age seeking health care when neonatal danger signs are present Percentage of women with a child under 5 years of age seeking health care when childhood illness signs are present 2013 Baseline 2018 Endline ns ns ns Significance for test of differences: ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 SO3: Local and municipal resilience systems in food security improved The PBS did not include indicators related to this strategic objective. Per the Indicator Performance Tracking Table, SEGAMIL met most of its targets related to strengthening community and municipal organizations. As of fiscal year 2017, it surpassed the targeted number of organizations that applied technologies or management practices with U.S. Government assistance. These organizations included women’s groups, water users’ associations, community organizations like the COCODEs, and producer organizations. All the observation points—interviews with SEGAMIL, municipal staff, COCODEs, Municipal Development Committees (Comités de desarrollo municipal [COMUDEs]), etc.—confirmed that this component performed well and strengthened the capacity of the various entities to function better around their mandates and most notably improved their interactions with their constituencies—the food insecure. Water Management Committees Each community water supply system has a WMC that is responsible for operating, maintaining, and repairing its water system. WMCs consist of a variety of positions, each with unique responsibilities but structured so that any one position can cover for another when necessary, especially when making repairs. General positions and their primary responsibilities include President (coordinates activities, conducts community meetings, manages materials, maintains records), Secretary (keeps list of households using the water system, keeps meeting notes), Treasurer (collects and stores water usage fees, maintains financial accounting book, disburses money for repairs), and Vocals (plumbers/generalists) (make repairs, manage repair tools and supplies, organize support from community members, operate the chlorination system, conduct residual chlorine testing of treated water, read water meters). The positions are voluntary but elected. Few members are women. SEGAMIL provided a variety of technical support to the WMCs, including providing operations and maintenance manuals and conducting trainings in the repair of pipelines, use of the chlorination equipment, and use of residual chlorine testing kits. SEGAMIL also provided office supplies and pipeline repair materials. Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 27 Municipal Technical Office for Water and Sanitation SEGAMIL created new or strengthened existing OMAS. The purpose of the office is to provide water system technical support to WMCs. The responsibilities include training WMCs, visiting communities to conduct water chlorination sensitization, providing chlorine tablets to WMCs, training and supporting WMCs in the use of the chlorination equipment, and conducting residual chlorine testing. The OMAS is led by a Coordinator or Technician and may have an assistant, plumbers, and water meter readers on staff. Terms are typically for one year, but some staff have been kept longer by the Mayor’s office because they have been recognized for good performance. SEGAMIL initially provided 100 percent of the funding for the operations of the office. This was scaled down by 25 percent per year until the municipality took over full funding responsibility. OMAS supports the sustainability of urban water supply systems in addition to rural water supply systems, and supports communities within and outside SEGAMIL. SEGAMIL provided technical support through trainings, which included providing operations and maintenance manuals. Training topics included the following: protecting water resources in the watershed, particularly the water sources used by communities; operations and maintenance, particularly pipelines and chlorination equipment; advocating women’s involvement in water supply services; and developing and enforcing water user rules, such as how to use the water system, payment of water usage fees, payment of fines for late payments, and administrative responsibilities. Sanitation and more comprehensive water quality testing activities are the responsibility of the MOH and the water and environmental sanitation inspectors. Cross-cutting Factors Affecting Achievement of Project Objectives Seasonal migration was reported by promoters in various communities as a hindrance to working with families. Sometimes entire families would leave the project area and not be able to complete the trainings and BCC learning sequence, thus negatively affecting behavior change. Seasonal migration also interrupted growth monitoring and the promoters’ ability to understand particular barriers or issues that may be affecting a particular child. High levels of alcoholism and domestic violence in some communities were cited as barriers to participation in project activities. Staff indicated that often both parents abused alcohol and could not effectively participate in the project or provide the care needed to the child. Project’s targeting approach. The SEGAMIL project used an approach that targets children in the first 1,000 days of life (from conception to 2 years of age), which represents a period of high vulnerability to stunting. This targeting approach is an accepted best practice by nutrition professionals, given the strong evidence that nutrition deficiencies in this window cause grave and irreversible damage to child development. This targeting strategy was also selected to align with existing MOH priorities. Although technically sound, the targeting strategy created conflicts and resentment in some communities in which poor mothers felt a need for services but their children were not eligible to participate. Some communities opted out of the project altogether because they felt that such selective targeting was too controversial. They preferred that all needy families, regardless of the child’s age, benefit from the interventions. The project also aimed to reach the poorest and most food insecure households. The MTE found that the use of the first 1,000 days as a criterion led to the inclusion of families who were of higher socioeconomic status and therefore not as vulnerable to food insecurity. The MTE recommended adding a new socioeconomic status criterion to ensure that the project targeted only the poorest and most vulnerable households. Conversely, in two municipalities with very high food insecurity, SEGAMIL shifted to include all vulnerable families with children under 5 years of age. The package of interventions for each group was adapted to better address the specific vulnerability factors of each group. Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 28 The focus on the first 1,000 days also created the impression among communities that this was a project exclusively for women. SEGAMIL project documents indicate that this discouraged some men from participating in training and other activities provided by the project. For the value chain activities, SEGAMIL elected to target a different group of farmers who had larger plots and had more potential to scale up production for markets. This decision led to the success of the value chain activities but also eliminated opportunities for the poorest households to diversify their incomes. Changes in Food Security The PBS data showed significant improvements in the average household dietary diversity score, which increased from 6.2 to 6.8 between 2013 and 2018 (Annex 4 and Annex 5, Table A5.14). The proportion of households reporting moderate to severe hunger did not significantly change between baseline (7.2 percent) and endline (10.1 percent) (Annex 4), but the prevalence was already low at baseline. It is noteworthy that there was not an increase in hunger during this six-year period that saw prolonged and repeated droughts, crop losses, the breakdown of the health system, and significant earthquakes. Throughout the SEGAMIL project area, poverty decreased significantly between 2013 and 2018 (Figure 13). The proportion of people living on less than $1.25 per day decreased by half, from 43.8 percent in 2013 to 19.2 percent in 2018. The mean depth of poverty also decreased significantly, from 11.9 percent in 2013 to 3.5 in 2018. Multivariate analyses were conducted to identify factors associated with per capita consumption expenditures (Annex 6). Improvement in daily per capita consumption expenditures between baseline and endline can be partially explained by households’ use of agricultural practices promoted by the project. Making savings, using at least two value chain activities, and using at least two sustainable livestock practices are associated with higher daily per capita consumption expenditures (Annex 8, Tables 8.1-8.3). This corroborates qualitative findings that many of the SILC and ECADI seed fund activities are generating supplemental household income and thus increasing household consumption expenditures and food security. However, it is not possible to attribute changes in poverty to SEGAMIL directly, and caution should be taken in interpreting these results, given the presence of other value chain projects25 in five of the eight SEGAMIL municipalities. Anecdotal evidence from respondents indicates that the reduction of poverty in these communities might also be due in part to migrant remittances. Higher daily per capita consumption is also associated with lower odds of stunting and underweight. These findings support the project’s hypothesized pathways of how the uptake of sustainable agriculture practices contributes to increased consumption expenditures, which in turn translate into improved nutrition. 25 The Rural Value Chains Project operated from 2012 to 2017 in these areas. Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 29 Figure 13: Prevalence of Poverty: Percentage of People Living on Less Than $1.25 per Day, and Mean Depth of Poverty 43.8% 19.2% 0% 20% 40% 60% 80% 100% Percentage of people living on less than $1.25/day 2013 Baseline 2018 Endline *** 11.9% 3.5% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Mean depth of poverty 2013 Baseline 2018 Endline *** Significance for test of differences: ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 EQ2: FOR EACH TECHNICAL SECTOR, WHAT ARE THE STRENGTHS OF AND CHALLENGES IN THE PROJECT’S DESIGN, INCLUDING THEORIES OF CHANGE THAT INFLUENCE THE EFFECTIVENESS OF THE INTERVENTIONS? The SEGAMIL design was largely based on the Preventing Malnutrition in Children Under 2 Approach, 26 which calls for the integration of food assistance with improved food production and the promotion of optimal health and nutrition practices, specifically targeting the first 1,000 days of life. Some of the primary concerns with the initial design were raised by the MTE, including the lack of WASH strategy given the high levels of diarrheal diseases, insufficient linkages to local and governmental organizations, and the need for more context-specific livelihoods assistance. The MTE led to a major realignment of project interventions. This shift was extremely demanding of project staff and disruptive to field operations. In the evaluation team’s perspective, many of the changes were positive and ultimately contributed to the project’s successes. Following the MTE, there was a renewed focus on WASH activities. SEGAMIL integrated WASH into the project’s BCC strategy, focusing on handwashing, safe drinking water, proper management of excreta, and household waste management. SEGAMIL also started working closely with WMCs and OMAS to improve the management of water systems. The project adopted the Participatory Hygiene and Sanitation Transformation (PHAST)/Self-esteem, Associative strength (of a group), Resourcefulness, Action-planning and Responsibility (SARAR) methodology.27 The project’s BCC strategy was revamped, leading to the simplification of the messaging, a reduction in the number of practices promoted, and the adoption of adult education techniques. SEGAMIL also began 26 FANTA-2. Title II Technical Reference Materials. TRM-01: Preventing Malnutrition in Children Under 2 Approach (PM2A): A Food￾Assisted Approach. Version 1: October 2009. Washington, D.C.: Food and Nutrition Technical Assistance II Project, Academy for Educational Development, 2009. 27 See page 35 of this report for more details on this methodology. Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 30 working more closely with municipal and local organizations to improve awareness of food insecurity causes and strengthen their ability to respond. These efforts were all warranted and proved successful. The SEGAMIL staff’s dedication to getting it right was apparent in the quality of the materials and approaches developed and evidenced by the results obtained in these areas specifically. SO1: Given the susceptibility of the target geographies to climatic variation (e.g., El Niño), high dependency on seasonal migration for income generation, small landholdings, and limited crop diversification, it seems that the design could have anticipated these constraints a bit more to include greater flexibility in how the package of interventions was implemented in any given community so that participants could choose among technologies and practices promoted, or these technologies and practices could have been better adapted to the local agro-ecological and market conditions, and the agricultural technicians could have been more creative and innovative in finding local solutions and leveraging existing resources (include positive deviants). The value chain component did anticipate this through analyses to determine which value chains were best suited for particular communities, which accounts to a large extent for their success and the high probability of sustainability. Some agriculture technicians were quite proactive and convinced project management to be creative in the local context and partner with other projects in the community to build on existing activities to enhance SEGAMIL investments. Those communities had some of the best applications of SO1 activities, especially gardens, livestock, greenhouses, and income diversification. This creativity and innovation was unfortunately not widespread, and most agricultural technicians lamented the rigidity of the proscriptive activities of the project. Investing in better understanding the current farming practices and their underlying rationale would have helped a lot in determining which practices to promote as well as how to adapt them better (e.g., lower-cost rain catchment systems, appropriate-sized grain storage equipment, dry land gardening techniques). As one agriculture technician from one of the partners said, “If we had done our homework better as to what people are currently doing and why, we wouldn’t have to promote or convince people to adopt anything. It would be a natural fit with the existing practices.” This could have included a process to identify positive deviants, farmers who were already doing interesting things upon which the project could build. For example, in one community a female non-participant was actively producing vegetable and tree seedlings, which she had learned on her own. Her expertise and experience were not used by the project at all. Most certainly other local innovators exist in other communities. The impact pathway of SO1 on nutrition could have been better researched. Some CRS technical staff observed in hindsight that some nutrition-sensitive agricultural activities could have been emphasized more than others to achieve a stronger nutrition impact. They felt that small animal production could have been strengthened more so that households could both sell (for income) and consume (for nutrition) at levels sufficiently significant. Household gardens, it was observed, were perhaps not the most effective pathway for positively affecting nutritional intake by the most vulnerable. By investing more in livestock, it was felt that participant households would not have been forced into trade-offs between income generation and improved nutrition. Findings from this evaluation also raise questions as to whether a focus on increasing agricultural yields is the best approach in areas where landholdings are small and water is increasingly scarce. These questions include whether production could be increased sufficiently on these micro plots to address food insecurity and whether other non-agriculture income-generation activities could have been explored. Related to this is whether project participants who did not have the required conditions to participate in the value chain activities could have been involved in other ways that did not involve having access to land, such as packing, processing, marketing, etc. These are important questions for FFP to consider as the global population rapidly shifts from rural areas to urban areas, and the concentration of food insecurity concomitantly shifts with it. Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 31 Finally, the role of youth, which is a critical demographic, and their aspirations were not systematically addressed in the design of the project, nor did they figure predominantly in the implementation. This was due, in part, to the focus on the first 1,000 days and reflects a deficit in the broader development narrative about this age group. Given the large out-flux of youth from the region as long-distant migrants to Mexico and the United States, it would have made sense to include them more systematically in the project. Good examples are their participation in the value chain activities and some other income-generating activities that certain technicians were able to set up. It was not a broad-based, project-wide endeavor, however. Given the growing youth bulge in many USAID focus countries, this is an important lesson learned. SO2: SEGAMIL’s approach to improve the nutritional status of children under 5 years of age was based on the first 1,000 days approach and the adoption of beneficial health, nutrition, and hygiene behaviors by mothers. The project’s implied28 theory of change posits that education, behavior change messages, and skills building through peer learning groups, individual counseling, and demonstrations will lead to the adoption of heathy behaviors. To facilitate the adoption of these behaviors, the project strategy included the following:    Improving the availability of nutrient-rich foods through rations, home gardens, and livestock ownership and by increasing household income Improving access to health services Involving men in the educational activities so they can support their partners in the adoption of healthy practices The health and nutrition educational and behavior change components were appropriate in that they addressed key determinants of stunting in the targeted communities, such as infant and young child feeding habits, hygiene, and the treatment of childhood illnesses. They filled a knowledge gap as evidenced by testimony from participants and promoters who claimed to have learned a great deal, and who recognized the benefits of the practices they had been taught. Given the deficiencies in the national health system, the cessation of the PEC program, and poor access to health care, establishing growth monitoring and nutritional follow-up visits directly in the community was wise. The design, however, did not address problems of access to antenatal care or folic acid supplementation. A shortcoming in the design involves the assumptions on pathways to sustain improved feeding practices after food transfers or vouchers were withdrawn. The assumption that household agricultural production (plant and animal) could fill this gap and allow families to maintain similar levels of consumption and dietary diversity is unrealistic given the small scale of production. Furthermore, the seasonality of agricultural production also affects home gardens and households’ ability to support animals (availability of fodder), and thus does not adequately resolve food shortages in the lean season. As noted under SO1, greater diversification of income is needed to allow families to buy foods and supplement their maize diet. With regards to the WASH activities, SEGAMIL could have focused more on sanitation activities, especially improving the construction and use of household latrines. Proper sanitation and handwashing are key to reducing the incidence of diarrhea, which in turn, reduces malnutrition in children under 2 years of age. Repeated episodes of diarrhea can alter the gut, leading to the poor absorption of essential nutrients, which puts children at risk of stunting and even death. SO3: A challenge to the sustainable effectiveness of municipal offices, including OMAS, is the possibility of staff turnover due to mayoral changes. Many positions are mayoral appointees, and staff can be 28 SEGAMIL did not develop an explicit theory of change, but it can be constructed from the results framework and suite of interventions. Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 32 replaced when a new mayor comes into office. A clear strategy for the transfer of knowledge during transitions in personnel or administration would have been beneficial. EQ3: IN EACH TECHNICAL SECTOR, WHAT ARE THE STRENGTHS OF AND CHALLENGES OF THE EFFECTIVENESS OF THE INTERVENTIONS’ IMPLEMENTATION? Implementation Approaches and Methodologies That Contributed to Achieving Project Outputs Women’s Empowerment and Self-esteem Trainings The trainings that women received on self-esteem and leadership have by their own account made a big impact on their lives and were greatly valued by many participants to whom we spoke. Women report the trainings as being an awakening that enabled them to recognize that they have a voice and a right to advocate for themselves and their families. The ECADIs provided a space for them to learn and practice technical and managerial skills and demonstrate their ability to lead an organization. Some women eventually managed to integrate COCODEs, which are traditionally male organizations, although the process was not always welcome or easy.       “Now we feel that we are recognized in the community as leaders and our voice matters.” (ECADI board member) “SEGAMIL helped so the women could express themselves and seek help, and not be afraid to ask for help.” (Deputy Mayor) Some women said that they now feel comfortable advocating resources for their respective municipalities. One promoter described to the team that she prepared and presented proposals to municipal offices seeking support for their activities. Connections with women in other communities created a sense of social solidarity and connectivity and reinforced promoted themes. The accreditation program for promoters further validated their leadership. “For many female promoters, the accreditation was the first diploma they ever received.” (CRS staff) Cascade Trainings Cascade trainings were conducted by CRS and involved the entire coordination and technical team across all partners. It was important that the same information was communicated at each level, from the technicians to the promotors and to the households, but the content was adapted to each level. Promoters were able to convey the information in their own language using terminology that their neighbors would understand. This training approach was generally perceived as effective by partner technical staff, promoters, and staff from municipal offices. Partner staff noted that the technical content was strong and the trainings were well prepared. The promoters exhibited good knowledge of subject matter and noted that they had learned a lot, which they were applying in their own lives. It was evident to the evaluation team that the promoters in the communities visited were effective at delivering the correct messaging to the participants. “I learned a lot from the trainings. They gave me the tools so I could do the work.” (health promoter) “At first I thought it was hard talk with the technicians and learn the topics, but they had patience with me and now I have learned. It isn’t difficult.” (health promoter) The curriculum was revisited after the MTE, and the number of topics was reduced, which many of the technical staff and promoters appreciated. Government institutions also expressed interest in replicating SEGAMIL trainings. “I was very impressed with the training approach and the ECADI model. I told my agricultural extension workers to emulate it.” (Ministry of Agriculture staff) Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 33 The MOH in San Marcos noted that it had received paper and electronic copies of all the training materials and was planning to implement that same month one of the trainings related to engaging mothers-in-law in health and nutrition decisions. Some recommendations for improving the training process were provided by partner technical staff. The recommendations included increasing the frequency of the trainings and providing some periodic refresher trainings. Others pointed out that the training duration was too short and felt rushed. One respondent said that an increased CRS presence at the facilitator trainings would help improve the training and facilitation skills of those who had to replicate the training by providing more opportunity to assess and provide feedback. ECADIs The ECADIs showed potential for facilitating the integration of health, agriculture, and income diversification interventions, and they provided a central location where activities could be coordinated. The extent of integration was dependent on the individual promoters and the extent to which they chose to collaborate. One agriculture technician noted that, “Not all agricultural activities were well integrated and supported directly by the health and nutrition activities.” Perhaps more could have been done to foster that collaboration and the integration of messaging, such as through joint home visits. The integration of SILC into the ECADIs was very successful and a major impetus for the continued participation of ECADI members. The ECADIs opened a space for women in the community where they could meet, share experiences, and take on leadership roles. Participants felt that the ECADI model improved social cohesion, and an ECADI board member noted, “Now we know each other better, there is more trust among us.” One women recalled that the ECADI members came to visit her home to provide support after she lost her baby. In addition, the ECADI structure facilitated the women’s groups to become formal entities. This in turn facilitated their ability to engage with municipal offices. A Municipal Department for Women (DMM) representative noted that it was easier to work with ECADIs than other women’s groups; the fact that they were formalized facilitated supporting them with municipal resources. DMM representatives in a few municipalities stated that they were grateful for the work SEGAMIL had done linking these groups to their office. The “model home” approach, whereby the household housing the ECADI would demonstrate the ideal practices, was not necessarily effective. Several of the items provided to the ECADIs for them to demo (e.g., grain silos, rainwater storage tanks, animal pens, agriculture tools) were not affordable to the target population. It was not effective in creating demand because the items were beyond their reach economically. Some promoters and participants shared that some community members resented the fact that the promoters received materials for free. On the other hand, these donations were a good incentive to motivate the promoters who are otherwise not compensated for their work. Market-oriented Crop Production The production of market-oriented crops for the SEGAMIL value chain component worked well and should have had a larger budget. All the value chain participants interviewed gave this component high marks and wished the project could continue. However, they felt confident that they could continue on their own and understood both production and marketing well. This is a sure sign of sustainability. Given the solid value chain analysis done by SEGAMIL prior to selecting these value chains, there is great potential for these participants to expand. All value chain participants interviewed were actively reinvesting their earnings back into their businesses, knew how to track their expenses and income, and exhibited comprehensive knowledge about production and management practices. The assumption that the most vulnerable households without productive resources could not participate should be questioned because, as noted above, perhaps they could have participated in other aspects of the value Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 34 chain. If land was the primary constraint, there could have been creative ways to access the land, such as leasing or sharecropping the land with the owners for a percentage of the profits. This selection criteria unfortunately restricted the number of participants involved in this activity. The desire for income￾generation opportunities was clearly expressed in the interviews with many of the project participants. Food Transfers—Rations and Vouchers The participants reported that the food transfers had greatly benefitted their families and helped provide more and different foods for their children. Receiving food transfers freed up income so the participants could buy other foods (e.g., fruit) or other household goods.  “The food provisions are what helped our family the most when I was sick.” (project participant) Both food transfer types helped households meet the nutritional needs of the target population because they directly addressed the issue of poor food availability (e.g., food deficits and non-diversified diets). Project staff reported that the voucher system was better because it was more efficient than the food ration in terms of cost to the project (transportation, storage, pest control, accounting, management, etc.) and to the participants (they no longer had to pay for local transportation from the regional warehouse). It gave participants choice, reduced stigma, reduced transport and storage costs, which reduced the incidence of food spoilage, and offered more foods that were culturally acceptable and known and that could continue to be purchased in the marketplace. The voucher system was also nutritionally superior to the donated commodities because it provided eggs. In addition, the voucher system stimulated the local economy through national procurement and employment generation. For example, the increase in institutional demand for food commodities from SEGAMIL stimulated the supply side of egg production through a project supported by the Swiss nongovernmental organization Helvetas that was implementing a four-year program to improve the productive capacities in several animal value chains. The switch from food distribution to vouchers improved the agility and efficiency of households receiving food products, involved the local private sector, and motivated suppliers by the social mission of the project. It also built social relations between project participants and local merchants, and many participants continued to make their food purchases with voucher providers after the project ended. Behavior Change Strategy As discussed in EQ1, many of the practices promoted by the project showed significant improvements between 2013 and 2018. The PBS measured only a few of the behaviors promoted by SEGAMIL, and these data showed improvements in the following behaviors specifically:        Use of household gardens and the production of native herbs Use of improved livestock facilities Use of improved storage practices Consumption of a more diverse diet by women and children Treatment of water prior to drinking Trainings in the PHAST/SARAR methodology improved the skills of the SEGAMIL staff and promoters in using participatory and adult learning activities to successfully promote behavior change. According to SEGAMIL staff and promoters, this methodology and the accompanying materials greatly helped the project’s BCC efforts. The training also focused on improved interpersonal communication. “We emphasized listening to participants, not just talking at them.” (CRS staff member) “The trainings focused on how to work with families in addition to the technical content.” (SEGAMIL partner staff member) Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 35 The combination of instruction at the ECADIs and home visits were seen as beneficial by promoters and participants alike. The promoters noted that home visits to participant households allowed them to assess practices, identify barriers impeding behavior change on a personal or household level, and support the household in identifying solutions. Home visits allowed promoters through observation and counseling to identify causes of poor weight gain for a given child and rectify the issues (e.g., poor feeding practices, poor hygiene). Some promoters described how they encouraged the participants to make vows (promesas) to change their behaviors, which created social obligation to comply. 29 Several participants also described the benefits of these home visits:   “They (health promoter and mother monitor) come to see the hygiene, the latrines and how we feed our children. They showed me how to prepare Incaparina. When my baby lost weight they made sure I added three spoonfuls of Incaparina and a little oil.” (project participant) “When I had my child I did not know how to care for him. The promoter would weigh him and helped me when he lost weight. They kept him healthy.” (project participant) The purpose of PHAST is to get communities to take leadership in planning their own water and sanitation projects. PHAST is an offshoot of the SARAR methodology, pioneered by Lyra Srinivasan. These are all attributes to be developed in the planning group and the wider community to help people solve their own problems. SARAR is based on adult learning principles, in particular that adults learn and change best in groups and that adults learn best when they have a self-identified problem to solve. When adults have defined the problem and solution, they will own it, as long as no one takes it away from them. PHAST uses the philosophy and many of the SARAR activities to create a tool specifically for the water and sanitation sector. It is empowering but requires training and care in its implementation. PHAST is one of several participatory tools used in development called Participatory Learning and Action. PHAST is a series of learning and planning activities that follow a specific logical sequence, leading to making final concrete decisions. These final decisions are owned by the community planning group, and they are responsible for ensuring that these decisions are implemented by the community at large. The idea behind using PHAST is to assist a selected community planning group to decide for itself what changes it would like to make, for example, in community behavior, where it would like water points sited, and how it will go about getting sanitation coverage. The planning group should include project staff and relevant government representatives. Food Demonstrations Food demonstrations introduced recipes and preparation techniques, such as adding vegetables and fruit to meals and adding a spoonful of oil to increase calories. Local and seasonal ingredients were used. Participants across the communities visited by the evaluation team said that these demonstrations were critical in getting them to incorporate new foods in their diets. The demonstrations were very dynamic and motivated women’s participation, according to technical staff and participants alike. Organizational Strengthening The approach used to strengthen municipal and community organizations was appropriate, combining technical and organizational trainings. The main challenge was resource mobilization because these local organizations, especially the technical units in the municipal governments, were dependent on the municipal annual budget. The institutions were taught advocacy skills, which many applied successfully to get a line item in the municipal annual operational plan and accompanying budget, but support with mobilizing external resources (direct to international donors) might have been useful. Staff from the targeted organizations reported that organizational strengthening was effective, especially with helping organizations understand their role, their obligations, how they are supposed to work with 29 This is a common and culturally accepted practice in Guatemala. Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 36 different entities or institutions, and how to better serve their communities. This was reported to the evaluation team by different organizations at both the community and municipal levels. Members and employees of these groups generally felt better able to do their jobs. Training on the development council system and the related laws in particular was greatly valued. One mayor, who leads the COMUDE in his municipality, felt that SEGAMIL strengthened the COMUDE by training the COCODEs on their roles and responsibilities. The COCODEs could now better perform their responsibilities to produce community development proposals, which made it easier for the COMUDE to review them and make better funding decisions. The CRS staff dedicated to this component were highly qualified and extremely motivated. Much of the advancement of this revised SO post-MTE is due to the hard work and inspiration of this team. SEGAMIL helped create linkages across entities at the municipal level, which led to better, regular coordination. For example, the DMM in one municipality stated that it now has better coordination with the Presidential Secretariat for Women, the Municipal Planning Office, and the Municipal Directorate for Integrated Financial Administration. At the community level, the DMM found it easier to work with ECADI groups because they were formally recognized in the communities, whereas the women’s groups they had worked with before were not. One DMM representative said that, “The ECADI groups are now better organized and trained than other women’s groups and are able to achieve more.” Regarding WASH activities, OMAS staff appreciated the technical trainings and support received from SEGAMIL but felt that they did not have the resources to undertake future work with their communities. One respondent stated that he would like more technical staff and water system supplies to be able to support more communities. He also stated that he would like more support in speaking with communities about the sustainability of their water supply system. This support would help achieve better communication about the issues surrounding the use, operations, maintenance, and repairs of the communities’ water supply systems, leading to better support (e.g., labor, payment of appropriate fees) from the community. Also, the possibility of staffing changes as a result of future mayoral elections was often reported by OMAS staff interviewed. Implementation and Contextual Factors Associated with the Project’s Effectiveness in Producing Outputs of Higher or Lower Quality Program Staff Technicians were, for the most part, able to gain the trust of the community and organizations they supported. All respondents spoke highly of SEGAMIL staff. The technicians reported that creating a good rapport with the participants was the key factor to determine the success of SEGAMIL’s activities. It was noted that the technical and interpersonal skills of technical staff varied. Some were more engaged with communities, some were less effective. In the early phases of the project, there were some tensions between staff and community members, leading to some staff having to be replaced. The project, through its trainings, emphasized interpersonal relationships, especially in the second half of implementation. This was later included in the trainings provided to municipal civil servants at the municipal level. Many of the municipal staff interviewed noted the eye-opening nature of these trainings and that the trainings changed how they engaged with the public. It was also noted by some respondents that there were too few technical staff and that they were spread too thin, resulting in an inability to spend enough time in each community supervising and supporting the promoters and community organizations. For example, there were only two WASH facilitators in one department. They noted that reviewing and using the water system manuals with WMCs in every community was challenging because of the volume of information to review. This meant that important guidance considerations could not be communicated well to the WMCs. In addition, the WASH team felt pressed to provide sufficient trainings to the WMC—whose staff turned over Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 37 frequently—and were therefore not able to focus as much on issues related to water supply system operations and maintenance. Physical access to the communities was also a challenge, especially the remote ones during the rainy season. This resulted in less-than-favorable exposure time in the communities in some cases and long hours in the field. In some areas, drug trafficking activities and civil unrest also impaired access. SEGAMIL Design It was also noted by different sources that SEGAMIL tried to implement too many interventions. The original project design seemed to have been overly complex and complicated for a variety of reasons, possibly including the desire by CRS to win the request for application. The MTE reduced the complexity of the design and focused the interventions on key outcomes (with some new outcomes, such as organizational strengthening and sustainability), but the timeline for implementation was short. The post-MTE design seems to have been better appreciated by the implementing staff of both partner organizations and allowed them to focus their already constrained time on project activities. It was difficult to determine whether the integration of the three SOs was achieved because it was not clear how to measure the synergistic effects. It seems that customization or adaptation to the local agro-ecological and market conditions was mixed. Some SEGAMIL field staff said that the implementation requirements were rigid, and management staff said that the requirements were flexible and adaptive. Partnership The interviews revealed a good and collaborative working relationship between CRS and its partners. There were regular meetings to review progress and opportunities to learn across municipalities. Some partner technical staff mentioned that trainings were good, but follow-up or “acompañamiento” could have been stronger. Implementing partner staff would have liked more mentoring and supervision from CRS technical staff to ensure that they were doing things correctly and help improve their skills (e.g., facilitation when they had to replicate trainings with promoters or with other organizations). Program Monitoring SEGAMIL staff at all levels felt that the SEGAMIL monitoring and evaluation (M&E) system was very good because it was rigorous. The CRS software worked well, and both partners were well-trained in its operation. There was constant control of activities as the work was implemented and was closely followed by both CRS and the implementing partners. For example, CRS would visit an ECADI board of directors unannounced to check on the topics they were supposed to learn. CRS would then have monthly meetings during which feedback would be provided to the implementing partner managers. The managers then would discuss the feedback in internal meetings and identify ways forward if there were issues to address. Some staff, however, reported that the project reporting requirements were excessively time consuming and sensitive to input errors. Partner M&E staff were constantly evaluating the work of their organization. For example, M&E staff would be present at trainings and would evaluate the actual training, materials used, snacks, per diem, and transportation. The M&E staff would provide feedback after the trainings. It was also reported that partner Field Supervisors also participated in the M&E process. Technical Coordinators were required to develop their community engagement plans, including specific activities and dates for when the activities would be implemented. The Field Supervisors would then show up unannounced to check on their work. Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 38 Migration Many families in the SEGAMIL target communities are not able to subsist on their own agricultural production and need to find wage labor to support their families. This situation was aggravated by the prolonged and repeated droughts experienced in the Western Highlands during the six years the project operated, which led to widespread loss of crops. There are few wage labor opportunities in these rural areas, so permanent or seasonal migration is often the only solution. Men often go to the coast to work on plantations or to the cities in search of temporary jobs, or migrate to the United States. The absence of men from the community limited the project’s ability to engage them as initially intended, and men’s participation in trainings and educational activities was consistently low. Because men are traditionally the primary farmers in the household, the lack of participation likely restrained the uptake of agricultural practices. It also limited the project’s ability to more fully engage men in supporting the behavior change activities targeted to women. Many promoters also stated that increasingly, entire families are temporarily migrating together with their children. Promoters noted that this affected the continuity of growth monitoring and of the BCC activities with the mothers, limiting the impact of the project for those individuals. Water Availability The prolonged drought in the first half of the project affected a broader geographic area and severely impacted agricultural production, limiting households’ ability to maintain home gardens or diversify crops. The seasonal availability of rainfall restricted year-round production for many Highland communities,30and increasing climate variability makes it hard to predict when is the best time to plant. Although the proportion of households with access to piped water has increased, the participants in some communities noted that it was illegal to use tap water for irrigation. Drinking water was mostly available during the dry season, although for less time during the day. Many systems had a daily operating schedule. Gender Issues A few female participants told the evaluation team directly that they had difficulties obtaining permission from their husbands to attend meetings, and the promoters and SEGAMIL technical staff indicated that this was a problem for many women. This was especially true in Mam areas. Some women had difficulty attending meetings or receiving home visits when the technical staff or promoters were men. However, other women reported that they had the full support of their husbands to participate and that they shared with them the things they were learning from the ECADI trainings. Many men saw the direct benefits of participation (i.e., food transfers) as well as the application of the trainings to agricultural activities. Many men interviewed in the FGDs could clearly articulate the multiple benefits derived from the project (healthier children due to improved diets and cleaner home environments, better agricultural practices, etc.), and they reported helping out their wives more with household chores and child care so their wives could participate in SEGAMIL activities. In several communities, evaluation team members observed men actively caring for children while their wives participated in the evaluation activities. Almost all of the female interviewees expressed their appreciation for the leadership training that they had received, which was a new experience for them, and said that now “they can speak in public without being embarrassed and they know now how to access municipal services for their communities.” Participation in project activities did increase the burden on women’s time. Not only did they need to find time to attend project meetings, but several of the promoted behaviors required a substantial time investment from women. For example, caring for animals, implementing hygiene practices in the home, preparing specialized foods for young children, and taking more time to feed young children all added to 30 Except in the communities in the coastal areas, where sometimes up to three maize crops were possible Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 39 women’s daily obligations. Some women reported getting up earlier to be able to do all this. SEGAMIL attempted to engage men and older children to provide more support with household chores, and the evaluation team heard from a few participants that this had helped. Given the low male participation in project activities, however, it is unlikely that this had a large impact. BCC Revisions Post-MTE The revision of the BCC strategy was warranted and successful. The reduction in the number of practices promoted and the simplification of materials (from a manual to a two-page guide for promoters) were cited at all levels as a major factor in success in second half of project. SEGAMIL worked closely with the MOH and promoters in developing the messaging and BCC materials. The visual wheel aide (rueda) was a useful tool to remind promoters and participants of practices that needed to be addressed. Having men represented in the images helped convince husbands or partners that they had a role to play in the behaviors and practices (such as feeding children or cleaning the house). It was helpful for partner technical staff and the promoters to participate in the redesign because they understood the culture. The project also took the time to conduct effective barrier analyses to understand the particular challenges and constraints to behavior change. The incorporation of behavior change educators increased the frequency of follow-up home visits. The participants expressed appreciation for the opportunity for more personalized learning. Health Services Linkages and referrals to health systems helped get treatment for malnourished children. Participants sometimes feel that seeking care at health facilities wastes their time. There are long wait periods and often stockouts at the public clinics, so they either have to buy medicines or revert to herbal remedies. Participants reported that stockouts of medicines and vaccines impeded progress in child health. There were also reports of language differences between health center staff and participants. EQ4: WHICH PROJECT OUTCOMES ARE LIKELY TO BE SUSTAINED? The SILC groups are very likely to continue to operate. They are well established, and there is a high level of demand for this service among participants because of the access to credit and the safety net that the SILC groups provide for families. In addition, these groups are self-sustaining and can easily bring in new members. Many participants and ECADI board members indicated that the SILC are an important reason why the ECADIs are still functioning despite the project closeout. SEGAMIL trained a cadre of SILC auxiliaries who can provide technical assistance to existing or new groups related to SILC management and financial literacy. A key factor in the sustainability of the ECADIs is having received seed funds and using these funds to allow member households to invest in productive activities. These funds were an important factor motivating participants to continue being involved in ECADI activities, including growth monitoring and educational activities. SEGAMIL used clear criteria to determine which ECADIs could receive these funds, such as number of members, group dynamics, and leadership ability. Based on the evaluation findings, it is likely that households linked to ECADI revolving funds will continue to invest in animal rearing as a source of income. These funds were generating small and much appreciated profits for the participants. The ECADIs reported that loans were being paid back in a timely fashion, and members reported that being well-known to the group and social obligations to the community would make it troublesome to default on payments. Some ECADIs had begun charging small interests, replicating the SILC model, to allow the fund to grow, which could lead to larger loans or to more members accessing the funds in a given cycle. To successfully generate income from the animal-rearing activities, the members will require ongoing vaccination services, which under the SEGAMIL project were being conducted by the agriculture promoters. Little additional external support Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 40 is needed unless the ECADIs decide to invest in other types of income-generating activities in the future. In such cases, the connections that have been established between the ECADIs and municipal offices, such as the DMM and the Municipal Agricultural Technical Office (UTAM), can be beneficial. The sustainability of the agriculture promoters post-project will depend on a strong demand and willingness to pay for their services. Some demand was generated for the vaccination services, and various participants indicated that they were willing to pay for this service because it protected their investment in animal-rearing activities. The agriculture promoters are able to recoup their costs through the fees they charge and make a nominal profit. It remains to be seen whether the demand is large enough to make it worthwhile for the agriculture promoters. This is not a full-time activity for promoters, so they could continue providing this service even if engaged in other occupations. It is not clear that the project has created demand for other services offered by the agriculture promoters, especially from non-project participants, because there were limited operational mechanisms to spread the promoted practices beyond the targeted participants. Most promoters we spoke to were eager to continue working in the community and were looking for ways to continue serving the ECADI members. For example, some were reported to be actively looking for seeds or preparing solicitations to request building materials for animal pens from the UTAM. Because the project had just ended, it is unclear how successful they will be in obtaining that support. Various health promoters and mother monitors indicated that they will continue with their growth monitoring activities and household visits to encourage women to continue practicing the behaviors SEGAMIL had promoted. They are willing do to so on a voluntary basis and will likely be able to do this without supervision with the knowledge already acquired. To perform these tasks, the health promoters require some resources, such as measuring sticks, scales, and the registers and notebooks used to track children and households. The project has left these behind, but the promoters do not have a means of replacing them if they get lost, broken, or completely filled in. A few promoters reported that they had paid out of pocket to make photocopies of the growth monitoring notebooks so they could continue their work. The health promoters who were asked about their future role did not envision expanding their services to newly pregnant women in the community. When asked why they did not plan to reach out to these women, the promoters stated that it would be impossible to recruit new members if food vouchers were not being offered. It seems that this was a lost opportunity to expand coverage and reach more people with the BCC messages and growth monitoring activities. It also affects the sustainability of these activities because the current participants’ children will soon outgrow the need for these services. The agriculture promoters have been accredited by the Instituto Técnico de Capacitación y Productividad, and the health promoters have been accredited by the MOH. This was an important component of SEGAMIL’s sustainability strategy. Staff from the MOH, the Ministry of Agriculture (MAGA), the Food Security and Nutrition Department, and various municipal offices recognized the value of having a well-trained cadre of promoters in the communities. They all expressed a desire to support the promoters. However, all noted that they do not have the capacity to absorb or support these promoters due to funding restrictions. They lack the funds and human resources to supervise the promoters, let alone provide them with remuneration or inputs for community activities. Without a clear source of technical assistance from government institutions, it is unlikely that the promoters will be able to continue offering services in the longer term. One agriculture promoter indicated that MAGA had reached out to invite her to a training related to preparing parasite treatment for animals using locally resourced plants. She had to pay for her own transport to attend. She did plan on disseminating the information in the community after the training was completed. Although welcome, these ad hoc activities are insufficient to promote sustainability. Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 41 The value chain activities observed appear to have been well-designed and successful, and all members interviewed expressed strong conviction that they would continue activities post-project. The market linkages for all value chains were well-researched and strengthened, production technologies were affordable and accessible, and transportation from farm to market was feasible. The participants were generating revenues, which motivated them to continue working and expanding their production. In the case of honey, producers were organized into a formal cooperative and linked to a honey exporter. The group had clear business plan to grow the cooperative. In the case of coffee, linkages to Anacafé and other coffee institutions have been established and show promise for continued technical support for these producers’ associations. The sustainability of the nutrition outcomes is uncertain. The participants have acquired a wealth of knowledge regarding proper nutrition and feeding practices, and as they repeatedly told the evaluation team, “no one can take that away.” The women interviewed demonstrated a clear understanding of the importance of diverse diets for their children and because of this were committed to continuing to provide a healthy diet. The key question is whether they will have continued access to the quantity and diversity of foods needed to sustain the nutritional gains. Maintaining these diets will require either additional income to buy the foods or having the households produce the foods themselves. There are indications that SEGAMIL has made progress on both these fronts. Although these efforts will help diversify household diets to an extent, it is unlikely that households will be able to maintain the same levels as when they were receiving food transfers. For those participants who are motivated to maintain home gardens, these will help provide needed herbs and vegetables, at least for part of the year when rain is available. Given the plot sizes and seasonality, it is unlikely that the home gardens will produce enough quantity to maintain current levels. Animal production will generate income that can then be spent on food, but women specifically noted that they rarely consume the meat of these animals, choosing instead to monetize them. Thus, although diets will likely be better than they were at baseline, it remains to be seen whether they will be good enough to avert malnutrition problems in the future. There was no indication that the nutrition gains will affect non-participants because the ones we spoke to had no familiarity with the project activities. Regarding the WASH activities, a major accomplishment was strengthening the WMCs and creating additional OMAS. Staff from both the OMAS and WMCs felt that the various trainings and operations and maintenance guidance and pipeline repair materials provided to them by SEGAMIL were useful to their work. SEGAMIL staff were confident that the OMAS will continue because funding for OMAS is now part of the municipal budget. The water system supplies (e.g., pipeline materials and chlorine tablets) used by the WMCs are funded by household fees, and the collection of tariffs has been standardized and contributes to the sustainability of the water systems. However, it was reported that the amount of fees collected may not be enough to cover the future costs of purchasing water system supplies (e.g., pipeline materials, chlorine tablets). Both institutions have heavily relied on subsidized pipeline supplies and have expressed their sadness at SEGAMIL ending. It does not appear that there are strong enough linkages between the WMCs, OMAS, and COCODEs to ensure the availability of adequate finances for water supply system operations, maintenance, and repairs. Regarding the communities’ willingness to pay for water services, all WMCs and OMAS offices visited reported that communities are willing to pay for potable water services. However, it was not possible to review and comment on WMC or OMAS financial resources. The evaluation team asked to review the financial accounting books for each WMC visited, but the materials were not brought to the meeting. Limited interview time did not allow the member to retrieve the materials, return to the meeting site, and allow for review. Overall, each WMC visited did not bring any materials (e.g., financial accounting books, operations and maintenance manuals) to the meeting. They were not prepared for these discussions. As reported from the OMAS visits, all are allocated money from their municipality’s annual budget. It remains to be seen whether the amount is sufficient to support the costs of office and water system repair supplies after SEGAMIL ends. Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 42 The project made a concerted effort to build up municipal government offices to prepare them to take over the leadership of many of the food security activities implemented by SEGAMIL. The project created coordination bodies through the Municipal Food and Nutrition Security Commission that would be able to advocate and mobilize resources. In three of the municipalities, SEGAMIL left official food security plans and policies that outline concrete actions and name the entities responsible. Building the technical and managerial capacity of municipal offices was a key strategy of the SEGAMIL sustainability plan. The idea was to strengthen coordination between community organizations and government institutions to better respond to the food security needs of their constituents. SEGAMIL leaves behind the following:   Community organizations, including ECADIs, COCODEs, and WMCs, that have been trained to recognize the nutrition, health, food security, and drinking water needs of their communities and to understand the avenues through which they can advocate support. A network of municipal and departmental offices with the technical and managerial knowledge to support communities with their food security needs. These include the MOH, MAGA, the Food Security and Nutrition Department, DMM, UTAM, Municipal Planning Department, and OMAS (the latter of which were established by SEGAMIL in some municipalities and are now receiving direct funds from the municipality). Although the capacity is well established, and the will to support food security activities exists, the government offices have limited resources to allocate. A few respondents in the community noted that they had reached out to these offices and submitted proposals for support but had not received a response. Nevertheless, there have been a few instances in which resources have been successfully leveraged to sustain project outcomes. The evaluation team heard of cases in which ECADIs were able to obtain seeds from the UTAM, and a Municipal Food and Nutrition Security Commission mobilized food rations for distribution. Although the government funding crisis is a major barrier, SEGAMIL has built the advocacy skills and networks that have potential to generate support. An additional threat to sustainability involves turnover of staff or volunteers who work in these various entities. The project has in many cases left documentation and training materials behind, but it is not clear that the project helped these organizations plan for future transitions and how the knowledge and capacity can be transferred to new members. This is particularly critical for COCODEs, in which staff turnover happens every one to two years. It was also reported that following elections, new personnel arrive with different priorities and are not always willing to learn what was being done before. SEGAMIL collaborated with other Feed the Future projects operating in the region, including Buena Milpa and MásFrijol, to improve the services and trainings offered to SEGAMIL participants. These collaborations led to the use of better seed varieties by some SEGAMIL participants and also helped create local seed storage centers to increase access to seeds. The long-term effects of these collaborations are unknown. Like SEGAMIL, these projects are ending and are not able to continue supporting the SEGAMIL participants. EQ5: WHAT ARE THE KEY LESSONS AND BEST PRACTICES THAT SHOULD INFORM FUTURE PROJECTS IN THE COUNTRY? Best Practices Hands-on experiential learning: Overall, the ECADI system was a good model for community engagement and the transfer of knowledge, technologies, and practices around household food security and nutrition using participatory learning techniques. As an expansion on the well-proven Food and Agriculture Organization farmer field school approach, the ECADIs offered both a physical and social gathering space for project participants to learn about food and nutrition. Having the right support from Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 43 the project technician seems to be a key factor in why some ECADIs were more successful (active participation by members) than others in spite of food transfers ending. Work with local markets: The voucher system was well-designed and executed. It served the target population well, created new markets for vendors, and stimulated the local egg production. Participants, partner organizations, and suppliers all gave it high marks. It deserves closer attention and would serve well as a case study for future projects. This is the future of food assistance. Personalized BCC: For the health and nutrition component, the household visits and personalized social and behavior change communication worked well with positive, and most likely sustainable, impact. Perhaps if a similar approach had been applied to the SO1 component, there may have been stronger reinforcement of the promoted sustainable agricultural practices. Everyone counts: The M&E system captured all the important information about the project and provided timely and systematic feedback to partners, which was analyzed and discussed in a participatory fashion. Community promoters were well integrated into the M&E system, which included well-designed tools adapted for their level of literacy and language preference. Cascade training: The cascade training system appears to have worked well in terms of the transfer of knowledge. All participants demonstrated high levels of knowledge related to the promoted behaviors, practices, and technologies. The community and municipal organizations said that they benefited from the trainings received and demonstrated good understanding of the training topics (e.g., drivers food security, roles of various government institutions). MAGA staff gave the process high marks and adopted and adapted the training materials and approach. Promoters were given the freedom to communicate in their own style, language, and creativity to their neighbors. Savings and loans: The savings and loan component was well designed and implemented. It was warmly received by project participants and replicated with additional non-SEGAMIL groups due to its popularity. This approach fills an important gap in access to loans and credit for rural communities, and because of this, it is likely to be sustainable. The financial literacy trainings are key for populations with little formal education. Women’s empowerment: Overall, the female promoters and elected board members of the ECADIs and SILC groups noted their evolved, positive self-perception, increased abilities, and greater self-confidence. Given the political economy, history, and culture of the Western Highlands, this experience and process deserves closer attention and documentation for possible replication and scale up. Seed funds: Providing seed funds to help poor households make initial investments in productive activities was successful and after a few months is already generating returns. This did not require large amounts of money, and members took turns borrowing against the fund. Small interests on the loans will allow the fund to grow and reach more people. This income generation has also proven to be a strong motivator for continued participation in the ECADIs. Lessons Learned Keep it simple: There was a strong desire at the onset to integrate many interventions across various sectors to address food insecurity from many angles. The MTE and final evaluation found that this complex design hindered quality by spreading project staff too thin. It required staff to spend a lot of time in trainings instead of doing follow-up at the community level and also created a large burden on participants, who were expected to carry through with all activities. For agriculture practices, focusing on two or three interventions that are known to have a more direct, positive impact on the nutrition of women and children, and making sure that they worked well and reached a scale of production to reach a threshold impact would have been more effective. Such interventions could have included household gardens (conducting home visits and ensuring personalized follow-up); small livestock production Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 44 (reducing cost barriers, increasing production levels, and making it affordable); and adaptive post-harvest food storage, handling, and hygiene (creating alternatives to the metal silos and low-cost appropriate technologies for vegetable storage). Targeted interventions should be carefully selected and customized to the circumstances of the particular communities. The project should weigh the potential, direct return on investment and avoid too many activities by making the menu of options too complex. Never underestimate the value of local knowledge and experience: The farmer-to-farmer methodology worked well and expanded participants’ knowledge through direct contact with farmers from other communities who were successful. It also expanded people’s social networks and stimulated their creativity. More could be done to leverage existing community knowledge and expertise (i.e., positive deviants) and enlist experienced community members to jump start the agriculture component and build on this foundation. It takes a village: The promoted agriculture practices were not disseminated beyond ECADI members, which limited the potential demand for services of the agriculture promoters. A broader reach might have facilitated uptake and sustainability. Encouraging promoters to offer services beyond existing ECADI members, and offering SILC participation as an incentive, may have led to more sustainable service provision in the communities. Everything has a price: How can households with limited income adopt practices and technologies promoted at the ECADI that may be too expensive given their resource limitations? A seed capital fund introduced early on might be one way to address this after the group has solidified and capacity has been built. Getting off the farm: Given the limited access to land and its low productivity, non-farm income-generation activities could have been explored more using the approach and methodology implemented for the value chain component. These activities should be coupled with BCC on household budgeting and resource allocation for prioritizing the health and nutrition of women and children.  For example, sewing is a livelihood for some, but that value chain remains unexplored, and some of the individuals participating in production (sewing at home) seem to be getting low returns. Nonetheless, the earnings were highly valued. The weak link in the value chain: The majority of target households are net purchasers of food rather than net producers. The assumption that many SEGAMIL participants “lacked the conditions” to participate in value chains may not be valid because there are many ways people can participate in value chains that do not include owning land for production purposes. The value chain component could have been expanded and adapted to involve more SEGAMIL participants because there was a strong demand for and interest in income-generating activities.   Their involvement could have been in processing or in the construction of structures or some other creative mechanism. Markets can be the driver out of poverty, especially if coupled with appropriate technical, vocational, and educational training that target youth. The political economy of poverty: Economics of poverty can be brutal, and households are forced to make difficult—and life-limiting—choices about how to allocate scarce resources. Therefore, it is important to evaluate carefully beforehand which project activities will be net contributors to improving the household economy immediately and in measurable, direct ways that participants can observe and will value. Longer-term investments (e.g., NRM), although important, may not represent the interests of those whose main concern is meeting their immediate needs (e.g., food, water, shelter, clothing, medicine). Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 45 Learning to learn: Various lessons can be learned from the BCC component:      Simplification of BCC materials was critical, such as developing a two-page guide for the promoters that did not require high levels of literacy. Involving the MOH, field staff, and promoters in the redesign of the BCC materials was helpful. Using the participatory education approach with adult learning principles rather than a top-down traditional education model worked well. Didactic materials: After years of use, the educational materials and growth monitoring ledgers used by promoters need replacement. Linkages with the MOH will be critical for this aspect of sustainability. Close follow-up of malnutrition cases in the community, focused on household adoption of optimal behaviors, can be a successful case management solution. Building the capacity of health promoters, mother monitors, and community educators was a key part of the success, as well as reinforcing messaging with the agriculture promoters. Women and WMCs: Including women in the WMCs can help generate acceptance and community support for water chlorination activities. Women are the main collectors and users of water in the household, and it is important to convince them of the safety of water treatment processes. Among traditional indigenous communities, it is easier for women to talk to other women than it is for men. CRS technical staff described an example in one Mam community in which such an approach was successful, noting that, “Culturally, it is difficult for men to convince women to change their behavior.” Water system operating regulations and fees: There should be more focus on water system regulation, in particular household fee contribution. Water user regulations have been developed that describe how a household should use its community’s water supply system, the need to pay a water usage fee, and the need to charge a fine if the fee is not paid on time. However, changing the fee structure, in particular increasing the fees, is a complicated process that cannot happen without a sufficient understanding of the costs involved and a transparent dialogue with the communities to gain their acceptance of the issues involved. Fees generally range from Q5–Q15 per household per month. It appears the amount is arbitrary. WMCs should go through the Life-Cycle Cost Analysis to properly identify the costs of anticipated operations, maintenance, and repairs that will need to be made as well as the eventual replacement of the system when it has reached its lifespan. These costs should then be used to determine the appropriate household fee to be paid for these purposes. The household payment of higher fees may also be achieved if there is confidence in the water quality and reassurance that the water supply system is well maintained by the WMC. Organizational capacity: Despite the funding crisis facing the municipal offices and the departmental offices that represent the Secretariats and Ministries of the central government, it is important to continue engaging them to build sustainable processes. The framework of the Development Council System provides an important opportunity to create linkages and reinforce existing national support networks. Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (Seguridad Alimentaria Enfocada en los Primeros 1,000 Días [SEGAMIL]) 46 5. RECOMMENDATIONS The evaluation team offers the following recommendations for future FFP projects:        Sustainable livelihoods and income generation need to be a central component of the project. Given the natural resource constraints and costs of production, agricultural yields for maize and beans will top out and offer low rates of return to households. New food security solutions need to be identified that are more market-based. Therefore, it is critical to find ways to incorporate more small landholders into value chains, even if their participation is not in agricultural production. In addition, other non-agricultural opportunities of income generation in these rural communities need to be explored that can provide the population with viable and sustainable sources of revenue. This is particularly relevant for the youth of the area, who are keen to find employment locally rather than to embark on long-distance migration. Technical training, vocational education, and training that is well-tuned to the growing labor demands of the regional and national market is particularly important. SILCs provide an important safety net for families in extreme poverty and a source of credit for productive investments. They take relatively few resources to implement and can easily be self-sufficient after they are established. These should be incorporated in future projects. Future projects would benefit from a stronger focus on household sanitation coverage and usage. This work should be implemented jointly with the environmental sanitation inspectors (inspectores de saneamiento ambiental) and the Ministry of Public Health and Social Services (Ministerio de Salud Pública y Asistencia Social). The reinforcement of messaging at the household level and personalized support in the adoption of behaviors appears to be a promising avenue for behavior change. The cost-effectiveness of this more labor-intensive strategy should be assessed. More linkages should be established with the Ministry of Education Family Development Public Education Program (Nucleos Familiares Educativos para el Desarrollo), which seeks to engage secondary school students in development activities. Engagement of youth in productive activities through these projects can increase school retention and increase the technical skills of youth at the same time. In future FFP projects, it is worthwhile to explore the agricultural analogy approach to social and behavior change communication as a possible way to better integrate agricultural activities with nutrition and health. This methodology, developed by the International Institute for Rural Reconstruction in the Philippines and used extensively in South Asia and Africa, 31 involves building appropriate analogies between agricultural practices and health care practices to facilitate both understanding and adoption of promoted practices. The methodologies used by the project that were particularly successful should be documented, and the approaches and training materials should be made available for wider use. These include the voucher experience and the BCC strategy, including how it was monitored. 31 Nichter, M., and M. Nichter. 2003. Education by Appropriate Analogy. University of Arizona. https://arizona.pure.elsevier.com/en/publications/education-by-appropriate-analogy Annex 1: Evaluation Statement of Work April 27, 2018 1 Statement of Work Population-Based Final Evaluations of Food for Peace Development Food Assistance Projects SEGAMIL and PAISANO in Guatemala INTRODUCTION The final evaluation of the 2012 Guatemala Title II Food for Peace (FFP) projects is the second and final phase of a pre-post evaluation survey. The baseline study, conducted from April-June 2013, employed a mixed-method approach, and was designed to provide information on all four aspects of food security – availability, access, utilization and stability. The study investigated household food access, sanitation and hygiene, agriculture, household expenditures and assets, dietary diversity, and anthropometry among women and children. As with the baseline study, the Guatemala performance evaluation will also use a mixed-method approach, but will also utilize and integrated secondary data and project performance monitoring data. Methods will be chosen in order to generate the highest quality and the most credible and robust evidence possible to answer evaluation questions. BACKGROUND In fiscal year (FY) 2012, USAID/Food for Peace (FFP) awarded Catholic Relief Services (CRS) and Save the Children (SC) six-year Title II development food assistance programs (DFAP) in Guatemala with an overall investment of $79.3 million. The CRS Food Security Program Focused on the First 1,000 Days (SEGAMIL) aims to improve food security of 23,500 families in the San Marcos and Totonicapán departments, while the SC Western Program of Integrated Food Security and Food Actions (PAISANO) seeks to reduce the food insecurity of 26,500 families in Quiche, Quetzaltenango, and Huehuetenango departments. Both programs implement activities in three technical sectors – Agriculture/Natural Resource Management (Ag/NRM), Health/Nutrition (H/N), and Disaster Risk Reduction (DRR) – and have female empowerment as a cross-cutting focus. FFP partners implement in 18 of the 30 Feed the Future (FtF) priority areas in the Western Highlands region; PAISANO programs entirely within the zones of influence (ZOI), while approximately 60 percent of SEGAMIL’s programming is within the ZOI.1 CRS collaborates with the local non-governmental organization Caritas in San Marcos department and with ADIPO in Totonicapán department. They will improve food security among rural populations by (1) increasing food availability and access at the household level through sustainable agricultural production practices, sustainable natural resources management, diversification of income sources, and linkages between farmers’ groups and market processes; (2) reducing chronic malnutrition among children under five through activities such as improved health hygiene practices, nutrition and care practices, and increased household use of health services; and (3) strengthening local and municipal resilience systems through improved community and municipal management of risk and vulnerability to food insecurity. SC implements PAISANO in consortium with Project Concern International (PCI). SC is the prime and implements in Quiche and Quetzaltenango, where 20 percent of the communities participated in the prior multi-year assistance program PROMASA, while PCI implements in Huehuetenango. The program will reduce food insecurity among vulnerable rural populations by (1) increasing food 1 Santa Maria Chiquimula and Totonicapán in Totonicapán department and Comitancillo in San Marcos are not in the FtF zone. These communities were selected by CRS and ultimately approved by FFP based on extremely high levels of chronic malnutrition and poverty. April 27, 2018 2 access at the household (HH) level through increased and diversified agricultural production, fostering women’s business and leadership opportunities, and increasing access to markets; (2) reducing chronic malnutrition among children under five (CU5) by improving the use and quality of maternal and child health and nutrition care at the community level, and improved practices at the HH level; (3) improving resiliency at the community level via improved community and municipal governance and disaster response. Both projects are preceded by FFP-funded multi-year assistance programs (MYAPS) - SEGAMAYA (CRS) and PROMASA II (SC) and Asociacion Share de Guatemala (SHARE) - implemented in the Western Highlands from 2006-2011. EVALUATION PURPOSE & OBJECTIVES The overarching purpose of the final performance evaluation is to measure the development of outcomes of the SEGAMIL and PAISANO projects. Findings from the final evaluation will be used to determine the performance of the two DFAPs. Additionally, findings will help to inform the selection, design, and implementation of future USAID projects in the Western Highlands of Guatemala. EVALUATION QUESTIONS In the table below, FFP has identified the evaluation questions and provided the fundamental elements that should shape the Evaluation Team’s (ET) research. It is anticipated that the ET will use these, but it is not limited to working solely within this guidance. Q1: To what extent have the projects met their defined goals, purposes and outcomes? The ET will evaluate the contribution of SEGAMIL and PAISANO to USAID’s efforts to reduce food insecurity among vulnerable rural populations in five departments of the Western Highlands of Guatemala.2 The ET will support its determination to using both quantitative and qualitative methodologies when discussing the following: (1) project performance on indicators against targets set by both the partners and the key FFP indicators* Depth of Poverty, Stunting, and Underweight; (2) factors that promoted or inhibited the achievement of the project objectives, including, but not limited to the effectiveness of food-for-asset and/or cash-for-asset interventions; (3) plausibility of pathways and the determinants of achieving the key outcomes; (4) targeting strategies and their contribution to achieving project goals (especially with regard to gender and reaching the most vulnerable); (5) the appropriateness and effectiveness of interventions on the poorest individuals. * FFP’s global targets are: a minimum of 2 to 2.5 percentage point annual reduction of prevalence of stunting, a minimum of 3 to 4 percentage point annual reduction of prevalence of underweight, and a minimum of 4 percentage point annual reduction of depth-of-poverty. Q2: Based on the evidence, which project outcomes are likely to be sustained? The ET will evaluate the functionality and performance of systems and services established independently by the projects, as well as in collaboration with the private sector, Government of Guatemala, and non-governmental organizations and academic organizations to achieve project outcomes and sustainability. It will support its determination using both quantitative and qualitative methodologies that explore the following: (1) the quality of the processes, systems, and institutional arrangements developed and/or strengthened to sustain the necessary and critical services; (2) communities’ perceptions on the quality, frequency, effectiveness, and sustainability of the services provided by the project; (3) the likelihood that service providers will continue providing services after the project ends; (4) the motivation of the community and 2 2012 Food for Peace Country Specific Information: Guatemala http://pdf.usaid.gov/pdf_docs/pdacu283.pdf April 27, 2018 3 beneficiaries to demand and pay (or invest time) for the services; (5) whether the necessary resources and capacity strengthening will exist to sustain service providers; (6) the extent to which the projects leveraged other USG and non-USG investments to achieve sustained outcomes as identified in the theories of change; (7) evidence of enhanced linkages with other service providers. Q3: In each technical sector, what are the strengths of and challenges to the effectiveness of the interventions’ implementation and their acceptance in the target communities? The ET will evaluate the effectiveness and relevance of the technical interventions, including food-for-asset and/or cash-for-asset interventions, to achieve project outcomes, and discuss those findings in relation to the projects’ theories of change. It will support its determination using both quantitative and qualitative methodologies when discussing the following: (1) factors in the implementation and context associated with greater or lesser effectiveness in producing Outputs of higher or lower quality; (2) the interventions and implementation processes deemed more/less acceptable to members of the target communities. Q4: What key lessons learned and best practices should inform future projects in the country? During the course of its research, the ET should identify best practices, strengths, and challenges in the projects’ designs (including theories of change), supporting project achievements, as well as approaches that should be considered in designing future food and nutrition security projects and strengthening household and communities’ resilience capacities. The ET will support its determination using both quantitative and qualitative methodologies when discussing the following: (1) the unintended positive and/or negative consequences of the projects; (2) ways to minimize potential unintended negative consequences and systematically capture positive consequences. AUDIENCE & INTENDED USES The primary audience of the evaluation reports will be CRS and SC (and their sub partners), and USAID (FFP/Washington, USAID/Guatemala). The reports will also be shared with the Government of Guatemala. USAID will make extensive use of findings from the evaluation to make different presentations and bulletins as part of a wider dissemination of best practices and lessons learned. The evaluation recommendations may be used by FFP to refine proposal guidelines and project policy. PERFORMANCE EVALUATION METHODOLOGY The final evaluation will use a mixed-methods approach. The ET will begin with a desk review of project documents, validate its understanding of the projects via consultations with USAID and implementing partners, conduct a population-based household survey using all implementation villages as the sampling frame, and lead a purposively sampled qualitative study. It is preferred that, if possible, the quantitative and qualitative components are conducted sequentially so as to allow the quantitative data to inform the qualitative research. a) Desk Review The evaluation team should review the following documents to contextualize and refine the evaluation questions, as well as to gain an in-depth understanding about the project design, implementation, and the food security situation in the area. Partner annual monitoring data should be reviewed when preparing for qualitative research, considered in relation to the evaluation findings, and incorporated into the report in support of/contrast to select evaluation findings. While FFP recommends the below documents for pre-evaluation learning, the literature review should not be limited to the following: April 27, 2018 4 a) Project proposals b) Pipeline Resource Estimate Proposals (PREPs) c) Annual results reports (ARR), including Indicator Performance Tracking Tables (IPTT) for performance against targets d) Midterm review reports and corresponding action plans developed by the two projects e) Baseline Study for the Title II Development Food Assistance Projects in Guatemala, 2013 f) Guatemala Demographic Health Survey 2014/2015 g) 2015 Nutrition, Mortality, Food Security, and Livelihoods Survey (based on SMART methodology) h) Partner formative research and barrier analyses so as to better understand the context and if/how the studies influenced programming i) Monitoring data and reports j) Sustainability and Exit Strategy plans b) Quantitative Endline Survey FFP strongly prefers that tablets are used for quantitative data collection (e.g., computer-assisted personal interviews (CAPI)). The quantitative endline household survey will collect data on the same population-level impact and outcome indicators (presented below) that were collected during the baseline survey. Baseline data were collected between April 1 and June 30, and data collection for the final evaluation must occur during the same time period. The ET should use the same data collection instruments and parameters as the baseline study (level of statistical precision (95 percent confidence intervals), statistical power (80 percent), 6 percentage point reduction to estimate the sample size, Design Effect of 2, etc.)3. The quantitative endline survey design does not need to be identical to the baseline, however; if the projects reduced their target areas, for example, the sampling frame of 6,000 households used for the baseline will need to be adjusted. Data will be collected to support the calculation of the following indicators: 1 Prevalence of underweight children under five years of age 2 Prevalence of Poverty: Percent of people living on less than $1.25/day* 3 Mean Depth of Poverty* 4 Per capita expenditures (as a proxy for income) of USG-targeted beneficiaries* 5 Prevalence of stunted children under five years of age 6 Prevalence of underweight women (of reproductive age)* 7 Percentage of farmers who used at least [a project-defined minimum number of] sustainable agriculture (crop/livestock and/or NRM) practices and/or technologies in the past 12 months 8 Percentage of farmers who used improved storage practices in the past 12 months 9 Percentage of farmers who used financial services (savings, agricultural credit, and/or agricultural insurance) in the past 12 months 10 Percentage of farmers who practices the value chain activities promoted by the project in the past 12 months 11 Prevalence of households with moderate or severe hunger (Household Hunger Scale - HHS) 12 Average Household Dietary Diversity Score (HDDS) 13 Prevalence of children 6-23 months receiving a minimum acceptable diet (MAD) 3 Baseline Study for the Title II Development Food Assistance Projects in Guatemala http://pdf.usaid.gov/pdf_docs/pnaed236.pdf April 27, 2018 5 14 Women’s Dietary Diversity (Score): Mean number of food groups consumed by women of reproductive age (WDDS)* 15 Prevalence of exclusive breastfeeding of children under 6 months of age 16 Percentage of children under age five who had diarrhea in the prior two weeks* 17 Percent of children under five years old with diarrhea treated with Oral Rehydration Therapy (ORT)* 18 Percent of households using an improved drinking water source 19 Percent of households using an improved sanitation facility 20 Percent of households with soap and water at a handwashing station commonly used by family members 21 Percentage of pregnant women Seeking Health Services when danger signs are present 22 Percentage of mothers or caretakers Seeking Health Services when danger signs are present (childhood) 23 Percentage of mothers and caretakers seeking health services when neonatal danger signs are present 24 Percentage of pregnant women that recognize at least 2 pregnancy danger signs that indicate need of seeking Health Services 25 Percentage of women that recognize at least 2 neonatal (<28 days) danger signs that indicate need of seeking Health Services 26 Percentage of mothers that recognize at least 2 childhood illness signs that indicate need of seeking Health Services 27 Percentage of mothers of children 0-59 months reporting that she makes decisions either by herself or jointly with her partner regarding seeking health services for themselves 28 Percentage of mothers of children 0-59 months reporting that she makes decisions either by herself or jointly with her partner regarding seeking health services for their children’s health 29 Food Insecurity Experience Scale (FIES (12 month and 30 day recall)) *Partners were not required to set targets for contextual indicators. Note: Based on USAID’s strategic interests, USAID will add a few additional questions to predict sustainability and test poverty probability index (PPI) into the household questionnaire (and any corresponding indicators added). All quantitative data is owned by USAID and made available to the public barring rare exceptions. c) Consultations As a supplement to the desk review, and in preparation for qualitative research, consultations with FFP and partner staff in Washington, DC and Guatemala will allow the ET to corroborate its understanding of the design, approaches and interventions employed by each DFAP and acquired through the desk review. It is recommended that the ET engage with the staff at each organization prior to beginning fieldwork. Equally important to engaging pre-data collection is to reconnect post-data collection to “ground truth” findings with FFP/Guatemala and the partner staff. In the case of major disagreements, the program staff should provide evidence in support of the argument, and pending time constraints, the ET may revisit the field. d) Qualitative Study Qualitative methods will be used to collect information to answer evaluation questions and also to support the interpretation of the quantitative data. The ET will design the overall qualitative study approach and should consider a variety of primary data collection methods, such as semi-structured in-depth interviews, group discussions, key informant interviews, direct observations, and case studies (the ET may choose to use the most significant change methodology to identify a selective set of case studies). These methods - to the maximum extent possible - will ensure that if a different, April 27, 2018 6 well-qualified evaluator were to undertake the same evaluation, he or she would arrive at the same or similar findings and conclusions. The ET should decide on specific methodologies before traveling to Guatemala, and present those to FFP along with the number of interviews, FGDs, etc., per project, in the inception report. Following discussion and agreement, the ET will finalize the methods during the team meeting in-country. The evaluation team leader and members will be responsible for interviewing the direct, indirect and non-beneficiaries in their households and communities, as well as look for evidence of ongoing learning and activities (such as home gardens, etc.). The ET will also be responsible for interviewing relevant stakeholders for the evaluation and analyzing the qualitative data. Should the ET decide to hire additional researchers to complement the data collection effort, they cannot replace the evaluation team members’ role of collecting primary data using qualitative methods. Note: the intervention areas have large Mayan populations; the evaluators will need to take special care to communicate using the local language and in a culturally sensitive way to ensure the most accurate responses. The ET will contribute to the interpretation of the quantitative results using qualitative findings. In addition to the factors specifically identified earlier as essential to responding to the evaluation questions, during its research during the qualitative study the ET should also consider the efficacy of the following cross-cutting interests: project management; performance monitoring; strategies to improve gender equality at the participant and project management levels; environmental considerations; and conflict sensitivity (i.e., do no harm). Lastly, it is expected that the qualitative study will speak to lessons learned and best practices. The ET may find it useful to apply non-probability sampling methods to select a sub set of enumeration areas from the quantitative survey. In selecting interview sites, the evaluation team should strategically select large-enough-yet-manageable interview sites that generally represent the target area. As with the quantitative household survey, qualitative sampling should include both individuals who directly participated in the DFAP (beneficiaries) and those not specifically targeted with any intervention (non-beneficiaries). In addition, the qualitative team should interview USAID personnel, project staff, host Government officials, local government staff (including, but not limited to, municipalities, ministries, local and central authorities), community leaders and other knowledgeable people in the communities, community development councils (COCODES), and other agencies and individuals as appropriate. e) Data Analysis and Interpretation The evaluation team will statistically compare the endline data for each of the two strata (PAISANO and SEGAMIL) with that of the baseline for that stratum, and also for the overall country level, in order to detect changes (if any) for all key indicators. The ET will conduct descriptive and inferential analyses to describe the results, as well as various econometric analyses to identify the determinants of key outcomes and the magnitude and direction of changes. In advance of fieldwork, the evaluation team should develop a data analysis plan and submit to the COR for approval. When analyzing the data, however, the evaluation team should not limit itself to the data analysis plan; rather, the evaluation team should keep an open and curious mind to look for correlations between variables. In presenting the analysis, the evaluation team should be cognizant about the readers’ familiarity with the statistical presentation. FFP suggests avoiding jargons, but rather describe the statistical terms in a common language. Interpreting the results is as critical as the analysis. Oftentimes, it can be difficult for a reader to fully understand the key points and utility of the findings conveyed in a report. The analysis and April 27, 2018 7 interpretation should be presented in a “story telling format” so that the readers can see a human face as they read the report. While it is important for the reader to understand whether level of stunting is reduced in the area, it is equally important to understand the pathway; for example, how learning derived from project participation influenced people’s practices, which in turn resulted in positive changes in food security outcomes at the household and/or community level. Similarly, it is equally important for the readers to know some of the challenges participants faced that might have prevented them from reaping the full benefits of the projects. REPORT The ET will produce two reports in English, not to exceed 60-pages, for each DFAP. The draft reports will be shared within FFP, the USAID/Guatemala, and the partners for review and comment over a two-week period. The final report should include a table of contents, table of figures (as appropriate), acronyms, executive summary, introduction, purpose of the evaluation, research design and methodology, limitations, findings, conclusions, lessons learned, and recommendations. All evaluation questions should be answered, and the evaluation methodology should be explained in detail. To ensure a high quality deliverable, the reports should reflect a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the project, what did not, and why. Where noteworthy, the discussion should highlight and discuss the outcomes and impacts on males versus females. The report must integrate the quantitative analysis from the PBS and include a statistical comparison between baseline and endline results. Learning from the qualitative research will help to contextualize and interpret the quantitative data. The ET should also draw from partners’ annual monitoring data, where noteworthy, to substantiate and/or compare with evaluation findings. For example, if a qualitative finding is that beneficiaries report increased yields, the ET should compare/contrast that finding against data collected by the partners. Findings should be specific, concise, and supported by strong quantitative and/or qualitative evidence, and presented as analyzed facts/evidence/data, and not be based on anecdotes, hearsay or a compilation of people’s opinions. It should include analytical methods to include appropriate tests of differences; econometric analysis to evaluate the theories of changes and to explore the causal relation between the outcome and activities/variables based on the theoretical models; it is expected that the contractor will interpret the analytical findings. The report should disclose limitations to the evaluation, with particular attention to the limitations associated with the evaluation methodology, e.g., selection bias, recall bias, unobservable differences between comparator groups, etc. Recommendations should be supported by a specific set of findings, and be action-oriented, practical, and specific. It is expected that the final reports will address and incorporate feedback, as appropriate, from the reviewers. Should the ET disagree with any of the comments, it should raise this with the COR immediately for discussion. EVALUATION TEAM The Senior Evaluation Specialist will be responsible for designing and managing the evaluations and overseeing the work of the evaluation team members; coordinating with the implementing partners, USAID Mission and other stakeholders; coordinating with the endline PBS team; analyzing the findings and ensuring the quality of the report. As this is a mixed-method performance evaluation, in addition to the evaluation team, the endline survey will require extensive participation of the following personnel: Senior Survey Method Specialist, Data Analyst, Survey Coordinator, Data April 27, 2018 8 Management Specialist, Anthropometry Specialist, Country Operations Manager, and Survey Monitors. The PBS data collection team should be hired locally, if at all possible. The qualitative study will require in-country participation from the Evaluation Team Leaders and up to three subject matter specialists. As FFP projects are multi-sectoral, the evaluation team must possess expertise and field experience with food security and rural integrated programming, and demonstrate an in-depth knowledge of the following technical sectors and cross-cutting areas: agriculture and off farm livelihoods; nutrition; water, sanitation, and hygiene (WASH); gender, youth, resilience, and disaster risk management. The subject matter specialists must also possess experience and knowledge about the specific processes used by the projects (e.g., Care Groups, Farmer Field Schools, etc.). The qualitative study should also, to the extent possible and appropriate, include local representation. FIELD LOGISTICS The ET is responsible to arrange and pay for all logistics including anthropometric equipment, and transportation. It is strongly recommended that the ET consults the Mission on identifying a reputable car-rental firm. The Mission and partners should also be consulted on identifying interpretation services. The ET should request assistance from the partners on making introductions, as necessary, to local ministry representatives and community leaders. DELIVERABLES The ET shall produce the following deliverables during the evaluation and submit to the Contracting Officer’s Representative (COR). All draft documents should be submitted in Microsoft Word or Microsoft Excel, or in the rare occasion both .pdf and Word/Excel. The COR must review and approve all deliverables. Deliverable Timeline Work Plan • includes a brief synthesis and timeline for the Guatemala final evaluation, with the timeline including major activities throughout the study, including dates by which field guides and training materials will be completed. TBD PBS Enumerator Guide, Supervisor Manual, and Anthropometry Guide • provide revised detailed instructions on supervisor, enumerator and anthropometry trainings. Note that the PBS should use the supervisor, enumerator and anthropometry training guides developed for the baseline. Minor adjustments will be needed to accommodate the new indicators. TBD PBS Data Treatment and Analysis Plan • details how the data will be cleaned, weighted, and analyzed and must include: programming specifications and editing rules for cleaning data, data dictionary codebook, SPSS syntax and output for all data analyses and variable transformations into indicators; and • includes a descriptive, inferential, and econometric analyses plan. TBD PE Inception Report (~20 pages) • briefly synthesizes the literature review; • describes the qualitative evaluation methodology (including evaluation questions contextualized based on the literature review, sample site selection strategy and number of sites to be selected, number of interviews/discussions per project types of interviewees) and how qualitative data will be analyzed and coded; • introduces the evaluation team members and their roles; and TBD April 27, 2018 9 Deliverable Timeline • details how the qualitative information will be analyzed and integrated with quantitative. Final Evaluation Protocol (a combined protocol that discusses both the quantitative and qualitative components) (~30 pages) • describes the evaluation questions to be answered; • presents specific data collection methods by evaluation question; • identifies indicators to be collected; • introduces the local partner (data collection firm); • discusses the quantitative and qualitative analysis methods and plan; • presents PBS sample size, design and plan, survey design, questionnaire design, site selection plan for qualitative research; and • presents the fieldwork plan (including trainings and field support/supervision, data management, quality control, recording, analysis and reporting). TBD Pertinent Permissions and Approvals • demonstrate official approval from all relevant institutional review boards and from host country institutions to collect data, conduct the evaluation, and release data and reports, as required, as well as a statement affirming adherence to all requirements specified in USAID’s Scientific Research Policy. TBD PBS Quantitative Survey and Qualitative Instruments • include both English and Ngakarimojong versions of the household survey (note: the instrument must be back-translated to English via a second translator to ensure accurate translation. Following the pilot of the survey, any modifications based on field experience will again require translation and back translation to ensure accuracy). • describe the site selection methodology and factors used to select sample communities; • list sample communities; • discusses groups to be interviewed; and • explain criteria used to select respondents. TBD [preliminary estimates submitted in advance to support qualitative research] Draft Evaluation Reports (one for PAISANO and one for SEGAMIL) • will be no more than 50 pages per DFAP; • include as an annex tables for FFP indicator estimates, including comparisons between baseline/endline values. October 2018 In-country Briefings to USAID Guatemala and FFP Stakeholders in Guatemala • Two 60-minute presentations of the major findings of the baseline study to provide an opportunity for immediate stakeholder feedback that can be considered for revision (as appropriate and without compromising the validity or independence of the evaluation): o One presentation to USAID/Guatemala; o One presentation to FFP stakeholders in Guatemala, including the DFSA partners, donors, and Government of Guatemala (invited by USAID/Guatemala and the partners), via a 60-minute PowerPoint presentation. November 2018 Briefing to FFP/Washington • comprises a 60-minute PowerPoint presentation on the evaluation, major findings, lessons learned, and recommendations. November 2018 April 27, 2018 10 Deliverable Timeline Final Evaluation Reports • include items identified in the draft report as well as a three- to five-page executive summary of the purpose, background of the project, methods, findings, conclusions and recommendations, and the following annexes: the scope of work, tools used in conducting the evaluation (questionnaires, checklists, and discussion guides), and any substantially dissenting views by any Team member, USAID or the PVOs on any of the findings or recommendations; and • must be 508 compliant and uploaded to the Development Clearinghouse following COR approval. December 2018 Data (to be submitted at the time of the final report*) • include a separate electronic file of all quantitative data in an easily readable format that is organized and fully documented so as to facilitate use by those not fully familiar with the project or the evaluation; • provides cleaned data, sampling weights at each stage, final sampling weights, and all derived indicators; • includes a second final data set in CSV format that has been anonymized to protect individual confidentiality for use as a public data file in the USAID Open Data; and • include a separate file detailing GPS coordinates of households that participated in the PBS. *FFP may request data sets earlier for internal use only December 2018 Annex 2: Population-based Survey Methods July 16, 2018 This publication was produced for review by the U.S. Agency for International Development. It was prepared by ICF Macro, Inc. Guatemala Endline Population-Based Survey Data Treatment and Analysis Plan - FINAL Office of Food for Peace (FFP) Contract #: GS-00F-189CA/7200AA18M00002 Contract #: GS00F189CA/AID-OAA-M-15-00022 Guatemala Endline PBS Data Treatment and Analysis Plan i ABBREVIATIONS ANC Ante-natal care BL Baseline BMI Body mass index CRS Catholic Relief Services DFAP Development food assistance program DTAP Data treatment and analysis plan EL Endline FANTA Food and Nutrition Technical Assistance Project II FFP Office of Food for Peace FIES Food Insecurity Experience Scale FY Fiscal year GHT HDDS HHS Gendered household type Household dietary diversity score Household hunger scale ICF ICF Macro, Inc. IP Implementing partner ORT Oral rehydration therapy PBS Population-based survey PE Performance evaluation PPI Poverty Probability Index SC Save the Children SD Standard deviation USAID U.S. Agency for International Development WASH Water, sanitation and hygiene Guatemala Endline PBS Data Treatment and Analysis Plan ii TABLE OF CONTENTS 1. BACKGROUND ..........................................................................................................................................................................1 2. STUDY DESIGN AND SAMPLE ..............................................................................................................................................2 3. QUESTIONNAIRE.......................................................................................................................................................................2 4. QUALITY CONTROL AND DATA PROCESSING ..........................................................................................................3 4.1 Data Collection Mode and Data Transmission Procedure......................................................................................3 4.2 CAPI Data Entry Training.................................................................................................................................................4 4.3 Field Quality Control Procedures..................................................................................................................................4 4.4 Data Processing Quality Control Procedures.............................................................................................................5 5. DATA PREPARATION...............................................................................................................................................................8 5.1 Sampling Weights................................................................................................................................................................8 5.2 FFP Indicator Definitions ..................................................................................................................................................8 5.2.1 Anthropometry Indicators..............................................................................................................................10 5.2.2 Agricultural Indicators......................................................................................................................................11 5.2.3 Poverty Indicators.............................................................................................................................................13 5.2.4 Custom Indicators.............................................................................................................................................13 5.2.5 Handling of Missing Data and “Don’t know” Responses.........................................................................14 6. DATA ANALYSIS.......................................................................................................................................................................14 6.1 Calculation of FFP and Custom Endline Indicators..................................................................................................14 6.2 Comparison of FFP and Custom Baseline and Endline Indicators .......................................................................15 6.3 Comparison of Baseline and Endline Demographic Characteristics ...................................................................15 6.4 Comparison of Direct and Indirect Participants.......................................................................................................15 6.5 Additional Analyses to Support Performance Evaluations.....................................................................................16 APPENDIX A: Sampling Weights APPENDIX B: Methodology to Derive Poverty Indicators APPENDIC C: Overview of PPI Analyses Guatemala Endline PBS Data Treatment and Analysis Plan 1 1. BACKGROUND In Fiscal Year (FY) 2012, the U.S. Agency for International Development (USAID) Office of Food for Peace (FFP) awarded funding for two multi-year Title II development food assistance programs (DFAPs) in the most food-insecure regions of Guatemala with an overall investment of $79.3 million. The two DFAPs include: (1) Catholic Relief Services’ (CRS) Seguridad Alimentaria Enfocada en los Primeros 1,000 Días (SEGAMIL) program in San Marcos and Totonicapán departments; (2) Save the Children’s (SC) Programa de Acciones Integradas de Seguridad Alimentaria y Nutricional del Occidente (PAISANO) program in Quiché, Huehuetenango, and Quetzaltenango departments. The CRS Food Security Program Focused on the First 1,000 Days (SEGAMIL) aims to improve food security of 23,500 families in the San Marcos and Totonicapán Departments, while the SC Western Program of Integrated Food Security and Food Actions (PAISANO) seeks to reduce the food insecurity of 26,500 families in Quiche, Quetzaltenango, and Huehuetenango Departments. Both programs implement activities in three technical sectors - Agriculture/Natural Resource Management (Ag/NRM), Health/Nutrition (H/N), and Disaster Risk Reduction (DRR) – and have female empowerment as a cross￾cutting focus. FFP contracted with ICF Macro, Inc. (ICF) in 2012 to conduct a baseline (BL) study of the two DFAP areas in Guatemala as the first phase of a two-phase pre-post evaluation cycle. The BL study included a population-based household survey (PBS) and a qualitative component.1 Training for the PBS took place between January and March 20132, and data collection began on April 1, 2013 and ended on June 15 2013. The BL PBS was designed to collect data for 20 FFP indicators related to: food security; children’s nutritional status and feeding practices; women’s nutritional status and dietary diversity; water, sanitation, and hygiene (WASH); agricultural practices; and measurements of poverty. The PBS also included eight custom indicators to assess women’s health care awareness and decision-making for themselves and their children. In 2018, FFP awarded ICF a contract to conduct independent performance evaluations of the two FY 2012 DFAPS which are expected to close out in September 2018. The purpose is to improve understanding of how FFP projects are functioning, whether the projects are achieving targeted results, and how they are perceived by primary stakeholders. ICF will conduct an endline (EL) population-based survey (PBS) in the DFAP implementation areas as part of the performance evaluations. The results of the EL PBS will be used to evaluate change over time in the indicators that were collected at BL. The fieldwork for the EL PBS will be conducted in June-July 2018. ICF has subcontracted with Vox Latina, a local data collection firm, to support the field implementation of PBS. This report describes the data treatment and analysis plan (DTAP) for the EL PBS in Guatemala. The report is divided into six sections. The next section provides a brief description of the study design and sample, the third section describes the questionnaire, the fourth section illustrates quality control and 1 A complete description of the baseline study is provided in the “Baseline Study for the Title II Development Food Assistance Projects in Guatemala” report. Available at: https://www.usaid.gov/sites/default/files/documents/1866/Guatemala%20Baseline%20Study%20Report%2C%20March%202014.pdf 2 Due to delays in the start of field work a re-training of all field coordinators, supervisors and interviewers was conducted from March 18-23, 2013. Guatemala Endline PBS Data Treatment and Analysis Plan 2 data processing measures, the fifth section describes data preparation measures, and the final section focuses on the data analysis plan. 2. STUDY DESIGN AND SAMPLE The primary objective of the BL PBS was to assess the status of FFP and custom indicators prior to program implementation; the primary objective of the EL PBS is to assess the change in these indicators at the end of the six-year DFAP cycle by undertaking statistical tests of differences between BL and EL measures. The EL PBS will utilize the same data collection instruments as the baseline PBS with a module added on program participation, migration and remittances, and the Poverty Probability Index (PPI); and will collect data on the same population-level impact and outcome indicators (see Table 5.2 for a list of indicators) that were collected at BL. This pre-post design allows for the measurement of change in indicators between the BL and EL surveys; however, the design does not allow statements to be made about attribution or causation relating to program impact. The household samples selected for the BL and EL PBSs are based on a multi-stage clustered sample designed to power a test of differences between the BL and EL estimates for the FFP stunting indicator for each DFAP. 3 The sample size is derived based on detecting a difference in the prevalence of stunting from BL to EL of 6 percentage points with 95 percent confidence and 80 percent power. Based on this calculation, the EL PBS will be administered in 70 communities (or clusters) and 2,800 households across the two program areas (35 communities and 1,400 households for each of the two programs). The BL sample included a total of 150 communities and 6,000 households (75 communities and 3,000 households in each of the two program areas). Both of these sample size calculations include a pre-inflation factor to take into account anticipated household level non-response as well as an adjustment factor that takes into account the number of households that need to be contacted in order to achieve the required sample size of children under five years of age for each program. A detailed description of the EL sampling plan is provided in the “Guatemala Endline PBS Protocol” and a detailed description of the BL sampling plan can be found in Annex 1 of the BL study report (see footnote 1). 3. QUESTIONNAIRE The EL PBS will use the same questionnaire as was used for the BL PBS with one module (Module BB) added to include questions on participation in program interventions, mobility, remittances, and 3 Since the input parameters were somewhat different for the two time points, the sample size requirement for EL was updated from a BL value of 1,686 children per program to an EL value of 895 children per program, given that this combination of sample sizes for BL and EL (1,686 and 895) ensured an overall power of 80% for the two time points. In principle, the sample size of required children should be identical for the BL and EL PBSs, since the computation is made at the time of the BL to ensure that the pair of sample sizes provides an overall power of 80 percent. However, the actual realized sample size of children from the BL PBS (2,778 children per program) greatly exceeded the target sample size for the BL PBS (1,686 children per program). This implied that the overall power of the statistical test would likely be considerably greater than 80 percent had the same sample size used at BL also been used at EL, since the power of the statistical test is determined by the actual realized sample size of children at the two time points. Therefore, it was possible to reduce the sample size at EL relative to what it was at BL and still maintain 80 percent power overall. Guatemala Endline PBS Data Treatment and Analysis Plan 3 questions to calculate the Poverty Probability Index i (PPI). 4 Questions are included in Module C to calculate the Food Insecurity Experience Scale (FIES) using a 30 day recall period. The BL questionnaire was developed through a series of consultations with FFP, the Food and Nutrition Technical Assistance II Project (FANTA), and the implementing partners (IPs) before, during, and after the in-country baseline workshop in November 2012. The questionnaire consists of separate modules covering the following topics:           Module A: Household identification and informed consent Module B: Household roster Module BB: Program participation, mobility and remittances, and PPI Module C: Household food security Module D: Children’s nutritional status and feeding practices Module E: Women’s nutritional status and dietary diversity Module F: Water, sanitation and hygiene Module G: Agriculture Module H: Poverty Module I: Mothers’ health care awareness and decision-making Questions for Modules A through G (excluding Module BB) were adapted using questions from the FFP Standard Indicators Handbook and the Demographic and Health Survey (DHS) questionnaire.5 Questions for Module H were adapted from the Guatemala Encuesta Nacional de Condiciones de Vida (ENCOVI) survey, conducted by Instituto Nacional de Estadística (INE) in 2011. Questions for Module I were provided to ICF by the IPs after discussions during the BL planning workshop. Module BB questions on program participation, mobility and remittances were provided by FFP and USAID/Guatemala; and questions for the PPI were provided by Innovations for Poverty Action (IPA). The questionnaire was first developed in English and then translated into Spanish. The total time for completing the survey in each household is expected to be approximately two to three hours, depending on the size of the household. 4. QUALITY CONTROL AND DATA PROCESSING 4.1 Data Collection Mode and Data Transmission Procedure The 2018 EL PBS data will be collected with tablets using Computer-Assisted Personal Interviewing (CAPI) mode by local data collection subcontractor, Vox Latina. Tablets will be loaded with a CSPro data entry application developed at ICF for FFP surveys and tailored to fit the PBS questionnaire. All data will be entered directly into the tablets and edited dynamically while interviewing in the field. For transmission of data from the field, Vox Latina will use Internet File Streaming System (IFSS), a cloud￾based electronic file delivery web service. The primary objective of the service is to deliver files from one user to another in a way that is fast and secure. The ICF CSPro programmer will work in-country to set 4 The PPI is a poverty measurement tool based on questions about a household’s characteristics and asset ownership which are used to score households and compute their likelihood of living below the poverty line. More information is provided on this indicator in Appendix B 5 MEASURE DHS. DHS model questionnaire: Phase 6 (2008-2013) (English, French). Retrieved from http://www.measuredhs.com/publications/publication-dhsq6-dhs-questionnaires-and-manuals.cfm Guatemala Endline PBS Data Treatment and Analysis Plan 4 up and test the cloud-based data transmission system, as well as to provide technical support during the first week of data collection, to ensure that tablets and the IFSS transmission system are operating smoothly. The subcontractor, Vox Latina, will upload data to the IFSS on a weekly basis. Data transmissions will occur on a weekly basis and will begin after interviewers have completed all interviews in their first assigned cluster. For the final dataset, the CSPro programmer will develop a program to run quality control checks and convert the raw data exported from the CSPro application into the data format needed for analysis using Stata, SPSS or SAS. 4.2 CAPI Data Entry Training All interviewers and supervisors will participate in a CAPI data entry training prior to the start of fieldwork to ensure the successful use of tablets during data collection. ICF IT specialists will lead the CAPI training sessions, which will include:        Basic use of the tablet, including how to check and prepare the tablets, switching off/on, login, touch screen/keyboard, rotating screen, buttons to avoid, change of batteries, power management, click/double click, swiping, basic operating system tasks Review of different types of responses to questions, including predetermined numeric, open￾ended numeric, predetermined alpha, open text, and multiple response Trouble spots in the questionnaire and troubleshooting, error messages Anthropometry data entry with anthropometry measurement exercises Practice interviews with tablets in pairs, including starting/stopping the interview, reading questions, entering different types of responses, household rosters, use of calendar for age verification Workflow, including assigning interviews, receiving assignments and sending completed interviews back to supervisors, supervisors transferring updates to interviewers Bluetooth transfers of data to the central office via the IFSS 4.3 Field Quality Control Procedures ICF ensures high-quality data through a strong emphasis on training field staff, and monitoring data collection and quality control at the field level. During critical periods, including training, anthropometry standardization testing, pretesting, piloting, and at the beginning of fieldwork, the ICF survey coordinator will be in-country to coordinate and oversee these activities. When the ICF survey coordinator is not in the country, the local survey monitor will oversee fieldwork activities and closely update the ICF survey coordinator on fieldwork progress or any issues encountered during data collection. The quality control procedures established in the field include: Proper fieldwork oversight: Maximum ratio of one supervisor for every four interviewers, and one anthropometry specialist. Each interviewer will be accompanied by a team supervisor for one full interview, from start to finish, within the interviewer’s first three clusters of households. Inconsistency checks: These will be built into the CSPro data entry application and will include respondent eligibility checks, checks for questionnaire skip patterns and filters, valid response range checks and other quality control checks. Guatemala Endline PBS Data Treatment and Analysis Plan 5 Field-check tables: These tables will be run on raw survey data that are uploaded from the survey teams to the central office via the Internet file transfer system. Therefore, they represent a near real￾time snapshot of the status of the survey data quality. Field-check tables are designed to flag indicators that appear to be lower or higher than anticipated, such as the expected number of eligible women and children per household. The ICF CSPro programmer and supervisors will work together to review the tables and identify any problems. If data collection problems are discovered at the team level, individual-level tabulations can be run to determine whether problems are team-wide or restricted to one or two of the team members. Immediate action will be taken to address problems, either by contacting the team supervisor by telephone or by visiting the team to review the findings. In cases of serious problems, a brief written report will be produced detailing the teams with problems and the actions that were taken. The supervisors of teams whose data indicate serious problems in data collection will be informed immediately of the specific problems observed. Data review: Supervisors will review the electronic data received from interviewers and resolve error messages identified by the program. The review will be conducted on a daily basis to identify any missing or problematic data items. Supervisors will not be able to close a cluster and transmit the final data to the central office until all error messages identified are resolved. Re-interviews: During fieldwork, ten percent of the households interviewed per cluster (three households) will be randomly selected for a short re-interview by the team supervisor. The supervisor will visit the household and conduct a quick re-interview on paper comprising the first two sections of the household questionnaire (the cover page and the household roster). The team supervisor will then compare the manually collected responses to the responses in the CAPI system. Any significant discrepancies between the two will be followed up by the supervisor. Re-interviews can be effective in detecting issues, such as falsifying interviews and deliberate displacement of ages of household members to reduce workload. Completion of interviews: Interviewers will make up to three visits to the household to interview a respondent, and will plan one to two visits with the respondents to successfully complete the interview, as necessary. Closing the cluster: This is the last step for the field team and supervisors before leaving each cluster. This is an ongoing activity throughout the data collection period. After the supervisor receives all data from the team, s/he will run a program to check all data collected for completeness and structural integrity. The program will generate a report flagging any missing or incomplete data items. The supervisor will make sure that any problems are resolved before leaving the cluster. When there are no issues remaining, the system will archive the data and automatically upload them to the local subcontractor’s central office. Data transfers from the field to the central data office in Guatemala City will take place regularly during the data collection. 4.4 Data Processing Quality Control Procedures The CSPro data capture and processing program is designed to allow only valid data ranges, to check questionnaire logic (skips and filters) and to flag data inconsistencies during data entry. The CSPro program will also make comprehensive reviews of the data at the cluster level. Guatemala Endline PBS Data Treatment and Analysis Plan 6 Within CSPro, a hierarchical structure is used to store the survey data; each module corresponds to a unique record within the CSPro dictionary (codebook). For singly-occurring modules (i.e., one set of values per sampled household), such as BB, C, F, H, there will be one line of data in the ASCII file corresponding to the CSPro record where those variables have been defined. For modules where more than one person is included (such as the household roster (Module B), the anthropometry modules for children and women, and Modules D, E, G, and I), there will be one line of data for each person eligible for that roster/module. For example, if there are five persons in the household, there will be five lines of data in the data file corresponding to the record created to represent Module B. The complete suite of quality control checks used during the data processing cycle include the following: 1) Data Capture (During fieldwork in CAPI mode) a) Range checking for numeric responses: Based on all possible values being listed in the CSPro dictionary, CSPro automatically ensures that values cannot be entered outside that range. For example, once the variable "sex" has been assigned to the codes 1 (male) and 2 (female), no other value can be entered. b) Range checking for alphabetic responses: For questions that allow multiple responses to be selected (corresponding to the alphabetic responses), a specially-programmed function has been added, which ensures that: (1) only the letters listed can be entered; (2) allowable letters only appear once ("A", but not "AA"); (3) responses requiring an "other" text entry (generally indicated with the "X" and sometimes "W" characters) are captured; (4) responses that must appear in isolation from any other response (usually "Y" (no one) or "Z" (don't know)) do not appear in combination with any other letter; and (5) the field cannot be left blank. c) Consistency checks: In selected fields when applicable, answers will be cross-checked against other fields for validity. For example, in Modules D and E and the anthropometry sections, age and date of birth will be compared to one another to ensure agreement. In addition, in any module that asks for a person's age, this will be cross-checked against the age given in the household roster (Module B); if an age difference exists, a warning message is issued and the interviewer must verify the correct age. d) Skips: If a skip is present, then based on the respondent's answer to the question, the skip will be applied by the CAPI system. Responses that are skipped will be designated as missing by the CAPI system. For numeric responses, missing is indicated by filling the full field with "9"s. For alpha fields, missing is indicated by filling the field with “X" to indicate "text missing". e) Filters: If a question should not be asked, it will be skipped. For example, persons under the age of 15 are not asked their marital status in the household roster. Therefore, the question will be skipped over for those under-age persons. f) Identifier integrity: A file containing the geographic identifiers will be created for each country. The file provides, for any given cluster, all levels of geographic identifiers. This information will be prefilled from the sample files. This step ensures that the correct identifier is associated with each record. 2) Structure Checks (During fieldwork in CAPI mode) Guatemala Endline PBS Data Treatment and Analysis Plan 7 a) Files are created at the cluster level. They are concatenated into a single file at the very end of closing the clusters. The final data are then transmitted to the central office. When closing the clusters, the total number of households with complete (result=1) and incomplete (result <> 1) result codes are also logged in. A check is applied that compares the number of households found within their data file against what was expected from the sample file, with an error being generated if the two are not the same. Likewise, if the total number of households found is correct, but there are some partially completed households, an error message will be generated. The cluster cannot be closed until these problems have been resolved. b) In addition to checking for result codes and total number of households, the program will ensure for each household that the required number of individual records exist, based on the eligibility of the persons within Module B. For example, if the household roster indicates three persons should be administered Module D, then three records must exist in the file before the structure check can succeed. The cluster cannot advance to the consistency editing stage until any identified problems have been resolved. 3) Miscellaneous Data Quality Measures (During fieldwork in CAPI mode) a) Field-check tables will be run on a weekly basis during fieldwork that will report on several key items measuring fieldwork quality. These tables will show data at the team level. For example, a table will be generated that shows age distributions of female respondents between 12-18 years that allows survey managers to determine if teams are dropping respondents with ages below 15, in order to disqualify women from Module E. This helps to identify underperforming teams. b) Frequencies will be generated to ensure reasonable distribution of the data and that no out-of￾range values exist. 4) Consistency Checks (After fieldwork is complete) a) More complex issues are handled after fieldwork is complete. Once a cluster has been closed in the field and data have been transmitted to the central office, a secondary (consistency) edit program will be run against the data in the central office. Many of the checks made during the interviewing process will be repeated here. All error messages are assigned a unique number. b) The central office will be provided a secondary editing manual that lists all error messages in numerical order. It will describe the problem that prompted the error, and possible methods to resolve the conflict. In general, the method is to review the data collected, compare the variables (questions) involved, and look for any notes the interviewer may have made, or changes the field supervisor or field coordinators may have made, that created/exacerbated the problem. Checks for missing values are not made at this time, as it is too late for the field team to resolve this type of error. ICF will conduct a quality control review of the raw and edited data as the data is received from the central office in Guatemala City. Data transfers will take place weekly from the central office to ICF in Rockville, MD, via the IFSS secure file transfer protocol. Data cleaning will take place based on secondary (consistency) editing reports generated in-country, and per ICF feedback. Final review and data cleaning will take place at ICF in Rockville, MD, upon receipt of the final datasets. The final raw CSPro datasets Guatemala Endline PBS Data Treatment and Analysis Plan 8 will be accompanied by a data dictionary/codebook with all variables clearly labeled. The raw CSPro datasets will be converted to facilitate data analysis using SAS, Stata or SPSS Statistical Software. 5. DATA PREPARATION 5.1 Sampling Weights Sampling weights will be computed and used in the data analyses. This will involve computing an overall sampling weight for each indicator by taking the inverse of the product of the probabilities of selection from each stage of sampling (community selection; household selection; and, when relevant, individual selection). Weights will be derived according to the unique sampling scheme that is relevant to the associated sampled household or individual. Separate weights will be calculated for the following target sampling groups:       Households (used for indicators derived from Modules BB, C, F, H) Children under 5 years of age (Module D and children’s anthropometry) Women 15-49 years of age (Module E) Non-pregnant women 15-49 years of age (Women’s anthropometry) Farmers (Module G) Mothers of children under five years of age (Module 1) Weights will be calculated separately for each of the program areas and will be adjusted to compensate for household- and individual-level non-response, where appropriate. The household level nonresponse adjustment, relevant for all modules, is based on the total number of households with completed interviews and the total number of households in each cluster. Individual-level non-response adjustments for Modules D, E, G, and I are based on the total number of completed interviews for each group of individuals and the total number of eligible individuals from the household roster.6 A more detailed description of the calculation for sampling weights is provided in Appendix A. 5.2 FFP Indicator Definitions The 22 FFP indicators and eight custom indicators to be included in the EL data analysis are listed in Table 5.2. A definition and full description of each of the 22 FFP indicators is provided in the FFP Indicator Handbook. 7 Definitions and full descriptions of the eight custom indicators are included in section 5.2.4. Results for all indicators will be weighted to represent the full target population and tabulated for the combined program areas and for each program separately. 6 Strictly speaking, a separate non-response adjustment should be made for all indicator subgroups, e.g., children 0-5 months, children 6-23 months, children 0-59 months, etc. However, nonresponse for these subgroups very closely mirrors nonresponse for the entire group (i.e., children 0-59 months), so one nonresponse adjustments for the entire group is used. 7 USAID. (2011). FFP standard indicators handbook (baseline-final indicators). Retrieved from http://pdf.usaid.gov/pdf_docs/pnadz580.pdf Guatemala Endline PBS Data Treatment and Analysis Plan 9 Table 5.2 Guatemala Baseline and Endline PBS Indicators Indicator Disaggregatio n Level FOOD SECURITY 1. Average Household Dietary Diversity Score (HDDS) None 2. Prevalence of moderate or severe hunger (HHS) GHT 3. Prevalence of moderate or severe food insecurity (FIES) [30 day recall] GHT POVERTY 4. Per capita expenditures (as a proxy for income) of USG-assisted areas GHT 5. Prevalence of poverty: Percent of people living on less than $1.25 per day GHT 6. Mean depth of poverty: Mean percent shortfall relative to the $1.25 poverty line GHT WATER, SANITATION, AND HYGIENE 7. Percentage of households using an improved drinking water source None 8. Percentage of households using an improved sanitation facility None 9. Percentage of households with soap and water at a handwashing station commonly used by family members None AGRICULTURE 10. Percentage of farmers who used financial services (savings, credit or insurance) in the past 12 months Sex 11. Percentage of farmers who practiced the value chain activities promoted by the project in the past 12 months Sex 12. Percentage of farmers who used at least three sustainable agriculture [crop, livestock and natural resource management (NRM] practices and/or technologies in the past 12 months Sex, type of practice (crop, livestock, NRM) 13. Percentage of farmers who used improved storage practices in the past 12 months Sex WOMEN’S HEALTH AND NUTRITION 14. Prevalence of underweight women None 15. Women’s dietary diversity score (WDDS) None CHILDREN’S HEALTH AND NUTRITION 16. Prevalence of underweight children under five years of age Sex 17. Prevalence of stunted children under five years of age Sex 18. Prevalence of wasted children under five years of age Sex 19. Percentage of children under age five who had diarrhea in the past two weeks Sex 20. Percentage of children under five years old with diarrhea treated with oral rehydration therapy Sex 21. Prevalence of exclusive breast-feeding of children under six months of age Sex 22. Prevalence of children 6-23 months receiving a minimum acceptable diet Sex CUSTOM INDICATORS 23. Percentage of women with a child under 5 years who recognize two or more pregnancy danger signs None 24. Percentage of women with a child under 5 years who recognize two or more neonatal danger signs None 25. Percentage of women with a child under 5 years who recognize two or more childhood illness signs None 26. Percentage of women with a child under 5 years seeking health care when pregnancy danger signs are present None 27. Percentage of women with a child under 5 years seeking health care when neonatal danger signs are present None 28. Percentage of women with a child under 5 years seeking health care when childhood illness signs are present None 29. Percentage of women with a child under 5 years making decisions about health care for themselves None 30. Percentage of women with a child under 5 years making decisions about health care for their child None GHT=Gendered household type Guatemala Endline PBS Data Treatment and Analysis Plan 10 5.2.1 Anthropometry Indicators Children: Children’s nutritional status indicators will be computed following the method used by the DHS.8 To obtain anthropometric indicators on stunting, underweight, and wasting children from height/length, weight, and/or age, the World Health Organization (WHO) child growth standards (WHO Multicentre Growth Reference Study Group 2006) will be used to compute three nutritional scores described as z-scores. These z-scores are the HAZ (height-for-age), WAZ (weight-for-age), and WHZ (weight-for-height), relating to stunting, underweight and wasting, respectively. Each z-score is calculated by comparing the child’s height/length or weight with the median value of the WHO 2006 reference population. The difference is divided by the standard deviation of the reference population as shown in the following formula: Z-score = (Individual value of the child – median value of children in the reference population) / (standard deviation of the reference population) Using the above formula, each z-score for HAZ, WAZ, and WHZ is calculated as follows: Z-score for HAZ = (Height-for-age of children in the sample – median value of height-for-age of children in the reference population) / (Standard deviation of height-for-age of children in the reference population) Z-score for WAZ = (Weight-for-age of children in the sample – median value of weight-for-age of children in the reference population) / (standard deviation of weight-for-age of children in the reference population) Z-score for WHZ = (Weight-for-height of children in the sample – median value of weight-for-height of children in the reference population) / (standard deviation of weight-for-height of children in the reference population) After obtaining the z-scores, datasets will be cleaned by flagging cases with z-scores beyond specified lower or upper cutoffs and excluding them from the computation of all indicators. The purpose of flagging is to eliminate extreme values that are most probably due to measurement errors or data-entry errors. Flags used in cleaning anthropometric data prior to computing indicators are shown in Table 5.2.1a below. Cases with height-for-age z-scores that are less than -6 standard deviations (SD) from the median or greater than +6 SD above the median will be flagged and excluded from the calculation of the prevalence of stunting. Cases with weight-for-age z-scores that are less than -6 SD from the median or greater than +5 SD above the median will be flagged and excluded from the calculation of the prevalence of underweight. Cases with weight-for-height z-scores that are less than -5 SD from the median or greater than +5 SD above the median will be flagged and excluded from the estimation of the prevalence of wasting. 8 Assaf, Shireen, Monica T. Kothari, and Thomas Pullum. 2015. An Assessment of the Quality of DHS Anthropometric Data, 2005-2014. DHS Methodological Reports No. 16. Rockville, Maryland, USA: ICF International. Guatemala Endline PBS Data Treatment and Analysis Plan 11 Table 5.2.1a: Flags used in cleaning anthropometric data prior to computing indicators Z-score Cut-off point Height-for-age z-scores (HAZ) <-6 SD or >+6 SD Weight-for-age z-scores (WAZ) <-6 SD or >+5 SD Weight-for-height z-scores (WHZ) <-5 SD or >+5 SD Women: For women, body mass index (BMI) will be used to measure nutritional status. The BMI is the ratio of the weight in kilograms to the square of the height in meters (kg/m2). Criteria for measuring women’s nutritional status by BMI levels are shown in Table 5.2.1b. Cases with BMI less than 12 or greater than 60 are flagged and excluded from the calculation of the indicator. The purpose of flagging is to eliminate extreme values that are most probably due to measurement errors or data-entry errors. Table 5.2.1b: Indicators of women’s nutritional status by BMI levels Women’s Nutritional Status BMI Moderately & severely underweight < 17 Mildly underweight 17.0 - 18.49 Normal weight 18.5 - 24.9 Overweight 25.0 - 29.9 Obese >=30 5.2.2 Agricultural Indicators Country-specific adaptations of the FFP agricultural indicators were developed after discussions with FFP, FANTA, and the IPs during the Guatemala BL planning workshop in November 2012. Indicators relating to the use of financial services, value chain activities, sustainable agricultural practices, and improved storage practices were defined based on those activities and practices used and promoted by the programs. The following tabulation instructions will be used to calculate the agricultural indicators:    Percentage of farmers who used financial services (savings, agricultural credit, and agricultural insurance) in the past 12 months will be calculated based on the sample weighted number of farmers who reported using at least one financial service divided by the sample weighted total number of eligible farmers. Percentage of farmers who practiced the value chain activities promoted by the project in the past 12 months will be calculated based on the sample weighted number of farmers who reported using at least two value chain activities promoted by the project divided by the sample weighted total number of eligible farmers. Percentage of farmers who used a project-defined minimum number of sustainable agricultural practices (crop, livestock, and/or natural resource management [NRM]) in the past 12 months will be calculated based on the sample weighted number of farmers who reported using three sustainable agricultural practices and/or technologies promoted by the project divided by the Guatemala Endline PBS Data Treatment and Analysis Plan 12 sample weighted total number of eligible farmers. The minimum threshold is set at two for the individual sustainable agricultural practice sub-indicators specifically related to crops, livestock and NRM practices (see table 5.2.2).  Percentage of farmers using improved storage practices will be calculated based on the sample weighted number of farmers who reported using at least one improved storage practice and/or technology divided by the sample weighted total number of eligible farmers. Table 5.2.2 provides the definitions of the activities and practices that will be evaluated for these indicators. Table 5.2.2. Definition of Agricultural Indicators Indicator Definition Survey Questions Comments Percentage of farmers who used at least two sustainable agricultural (crop) practices and/or technologies in the past 12 months 1. Use of improved seeds for beans (SC) 2. Use of soil conservation practices (CRS, SC) 3. Use of organic fertilization (CRS, SC) 4. Household garden production (CRS, SC) 5. Management of fruit trees (SC) 6. Production of native herbs (CRS, SC) 1. G19 2. G49 3. G11, G22, G34, G44 4. G29 5. G40 6. G31 Only those farmers who grow corn (G05), beans (G18), herbs & vegetables (G29), or fruit trees (G39) were asked these questions except for G49. All farmers were asked question G49. Percentage of farmers who used at least two sustainable agriculture (livestock) practices and/or technologies in the past 12 months 1. Improved (roof, wall, water and food) livestock facilities for rabbits, goats, pigs and poultry (CRS, SC) 2. Vaccination of poultry (CRS, SC) 3. Management of goats (3 of 4: deworming, vaccination, hood removal, vitamin supplementation) (SC) 1. G60 2. G61 3. G62 Only those farmers involved in livestock breeding or caretaking (G56) were asked these questions. Percentage of farmers who used at least two sustainable agriculture (NRM) practices and/or technologies in the past 12 months 1. Management of watershed (CRS) 2. Agroforestry or cultivation of fruit trees (CRS, SC) 3. Reforestation (CRS, SC) 4. Management of forest plantations (CRS, SC) 5. Management of natural regeneration (CRS, SC) 6. Production of seedlings (CRS, SC) G55 Only those farmers who reported being involved in forest production (G52) were asked these questions. Percentage of farmer who used at least three sustainable agricultural practices and/or technologies in the past twelve months This indicator will be set based on whether three of the activities listed for all three sub￾indicators above are used. NA Percentage of farmers who used any improved storage practices in the past 12 months 1. Silos 2.Other improved practices (improved crib, drying crib, hermetic tonnels) G17 - CORN (CRS, SC) G28 – BEANS (SC) Only those farmers who grow corn (G05) or beans (G18) are asked these questions. Percentage of farmers who practiced the value chain activities promoted by the project in the past 12 months (at least two activities) 1. Market-oriented production 2. Calculation of cost of production for the market 3. Keeping production records 4. Developing production and sales plans 1. G68B 2. G70A 3. G70C 4. G75 Only those farmers who reported selling their products (G58B) were asked these questions. Percentage of farmers who used any financial services in the past 12 months 1. Savings 2. Agricultural credit 3. Agricultural insurance 1. G79 2. G77 3. G78 All farmers where asked these questions. Note: The denominator for each indicator includes all eligible farmers, even though a subgroup of farmers is asked the question. Guatemala Endline PBS Data Treatment and Analysis Plan 13 5.2.3 Poverty Indicators Calculation of the three poverty indicators involves a complex and time-consuming methodology that follows guidance from USAID and the World Bank. A detailed description of this methodology is provided in Appendix B. The EL study will include questions to calculate the PPI (developed by Innovations for Poverty Action) which will be used to construct two poverty indicators: prevalence of poverty and depth of poverty. These two indicators will be compared to the same two indicators constructed using the LSMS methodology as described in Appendix B. This is a pilot study for the PPI in order to test its viability as a replacement methodology for the LSMS variant for calculating the poverty indicators. The analyses for the PPI will be conducted by Innovations for Poverty Action (IPA). See Appendix C for a summary description of the proposed analyses for the PPI. 5.2.4 Custom Indicators Eight additional indicators were selected and defined based on the program strategic frameworks designed by the IPs. These indicators were discussed during the BL planning workshop and were finalized based on input from FFP, FANTA, and the IPs. These indicators will be tabulated based on the definitions and methodology provided by the IPs as shown in Table 5.2.4. Table 5.2.4. Definitions for Custom Indicators Custom Indicator Tabulation Instructions 1. Percentage of mothers of children 0-59 months that recognize at least 2 pregnancy danger signs that indicate need of seeking Health Services Numerator: I02=1 + 2 or more responses 1 to 8 in I03 Denominator: Total number of mothers of children 0-59 months 2. Percentage of mothers of children 0-59 months that recognize at least 2 neonatal (<28 days) danger signs that indicate need of seeking Health Services Numerator: I07=1 + 2 or more responses 1 to 10 in I08 Denominator: Total number of mothers of children 0-59 months 3. Percentage of mothers of children 0-59 months that recognize at least 2 childhood illness signs that indicate need of seeking Health Services Numerator: I12=1 + 2 or more responses 1 to 9 in I13 Denominator: Total number of mothers of children 0-59 months 4. Percentage of mothers of children 0-59 months seeking health services when pregnancy danger signs were present during their last pregnancy Numerator: I02=1 + any response 1 to 08 in I03 + I04=1 + I05=1 + I06=1 any response 1 to 6 Denominator: I02 =1 + any response 1 to 08 in I03 + I04= 1 5. Percentage of mothers of children 0-59 months seeking health services when neo natal danger signs are present Numerator: I07=1 + any response 1 to 10 in I08 + I09=1 + I10=1 + I11=1 any response 1 to 6 Denominator: I07 =1 + any response 1 to 10 in I08 + I09=1 Guatemala Endline PBS Data Treatment and Analysis Plan 14 6. Percentage of mothers of children 0-59 months seeking health services when childhood illness danger signs are present Numerator: I012=1 + any response 1 to 09 in I13 + I14=1 + I15=11 + I16=1 any response 1 to 6 Denominator: I012=1 + any response 1 to 09 in I13 + I14=1 7. Percentage of mothers of children 0-59 months that are married or living in a union and report making decisions about health care for themselves Numerator: I17=1 or 2 + (I18=1 or I18=3 or I18=4) Denominator: Total number of mothers of children 0-59 months married or living in a union (I17=1 or 2) 8. Percentage of mothers of children 0-59 months that are married or living in a union who report making decisions about health care for their children 0-59 months Numerator: I17=1 or 2 + (I19a=1 or I19a=3 or I19a=4 or I19b=1) Denominator: Total number of mothers of children 0-59 months married or living in a union (I17=1 or 2) 5.2.5 Handling of Missing Data and “Don’t know” Responses Missing data points will be excluded from both the denominator and the numerator for calculation of all indicators. “Don’t know” and “Refused” responses will be excluded from the numerators used in the calculation of the indicators. For example, for the HH dietary diversity (HDDS) indicator, “Yes,” “No,” and “Don’t Know” responses were included in the denominator, but only “Yes” responses will be counted in the numerator. For poverty indicators, there are special instructions to handle missing data (See Appendix B). 6. DATA ANALYSIS Data analysis for the EL PBS will include calculation of all FFP and custom indicators, examination of some key demographic characteristics, and comparisons of BL and EL data (FFP and custom indicators, and demographic characteristics) for each of the two programs separately and the combined program areas. Sampling weights will be used for all analyses and all analyses will be done using Stata version 14.9 Additional analyses of the PBS data will be conducted to support the performance evaluations. Illustrative examples of analyses of PBS data for the performance evaluations are provided below and will be refined in the evaluation protocols. The detailed plan of data analyses is given below. 6.1 Calculation of FFP and Custom Endline Indicators All EL indicators (22 FFP and 8 custom) will be generated using relevant sampling weights to represent the full target population and tabulated for the combined program areas and for each program separately. In addition to the 22 FFP and 8 custom indicators measured at BL, the PPI and FIES 30-day recall indicator will be calculated. All indicators will be disaggregated, as noted in Table 5.2. Point estimates and standard error estimates (derived using Taylor series expansion and taking into account the design effect associated with the complex sampling design) will be provided; 95 percent confidence intervals will also 9 StataCorp. 2015. Stata Statistical Software: Release 14. College Station, TX: StataCorp LP. Guatemala Endline PBS Data Treatment and Analysis Plan 15 be provided for all FFP and custom indicators at the aggregated program level and for each program separately. 6.2 Comparison of FFP and Custom Baseline and Endline Indicators As mentioned before, the main objective of the EL survey is to gather data to generate estimates of the same indicators that were collected at BL and to compare these EL estimates with the BL estimates to determine whether statistically significant changes have occurred over time.10 The results from this EL survey will inform the final performance evaluations for the two FFP DFAPs. EL data for each of the two programs will be statistically compared with that of the BL in order to detect significant changes (if any) for all indicators that were collected at BL. Appropriate statistical methods (e.g., independent sample t￾test, chi-square test) will be used to compare the differences between BL and EL estimates. 6.3 Comparison of Baseline and Endline Demographic Characteristics Analysis will be done to compare the demographic characteristics between BL and EL samples to assess if there are any statistically significant differences in selected demographic characteristics between these two samples. Because of the study design, general demographic characteristics within the program areas are expected to remain relatively similar over time for both the BL and EL samples. However, if the demographic characteristics change significantly between the two survey rounds, then explanations of changes in the indicators should take into consideration the nature of the changes in relevant demographic characteristics. Possible reasons for changes over time could be sampling bias, results of program activities or external factors. Household demographic characteristics that will be analyzed include the following:            Average household size Average age of household head Average age of primary caregivers of children under 5 years household headship (percent female) Percent of households with women 15-49 years Percent of household with currently married women 15-49 years Percent of households with at least one child under 5 years Percent of household with at least one child under 2 years Percent of household with at least one child under 6 months Educational level of household head Educational level of primary caregiver 6.4 Comparison of Direct and Indirect Participants For the EL PBS, a question was added to the questionnaire to ascertain whether each household received any services from the DFAP operating in their community, i.e., a proxy measure of whether the 10 Because data for the PPI was collected at EL only there will be no BL-EL comparison for this indicator. Guatemala Endline PBS Data Treatment and Analysis Plan 16 household contained any direct project participants. This question will allow for an analysis of direct project participants (those who received DFAP services) compared to indirect project participants (those who did not receive DFAP services). Project participation status will be included as a control variable in multivariate analyses of select outcome and impact indicators to evaluate if the effects of project interventions “spilled over” from direct participants to indirect participants – as is desired because the intent of the FFP programs is to expressly induce spillover effects from direct beneficiaries to indirect beneficiaries over the course of the DFAP. 6.5 Additional Analyses to Support Performance Evaluations Additional multivariate analyses will be conducted to support the performance evaluations of the two DFAPs. Multivariate analyses could focus on key impact indicators such as stunting, wasting, and the prevalence of hunger controlling for project participation status and a host of other confounding factors that can be included based on the data collected at baseline and endline. The selection of additional analyses of the PBS data for the performance evaluations, including specific indicators on which to focus, will be finalized following the baseline-endline comparison of FFP and custom indicators, and in tandem with the evaluation protocols, which will specify the evaluation questions and methods of analyses to be used to address those questions. Guatemala PBS Data Treatment and Analysis Plan APPENDIX A – Sampling Weights Guatemala Endline PBS Data Treatment and Analysis Plan 17 HOUSEHOLD WEIGHTS Household weights will be applied for household level indicators derived from Modules BB, C, F, and H and included in the construction of individual weights for all other modules. 11 Household design weights are calculated based on the separate sampling probabilities for each sampling stage and for each cluster. 𝑃1ℎ𝑖= first-stage sampling probability of the i-th cluster in stratum h 𝑃2ℎ𝑖= second-stage sampling probability within the i-th cluster (household selection). The probability of selecting cluster i in the sample is: 𝑃1ℎ𝑖= 𝑚ℎ × 𝑁ℎ𝑖 𝑁ℎ × 𝑏ℎ𝑖 The second-stage probability of selecting household in cluster i is: 𝑃2ℎ𝑖 = 𝑛ℎ𝑖 𝐿ℎ𝑖 ⁄ Where: 𝑚ℎ= number of sample clusters selected in stratum h. total households in the frame for the i-th sample cluster in stratum h (obtained from census or other outside information source). 𝑁ℎ𝑖= 𝑁ℎ= total households in the frame in stratum h. 𝑏ℎ𝑖= the number of selected segments divided by the total number of segments in the i￾th sample cluster in stratum h if the cluster is segmented; otherwise 𝑏ℎ𝑖= 1 𝑛ℎ𝑖 = number of sample households selected for the i-th sample cluster in stratum h. 𝐿ℎ𝑖= number of households for the i-th sample cluster in stratum h.12 The overall selection probability of each household in cluster i of stratum h is the product of the selection probabilities of the two (or three) stages: 𝑃ℎ𝑖 = 𝑃1ℎ𝑖 x 𝑃2ℎ𝑖 = 𝑚ℎ × 𝑁ℎ 𝑁ℎ × 𝑏ℎ𝑖 × 𝑛ℎ𝑖 𝐿ℎ𝑖 ⁄ The household design weight for each household in cluster i of stratum h is the inverse of its overall selection probability: 𝑊ℎ𝑖 = 1 𝑃ℎ𝑖 = 𝑁ℎ×𝐿ℎ𝑖 𝑚ℎ×𝑁ℎ𝑖×𝑛ℎ𝑖×𝑏ℎ𝑖 The household sampling weight is calculated using the household design weight corrected for household non-response in each of the selected clusters. Weighted response rates are calculated at the cluster level as ratios of the weighted number of interviewed households 11 Note that sampling weights for the baseline dataset were calculated using the same methodology and are described in the baseline study report – see footnote 1. 12 Ideally, this is the same as 𝑁ℎ𝑖 , but most often is not in practice. Guatemala PBS Data Treatment and Analysis Plan APPENDIX A – Sampling Weights Guatemala Endline PBS Data Treatment and Analysis Plan 18 divided by the weighted number of eligible households, where the weights used are the household design weights. The household sampling weight is calculated by dividing the household design weight by the weighted household response rate. INDIVIDUAL WEIGHTS Individual sampling weights will be applied for indicators derived from Modules D (children), E (women of reproductive age), G (farmers), and I (mothers of children under 5 years). Since all individuals will be selected for each Module (except Module I), these weights will include a non-response adjustment only. The nonresponse adjustment will be applied using the inverted proportion of the total number of completed interviews for each group divided by the total number of eligible individuals for each group. Individual weights for Module I will include a third stage sampling probability since one mother of children under 5 years is randomly selected per household. 𝑃3ℎ𝑖𝑗= third-stage sampling probability within the i-th cluster and j-th household (individual selection). The individual design weight for each individual in household j in cluster i of stratum h is the inverse of its overall selection probability: 𝑊ℎ𝑖𝑗 = 1 𝑃1ℎ𝑖 ×𝑃2ℎ𝑖×𝑃3ℎ𝑖𝑗 The same household level and individual level nonresponse adjustments will be applied to the individual design weight to arrive at the individual sampling weight for Module I. Guatemala PBS Data Treatment and Analysis Plan APPENDIX C – Overview of PPI Analyses Guatemala Endline PBS Data Treatment and Analysis Plan 19 The World Bank defines poverty as whether households or individuals have enough resources or abilities today to meet their needs. Poverty is usually measured based on consumption levels rather than other measures such as income. Actual consumption is more closely related to a person’s well-being in the sense of having enough to meet current basic needs. Also, in poor agrarian economies and in urban economies with large informal sectors, income may be difficult to estimate. It may be seasonal and erratic, and it may be difficult to estimate particularly for agricultural households whose income may not be monetized. The prevalence of household poverty will be measured using information on household consumption expenditures to compute a household consumption expenditures aggregate. The consumption expenditures aggregates will be constructed following guidelines from Deaton & Zaidi (2002)13 and Grosh & Muñoz (1996)14 by adding together the various goods and services consumed by each household during a period of 12 months. The various components of consumption will be grouped together into 6 main categories, including food, usual expenses (expenses in the last 7 days), occasional expenses (expenses in the last 30 days), unusual expenses (expenses in the last 12 months), housing, and durable assets. In general consumption expenditures will be calculated by adding the value in local currency units (LCU) of the items consumed by the household, as reported by household informants. These items will be collected according to different time horizons, but will be then transformed into daily per capita consumption expenditures. Whenever a household misses data on the value consumed for a given item, that value will be imputed using the closest local median value for that item. That is, if a household missed consumption information on a given item, they will be assigned the median value reported by other households in the vicinity. Whenever the item is reported frequently enough, this imputation will be done at the cluster level. However some items may be consumed by few households. In those cases the level of imputation would be at a higher level, depending on how rare the item is. These imputed amounts will be subject to checks that the imputed prices are plausible to avoid undue influence from outliers. The reported values for each item and each consumption component will be checked for outliers to detect possible coding errors or extreme values. Values that are 3 standard deviations (SD) over the average will be flagged and checked for plausibility. Values deemed implausible will be imputed using the methodology described above. Besides this general methodology, some components require specific computations.  Food Consumption Computation of food consumption expenditures is complex because it involves products that are purchased in the market, where price information is available, and products that are home-produced or received as a gift, where price information is not available. Even when products are purchased, it is often difficult for household informants to report the precise market value of the amounts consumed by the household over the reference period, which often results in missing data. 13 Deaton, A. and S. Zaidi (2002), A Guide to Aggregating Consumption Expenditures, Living Standards Measurement Study, Working Paper 135. Available at: http://siteresources.worldbank.org/INTPA/Resources/429966-1092778639630/deatonZaidi.pdf 14 Margaret Grosh and Juan Muñoz (1996). A Manual for Planning and Implementing the Living Standards Measurement Study Surveys. LSMS Working Paper #126, The World Bank. Available at: http://documents.worldbank.org/curated/en/1996/05/438573/manual-planning-implementing-living-standards-measurement￾study-survey Guatemala PBS Data Treatment and Analysis Plan APPENDIX C – Overview of PPI Analyses Guatemala Endline PBS Data Treatment and Analysis Plan 20 The value of non-purchased food (and of any food missing market value information), will be imputed by transforming the amounts consumed by the household to a common reference unit, and multiplying the local median value of that unit times the amount consumed. If a product is reportedly consumed, but amount information is missing, the median per capita amount consumed by local households will be imputed.  Assets Purchases of durable goods represent large and relatively infrequent expenses. While almost all households incur relatively large expenditures on these items at some point, only a small proportion of all households are expected to make such expenditures during the reference period covered by the survey. As indicated by Deaton & Zaidi (2002) “From the point of view of household welfare, rather than using expenditure on purchase of durable goods during the recall period, the appropriate measure of consumption of durable goods is the value of services that the household receives from all the durable goods in its possession over the relevant time period” (p. 33). Consumption of durable goods was calculated as the annual rental equivalent of owning the asset. As the value of the item when new was not available in the data sets, consumption of durable goods was calculated based on the estimated remaining life of the asset, as recommended by Deaton & Zaidi (2002): First, the average age for each durable good, 𝑇̅, was calculated from the data on the current age of the particular respondent’s asset recorded in the survey (𝑇 good was estimated as 2𝑇̅ under the assumption that purchases are uniformly distributed through time. ). The average lifetime of each durable This uniform distribution is defined over the continuum 0 to 2T̅ and has a mean of T̅. The remaining life of each good was calculated as 2𝑇̅ − 𝑇. A rental equivalent estimating the daily per capita flow of services from the durable goods is then derived by dividing the current replacement value of the good by its expected remaining life.  Housing The case of housing is similar to other durable goods, in that it is better measured as an annual consumption of housing services, either annual rent expenditures for renters, or an annual rental equivalent for non-renters. The survey collected information on rent paid among renters, and an estimated rental equivalent for non-renters. It is likely that the housing rental market is small and a significant amount of non-renters are unable to provide an estimated rental equivalent. Missing responses were imputed using two approaches. First, the age of the house and its current replacement value was used to estimate a housing rental equivalent, using the methodology described above for durable goods. For those cases where an estimated current value or age of the house were not available, a hedonic OLS regression model will be used, as suggested by Grosh & Muñoz (1996). The model will be built on the sample of households reporting non-zero rent or rental equivalents, with the log of rent paid by renters as a dependent variable, and several sets of independent variables, that may include: - - - Housing characteristics: number of members, type of water access, type of sanitation services. Socio-economic status: consumption sub-aggregates, asset ownership, Household Hunger Score. Location: Urban vs. Rural, department, community. Guatemala PBS Data Treatment and Analysis Plan APPENDIX C – Overview of PPI Analyses Guatemala Endline PBS Data Treatment and Analysis Plan 21 The final model will be estimated based on the following regression equation, log(𝑅𝑖) = 𝛽0 + 𝛽𝑋𝑖 + 𝜀𝑖 where 𝑅𝑖 represents the reported non-zero rent paid by household i, 𝛽0 is the constant term, 𝑋𝑖 is the final vector of independent variables and 𝜀𝑖 is the error term accounting for unexplained variance. The initial model will contain consumption variables in log form and a set of dummies for all categorical variables. In order to avoid problems with multi-collinearity, a forward stepwise regression approach will be used to exclude variables that do not contribute to model fit and were thus statistically redundant. The unstandardized beta weights resulting from this regression equation will be applied to the vector of independent variables among non-renting households to estimate their annual rent equivalent.  Average daily per capita expenditures The final consumption expenditures aggregate will be expressed as average daily per capita consumption expenditures in constant 2010 US dollars at 2005 PPP adjusted to 2010 US prices. The steps to convert daily per capita consumptions expenditure data collected in the country’s local currency units (LCU) to constant 2010 US$ (2005 PPP adjusted to 2010 US prices) will be: 1) Convert LCU at the time of the survey (June 2018) to LCU at 2005 prices by dividing by the ratio of the CPI of the survey month to the average annual CPI in 2005.15 1) Convert 2005 LCU to 2005 US$ by dividing by the 2005 PPP for private consumption conversion rate of 4.54. 16 2) Convert US$ in 2005 prices to US$ in 2010 prices by multiplying by 111.65, which is the ratio of the US CPI in 2010 to the US CPI in 2005.17 Note that average daily per capita consumption expenditures is expressed in US$ in 2010 prices in order to enable comparisons with other countries-so a common standard is essential.  Prevalence of Poverty The prevalence of poverty, or poverty headcount ratio, is the proportion of the population in the survey area living in extreme poverty, defined as having average daily consumption expenditures of less than US$1.25 per capita, converted into LCU at 2005 Purchasing Power Parity (PPP) exchange rates. Poverty lines will be calculated to estimate the proportion of the population living in extreme poverty, defined as having average daily consumption expenditures of less than US$1.25 per day, converted into LCU (i.e. Guatemalan quetzals) at 2005 Purchasing Power Parity (PPP) exchange rates. This is done in two steps: 1) The $1.25 line will be converted into LCU, using the 2005 PPP exchange rate for private consumption for Guatemala18. 15 CPI of the Guatemalan quetzale in the survey month (June 2018) is not available at the present time. Average annual CPI of the Guatemalan quetzale in 2005 is 74.57. Available at: http://data.imf.org/regular.aspx?key=61545861. 16 2005 PPP conversion factor, private consumption, 2005 International Comparison Program. Available at: https://data.worldbank.org/indicator/PA.NUS.PRVT.PP.05?locations=GT . 17 Annual average CPI of USD in 2005 is 195.30 (http://www.bls.gov/cpi/cpid05av.pdf), Annual average CPI of USD in 2011 is 218.06 (http://www.bls.gov/cpi/cpid10av.pdf). The ratio of the US CPI in 2010 to the US CPI in 2005 = 218.06/195.30 = 1.1165. 18 Average annual CPI of the Guatemalan quetzale in 2005 is 74.57. Available at: http://data.imf.org/regular.aspx?key=61545861. Guatemala PBS Data Treatment and Analysis Plan APPENDIX C – Overview of PPI Analyses Guatemala Endline PBS Data Treatment and Analysis Plan 22 2) The resulting figure will be adjusted for cumulative price inflation since 2005. The adjustment will be done using the consumer price index (CPI) for June 2018, the month in which the survey was fielded as the numerator, and the average annual CPI for 2005 as the base factor.19 Note that the poverty line is converted to LCUs to enable a computation of prevalence of poverty using per capita consumption expenditures figures in LCUs, given that the currency units must be standardized in the computation. It is also possible to compute the prevalence of poverty by using the $1.25 poverty line and converting the per capita consumptions expenditure figures from 2018 LCU to 2005 US dollars instead. This is because the prevalence of poverty figures that are reported do not explicitly state which currency underpinned the calculation.  Mean depth of poverty This indicator is useful to understand the average, over all individuals in the survey, of the gaps between their living standards and the poverty line. It indicates the extent to which individuals fall below the poverty line (if they do). Mean depth of poverty is sometimes also called the poverty gap index (PGI). The PGI is computed as the average of the differences between an individual’s total daily per capita consumption expenditures and the poverty line, divided by the poverty line, with individuals over the poverty line having a PGI = 0. The PGI is given by the formula: PGI = ( 1 𝑁 ∑ ( 𝑧− 𝑦𝑖 𝑧 ) 𝑞 𝑖=1 ) × 100 Where N is the total number of individuals in the population, z is the poverty line and yi is the daily per capita consumption expenditures of individual i. For individuals above the poverty line, set yi = zso so that contribution to PGI is 0 for those individuals. 19 CPI of the Guatemalan quetzal in the survey month (June 2018) is not available at the present time. See: http://data.imf.org/regular.aspx?key=61545861. Guatemala PBS Data Treatment and Analysis Plan APPENDIX C – Overview of PPI Analyses Guatemala Endline PBS Data Treatment and Analysis Plan 23 Prevalence of Poverty i) Prevalence of Poverty Scorecard: Innovations for Poverty Action (IPA) has developed household’s PPI score using INSTITUTO NACIONAL DE ESTADÍSTICA -INE -ENCUESTA NACIONAL DE CONDICIONES DE VIDA (ENCOVI 2014), which is then used with a look-up table to estimate the likelihood that the household is poor. This scorecard is calibrated to the 3.20 US PPP 2011 per person per day poverty line. Each question provides a reference to the question number in the ENCOVI 2014 instrument. ii) Prevalence of Poverty Look-up Table: IPA has also developed a look-up table to be used in conjunction with the prevalence of poverty scorecard. A household’s cumulative score is mapped to a corresponding probability that a household is below the 3.20 US PPP poverty line. Depth of Poverty i) Depth of Poverty Scorecard: Using the ENCOVI 2014, IPA has developed a household’s Poverty Depth score, which is then used with a look-up table to estimate the depth of poverty, which is also termed the poverty gap (the percentage by which the household’s consumption is below a poverty line, with households at or above the line being assigned a gap of zero). This scorecard is calibrated to the 3.20 US PPP 2011 per person per day poverty line. For each question, a reference to the question number in the ENCOVI 2014 instrument is provided. The interview guidance provided in that instrument should be used when asking those questions. ii) Depth of Poverty Look-up Table: IPA has also developed the depth of poverty look-up table which should be used in conjunction with the depth of poverty scorecard. A household’s cumulative score is mapped to a predicted poverty depth with respect to the 3.20 US PPP poverty line. Approach to Test the Accuracy of the PPI based Poverty Indicators i) When the 2018 data are available, ICF through TANGO will provide IPA with a Stata data file that contains just the responses to the relevant PPI questions from the PPI Module along with (anonymized) household identifiers. ii) The IPA team will return: a) A Stata do file (and related documents) demonstrating how the PPI scorecards and look up tables can be used to calculate the probability that a household is poor (or a household’s poverty depth). ICF will review the code, scorecard, and the look up tables prepared by IPA. b) IPA will also develop and share a separate Stata file with poverty predictions (along with household identifiers) based on an alternative non-linear approach to estimate poverty (that involves interactions between the responses), that uses responses to the questions previously requested by the PPI, and made available to PPI for the purpose of computing a). iii) Subsequently, ICF will calculate accuracy measure for each poverty category. To this end, ICF will use the mean predicted poverty headcount (or mean predicted poverty depth) and compare it to the actual poverty rate (or actual mean poverty depth). Annex 3: Evaluation Matrix Annex 3: Evaluation Matrix Evaluation Questions Sub- Questions Data Sources Collection Methods Data Analysis Methods Q1: To what extent have the projects met their defined goals, purposes, and outcomes? 1.1. Were targets set by the project and key FFP indicators met? Did program indicators improve over time? Qualitative: - SEGAMIL staff (all partners) - USAID Guatemala FFP staff - SEGAMIL documentation (e.g., DFAP proposals, annual and quarterly reports) - Mid-term Evaluation (MTE) Report Quantitative: - SEGAMIL PBS baseline/endline data - Indicator Performance Tracking Table (IPTT) data Qualitative: - Desk review - KIIs using semi￾structured instruments specific to given respondent category - FGDs using semi￾structured instruments specific to given respondent category Quantitative: - Baseline and endline surveys - Program monitoring processes Qualitative: - Content analysis of stakeholders responses (KIIs) to assess their views on the extent to which activities have been achieved - Content analysis of SEGAMIL annual reports that describes achievements to date and factors related to performance, taking into account programmatic shifts at mid-term - Content analysis of MTE describing findings and conclusions on achievements to date and factors related to progress to date Quantitative: - Statistical testing of baseline and endline indicators, test differences in endline indicators across direct and indirect beneficiaries - Comparison of targets versus actual results for PBS indicators and IPTT indicators 1.2. What factors promoted or inhibited the achievement of the project objectives, including but not limited to the effectiveness of sustainable agriculture, income diversification, NRM and behavior change communication interventions? Qualitative: - Program participants and non-participants - SEGAMIL staff (all partners) - USAID Guatemala FFP staff - SEGAMIL documentation (e.g., DFAP proposals, annual and quarterly reports) - MTE Report - Other reports Qualitative: - FGDs, KIIs, and in-depth interviews (IDIs) - Document review Qualitative: - Content analysis of participant and stakeholder responses to assess their views on the extent to which factors promoted or inhibited project activities and outcomes - Content analysis and information extraction from documents reviewed Evaluation Questions Sub- Questions Data Sources Collection Methods Data Analysis Methods 1.3. How did the targeting strategies contribute to the achievement of goals (especially in regard to gender and reaching the most vulnerable)? Qualitative: - Program participants and non-participants - SEGAMIL staff (all partners) - USAID Guatemala FFP staff - SEGAMIL documentation (e.g., DFAP proposals, annual and quarterly reports) - MTE Report Quantitative: - SEGAMIL PBS baseline/endline data - IPTT data Qualitative: - Desk review - KIIs using semi￾structured instruments specific to given respondent category - FGDs using semi￾structured instruments specific to given respondent category Quantitative: - Baseline and endline surveys - Program monitoring processes Qualitative: - Content analysis of beneficiary responses (FGDs) and stakeholders responses (KIIs) to assess their views on the extent to which activities have been achieved and who benefited most - Content analysis of SEGAMIL annual reports that describes achievements to date and factors related to targeting and performance, taking into account programmatic shifts at mid-term - Content analysis of MTE describing findings and conclusions on achievements to date and factors related to progress to date Quantitative: - Statistical testing of baseline and endline indicators, test differences in endline indicators across direct and indirect beneficiaries; further disaggregation across subgroups and by level of poverty - Comparison of targets versus actual results for PBS indicators and IPTT indicators 1.4. What has been the contribution of SEGAMIL to USAID’s efforts to reduce food insecurity among households in the Western Highlands? Qualitative: - Program participants and non-participants - SEGAMIL staff (all partners) - Community and municipal organizations - USAID Guatemala FFP staff Quantitative: - SEGAMIL PBS baseline/endline data - IPTT data Qualitative: - KIIs - FGDs - IDIs Quantitative: - Baseline and endline surveys - Program monitoring processes Qualitative: - Content analysis of FGDs, KIIs, and IDIs - Evidence of immediate and lasting impact of project activities Quantitative: - Examination of overall improvement in indicators, number of households reached by the program Evaluation Questions Sub- Questions Data Sources Collection Methods Data Analysis Methods Q2. For each technical sector, what are the strengths of and challenges in the project’s design, including the results framework, that influence the effectiveness of the interventions? 2.1. What is the relevance of the technical interventions to achieve project outcomes, especially to achieving food security and improved nutritional status for mothers and children under two? Qualitative: - Program participants and non-participants - SEGAMIL technical staff - Community promoters - USAID Guatemala FFP staff - SEGAMIL documentation (e.g., DFAP proposals, annual and quarterly reports) - MTE Report - Other reports Quantitative: - SEGAMIL PBS baseline/endline data Qualitative: - FGDs, KIIs, and IDIs - Document review Quantitative: - Baseline and endline surveys Qualitative: - Content analysis and information extraction from documents reviewed Quantitative: - Examination of overall improvement in indicators, that describe specific agriculture, nutrition, WASH and health practices - Correlation between practices and participation in different interventions 2.2. What is the plausibility and the determinants of achieving the key outcomes, highlighting any unforeseen pathways leading to unintended positive and negative consequences of the projects? Qualitative: - Program participants - SEGAMIL technical staff (all partners) - USAID Guatemala FFP staff - SEGAMIL documentation - Contextual literature Quantitative: - SEGAMIL PBS baseline/endline data - IPTT data Qualitative: - Desk review - FGDs, KIIs, and IDIs Qualitative: - Review of SEGAMIL’s results framework - Contextual literature on the causes of food insecurity and malnutrition in the Guatemala Highlands - Content analysis of FGDs, KIIs, IDIs, and documents to identify facilitators and barriers to desired outcomes and practices promoted by SEGAMIL Evaluation Questions Sub- Questions Data Sources Collection Methods Data Analysis Methods Q3: In each technical sector, what are the strengths of and challenges of the effectiveness of the interventions’ implementation? 3.1. What was the effectiveness and relevance of the approach and methods of the technical interventions to achieve project outputs of greater or lesser quality, and discuss those findings in relation to the projects’ results framework? Qualitative: - Program participants - SEGAMIL staff - Community promoters - USAID FFP staff - Community and municipal government organizations - SEGAMIL documentation (e.g., DFAP proposals, annual and quarterly reports) - Mid-Term Evaluation Reports Quantitative: - SEGAMIL PBS baseline/endline data - IPTT baseline/endline data Qualitative: - Desk review - KIIs and FGDs Quantitative: Baseline and endline surveys Qualitative: - Content analysis of participant responses and stakeholder responses to assess their views on effectiveness of projects' methods and approaches across the multiple technical sectors, as well as for specific project activities (e.g., quality of trainings) - Content analysis of SEGAMIL documents to understand changes in implementation approaches and relative success of each Quantitative: - Statistical analysis and comparison of data from PBS baseline/endline indicators, and analysis of IPTT baseline/endline indicators to assess the relative effectiveness of approaches selected 3.2. What are the factors in the implementation approach and methods, and the context associated with greater or lesser effectiveness in producing outputs of higher or lower quality? Qualitative: - SEGAMIL documents - SEGAMIL special studies reports - SEGAMIL staff - Community promoters - Community and municipal organizations - USAID Mission staff Qualitative: - Desk review - KIIs and FGDs Qualitative: - Content analysis of relevant KIIs (e.g., SEGAMIL staff, municipal government staff, community organizations, USAID FFP staff, community promoters) to assess their views on strengths and challenges associated with SEGAMIL interventions. This includes program management, partnerships, how information management system was used for decision-making, planning, coordination and communication mechanisms used by project. - Content analysis of annual reports to identify strengths and challenges and steps SEGAMIL took to address challenges - Review of MTE Reports that identify strengths, challenges, and weaknesses of project implementation approach, management, communication, collaboration. Evaluation Questions Sub- Questions Data Sources Collection Methods Data Analysis Methods Compare with subsequent annual reports to determine whether these strengths are still evident and what steps SEGAMIL has taken to address challenges. - Content analysis of KII with ECADIs, promoters, and community organizations to understand implementation challenges at those levels 3.3. What were the interventions and implementation processes deemed more or less acceptable to members of the target communities? Qualitative: - Program participants and non-participants - Community promoters Quantitative: - PBS baseline/endline data - IPTT baseline/endline data Qualitative: - FGDs, KIIs, and IDIs Quantitative: - Baseline and endline surveys Qualitative: - Content analysis of FGDs and IDIs with project participants, non-participants, and community promoters to identify interventions and activities that were most valued and appreciated and those that were perceived as more problematic Quantitative: - Statistical analysis and comparison of data from PBS baseline/endline indicators to identify which practices were more likely to be adopted by beneficiaries Q4: Which project outcomes are likely to be sustained? 4.1.The quality and functionality of systems, processes, and institutional arrangements established and strengthened independently by the projects, as well as in collaboration with the private sector, Government of Guatemala, community organizations, and research organizations to achieve project outcomes and sustainability Qualitative: - SEGAMIL project documentation - SEGAMIL staff - USAID FFP staff - USAID monitoring reports focused on sustainability - Government Ministries (health, social development, agriculture) - Municipal government staff - Community organizations Qualitative: - Desk review - KIIs and FGDs Qualitative: - Content analysis of implementing partner (IP) project documents and relevant KIIs (especially with leadership of community organizations strengthened by SEGAMIL) and with municipal government to assess the processes, systems, and linkages in place, and their level of importance in sustaining the projects - Content analysis of IP Sustainability Plans and Exit Strategies -Review of MTE Reports and review of subsequent IP Annual Reports to determine whether IPs followed up on findings, Evaluation Questions Sub- Questions Data Sources Collection Methods Data Analysis Methods conclusions, and recommendations from MTE to ensure sustainability of activities and outcomes 4.2. Will the necessary resources and capacity strengthening exist to sustain service providers, and the likelihood that service provision will continue after the project ends? Qualitative: - Program participants - SEGAMIL documentation - Municipal government staff - Community organizations - Community promoters - Review of municipal plans, household health plans, and community organization plans developed with support from SEGAMIL Qualitative: - Desk review - KIIs, FGDs, and IDIs Qualitative: - Content analysis of FGDs, KIIs, and IDIs to determine whether the conditions exist to sustain the outcomes. These include motivation, access to inputs and resources, necessary skills. and access to additional technical support if and when needed. We will examine this from the perspective of the beneficiaries to sustain their practices, the community promoters to continue providing services, for the ECADIs and SIL to continue their activities and grow their membership, and for the COCODEs and COCOSANs to continue to identify and respond to community needs related to food security. KII with municipalities will help inform whether they are able and motivated to support the community systems strengthened by SEGAMIL. 4.3. Communities’ perceptions on the quality, frequency, and effectiveness of the implementation processes and services provided Qualitative: - Program participants and non-participants - Community promoters - Leaders of community organizations Qualitative: - FGDs, KII Qualitative: - Content analysis of FGDs and KIIs to identify interventions and activities that were most valued and appreciated and those that were perceived as more problematic 4.4. What is the motivation of the community and beneficiaries to demand and pay (or invest time) for the services, and communities’ perceptions on sustainability of the services provided by the project? Qualitative: - Program participants and non-participants - SEGAMIL promoters’ - SEGAMIL documents - Leaders of community organizations Qualitative: - Desk review - KIIs, FGDs, and IDIs Qualitative: - Content analysis of FGDs with beneficiaries to assess motivation to invest (money or time) in providing and/or purchasing services, as well as their perception of the value of activities - Content analysis of relevant KIIs Evaluation Questions Sub- Questions Data Sources Collection Methods Data Analysis Methods (e.g., COCODEs, ECADI) to assess their motivation to continue services and fund them -Review of project documents to identify indications that beneficiaries are already investing time or money into project activities (e.g., cost-share, volunteer leader/ promoters, resumption of discontinued activity) 4.5. The extent to which the projects leveraged other U.S. Government and non￾U.S. Government investments to achieve sustained outcomes as identified in the theory of change 4.6. Evidence of enhanced linkages with other service providers Qualitative: - SEGAMIL documents - SEGAMIL staff - USAID staff - Government representatives at municipal and national levels Qualitative: - Desk review - KIIs Qualitative: - Review documents for evidence of linkages - Content analysis of KIIs to assess the nature and strength of linkages and coordination across programs and agencies Q5. What are the key lessons learned and best practices that should inform future projects in the country? 5.1. What are the lessons, best practices, and approaches that should be considered in designing and implementing future food and nutrition security projects? 5.2. What are the lessons learned with regard to achieving sustainable outcomes in food security programs? 5.3. Based on the lessons learned, how can potential unintended negative consequences be minimized and positive consequences be captured systematically? Qualitative: - Findings and conclusions from evaluation questions 1–4 - SEGAMIL staff - USAID FFP staff Qualitative: - Review of findings from other evaluation questions - Desk review - KIIs Qualitative: - Review of findings from previous evaluation questions and synthesis of main lessons from each - Content analysis of relevant KIIs to assess their views on lessons learned and best practices for future design of FFP programs Annex 4: SEGAMIL Impact Indicators 2013 Baseline 2018 Endline Raw Difference (Endline - Baseline) Significance Level1 FOOD SECURITY INDICATORS Average Household Dietary Diversity Score (HDDS) 6.2 6.8 0.6 *** 2,994 1,332 Prevalence of households with moderate or severe hunger (HHS) 7.3 10.7 3.4 ns 3,072 1,440 Male and female adults 7.2 10.0 2.8 ns 2,723 1,264 Adult female, no adult male 9.2 15.2 6.0 ns 305 164 Adult male, no adult female NA NA NA NA 42 12 Child, no adults NA NA NA NA 2 0 POVERTY INDICATORS Per capita expenditures (as a proxy for income) of USG-assisted areas $1.94 $2.38 $0.45 *** 19,905 9,100 Male and female adults $1.89 $2.36 $0.46 *** 18,537 8,416 Adult female, no adult male $2.51 $2.74 $0.23 ns 1,274 648 Adult male, no adult female NA NA NA NA 91 35 Child, no adults NA NA NA NA 3 1 Prevalence of poverty: Percent of people living on less than $1.25/day2 43.8 19.2 -24.6 *** 19,905 9,100 Male and female adults 44.8 20.0 -24.9 *** 18,537 8,416 Adult female, no adult male 29.6 8.3 -21.3 *** 1,274 648 Adult male, no adult female NA NA NA NA 91 35 Child, no adults NA NA NA NA 3 1 Mean depth of poverty (expressed as percent of poverty line) 11.9 3.5 -8.4 *** 19,905 9,100 Male and female adults 12.2 3.5 -8.7 *** 18,537 8,416 Adult female, no adult male 8.4 2.0 -6.4 ** 1,274 648 Adult male, no adult female NA NA NA NA 91 35 Child, no adults NA NA NA NA 3 1 WASH INDICATORS Percentage of households using an improved source of drinking water 18.5 49.8 31.3 *** 3,073 1,441 Percentage of households using improved sanitation facilities 51.4 66.7 15.3 *** 3,068 1,441 Percentage of households with soap and water at a handwashing station commonly used by family members 73.1 76.9 3.7 ns 2,964 1,441 AGRICULTURAL INDICATORS Percentage of farmers who used financial services in the past 12 months 12.6 12.2 -0.4 ns 2,942 1,381 Percentage of farmers who practiced value chain activities promoted by the project in the past 12 months 11.5 23.9 12.4 *** 2,951 1,381 Percentage of farmers who used three sustainable agricultural practices in the past 12 months 50.1 63.8 13.7 *** 2,973 1,381 Percentage of farmers who used at least two sustainable crop practices in the past 12 months 55.7 63.6 7.9 * 5,546 1,381 Percentage of farmers who used at least two sustainable livestock practices in the past 12 months 10.2 23.5 13.3 *** 5,546 1,381 Percentage of farmers who used at least two sustainable NRM practices in the past 12 months 1.3 1.8 0.5 ns 5,546 1,381 Percentage of farmers who used improved storage practices in the past 12 months 8.1 15.9 1.6 ** 2,967 1,381 WOMEN'S HEALTH AND NUTRITION INDICATORS Prevalence of underweight women 2.1 1.2 -0.9 † 2,402 1,871 Women’s Dietary Diversity Score (WDDS) 4.0 4.5 0.6 *** 2,771 2,092 CHILDREN'S HEALTH AND NUTRITION INDICATORS Prevalence of underweight children under 5 years of age (Total) 34.5 25.3 -9.2 ** 2,978 1,337 Male 37.6 25.7 -11.8 *** 1,491 699 Female 31.4 24.9 -6.5 † 1,487 638 Prevalence of stunted children under 5 years of age (Total) 79.6 72.2 -7.5 ** 2,978 1,333 Table A4.1 Comparison of Baseline and Endline Indicators - CRS SEGAMIL Project Area [Guatemala 2013, 2018] Number of Observations Baseline Endline 2013 Baseline 2018 Endline Raw Difference (Endline - Baseline) Significance Level1 Table A4.1 Comparison of Baseline and Endline Indicators - CRS SEGAMIL Project Area [Guatemala 2013, 2018] Number of Observations Baseline Endline Male 80.0 73.1 -6.8 ** 1,491 696 Female 79.3 71.0 -8.3 * 1,487 637 Prevalence of wasted children under 5 years of age (Total) 2.0 0.9 -1.1 ** 2,978 1,336 Male 3.0 0.7 -2.3 *** 1,491 698 Female 0.9 1.1 0.2 ns 1,487 638 Percentage of children under age 5 with diarrhea in the last two weeks (Total) 42.6 28.2 -14.4 *** 3,042 1,354 Male 44.1 30.4 -13.7 *** 1,533 709 Female 41.1 25.8 -15.3 *** 1,509 645 Percentage of children under age 5 with diarrhea treated with ORT (Total) 49.3 51.3 2.0 ns 1,240 404 Male 47.7 45.9 -1.8 ns 649 224 Female 51.0 58.6 7.7 † 591 180 Prevalence of exclusive breast-feeding of children under six months of age 61.1 63.7 2.6 ns 340 156 Male 61.6 70.2 8.7 ns 169 80 Female 60.7 56.8 -3.9 ns 171 76 Prevalence of children 6-23 months of age receiving a minimum acceptable diet (MAD) 17.9 40.1 22.2 *** 895 428 Male 17.4 35.2 17.9 *** 473 210 Female 18.5 44.9 26.4 *** 422 218 CUSTOM INDICATORS Percentage of women with a child under 5 years who recognize two or more pregnancy danger signs 44.7 73.2 28.5 *** 1,825 828 Percentage of women with a child under 5 years who recognize two or more neonatal danger signs 32.0 59.8 27.7 *** 1,826 828 Percentage of women with a child under 5 years who recognize two or more childhood illness signs 39.7 58.9 19.2 *** 1,825 828 Percentage of women with a child under 5 years seeking health care when pregnancy danger signs are present 66.9 71.4 4.5 ns 374 207 Percentage of women with a child under 5 years seeking health care when neonatal danger signs are present 75.7 71.0 -4.6 ns 333 123 Percentage of women with a child under 5 years seeking health care when childhood illness signs are present 84.2 84.0 -0.2 ns 839 293 Percentage of women with a child under 5 years making decisions about health care for themselves 63.7 61.1 -2.6 ns 1,613 718 Percentage of women with a child under 5 years making decisions about health care for their child 96.5 98.2 1.7 ns 1,614 718 1 ns = not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 2 Expressed in constant 2010 USD NA : Not available Lower Upper Average Household Dietary Diversity Score (HDDS) 6.2 6.0 6.4 2,994 33,846 0.1 2.8 Prevalence of households with moderate or severe hunger (HHS) 7.3 5.4 9.8 3,072 33,846 1.1 2.4 Adult Female no Adult Male 9.2 5.6 14.7 305 2,943 2.3 1.4 Adult Male no Adult Female 2.6 0.6 9.9 42 472 1.8 0.7 Male and Female Adults 7.2 5.3 9.9 2,723 30,405 1.2 2.3 Child No Adults * 0.0 0.0 0.0 2 26 0.0 0.0 Percentage of households using an improved drinking water source 18.5 14.3 23.6 3,073 33,846 2.3 3.4 Percentage of households using improved sanitation facilities 51.4 46.7 56.1 3,068 33,846 2.4 2.7 Percentage of households with soap and water at a handwashing station commonly used by family members 73.1 68.7 77.1 2,964 33,846 2.2 2.9 Prevalence of poverty: Percent of people living on less than $1.25/day 43.8 39.0 48.7 19,905 220,308 2.5 2.8 Adult Female no Adult Male 29.6 21.1 39.8 1,274 12,996 4.8 1.9 Adult Male no Adult Female 16.4 4.9 42.9 91 1,019 9.1 1.5 Male and Female Adults 44.8 40.0 49.8 18,537 206,248 2.5 2.7 Child No Adults * 0.0 0.0 0.0 3 45 0.0 0.0 Mean depth of poverty 11.9 10.3 13.6 19,905 220,308 0.8 2.6 Adult Female no Adult Male 8.4 5.0 11.7 1,274 12,996 1.7 2.0 Adult Male no Adult Female 3.1 0.0 7.1 91 1,019 2.0 1.6 Male and Female Adults 12.2 10.6 13.9 18,537 206,248 0.8 2.5 Child No Adults * 0.0 0.0 0.0 3 45 0.0 0.0 Per capita expenditures (as a proxy for income) of USG targeted beneficiaries 1.9 1.8 2.1 19,905 220,308 0.1 2.8 Adult Female no Adult Male 2.5 2.2 2.8 1,274 12,996 0.1 1.4 Adult Male no Adult Female 3.3 2.5 4.1 91 1,019 0.4 0.9 Male and Female Adults 1.9 1.8 2.0 18,537 206,248 0.1 2.7 Child No Adults * 5.7 5.1 6.3 3 45 0.1 0.6 Percentage of farmers who used financial services in the past 12 months 12.6 9.9 15.9 2,942 61,257 1.5 2.5 Percentage of farmers who practiced value chain activities promoted by the project in the past 12 months 11.5 9.4 14.1 2,951 61,257 1.2 2.0 Percentage of farmers who used two sustainable agricultural practices in the past 12 months 90.7 87.6 93.0 2,972 61,257 1.4 2.6 Percentage of farmers who used improved storage practices in the past 12 months 8.1 6.2 10.6 2,967 61,257 1.1 2.2 Prevalence of underweight women 2.1 1.4 3.2 2,402 42,208 0.5 1.5 Women’s Dietary Diversity Score 4.0 3.8 4.2 2,771 48,937 0.1 3.2 Prevalence of underweight children under 5 years of age (Total) 34.5 30.6 38.6 2,978 32,644 2.0 2.3 Prevalence of underweight children under 5 years of age (Male) 37.6 32.5 42.9 1,491 16,286 2.6 2.1 Table A4.2. Title II Baseline Indicators - SEGAMIL Indicators, 95% Confidence Intervals and Base Population [Guatemala, 2013] Indicator Value 95% C.I. Weighted Population Number of Records CHILDREN'S HEALTH AND NUTRITION INDICATORS Standard Error DEFT HOUSEHOLD LEVEL INDICATORS AGRICULTURAL INDICATORS* WOMEN'S HEALTH AND NUTRITION INDICATORS Prevalence of underweight children under 5 years of age (Female) 31.4 27.4 35.8 1,487 16,358 2.1 1.8 Prevalence of stunted children under 5 years of age (Total) 79.6 76.2 82.7 2,978 32,644 1.6 2.2 Prevalence of stunted children under 5 years of age (Male) 80.0 76.2 83.3 1,491 16,286 1.8 1.8 Prevalence of stunted children under 5 years of age (Female) 79.3 75.3 82.8 1,487 16,358 1.9 1.8 Prevalence of wasted children under 5 years of age (Total) 2.0 1.5 2.7 2,978 32,644 0.3 1.2 Prevalence of wasted children under 5 years of age (Male) 3.0 2.0 4.5 1,491 16,286 0.6 1.4 Prevalence of wasted children under 5 years of age (Female) 0.9 0.5 1.7 1,487 16,358 0.3 1.2 Percentage of children under age 5 with diarrhea in the last two weeks (Total) 42.6 38.7 46.6 3,042 33,385 2.0 2.3 Percentage of children under age 5 with diarrhea in the last two weeks (Male) 44.1 39.6 48.7 1,533 16,785 2.3 1.8 Percentage of children under age 5 with diarrhea in the last two weeks (Female) 41.1 36.7 45.6 1,509 16,600 2.2 1.8 Percentage of children under age 5 with diarrhea treated with ORT (Total) 49.3 43.8 54.8 1,240 14,204 2.8 2.0 Percentage of children under age 5 with diarrhea treated with ORT (Male) 47.7 40.8 54.7 649 7,398 3.5 1.8 Percentage of children under age 5 with diarrhea treated with ORT (Female) 51.0 44.7 57.3 591 6,806 3.2 1.6 Prevalence of exclusive breast-feeding of children under six months of age 61.1 55.0 66.9 340 3,739 3.0 1.2 Prevalence of exclusive breast-feeding of children under six months of age (Male) 61.6 52.2 70.1 169 1,807 4.6 1.2 Prevalence of exclusive breast-feeding of children under six months of age (Female) 60.7 53.6 67.4 171 1,931 3.5 0.9 Prevalence of children 6-23 months of age receiving a minimum acceptable diet (MAD) 17.9 13.8 22.8 895 10,030 2.3 1.8 Prevalence of children 6-23 months of age receiving a minimum acceptable diet (MAD) (Male) 17.4 12.9 23.0 473 5,365 2.5 1.5 Prevalence of children 6-23 months of age receiving a minimum acceptable diet (MAD) (Female) 18.5 13.5 24.7 422 4,665 2.8 1.5 Note: Prevalence of wasted children is included but is not a required FFP indicator *Although FFP requires that agricultural indicators are disaggregated by sex, they are not included due to the non-random selection of only one farmer in each household. Table A4.2 (cont). Title II Program-specific Indicators - SEGAMIL Indicators, 95% Confidence Intervals and Base Population [Guatemala, 2013] Lower Upper Awareness and Health Care Seeking (Mothers of children 0-59 months) Percentage who recognize 2 or more pregnancy danger signs 44.7 40.2 49.3 1,825 22,759 2.3 2.0 Percentage who recognize 2 or more neonatal danger signs 32.0 27.6 36.8 1,826 22,759 2.3 2.2 Percentage who recognize 2 or more childhood illness signs 39.7 35.3 44.2 1,825 22,759 2.2 2.0 Percentage seeking health care when pregnancy danger signs are present 66.9 59.9 73.2 374 4,854 3.4 1.4 Percentage seeking health care when neonatal danger signs are present 75.7 70.3 80.3 333 4,099 2.5 1.1 Percentage seeking health care when childhood illness signs are present 84.2 80.2 87.5 839 10,389 1.8 1.5 Decision-Making for Seeking Health Care (Mothers of children 0-59 months - married or in a union) Percentage making decisions about health care for themselves1 63.7 59.2 67.9 1,613 19,923 2.2 1.9 Percentage making decisions about health care for children 0-59 months1 96.5 94.2 97.9 1,614 19,923 0.9 2.0 Standard Error DEFT 1 Includes joint decision making. Indicator Value 95% CI Weighted Population Number of Records Lower Upper FOOD SECURITY INDICATORS Average Household Dietary Diversity Score (HDDS) 6.8 6.6 7.0 1,332 45,019 1.9 0.10 2.0 Prevalence of households with moderate or severe hunger (HHS) 10.7 5.7 15.8 1,440 48,935 30.9 2.48 3.0 Male and female adults 10.0 5.5 14.6 1,264 42,749 30.1 2.24 2.6 Adult female, no adult male 15.2 5.8 24.7 164 5,875 35.0 4.64 1.7 Adult male, no adult female NA NA NA 12 311 NA NA NA Child, no adults NA NA NA 0 0 NA NA NA Prevalence of moderate and severe food insecurity in the population, based on the Food Insecurity Experience Scale (FIES) [30 day recall] 34.4 26.5 42.3 1,440 48,935 39.3 4.03 4.0 Male and female adults 32.1 24.9 39.3 1,264 42,749 38.2 3.66 3.5 Adult female, no adult male 50.8 36.3 65.3 164 5,875 42.8 7.41 2.2 Adult male, no adult female NA NA NA 12 311 NA NA NA Child, no adults NA NA NA 0 0 NA NA NA POVERTY INDICATORS Per capita expenditures (as a proxy for income) of USG-assisted areas $2.38 $2.22 $2.55 9,100 311,489 $1.32 0.08 2.3 Male and female adults $2.36 $2.20 $2.52 8,416 288,129 $1.27 0.08 2.2 Adult female, no adult male $2.74 $2.41 $3.07 648 22,409 $1.78 0.16 1.2 Adult male, no adult female NA NA NA 35 846 NA NA NA Child, no adults NA NA NA 1 106 NA NA NA Prevalence of poverty: Percent of people living on less than $1.25/day 19.2 14.9 23.5 9,100 311,489 39.4 2.12 2.1 Male and female adults 20.0 15.3 24.6 8,416 288,129 38.8 2.28 2.1 Adult female, no adult male 8.3 2.1 14.6 648 22,409 34.8 3.08 1.1 Adult male, no adult female NA NA NA 35 846 NA NA NA Child, no adults NA NA NA 1 106 NA NA NA Mean depth of poverty 3.5 2.3 4.7 9,100 311,489 10.6 0.57 2.0 Male and female adults 3.5 2.3 4.7 8,416 288,129 10.1 0.60 2.1 Adult female, no adult male 2.0 0.0 3.9 648 22,409 10.9 0.95 1.1 Adult male, no adult female NA NA NA 35 846 NA NA NA Child, no adults NA NA NA 1 106 NA NA NA WASH INDICATORS Percentage of households using an improved source of drinking water 49.8 42.6 57.0 1,441 48,965 50.0 3.54 2.7 Percentage of households using improved sanitation facilities 66.7 62.3 71.2 1,441 48,965 47.1 2.17 1.7 Percentage of households with soap and water at a handwashing station commonly used by family members 76.9 71.0 82.7 1,441 48,965 42.2 2.88 2.6 AGRICULTURAL INDICATORS Percentage of farmers who used financial services in the past 12 months 12.2 8.8 15.6 1,381 81,707 32.7 1.68 1.9 Percentage of farmers who practiced value chain activities promoted by the project in the past 12 months 23.9 18.8 29.1 1,381 81,707 42.7 2.54 2.2 Percentage of farmers who used three sustainable agricultural practices in the past 12 months 63.8 59.9 67.6 1,381 81,707 48.1 1.89 1.5 Percentage of farmers who used at least two sustainable crop practices in the past 12 months 63.6 59.2 68.0 1,381 81,707 48.1 2.18 1.7 Percentage of farmers who used at least two sustainable livestock practices in the past 12 months 23.5 18.7 28.2 1,381 81,707 42.4 2.35 2.1 Percentage of farmers who used at least two sustainable NRM practices in the past 12 months 1.8 0.9 2.6 1,381 81,707 13.2 0.41 1.2 Percentage of farmers who used improved storage practices in the past 12 months 15.9 11.6 20.2 1,381 81,707 36.6 2.12 2.1 WOMEN'S HEALTH AND NUTRITION INDICATORS Prevalence of underweight women 1.2 0.7 1.7 1,871 66,467 10.8 0.25 1.0 Women’s Dietary Diversity Score (WDDS) 4.5 4.3 4.8 2,092 76,480 1.6 0.10 3.0 CHILDREN'S HEALTH AND NUTRITION INDICATORS Prevalence of underweight children under 5 years of age (Total) 25.3 20.8 29.9 1,337 45,169 43.5 2.24 1.9 Table A4.3. FFP Guatemala Endline Indicators - SEGAMIL Project Area Indicators, 95% Confidence Intervals and Base Population [Guatemala, 2018] Indicator Value Confidence Interval Number of Records Weighted Population Standard Deviation Standard Error DEFT Lower Upper Table A4.3. FFP Guatemala Endline Indicators - SEGAMIL Project Area Indicators, 95% Confidence Intervals and Base Population [Guatemala, 2018] Indicator Value Confidence Interval Number of Records Weighted Population Standard Deviation Standard Error DEFT Male 25.7 21.5 29.9 699 23,975 43.4 2.08 1.3 Female 24.9 18.6 31.2 638 21,194 43.7 3.11 1.8 Prevalence of stunted children under 5 years of age (Total) 72.2 68.3 76.1 1,333 45,149 44.8 1.92 1.6 Male 73.1 69.2 77.1 696 23,970 43.9 1.94 1.2 Female 71.0 65.7 76.3 637 21,179 45.8 2.61 1.4 Prevalence of wasted children under 5 years of age (Total) 0.9 0.4 1.4 1,336 45,151 9.4 0.27 1.0 Male 0.7 0.0 1.3 698 23,957 8.1 0.32 1.0 Female 1.1 0.2 2.1 638 21,194 10.8 0.48 1.1 Percentage of children under age 5 with diarrhea in the last two weeks (Total) 28.2 24.8 31.6 1,354 45,823 45.0 1.67 1.4 Male 30.4 27.1 33.7 709 24,439 45.6 1.61 0.9 Female 25.8 20.5 31.1 645 21,383 44.2 2.61 1.5 Percentage of children under age 5 with diarrhea treated with ORT (Total) 51.3 45.7 56.9 404 12,938 50.0 2.76 1.1 Male 45.9 36.0 55.8 224 7,424 49.9 4.87 1.5 Female 58.6 52.9 64.4 180 5,514 51.7 2.84 0.7 Prevalence of exclusive breast-feeding of children under six months of age 63.7 55.0 72.4 156 5,773 48.2 4.27 1.1 Male 70.2 56.0 84.5 80 2,975 43.5 6.98 1.4 Female 56.8 40.4 73.2 76 2,798 48.1 8.05 1.5 Prevalence of children 6-23 months of age receiving a minimum acceptable diet (MAD) 40.1 33.8 46.4 428 13,722 49.1 3.11 1.3 Male 35.2 29.2 41.3 210 6,810 48.2 2.97 0.9 Female 44.9 33.4 56.4 218 6,912 51.4 5.65 1.6 CUSTOM INDICATORS Percentage of women with a child under 5 years who recognize two or more pregnancy danger signs 73.2 67.9 78.5 828 31,835 44.3 2.61 1.7 Percentage of women with a child under 5 years who recognize two or more neonatal danger signs 59.8 54.7 64.8 828 31,835 49.1 2.48 1.5 Percentage of women with a child under 5 years who recognize two or more childhood illness signs 58.9 53.3 64.5 828 31,835 49.2 2.75 1.6 Percentage of women with a child under 5 years seeking health care when pregnancy danger signs are present 71.4 65.0 77.7 207 7,854 45.3 3.14 1.0 Percentage of women with a child under 5 years seeking health care when neonatal danger signs are present 71.0 61.8 80.2 123 4,987 45.5 4.51 1.1 Percentage of women with a child under 5 years seeking health care when childhood illness signs are present 84.0 80.0 88.0 293 11,823 36.7 1.98 0.9 Percentage of women with a child under 5 years making decisions about health care for themselves 61.1 52.4 69.7 718 28,004 48.8 4.26 2.3 Percentage of women with a child under 5 years making decisions about health care for their child 98.2 96.9 99.4 718 28,004 13.4 0.62 1.2 NA : Not available Annex 5: PBS Tables 2013 Baseline 2018 Endline Total population 220,181 311,489 Male 106,275 145,293 Female 113,906 166,197 Women of reproductive age (15-49 years) 48,937 76,480 Children ages 0-59 months 33,385 45,823 Males ages 0-59 months 16,785 24,439 Females ages 0-59 months 16,600 21,383 Children ages 0-5 months 3,739 5,773 Males ages 0-5 months 1,807 2,975 Females ages 0-5 months 1,931 2,798 Children ages 6-23 months 10,030 13,722 Males ages 6-23 months 5,365 6,810 Females ages 6-23 months 4,665 6,912 Total households 33,846 48,965 Male and Female Adults 30,405 42,779 Adult Female no Adult Male 2,943 5,875 Adult Male no Adult Female 472 311 Child No Adults 26 … Average household size 6.5 6.4 Percentage of households with children 0-59 months 60.5 58.6 Percentage of households with a child 6-23 months 27.7 25.9 Percentage of households with a child 0-5 months 10.6 11.2 Household headship (% male) 82.9 79.8 Education level of head of household No formal education* 40.4 41.8 Pre-primary 0.2 0.2 Primary* 53.9 50.4 Lower Secondary1 2.3 2.7 Upper Secondary2 2.7 3.8 Higher 0.5 1.1 2 Equivalent to "Diversificado" in Guatemala. Table A5.1. Population and Household Characteristics in the Project Area, CRS-SEGAMIL [Guatemala, 2013, 2018] Source: FFP Baseline (2013) and Endline (2018) PBS in Guatemala 1 Equivalent to "Básico" in Guatemala. Percent Number Regular participation in project activities Yes 37.1 529 No 62.9 913 Type of project activities in which households participated on a regular basis1 Received food rations 96.9 509 Attended nutrition training/meetings 91.1 485 Attended agriculture training/meetings 87.2 452 Received cash transfers 9.0 51 Other activities 17.4 118 Number of project activities in which households participated on a regular basis Did not participate in any activities 63.0 913 1 1.8 24 2 4.3 66 3 23.3 310 4 6.9 111 5 0.8 17 NOTE: A total of 25 of the 3,002 households in the sample did not report information on household participation in project activities. This includes 13 households in the CRS-SEGAMIL project area. Among the households reporting direct participation in project activities,1 household did not provide additional information on the type of activities in which they participated. 1 Percentages add up to more than 100 because households can participate in more than one type of activity. Percentage is proportion of all respondents who answered "Yes" to regularly participating in project activities. A total of 51 households or 9 percent of the sample reported receiving cash transfers although the CRS-SEGAMIL project did not provide this intervention. It is possible that those respondents may have confused the food vouchers that were provided via a card with cash transfers. Table A5.2 Project participation Percentage of households reporting regular participation in project activities, overall and by activity type [Guatemala 2018] CRS-SEGAMIL Baseline 2013 (BL) Endline Comparison by Participation Status (Sig.) All Households (I) All Households (II) Direct Participants (III)1 Indirect participants (IV)1 All Households (V) Direct Participants (VI)1 Indirect participants (VII)1 Direct vs. Indirect Participation (VIII)1 Household dietary diversity score (HDDS) 6.2 6.8 6.9 6.7 *** *** *** ns Per capita consumption expenditures (as a proxy for income) of USG￾assisted areas (constant 2010 USD at 2005 PPP) $1.94 $2.38 $2.19 $2.53 *** * *** ** Prevalence of poverty: Percent of people living on less than $1.25/day (%) 43.8 19.2 18.8 19.1 *** *** *** ns Mean depth of poverty (%) 11.9 3.5 2.8 3.8 *** *** *** ns Women's dietary diversity score (WDDS) 4 4.5 4.6 4.5 *** *** ** ns Prevalence of underweight children under five years of age (%) 34.5 25.3 22.2 29.6 ** *** ns † Prevalence of stunting children under five years of age (%) 79.6 72.2 74.1 69.6 ** † ** ns Prevalence of children 6-23 months with a minimum acceptable diet (MAD) (%) 17.9 40.1 46.4 31.7 *** *** ** ** Table A5.3. Comparison of Baseline and Endline Indicators by Project Participation Status - Catholic Relief Services SEGAMIL Program Guatemala FY 2012 FFP Development Food Assistance Programs - Select Indicators Endline 2018 (EL) BL-EL Comparison (Sig.) Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 NOTE: 1 The household survey asked respondents whether they participated regularly in any of the project activities. Respondents who answered "Yes" are considered direct participants. Respondents who answered "No" or "Don't know" are considered indirect participants. A total of 13 households in the CRS-SEGAMIL project areas did not report information on project participation. Households with missing information on project participation are not included in the BL-EL comparison by participation status (Columns VI and VII) nor in the endline comparison by participation status (Column VIII). Table A5.4. Baseline-Endline Comparison of Use of Sustainable Agricultural Practices Percentage of farmers using sustainable agricultural practices by type of practice [Guatemala 2013, 2018] 2013 Baseline 2018 Endline Sig. Crops1 Use of improved seeds for beans 2.7 2.7 ns Use of soil conservation practices 57.1 49.0 * Use of organic fertilization 69.9 75.7 ns Household garden production 31.2 48.7 *** Management of fruit trees 28.5 41.0 ** Production of native herbs 20.1 33.1 *** Use of at least two sustainable crop practices 55.7 63.6 * No sustainable crop practices 13.5 13.2 ns Livestock2 Improved (roof, wall, water, and food) livestock facilities for rabbits, goats, pigs and poultry 32.9 57.2 *** Vaccination of poultry 27.1 32.7 ns Management of goats (3 of 4 categories: deworming, vaccination, hoof removal, vitamin supplementation) 0.0 0.4 ns Use of at least two sustainable livestock practices 10.2 23.5 *** No sustainable livestock practices 50.3 33.6 *** Natural resource managemet3 Agroforestry 5.0 4.4 ns Natural forest regeneration 3.4 5.6 ns Reforestation - new plantations 16.1 10.8 † Reforesting of water resources 2.3 1.3 ns Management of forests and woods 0.9 1.7 ns Production of forest plants 0.3 0.3 ns Use of at least two sustainable NRM practices 1.3 1.8 ns No sustainable NRM practices 73.3 77.9 ns Number of farmers 2,972 1,381 CRS - SEGAMIL Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 NOTE: Multiple responses allowed so totals may add up to more than 100 percent. Each household is associated with one farmer's record. In households with multiple farmers, responses were consolidated into one record per household. If more than one farmer provided a response, the response of the first farmer (usually the household head) was considered. 1 Includes use of improved seeds for corn, soil conservation practices, use of organic fertilization, household garden production and production of native herbs. 2 Includes improved livestock facilities (roof, wall, water and food) for rabbits, goats, pigs and poultry; vaccination of poultry for both program areas. 3 Includes agro-forestry, reforestation, management of forest plantations, management of natural reforestation, reforesting of water resources and production of plants for reforestation. Table A5.5. Baseline-Endline Comparison of Use of Financial Services and Value Chain Activities Percentage of farmers by access to financial services and value chain activities [Guatemala 2013, 2018] 2013 Baseline 2018 Endline Sig. Financial services Saved money 7.3 2.5 *** Obtained agricultural credit 5.3 0.5 *** Had agricultural insurance 1 10.2 *** No financial services 87.6 87.8 * Value chain activities Market-oriented production 46.0 52.8 † Calculation of cost of production for the market 6.7 18.1 *** Keeping production records 1.9 2.7 ns Developing production and sales plans 7.3 15.2 ** No value chain activities 54.0 47.2 † Number of farmers 2,972 2,501 CRS - SEGAMIL Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 NOTE: Each household is associated with one farmer's record. In households with multiple farmers, responses were consolidated into one record per household. If more than one farmer provided a response, the response of the first farmer (usually the household head) was considered. 2013 Baseline 2018 Endline Sig. Prevalence of stunted children 79.6 72.2 ** <6 45.0 47.3 ns 6-11 64.6 55.0 ns 12-23 84.8 84.4 ns 24-35 88.9 80.7 * 36-47 85.2 78.9 ns 48-59 84.2 67.1 *** Number of children 2,978 1,333 Prevalence of underweight children 34.5 25.3 ** <6 12.7 16.3 ns 6-11 29.4 16.1 * 12-23 46.5 36.1 * 24-35 36.5 25.7 * 36-47 37.0 27.2 * 48-59 31.4 21.7 † Number of children 2,978 1,337 Prevalence of wasted children 2.0 0.9 ** <6 0.3 0.9 ns 6-11 2.1 1.1 ns 12-23 5.0 1.7 * 24-35 2.5 0.5 * 36-47 0.8 0.3 ns 48-59 0.2 1.1 ns Number of children 2,978 1,336 Table A5.6a. Baseline-Endline Comparison of the Malnutrition Status of Children Under Five 1 Prevalence of stunted, underweight and wasted children by age in months [Guatemala 2013, 2018] CRS-SEGAMIL Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 1 The prevalence of stunting is based on children with valid height for age measurements, the prevalence of underweight is based on children with valid weight for age measurements, and the prevalence of wasting is based on children with valid weight for height measurements. 2013 Baseline 2018 Endline Sig. Prevalence of stunted children 0-23 months 69.8 66.8 ns Number of children 1,204 575 Prevalence of underweight children 0-23 months 33.8 25.7 ns Number of children 1,204 578 Table A5.6b. Baseline-Endline Comparison of the Malnutrition Status of Children Under Two Prevalence of stunted and underweight children 0-23 months [Guatemala 2013, 2018] CRS-SEGAMIL Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 2013 Baseline 2018 Endline Sig. Prevalence of stunted children 12-59 months 85.9 78.3 ** Number of children 2,369 1,037 Table A5.6c. Baseline-Endline Comparison of the Prevalence of Stunting Among Children 12-59 months Prevalence of stunted children 12-59 months [Guatemala 2013, 2018] CRS-SEGAMIL Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 Percent Number of children Sig. Participation status1 Non-participant 69.6 599 ns Participant 74.1 734 Total 72.2 1,333 Food rations2 No 74.7 23 ns Yes 74.1 711 Total 74.1 734 Nutrition training/meetings2 No 73.0 62 ns Yes 74.2 672 Total 74.1 734 Agriculture training/meetings2 No 70.2 92 ns Yes 74.6 642 Total 74.1 734 Cash transfers2 No 73.8 663 ns Yes 77.8 71 Total 74.1 734 Other activities2 No 73.4 583 ns Yes 77.7 151 Total 74.1 734 Table A5.7. Prevalence of Stunting by Project Participation Status at Endline Prevalence of stunted children under five by project participation status [Guatemala 2018] Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 1 Direct participants refer to households that self-reported regularly participating in project activities. Indirect participants refer to households that self-reported not regularly participating in project activities. 2 Includes only children living in households that self-reported regularly participating in project activities. CRS-SEGAMIL 2013 Baseline 2018 Endline Sig. Percent with minimum meal frequency 57.3 66.8 † Percent with minimum dietary diversity 27.1 53.9 *** Grains, roots, and tubers 92.4 93 ns Legumes and nuts 32.4 48.4 ** Dairy products (milk, yogurt, cheese) 20.2 31.5 ** Flesh foods (meat, fish, poultry, and liver/organ meats) 14.7 21 * Eggs 36.2 51 ** Vitamin A-rich fruits and vegetables 46.4 58.7 ** Other fruits and vegetables 32.5 62 ** Number of children 6-23 months 895 428 Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 Table A5.8 Baseline-Endline Comparison of Minimum Acceptable Diet (MAD) Components of MAD indicator for children 6-23 months [Guatemala 2013 2018] CRS-SEGAMIL 2018 Endline Breastfed children 6-8 months Percent with minimum meal frequency (2 or more) 70.1 Percent with minimum dietary diversity (4 or more) 16.2 Grains, roots, and tubers 78.4 Legumes and nuts 16.2 Dairy products (milk, yogurt, cheese) 18.1 Flesh foods (meat, fish, poultry, and liver/organ meats) 5.3 Eggs 23.2 Vitamin A-rich fruits and vegetables 32.5 Other fruits and vegetables 38.1 Number of children 78 Breastfed children 9-23 months Percent with minimum meal frequency (3 or more) 78.5 Percent with minimum dietary diversity (4 or more) 63.3 Grains, roots, and tubers 95.5 Legumes and nuts 58.5 Dairy products (milk, yogurt, cheese) 33.4 Flesh foods (meat, fish, poultry, and liver/organ meats) 21.6 Eggs 59.6 Vitamin A-rich fruits and vegetables 61.9 Other fruits and vegetables 67.4 Number of children 273 Non-breastfed children 6-23 months Percent with minimum meal frequency (4 or more + 2 milk) 20.8 Percent with minimum dietary diversity (4 or more) 61.3 Grains, roots, and tubers 100.0 Legumes and nuts 47.1 Dairy products (milk, yogurt, cheese) 38.8 Flesh foods (meat, fish, poultry, and liver/organ meats) 36.0 Eggs 50.2 Vitamin A-rich fruits and vegetables 75.6 Other fruits and vegetables 68.6 Number of children 77 Table A5.9 Minimum Acceptable Diet (MAD) by Age Group and Breasfeeding Status Components of MAD indicator for children 6-23 months [Guatemala 2018] NOTE: The results for these subgroup analyses are based on small sample sizes and may be unreliable. 2018 Endline Not breastfeeding <2 4.3 2-3 2.7 4-5 1.1 6-8 1.0 9-11 1.5 12-17 10.8 18-23 38.7 Exclusively breastfed <2 74.3 2-3 61.9 4-5 57.7 6-8 14.4 9-11 0.0 12-17 0.0 18-23 0.7 Breastfed and plain water only <2 7.9 2-3 24.4 4-5 6.4 6-8 0.7 9-11 1.5 12-17 0.0 18-23 0.0 Breastfed and non-milk liquids <2 4.8 2-3 4.2 4-5 10.3 6-8 6.0 9-11 0.5 12-17 3.2 18-23 0.0 Breastfed and other milk <2 6.6 2-3 6.7 4-5 9.8 6-8 1.2 9-11 2.3 12-17 0.0 18-23 0.0 Breastfed and complementary foods <2 2.2 2-3 0.0 4-5 14.8 6-8 76.6 9-11 94.2 12-17 86.0 18-23 60.6 Number of children 584 Table A5.10. Baseline-Endline Comparison of Breastfeeding Status Breastfeeding status for children 0-23 months by age in months [Guatemala 2013, 2018] NOTE: The results for these subgroup analyses are based on small sample sizes and may be unreliable. Table A5.11. Baseline-Endline Comparison of Women's Dietary Diversity Food groups consumed by women 15-49 [Guatemala 2013, 2018] 2013 Baseline 2018 Endline Sig. Grains, roots and tubers 99.6 99.7 ns Legumes and nuts 49.9 59.5 ** Dairy products (milk, yogurt, cheese) 15.3 19.0 ns Organ meat 4.9 4.1 ns Eggs 45.1 59.5 *** Flesh foods and misc. small animal protein 33.5 39.2 * Vitamin A dark green leafy vegetables 49.5 53.3 ns Other Vitamin A rich vegetables and fruits 41.1 40.7 ns Other fruits and vegetables 58.6 79.6 *** Number of women 15-49 2,771 2,092 CRS-SEGAMIL Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 2013 Baseline 2018 Endline Sig. Improved, not shared sanitation facility Flush to piped sewer system 4.1 5.0 ns Flush to septic tank 3.2 5.0 ns Pit latrine with slab 44.1 56.8 ** Improved, shared sanitation facility Flush to piped sewer system 0.4 0.5 ns Flush to septic tank 0.4 0.3 ns Pit latrine with slab 4.5 7.1 * Non-improved sanitation facility Open pit 32.8 18.0 *** No facility 10.4 7.0 ns Other 0.1 0.3 * Improved source of drinking water Piped water into dwelling 12.8 32.2 *** Piped water into yard 15.3 38.4 ** Public tap 0.8 1.2 ns Private well 8.1 14.6 ns Public well 5.5 3.0 † Rainwater 0.1 0.6 ** Non-improved source of drinking water Surface water (river/dam/ lake/ponds /stream/canal/irrigation channel) 56.5 9.5 *** Tanker truck 0.0 … n/a Other 1.0 0.5 ns Water availability Water is generally available from this source (% Yes) 63.3 71.3 ns Water is not available a day or more during the last two weeks (% Yes) 53.3 37.7 ** Water treatment prior to drinking1 Boil 88.5 96.6 Filter 3.1 3.6 Bleach/chlorine added 3.7 7.6 Purchase purified water 0.2 0.8 Other 0.6 0.3 No treatment 8.7 1.7 Number of households 3,074 1,441 Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 1 Multiple responses allowed so totals may add up to more than 100 percent. Table A5.12. Baseline-Endline Comparison of Household Sanitation and Drinking Water Sanitation facility, source of drinking water, and treatment of drinking water [Guatemala 2013, 2018] CRS-SEGAMIL Percent Number of children Sig. Improved water source Household does not use an improved water source 32.5 725 ** Household uses an improved water source 23.9 629 Improved sanitation facility Household does not use an improved sanitation facility 32.2 472 † Household uses an improved sanitation facility 26.1 882 Handwashing station with water and soap or another cleansing agent Household does not have a handwashing station with water and soap or another cleansing agent 28.6 298 ns Household has a handwashing station with water and soap or another cleansing agent 28.1 1,056 Household water treatment Household does not treat water 51.1 22 ns Household treats water 28 1,332 Use of improved facilities for livestock Household does not use improved facilities for livestock 30.4 552 ns Household uses improved facilities for livestock 26.8 765 Project participation Household did not participate in any project activities 29.6 615 ns Household participated in at least one project activity 27.2 739 All households 28.2 1,354 Table A5.13. Diarrhea and household water, hygiene and sanitation (WASH) at Endline Prevalence of diarrhea among children under five by household WASH status [Guatemala, 2018] Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, ***0.001 NOTE: Chi squared tests were used to examine the statistical significance of the relationship between the prevalence of diarrhea and household WASH status. 2013 Baseline 2018 Endline Sig. Cereals 99.5 98.1 *** Roots and tubers 33.9 46.4 *** Vegetables 61.2 65.1 ns Fruits 39.9 60.0 *** Meat, poultry and organ meat 29.2 32.3 ns Eggs 45.9 60.4 *** Fish and seafood 4.8 8.7 ** Pulses/legumes/nuts 54.7 60.2 † Milk and milk products 12.9 17.5 ns Oils/fats 61.8 62.6 ns Sugar/honey 93.3 91.9 ns Miscellaneous (tea, coffee, condiments, etc.…) 80.1 78.5 ns Number of households 3,074 1,332 Table 5.14. Baseline-Endline Comparison of Household Dietary Diversity Food groups consumed by households [Guatemala 2013, 2018] Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 NOTE: Only includes households that reported that yesterday was not an unusual or special day. CRS-SEGAMIL Table 5.15. Baseline-Endline Comparison on Women's Nutritional Status Height and BMI levels of non-pregnant women 15-49 years of age [Guatemala 2013, 2018] 2013 Baseline 2018 Endline Sig. Percent less than 145 cm 52.0 52.2 ns Mean Body Mass Index (BMI) 24.6 25.1 *** Normal (%) 18.5-24.9 (total normal) 64.4 53.9 *** Underweight (%) <18.5 (total underweight) 2.2 1.2 † 17.0-18.4 (mildly underweight) 1.8 1.2 ns <17 (moderately and severely underweight) 0.3 0.0 * Overweight/obese (%) ≥25 (total overweight or obese) 33.5 44.9 *** 25.0-29.9 (overweight) 27.2 33.6 ** ≥30.0 (obese) 6.4 11.3 *** Number of non-pregnant women 15-49 2,773 1,871 CRS-SEGAMIL Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 54% 0% 1% 34% 11% BMI levels of non-pregnant women of reproductive age, CRS-SEGAMIL [Guatemala 2018] 18.5-24.9 (total normal) 17.0-18.4 (mildly underweight) <17 (moderately and severely underweight) 25.0-29.9 (overweight) ≥30.0 (obese) 64% 2% 0% 27% 7% BMI levels of non-pregnant women of reproductive age, CRS-SEGAMIL [Guatemala 2013] 18.5-24.9 (total normal) 17.0-18.4 (mildly underweight) <17 (moderately and severely underweight) 25.0-29.9 (overweight) ≥30.0 (obese) Percent Number of women Sig. Participation status1 Indirect 44.5 1,124 ns Direct 45.3 746 Total 44.9 1,870 Food rations2 No 44.4 37 ns Yes 45.4 709 Total 45.3 746 Nutrition training/meetings2 No 49.3 61 ns Yes 45.0 685 Total 45.3 746 Agriculture training/meetings2 No 51.9 110 ns Yes 44.4 636 Total 45.3 746 Cash transfers2 No 45.3 670 ns Yes 45.4 76 Total 45.3 746 Other activities2 No 43.3 570 * Yes 54.9 176 Total 45.3 746 Table 5.16. Prevalence of Overweight of Obese Women 15-49 years by Project Participation Status at Endline Prevalence of women's overweight or obesity by project participation status [Guatemala 2018] CRS-SEGAMIL Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 1 Direct participants refer to households that self-reported regularly participating in project activities. Indirect participants refer to households that self-reported not regularly participating in project activities. 2 Includes only women living in households that self-reported regularly participating in project activities. Annex 6: Summary of Methodology and Results of Supplemental Multivariate Analyses Annex 6: Summary of Methodology and Results of Supplemental Multivariate Analyses This annex provides the methodology and summary of results of the supplemental multivariate analyses that the evaluation team conducted to explore the factors that promoted or inhibited the achievement of the SEGAMIL goal and objectives. Multivariate analyses controlling for key socioeconomic and project-specific factors as covariates was used to understand the factors that are associated with daily per capita consumption expenditures, and the prevalence of stunting and underweight among children under five. These indicators were selected for additional analyses to help inform learning on key outcomes and impact indicators that showed improvement over time. Daily per capita consumption expenditures increased markedly over the life of the award from $1.94 at 2013 baseline to $2.38 at 2018 endline. Daily per capita consumption expenditures improved for both direct and indirect participants. The prevalence of stunting declined since baseline from 79.6% to 72.2%. The prevalence of underweight also improved over the life of the project; at baseline about one-third of children were underweight compared to one-quarter of children at endline. The following sections describe the data used, provide variable definitions, and describe the statistical methods, results and conclusions. Daily per capitation consumption expenditures Data used Data used in these analyses come from population-based household surveys (PBS) implemented at project baseline (BL) (2013) and endline (EL) (2018) and utilized for the 2018 performance evaluation of the project. The surveys collected standard information on household and respondent characteristics; food security and poverty; agricultural practices; children’s health and nutrition; and women’s health and nutrition. The analyses are restricted to cases with nonmissing information on the dependent and explanatory variables. The final sample size for the analyses of daily per capita consumption expenditures was 4,264 households. Variable definitions Dependent variables The main outcome of interest is daily per capita consumption expenditures. Annex 2 provides the Data Treatment Analysis Plan and details on the methodology for calculating of daily per capita consumption expenditures. To deal with normality, daily per capita consumption expenditures were transformed by taking its natural logarithm. Explanatory variables The analyses controlled for individual, household and project-level characteristics that can influence daily per capita consumption expenditures. The selection of covariates is based the availability of data collected at baseline and endline. Variables with statistically significant bivariate associations with the outcome are excluded from the multivariate analysis if post-estimation tests indicated multicollinearity. Bivariate analyses of the daily per capita consumption expenditures were conducted for each of the following household agricultural practices that were promoted by the project: acquiring agricultural credit in cash or in kind, making savings, use of value chain activities (any two), use of at least two sustainable crop practices, use of at least two sustainable livestock practices, and use of at least two sustainable NRM practices. Department dummies are added to capture variations in agro-ecological zones and other unobserved regional factors such as distance to the market and differences in the implementation of activities between departments. To examine and control for potential differences in effects for direct and indirect participants, a dummy is included for households in which any member participated in the project activities on a regular basis. The designation of the household as a direct participant is based on the household survey respondent’s reply and is not verified using project documents. Additionally, the analyses explored the association of daily per capita consumption expenditures with the type of intervention which could include food rations, agriculture trainings/meetings, and nutrition training/meetings, and other activities.1 Most direct participants generally participated in all interventions. The number of interventions received by each household was also explored. The benefits of increased consumption expenditures may be higher for households participating in a higher number of interventions. Cases that comprise the baseline sample are coded “0” for participation variables, since no project activities were implemented at the time of the baseline survey. The following household sociodemographic characteristics were considered: number of children under five, number of adult females, number of adult males, household gendered type and the age, sex and educational attainment of the household head. Because the projects targeted children under the age of two, households with a higher number of children under the age of five are likely to benefit from project interventions compared to households where children have aged out of the under two age brackets Households with working age males and male-headed households, compared to adult female only households or female-headed households, may be more likely to engage in market-oriented production or income generating opportunities and/or may be more likely to access credit to purchase productivity￾enhancing agricultural inputs and equipment, make investments in household water and sanitation infrastructure, or purchase food items that cannot be home grown. Alternatively, because the Women’s savings and loan groups aimed to increase women’s use of credit and making savings, households with a participating adult female may have been more likely to pursue practices that contribute to increases in daily per capita consumption expenditures. The results of the bivariate analyses are presented in Annex 7, Tables 7.1 and 7.2 and Annex 8, Table 8.1. For the purposes of parsimony only variables whose bivariate association was found to be statistically significant were included in the multivariate model. 2 Statistical Methods The analyses used ordinary least squares (OLS) regression model to analyze the correlates of daily per capita consumption expenditures. The analyses accounts for the two-stage stratified cluster sampling design. All analyses were conducted using STATA 14. The r-squared of the models is reported as a measure of goodness of fit. Additional post-estimation tests were conducted to check for the influence of outliers and the model was re-run excluding influential cases (i.e. cases with high residuals and leverage). Results Table 8.2 illustrates the results of the OLS regression of the natural logarithm of daily per capita consumption expenditures. The results are reported as coefficients with the 95% confidence interval. 1 A total of 51 households or 9 percent of the sample reported receiving cash transfers although the CRS￾SEGAMIL project did not provide this intervention. It is possible that those respondents may have confused the food vouchers that were provided via a card with cash transfers. 2 Number of adult males, number of adult females and number of children under five were excluded because they are collinear with other covariates such as household size and with gendered household type. The baseline model (Model 1) controls for the survey round variable only and indicates that daily per capita consumption expenditures are higher at endline compared to baseline. Improvement in daily per capita consumption expenditure between BL and EL can be partially explained by households’ use of agricultural practices promoted by the project. Making savings, using at least two value chain activities, at least two sustainable crop practices, and at least two sustainable livestock practices are associated with higher daily per capita consumption expenditures (Model 2). Acquiring agricultural credit is associated with higher daily per capita consumption expenditures but this association is only marginally statistically significant (Model 2). These associations remain statistically significant net of all other factors in the final model (Model 5). Model 4 controls for project participation and illustrates a negative association with daily per capita consumption expenditures. The final model (Model 5) controls for a host of household sociodemographic characteristics that could influence daily per capita consumption expenditures. The r￾squared indicates that the final model explains 27.9 percent of the variance in daily per capita consumption expenditures. We performed post estimation diagnostics to identify influential cases (i.e. with high residuals or leverage) and re-ran the analyses excluding those influential cases. Cook's D was used to identify cases (households) with high residuals and leverage using the conventional cut-off point of 4/n where n is the number of observations (in this case 4/4,264). A total of 158 households from the baseline sample and 78 households from the endline sample were flagged as influential observations and were excluded from this analysis. The results of the analysis excluding influential cases are provided in Table 8.3 and are consistent with the findings reported in Table 8.2. Discussion and conclusions The objective of the additional multivariate analyses was to better understand the factors associated with improvements in daily per capita consumption expenditures. The results underscore the positive association of participation in agricultural interventions promoted by the project with improvements in consumption expenditures. Specifically, acquiring agricultural credit, making savings, using at least two value chain activities, and at least two sustainable livestock practices promoted by the project is associated with an increase in daily per capita consumption net of other factors. Household agricultural status may be contributing to an increase in daily per capita consumption expenditures either directly through the consumption of more diverse home-grown foods and/or indirectly by increased income from higher yields and profits. It is not unexpected that the association between project participation is negative since the project targeted the poorest of the poor. Inclusion of household sociodemographic characteristics increases the goodness of the fit of the model from about 9.8 percent to 27.9 percent. The inclusion of these variables also washes out the association with project participation which is no longer statistically significant in the final model. The size and composition of the household can affect daily per capita consumption expenditures by influencing the availability of labor to participate in income generating activities, and indirectly by determining eligibility for participation in project interventions since the project targeted households with pregnant and lactating women, children under two, and the poorest of the poor. The results of these supplemental analyses are intended to identify potential correlates that are associated with daily per capita consumption expenditures that may support the future design of project activities and/or beneficiary targeting based on household demographic and socioeconomic factors. However, the pre-post design of the data collection does not allow statements to be made about attribution relating to project impact. PREVALENCE OF STUNTING AND UNDERWEIGHT AMONG CHILDREN UNDER FIVE Data Used in the Analysis Data used in these analyses come from population-based household surveys (PBS) implemented at project baseline (BL) (2013) and endline (EL) (2018) and utilized for the 2018 performance evaluation of the project. The surveys collected standard information on household and respondent characteristics; food security and poverty; agricultural practices; children’s health and nutrition; and women’s health and nutrition. The analyses are restricted to cases with nonmissing information on the dependent and explanatory variables. The final sample size for the analyses of children’s stunting and underweight is 3,683 children under five. Definitions of Variables Dependent variables The main outcomes of interest are the prevalence of stunting and underweight among children under five. The baseline and endline surveys collected weight and height measurements for children under five. Data are collected for all children under. The World Health Organization (WHO) child growth standards (WHO Multicentre Growth Reference Study Group 2006) were used to compute z-scores for height-for-age (HAZ) and weight-for-age (WAZ). Each z-score is calculated by comparing the child’s height/length or weight with the median value of the WHO 2006 reference population. The difference is divided by the standard deviation of the reference population. datasets will be cleaned by flagging cases with z-scores beyond specified lower or upper cutoffs and excluding them from the computation of all indicators. Cases with height-for-age z-scores that are less than -6 standard deviations (SD) from the median or greater than +6 SD above the median were flagged and excluded from the calculation of the prevalence of stunting. Cases with weight-for-age z-scores that are less than -6 SD from the median or greater than +5 SD above the median were flagged and excluded from the calculation of the prevalence of underweight. Children with HAZ or WAZ below – 2 SD from the median are considered moderately-to-severely stunted or underweight, respectively. Explanatory variables The analyses controlled for individual, household and project-level characteristics that can influence the prevalence of stunting and underweight. The selection of covariates is based on the availability of data collected at baseline and endline. The analysis of children’s stunting and underweight included the child’s age, sex, and whether the child experienced diarrhea in the two weeks prior to the survey. The bivariate analyses also explored the association of child nutritional outcomes with the caregiver’s educational level and age to capture differences in basic literacy and experience with caregiving since in some case children may be looked after by older siblings rather than grandmothers or adult women. Children’s dietary diversity was not included since this would result in a reduction in sample size because information on food groups consumed and meal frequency were collected only for children 6 -23 months. Household consumption of food groups was used as a proxy for children’s dietary diversity. Knowledge of childhood illness danger signs by a mother of a child under the age of five in the household was also explored. Children are more likely to receive treatment if their mother, or another mother living in the same household, can correctly identify childhood illness danger signs that require medical treatment. The analyses considered the following household sociodemographic characteristics: number of children 0-4 other than the child, number of children 5-17, number of adult females, number of adult males, household gendered type, and the age, sex and educational attainment of the household head. Because the project targeted children under the age of two, households with a higher number of children under the age of five are likely to benefit from project interventions compared to households where children have aged out of the under two age brackets. The size and composition of the household may also affect how often children are fed, what they are fed, and how much. Households with working age males and male-headed households, compared to adult female only households or female-headed households, may be more likely to engage in market-oriented production or income generating opportunities and/or may be more likely to access credit to purchase productivity-enhancing agricultural inputs and equipment, make investments in household water and sanitation infrastructure, or purchase food items that cannot be home grown. Alternatively, because the savings and loan groups aimed to increase women’s use of credit and making savings, households with a participating adult female may have been more likely to pursue practices that contribute to increases in daily per capita consumption expenditures. Households’ water, sanitation and hygiene status can influence children’s nutritional status since the use of unhygienic practices can lead to diarrheal disease and loss of important minerals and vitamins and contribute to weight loss. Variables explored for inclusion in the multivariate analyses included whether the household used an improved drinking water source, an improved sanitation facility, whether the household has soap and water at a commonly used handwashing station, and whether the household treated its water. Household socioeconomic status was included by daily per capita consumption expenditures. Since daily per capita consumption expenditures is skewed it was transformed by taking its natural logarithm to avoid the influence of outliers on the outcome. The analyses also explored the association of children’s stunting and underweight with several key activities promoted by the project, and undertaken by the primary farmer in the household, that aimed to increase household food security through increased food production, food availability, and economic resources: acquiring agricultural credit, making savings, use of at least two value chain activities; use of at least two sustainable crop practices; use of at least two sustainable livestock practices; and use of at least two sustainable natural resource management practices. The analyses also examined the relationship of stunting and underweight to specific practices promoted by the project – namely, whether the household engaged in market-oriented production, household garden production, or used improved facilities for it livestock. The underlying aim for the inclusion of such variables is to better understand their potential role in fostering improvements in food security and children’s nutritional status. Department dummies are added to capture variations in agro-ecological zones and other unobserved regional factors such as distance to the market or health services and differences in the implementation of activities between departments. To examine and control for potential differences in effects for direct and indirect participants, a dummy is included for households in which any member participated in the project activities on a regular basis. The designation of the household as a direct participant is based on the household survey respondent’s reply and is not verified using project documents. Additionally, the analyses explored the association of children’s underweight and stunting with the type of intervention which could include food rations, agriculture trainings/meetings, nutrition training/meetings, and other activities.3 The number of interventions received by each household was also explored. The benefits to 3 A total of 51 households or 9 percent of the sample reported receiving cash transfers although the CRS￾SEGAMIL project did not provide this intervention. It is possible that those respondents may have confused the food vouchers that were provided via a card with cash transfers. children’s nutritional outcomes may be higher for households participating in a higher number of interventions. Cases that comprise the baseline sample are coded “0” for participation variables, since no project activities were implemented at the time of the baseline survey. The results of the bivariate analyses are presented in Annex 7, Tables 7.1 and 7.2. For the purposes of parsimony only variables whose bivariate association was found to be statistically significant were included in the multivariate model and some variables were excluded after post-estimation tests indicated multicollinearity. 4 Statistical Methods The analyses used logistic regression models to analyze the correlates of the prevalence of stunting and underweight. The analyses accounts for the two-stage stratified cluster sampling design. All analyses were conducted using STATA 14. We ran post-estimation model specification to assess goodness of fit and model specification. Results Stunting Table 7.3 illustrates the results of the multivariate logistic regression of the prevalence of stunting. The results are reported as odds ratios with the 95% confidence interval. The baseline mode (Model 1) indicated lower prevalence of stunting at endline compared to baseline. Model 2 controls for household WASH status and indicates that the odds of stunting for children in households with water and soap at a commonly used handwashing station are lower than that of children in households lacking this facility Use of at least two sustainable NRM practices is associated with lower odds of stunting (Model 3 and 4). Children living in households that consume meat and milk have lower odds of stunting compared to children who live in households that do not consume this food group, but this association is marginally statistically significant (Model 5). Model 6 indicates that an increase in daily per capita consumption expenditures is associated with lower odds of stunting and this association remains statistically significant across all models. The baseline-endline difference in stunting as indicated by the odds ratio of the survey round variable is no longer statistically significant when the model controls for consumption expenditures (Model 5). The odds of stunting are about 1.3 times higher for children living in households that participated in the project compared to children in households that are indirect project participants (Model 7) but this association is marginally statistically significant and washes out when the model controls for the child’s and household characteristics (Model 8). Underweight Table 7.4 illustrates the results of the multivariate logistic regression of the prevalence of underweight. The results are reported as odds ratios with the 95% confidence interval. The baseline mode (Model 1) indicated lower prevalence of underweight at endline compared to baseline. Model 2 controls for household WASH status and indicates that the odds of underweight for children in households with water and soap at a commonly used handwashing station are 0.708 times that of children in households lacking this facility. Use of at least two sustainable NRM practices is also associated with lower odds of underweight (Model 3 and 4). Children living in households that consume meat have lower odds of underweight compared to children who live in households that do not consume this food group (Model 5). The odds of underweight are about 1.7 times higher for children living in households that consume 4 HDDS was not included because it is collinear with household consumption of foods groups. fish compared to children who live in households that do not consume this food group (Model 5). Model 6 indicates that an increase in daily per capita consumption expenditures is associated with lower odds of underweight and this association remains statistically significant across all models. The baseline￾endline difference in underweight as indicated by the odds ratio of the survey round variable is no longer statistically significant when the model controls for consumption expenditures (Model 6). The final model controls (Model 8) for the child and household characteristics and indicates that children who live in households that are direct participants are less likely to be underweight. Discussion and Conclusions The objective of the additional multivariate analyses was to better understand the factors associated with improvements in the prevalence of stunting among children under five and the factors that impeded the achievement of improvements in children’s underweight. The results of the multivariate analysis of children’s stunting and underweight underscore the role of daily per capita consumption expenditures in contributing to a reduction in the odds of malnutrition. The results of the multivariate analyses do not indicate a statistically significant effect of household agriculture status net of other factors, with the exception of the use of NRM practices. However, multivariate analysis of daily per capita consumption expenditures indicates that household participation in agricultural interventions promoted by the project is associated with improvements in consumption expenditures. Specifically, acquiring credit, making savings, and use of value chain activities, or at least two sustainable livestock practices promoted by the project is associated with an increase in daily per capita consumption (see Table 8.2 for the results of the multivariate analyses of daily per capita consumption expenditures). This suggests that household agricultural status may be contributing to a reduction in vulnerability to malnutrition, either through the consumption of more diverse home-grown foods and/or by increased income from higher yields and profits since the calculation of consumption expenditures includes both. The project aimed to reduce malnutrition by increasing use of improved health and nutrition practices. The results of the multivariate analyses generally did not indicate a statistically significant association between consumption of food groups promoted by the project and children’s stunting or underweight except for meat. This may partially be examined by the fact that these variables measure household-level consumption and not the food groups consumed by the child. The higher odds of underweight among children who consume fish and seafood compared to the odds of those who do not may be explained by the fact that fish and seafood are likely to be consumed by poorer households and the prevalence of malnutrition tends to be higher in poorer households. The results of these supplemental are intended to identify potential correlates that are associated with child malnutrition which may support the future design of project activities and/or beneficiary targeting based on household demographic and socioeconomic factors. However, the pre-post design of the data collection does not allow statements to be made about attribution relating to project impact. These analyses leveraged data available at baseline and endline to include a range of factors that are expected to be associated with the stunting and underweight while excluding variables that exhibited multicollinearity. However, some key variables are not included. Postestimation diagnostics of the stunting and underweight models indicated goodness of fit and did not detect model misspecification. However, children’s malnutrition is often closely associated with the mother’s health (e.g. weight of the mother during pregnancy) and the child’s birth weight but it was not possible to include such variables in these analyses as the survey instrument did not collect this information. Annex 7: Tables for Multivariate Analyses of Stunting and Underweight VARIABLE % N Sig % N Sig Child's characteristics Child's sex Male 77.3 1,874 ns 31.4 1,874 ns Female 75.4 1,809 28.9 1,809 Child's age in years < 6 months 44.7 383 *** 15.2 383 *** 6-11 57.4 398 22.4 398 12-23 86.6 761 41.8 761 24-35 85.8 756 32.2 756 36-47 82.5 709 30.9 709 48-59 77.9 676 27.8 676 Prevalence of diarrhea Child did not experience diarrhea in the past seven days 73.7 2,286 *** 27.9 2,286 ** Child experienced diarrhea in the past seven days 81.4 1,397 34.5 1,397 Caregiver's characteristics Educational attainment Never attended school 81.9 1,262 *** 34.9 1,262 ** Primary or less 76.8 2,101 29.6 2,101 Secondary, vocational, or higher 56.5 319 18.6 319 Caregiver's age (years) 15-19 72.3 1,180 ** 29.1 1,180 ns 20-24 74.3 822 27.3 822 25-29 78.9 753 31.6 753 30/34 77.9 526 31.0 526 35-39 87.3 269 33.7 269 40-49 85.3 93 39.2 93 50 + 71.8 40 43.1 40 Mother's health seeking knowledge Percentage of women with a child under 5 years who recognize two or more childhood illness signs Does not recognize two or more childhood illness signs 75.8 2,032 ns 32.3 2,032 ns Recognizes two or more childhood illness signs 76.9 1,651 28.3 1,651 Household sociodemographic characteristics Household head sex Male 76.6 3,281 ns 30.4 3,281 ns Female 74.7 402 28.3 402 Household head age (years) 18-24 74.1 213 ns 32.5 213 ns 25-34 78.0 1,060 30.0 1,060 35-44 77.9 1,004 29.9 1,004 45-54 75.5 656 32.7 656 55-64 73.1 463 28.3 463 65+ 74.2 287 27.5 287 Household head educational attainment Never attended school 78.5 1,444 ns 32.2 1,444 ns Primary or less 76.3 2,019 29.3 2,019 Secondary, vocational/technical or higher 66.4 220 26.7 220 Gendered household type Adult male and female 76.4 3,504 ns 30.3 3,504 ns Adult female only 75.0 173 28.9 173 Adult male only 85.7 6 39.4 6 Number of adult males 0 75.0 173 ns 28.9 173 ns 1 77.4 2,145 31.9 2,145 2 77.2 803 29.4 803 3 72.7 377 26.3 377 4 68.6 138 21.2 138 5 or more 69.4 47 23.9 47 Number of adult females 1 78.1 1,933 * 30.8 1,933 ns 2 79.3 917 32.3 917 3 70.0 480 29.0 480 4 66.8 238 21.2 238 5 66.1 87 27.8 87 6 or more 72.3 28 18.7 28 Number of children 0 to 4 other than the child None 71.9 1,186 * 26.5 1,186 * 1 79.1 1,673 33.6 1,673 2 77.8 598 28.1 598 3 79.1 181 30.1 181 4 or more 76.3 45 37.8 45 Number of children 5-17 0-1 66.0 541 *** 28.6 541 ns 1 74.9 594 25.9 594 2 73.2 620 28.4 620 3 78.2 702 33.4 702 4 84.5 503 31.1 503 5 84.2 395 35.8 395 6 or more 79.8 328 30.7 328 Table A7.1. Results of bivariate analyses of the prevalence of moderate-to-severe stunting and moderate-to-severe underweight among children under five , CRS￾SEGAMIL [Guatemala 2013, 2018] Stunting Underweight VARIABLE % N Sig % N Sig Table A7.1. Results of bivariate analyses of the prevalence of moderate-to-severe stunting and moderate-to-severe underweight among children under five , CRS￾SEGAMIL [Guatemala 2013, 2018] Stunting Underweight Household water, sanitation, and hygiene (WASH) status Use of improved water source Household does not use an improved water source 77.7 2,801 ns 31.0 2,801 ns Household uses an improved water source 73.9 882 28.7 882 Use of improved sanitation facility Household does not use an improved sanitation facility 77.1 1,599 ns 33.3 1,599 * Household uses an improved sanitation facility 75.9 2,084 27.9 2,084 Water and soap at a commonly used handwashing station Household does not have water and soap at a commonly used handwashing station 81.6 1,022 ** 36.3 1,022 ** Household has water and soap at a commonly used handwashing station 74.6 2,661 28.1 2,661 Water treatment Household does nothing to treat water 85.5 91 † Household treats water 76.2 3,592 43.3 91 * Household consumption of diverse food groups 30.0 3,592 Cereals No 93.1 34 † 16.7 34 ns Yes 76.2 3,649 30.4 3,649 Roots and tubers No 77.6 2,306 ns 29.1 2,306 ns Yes 74.8 1,377 31.6 1,377 Vegetables No 77.3 1,391 ns 30.3 1,391 ns Yes 75.9 2,292 30.2 2,292 Fruits No 79.1 2,140 ** 33.2 2,140 ** Yes 73.5 1,543 27.1 1,543 Meat, poultry, and organ meat No 78.1 2,620 ** 32.0 2,620 ** Yes 72.1 1,063 25.7 1,063 Eggs No 78.6 1,924 * 31.3 1,924 ns Yes 74.3 1,759 29.1 1,759 Fish and seafood No 76.7 3,462 ns 29.5 3,462 * Yes 72.2 221 39.2 221 Pulses/legumes/nuts No 76.9 1,591 ns 31.3 1,591 ns Yes 75.9 2,092 29.4 2,092 Milk and milk products No 77.5 3,280 ** 31.0 3,280 * Yes 68.7 403 24.6 403 Oils/facts No 78.2 1,455 ns 32.5 1,455 ns Yes 75.3 2,228 28.9 2,228 Sugar/honey No 74.0 262 ns 31.7 262 ns Yes 76.6 3,421 30.1 3,421 Miscellaneous No 77.3 726 ns 31.3 726 ns Yes 76.2 2,957 29.9 2,957 Household agriculture status Use of credit Did not use credit 76.1 3,525 † 30.3 3,525 ns Used credit 84.6 158 27.9 158 Use of savings Did not make any savings 76.2 3,488 ns 30.1 3,488 ns Made savings 80.7 195 33.1 195 Use of value chain activities (any two) Did not use value chain activities 76.6 3,179 ns 30.2 3,179 ns Used value chain activities 75.7 504 30.1 504 Market-oriented production Did not practice market-oriented production 75.5 2,081 ns 28.4 2,081 ns Practiced market-oriented production 77.3 1,602 32.0 1,602 Use of at least two sustainable crop practices Did not use at least two sustainable crop practices 74.6 1,586 † 27.8 1,586 ns Used at least two sustainable crop practices 77.5 2,097 31.7 2,097 Household garden production Did not practice household garden production 76.1 2,227 ns 30.4 2,227 ns Practiced household garden production 76.7 1,456 29.9 1,456 Use of at least two sustainable livestock practices Did not use at least two sustainable livestock practices 76.5 3,195 ns 31.0 3,195 ns Used at least two sustainable livestock practices 75.7 488 26.3 488 Use of improved livestock facilities Did not use improved livestock facilities 76.2 2,281 ns 31.5 2,281 ns Used improved livestock facilities 76.7 1,402 28.6 1,402 Use of at least two sustainable NRM practices Did not use at least two sustainable NRM practices 76.6 3,595 * 30.5 3,595 * Used at least two sustainable NRM practices 67.4 88 13.2 88 VARIABLE % N Sig % N Sig Table A7.1. Results of bivariate analyses of the prevalence of moderate-to-severe stunting and moderate-to-severe underweight among children under five , CRS￾SEGAMIL [Guatemala 2013, 2018] Stunting Underweight Household poverty status ($1.25/day at 2005 PPP) Household is at or above the poverty line 72.3 2,159 *** 25.6 2,159 *** Household is below the poverty line 84.4 1,524 39.2 1,524 Project-level factors Project participation Household did not participate in at least one project activity 76.4 3,109 ns 33.0 3,109 * Household participated in at least one project activity 76.3 574 24.1 574 Type of participation Food rations Did not receive food rations 76.4 3,122 ns 32.9 3,122 * Received food rations 76.3 561 24.2 561 Nutrition training/meetings Did not participate in nutrition trainings/meetings 76.3 3,163 ns 33.1 3,163 * Participated in nutrition trainings/meetings 76.7 520 23.0 520 Agriculture training/meetings Did not participate in agriculture trainings/meetings 76.2 3,179 ns 32.7 3,179 * Participated in agriculture trainings/meetings 76.8 504 23.6 504 Other activities Did not participate in any other project activities 76.2 3,569 ns 30.6 3,569 ns Participated in other activities 79.3 114 22.3 114 Department San Marcos 74.4 1,676 ns 31.5 1,676 ns Totonicapan 78.1 2,007 29.1 2,007 Survey round 2013 79.6 2,641 * 34.6 2,641 * 2018 73.8 1,042 26.6 1,042 All children under five 76.4 3,683 30.2 3,683 *** p<0.001, ** p<0.01, * p<0.05, + p<0.10 Not stunted Stunted Sig. Not underweight Underweight Sig. Household size 7.6 8.0 ns 7.9 7.9 ns Household dietary diversity score 6.7 6.4 *** 6.5 6.3 † Average daily per capita consumption expenditures (constant 2010 USD) $2.33 $1.91 *** $2.10 $1.79 *** Number of project activities in which household participated 0.9 0.9 ns 1.0 0.7 * *** p<0.001, ** p<0.01, * p<0.05, + p<0.10 Table 7.2. Relationship between prevalence of children's stunting and underweight and select household characteristics, CRS￾SEGAMIL [Guatemala 2013, 2018] Stunting Underweight VARIABLE OR 95% CI OR 95% CI OR 95% CI OR 95% CI OR 95% CI OR 95% CI OR 95% CI OR 95% CI Survey round (ref.:2013) 2018 0.721* 0.559 - 0.930 0.735* 0.572 - 0.945 0.714** 0.554 - 0.920 0.725* 0.566 - 0.929 0.792+ 0.616 - 1.019 0.920 0.711 - 1.189 0.795 0.578 - 1.093 0.869 0.619 - 1.218 Household WASH status Household has water and soap at a commonly used handwashing station (ref.: No) 0.676* 0.501 - 0.912 0.666** 0.497 - 0.893 0.678** 0.507 - 0.906 0.733* 0.538 - 0.998 0.733+ 0.537 - 1.001 0.769 0.522 - 1.133 Household treats water (ref.: No) 0.611 0.308 - 1.215 0.612 0.308 - 1.214 0.602 0.301 - 1.205 0.574 0.274 - 1.204 0.547 0.259 - 1.155 0.538 0.242 - 1.197 Household agriculture status Obtained agricultural credit (ref.: No) 1.441 0.795 - 2.612 1.436 0.796 - 2.592 1.466 0.822 - 2.614 1.539 0.874 - 2.710 1.561 0.887 - 2.747 1.389 0.787 - 2.450 Used at least two sustainable crop practices (ref.: No) 1.238* 1.029 - 1.490 1.263* 1.052 - 1.516 1.226* 1.022 - 1.470 1.237* 1.034 - 1.480 1.198+ 0.997 - 1.439 1.292* 1.015 - 1.644 Used at least two sustainable NRM practices (ref.: No) 0.606* 0.400 - 0.918 0.602* 0.384 - 0.942 0.642+ 0.407 - 1.012 0.607* 0.382 - 0.966 0.619* 0.396 - 0.970 0.671+ 0.422 - 1.067 Household food consumption Fruits 0.866 0.704 - 1.066 0.949 0.763 - 1.180 0.943 0.756 - 1.175 1.001 0.787 - 1.272 Meat 0.805+ 0.639 - 1.015 0.905 0.716 - 1.145 0.907 0.718 - 1.147 0.969 0.752 - 1.249 Eggs 0.850 0.680 - 1.062 0.912 0.733 - 1.135 0.914 0.733 - 1.141 0.876 0.684 - 1.123 Milk 0.758+ 0.567 - 1.015 0.880 0.658 - 1.175 0.899 0.663 - 1.220 0.707* 0.523 - 0.956 Household poverty status Natural logarithm of daily per capita consumption expenditures in constant 2010 USD 0.509*** 0.377 - 0.687 0.508*** 0.377 - 0.685 0.508*** 0.351 - 0.735 Project-level factors Participated in at least one activity regularly (ref.: did not participate in any project activities) 1.313+ 0.970 - 1.779 1.025 0.758 - 1.387 Child's characteristics Child's sex (ref.: male) Female 0.879 0.624 - 1.239 Child's age in years (ref. less than 6 months) 6-11 1.783** 1.158 - 2.745 12-23 9.042*** 5.697 - 14.351 24-35 8.493*** 6.109 - 11.808 36-47 6.507*** 4.330 - 9.780 48-59 4.287*** 3.158 - 5.821 Child had diarrhea (ref.: No) 1.089 0.821 - 1.444 Caregiver's educational level (ref.: None) Primary or less 0.849 0.639 - 1.128 Secondary, vocational, or higher 0.495** 0.320 - 0.766 Number of adult females 0.853* 0.754 - 0.965 Number of children under five other than child (ref.: none) One 1.341* 1.006 - 1.787 Two 1.349 0.866 - 2.102 Three 2.096* 1.127 - 3.901 Four or more 1.499 0.628 - 3.578 Department (ref.: San Marcos) Totonicapán 1.395* 1.064 - 1.828 Constant 3.901*** 3.230 - 4.712 8.455*** 4.213 - 16.968 3.450*** 2.860 - 4.162 7.484*** 3.741 - 14.975 9.403*** 4.515 - 19.582 11.199*** 5.197 - 24.132 11.929*** 5.501 - 25.865 3.100* 1.227 - 7.833 N 3,683 3,683 3,683 3,683 3,683 3,683 3,683 3,682 *** p<0.001, ** p<0.01, * p<0.05, + p<0.10 Model 5 Model 6 Model 7 Model 8 Table A7.3. Results of logistic regression of the prevalence of moderate-to-severe stunting, CRS-SEGAMIL [Guatemala 2013, 2018] Model 1 Model 2 Model 3 Model 4 VARIABLE OR 95% CI OR 95% CI OR 95% CI OR 95% CI OR 95% CI OR 95% CI OR 95% CI OR 95% CI Survey round (ref.:2013) 2018 0.685* 0.513 - 0.915 0.712* 0.542 - 0.935 0.687* 0.513 - 0.919 0.715* 0.542 - 0.941 0.741* 0.560 - 0.981 0.845 0.633 - 1.128 0.996 0.694 - 1.429 1.079 0.741 - 1.571 Household WASH status Household has improved sanitation facility (ref.: No) 0.812+ 0.652 - 1.010 0.807+ 0.649 - 1.005 0.814+ 0.658 - 1.005 0.837+ 0.677 - 1.034 0.837+ 0.679 - 1.032 0.850 0.681 - 1.062 Household has water and soap at a commonly used handwashing station (ref.: No) 0.708** 0.561 - 0.895 0.708** 0.559 - 0.897 0.721** 0.571 - 0.911 0.765* 0.611 - 0.958 0.766* 0.610 - 0.961 0.785+ 0.607 - 1.015 Household treats water (ref.: No) 0.623+ 0.355 - 1.094 0.632 0.361 - 1.108 0.608 0.332 - 1.111 0.587+ 0.335 - 1.027 0.615+ 0.355 - 1.065 0.650 0.374 - 1.129 Household agriculture status Used at least two sustainable NRM practices (ref.: No) 0.352** 0.162 - 0.765 0.348** 0.159 - 0.762 0.375* 0.177 - 0.797 0.364** 0.174 - 0.764 0.360** 0.173 - 0.752 0.356** 0.173 - 0.732 Household food consumption Fruits 0.867 0.699 - 1.074 0.938 0.755 - 1.166 0.944 0.763 - 1.167 1.018 0.815 - 1.270 Meat 0.788* 0.648 - 0.959 0.861 0.699 - 1.059 0.857 0.701 - 1.047 0.886 0.699 - 1.123 Fish 1.693** 1.168 - 2.455 1.712** 1.198 - 2.448 1.739** 1.202 - 2.517 1.921** 1.307 - 2.822 Milk 0.829 0.588 - 1.170 0.932 0.665 - 1.306 0.905 0.642 - 1.276 0.868 0.603 - 1.251 Household poverty status Natural logarithm of daily per capita consumption expenditures in constant 2010 USD 0.589*** 0.462 - 0.751 0.595*** 0.462 - 0.764 0.619*** 0.477 - 0.804 Project-level factors Participated in at least one activity regularly (ref.: did not participate in any project activities) 0.739 0.468 - 1.166 0.653* 0.426 - 1.000 Child and household characteristics Child's sex (ref.: male) Female 0.876 0.698 - 1.099 Child's age in years (ref. less than 6 months) 6-11 1.598+ 0.969 - 2.636 12-23 4.181*** 2.935 - 5.956 24-35 2.758*** 1.812 - 4.196 36-47 2.559*** 1.642 - 3.990 48-59 2.110*** 1.410 - 3.157 Child had diarrhea (ref.: No) 1.095 0.903 - 1.329 Caregiver's educational level (ref.: None) 0.866 0.683 - 1.098 Primary or less 0.542** 0.345 - 0.853 Secondary, vocational, or higher Number of children under five other than child (ref.: none) 1.328* 1.063 - 1.658 One 1.072 0.776 - 1.482 Two 1.207 0.711 - 2.047 Three 1.572 0.547 - 4.520 Four or more Department (ref.: San Marcos) 0.846 0.660 - 1.083 Totonicapán Constant 0.529*** 0.440 - 0.637 1.192 0.637 - 2.232 0.537*** 0.446 - 0.646 1.196 0.641 - 2.230 1.339 0.696 - 2.578 1.500 0.815 - 2.763 1.429 0.779 - 2.621 0.549 0.254 - 1.186 N 3,683 3,683 3,683 3,683 3,683 3,683 3,683 3,682 Model 6 Model 7 Model 8 Table A7.4. Results of logistic regression of the prevalence of moderate-to-severe underweight for children five, CRS-SEGAMIL [Guatemala 2013, 2018] Model 1 Model 2 Model 3 Model 4 Model 5 Annex 8: Tables for Multivariate Analyses of Per Capita Consumption Expenditures Table A8.1. Results of bivariate analyses of daily per capita consumption expenditures , CRS-SEGAMIL [Guatemala 2013, 2018] Baseline 2013 Endline 2018 BL-EL sample constant 2010 USD Sig. constant 2010 USD Sig. constant 2010 USD Sig. Household sociodemographic characteristics Household head sex Male $1.89 *** $2.37 ns $2.17 ** Female $2.23 $2.44 $2.36 Household head age 18-24 $2.02 ** $2.53 * $2.29 *** 25-34 $2.03 $2.35 $2.21 35-44 $1.83 $2.30 $2.10 45-54 $1.81 $2.24 $2.05 55-64 $2.01 $2.57 $2.36 65+ $2.18 $2.60 $2.46 Household head educational level Never attended school $1.78 ** $2.25 *** $2.07 *** Primary or less $1.92 $2.37 $2.17 Secondary, vocational/technical or higher $3.08 $3.47 $3.32 Number of adult males 0 $2.51 *** $2.67 *** $2.61 *** 1 $2.03 $2.51 $2.31 2 $1.84 $2.26 $2.08 3 $1.62 $2.19 $1.98 4 $1.71 $2.09 $1.92 5 or more $0.88 $1.80 $1.50 Number of adult females 1 $2.08 *** $2.46 *** $2.29 *** 2 $1.95 $2.45 $2.25 3 $1.65 $2.37 $2.12 4 $1.56 $2.16 $1.90 5 $1.29 $1.86 $1.70 6 or more $1.00 $1.62 $1.52 Number of children under five None $2.44 *** $2.82 *** $2.66 *** 1 $1.93 $2.41 $2.21 2 $1.55 $1.99 $1.82 3 $1.38 $2.13 $1.79 4 or more $1.32 $1.88 $1.66 Number of children 5-17 0-1 $2.92 *** $3.20 *** $3.10 *** 1 $2.40 $2.80 $2.65 2 $2.09 $2.65 $2.42 3 $1.99 $2.25 $2.15 4 $1.67 $1.91 $1.81 5 $1.44 $1.79 $1.59 6 or more $1.33 $1.88 $1.61 Gendered household type Adult male and female $1.89 *** $2.36 *** $2.16 *** Adult female only $2.51 $2.67 $2.61 Adult male only $3.18 $4.05 $3.49 Household agriculture status Use of credit Did not use credit $1.92 ns $2.38 ** $2.19 ns Used credit $2.13 $3.53 $2.29 Use of savings Did not make any savings $1.89 ** $2.37 ns $2.18 * Made savings $2.46 $2.74 $2.56 Use of value chain activities (any two) Did not use value chain activities $1.88 * $2.34 + $2.14 *** Used value chain activities $2.27 $2.52 $2.46 Use of at least two sustainable crop practices Did not use at least two sustainable crop practices $1.92 + $2.37 + $2.16 + Used at least two sustainable crop practices $1.93 $2.39 $2.22 Use of at least two sustainable livestock practices Did not use at least two sustainable livestock practices $1.92 + $2.34 ** $2.15 *** Used at least two sustainable livestock practices $2.05 $2.52 $2.41 Use of at least two sustainable NRM practices Table A8.1. Results of bivariate analyses of daily per capita consumption expenditures , CRS-SEGAMIL [Guatemala 2013, 2018] Baseline 2013 Endline 2018 BL-EL sample constant 2010 USD Sig. constant 2010 USD Sig. constant 2010 USD Sig. Did not use at least two sustainable NRM practices $1.93 ns $2.37 ns $2.19 ns Used at least two sustainable NRM practices $1.90 $2.84 $2.54 Project-level factors Household participated in at least one project activity regularly Household did not participate in at least one project activity $1.93 $2.53 ** $2.20 ns Household participated in at least one project activity n/a $2.19 $2.19 Type of intervention Food rations Did not receive food rations $1.93 $2.52 ** $2.20 ns Received food rations n/a $2.19 $2.19 Nutrition training/meetings Did not participate in nutrition trainings/meetings $1.93 $2.52 ** $2.20 ns Participated in nutrition trainings/meetings n/a $2.18 $2.18 Agriculture training/meetings Did not participate in agriculture trainings/meetings $1.93 $2.53 ** $2.21 ns Participated in agriculture trainings/meetings n/a $2.15 $2.15 Other activities Did not participate in any other project activities $1.93 $2.39 ns $2.19 ns Participated in other activities n/a $2.28 $2.28 Number of project activities in which HH regularly participated 0 $1.93 $2.53 + $2.20 ns 1 n/a $2.04 $2.04 2 n/a $2.50 $2.50 3 n/a $2.16 $2.16 4 n/a $2.13 $2.13 5 n/a $2.03 $2.03 Department San Marcos $2.11 ns $2.45 $2.27 ns Totonicapán $1.69 $2.35 $2.13 Total $1.93 $2.38 $2.20 *** p<0.001, ** p<0.01, * p<0.05, + p<0.10 Table A8.2. Results of OLS regression of the natural logarithm of daily per capita consumption expenditures (2010 constant USD at 2005 PPP), CRS-SEGAMIL [Guatemala 2013, 2018] VARIABLES Coef. 95% CI Coef. 95% CI Coef. 95% CI Coef. 95% CI Coef. 95% CI Survey round (ref.:2013) 2018 0.275*** 0.184 - 0.366 0.267*** 0.174 - 0.360 0.287*** 0.192 - 0.383 0.334*** 0.224 - 0.443 0.268*** 0.165 - 0.371 Household agriculture status Accessed agricultural credit in last 12 mos. 0.123+ -0.000 - 0.245 0.100 -0.020 - 0.220 0.097 -0.023 - 0.217 0.120* 0.009 - 0.230 Made savings in the last 12 mos. 0.178** 0.063 - 0.293 0.157** 0.043 - 0.272 0.155** 0.040 - 0.270 0.139** 0.037 - 0.241 Used (at least two) value chain activities(ref.: No) 0.095** 0.036 - 0.153 0.103*** 0.045 - 0.161 0.097** 0.040 - 0.153 0.088** 0.029 - 0.147 Used at least two sustainable crop practices (ref.: No) 0.015 -0.048 - 0.077 -0.014 -0.069 - 0.041 -0.003 -0.057 - 0.052 0.001 -0.049 - 0.051 Used at least two sustainable livestock practices (ref.: No) 0.062* 0.008 - 0.117 0.085** 0.035 - 0.135 0.103*** 0.050 - 0.156 0.070** 0.017 - 0.123 Department (ref.: San Marcos) Totonicapán -0.122* -0.215 - -0.029 -0.125** -0.217 - -0.032 -0.079+ -0.167 - 0.010 Project-level factors Participated in at least one activity regularly (ref.: did not participate in any project activities) -0.115** -0.199 - -0.031 -0.001 -0.070 - 0.068 Household sociodemographic characteristics Household head age (continuous; 17 - 94 years) 0.006*** 0.005 - 0.008 Female-headed household (ref.: Male) 0.064+ -0.002 - 0.129 Household head educational level (ref.: none) Primary or less 0.092*** 0.048 - 0.136 Secondary, vocational/technical or higher 0.414*** 0.312 - 0.516 Gendered household type (ref.: adult male and adult female) Adult female only -0.025 -0.122 - 0.072 Adult male only 0.157+ -0.017 - 0.331 Household size (continuous; 1 -32) -0.061*** -0.070 - -0.052 0.477*** 0.412 - 0.542 0.433*** 0.362 - 0.504 0.502*** 0.421 - 0.584 0.497*** 0.414 - 0.579 0.519*** 0.384 - 0.654 Constant 4,264 4,264 4,264 4,264 4,264 N 0.065 0.080 0.092 0.098 0.279 R-squared *** p<0.001, ** p<0.01, * p<0.05, + p<0.10 Model 1 Model 2 Model 3 Model 4 Model 5 VARIABLES Coef. 95% CI Survey round (ref.:2013) 2018 0.259*** 0.172 - 0.347 Household agriculture status Accessed agricultural credit in last 12 mos. 0.073+ -0.002 - 0.149 Made savings in the last 12 mos. 0.146*** 0.067 - 0.226 Used (at least two) value chain activities(ref.: No) 0.064** 0.018 - 0.110 Used at least two sustainable crop practices (ref.: No) 0.003 -0.044 - 0.049 Used at least two sustainable livestock practices (ref.: No) 0.059+ -0.006 - 0.124 Department (ref.: San Marcos) Totonicapán -0.057 -0.141 - 0.027 Project-level factors Participated in at least one activity regularly (ref.: did not participate in any project activities) 0.016 -0.038 - 0.069 Household sociodemographic characteristics Household head age (continuous; 17 - 94 years) 0.007*** 0.005 - 0.008 Female-headed household (ref.: Male) 0.060* 0.001 - 0.120 Household head educational level (ref.: none) Primary or less 0.108*** 0.068 - 0.148 Secondary, vocational/technical or higher 0.380*** 0.292 - 0.468 Gendered household type (ref.: adult male and adult female) Adult female only -0.013 -0.109 - 0.083 Adult male only 0.300*** 0.185 - 0.415 Household size (continuous; 1 -32) -0.074*** -0.080 - -0.068 Constant 0.591*** 0.465 - 0.716 N 4,028 0.356 R-squared 0.356 *** p<0.001, ** p<0.01, * p<0.05, + p<0.10 Table A8.3. Results of OLS regression of the natural logarithm of daily per capita consumption expenditures (2010 constant USD at 2005 PPP) excluding influential cases, CRS-SEGAMIL [Guatemala 2013, 2018] NOTE: Model excludes influential observations that can affect the estimation of the coefficients. Cook's D was used to identify cases (households) with high residuals and leverage using the convention cut-off point 4/n where n is the number of observations (in this case 4/4,264). A total of 158 households from the baseline sample and 78 households from the endline sample were flagged as influential observations and were excluded from this analysis. Annex 9: Data Collection Instruments Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 1 1. Focus Group Discussion Guides for Program Participants Separate FGDs will be held to address: 1) nutrition and health and 2) agriculture and livelihoods. The general questions are asked of both. General Questions: 1. What SEGAMIL activities have you participated in? a. Probe: any others? 2. Which project activities made the biggest positive changes in your life? a. Please explain why. What examples of success can you share? b. What have been the challenges you overcame to be successful? c. Probe: do others have different experiences? Please share your experiences. 3. What aspects of the project did you not like? a. Please explain why. What examples can you share? b. What have been the challenges that were difficult to overcome? c. Probe: do others have different experiences? Please share your experiences. 4. Are your food and health conditions better or worse now than you had 4 or 5 years ago? a. Please explain why. What examples can you share? b. What has changed? c. Do others have a different experience? 5. Did any SEGAMIL activities help you increase your family income? Which ones helped the most? 6. Did all or only some households in your community benefit from the project? a. Please explain why or why not. b. Who benefited the most? 7. After the SEGAMIL project ends, do you think you will be able to continue the activities you participated in? a. Which activities will you continue? i. Why will you be able to continue the activities? b. Which activities will you not continue? i. What will prevent you from being able to continue the activities? Questions for the Nutrition and Health group Nutrition and Health 1. Before SEGAMIL, describe your typical diet (the foods you eat)? a. How did you provide food for your family? b. Are there times during the year where feeding your family is more difficult? ii. i. When? How were you able to feed your family during these times? 2. How did you like the foods that SEGAMIL provided you with? a. Did you like the flavor/taste, texture, quality, etc. of the ration mix? b. Was it enough to meet the food needs of your family? Was it too much? c. What foods would you eliminate? Why? d. PLEASE BE HONEST: Did you ever have the need to trade or sell part of your food assistance to meet other needs of your family? 3. How did you like the voucher distribution compared to direct commodity distribution? 4. After SEGAMIL, do you feel you will have skills to grow enough food to feed your family? a. Describe the skills you learned. b. Do you receive any external assistance from any other public or private sources? Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 2 i. Who provides the assistance? ii. What assistance is provided? c. Do you have a family member who has migrated and is sending financial support? i. If so, how do you use those resources? Are you able to invest anything or does it go entirely to support the family? 5. Has your family’s diet changed for the better or worse in the past few years? a. How has it changed? b. Why has it changed? 6. What kinds of foods do you eat more of now? a. What helped you to be able to eat these foods? 7. What can pregnant women do to improve their health? (Probe: any others?) a. What makes it difficult for women to do these things? b. Did the SEGAMIL project make it easier for women to practice these behaviors? How? Give examples. 8. What can families do to improve the health and nutrition of their babies? (Probe: any others?) a. What makes it difficult for families to do these things? 9. Did the SEGAMIL project make it easier for women to practice these behaviors? How? Give examples 10. Did your family prepare a “Healthy Home Plan”? a. Were these plans helpful? Can you give examples of how it helped your family? b. What difficulties did you face in implementing the plans? Homestead Gardens and Livestock 1. How many of you have a home garden? a. Did you have a homestead garden before SEGAMIL? b. What do you like best about the garden? c. How has the homestead garden affected your life? d. How much food do you consume? i. Did you learn to prepare them or did you already know how to do it? ii. Do your children like to eat vegetables from the garden? 1. Which vegetables do they like most? 2. Which vegetables do they like least? e. How much produce from the garden do you sell? i. How do you get income? ii. Where do you sell? iii. How often do you sell? f. Who tends the garden? i. Does anyone else help? If yes who helps? ii. How much time and effort does it require to maintain the garden? g. Describe any pest problems you might have. h. What do you recommend SEGAMIL do differently in the future? 2. How many of you have livestock? a. Did you have livestock before SEGAMIL? i. Did you increase the number of livestock thanks to SEGAMIL? b. What do you like best about having livestock? c. In what ways has having animals affected your life? d. How much livestock do you consume? i. What products do you consume? ii. Did you learn to prepare them or did you already know how to do it? Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 3 iii. Do your children like consuming livestock (e.g., meat, milk, organs)? e. How much livestock do you sell? i. How do you get income? ii. Where do you sell? iii. How often do you sell? f. Who tends the livestock? i. How much time and effort does it require to tend the livestock? ii. Does anyone else help? If yes who helps? iii. How do you care for your animals when they get sick? g. Describe any pest problems you may have. h. Anything you would like different? 3. Do you have resources needed to continue? 4. Do you consider that there have been changes in your family's nutrition by diversifying the foods you eat? WASH Drinking Water 1. What community water supply activities were implemented during SEGAMIL? a. Which activities did you participate in? b. Did the activities achieve what they were supposed to? i. Why? c. What activities worked well? d. What challenges did you face implementing the activities? i. Why were they difficult? e. What good changes or bad changes have the water supply activities made in your life? i. Why? ii. How will you sustain those good changes once SEGAMIL has ended? f. What advice do you have for implementing water supply activities in future? 2. Who owns the water supply system? 3. How is the water supply system managed? a. What are the roles and responsibilities of the Water Management Committee (WMC)? b. What motivates someone to participate in the WMC? c. What will ensure the sustainability of the roles and responsibilities of the WMC? d. What challenges will the WMC experience once SEGAMIL has ended? i. What external support will they need to continue? ii. How will they obtain that support? e. Has your water system been broken or non-functional in the previous two years? i. If so, for how long? ii. By what process was the system repaired? iii. Who paid for the repair? 1. Do you contribute money for repairs? 2. How much do you contribute? 3. Do all households pay the same amount? Is anyone exempted from paying for water? iv. When are fees collected? v. Do you know where the money goes and what it is used for? f. Where can spare parts be purchased? i. Who purchases them? 4. Did you receive a water filter from SEGAMIL? a. If yes, what type of filter is it? Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 4 b. What training on its use and maintenance was conducted? i. Who conducted the training? ii. Do you feel you have the skills to continue using and maintaining the filter? Why or why not? c. Where can you find support if you need it? Sanitation 5. What household sanitation activities were implemented during SEGAMIL? a. Which activities did you participate in? b. Did the activities achieve what they were supposed to? i. Why? c. What worked well when implementing the sanitation activities? d. What challenges did you face implementing those activities? i. Why were they difficult? e. What good changes or bad changes have the sanitation activities made in your life? i. Why? ii. How will you sustain those good changes once SEGAMIL has ended? f. What advice do you have for implementing future sanitation activities? 6. Were you provided a latrine during SEGAMIL? a. If no, where do you defecate? b. If yes, what did SEGAMIL provide — technical support, financial support, education? c. When did you install this latrine/toilet? d. Were you provided any subsidy for this latrine/toilet when it was installed? e. What maintenance does it require, and how often? i. What did you pay or provide for the latrine construction? i. Do you have any problems with the latrine during the rainy season? f. Are there times that you or your family members are not able to use this latrine? i. If yes, when? ii. What do you do when this happens? g. How satisfied are you with this latrine/toilet? i. Does it meet the needs of your family? ii. Do you have any improvements planned you would like to complete for the toilet/latrine? 7. If there are children in the household: The last time a child passed stools, what was done to dispose of the stool? a. At what age did/will your child begin to use a latrine? 8. If you participated in Participatory Hygiene and Sanitation Transformation (PHAST) activities, which hygiene behaviors do you currently practice? a. What changes have the sanitation and hygiene activities caused you to make in your life? i. Why? ii. How will you sustain those changes once SEGAMIL has ended? iii. What challenges will you have continuing the practices once SEGAMIL ends? Handwashing Stations 1. Do you have a hand washing station [in your home/near latrine/near kitchen]? a. If yes, describe the handwashing station. i. Where do you get water for hand washing and personal hygiene? b. If no, where do you wash your hands? Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 5 Questions for agriculture and livelihoods group Sustainable Agriculture 1. Please describe what agriculture practices you have learned from SEGAMIL? a. Any others? 2. Which agriculture practices do you practice on your land? a. Any others 3. Which practices have you decided not to practice? Why? 4. How has your production changed as a result of these practices? a. Are you planting different kinds of foods? Which ones? b. Has the yield changed? c. Have you noted any other changes? 5. What challenges have you experienced in implementing these practices? 6. What do you think of the post-harvest practices promoted by SEGAMIL? a. Is anyone using these practices? Why or why not? b. What difficulties have you faced in trying to use these practices? 7. What animal rearing practices have you learned from SEGAMIL? 8. Which animal rearing are you practicing in your household? 9. Which animal rearing have you decided not to practice? Why? 10. Have these practices made any changes to the number of animals you keep? 11. How have these practices affected your household? a. Have they affected your family in any other way? b. Which practices have made the most difference in your lives? 12. Thinking of all the agriculture and animal practices, which ones will you continue using in the future? a. Which ones will you stop using? Why? 13. What support did you get from SEGAMIL for the agricultural practices? a. Probe: training? Equipment? Materials? Seeds? b. Do you still need this support to continue using these practices? 14. How will you get this support in the future? Natural Resource Management 1. What activities are you practicing to improve your natural resources? a. What practices are you using on your own land? b. What practices are used in communal/public land? 2. How has SEGAMIL supported these activities> 3. Do you plan on continuing these activities in the future? Why or why not? 4. What activities are you implementing to improve your water resources? 5. How has SEGAMIL supported these activities? 6. Do you plan on continuing these activities in the future? Why or why not? 7. How has SEGAMIL supported you in understanding your natural resources? 8. Have you or your community developed an NRM plan? a. Who is responsible for implementing it? b. How has SEGAMIL supported you with preparing your NRM plan and in implementing it? c. Where else did you learn about these activities? 9. What resources are needed for your to improve your natural resources? a. What local resources are needed? b. What outside resources are needed? Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 6 c. How will you acquire these outside resources? 10. Describe who else (i.e., ECADI, municipality, OMAS, other) is supporting you to implement your NRM plan? Describe the support they provide. 11. What is your strategy for carrying through with your NRM plan once SEGAMIL has been completed? 12. What suggestions do you have for a similar food security project in the future? a. What activities should be implemented again? Why? b. Which activities should be changed? Why? c. Which activities should be eliminated? Why? Value Chains 1. Did anyone participate in any value chain activities? a. If so, which ones? If not, why not? 2. How has this activity benefited your household? 3. What were your biggest challenges? 4. Are you happy with the way your group is organized and managed? 5. Will you continue participating in this value chain after the project? a. Why or why not? 6. What support did you receive from SEGAMIL for this? a. Training? Equipment? Materials? 7. What is needed for your group to continue participating in value chain in the future? 8. Is your group receiving support from any other institution or project? a. What type of support? b. Who is providing it? c. Is this support sufficient for your group to continue? d. Do you have formal agreements with these organizations? 9. If you are earning income from value chain participation, how do you spend it? 10. Who participated more in the value chain: Men? Women? The whole family? 11. Was opportunity given to women producers if they wanted to participate? If yes: In which value chains did they participate the most? If it is not: Why do you think you were not allowed to participate? Saving and Loans 1. Did you participate in a savings and loans group? a. How did it work for you? Did you borrow money? What did you use it for? b. Did you ever save money before SEGAMIL? If so, how? 2. What are the benefits to your family of participation in the savings and loan group? 3. Are there any negative effects of participating in these groups? 4. Will you continue to participate in the future? Why or why not? 5. Do you have enough training to borrow and investing loans in the future? Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 7 2. Focus Group Discussion Guide for Community Organizations Supported by SEGAMIL Promoters Group General Questions 1. Ask each participant to describe what type of promoter they are. 2. What are the greatest needs of your community? 3. What do you think the SEGAMIL project has been trying to help your community achieve? a. What are the objectives of the SEGAMIL project? 4. In your opinion how has the project helped your community? a. What has the project achieved? b. What were some of the most important results? 5. Were there any negative effects of the project? 6. What SEGAMIL activities were the most successful? Why? 7. What SEGAMIL activities were the least successful? Why? 8. Do you know how SEGAMIL selected participants for its project activities? a. Do you think the selection of program participants was fair? Why or why not? b. Did SEGAMIL consider the needs of the people? 9. What did the community think of the activities that SEGAMIL implemented? 10. What is your role in the SEGAMIL project? a. What are your responsibilities? b. Please describe the work that you do. 11. What practices do you promote in the community? a. Any others? 12. What behavior change have you observed in the families with the SEGAMIL activities and especially in the area that you supported? 13. Which practices have been easiest to change? 14. Which practices have been the most difficult to change? a. Why? 15. Please tell me about the healthy home plans. a. Did these plans work? b. What difficulties did families have in implementing them? 16. What support did you receive from SEGAMIL to promote the practices? a. Please describe how you worked with SEGAMIL? b. What did SEGAMIL provide to you to help you do your job? 17. Was the support received what you needed to make the activities possible? 18. What do you think about the training you received? 19. What additional training would you have liked to have received? a. Describe what training topics would have helped improve your work? 20. What additional support would you have liked to get? a. Do you think you got enough training to do your job? Materials? Equipment? 21. What challenges have you faced in the implementation of your activities? 22. What have been the benefits of working with SEGAMIL? What have been the difficulties of working with SEGAMIL? 23. Have you, as promoters, benefited from collaboration with SEGAMIL? What are these benefits? a. Do you receive any economic compensation from your promotion work? 24. Do you work with or receive support from any government offices to carry out your promotion activities? a. Please describe which offices and what support you get. Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 8 b. Would this relationship still exist without SEGAMIL? 25. Do you think municipal authorities will provide support to your activities when SEGAMIL is no longer there? a. How will you get this support? 26. How many of you have received certification through the SEGMAIL project? a. What is the certification you received? b. Has this certification helped you in your work? How has it helped? 27. Do you think you will continue carrying out promotional activities in your community after SEGAMIL ends? a. Why or why not? b. What do you need to continue doing this work? 28. Do you think it is better to have men or women be community promoters? Is it the same for all sectors? 29. Please tell us how well you think these interventions worked in your community: For each intervention ask what successes have they had? What negative impact did they have? Who benefited the most from these activities? a. ECADIs b. Savings and loans groups c. Natural resource management d. Food voucher system e. Sustainable agriculture activities f. Activities to improve water sources g. Activities to improve sanitation and hygiene h. Economic development activities i. Famer or producer groups Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 9 3. Interview Guide for Community Organizations Supported by SEGAMIL General Questions ASKED OF ALL COMMITTEES 1. What are the greatest needs of your community? 2. What is the role of this committee in your community? a. How does the committee support the community? 3. How long has the committee been functioning? a. How many members are in the committee? b. How many women are in this committee? How many men? c. How do you become a member of the committee? 3. Who decides the activities of this committee? a. How are those decision made? 4. What do you think the SEGAMIL project has been trying to help your community achieve? a. What are the objectives of the SEGAMIL project? 5. In your opinion how has the project helped your community? a. What else has the project achieved? b. What were some of the most important results? 6. What SEGAMIL activities were the most successful? Why? 7. What SEGAMIL activities were the least successful? Why? 8. Do you know how SEGAMIL selected participants for its project activities? a. Do you think the selection of program participants was fair? Why? b. Did SEGAMIL consider the needs of the people? 9. What are the priorities for this committee? 10. Do you think the SEGAMIL project helped address the priorities of this committee? a. Can you provide examples? 11. How did this committee work together with SEGAMIL? a. How did the committee leaders work with SEGAMIL? b. What were the institutional arrangements for implementation of the project between you and SEGAMIL? 12. Did the SEGAMIL program support this committee? a. How did they support this committee? b. Please describe any trainings or assistance the project provided to this committee? 13. What support did this committee receive for SEGAMIL to promote its activities? a. What else did the project provide to help this committee do its work? b. How helpful was this support? 14. What restrictions / challenges have you faced, as a committee, in the implementation of your activities? 15. What type of support does the committee need in the future to carry out its activities? 16. How will this committee continue to work after the SEGAMIL project ends? a. Will you able to continue with all the same activities? Why or why not? b. Which activities will continue? Which will not? 17. Are there other organizations or projects that will support your committee in the future? a. Which organizations? b. Have you already started working with these organizations? c. Do you have formal agreements with them? d. What type of support have they given your committee? 18. Has the SEGAMIL project supported you to create institutional agreements with any government offices? Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 10 a. Why which offices? b. Please describe those agreements. Additional questions for specific committees Questions for ECADI Leaders 1. Were the activities and trainings carried out by the ECADI those that are really needed in the community? Did the beneficiaries agree with these? 2. What are the biggest successes of the ECADI? a. What behavioral changes have you had? b. How has the work of the ECADI affected the community? 3. Who planned the trainings activities? 4. Who was responsible for providing the trainings? a. Anyone else? b. How many people provided training activities 5. Do you think the promoters were well prepared to provide trainings to the community? a. Do you think they need more training and support to do their work? 6. How were the groups receiving the trainings selected or organized? a. Who participated more in the ECADI? Men? Women? Families? 7. Was it ever difficult to get people to participate in the activities of the ECADI? Why? 8. How did the different promoters work together to plan the trainings? 9. What activities do you do in the ECADI in addition to the training? Which of these activities do you think were most successful? 10. Which training topics were most popular (better received)? a. Which were less well received by the community? 11. What are the main challenges that the ECADI faces? 12. Will you continue training community members in the future? a. Who will be invited to participate in the trainings? 13. Do you have any formal agreements with institutions get technical assistance for the ECADI? What about financial support? 14. How will the ECADI support the trainers/promoters in the future? 15. Have the promoters benefited from collaboration with SEGAMIL? a. What are these benefits? 16. Is the ECADI part of the COCODE? Why or why not? 17. Do you think the certification of agricultural and health promoters will be helpful? In what ways? Questions for COCODES Leaders: 1. What committees are formed and supported by COCODE? Could you list each one? 2. Which committees have been strengthened with the intervention of the project? Give some examples of work done by the committee. 3. What is the relationship between these community committees and municipal committees (COMUDES, COMUSAN, etc.)? What type of support do the municipal committees provide? 4. Has your organization/committee benefited from collaboration with SEGAMIL? 5. Do you think you are more or less able to serve your community now than you were 4 years ago? Why? 6. What are the challenges that your organization/committee encounters working with SEGAMIL? 7. Please tell us how well you think these interventions worked in your community: For each intervention ask what successes have they had? What negative impact did they have? Who benefited the most from these activities? Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 11 a. ECADIs b. Savings and loans groups c. Natural resource management d. Food voucher system e. Sustainable agriculture activities f. Activities to improve water sources g. Activities to improve sanitation and hygiene h. Economic development activities i. Famer or producer groups Savings and Loan Group Leaders: 1. What was the process of forming the savings and loan group? 2. How were the members selected? a. Can men and women participate? b. Is participation individual or as a family? 3. What type of training the members receive as part of this group? 4. Will the group add new members in the future? a. How will they be selected? b. Who decides who can become a member? 5. How will you train new members? a. Who will train them? b. What support will you receive in the future to support the SIL group? c. Have you worked with any municipal or departmental organizations? d. Do you have formal agreements with these or other organizations for future support? 6. How many loans do you give out each month? 7. What are the loans used for? 8. Have you had problems getting members to pay back loans? a. Why? 9. Have there been difficulties with the loan process? Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 12 4. Interview Guides for SEGAMIL Program Staff Interview Guide for CRS Management 1. What are some of the most tangible, positive changes you have witnessed in household and community food security as a result of the project’s interventions? a. Are you satisfied with these changes or could more have been accomplished? b. What would have been need to reach this added-value? 2. What is your perception of the positive impacts of the project? Negative impacts? 3. For each of the three SOs: 1. Which strategies were most successful? Why? 2. What factors facilitate the achievement of your targets? 3. Which strategies were least successful? Why? 4. What were the main challenges that affected the implementation of the program? 4. Overall were you able to achieve the targets you set for the program? Were there any that were not met? If so which ones? 5. What were the primary constraints or challenges faced in the implementation of the program? a. How did they affect the ability to achieve results? b. What sort of programmatic adjustments did you have to make? 6. SEGAMIL is a multi-sector program design to address the multiple causes of food insecurity in the target geographies: a. How successful do you feel you were in integrating the different elements of the project? b. What actions enhanced integration and cross-sector collaboration within the project? c. What were some of the challenges? (i.e. getting aggies to work/communicate with) nutritionists? d. Were there any gender dynamics within the project staff that either facilitated or hindered the project implementation? 7. Was the shift in strategy following the mid-term evaluation beneficial or a hindrance to achieving program goals? Why? 8. Did the project attempt to impact all households within a target community, beyond the direct participants in program activities? How? a. Why were some communities not willing to participate in the program? 9. Given that the project focuses on the most vulnerable which are often those without land or with limited land, how did you cope with this issue especially for the agricultural component? 10. Often among the poor, the harsh consequences of poverty are socialized (shared among many) to reduce the impact/risk to any one household. How did SEGAMIL address the issue of strengthening social capital in the communities that were targeted? What worked well and what were the challenges? 11. Some of the videos show youth engagement in agriculture. Given the attractive of out￾migration to young people who have the most to gain from it, how successful do you feel you were in attracting youth to agriculture through the SEGAMIL activities? 12. Can you estimate what were the cost differences in a food voucher distribution system compared to a food commodity distribution system? In your estimation, was the change good/useful/beneficial? If there were cost savings, how did you redirect those? 13. Describe how the phase out of food assistance was perceived by the recipients? a. How did they react? b. Do you feel that that the majority of target households were ready to phase of food assistance? Why? Why not? c. If not, what else should have been done to achieve this? Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 13 14. How successful do you feel you were in integrating the different elements of the project? a. What actions enhanced integration and cross-sector collaboration within the project? i. What were some of the challenges? 15. How did the program collaborate with other USAID funded mechanisms? a. How well did the program link with municipal organizations? b. Do you tin they can sustain the work of the SEGAMIL Project? c. Describe the effectiveness of the collaboration between SEGAMIL, USAID/FFP, USAID/Guatemala, and PAISANO. 16. Which aspects of the program do you think are most likely to be sustained in the future? Why? 17. Which aspects of the program do you think are least likely to be sustained in the future? Why? 18. Besides the issue of the quality of primary health services in the target geographies and the prolonged drought, are there any other possible confounding factors (outside the control of the project) that we should be aware of that could affect the results, either positively or negatively? (i.e. coffee rust, coffee prices [which have been dropping recently], other market forces, etc.) 19. Given the importance of sustainability of program interventions for lasting impact on the food security of program participants, do you feel that enough effort, time, and resources were dedicated to this given the complexity of the operating environment and the plethora of partners that need to take on the different elements of the project? What worked well? What could have been done better? 20. In retrospect, is there anything you would have done differently with regards to: a. The program design? Why? b. The partnerships formed through the program? Why? c. Targeting strategies? Why? d. Implementation strategy? Why? 21. Are there any lessons learned or best practices that can be up-scaled? ADDITIONAL QUESTIONS WILL BE GENERATED BASED ON FINDINGS FROM FIELD WORK Interview guide for SEGAMIL Technical Staff (CRS/ADIPO/CARITAS) Technical staff will be interviewed in groups according to technical area. General questions will be asked of each group, followed by questions specific to their technical area. General Questions 1. How did SEGAMIL engage the communities? 2. How did SEGAMIL take into consideration the needs of the communities? 3. Which interventions were most successful? Why? a. Which interventions were most acceptable to the community? Why? b. Which practices and changes were easiest to adopt by community members? Why? 4. Which interventions were least successful? Why? a. Which interventions were most acceptable tot eh community? Why? b. Which practices and changes were easiest to adopt by community members? Why? 5. What factors facilitated the achievement of your targets? 6. What were the main challenges that affected the implementation of the program? 7. Was the shift in strategy following the mid-term evaluation beneficial or a hindrance to achieving program goals? Why? 8. What was done by the project to impact all households within a target community, not only those of the beneficiaries of a specific program activity? a. Why were some communities not willing to participate in the program? Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 14 9. How did the program collaborate with other USAID funded mechanisms? a. Were those collaborations helpful or not? Why? 10. Which aspects of the program do you think are most likely to be sustained in the future? Why? 11. Which aspects of the program do you think are least likely to be sustained in the future? Why? 12. In retrospect, is there anything the project should have done differently? Explain. 13. What is your perception of the positive impacts of the project? 14. Were there any negative effects of the program in the communities? 15. Are there any lessons learned or best practices that can be up-scaled? 16. Besides the issue of the quality of primary health services in the target municipalities and the prolonged drought, are there any other possible factors (outside the control of the project) that we should be aware of that could affect the results, either positively or negatively? For example, coffee rust, food prices, other market forces, changes in migration, etc. Food Security 1. Overall how do you think the program has affected households’ food security? a. Are you satisfied with these changes or could more have been accomplished? b. What would have been need to reach this added-value? 2. What interventions contributed most to improving household food security? 3. What is needed for those improvements to last after the project ends? a. Can most households continue to meet their dietary needs? 4. Given that the project focuses on the most vulnerable which are often those without land or with limited land, how did you cope with this issue especially for the agricultural component? a. What were the particular challenges and issues with the homestead gardens? b. Was access to inputs (particularly vegetable seeds) an issue? If so, how did you resolve that? 5. In your estimation, how important is long-distance and seasonal migration in the livelihoods of the households in the target municipalities? a. Have you been able to notice any visible impacts of remittances on the livelihoods and living conditions of the households? 6. Often among the poor, the harsh consequences of poverty are socialized (shared among many) to reduce the impact/risk to any one household. How did SEGAMIL address the issue of strengthening social capital in the communities that were targeted? What worked well and what were the challenges? a. Besides the ECADIs, what other community mechanisms are in place to continue the work of SEGAMIL to support households? 7. SEGAMIL provided rainwater collection tanks for use in irrigation systems for vegetable and herb production. Describe the effectiveness of the rainwater collection tanks. Describe how the tanks were sized. Describe how they were procured. Describe the potential for replicability after SEGAMIL is completed. 8. Since the target area is known to be arid, what ensures that households will not divert the rainwater for domestic uses, such as drinking, cooking, washing and bathing? 9. Which income generating activities had the biggest impact on your target population? Why? Value Chain 1. How did you identify the particular value chains to target? a. Were there some value chains that you discarded? b. What was the selection process? c. In hindsight, are there some value chains that you wished you had selected? If so, why? 2. What criteria did you apply to incorporate producers / families in rural value chains? Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 15 a. Generally, would you say, the selected participants were vulnerable to food insecurity before joining the value chains? Did that vulnerability improve? 3. Which value chains are the most successful in the framework of the project? 4. Given the limitations to production of most households, how well do you feel that SEGAMIL was able to connect beneficiaries with market-linked value chains? What worked well? What were the challenges? What could have been done better? 5. Some value chain groups include only women, some included only men and some were mixed. What criteria were applied to identify the value chains promoted for each type of group? a. Were women encouraged to participate in specific types of value chain activities? Or were they able to choose which chains to be a part of? b. Was there resistance on the part of men (neighbors, husbands, family, etc.) for the participation of women in value chains? 6. How did the project support producer groups? 7. After the MTE, what changed in the strategy to work with groups of producers and value chains? a. Were those changes successful? 8. What were the main challenges related to forming value chain groups? 9. With the foundation laid by SEGAMIL, do you feel that the participating farmers will be able to continue and perhaps expand their involvement in the targeted value chains? a. What additional support to these groups need to continue this work? b. Who will provide them this support? 10. How confident are you that participating farmers will get the technical and financial assistance needed to continue/expand their participation in these value chains? Why? a. What are the strengths and weaknesses of the government and private sector agricultural extension services? b. Will the community savings and loan associations have enough capital to loan to local entrepreneurs who want to expand their participation in the value chains or can they get commercial credit? 11. Do think other non-targeted households will spontaneously enter these value chains in the future? Why or why not? 12. Have you seen any evidence of migrant remittances being invested in higher-value agriculture in the target communities? Can you give specific examples? 13. What is the extent and severity of the coffee rust problem in the SEGAMIL project area? 14. How successful do you feel your interventions in this area were? What more could be or needs to be done? Savings and Loans 1. Compared to the other interventions, to what extent did the savings and loans strategy contribute to improvements in food security? 2. What significant achievements did this initiative generate? 3. Are there cases, communities or families where this strategy did not work? 4. What were the main challenges related to forming SIL groups? 5. Did you track the expenditures of loans that participants took out? a. What did they spend their loans on? b. What percentage of the loans went for productive activities and what percentage went for consumption activities (lean season food security)? 6. The savings and loans focused primarily on women? Do you think this approach was beneficial for helping household food security? Why or why not? Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 16 7. How confident are you that SIL groups will be able to continue participating women will get the training and assistance needed to continue /expand their participation in these value chains? Why? 8. Do think other non-targeted households will join these savings and loans groups in the future? Why? Or why not? 9. What were the key challenges with this intervention? Agriculture Strategies 1. What were the successes and challenges associated with the sustainable agriculture practices? 2. Which of the practices were most likely to be adopted by the target population and why? a. Did this vary by region? 3. Please describe any challenges encountered in the promotion of the following: a. Post-harvest practices b. Use of organic fertilizers c. Livestock rearing d. Market-oriented crop production 4. Did ideas about gender roles affect the adoption of agricultural practices by men and women in these communities? How? 5. Were there other challenges related to gender affect the implementation or success of the agriculture interventions? a. What was the participation rate of women in the higher-value agricultural trainings and initiatives (e.g., for apiculture)? 6. Entrepreneurship: género, women, youth, etc. (For example, in addition to the livestock kits which others have been promoted). a. Were there specific actions / strategies to strengthen the participation of women at the community / municipal level? b. What achievements are made visible in terms of gender? What has been the level of resistance in terms of promoting behavioral changes with respect to gender stereotypes? c. Was the economic participation of women promoted and strengthened? How? What significant achievements can you mention? How have these achievements measured? They implied behavioral changes in the food security of the families? How? d. Actions in favor of young people, which ones can you mention? Achievements generated by these actions, could you mention some? e. Are there cases that generate learning (favorable or resistant) in terms of actions focused on women and young people? Vouchers 1. How did the beneficiaries feel about the voucher systems compared to traditional commodity distribution? 2. How did the shift from food rations to a voucher system impacted women’s participation in various interventions? 3. What challenges were encountered in the implementation of the food voucher system? a. Was the project able to overcome these challenges 4. How could the voucher system be improved? 5. What mechanisms are in place to sustain the food voucher system moving forward? 6. Food transfers are designed to fill food gaps temporarily in food deficit households while other project interventions, i.e. food production and income generation, take hold and allow households to transition off food assistance. Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 17 a. How well do you feel that SEGAMIL accomplished this? Do you feel that that the majority of target households no longer need food assistance? b. If not, what else should have been done to achieve this? 7. Did SEGAMIL identify cases of families wanting to monetize the products received by SEGAMIL? 8. Did the project monitor whether the products received (by food rations or through the voucher system) were consumed by the targeted beneficiaries? 9. What criteria did you apply for the selection of the food that was delivered to the families? a. How acceptable were the foods to these families? Natural Resource Management 1. The CRS Agriculture sector has developed a set of SMART skills to assist poor, rural farmer get out of poverty. SMART stands for Skills for Marketing and Rural Transformation. There are modules on understanding natural resources and managing natural resources and use adult learning principles. It appears that these guidance documents were not used and other agricultural technical guides were developed by the project. Why? 2. What were the main achievements with regards to NRM? 3. Did the communities value the importance of these interventions? 4. How effective were the food for work strategies to motivate community members to support these activities? 5. Describe the sustainability of the nursery and reforestation activities. Describe the support required after SEGAMIL closes and describe how this support will be realized. 6. Describe how the effectiveness of the interventions will be monitored after SEGAMIL is completed. 7. Are you aware of any plans to replicate these activities in other communities in the future? 8. Describe the Integrated Water Resources Management (IWRM) process used in SEGAMIL. 9. Describe the process for developing community disaster risk mitigation and local response plans. 10. What were the main challenges faced in developing and implementing these plans? 11. What support and resources are needed to continue the implementation of the mitigation plans once SEGAMIL has been completed? 12. SEGAMIL helped communities to develop 120 food security and nutrition plans based on information obtained from conducting risk diagnostics and vulnerability assessments in previous years. Why did SEGAMIL not conduct more current assessments where the updated information may be more relevant for the plans? ECADIs 1. Please describe how the ECADI were formed and how they function. 2. What has been the contribution of the ECADIs? 3. What ECADI activities have been most successful? Why? 4. Were the ECADIs effective at generating behavior change? a. In which areas was behavior change most noticeable? 5. What has represented the greatest difficulty in the work of the ECADIs? 6. What activities have represented the greatest challenge within the ECADI? Why? 7. Is there support for the ECADI by community members? a. Why do community members participate in the ECADI? b. Do the community members value the trainings and instruction provided by the ECADI? 8. Were cascade trainings an effective way to build skills at the household and community level, and to build capacity of local partner agencies (across interventions)? a. Was the quality of information maintained at the community level? 9. What type of support has the SEGAMIL project given the ECADI? Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 18 10. Is anyone else supporting the ECADI? Who? 11. Will anyone support the ECADI after SEGAMIL closes? a. Is there support for ECADIs by municipal authorities? Is this the case in all SEGAMIL municipalities? 12. How have the ECADI been integrated into COCODE? a. Have ECADI representatives been able to integrate into the COCODE? If the answer is YES, what has been more complicated in the integration process? If it is No. Which do you consider to be the reason why this integration was not possible? Nutrition and Health Behavior Change 1. Which nutrition and health practices were promoted by the program? 2. Which of these were most easily changed? a. Which were the most difficult to change? Why? 3. Which strategies were most effective in achieving the desired changes in behaviors? a. What were the most effective strategies for improving dietary diversity? b. How acceptable were the plant and animal foods introduced by the program to the community members? 4. Can households sustain these dietary gains without the food rations and vouchers provided by SEGAMIL? 5. Can households sustain these dietary gains without other agricultural inputs from the program? 6. Please describe the pros and cons of the following strategies in achieving behavior change. a. ECADI trainings b. Demonstrations c. Home visits d. Growth monitoring 7. How successful were the use of household improvement plans for bringing about the desired changes? a. What difficulties were associated with their implementation? b. What could have been done to make this more effective? 8. How effective were the promoters in promoting nutrition and health behaviors? a. Did the project monitor the quality of instruction provided by the promoters? 9. During the implementation of the project: How was the nutritional surveillance of the project implemented? What actions or strategy was worked to strengthen this nutritional surveillance system? At what level was this system strengthened? What factors influenced / affected this process? For the implementation of this strategy, how did it affect the municipal level and the providers of health services and primary care? What actions were carried out with municipal and central government bodies (municipal offices, committees, MSPAS, MAGA)? 10. How nutritional surveillance is assured at the family level and at the community level (municipal inclusive) once the project no longer exists. If there is no strengthened surveillance system (at the family, community, municipal level); what factors are the ones that did not allow this practice in a sustainable way? 11. What will it take for the ECADIs (and other SEGAMIL initiatives) to become sustainable and possibly scaled-up to the benefit of other communities? WASH 1. What interventions in WASH were prioritized after the MTE? What interventions were implemented? Where were they implemented? Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 19 2. Introduction of Water Systems and Sanitation: Where were they introduced? What types of communities were favored? What criteria were used for the implementation and selection of these communities? 3. Changes in behavior regarding sanitation and hygiene: how were they measured? What interventions were made to promote behavioral change? 4. Is it possible to differentiate the characteristics of families that have manifested behavioral changes in this topic and which are not? What do you think influenced it? 5. What aspects of WASH (at the community level and at the family level) were most open to manifest behavioral changes? 6. Are there more communities open to participate? What characteristics do you identify in these communities? 7. Was a SEGAMIL WASH strategy developed? Is it truly focused on community drinking water supply systems, household sanitation and personal hygiene behavior change? Or, it is mainly focused on personal hygiene behavior change (handwashing, safe drinking water, proper management of excreta of the baby and the family, and waste management within the household)? 8. If a WASH strategy was developed, how well is it integrated with health/nutrition and agricultural activities? 9. In hindsight, describe the positive and negative impacts of subsidizing WASH items. 10. Describe the effectiveness of the linkages created between COCODE, the Community Water Committees and OMAS. Describe how these linkages will be sustained past the life of SEGAMIL. Community Water Supply 11. What community drinking water activities were implemented? 12. Describe the linkages between SEGAMIL and local community enterprises which supply water filters and replacement filters. Describe the viability of these markets after the completion of SEGAMIL. 13. What will ensure the sustainability of the roles and responsibilities of the water technicians? What incentives did SEGAMIL create or instill for the technicians to continue in their role beyond the life of the project? 14. By definition, PHAST is a methodology to transform hygiene and sanitation practices. Describe how the methodology was used “to build the capacity of the community water committees on their roles and responsibilities as committee members, advocacy skills to seek support for their projects, and the operation and maintenance of the water systems.” 15. It appears that water supply system-based chlorination (as opposed to household chlorination at point of use) is being used to disinfect water supplied to communities and that this was implemented prior to SEGAMIL. Describe how SEGAMIL supported this pre-existing technology and the mechanisms put into place to ensure its functionality beyond the life of the project? 16. Describe the private entities that established agreements with the public sector to promote access to WASH services. 17. It was reported that SEGAMIL’s collaboration with OMAS, members of community water committees and Ministry of Health has been key to improving water systems functionality of the communities where the program works. Why did it take the involvement of SEGAMIL for the systems to function properly? What ensures that the entities outside of SEGAMIL will carry on with their responsibilities? 18. It is good that SEGAMIL is concerned with water quality and tests the community water to ensure it is fit for human consumption. What will ensure water quality sampling and testing will occur beyond the life of the project? Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 20 Sanitation 19. What household sanitation (latrines) activities were implemented? Latrine construction? Latrine usage messaging? 20. How has behavioral change been measured in the different interventions related to health, hygiene, nutrition, sanitation, agricultural practices, etc.? 21. It was reported that “85% of HH have a latrine.” This seems very high (which is a good thing) but usage is low (which is not a good thing). Describe the reasons for this high coverage rate. 22. Describe the reasons households are not using their latrine. Hygiene 23. Which areas (e.g., municipalities and communities) were better open to manifest behavior changes? What were the criteria for implementation and selection of these areas? 24. Why were WASH behavior change activities not implemented in all areas of intervention? 25. Participatory Health and Sanitation Transformation (PHAST) is a participatory methodology used to create household demand for an improved sanitation facility as well as improve personal hygiene behaviors, such as using the sanitation facility, constructing a handwashing station and washing hands at key times during the day and using safe water. Since the PHAST activity was used later in the SEGAMIL timeline, how will its effectiveness be monitored? Describe if there will be longer-term monitoring past the completion of the project. 26. Regarding PHAST, it is a methodology that is facilitated by using a series of activities to get participants to understand their poor sanitation and hygiene practices and then uses a series of activities for the participants to realize and take ownership of good practices. Several SEGAMIL documents state that there were PHAST trainings. Did SEGAMIL mean to say “facilitated” or were staff trained in facilitating the methodology? Please clarify. 27. SEGAMIL used much messaging on proper hygiene, specifically using a latrine and washing hands. What did the project do for those households that did not have a latrine or a handwashing station? Community Resilience and Early Warning System 1. How were communities engaged in the implementation of early warning systems? a. What factors contributed to their successful implementation? b. What were the main challenges faced? 2. Where did not this system not work? 3. Why did the program discontinue its support to the COLRED? 4. How integrated to COLRED/COMRED to early warning system? 5. What proportion of communities developed community resilience plans? 6. What challenges did the program face in the development and implementation of these plans? 7. Can you provide examples of successes in this area? 8. What mechanisms are in place for continued funding of community resilience plans? Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 21 5. Interview Guides for Municipal Offices General Questions 1. Can you describe how your office supports the communities in this municipality? 2. Are you familiar with the SEGAMIL project? 3. What do you think the SEGAMIL project is trying to achieve? 4. Has your office collaborated with SEGAMIL? a. Please describe that collaboration. b. What activities did you work on together? 5. Can you provide examples of things you have achieved together? 6. How long have you been collaborating with SEGAMIL? 7. What were the institutional arrangements for implementation of the project between you and SEGAMIL? 8. What support did you receive from SEGAMIL? a. How effective was this support? 9. Did the project contribute to capacity building within your office? a. In what areas? 10. How has your office benefited from collaboration with SEGAMIL? 11. What have been the difficulties of working with SEGAMIL? 12. Has working with SEGAMIL changed the way your office works? 13. What is your perception of the impact of the project? a. What have been the biggest achievements of the SEGAMIL project? b. What aspects of the project did not work well? 14. How did the program contribute to municipal plans? 15. What is your opinion of the SEGAMIL project with respect to sustainability? 16. Has your office worked with SEGAMIL to support community organizations? a. Which ones? b. Probe: COCODES, COCOSAN? Water Committees, ECADI, Value chain groups? 17. Can you describe this support? Municipal Agricultural Technical Office 1. In general, what is your opinion of the agricultural promotion work done by SEGAMIL in the target communities? a. What seemed to work well and what were the challenges? b. What could have been done better? 2. How well do you feel the agricultural activities contribute to improving household food security? a. Anything in particular you think was especially good and could be scaled up? b. How sustainable do you think the promoted practices will be after the project? 3. What about the value chain activities? How well did those work? a. Are there specific activities that can be scaled up to other communities? b. How sustainable do you think the promoted practices will be after the project? 4. How strong is rural out-migration within your jurisdiction? a. Is out-migration gender-selective? b. What are the most notable impacts? c. Are there any benefits? d. Have you noticed any impacts of remittances especially on agriculture? What are those? 5. How well do you feel the ECADI agricultural schools functioned? a. What worked and what were the challenges? b. What could be done better? Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 22 6. Will the municipality be able to support these schools after the project closes? a. In what ways? 7. Are there any plans to for these schools to be scaled up to other non-project communities? Municipal Women's Office 1. How have women’s roles in agriculture and overall decision-making, including community leadership participation, changed as a result of SEGAMIL? a. Have intra-household dynamics changed in anyway because of the project? i.e. who decides about the care and feeding of infants and young children? 2. What are the biggest challenges that women face in your jurisdiction? (i.e. the triple workload) 3. Is it different by age-group (i.e. youth)? 4. How did SEGAMIL help address these challenges? a. What worked well? What could have been done better or differently? 5. Do you think the positive changes will be sustainable? 6. Can the successes be scaled up to non-project communities? If so, how exactly? 7. How has male out-migration impacted the lives of women? 8. What challenges has it produced for women? Opportunities? 9. Did SEGAMIL have any mitigating consequences on outmigration? 10. How important are remittances to the household economy? a. Who decides how they are spent? Municipal Water and Sanitation Office 1. How did the project integrate aspects of water resource management? 2. What stakeholders were involved in these activities? 3. What challenges do you face in water service provision in your municipality? 4. How does the Municipal Council support communities to sustain their water sources? What services do you offer? 5. What challenges do you face with other support structures that support community water supply? 6. Did the SEGAMIL project help address these challenges? How? 7. Can you give examples of the activities conducted by SEGAMIL in this area? 8. Was there work effective in improving water sources? 9. What challenges do you face in sanitation provision in your municipality? 10. How is your office responding these challenges? 11. Did the SEGAMIL project help address these challenges? How? 12. Can you give examples of the activities conducted by SEGAMIL in this area? Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 23 6. Interview Guide for USAID 1. What has been SEGAMIL’s contribution to USAID’s efforts to reduce food insecurity in the Western Highlands? 2. In your estimation, which SEGAMIL activities have made the largest contributions to the USG strategy for Guatemala? 3. In your estimation, which SEGAMIL activities were more marginal in terms of their contributions to the USG strategy? 4. What were the main implementation challenges faced by CRS? a. Were CRS able to adequately address these challenges? 5. How effective was CRS at managing the SEGAMIL program? a. Was adaptive learning strategically used by CRS and its partners over the course of the project? 6. Generally, what do you think of the functionality and quality of the systems and processes established by the SEGMAIL project? a. For example, its system for cascade training b. The institutional arrangements to support program activities c. The establishment and strengthening of community and municipal organizations 7. How well did SEGAMIL coordinate with and leverage other USAID-funded programs? a. In your opinion did these collaborations produce synergistic results? Please describe? What worked well? What could have been done better? What considerations need to be made for future collaborations? 8. In your opinion, was sufficient effort dedicated to creating sustainable systems? a. Will the necessary resources and capacity strengthening exist to sustain service providers? 9. How did USAID/Guatemala determine the programmatic parameters for SEGAMIL? Based on what data/evidence? a. What were the factors that influenced taking a 1,000 Day approach vis-a-vis other possible approaches? b. In hindsight, are you satisfied with the results from this programmatic approach? c. Would you have done anything differently? 10. How effective do you feel that SEGAMIL, and Title II programming in general, has been at addressing both the immediate and underlying causes of food insecurity in the Guatemalan highlands? a. Given that poverty--and consequently food insecurity--in the highlands has multiple causes, including insecure and limited landholdings, how effective have Title II programs been in addressing these multifactorial causes? 11. Please describe the relationship between USAID/Guatemala and SEGAMIL. What worked well? What did not work well? What considerations need to be made for future collaborations? 12. Food assistance invokes a lot of intense responses. Based on the 20 year + experience of USAID/Guatemala in Title II programming and taking SEGAMIL as the most recent example, what were/are the most outstanding benefits of using food as a programmatic resource? What were the most outstanding challenges to using direct distribution? What were the reasons for switching from direct commodity distribution to a voucher systems? Benefits and challenges of the transition? 13. Does USAID consider that activities focused on risk and disaster prevention are key to improving the nutrition, health and hygiene of rural families? Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 24 7. Interview Guide for In-depth Interviews (First, note number of participants, number of women and number of men participants) 1. Did you participate in any SEGAMIL activities? IF yes, which ones? 2. Which agriculture or livelihood activities of the project were most beneficial to you? [list] 3. Did any SEGAMIL agriculture or livelihood activities help you increase the amount of food your family had to eat or the number of different types of food your family had to eat? a. If yes, which activities? b. Please describe how the agriculture and livelihoods activities helped your household. 4. Did any SEGAMIL agriculture or livelihood activities help you increase your household’s income? a. If yes, which activities? Probe: did any other activities helped your household? 5. Do you think there are things SEGAMIL could have done to help you improve your food security that it didn’t do? a. If yes, what are they? 6. Were there any agriculture or livelihood activities promoted by SEGAMIL that you thought were good but were too difficult or time-consuming or expensive for you to do? a. If yes, which ones? b. Explain why they were too difficult. 7. How did the project help you improve the health of your children? a. Please give specific examples. 8. Please describe your experience with     ECADI Monitor mothers Growth monitoring Sustainable agriculture 9. Describe how the voucher system worked for your family 10. Do you think the project helped address your needs? a. Did the project listen to you about what you needed? 11. Are there activities that you started to do during the SEGAMIL project that you will find difficult or impossible to continue now that the project is finished? a. If yes, which ones and why will they be difficult to continue? b. Probe (based on response’s given above) 12. Do you know of other sources of support that can help you to continue in future the activities you began under SEGAMIL? a. If yes, what are they? [List] Final Performance Evaluation of the Food Security Program Focused on the First 1,000 Days (SEGAMIL) Development Food Assistance Project—Draft Evaluation Protocol 25 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523