[Add Date] Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala Final Evaluation Report January 17, 2019 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by ICF Macro, Inc. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala Final Evaluation Report Contract Number: GS-00F-189CA/7200AA18M00002 Submitted by: ICF Macro, Inc. 530 Gaither Rd., Suite 500 Rockville, MD 20850 Contributors: Sandra Wilcox, Team Leader Kali Erickson, Maternal, Child Health and Nutrition Specialist John Russell, Agriculture and Livelihoods Specialist Carlos Flores, WASH, Resilience, Disaster Risk Reduction and Governance Specialist Gheda Temsah, Lead Analyst Benita O’Colmain, Senior Survey Methods Specialist January 17, 2019 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report i ACRONYMS ARR Annual Results Report BMI body mass index CEPROCAL Centro de Producción Caprina Del Altiplano CGAP Consultative Group to Assist the Poor CIPCA Centro Integral de Producción y Capacitación Agroambiental COCODE Community Development Council (Consejo Comunitario de Desarrollo) COMUDE Municipal Development Council (Consejo Municipal de Desarrollo) CONRED Coordinadora Nacional para la Reduccion de Desastres (National Coordination Body for Disaster Risk Reduction) COSAN Comision de Seguridad Alimentaria Nutricional (Food Security Committee) DFAP Development Food Assistance Project DIP Detailed Implementation Plan DHS Demographic and Health Survey DPCCE daily per capita consumption expenditures EBF exclusive breastfeeding EQ evaluation question FFP Office of Food for Peace FFW Food for Work FGD focus group discussion FY fiscal year GDP gross domestic product GMP Growth Monitoring and Promotion HDDS household dietary diversity score IDI in-depth interview IP implementing partner IPTT Indicator Performance Tracking Table KII key informant interview LOA life of activity M&E monitoring and evaluation MAD minimum acceptable diet MAGA Ministry of Agriculture MCHN maternal and child health and nutrition MIS management information system MOH Ministry of Health MTE mid-term evaluation NGO nongovernmental organization ORT oral rehydration therapy PAISANO Programa de Acciones Integradas de Seguridad Alimentaria y Nutricional del Occidente PBS population-based survey PCI Project Concern International PE performance evaluation Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report ii PEC Programa para Extension de Cobertura (Coverage Extension Program) PHAST Participatory Hygiene and Sanitation Transformation PIRR plan for integrated community risk reduction PPI Poverty Probability Index SAMI sistema automatizado de monitoreo de informacion (automated information monitoring system) SBC social and behavior change SBCC social and behavior change communication SESAN Secretaría de Seguridad Alimentaria Nutricional (Secretariat for Food and Nutrition Security) SO strategic objective TOC theory of change USAID United States Agency for International Development USG U.S. Government WASH water, sanitation, and hygiene WDDS women’s dietary diversity score Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report iii CONTENTS Acronyms.............................................................................................................................................................................. i Executive Summary .......................................................................................................................................................... vi 1. Introduction ............................................................................................................................................................... 1 1.1 Background ....................................................................................................................................................... 1 1.2 Project Description ......................................................................................................................................... 2 1.3 Theory of Change ........................................................................................................................................... 3 2. Evaluation Overview ................................................................................................................................................ 5 2.1 Evaluation Purpose .......................................................................................................................................... 5 2.2 Evaluation Questions ...................................................................................................................................... 5 3. Evaluation Methods and Limitations .................................................................................................................... 7 3.1 Data Collection Methods .............................................................................................................................. 7 3.1.1. Quantitative Data Collection Methods and Analysis: PBS ....................................................... 7 3.1.2. Qualitative Data Collection ............................................................................................................. 9 3.2 Limitations ....................................................................................................................................................... 12 4. Evaluation Findings, Conclusions, and Recommendations ............................................................................ 14 4.1 SO.1: Household Access to Food ............................................................................................................. 14 4.1.1 EQ1: To what extent has PAISANO met its defined goals, objectives and outcomes? . 14 4.1.2 EQ2: Based on the evidence, which project outcomes are likely to be sustained? ......... 18 4.1.3 EQ3. In each technical sector, what are the strengths and challenges of the effectiveness of the interventions’ implementation?............................................................................. 19 4.1.4 EQ4. What are the key lessons learned and best practices that should inform future projects in the country? .............................................................................................................................. 21 4.2 SO2: Reduction of Malnutrition ................................................................................................................. 23 4.2.1 EQ1: To what extent has PAISANO met its defined goals, objectives and outcomes? . 23 4.2.2 MCHN: EQ2: Based on the evidence, which project outcomes are likely to be sustained? ........................................................................................................................................................ 36 4.2.3 MCHN: EQ3: In each technical sector, what are the strengths of and challenges of the effectiveness of the interventions’ implementation?............................................................................. 38 4.2.4 MCHN: EQ 4: What are the key lessons learned and best practices that should inform future projects in the country? .................................................................................................................. 42 4.2.5 WASH: EQ1: To what extent has PAISANO met its defined goals, objectives, and outcomes? ....................................................................................................................................................... 44 4.2.6 WASH: EQ2: Based on the evidence, which project outcomes are likely to be sustained? ........................................................................................................................................................ 47 4.2.7 WASH: EQ3: In the SO2 technical sector, what are the strengths and challenges of the effectiveness of the interventions’ implementation?............................................................................. 49 4.2.8 WASH: EQ4: What are key lessons learned and best practices that should inform future projects in the country? .................................................................................................................. 51 Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report iv 4.3 SO3: Improved Resilience to Food Insecurity ........................................................................................ 53 4.3.1 EQ1: To what extent has PAISANO met its defined goals, objectives, and outcomes? 53 4.3.2 EQ2: Based on the evidence, which project outcomes are likely to be sustained? ......... 55 4.3.3 EQ3: In the SO3 technical sector, what are the strengths and challenges of the effectiveness of the interventions’ implementation?............................................................................. 56 4.3.4 EQ4: What are key lessons learned and best practices that should inform future projects in the country? .............................................................................................................................. 58 4.4 Cross-cutting Issues ...................................................................................................................................... 59 4.4.1 Project Management ....................................................................................................................... 59 4.4.2 Gender ............................................................................................................................................... 63 4.4.3 Monitoring and Evaluation ............................................................................................................. 65 5. General Conclusion and Challenges .................................................................................................................. 68 Annexes: Annex 1: Results Framework Annex 2: Statement of Work Annex 3: Evaluation Protocol Annex 4: Tabular Summary of Indicators and Change from Baseline to Endline Annex 5: Descriptive and Bivariate Analyses Tables Annex 6: Methodology and Summary of Multivariate Regression Analysis Annex 7: Multivariate Tables Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report v FIGURES Figure 1: Food Groups Consumed by Households ................................................................................................. 14 Figure 2: Prevalence of Poverty: Percentage of People Living on Less Than $1.25 per Day ........................ 15 Figure 3: Mean Depth of Poverty ................................................................................................................................. 15 Figure 4: Baseline–Endline Comparison of Agriculture Indicators ..................................................................... 16 Figure 5: BMI Levels of Non-pregnant Women of Reproductive Age, PAISANO 2013 and 2018 ............. 23 Figure 6: Baseline-Endline Comparison of Women’s Dietary Diversity as Food Groups Consumed by Women 15–49 Years of Age ........................................................................................................................................ 24 Figure 7: Population-based Infant and Young Child Feeding Indicators .............................................................. 27 Figure 8: Population-based Meal Frequency and Dietary Diversity Indicators for Children 6–23 Months of Age ................................................................................................................................................................................. 28 Figure 9: Mother’s Recognition of Danger Signs in Pregnancy, Neonates, and Children under 5 Years of Age 29 Figure 10: Mother’s Reported Care-seeking for Danger Signs in Pregnancy, Neonates, and Childhood Illness .................................................................................................................................................................................. 30 Figure 11: Women’s Participation in Health Care Decisions for Themselves and for Their Child............. 31 Figure 12: Baseline-Endline Comparison of Percentage of Households Using Improved Drinking Water Source ................................................................................................................................................................................. 45 Figure 13: Baseline-Endline Comparison of Percentage of Households Using Improved Sanitation Facilities .............................................................................................................................................................................. 46 Figure 14: Baseline-Endline Comparison of Percentage of Households with Water and Soap at a Commonly Used Handwashing Station ...................................................................................................................... 46 Figure 15: Baseline-Endline Comparison of Percentage of Children under 5 Years of Age with Diarrhea in the Last Two Weeks and Children under 5 Years of Age with Diarrhea Treated with ORT ................ 49 Figure 16: Waterborne diseases in Guatemala between Week 01/2012 and Week 26/2016 ...................... 50 TABLES Table 1: Daily Per Capita Consumption Expenditures by Household Use of Financial Services, Value Chain Activities and Sustainable Crop and Livestock Practices at Endline (constant 2010 USD and 2005 PPP) 17 Table 2: Population-level Child Nutritional Indicators for PAISANO Communities ...................................... 25 Table 3: Baseline to Endline Indicators for Recognition of Danger Signs, Care-seeking, and Women’s Decision-making about Health Care ........................................................................................................................... 31 Table 4: Overview of PAISANO’s Approaches to Promote High-nutrient Foods .......................................... 32 Table 5: Comparison of Baseline and Endline Indicators by Project Participation Status .............................. 35 For Select Indicators ....................................................................................................................................................... 35 Table 6: Population-level WASH Indicators for PAISANO Intervention Communities ................................ 44 Table 7: Community Resilience Indicators for PAISANO Participants Reported in ARRs ........................... 54 Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report vi EXECUTIVE SUMMARY Save the Children was awarded the PAISANO (Programa de Acciones Integradas de Seguridad Alimentaria y Nutricional del Occidente) Development Food Assistance Project (DFAP) by the United States Agency for International Development (USAID) Office of Food for Peace (FFP) in August 2012. The approximately $48 million project was implemented by Save the Children in partnership with Project Concern International (PCI). PAISANO was a six-year project conceived to build on lessons learned and strategies that had been successful in previous Save the Children and PCI projects. PAISANO’s goal has been to reduce food and nutrition insecurity among 26,500 vulnerable rural households in 13 municipalities in 3 departments of the Guatemala Western Highlands (Save the Children in 8 municipalities in Quetzaltenango and Quiche, and PCI in 5 municipalities in Huehuetenango). The project targets rural communities suffering from food insecurity and at risk of food insecurity, and communities with a high prevalence of chronic malnutrition. To achieve its goal, the project aimed to increase household access to food, reduce malnutrition among children under 5 years of age, and improve community resilience. The project also worked to ensure gender equality, mitigate environmental consequences, and influence behavior change. The findings, conclusions, and recommendations of the evaluation are expected to have primary utility and learning value for USAID stakeholders (FFP/Washington, USAID/Guatemala, and the FFP learning network). Additional stakeholders include the two implementing partner organizations (Save the Children and PCI), as well as Guatemalan government officials from key municipalities and departments. THEORY OF CHANGE PAISANO was designed to address critical problems of food insecurity, poverty, and malnutrition, as well as to increase the resilience of communities and households in targeted areas in the Western Highlands. Key interventions include promotion of positive behavior change in nutrition, health, hygiene, sanitation, and agriculture; involvement of women in project activities as participants and beneficiaries; and interventions to diversify livelihoods through livestock and savings and lending activities. The stated goal is that food insecurity and malnutrition will be reduced among poor, vulnerable households in the Western Highlands of Guatemala. The working hypothesis is that if access to food by vulnerable rural households is increased (SO1), if malnutrition among children under 5 years of age is reduced and the nutritional status of pregnant and lactating women is improved (SO2), if vulnerability to food security shocks is decreased and community resilience is increased (SO3), then decreases in food insecurity and malnutrition among poor households should be achieved. The hypothesis stresses the close interrelationship between malnutrition and food insecurity in reaching the overall goal. EVALUATION QUESTIONS The principal evaluation questions (EQs) are as follows:  EQ 1: To what extent has PAISANO met its defined goals, purposes, and outcomes, and what factors promoted or inhibited their achievement?  EQ 2: Based on the evidence, which project outcomes are likely to be sustained?  EQ 3: In each technical sector, what are the strengths and challenges of the effectiveness of the interventions’ implementation?  EQ 4: What are the key lessons learned and best practices that should inform future projects in Guatemala? Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report vii EVALUATION DESIGN The PE was conducted using a mixed methods design by a team assembled by ICF International. It relied on a broad database of information derived from two population-based surveys (PBSs) of households in areas supported by the project—a total of 3,000 households were surveyed at baseline (2013), and 1,400 households at endline (2018), distributed proportionally across the project communities—as well as extensive key informant interviews (KIIs), in-depth interviews (IDIs), observations, and focus group discussions (FGDs) conducted by the evaluation team. The PBS data were used to derive and compare standard FFP indicators across multiple sectors from baseline to endline. The PBS data were supplemented in 2018 by FGDs with 217 participants across 8 communities, as well as 91 KIIs and IDIs, conducted at the department, municipal, and community levels. The evaluation team also drew on key implementing partner documents, monitoring information, mid-term evaluation (MTE) information, and other information from secondary sources to help interpret findings and provide support for recommendations. The mixed methods design allowed the evaluation team to triangulate findings from several different sources, providing evidence-based confirmation of findings and conclusions while helping mitigate the limitations of each source of data considered in isolation. SUMMARY FINDINGS AND CONCLUSIONS SO1: Household Access to Food PAISANO’s SO1 was designed in accordance with best practices at the time of its inception, and implementation was generally effective. Distribution of small animals and improved seeds, together with adequate technical support, allowed significant numbers of participants to increase the amount of nutritious food available to their families and provide occasional and modest increases to their household incomes. Increased incomes from the sale of project-promoted crop and animal products allows increased economic access to food, and increased home consumption of such food represents increased physical access to food. Despite difficulties with availability of water and a lack of improved vegetable seeds, the home gardening activity is likely to be sustained due to the positive attitude of participants toward it. The distribution of goats for milk is popular with participants but may prove less sustainable than the small animals (poultry and rabbits), given the value of individual animals and the relative complexity of management. Housing provided for animals was rudimentary at best. FFP would be well advised to consider something similar to the Consultative Group to Assist the Poor (CGAP)- Ford Foundation graduation approach to small animal asset transfers, which might help participants leverage the reception of a small number of animals and associated inputs into enterprises that generate more income as well as provide for home consumption. The training that agriculture leaders received allowed them to provide competent extension advisory services to participants, and most appear on their way to establishing viable, sustainable, fee-for-service enterprises. Their income, however, comes only from support to households’ small animal production; there is no market for crop advisor services. The Western Highlands Goat Production Center and the Integrated Production and Agro-Environmental Training Center were well-designed and well-constructed and appear to be functioning as intended. The PE could not evaluate their eventual sustainability because they are both still finalizing their business and sustainability plans. PAISANO’s value chain activities were successful but on a necessarily small scale, given that most FFP participants do not have the minimum economic resources, market experience and savvy, and household resilience to be integrated into them effectively. FFP should concentrate more on activities appropriate to its target population, rather than those appropriate to recipients of Feed the Future activities. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report viii SO2: Reduction of Malnutrition PAISANO’s strong contribution to improved food security is documented by statistically significant improvements in dietary diversity in households, women’s diets, and children’s diets, particularly for the high-nutrient foods promoted by PAISANO to address dietary gaps. Improved complementary feeding indicators testify to a major, multi-channel effort, including hands-on activities to try to practice new behaviors and foods that participants consider valuable for their improved nutrition. Decreases in population-based stunting prevalence were statistically significant, although under the project target, and child underweight remains an important concern due to the lack of progress in reducing this indicator. The extremely high prevalence of stunting requires addressing the nutritional and health contributors over a long-term, intergenerational period, and PAISANO makes a positive contribution to this effort. Project maternal and child health and nutrition volunteers expressed interest in continuing to serve their communities beyond the life of the project, although most volunteers described a substantial decrease in their activities if the project had already concluded in their communities. It is unlikely that volunteers will continue to carry out the range of activities they had done during PAISANO’s implementation due to limited mechanisms for sustained capacity-building1 and motivation, which are recognized as important contributors to sustainability of volunteer services.2 As a result, referrals and social and behavior change (SBC) are likely to be reduced without PAISANO’s support, but participants report motivation to continue attending health services. Most mothers consulted reported that they will face challenges to maintain dietary diversity without cash transfers, but many cited their motivation to prioritize the purchase of nutritious foods and mentioned their improved animal and home garden production due to PAISANO’s support. PAISANO’s linkages are strongest at the health facility level, where many health staff expressed their intention to use project SBC materials to continue health promotion after project close-out. PAISANO’s SBC approach, based on hands-on activities and close individual follow-up by volunteers and staff, appears to have been effective at supporting behavior change, particularly for dietary diversity and water, sanitation, and hygiene (WASH). Food/cash transfers and productive activities also appear to have addressed some of families’ requirements for increased access to and availability of diverse foods. Cash assistance, as opposed to food assistance, was mentioned by nearly all participants consulted during focus groups as supporting greater dietary diversity. Quantitative analysis supported this finding, and statistically significant associations were found between households that received a cash transfer and lower prevalence of child stunting in children under 5 years of age, as well as improved prevalence of minimum acceptable diet. A gender lens was integrated across the three SOs, but specific gender activities had limited reach, although they were well-received by those exposed to activities. Lessons can be learned from PAISANO’s successful experiences in increasing dietary diversity and improving complementary feeding based on interventions that synergize and complement each other, including the use of effective SBC methods and cash transfers. Qualitative consultations with project participant mothers found nearly universal preference for cash transfers rather than direct food assistance, due to the perceived benefit that cash assistance facilitates access to diverse foods promoted by the project. In addition, PAISANO’s structured process for spending cash transfers was considered by participants to be effective for guiding the purchase of diverse foods and acceptable to their families. Volunteer subsidies funded by service fees provide examples of challenges for the stability of the funds, as well as for repercussions for sustainability. 1 For example, refresher trainings, feedback, and supervision 2 Rogers, B.L., and Coates, J. 2015. Sustaining Development: A Synthesis of Results from a Four-Country Study of Sustainability and Exit Strategies among Development Food Assistance Projects. Washington, DC: FHI 360/Food and Nutrition Technical Assistance III Project (FANTA). Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report ix WASH The most sustainable WASH interventions are those related to behavior change and capacity building among leaders, rather than those involving economic factors, such as Ecofilters and the chlorination of water systems. Both participants and local authorities agree that people saw the importance of WASH actions in terms of the health benefits they bring to households. For the most part, members of the water committees saw their capacities strengthened and were motivated by the practicality of the training sessions. As a result, they have adopted the plans they designed with the project in their regular activities. Despite this, there are strong cultural barriers and a reluctance to pay for an efficient and quality service, which leads to decreased participation in these interventions or reduces the outcomes that these interventions strive to achieve. An important lesson learned is that WASH activities should be included in the design of any food security project, including educational activities and the provision of improved services for the participants. SO3: Improved Resilience to Food Insecurity SO3 focused on strengthening community resilience in the project areas. At the beginning of the project, the focus was to establish linked committees within the Coordinadora Nacional para la Reduccion de Desastres (National Coordination Body for Disaster Risk Reduction) system. The actions were refocused after the MTE, directing the project interventions toward community leaders and their link with government authorities. The MTE found that this strategy was more focused on community resilience related to natural disasters but not to food security. Yet no clear evidence could be established that these changes in the PAISANO strategy would bring about an increase in community resilience. The municipal authorities agreed that the leaders who were strengthened by PAISANO stand out for their strategic vision and proactive approach for financing management activities. For its part, the project made significant progress in establishing signed municipal commitments to follow up on project activities. All these accomplishments, however, can be undercut by the short terms and recurring election of authorities and leaders. It was also acknowledged by authorities and project partners that because the changes did not occur until after the MTE, there was little time to consolidate sustainable processes or to improve the scope of the governance. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 1 1. INTRODUCTION 1.1 BACKGROUND Guatemala is a Central American country of 15,460,732 inhabitants (July 2017 estimate) with a land area of 108,889 square kilometers. 3 Mexico and Belize border on the north, and Honduras and El Salvador on the south. As the most populous country in Central America, Guatemala has a gross domestic product (GDP) per capita that is roughly half the average for Latin America and the Caribbean. The agricultural sector accounts for 13.5 percent of the GDP and 31 percent of the labor force; key agricultural exports include sugar, coffee, bananas, and vegetables. Guatemala is the top remittance recipient in Central America as a result of Guatemala’s large expatriate community in the United States. These inflows are a primary source of foreign income, equivalent to two-thirds of the country’s exports and about a tenth of its GDP.4 The distribution of income remains highly unequal, with the richest 20 percent of the population accounting for more than 51 percent of Guatemala’s overall consumption. More than half of the population is below the national poverty line (59.3 percent), and 23 percent of the population lives in extreme poverty. Poverty among indigenous groups, which make up more than 40 percent of the population, averages at 79 percent, with 40 percent of the indigenous population living in extreme poverty. Nearly one-half of Guatemala’s children under 5 years of age are chronically malnourished, one of the highest malnutrition rates in the world.5 Guatemala is among the most unequal countries in Latin America. It also has the highest incidence of poverty, especially for the indigenous population.6 It struggles in several areas of health and development, including infant, child, and maternal mortality, malnutrition, literacy, and contraceptive awareness and use. The country’s large indigenous population is disproportionately affected. Guatemala is the most populous country in Central America and has the highest fertility rate in Latin America. It also has the highest population growth rate in Latin America, which is likely to continue because of its large reproductive-age population and high birth rate. Almost half of Guatemala’s population is under 19 years of age, making it the youngest population in Latin America. Guatemala’s total fertility rate has slowly decreased during the last few decades due in part to limited government-funded health programs; however, the birth rate is still close to three children per woman and is markedly higher among the rural and indigenous populations.7 Guatemalans have a history of emigrating legally and illegally to Mexico, the United States, and Canada because of a lack of economic opportunity, political instability, and natural disasters. Emigration, primarily to the United States, escalated during the 1960 to 1996 civil war and accelerated after a peace agreement was signed. Thousands of Guatemalans who fled to Mexico returned after the war, but labor migration to southern Mexico continues. Recent migration to these countries has increased due to drought, poor crop production, and lack of employment opportunities.8 Migration from the Western highlands of Guatemala has been increasing in recent years due to poverty and lack of economic opportunities; 14 percent of this population have relatives who have migrated.9 3 CIA World Factbook. Guatemala. 2017. https://www.cia.gov/library/publications/the-world-factbook/geos/gt.html 4 Ibid 5 Ibid 6 Cabrera M., Lustig N., Moran H. Fiscal Policy, Inequality, and the Ethnic Divide in Guatemala. World Development Vol. 76, pp. 263–279, 2015. http://dx.doi.org/10.1016/j.worlddev.2015.07.008. 7 CIA World Factbook. Guatemala. 2017. https://www.cia.gov/library/publications/the-world-factbook/geos/gt.html 8 Miroff, N., Sieff, K. “Hunger not violence, fuels Guatemala migration surge, U.S. says.” Washington Post. September 22, 2018. 9 Orozco, M. Inter-American Dialogue. Patterns of Central American Migration. July 2017. https://www.thedialogue.org/wp￾content/uploads/2017/08/Patterns-of-Central-American-Migration-Final.pdf Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 2 Guatemala is among the 10 countries that are most vulnerable to natural disasters and the effects of climate change.10 Four consecutive years of drought in Central America’s Dry Corridor, a region encompassing a large portion of southern Guatemala, have exacerbated food insecurity among poor households, particularly in the Western Highlands. The continuing poor harvests have led to limited agricultural production, lost labor options, and reduced household incomes, making it difficult for families to access food. In addition, poor soil conditions, over-exploitation of forest resources, degraded lands, small size of plots, and lack of access to credit, agricultural supplies, and technical assistance drive agricultural productivity and profitability further down. Although increased food production during the summer months alleviates some conditions, poor households still remain stressed due to indebtedness and loss of assets from the previous years of low agricultural production. Although the economy has diversified somewhat in recent decades, agriculture is still the country’s largest employment sector, now accounting for about a third of the active labor force. But national data suggest that poverty rates are nearly twice as high among agricultural workers (74 percent poor) as among those working in industry (44 percent poor), and more than three times higher than among those working in commerce (23 percent poor).11 The Western Highlands, which is primarily agricultural, is one of the poorest regions. Despite government and donor investments in the Western Highlands, the rate of malnutrition has remained high. The national prevalence of stunting has shown alarmingly slow improvement during more than 20 years of peace. Stunting was at 55 percent in 1995, at the end of the decades-long civil war. The most recent 2014-15 Demographic and Health (DHS) survey (known by its Spanish language acronym ENSMI) found that 46.5 percent of children under age 5 were stunted, representing, the fourth highest ratio in the world, with no improvement since the previous DHS for the stunting rate of 58 percent among indigenous children.12 Among the poorest fifth of the population, two-thirds of children are stunted. The persistent lack of clean water and adequate hygiene and sanitation infrastructure and practices have been cited as a primary driver of stunting outcomes.13 1.2 PROJECT DESCRIPTION Save the Children was awarded the PAISANO (Programa de Acciones Integradas de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) development food assistance project (DFAP) by the United States Agency for International Development (USAID) Office of Food for Peace (FFP) in August 2012. The approximately $48 million project was implemented by Save the Children in partnership with Project Concern International (PCI). Save the Children was building on its experience of more than 10 years of implementing food security programs in Guatemala’s Western Highlands, and PCI had 37 years of implementing programs in the Western Highlands. The PAISANO project was conceived to build on lessons learned and strategies that had been successful in previous Save the Children and PCI projects. PAISANO’s goal has been to reduce food and nutrition insecurity among 26,500 vulnerable rural households in 13 municipalities in 3 departments of the Guatemala Western Highlands (Save the 10 United States Agency for International Development. Food Assistance Fact Sheet—Guatemala. 2018 11 Hamel, R. “Tracking Promises: Analyzing the Impact of Feed the Future in Guatemala.” June 2017. Center for Strategic International Studies. https://csisprod.s3.amazonaws.com/s3fspublic/publication/170614_Hamel_TrackingPromisesGuatemala.pdf?c8wlSLL9Mb.Gq0Lp w1m7lw9G435teJv7 12 MSPAS. VI Encuesta Nacional de Salud Materno Infantil 2008 (ENSMI2014-20015). Ministerio de Salud Publica y Asistencia Social (MSPAS), Institucion Nacional de Estadistica (INE), Secretaria de la Planificacion y Programacion de la Presidencia (SEGEPLAN), Guatemala 2017. 13 Hamel, R. “Tracking Promises: Analyzing the Impact of Feed the Future in Guatemala.” June 2017. Center for Strategic International Studies. https://csisprod.s3.amazonaws.com/s3fspublic/publication/170614_Hamel_TrackingPromisesGuatemala.pdf?c8wlSLL9Mb.Gq0Lp w1m7lw9G435teJv7 Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 3 Children in 8 municipalities in Quetzaltenango and Quiche, and PCI in 5 municipalities in Huehuetenango). The project targets rural communities suffering from poverty, food insecurity and at risk of food insecurity, and communities with a high prevalence of chronic malnutrition. It initially selected project participants from households with pregnant and lactating women and children under 2 years of age (1,000 days). To achieve its goal, the project aimed to increase household access to food, reduce malnutrition among children under 5 years of age, and improve community resilience. The project also worked to ensure gender equality, mitigate environmental consequences, and influence behavior change. In fiscal year (FY) 2015, during its third year of the project, PAISANO underwent a mid-term evaluation (MTE). The MTE was led by FFP and conducted alongside the Catholic Relief Services project, Seguridad Alimentaria Enfocada en los Primeros 1,000 Dias. The MTE report made several recommendations for the project. Among the most significant strategic objective 1 (SO1) and SO2 recommendations were the following: refining the beneficiary targeting strategy to better tailor and differentiate interventions to poor and extremely poor households; increasing consistency and quality of support; increasing linkages to agricultural value chains; placing a stronger emphasis on adult education and social and behavior change communication (SBCC) techniques; implementing graduation elements and sustainability; formally incorporating water, sanitation, and hygiene (WASH) activities; and improving links with the country’s health systems. In SO3, the recommendations were focused on emphasizing work with municipal and community coordinating bodies, such as Municipal Development Committees (Committees Comités de desarrollo municipal [COMUDEs]) and Community Development Councils (Consejo Comunitario de Desarrollo [COCODEs]) for disaster risk reduction to improve governance and resilience, as well as better integrating youth into the project. During FY 2016 and FY 2017, PAISANO made a number of adjustments to its project in response to the recommendations. It collaborated with the USAID Guatemala Mission, the Western Highlands Integrated Program, and other USAID projects on water, in particular regarding standardizing messages and revising materials for safe water consumption and water chlorination. In 2017, PAISANO replaced the food rations that it was giving to pregnant women and mothers of children under 23 months of age with cash to allow participants to buy local food. The cash transfer program reached 7,500 households. During both years, PAISANO improved access to nutritious foods through improved bean production, small livestock, and home gardens. To promote sustainability, the project coordinated with the Government of Guatemala and nongovernmental organizations (NGOs)14 to promote continued access to financial support for community projects. This review will focus on the PAISANO project from 2013 to 2018, noting its progress up to and after the MTE as well as any changes in interventions and activities. 1.3 THEORY OF CHANGE In FY 2014, FFP began to institute a more robust evidence-based process of project design, by requiring that implementing partners (IPs) develop a comprehensive theory of change (TOC) for programming their activities.15 Although the PAISANO award granted in 2012 preceded FFP’s new strategic framework on food security, the Results Framework developed by the project enables one to posit a 14 Municipalities; Municipal Development Committee; Municipal Commission for Health, Food and Nutrition Security (Comisión Municipal de Salud, Seguridad Alimentaria y Nutricional); Ministry of Health; Ministry of Agriculture (MAGA); Ministerio de Educacion; Secretaría de Seguridad Alimentaria y Nutricional; CARITAS (charity); ECOFILTRO; Agua del Pueblo; Liga de la Leche Guatemala; Engineers without Borders; Escuela de Formacion Agricola, a MAGA academic center; Communities Leading Development project; Area Directorate of Health (Dirección del Area de Salud); among others 15 USAID. 2016. 2016–2025 Food Assistance and Food Security Strategy. Democracy, Conflict, & Humanitarian Assistance (DCHA)/Office of Food for Peace. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 4 comprehensive development hypothesis, or TOC, that reflects the goals, objectives, and outcomes to be achieved by the implementing organization. The stated goal is that food insecurity and malnutrition will be reduced among poor, vulnerable households in the Western Highlands of Guatemala. The working hypothesis is that if access to food by vulnerable rural households is increased (SO1), if malnutrition among children under 5 years of age is reduced and the nutritional status of pregnant and lactating women is improved (SO2), vulnerability to food security shocks is decreased and community resilience is increased (SO3), then decreases in food insecurity and malnutrition among poor households should be achieved. The hypothesis stresses the close interrelationship between malnutrition and food insecurity in reaching the overall goal. The goals and strategic objectives and intermediate results and supporting activities are found in the project’s Results Framework (see Annex 1). Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 5 2. EVALUATION OVERVIEW 2.1 EVALUATION PURPOSE This report details the findings and recommendations of the final performance evaluation (PE) of the PAISANO project, funded by USAID/FFP. The purpose of the PE was to measure the performance and development outcomes of the project, provide an evidence base for the project’s effects on key objectives, and provide recommendations to FFP for the future design of Development Food Security Activities. The evaluation was conducted from July 2018 to January 2019 by ICF International. A field team was assembled for the qualitative component of this work and was composed of Sandra Wilcox (team lead), John Russell (agriculture and livelihoods), Kali Erickson (nutrition and SBCC), and Carlos Flores (WASH and community resilience). The analysis of the quantitative population-based survey (PBS) data to inform the evaluation was led by Dr. Gheda Temsah (ICF). ICF provided logistic and quality control support throughout the evaluation. The evaluation’s results are aimed at multiple audiences. The findings are expected to have primary accountability and learning value to USAID (FFP/Washington, USAID/Guatemala, and the FFP learning network) as well as Save the Children and PCI and their sub-partners. Additional stakeholders include the Guatemalan municipal government officials as well as Ministry of Health (MOH) and Ministry of Agriculture (MAGA) department officials. USAID/FFP will use the findings, conclusions, and recommendations of the evaluation to extract lessons learned and generate insights to inform the design of future FFP activities. Evaluation recommendations and findings may also be used by FFP internally to refine Development Food Assistance Project (DFAP) proposal guidelines and policies. 2.2 EVALUATION QUESTIONS This section provides the evaluation questions (EQs) based on the Scope of Work (See Annex 2) for the final evaluation of the FFP DFAPs in Guatemala. The summative PE provides a substantive analysis of four key topical areas. EQ1: Overall Achievements: To what extent has PAISANO met its defined goals, purposes, and outcomes, and what factors promoted or inhibited their achievement? EQ1.1: Change in indicator values between baseline and endline and project performance on final indicator values against targets set by both the partners and the key FFP indicators that include poverty, stunting, and underweight EQ1.2: Factors that promoted or inhibited the achievement of the project objectives, including, but not limited to, the effectiveness of food-for-asset or cash-for-asset interventions EQ1.3: Targeting strategies and their contribution to achieving project goals (especially with regard to gender and reaching the most vulnerable) EQ 1.4: Based on evidence from sub-questions 1–3, what has been the contribution of PAISANO to USAID’s efforts to reduce food insecurity among vulnerable rural populations in five departments of the Western Highlands of Guatemala? EQ 2: Sustainability: Based on the evidence, which project outcomes are likely to be sustained? EQ2.1: Quality and functionality of the processes, systems, and institutional arrangements developed or strengthened to sustain the necessary and critical services EQ2.2: Communities’ perceptions on the quality, frequency, effectiveness, and sustainability of the services provided by the project Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 6 EQ2.3: Motivation of the community and participants to demand and pay (or invest time) for the services EQ2.4: Whether the necessary resources and capacity strengthening will exist to sustain service providers EQ2.5: Extent to which the projects leveraged other U.S. Government (USG) and non-USG investments to achieve sustained outcomes as identified in the TOC EQ2.6: Evidence of linkages with other service providers EQ3: Effectiveness: In each technical sector, what are the strengths and challenges of the effectiveness of the interventions’ implementation? EQ3.1: The effectiveness and relevance of the technical interventions, including food-for asset and cash-for-asset interventions, to achieve project outputs of greater or lesser quality, and discussion of those findings in relation to the project’s TOC EQ3.2: Factors in the implementation approach and methods, and the context associated with greater or lesser effectiveness in producing outputs of higher or lower quality EQ3.3: The interventions and implementation processes deemed more or less acceptable to members of the target communities EQ4: Lessons Learned and Best Practices: What are the key lessons learned and best practices that should inform future projects in Guatemala? EQ4.1: What are the lessons, best practices, and approaches that should be considered in designing future food and nutrition security projects? EQ4.2: What are the lessons, best practices, and approaches that should be considered in implementing and implementation approaches for future food and nutrition security projects? EQ4.3: What are the risks to the effective implementation of these approaches to support project achievements? EQ4.4: Based on the lessons learned, how can potential unintended negative consequences be minimized and positive consequences be captured systematically? Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 7 3. EVALUATION METHODS AND LIMITATIONS 3.1 DATA COLLECTION METHODS The PE for the PAISANO project used a mixed methods design that integrated data from multiple sources (See Annex 3 for a full description of the evaluation protocol). The evaluation drew on data collected from a PBS to measure current status and change over time for 28 key FPP indicators and a companion qualitative data collection effort conducted at endline to provide the following: additional context and understanding of beneficiary perceptions of project achievements; changes in key outcomes and reasons why; and additional information from project IPs and related stakeholders on issues of project effectiveness, sustainability, unintended consequences, and lessons learned. In addition to the quantitative PBS data and primary qualitative data collected from project participants and other key stakeholders, the final PE also drew on secondary data from IP performance monitoring, key IP documents, and the MTE. The availability of pre-post quantitative data on FFP indicators substantially increased the ability of the evaluation team to derive evidence-based findings, conclusions, and recommendations for this evaluation. 3.1.1. Quantitative Data Collection Methods and Analysis: PBS The quantitative results for the PE use the data collected through baseline and endline PBSs that are statistically representative of the areas where the PAISANO project was implemented. The baseline PBS was conducted from April to mid-June 2013, and the endline PBS was conducted from June to July 2018. The primary objective of the baseline PBS was to assess the status of FFP and custom indicators prior to project implementation; the primary objective of the endline PBS was to assess the change in these same indicators at the end of the six-year DFAP cycle by undertaking statistical tests of differences between baseline and endline measures. This pre-post design allows for the measurement of change in indicators over time; however, this design does not allow statements to be made about attribution or causation relating to project impact. The indicators are related to food security, poverty, women’s and children’s health and nutritional status (including anthropometry), WASH, and agricultural practices. Custom indicators were added by the projects to assess women’s health care awareness and decision-making for themselves and their children. The endline PBS used the same data collection instrument as the baseline PBS, with additional questions to calculate the Food Insecurity Experience Scale and the Poverty Probability Index (PPI) as well as questions on mobility, remittances, and exposure to project interventions. It is important to note that the population-based data from PAISANO communities included data on PAISANO participants and non-participants. Comparison of endline data for participants and non-participants is complicated by the fact that after the MTE, PAISANO began to apply a set of poverty criteria for selection of new families. As a result, participating families who entered after the MTE were more likely to be impoverished and have other risk factors for food insecurity and malnutrition16 compared to non-participants.17 16 For example, more dependent children in the family and poor household infrastructure 17 In addition, although all families with pregnant women or children under 23 months of age were invited to participate in PAISANO through the MTE, not all invited households chose to participate. Project volunteers mentioned that a primary reason why some households declined to participate was the perceived time commitment required, and a secondary reason was not wanting to receive home visits from PAISANO volunteers and staff. These factors distinguish PAISANO participants and non-participants related to time constraints or social factors that they face. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 8 Sampling The household samples selected for the baseline and endline PBSs are based on a multi-stage clustered sample designed to power a test of differences between the baseline and endline estimates for the FFP child stunting indicator. The sample size is derived based on a formula to detect a difference in the prevalence of stunting from baseline to endline of 6 percentage points with 95 percent confidence and 80 percent power. Based on this calculation, the baseline PBS included 75 communities and 3,000 households, and the endline PBS included 35 communities and 1,400 households.18 These sample size calculations included a pre-inflation factor to take into account anticipated household-level non-response as well as an adjustment factor to take into account the number of households that need to be contacted to achieve the required sample size of children under 5 years of age. A detailed description of the endline sampling plan is provided in the Guatemala Endline PBS Protocol, and a detailed description of the baseline sampling plan can be found in Annex 1 of the baseline study report.19 Data Analysis The baseline and endline PBS data were used to assess change over time in the 20 FFP and 8 custom indicators (see Annex 4 for a tabular summary of these indicators and tests of differences for change over time from baseline to endline). The data were also used, as appropriate to the EQ, to conduct additional descriptive and bivariate analyses to help strengthen findings established from the quantitative and qualitative data. Multivariate analyses controlling for key socioeconomic and project-specific factors as covariates were used to explore the factors that are associated with the prevalence of stunting among children under 5 years of age, and daily per capita consumption expenditures. These indicators were selected for additional analyses to help inform learning on key impact and outcome indicators. The multivariate models pool data from the baseline and endline surveys and control for a number of confounding factors in addition to project participation status to empirically explore hypothesized relationships between indicators and to determine whether project participation is associated with change over time net of other factors. Additional analyses explored indicator estimates at endline by participation status and compared the endline estimate of direct and indirect participants to baseline values to evaluate if the effects of project interventions “spilled over” from direct participants to indirect participants – as is desired because the intent of the FFP projects is to expressly induce spillover effects from direct beneficiaries to indirect beneficiaries over the course of the DFAP. The designation of direct versus indirect participant households was based on household respondents’ answers to a set of questions about exposure to the project interventions. One respondent per household was asked, “Have you or someone from your household participated in PAISANO project activities?” Respondents who answered “Yes” were considered to be project participants. Respondents who answered “No” were considered to be indirect participants. Respondents who reported that they or someone in their household participated in the project were asked about the type of assistance 18 Because the input parameters were somewhat different for the two time points, the sample size requirement for endline was updated from a baseline value of 1,686 children per project to an endline value of 895 children per project, given that this combination of sample sizes for baseline (1,686) and endline (895) ensured an overall power of 80 percent for the two time points. In principle, the sample size of required children should be identical for the baseline and endline PBSs, because the computation is made at the time of the baseline to ensure that the pair of sample sizes provides an overall power of 80 percent. However, the actual realized sample size of children from the baseline PBS (2,778 children per project) greatly exceeded the target sample size. This implied that the overall power of the statistical test would likely be considerably greater than 80 percent had the same sample size used at baseline also been used at endline, because the power of the statistical test is determined by the actual realized sample size of children at the two time points. Therefore, it was possible to reduce the sample size at endline relative to what it was at baseline and still maintain 80 percent power overall. 19 A complete description of the baseline study is provided in the Baseline Study for the Title II Development Food Assistance Projects in Guatemala report. Available at: https://www.usaid.gov/sites/default/files/documents/1866/Guatemala%20Baseline%20Study%20Report%2C%20March%202014.pd f Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 9 received. Project assistance could include food rations, cash transfers, nutrition training or meetings, agriculture-related training or meetings, or other types of assistance. The responses were not validated by the projects, so it was not possible to determine definitively that the respondents accurately reported the source and type of project assistance. Respondents or households were likely to be recipients of more than one type of assistance, so it was not possible to disentangle the effects of different types of project interventions. 3.1.2. Qualitative Data Collection The qualitative component of the PE drew on the following data collection methods:  Desk review of IP quarterly and annual performance reports, monitoring and evaluation (M&E) data, MTE report, and relevant secondary information  Focus group discussions (FGDs) conducted at endline with project direct and indirect participants, as well as community, agricultural leaders, and mother leaders in eight communities  Key informant interviews (KIIs) conducted at endline at the community, municipal, and department levels  In-depth interviews (IDIs) conducted selectively at endline at the community, municipal, and departmental levels  Observations conducted at endline during community home visits The qualitative data collection process used purposive sampling and semi-structured FGD and KII protocols. The qualitative fieldwork for the endline PE was conducted between September and October 2018. Full quantitative results were not available in time to fully inform the design of the qualitative instruments (such as for the poverty indicators), which limits the ability of the qualitative data collection to fully explain some of the quantitative results. However, some preliminary results from the PBS were available during qualitative data collection (for example, with respect to the WASH and diarrhea indicators), and those findings were used to help direct follow-up probing on those issues. (See Annex 3 for field visit plans and groups and individuals interviewed.) The four-person evaluation team was composed of an experienced evaluation team lead and three subject matter experts, whose areas of expertise included agriculture and livelihoods, health and nutrition, water and sanitation behavior change and infrastructure, resilience and disaster risk reduction, community mobilization, governance, behavior change communications, and adult education. Team members focused on their specific areas of expertise and conducted data collection individually and in pairs in cases in which interviews, group meetings, or observations presented the opportunity to collect information across thematic areas. The evaluation team consisted of the following members: Sandra Wilcox assumed overall team coordination for the evaluation, providing guidance on following the scope of work to ensure that information was collected and analyzed as planned and that the team maintained objectivity and followed the proposed timeline. She reviewed the M&E systems and management structure and functionality. Kali Erickson, with expertise in maternal and child nutrition and health, reviewed activities related to infant and young child feeding practices, maternal health and nutrition practices, growth monitoring, and behavioral change communication strategies to promote behaviors to reduce malnutrition. She also reviewed the use of food assistance and cash assistance promoted to improve household and maternal and child nutrition. John Russell, with expertise in agriculture and livelihoods, reviewed interventions related to agriculture and livelihoods promotion. He reviewed approaches designed and implemented by Save the Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 10 Children and PCI and partners to transfer knowledge and practices related to agriculture and other livelihood strategies. Carlos Flores, with expertise in WASH, resilience, disaster risk reduction, and governance, reviewed interventions related to these areas, including community development and WASH- and resilience-related behavior change strategies. Dr. Gheda Temsah (ICF) provided the analysis of the quantitative PBS data to inform the evaluation. Focus Group Discussions The evaluation team conducted 34 FGDs at endline, across 8 communities (distributed in 7 different municipalities over the three departments). FGDs comprised 217 participants in total. This is higher than the number planned (21 FGDs with 96–185 participants) due to having more groups and participants than initially expected.20 The initial plan included more KIIs with participants and leaders in several of the communities visited. Once in the field, the evaluation team changed this interview structure to FGDs, allowing the team to capture opinions of a larger number of participants.21 The different groups included the following: agriculture and health and nutrition participants; men’s groups (participants and non￾participants); youth groups (participants and non-participants); women’s empowerment groups (participants and non-participants); community leader groups (COCODE, the Municipal Food and Nutrition Security Commission, and the Food Security Committee [Comision de Seguridad Alimentaria Nutricional (COSAN)]); mother leader and nutrition promoter groups (participants from several surrounding communities); and agriculture leaders and promoters (participants from several surrounding communities per group). FGDs were organized with the assistance of community leaders and PAISANO field agents and comprised between 3 and 19 participants, with a median of 8 participants. A Maya-speaking translator conducted the FGDs in the local language predominantly spoken in the communities’ areas. Evaluation team members took notes of interview discussions. All participants were informed and requested to consent to the interviews before beginning the FGDs. FGDs were generally held in meeting halls and other common gathering areas in the respective communities. Semi-structured instruments with questions organized by evaluation theme were used to guide the discussion (see Annex 3 for the FGD interview guides). Key Informant Interviews At endline, the evaluation team conducted a total of 78 KIIs. This includes 18 KIIs at the municipal and departmental levels (including representatives from MAGA, MOH, Secretaría de Seguridad Alimentaria Nutricional (Secretariat for Food and Nutrition Security) [SESAN], Woman’s Resource Program, mayors, partner agencies, and health post staff) and 60 KIIs at the village level (including mother leaders, agricultural leaders, project health and agriculture participants, community leaders, and project staff). The interviews followed a semi-structured format to allow for follow-up questions and flexibility in the discussion. The KII protocol was structured to gather information the extent to which activities have been achieved and on which factors promoted or inhibited project activities and outcomes, perspectives on the effectiveness of project interventions and targeted groups, quality of services provided, motivations and capacity to demand and sustain services, and the sustainability of project interventions and outcomes (see Annex 3 for the KII interview guides). In-depth Interviews To more thoroughly explore certain themes and aspects of the PAISANO project, the team also conducted 13 IDIs with project technical staff, (nutrition, agriculture and livelihoods, WASH, gender) 20 Final Evaluation Protocol. Final Performance Evaluation of Integrated Food Security and Food Actions Project/Programa de Acciones Integradas de Seguridad Alimentaria y Nutricional del Occidente (PAISANO). September 2018. 21 Final Evaluation Protocol. Final Performance Evaluation of Integrated Food Security and Food Actions Project/Programa de Acciones Integradas de Seguridad Alimentaria y Nutricional del Occidente (PAISANO). September 2018. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 11 agricultural leaders, mother leaders, nutrition promoters, project participants, and community leaders. The interviews followed a semi-structured format, and probing follow-up questions gave the respondents an opportunity to respond more in depth about the project interventions, strategies, challenges, and accomplishments. Community and Household Observations The evaluation team conducted 18 community and household observations in the 8 communities. These observations involved review of the following areas: water and sanitation conditions related to practices, such as tippy taps, safe, clean play spaces for child; agricultural fields or livestock systems, such as animal pens, etc.; and water and sanitation infrastructure, such a water tanks or latrines. In each community visited, usually at the end of a group discussion, evaluation team members asked to visit the houses of a small number of individual group participants. These visits also provided an opportunity to talk to neighbors of the visited household to include information from non-participants. Community Selection for Qualitative Data Collection Community selection for qualitative data collection was based on maximizing coverage on a range of different project activities, drawn from IP data on activity implementation by project regions. The team chose areas representative of the geographic regions and linguistic communities in which the PAISANO project was implemented. Eight communities, belonging to seven municipalities of the departments of Huehuetenango, El Quiché, and Quetzaltenango, were selected. In these communities, the K'iché, Ixil and Mam language communities were represented. Because the randomly selected sample in which the PBS was conducted is statistically representative of the complete area where PAISANO worked, the evaluation team selected sites from these communities. The evaluation team obtained community lists for the PBS endline survey with community population size, number of households, language spoken, and distance from the municipal head. The team then mapped the location of the communities that participated in the project and chose a route that included communities that met the criteria previously mentioned, considering travel and time to conduct the field work. The evaluation team then verified with Save the Children staff that this was a feasible route and that the communities were safe to visit. Save the Children staff indicated the interventions they had carried out in each community. Initially all communities received the same intervention activities. After the MTE, however, interventions were prioritized based on community needs and appropriateness (see discussion in Section 4.4). In communities that had experienced most of the project interventions (maternal and child health and nutrition [MCHN], WASH, and livelihoods), the evaluation team randomly selected communities among three or more possibilities. In cases of communities with specific activities (such as Training for Water Committees, Groups of Empowered Women, and Marketing Groups), it was not possible for the evaluation team to have more than one option, and these communities were purposively selected. As a result of this process, the team selected eight communities (distributed in seven different municipalities over three departments), as well as municipal level interviews in three municipalities (see Annex 3 for the list of communities and municipalities visited.) Due to a national strike, however, the team was not able to visit one of the planned communities (Chiul in Cunen department), and instead, through the help of Save the Children, staff were able to arrange last-minute interviews with the municipality and department offices in Chichicastenango. During the second week of field visits, when visiting Huehuetenango, PCI was able to arrange for the team to visit two communities instead of the one planned, so the evaluation team was able to complete the planned number of communities to be visited and add one municipality, for a total of eight communities and three municipalities visited. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 12 Desk Review of IP and Secondary Documentation The purpose of the desk review of IP reports and other available secondary documentation was to identify key findings and explanatory factors from IP reports and internal M&E data pertaining to each of the EQs. Examination of key documents was also used as a source of triangulation for qualitative data provided by project participants and KIIs to help interpret or provide explanatory context for both PBS and qualitative results. A discussion of project reports and secondary documents is provided in the PAISANO Inception Report.22 Analysis, Coding, Interpretation Methods The evaluation team documented data from semi-structured key informant and group interviews in interview notes with consent of participants (see the Protocol Report for consent forms). Following initial creation of a “story line” describing the goals of the analysis, notes of the FGDs were coded manually, using a hybrid of both pre-set and emergent codes. Content analysis identified themes relevant to answering the PE EQs, as related to the goals and activities of the project and of each sector (agriculture/livelihoods, MCHN, disaster risk reduction, and WASH), the expected outcomes, and the perceived effectiveness of interventions.23 The analysis also recorded the frequency of themes and their content to better understand the meaning and context in which statements were made. Each interview tool was coded separately. KII notes were summarized using standard content analysis techniques.24 For data from the KIIs, FGDs, and IDIs, content analyses was used to summarize common trends and patterns to highlight project-, sector-, and gender-differentiated trends, and to identify examples of positive deviance and most significant change. All interim data (i.e., field notes) generated during the evaluation were deposited in a Dropbox to which only the evaluation team and necessary ICF staff had access. The data will be removed at the close of the evaluation. 3.2 LIMITATIONS The primary methods of data collection were key informant and group interviews. If respondents did not consistently interpret questions as the evaluation team intended, the resulting data may not be completely valid or reliable. In addition, because many respondents communicated in local Maya languages, the evaluation team needed to rely on translators. Translation usually introduces some loss of fidelity of data. To minimize this, the evaluation team trained the translators and worked with them to verify information during and after interviews to ensure that responses were documented and interpreted as accurately as possible. The evaluation team also triangulated evidence from a variety of methods and sources (Section 3.1.2) to ensure validity and reliability. Respondents may not have accurately recalled whether past project activities influenced their knowledge, attitudes and behavior. Thus, poor recall may have affected data reliability. To mitigate this source of error, the evaluation team designed the interview instruments to help respondents recall their participation in activities and the effect of this participation on their knowledge and behaviors. Although the final selection of sites was done independently and randomly by the evaluation team, based on specific objective selection criteria, the accessibility of sites was a selection criteria and was done to facilitate and ensure feasibility of travel, given the time constraints faced by the PE. However, this may have introduced a small amount of bias into the overall procedure. In addition, as discussed in 22 USAID. Food for Peace. Final Performance Evaluation of the Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nacional de Occidente [PAISANO]). Inception Report. August 2018. 23 Saldaña, J. (2015). The Coding Manual for Qualitative Researchers. (3th Ed.). SAGE Publications. 24 Kippendorf, K. (2018). Content Analysis: An Introduction to Its Methodology (4th Ed.). SAGE Publications. Content analysis is a technique for systematically describing written, spoken, or visual communication. It provides a quantitative (numerical) description. Many content analyses involve media—print (newspapers, magazines), television, video, movies, the Internet. Content refers to the meaning of the information. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 13 Section 3.1.2, due to a national strike, the team was unable to travel to one of the planned communities; to mitigate this, the team was able to substitute another community later in the visit. To address and mitigate these limitations, the evaluation team limited the evaluation activities to eight communities and three municipalities in the three departments of Quiche, Huehuetenango, and Quetzaltenango, as described in Section 3.1.2, which is the number of representative communities and municipalities that could be accessed in the time available. The team strengthened the findings from interview responses by triangulating the findings relating to each SO through a comparison of responses for selected evaluation questions across the different regions and municipalities. Using a mixed methods evaluation approach, the evaluation findings were further strengthened by triangulating them with quantitative findings from the analysis of the 2018 PBS endline data, comparisons of the baseline and endline data, and analysis of the life of activity (LOA) indicator data comparing project targets to actual performance monitoring data from the Indicator Performance Tracking Tables (IPTTs) in the IPs’ Annual Results Reports (ARRs). Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 14 4. EVALUATION FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 4.1 SO.1: HOUSEHOLD ACCESS TO FOOD 4.1.1 EQ1: To what extent has PAISANO met its defined goals, objectives and outcomes? Findings: Household Dietary Diversity, Poverty, and Agriculture Household dietary diversity score (HDDS) increased from 6.1 to 7.0 over the LOA. Improvements in HDDS were assessed by participation status. Households that reported participating regularly in the PAISANO project activities are considered direct participants, and households that did not participate regularly are considered indirect participants (see Annex 5, Table A5.1b for details on participation status). Improvements in HDDS were experienced by both direct and indirect participants (see Annex 5, Table A5.2 for details on baseline-endline change in HDDS by participation status and a comparison of HDDS at endline by participation status). Figure 1 illustrates an increase in the consumption of almost all the “strength foods” and “protection foods” food groups promoted by the project—namely, plant and animal-based proteins as well as vegetables (see Annex 5, Table A5.3 for details). Figure 1: Food Groups Consumed by Households ** ** † *** ** *** ** *** *** † 0 20 40 60 80 100 Cereals Roots and tubers Vegetables Fruits Meat, poultry and organ meat Eggs Fish and seafood Pulses/legumes/nuts Milk and milk products Oils/fats Sugar/honey Miscellaneous (tea,coffee, condiments, etc…) Percentage 2013 Baseline 2018 Endline no symbol=not significant, † p<0.1, * p<0.05, ** p<0.01, *** p<0.001 Data source: PAISANO Baseline and Endline PBS Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 15 HDDS is a count of the number of food groups that households on average consume from a total of 12 groups but is generally considered a proxy for socioeconomic status. Improvements in households’ socioeconomic status are further underscored by parallel improvements in the poverty indicators. Per capita consumption expenditures, a proxy for income, increased by 34 percent over the life of the project, from $1.89 at baseline to $2.54 at endline. This increase in daily per capita consumption expenditures exceeded the project target. There were also marked decreases in the prevalence of poverty (which decreased by more than two-thirds) (Figure 2) and mean depth of poverty (which decreased by more than three-fourths) (Figure 3). Improvements in poverty were experienced by all households in the PAISANO project area, regardless of whether households participated directly in the project activities (direct participants) or did not (indirect participants). At endline the daily per capita consumptions of direct participants ($2.80 at constant USD 2010) was higher than that of indirect participants ($2.29 at constant USD 2010), and the prevalence of poverty for direct participants (11.9 percent) was lower than that of indirect participants (17.2 percent). Annex 5, Table A5.2 provides details on the baseline-endline change in poverty by participation status and a comparison of poverty at endline by participation status. Figure 2: Prevalence of Poverty: Percentage of People Living on Less Than $1.25 per Day 44.2% 14.7% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Percentage 2013 Baseline 2018 Endline *** *** p<0.001 Data source: PAISANO Baseline and Endline PBS Figure 3: Mean Depth of Poverty 11.8% 2.6% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Percentage 2013 Baseline 2018 Endline *** *** p<0.001 Data source: PAISANO Baseline and Endline PBS There is evidence both from the qualitative study and from quantitative analyses of the PBS data that PAISANO’s SO1 activities may have contributed to these positive results for poverty. Sales of project￾promoted crop and animal products were cited by project participants and project staff as welcome Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 16 sources of additional income, which contributes to improved economic access to food. In addition, the fruit and vegetable gardening and the goat, rabbit and poultry production activities of the project directly improved household nutrition through increased home consumption, that is, increased physical access to food. In particular, animals distributed by the project represented valuable productive long-term assets for households. The Centro de Producción Caprina Del Altiplano (CEPROCAL) and the Centro Integral de Producción y Capacitación Agroambiental (CIPCA) were established and produced forecasted results, both for production and educational purposes. A number of seed and seedling enterprises were established, producing income for producers and high-quality inputs for household gardens and tree￾planting efforts. Although the evaluation team did not visit any Product Aggregation Centers, project documents examined indicated that they were independently and competently managed by grower associations, some of which benefited from training in value chain management, linkages with Feed the Future value chain projects, and sales contracts facilitated by PAISANO, which led to increasing sales, particularly of potatoes. Quantitative analysis indicated that use of value chain practices, sustainable agriculture practices, and improved storage increased over the life of the project (see Figure 4). Each of the value chain activities promoted by the project increased over time (see Annex 5, Table A5.4 for details). The increase in the use of at least three sustainable agricultural practices is underlined by an increase in the use of at least two sustainable livestock practices. The use of at least two sustainable crop practices and at least two sustainable natural resource management practices did not change over the life of the project. Annex 5, Table A5.5 provides baseline-endline comparison of use of sustainable agriculture practices by type. The endline results also indicate that the project achieved LOA targets for use of improved storage practices (target 12 percent), three sustainable agriculture practices (target 60 percent), and use of value chain practices (target 20 percent). The use of financial services did not change over time, and at endline fell slightly short of the LOA target of 25 percent. Figure 4: Baseline–Endline Comparison of Agriculture Indicators 20.2% 15.9% 50.2% 8.7% 21.1% 28.3% 60.6% 18.9% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Percentage of farmers who used financial services in the past 12 months Percentage of farmers who practiced value chain activities promoted by the project in the past 12 months Percentage of farmers who used three sustainable agricultural practices in the past 12 months Percentage of farmers who used improved storage practices in the past 12 months 2013 Baseline 2018 Endline * *** *** ns ns=not significant, † p<0.1, * p<0.05, ** p<0.01, *** p<0.001 Data source: PAISANO Baseline and Endline PBS Table 1 illustrates a correlation between the adoption of recommended agriculture practices and daily per capita consumption expenditures (DPCCE), which is consistent with reports from the majority of respondents who said that they had gained modest income benefits from project agricultural and livelihood activities. See Annex 5, Tables A5.6–A5.8 for additional details on the results of the bivariate analyses of the relationship between daily per capita consumption expenditures, the prevalence of poverty, and mean depth of poverty with household agriculture status at endline. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 17 Multivariate analyses suggest that improvement in daily per capita consumption expenditure between baseline and endline can be partially explained by households’ use of agricultural practices promoted by the project. Specifically, acquiring agricultural credit, making savings, using at least two value chain activities, using at least two sustainable crop practices, and using at least two sustainable livestock practices promoted by the project are associated with an increase in daily per capita consumption net of other factors and account for about 13 percent of variability in DPCCE. These associations remain statistically significant even after controlling for project participation and household sociodemographic factors. See Annex 6 and Annex 7, Tables A7.5–A7.7 for a detailed description of the methodology and summary of results of the multivariate analyses of DPCCE. Table 1: Daily Per Capita Consumption Expenditures by Household Use of Financial Services, Value Chain Activities and Sustainable Crop and Livestock Practices at Endline (constant 2010 USD and 2005 PPP) Constant 2010 USD Number of Householdsab Sig. Use of financial services No $2.42 1,101 ** Yes $2.90 317 Use of value chain activities No $2.43 1,019 ** Yes $2.75 399 Use of at least two sustainable crop practices No $2.36 510 * Yes $2.60 908 Use of at least two sustainable livestock practices No $2.45 1,213 *** Yes $2.91 205 Sig. = significance level of p-value for test of differences. * p<0.05, ** p<0.01, *** p<0.001 a Each household is associated with one farmer's record. In households with multiple farmers, responses were consolidated into one record per household. If more than one farmer provided a response, the response of the first farmer (usually the household head) was considered. b The sample size in the table refers to the number of households; however, the percentage of people living under the $1.25/day is a weighted average where the sampling weights consider the number of members per household. Conclusions In the absence of true counterfactuals, few definitive conclusions can be drawn about the degree to which asset- and income-generating SO1 activities helped bring about the positive economic outcomes reflected in the quantitative findings. Project participants generally said that the assets gained from the project were valuable but that they produced only occasional income and that the income did not comprise a truly substantial proportion of overall household income. Given the great importance of remittances from both internal and external out-migration from the project zone, and the fact that much of the agricultural production was promoted for home consumption, this is not surprising. The field findings, however, correspond with generally accepted thinking about the role of agricultural development as a key driver of broader economic development and poverty reduction, often the key driver in agriculture-based economies.25 25 Dethier, J.-J., Effenberger, A., Agriculture and development: A brief review of the literature. Econ. Syst. 2012, doi:10.1016/j.ecosys.2011.09.003 Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 18 Recommendations For contexts like the Western Highlands of Guatemala, in which other sources of income such as remittances are highly variable but also highly important, and in which increased incomes and poverty reduction are key project objectives, FFP should consider making random control trials of livelihoods and economic indicators a standard part of its quantitative evaluation. 4.1.2 EQ2: Based on the evidence, which project outcomes are likely to be sustained? Findings Based on the interview responses of project participants, including agriculture leaders and agriculture promoters, it is likely that a large number of participants will continue with household fruit and vegetable gardening and raising chickens for eggs and meat. The evidence is participants’ opinions that these activities were easy to implement, were relatively familiar, did not require a lot of time or effort, and produced nutritious food both for households and for sale at local markets. A possible constraint to sustaining vegetable garden production at current levels is the unavailability of the improved seed distributed by PAISANO at local input suppliers. Because the feeding of goat’s milk to children was cited by women interviewed as hugely beneficial, and the goat-raising enterprise was therefore very popular, it is probable that positive outcomes from this activity will be sustained. The potential constraints are that for many the enterprise is a new and somewhat unfamiliar one and that the relatively high value of the asset makes selling the household’s only animal a tempting option when there is a need for cash. About a quarter of SO1 participants interviewed either knew of or were involved in the PAISANO collaboration with Másfrijol. Those who had participated appreciated the new varieties disseminated, reported moderate to “very good” increases in yields, and said that the varieties were adapted, relatively pest tolerant, and had acceptable quality. All said that they saved seeds and replanted, but none knew of local sources of new certified seed. A small number of participants reported purchasing “new seeds” from neighbors, or receiving them as gifts, but they did not know whether these were varieties promoted by Másfrijol. On a wider scale, the PAISANO PBS indicated that use of improved bean seed remained low and had not increased significantly (used by 2.4 percent of farmers at baseline and 4.2 percent of farmers at endline). The sustainability of rabbit production is possible but may be even more constrained than for goats, given the novelty of the enterprise, uncertainty over how to maintain animals’ health, the small size of the market (due to buyers’ unfamiliarity with the product), and the relatively low quality of feed and housing observed by the evaluation team. There was some evidence that the participants in Huehue were more familiar with rabbit raising than in other areas. It should be noted that for most participants receiving rabbits, the primary objective was home consumption. Certain “elite producers” were to sell at market to increase their incomes and introduce the product to consumers. Although in many ways impressive, CEPROCAL and CIPCA both have uncertain futures. At the time of the PE, both were still finalizing their medium- to long-term business plans, which are to include the financial forecasts underpinning their sustainability. The evaluation team could not visit seed production sites or seedling nurseries, and although they are reportedly generating income, no evidence of profitability and no sustainability plans were available, so their future viability cannot be estimated. Many agriculture leaders reported (and their records indicated) incomes from their advisory services, representing up to 20–25 percent of their total household income, and they could cite convincing, if informal, evidence of an understanding of their clientele and their ability to grow the market for services. Agriculture leaders believed that livestock advisory services (vaccinations, diagnosis of health Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 19 problems, administering medication, castration, etc.) were both profitable and sustainable. There was no evidence of a market for crop advisory services. Conclusions Household fruit and vegetable gardening and animal asset transfer activities can contribute significantly to integrated food security projects in a context such as the Western Highlands when they are paired with nutrition education, are not difficult to master or too time consuming, and are easily integrated into existing production systems. Smaller animals (poultry and rabbits) appear to fit these criteria, especially when promoted to the poorest households, which are usually the target participants of FFP development projects. The potential sustainability of CIPCA and CEPROCAL would have been easier to judge had they had formal business plans and sustainability plans available for review. The sustainability plans had been long in the planning, but completion was repeatedly delayed. Recommendations FFP should require IPs planning new project-initiated entities such as CEPROCAL and CIPCA to develop business plans early in their implementation and have clear and viable plans for financial sustainability well before the project ends. Business plans should be modified annually or semi-annually as economic conditions change and lessons are learned. Sustainability plans should include financials to allow forecasting future viability. Partnering with the private sector early on will likely help in this sustainability planning. Projects that incorporate input giveaways, or highly subsidized input provision, should design the interventions in ways that ensure that participants have sufficient income streams by the end of the LOA to purchase inputs at market prices and ensure that private sector input suppliers can continue to provide inputs locally. 4.1.3 EQ3. In each technical sector, what are the strengths and challenges of the effectiveness of the interventions’ implementation? Findings PAISANO identified agricultural interventions of an appropriate scale for the farming systems in question. Activities were generally well-designed and effectively administered to meet expected outcomes. The majority of participants in SO1 activities grew fruits and vegetables in home gardens or raised livestock promoted by the projects (goats, poultry, and rabbits). None of the gardens observed had irrigation, although a few participants said that they had received drip irrigation materials and training from the project. This may be because many areas had water tables too deep to drill for easily, and because many areas have norms that limit the use of water for irrigation. Almost all participants said that most of the production from these activities was consumed by the household and contributed to improved access to nutritious foods and improved nutrition as they understood it. The majority of participants said that some production was sold “occasionally,” either when there was surplus production or when the household had a significant need for cash. Goat milk tended to be sold less than poultry or rabbits, and eggs tended to be sold most often. Offspring of goats were passed on according to the project’s pass-on protocol, but they were not sold at the market often, at least among those participants interviewed. Male kids tended to be consumed by households or sometimes sold; female kids were used as additional sources of milk. Vegetables were sold at local markets and only very rarely at larger markets. SO1 activities, including associated training on nutrition, were universally cited as helping families to better support their children, and these activities were seen as sources of improved nutrition and as sources of income. There was little evidence that income from Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 20 sales of project-promoted enterprises was directly reinvested in productive assets. Participants mentioned purchase of food, health services, and education-related items from their additional income. Only a few said that they purchased more animals or production inputs. Members of Women’s Empowerment savings and lending groups did however use significant amounts of loan proceeds for productive investments, and it is possible that increased incomes from project activities contributed to increased ability to save. PAISANO resources invested in CEPROCAL and in CIPCA produced institutions of appropriate scale, which were well-designed, well-constructed, and apparently well-managed. Both CERPROCAL and CIPCA have production and educational objectives. Inspection of the facilities and discussions with key staff indicated that in the longer term, CEPROCAL appears better suited for specialization in the production of improved goat breeding stock, and CIPCA may be better suited for concentration on educational activities, especially if it can monetize these, which it has begun to do on a small scale. Staff suggested that these are the directions their forthcoming sustainability plans will emphasize. They appear to recognize that post-PAISANO, they might be more viable if they can attempt to concentrate on marketable core competencies. The key weakness of both is that they were project initiated and developed in an NGO-led context. These origins may make it difficult to transition to funding by either the public sector or the private sector. Lack of water was the biggest constraint to crop and garden production, according to more than half of the agriculture/livelihoods participants interviewed. Gardens observed ranged from very small (≈10 m2) and somewhat unkempt to large (>100 m2) and well-tended. Fewer than five participants interviewed had received drip irrigation equipment and training, for reasons explained above. Few constraints were mentioned concerning livestock production, although the quality of animal shelters observed was highly variable and a bit ramshackle, except for those of agriculture leaders and agriculture promoters. Most recipients of animals (goats and rabbits) received only a few meters of chicken wire or a roof of corrugated metal to facilitate construction of a shelter. Poultry recipients were expected to supply all housing construction materials themselves. All interviewed participants fed animals locally grown maize or collected fodder. None of the participants fed concentrated feed to the distributed animals, as this practice was not promoted by PAISANO. Not giving concentrated feed to the distributed animals potentially limited the effectiveness of the animal distribution intervention as it may have slowed and limited animal growth; thus limiting milk and meat production both for home consumption and for sale. In a few cases, unexplained deaths of rabbits were cited, but otherwise animal health was not considered a significant issue. Some bought and administered medications and vaccines themselves, some paid veterinarians, and many paid project agriculture leaders. Livestock management was rudimentary. A curious MTE finding was that farmers greatly appreciated goat urine as a fertilizer and reported unexpectedly large yield increases from its use. About half of the goats observed in homesteads by the evaluation team had urine collectors (made from feed or seed sacks arranged in a funnel shape under the goat enclosure). Pens at both CIPCA and CEPROCAL also had larger-scale urine collection mechanisms (sloped concrete slabs). Individual participants reported being happy with goat urine as a fertilizer, but none reported yield increases much greater than the nutrient content of urine would lead one to expect. Save the Children, CEPROCAL, CIPCA, and MAGA staff interviewed all confirmed farmers’ satisfaction with goat urine, particularly when used as a foliar fertilizer on legumes and vegetables. None knew of any formal research on the subject, but CIPCA and CEPROCAL staff interviewed said that they were hoping to find funding for such research. Agriculture promoters and agriculture leaders trained by the project all expressed appreciation for the training they received. Participants generally were satisfied with the advisory services provided by the agriculture leaders especially, citing the lack of presence of MAGA extension at the community level. PAISANO’s extension work was also recognized by government staff as a useful, even necessary supplement to government’s own efforts, which are constrained by inadequate funding. Agriculture Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 21 leaders interviewed could describe their technical activities and report their monthly incomes, but none reported keeping records on expenses or calculating enterprise profits and losses. Conclusions Distribution of inputs, including seeds and animals, combined with nutrition education, contributed to improved availability of and access to more nutritious food by participants’ families, especially children, according to more than 80 percent of agriculture/livelihoods participants interviewed. It may also have contributed to increased incomes, as correlations of income and poverty with agriculture and value chain practices (see Section 4.1.1) indicate. Positive outcomes, especially including potentially increased incomes from animal asset transfer activities, could likely have been improved by a more integrated approach to input and asset distribution, i.e., by promoting savings and providing cash transfers for income support and provision of higher quality feed and housing to every animal asset recipient. Agriculture leaders’ training allowed them to support SO1 participants effectively, but more in-depth business training might have permitted them to meet the end of project better prepared to maintain and grow their own extension advisory enterprises. Recommendations For projects using agricultural leaders as trainers of trainers, as project extension workers, and as future fee-for-service providers, FFP should require more intensive, in-depth business skills development than appeared to be provided in PAISANO. For projects with animal asset transfer activities, FFP should consider adopting the best practices recommended by the CGAP-Ford Foundation graduation approach, which has shown—from study of numerous asset transfer initiatives in many contexts—to effectively allow in-kind asset recipients, even the “ultra-poor,” to graduate out of poverty in time frames similar to those of DFAPs. It should also consider the extent to which direct cash transfers can attain economic and nutrition objectives as well as or better than in-kind asset transfers. An integrated approach might consider combining CGAP’s supplemental distribution of animal assets with (1) materials for construction of proper shelters; (2) frequent provision of training and close technical support; (3) promotion of savings; and (4) provision of cash stipends during an initial time period to purchase medicine, vaccines, vet services, and high￾quality feed, and to serve as consumption support while recipients are mastering improved asset management practices.26 4.1.4 EQ4. What are the key lessons learned and best practices that should inform future projects in the country? Findings Training agricultural leaders in marketable skills and having them practice those skills during the life of the project allowed them to build credibility with project participants, who represent the core of a future, post-project clientele. Structuring the projects extension efforts with teams of agriculture promotors supervised and trained by agriculture leaders capitalized on the training-of-trainers approach to effectively extend outreach efforts. Of the agriculture leaders and promoters encountered by the evaluation team, the majority (more than 90 percent) were men; only five female agriculture promoters and one female agriculture leader were interviewed. It should be noted that leaders and promoters were nominated by communities themselves, the criteria being that the nominated individuals be locally respected, responsible and knowledgeable. There was no evidence that communities took gender into account or were advised to do so. The project filled the roles of the agriculture leaders and promoters mainly with males because mostly males were recommended by community members. Most gardening 26 Hashemi, Syed M., and Aude de Montesquiou. 2011. “Reaching the Poorest: Lessons from the Graduation Model.” Focus Note 69. Washington, D.C.: CGAP, March. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 22 and livestock raising, though, is done by women, so recruitment of more women agriculture leaders and promoters might have been appropriate. Although in Guatemala there are few if any cultural reasons why men cannot serve as extension agents advising women on technical matters, small animal raising and household gardening generally are women’s activities, and in other contexts it has been shown that women extension agents often understand time and energy constraints of women clients better than male agents do. Pairing nutrition education with agriculture extension services targeting production of nutrient-rich food products was thought by participants to be one of the most valuable aspects of the project. The majority said that it led to noticeable improvements in the nutrition of their children. Linking of PAISANO SO1 activities to Feed the Future projects, particularly the value chain activities, was in line with best practices for programmatic cross-fertilization, but it proved to be limited by the inability of most PAISANO participants to integrate themselves into value chain approaches. Goat milk production (through animal asset transfers and the development of CEPROCAL) was highly popular, both with participants benefiting from the activity and with project staff. The high value of individual animals and the associated complications posed by offspring pass-on requirements, however, made it difficult to extend the activity to large numbers of participants—relative to poultry and rabbit transfers—and thus the activity was less able to achieve rapid increases in animal populations during the LOA time frame. Pass-ons of high-value animals are inherently less simple that for smaller, less costly animals. The difficulties seen with the pass-on of goats are similar to those that FFP and other donors have for some time recognized in many projects. Most difficulty came from resentment that some people received very high value goats (either originally or as pass-on) while others received lower-value animals and the decisions over who received what were not apparently not well understood. Project tracking and technical support of recipients of pass-on goats was inevitably poorer than for the goats originally transferred. For the purposes of milk production, female goats are more desired, and birth of male goats complicated the pass-on model more than it did for smaller animals. Conclusions Despite moderate implementation setbacks and complications arising from mid-stream refocusing of activities, which are common in projects of this size, PAISANO generally incorporated a suite of agriculture and livelihood activities that were considered best practices at the time of project design, although they may no longer be at present.27 Implementation was generally effective but on a somewhat limited scale, especially for goat milk production and value chain activities, for which the vast majority of participants were not qualified. Small animal asset transfers appear more likely to contribute to improvement of nutrition and economic outcomes for large numbers of poor in a five-year time frame than transfers of larger animals. Recommendations FFP should consider whether the target populations of DFAPs are so different from those of Feed the Future projects that any advantages from add-ons of small-scale value chain initiatives are more than outweighed by the potential advantages of devoting more resources to activities within the capacities of the very poor, allowing larger numbers of participants to escape from levels of extreme poverty, might best be represented by direct cash transfers rather than by prematurely promoting higher levels of value chain market participation. 27 Technical References for FFP Development Food Assistance Projects, USAID, Bureau of Democracy, Conflict, and Humanitarian Assistance, Office of Food for Peace, Washington, April 2015. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 23 4.2 SO2: REDUCTION OF MALNUTRITION 4.2.1 EQ1: To what extent has PAISANO met its defined goals, objectives and outcomes? Findings: Women’s Health and Nutrition Prevalence of underweight in women of reproductive age. Although underweight among women of reproductive age is a required FFP indicator, studies indicate that among Guatemalan women of reproductive age living in the Western Highlands, insufficient caloric intake is less of a public health concern than the high prevalence of overweight and poor consumption of key nutrients (such as iron, zinc, B12, and folic acid).28 Because few women living in PAISAO intervention communities had low body mass index (BMI)29 at baseline, little change in underweight was expected. There was no change in the prevalence of underweight women in the PAISANO project area (see Annex 5, Table A5.9). However, the percentage of overweight and obese women of reproductive age increased, from 39.2 percent at baseline to 47.8 percent at endline (see Figure 5). The increase in the prevalence of overweight and obese women is notable and concerning, but it cannot be associated with project participation. See Annex 5, Table A5.10 for details on the comparison of the prevalence of overweight-to-obese women at endline by participation status. Figure 5: BMI Levels of Non-pregnant Women of Reproductive Age, PAISANO 2013 and 2018 58.7% 1.6% 0.5% 29.7% 9.5% 2013 18.5-24.9 (total normal) 17.0-18.4 (mildly underweight) <17 (moderately and severely underweight) 25.0-29.9 (overweight) ≥30.0 (obese) 50.9% 0.0% 34.2% 13.6% 2018 18.5-24.9 (total normal) 17.0-18.4 (mildl underweight) <17 (moderatel and severely underweight) 25.0-29.9 (overweight) ≥30.0 (obese) 1.3% y y Data Source: PAISANO Baseline and Endline PBS The increase in the prevalence of overweight and obesity is important as it increases women’s health risks due to cardiovascular disease, diabetes, and health conditions during pregnancy, among other negative consequences. In addition, maternal obesity affects both production of milk and effective positioning at the breast, causing some women to stop breastfeeding early or lessen the intensity of breastfeeding.30 During qualitative consultations with PAISANO participants, mothers reported learning from PAISANO about the importance of consuming more of the “growth foods” (plant and animal protein) and “protection foods” (fruit and vegetables), and they commented that before PAISANO they were focused only on filling the stomach, rather than on foods that the body needs. Mothers mentioned that during PAISANO they started preparing healthy snacks to carry with them to avoid buying junk foods, which could save money and would be more nutritious. They described a memorable PASIANO 28 FANTA. 2014. Development of Evidence-Based Dietary Recommendations for Children, Pregnant Women, and Lactating Women Living in the Western Highlands in Guatemala. Washington, DC: FHI 360/ FANTA. 29 BMI is classified as underweight (BMI<18.5), normal (BMI≥18.5 to <25.0), overweight (BMI≥25.0 to <30.0), and obese (BMI≥30.0). 30 Bever Babendure, Jennie; Reifsnider, Elizabeth, Mendis, Enora et al. “Reduced breastfeeding rates among obese mothers: a review of contributing factors, clinical considerations and future directions.” International Breastfeeding Journal (2015) 10:21 DOI 10.1186/s13006-015-0046-5. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 24 group activity of setting candies on fire and seeing how they burst into flame, which they perceived as showing these to be harmful for the body. Although women clearly expressed increased awareness about the value of eating diverse foods and the need to avoid junk foods, mothers appeared more focused on how these practices would help their children grow and less focused on avoiding weight gain in adults. Women’s dietary diversity. To address nutrient gaps in women’s diets and promote healthy diets, PAISANO promoted improved knowledge and selection of diverse foods, as well as home production and access. Women’s dietary diversity score (WDDS) increased, from 3.8 groups at baseline to 4.8 groups at endline.31 Figure 6 underscores an increase in women’s consumption of grains, roots and tubers, legumes and nuts, dairy products, eggs, flesh foods, vitamin A-rich dark green leafy vegetables, and other fruits and vegetables, which are all foods promoted by PAISANO.32 Annex 5, Table A5.11 provides details on the baseline-endline comparison of food groups consumed by women. WDDS increased for women in direct and indirect participant households (see Annex 5, Table A5.2). At endline, WDDS for women in direct participant households (4.9) was higher than that of women in indirect participant households (4.7). Figure 6: Baseline-Endline Comparison of Women’s Dietary Diversity as Food Groups Consumed by Women 15–49 Years of Age * *** *** *** ** ** * *** 0 10 20 30 40 50 60 70 80 90 100 Grains, roots and tubers Legumes and nuts Dairy products (milk, yogurt, cheese) Organ meat Eggs Flesh foods and misc. small animal protein Vitamin A dark green leafy vegetables Other Vitamin A rich vegetables and fruits Other fruits and vegetables Percentage 2013 Baseline 2018 Endline No symbol = not significant, * p<0.05, ** p<0.01, *** p<0.001 Data source: PAISANO Baseline and Endline PBS Discussions with project participants support these findings and portray PAISANO interventions as a logical facilitator for women’s improved dietary diversity. Many discussions with project participants mentioned the following: 31 The women’s dietary diversity indicator assesses the number of (pre-determined) food groups that were eaten by a woman the previous day or night. It indicates changes in the micronutrient adequacy of women's diets, an important dimension of diet quality. The nine food groups include: (1) grains, white roots and tubers, and plantains; (2) pulses (beans, peas and lentils); (3) nuts and seeds; (4) dairy; (5) meat, poultry, and fish; (6) eggs; (7) dark green leafy vegetables; (8) other vitamin A-rich fruits and vegetables; and (9) other fruits and vegetables. More information can be found at: Kennedy, Ellen; Ballard, Terri and Dop, MarieClaude. 2010, reprint 2013. Guidelines for Measuring Household and Individual Dietary Diversity. Rome: Nutrition and Consumer Protection Division, Food and Agriculture Organization of the United Nations.http://www.fao.org/docrep/014/i1983e/i1983e00.pdf 32 FANTA. 2013. Summary Report: Development of Evidence-Based Dietary Recommendations for Children, Pregnant Women, and Lactating Women Living in the Western Highlands in Guatemala. Washington, DC: FHI 360/FANTA. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 25  Appreciation for learning from PAISANO how to integrate new foods into diets and combine energy and strength foods (i.e., grains, roots, and tubers), growth foods (i.e., animal and vegetable protein), and protection foods (i.e., fruits and vegetables)  The value of using the cash transfer to purchase a greater variety of foods  Greater availability of eggs due to improved poultry management and poultry containment  Home gardens as a source of vegetables, at least when rain water is available Findings: Children’s Health and Nutrition Population-based anthropometric data for children under 5 years of age living in PAISANO intervention communities showed a 5 percent decrease in stunting prevalence, from 74.7 percent at baseline to 69.7 percent at endline (see Table 2), although PAISANO did not achieve its LOA goal of a 10 percentage point decrease. There was no change in prevalence of underweight or wasting for children among 5 years of age. Table 2: Population-level Child Nutritional Indicators for PAISANO Communities Indicator Baseline (2013) Endline (2018) Significance LOA Targets Comment Underweight children <5 years of age (WAZ <-2) 26.4% 25.8% ns 20% FFP target 15– 20% decrease Stunting children <5 years of age (HAZ <-2) 74.7% 69.7% * 65% FFP target 10– 15% decrease Wasting children <5 years of age (WHZ <-2) 1.0% 1.4% ns NA No target (due to low prevalence) ns=not significant, * p<0.05, HAZ=height for age Z score, WAZ=weight for age Z score, WHZ=weight for height Z score Data source: PAISANO Baseline and Endline PBS To provide context for PAISANO’s advances toward reducing malnutrition in children under 5 years of age, it is important to note that stunting begins early in children’s lives or before birth. Annex 5, Tables A5.12–A5.15 illustrate the baseline-endline comparison of stunting and underweight rates for select age groups. Almost half of children were already moderately or severely stunted before 6 months of age (43.9 percent at baseline and 46.2 percent at endline, a statistically non-significant change, see Annex 5, Table A5.12), suggesting that maternal health and nutrition status, as well as exclusive breastfeeding, are among the critical factors. Maternal short stature, which affects almost half of women of reproductive age (51.2 percent at baseline, 47.7 percent at endline, non-significant change, see Annex 5, Table A5.9) is a risk factor for stunting. Bivariate analyses of the prevalence of stunting illustrate other contributory risk factors for stunting that include low educational levels of primary care givers and household heads, poor water and sanitation infrastructure, and household size and composition (see Annex 7, Tables A7.1 and A7.2). In keeping with USAID’s Nutrition Strategy framework, PAISANO’s MCHN approach included promotion of improved access to health and nutrition services and promotion of optimal health, nutrition, and WASH behaviors in the home and community. As part of a preventing malnutrition in children under 2 approach, these interventions were paired with food assistance (provided until the MTE), followed by cash assistance (provided after the MTE) to address gaps in dietary intake of women and children during the critical 1,000 day period of pregnancy and early childhood. After the MTE, support for WASH infrastructure was enhanced through efforts to promote community-level water availability and water treatment. Bivariate analysis of PBS data found that among PAISANO’s Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 26 interventions,33 only receiving a cash transfer was associated with lower prevalence of stunting at endline (67.3 percent stunting rate for those receiving a cash transfer, compared to 75.7 percent stunting rate for those who did not receive cash transfer, see Annex 5, Table A5.16). The association between cash transfers and stunting was also conducted for a pooled sample that combined children from the baseline with children from the endline and compared children in households that received cash transfers to children in households that did not participate directly or participated directly but did not receive cash transfers. These results indicate that cash transfers are associated with a lower prevalence of stunting, but this association is marginally statistically significant in the pooled sample (see Annex 7, Table A7.1). Qualitative consultations support the importance of the cash transfer, and most participants noted how the cash transfer allowed them to purchase a greater diversity of foods, especially “growth” and “protection” foods. Reinforcing the importance of improving access to diverse food, multivariate analysis of stunting found that higher consumption expenditures were a key factor in reducing the odds of stunting. Further details of multivariate analysis of stunting performed are included in Annex 6 and Annex 7, Table A7.3. Underweight in children under 5 years of age remained static from baseline to endline, but the overall indicator masks differences by age group (see Annex 5, Tables A5.12–A5.15). For example, the prevalence of underweight decreased for children 24–35 months of age, from 37.1 percent to 27.3 percent (see Annex 5, Table A5.12). However, the prevalence of underweight almost tripled among children under 6 months of age, from almost 6.6 percent at baseline to 18.3 percent at endline (see Annex 5, Table A5.12). The endline survey was conducted later in the rainy season than the baseline survey, and diarrhea prevalence typically increases during the rainy season, so data were analyzed to explore whether there was an association between the prevalence of underweight and the prevalence of diarrhea, both for children under 5 years of age and for children under 6 months of age. From baseline to endline, there was no change in diarrhea prevalence for children under 5 years of age. For children under 6 months of age at endline, the spike in underweight cannot be explained by prevalence of diarrhea, because the prevalence of diarrhea for children under 6 months of age actually decreased from 22 percent to 12 percent over the life of the project. An attempt was made to explore underweight among children under 6 months of age with exclusive breastfeeding, but sample sizes were not sufficient to permit analysis. It was noted that the endline sample of mothers of children under 6 months of age were unlikely to have participated in PAISANO before the birth of the surveyed child, because project recruiting protocols restricted enrollment to mothers who would have 12 months of participation prior to the end of the project. Some mothers of children under 6 months of age may have participated in PAISANO with a previous pregnancy. Overall, women with children under 6 months of age were unlikely to have received cash assistance when pregnant with the surveyed child or to have participated in other project benefits. Child Feeding Indicators Exclusive breastfeeding. Despite PAISANO’s promotion of exclusive breastfeeding (EBF) in children under 6 months of age, EBF did not change. At endline, 66.7 percent of children under 6 months of age were exclusively breastfed, below the LOA target of 74 percent (Figure 7). Lack of improvement in the prevalence of underweight in children under 6 months of age may be linked to the lack of progress with EBF, but it was not possible to determine a correlation due to small sample sizes (as previously noted). Annex 5, Table A5.17 illustrates the breastfeeding status for children 0–23 months by age in months at endline. Although nearly all children under 6 months of age (more than 96 percent) were reportedly breastfed, some children were also fed plain water, milk, and non-milk liquids, as well as complementary foods. EBF remained high at 80 percent for children under 2 months of age but decreased for children 33 Participation status in PAISANO interventions was by self-report and included: (1) food rations, (2) nutrition training and meetings, (3) agriculture training and meetings, (4) cash transfer, and (5) other activities. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 27 between 2 and 6 months of age.34 The effects of women’s obesity on milk production and effective positioning at the breast are recognized internationally, but the implications are not clear for the PAISANO communities.35 Discussions with project participants, volunteers, and staff explored PAISANO’s support for EBF and why gains were not made despite the project’s interventions. In keeping with PAISANO’s EBF barrier analysis, project participants and volunteers cited the common practice of giving teas to help with colic or fever and the influence of grandparents and mothers-in-law in maintaining these practices. Project staff and volunteers cited PAISANO’s efforts to engage mothers and other family members in changing these norms but noted that such practices are deeply engrained, suggesting that additional programmatic strategies are required.36 Figure 7: Population-based Infant and Young Child Feeding Indicators 66.1% 22.1% 66.4% 47.7% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Prevalence of exclusive breast-feeding of children under six months of age Prevalence of children 6-23 months of age receiving a minimum acceptable diet (MAD) 2013 Baseline 2018 Endline *** ns ns=not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 Data source: PAISANO Baseline and Endline PBS Children’s complementary feeding indicators. Impressive gains were made in complementary feeding indicators, across all age groups and most parameters of child feeding. A minimum acceptable diet (MAD)37 for children 6–23 months of age is an aggregate indicator requiring (1) breastfeeding (or at least two feedings of animal milk for non-breastfed children), (2) minimum feeding frequency,38 and (3) minimum diet diversity (four or more food groups). Figure 7 illustrates that the prevalence of MAD for children 6–23 months of age more than doubled, from 22.1 percent at baseline to 47.7 percent at endline, far surpassing the project’s LOA goal of 27 percent.39 34 For children 2–3 months of age, EBF decreased, from 73 percent at baseline to 66 percent at endline. For children 4–6 months of age, EBF decreased, from 61 percent at baseline to 55 percent at endline. 35 Bever Babendure, Jennie; Reifsnider, Elizabeth, Mendis, Enora et al. “Reduced breastfeeding rates among obese mothers: a review of contributing factors, clinical considerations and future directions.” International Breastfeeding Journal (2015) 10:21 DOI 10.1186/s13006-015-0046-5. 36 PCI collaborated with La Leche League to offer enhanced training to ML to strengthen their skills for supporting breastfeeding women, and certified graduates of the training. 37 A full description of the components of the minimum dietary diversity indicator is available at: https://usaidlearninglab.org/sites/default/files/resource/files/Nutrition%20Indicator%20Reference%20Sheets.s2.pdf 38 Two or more times per day for children ages 6–8 months, three or more times per day for children ages 9–23 months, and four or more times per day for non-breastfed children ages 6–23 months 39 MAD increases were even more dramatic for girls. At baseline, MAD was the same (22 percent) for girls and boys; at endline, MAD for girls was 51 percent, and MAD for boys was 44 percent. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 28 Annex 5, Table A5.18a provides details on the components of MAD indicator for children 6–23 months of age by age and breastfeeding status. However, to ensure adequate sample size, meal frequency, dietary diversity, and food consumption patterns were compared for all children 6–23 months of age, rather than by age group and breastfeeding status (see Annex 5, Table A5.18b). Figure 8 indicates that the slight increase in frequency of feeding improved is marginally statistically significant, although during qualitative consultations participants frequently mentioned learning that young children require an additional morning snack between meals, and in two FGDs mothers mentioned that they had begun carrying healthy snacks to feed their children, instead of spending money on junk food. However, participants focused more on learning about new foods and recipes and less on age-appropriate frequency of feeding. Women’s time constraints may have been a limitation for frequency of feeding, and staff and volunteers frequently mentioned women’s workloads and exhausting schedules, but participating mothers said that project recipes were not difficult or too time-consuming for them. In agreement with participants’ comments about integrating new foods in their children’s diets, a large increase was documented for dietary diversity, along with increases40 for specific foods and food groups that PAISANO promoted (Figure 8). These foods included dairy products such as goat milk and cheese, poultry and rabbit, eggs, and fruits and vegetables, which were promoted through a multi-channel set of activities, particularly individual and community-level social and behavior change (SBC), technical assistance for home production of animals and crops, along with cash transfers to facilitate purchase of specific foods. In all group discussions, project participants mentioned examples of using additional foods in their children’s diets due to participation in specific project activities. Bivariate analyses show that among direct participant households, cash transfer were effective in supporting improvement in MAD; 58.4 percent of children in households that received cash transfers achieved a MAD, compared to 42.2 percent of children in households that participated in the project but did not receive cash transfers (see Annex 5, Table A5.19 for details). Figure 8: Population-based Meal Frequency and Dietary Diversity Indicators for Children 6–23 Months of Age 60.7 28.9 90.4 36.9 15.6 15.5 34.2 48.0 30.5 67.1 62.6 94.9 52.0 35.8 28.2 58.3 69.8 59.6 0 20 40 60 80 100 Percent with minimum meal frequency Percent with minimum dietary diversity Grains, roots, and tubers Legumes and nuts Dairy products (milk, yogurt, cheese) Flesh foods (meat, fish, poultry, and liver/organ meats) Eggs Vitamin A-rich fruits and vegetables Other fruits and vegetables Percentage 2013 Baseline 2018 Endline † *** *** *** *** *** ** *** ** † p<0.1, ** p<0.01, *** p<0.001 Data source: PAISANO Baseline and Endline PBS 40 Statistical significance of change over time could not be determined for individual food groups for breastfed children 6–8 months of age, breastfed children 9–24 months of age, or non-breastfed children 6–23 months of age due to small sample size. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 29 Findings: PAISANO Custom Indicators on Recognition of Danger Signs, Health Care-seeking, and Women’s Decision-making for Mothers with Children under 5 Years of Age Recognition of danger signs. Indicators for mother’s knowledge of danger signs during pregnancy, the neonatal period, and childhood showed an impressive and statistically significant increase from the baseline, surpassing LOA targets (see Figure 9 and Table 3). Project participants mentioned learning about danger signs during PAISANO’s activities, such as Growth Monitoring and Promotion (GMP) sessions and home visits. Figure 9: Mother’s Recognition of Danger Signs in Pregnancy, Neonates, and Children under 5 Years of Age 37.4% 26.6% 31.7% 71.2% 63.0% 66.5% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Percentage of women with a child under 5 years who recognize two or more pregnancy danger signs Percentage of women with a child under 5 years who recognize two or more neonatal danger signs Percentage of women with a child under 5 years who recognize two or more childhood illness signs 2013 Baseline 2018 Endline *** *** *** *** p<0.001 Data source: PAISANO Baseline and Endline PBS Care-seeking for danger signs. Despite mothers’ increased awareness of danger signs, the percentage of mothers who sought health care in case of danger signs did not change from baseline for pregnancy, neonatal, or child illness danger signs (see Figure 10). Discussions with participants sought to explore this issue to better understand the reasons why health care was not sought, despite recognizing danger signs. Focus groups with parents asked, “What do parents do when their children present danger signs? What type of medical help is available, at what distance and with what schedule? What are the reasons that some parents do not seek medical help (although they require assistance)?” Respondents could easily mention the closest health posts where they could seek medical care and the alternative location if the closest option was closed, such as hospitals for 24-hour care. A few respondents mentioned that children could also be treated at home with home remedies (for example, lime juice for fever) or visit a pharmacy directly, if a remedy was needed. This seems to suggest that mothers consider alternatives to seeking immediate medical assistance from a health facility. Most mothers consulted during group discussions emphasized that PAISANO taught them to seek medical help in case of danger signs and that parents do seek medical help when needed, including contacting the community health committee for emergency transportation, if required. In group discussions, mothers mentioned that women who do not visit health services must want to avoid having their children vaccinated. One project staff member noted that accessing health care from MOH services was generally not appealing to project participants due to the low quality of care and the likelihood of finding the facility closed or without medicines to prescribe. Although project monitoring documented that a high percentage of families who received referrals for danger signs did seek medical care, it is not known whether families that did not receive referrals from PAISANO volunteers also sought care. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 30 To understand the range of factors that influence care-seeking, it is important to note changes in the provision of health services during the life of the project, particularly the end of community health care provided by NGOs contracted under the Coverage Extension Program (Programa para Extension de Cobertura [PEC]). Along with human resource and budget shortages, the health model changes resulted in medical care no longer being available near many PAISANO implementation communities. Group discussions about changes in access to health care did not result in conclusive findings, perhaps due to the considerable variation in communities’ access to health services. Among the notable factors, the increased challenges of access, funding, and human resources for health care after the closing of PEC community health services have been well-documented.41 Figure 10: Mother’s Reported Care-seeking for Danger Signs in Pregnancy, Neonates, and Childhood Illness 64.0% 76.4% 85.0% 69.5% 66.1% 80.1% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Percentage of women with a child under 5 years seeking health care when pregnancy danger signs are present Percentage of women with a child under 5 years seeking health care when neonatal danger signs are present Percentage of women with a child under 5 years seeking health care when childhood illness signs are present 2013 Baseline 2018 Endline ns ns ns ns=not significant Data source: PAISANO Baseline and Endline PBS Women’s participation in decision-making about health care. The percentage of women with a child under 5 years of age making health care decisions for their child was already 97 percent at baseline and therefore can be considered an established social norm (see Figure 11 and Table 3). In contrast, the percentage of women who reported making health decisions for their own health care decreased, from 61.5 percent at baseline to 50.4 percent at endline. The PBS questionnaire for the indicator defines women’s participation in decisions about their own health if they indicate any of the following: (1) they normally decide for themselves, (2) they and husband/partner normally decide together, or (3) sometimes they decide and sometimes their husband/partner decides. Although the percentage of women deciding alone on their own health care nearly doubled, from 11.0 percent to 20.9 percent, the percentage of women deciding jointly with their husbands or partners decreased, from 50.8 percent to 35.7 percent (see Annex 5, Table A5.20). PAISANO’s gender strategy focused on the positive results of spouses working together to jointly meet their family’s needs, including men assisting with household responsibilities. Discussions with project participants highlighted PAISANO’s promotion of collaboration between spouses/partners, although frequently participants mentioned that women, more often than their husbands, were the recipients of such messages, due to men’s schedules not allowing them to participate in PAISANO activities. Given the focus on joint decision-making by spouses, it is not clear why the indicator may have decreased. Project staff indicated that they were surprised by the baseline finding that 61.9 percent of women 41 Avila, Carlos, Rhea Bright, Jose Gutierrez, Kenneth Hoadley, Coite Manuel, Natalia Romero, and Michael P. Rodriguez. Guatemala Health System Assessment, August 2015. Bethesda, MD: Health Finance & Governance Project, Abt Associates Inc, p. 65-68. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 31 participated in decisions about their health care, assuming that it would have been lower. The IPTT did not include a specific indicator on promoting women’s decision-making for her own health, so this indicator was not monitored over time. The evaluation team theorized that if the distances to health services increased after the close of the PEC, it could have required more women to need their husband’s approval (and perhaps transportation fees) before departing for medical care at a greater distance from the community).42 Figure 11: Women’s Participation in Health Care Decisions for Themselves and for Their Child 61.5% 97.0% 50.4% 95.8% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Percentage of women with a child under 5 years making decisions about health care for themselves Percentage of women with a child under 5 years making decisions about health care for their child 2013 Baseline 2018 Endline ** ns ns=not significant, **p<0.01 Data source: PAISANO Baseline and Endline PBS; Table 3: Baseline to Endline Indicators for Recognition of Danger Signs, Care-seeking, and Women’s Decision-making about Health Care Population-level Indicator for PAISANO Intervention Communities Baseline (%) Endline (%) Sig LOA Target LOA Target Met or Unmet Recognition of danger signs Women with a child <5 years who recognize two or more pregnancy danger signs 37.4 71.2 *** 65 Met LOA target Women with a child <5 years who recognize two or more neonatal danger signs 26.6 63.0 *** 40 Met LOA target Women with a child <5 years who recognize two or more child danger signs 31.7 66.5 *** 60 Met LOA target Care-seeking indicators Women with a child <5 years seeking health care when pregnancy danger signs are present 64.0 69.5 ns 85 Did not meet LOA target Women with a child <5 years seeking health care when neonatal danger signs are present 76.4 66.1 ns 85 Did not meet LOA target Women with a child <5 years seeking health care when child illness signs are present 85.0 80.1 ns 88 Did not meet LOA target Care-seeking indicators Women with a child <5 years making decisions about healthcare for their child 97.0 95.8 ns N/A Virtually 100%, no target Women with a child <5 years making decisions about health care for herself 61.5 50.4 ns N/A — N/A=not applicable, ns=not significant, *** p<0.001 42 When health services were available in the community, it may have been easier for women to seek reproductive health services such as family planning without seeking their partner’s approval. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 32 Activities that had the greatest impact on affecting change in nutritional status. Plausible pathways for PAISANO’s impressive statistically significant improvements in household dietary diversity, women’s dietary diversity, and improved complementary feeding indicators include the multi-channel and multi-sectoral range of interventions presented in Table 4, such as SBC focused on education and practice of new behaviors and recipes, promotion of the cash transfer to purchase high-nutrient foods, and PAISANO’s support for household availability of key foods through gardens and animal production. During group discussions, some participants said that they were unable to participate in some of PAISANO’s activities due to limits on space, time, or interest, or unavailability of assets, but that participation in other PAISANO interventions allowed them to improve consumption of diverse foods, particularly their child’s feeding practices. Table 4: Overview of PAISANO’s Approaches to Promote High-nutrient Foods Promoted Food Related SBC Activities Cash Transfers to Promote the Food Related Productive Activities Legumes and nuts Dairy products (milk, yogurt, cheese) Flesh foods (meat, fish, poultry, liver/organ meats) Eggs Vitamin A-rich fruits and vegetables Other fruits and vegetables Promotion of feeding “growth foods” (animal and plant protein) and “protection foods” (fruits and vegetable), recipes in SBC materials, featuring specific locally available foods Recipe demonstration with hands-on preparation and child and parent testing Group promotion during GMP, nutrition schools, and community-level outreach Promotion during home visits based on assessing household food security and child feeding, agreements with families to try new behaviors and subsequent follow-up Shopping lists for spending of cash transfers included key foods, for family to select according to their needs Monthly planning process and group purchase of the following foods, if not sufficient through home production:  Goat’s milk or cheese  Animal flesh foods  Incaparina  Eggs  Fruits  Vegetables Support from USAID’s Mas Frijol Project for some participant families Goats provided to some families, sale and purchase of goat’s milk encouraged Rabbits provided to some families for consumption and sale Chicken housing materials and technical assistance provided to increase egg production and minimize egg loss Seeds and technical support to increase diversity and productivity of vegetable gardens (when water available) During interviews and focus groups, project participants cited the following specific examples of PAISANO’s activities that they found most useful to improve their children’s diets:  Recipe demonstrations using foods that they had not previously given their children, particularly goat’s milk, egg, rabbit meat, and combining foods in ways that children like and help them grow  Cash transfers to purchase a variety of foods that their children were not used to eating frequently or not at all, but they now like, such as goat’s milk (frequently mentioned), fruit, eggs, vegetables and Incaparina. “Growth” (i.e., protein) foods and “protection” (i.e., fruits and vegetables) were cited frequently as the most important foods to buy, because tortillas and potatoes are “strength and energy foods.” Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 33  Technical support and seeds that PAISANO provided for gardens to start or improve production of green leafy vegetables, habas, carrots, and others  Containing chickens behind wire provided by PAISANO to prevent losing eggs (also mentioned for helping avoid diarrhea). Some participants highlighted the value of learning how to raise and prepare rabbit after receiving rabbit stock from PAISANO. Many mothers noted learning from PAISANO about the nutritional value of goat’s milk and starting to give it to their children, but fewer participants mentioned raising goats received from PAISANO. Effectiveness of food or cash assistance. As mentioned above, analysis of PBS data found statistically significant associations between lower prevalence of child stunting in children under 5 years of age for households that received cash transfers (67.3 percent), compared to direct project participants that did not receive cash transfers (75.7 percent) (see Annex 5, Table A5.16). Positive associations were also reported for improved prevalence of MAD for households that received cash transfers (58.4 percent), compared to direct project participants who did not receive cash transfers (41.2 percent) (see Annex 5, Table A5.19). No such associations were found for families that received food assistance. Consultations with project participants provided additional evidence that PAISANO’s transition from food assistance to cash assistance post-MTE was more effective for meeting objectives related to improved children’s diets. Nearly all participants, project staff, and even MOH staff mentioned that cash assistance allowed families to expand the variety of foods, exposing children to new foods, especially “growth foods” (i.e., protein foods) and “protection foods” (i.e., fruits and vegetables). Parents mentioned that young children get bored with the base diet of corn or potatoes, and the cash transfer allowed children to become accustomed to eating foods such as goat’s milk, Incaparina, green leafy vegetables, and fruit. Use of cash assistance for productive assets was not mentioned by project participants. The likelihood that participants can continue to access diverse foods after the end of the cash transfer and other project activities is explored further under EQ2. Women’s time burden for participation in MCHN activities. We observed rigid gender norms around men’s and women’s roles, which sharply defined women’s roles in the household and their economic participation and decision-making. For example, project volunteers cited women’s excessive time burden on unpaid tasks such as water collection, firewood collection, and agricultural work, in addition to being responsible for most of the cooking, cleaning, and childcare. Project staff described their efforts to recognize women’s time constraints and streamline MCHN activities to accommodate their time, such as varying the schedule and length of time for group activities or home visits. Still, preparation of the new foods and recipes requires time, mostly for women. Several volunteers and staff noted assisting overburdened women at home during home visits. Nevertheless, staff members cited time constraints as the primary reason that some participants declined to participate.43 Contribution of targeting strategies to achieving project goals. Targeting pregnant women and families with children under 2 years of age was effective for focusing interventions on women’s and children’s dietary diversity, as shown by increases in consumption of the food groups specifically promoted by PAISANO (detailed in Table 4). During field visits, participants in Huehuetenango shared experiences of PAISANO activities that successfully engaged the wider community and provided examples of families that adopted new behaviors (such as rabbit raising or use of Ecofilters). External factors that inhibited or promoted progress. PAISANO’s approach to reducing the high rate of childhood malnutrition relied on increasing families’ use of preventive health services, ensuring the quality of the services offered at the community level, and supporting optimal child health and nutrition behaviors. As a result, PAISANO’s health interventions relied on building a functional and 43 However, field discussions with non-participants focused on not meeting the project criteria to have family members during the 1,000 days. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 34 stable partnership with the MOH and harmonizing actions with the existing community health model. However, PAISANO faced a volatile context with evolving national health care models, along with budget and staff shortages, which limited the project’s ability to coordinate activities with the MOH and build the capacity of community health staff, as defined in the proposal and early Detailed Implementation Plans (DIPs) for FY13 and FY14. When PAISANO began, the PEC delivered primary health care through contracted NGOs and included a community-level cadre of paid volunteers, including “community facilitators,” who were intended to support and interact with PAISANO’s volunteers. The abrupt end of the PEC and short duration of the Inclusive Health Model particularly affected intermediate result 2.1, “Use of quality MCHN preventive services increased,” which required training community heath facilitators. With PEC community-based health positions eliminated, many families could not receive health services in their community at “convergence centers,” which increased the distances to receive health services. Given that prompt treatment of child illness (including diarrhea) and access to prenatal care, including provision of multivitamins and prompt treatment of infections, are important factors for preventing malnutrition in Guatemala, these challenges likely inhibited PAISANO’s progress in reducing malnutrition. To mitigate the loss of community health workers under the PEC, PAISANO re-focused intermediate result 2.1 on preventive care provided by PAISANO volunteers. The project facilitated joint service delivery provided by MOH staff and PAISANO during quarterly GMP sessions in each community, with periodic health fairs and recipe contests and through joint home visits as feasible. PAISANO staff and volunteers also monitored MOH services provided after PAISANO’s referrals and conducted follow-up visits to children and pregnant women who had danger signs. PAISANO’s modifications to its original strategy of capacity-building, as documented in ARRs and annual DIPS (FY12, FY13, and FY14), was logical given the volatile environment. However, the change limited PAISANO’s influence on the quality of facility-based preventive services provided by MOH staff, because few MOH staff were trained by PAISANO.44 Conclusions PAISANO’s strong contribution to improved food security is documented by statistically significant improvements in dietary diversity in households and women’s diets and children’s MAD, particularly for the high-nutrient foods that PAISANO promoted to address dietary gaps. Table 5 illustrates that improvement in HDDS, WDDS and MAD were experienced by both direct and indirect participant households (see Annex 5, table A5.2 also). Improved complementary feeding indicators testify to a major, multi-channel effort, including hands-on activities to practice new behaviors and foods that participants consider valuable for their improved nutrition. Decreases in population-based stunting prevalence were statistically significant, although less than the project target, while child underweight remains an important concern, given the lack of progress. The extremely high prevalence of stunting requires addressing the nutritional and health contributors over a long-term, intergenerational period (including maternal nutrition), and PAISANO makes a positive contribution to this effort. 44 Limited training of community facilitators took place before the collapse of the PEC (although additional trainings were planned). The project reported that 368 MOH staff were trained during the project. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 35 Table 5: Comparison of Baseline and Endline Indicators by Project Participation Status For Select Indicators Baseline 2013 (BL) Endline 2018 (EL) BL-EL Comparison (Sig.) Endline Comparison by Participation Status (Sig.) All Households (I) All Households (II) Direct Participants (III) Indirect Participants (IV) All Households (V) Direct Participants (VI) Indirect participants (VII) Direct vs. Indirect Participation (VIII) Household dietary diversity score 6.1 7 7.0 7.0 *** *** *** ns Women's dietary diversity score 3.8 4.8 4.9 4.7 *** *** *** * Prevalence of underweight children under 5 years of age (%) 26.4 25.8 26.6 24.4 ns ns ns ns Prevalence of stunting children under 5 years of age (%) 74.7 69.7 72.0 65.4 * ns ** * Prevalence of children 6–23 months of age with a minimum acceptable diet (%) 22.1 47.7 49.7 44.2 *** *** *** ns Sig. = significance level of p-value for test of differences. ns=not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 NOTE: The household survey asked respondents whether they participated regularly in any of the project activities. Respondents who answered "Yes" are considered direct participants. Respondents who answered "No" or "Don't know" are considered indirect participants. A total of 12 households in the SC-PAISANO project areas did not report information on project participation. Households with missing information on project participation are not included in the BL-EL comparison by participation status (Columns VI and VII) nor in the endline comparison by participation status (Column VIII). Recommendations To address the intractable problem of child undernutrition in the Western Highlands, it is recommended that future projects explore the issues related to lack of progress at improving EBF, frequent diarrheal disease, deficient water and sanitation infrastructure, and the availability of culturally acceptable, quality health care which facilitates healthcare-seeking. For child feeding in particular, the significant progress with complementary feeding contrasts with the inability to influence EBF rates, suggesting a need to more deeply understand the constraints and opportunities faced in these two different domains which warrant different approaches to support changes in behavior. To facilitate analysis of the benefits of project participation and better understand the most effective interventions, endline surveys would benefit from using lists of project participants, rather than self-reports of project participation, to better distinguish direct participants from indirect participants, and to provide accurate data on how specific interventions were associated with changes in outcome and impact indicators. In addition, having a control group in non-intervention communities would allow Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 36 greater precision in understanding the project’s effects on the intervention communities. Given the double burden of malnutrition in Guatemala (i.e., the concerning increase in overweight among adults and children, as well as the persistently high prevalence of stunting), it would be valuable to include relevant indicators related local trends for consumption of high-calorie, sugary foods (such as soda), high-fat, salty snack foods (such as chips), and processed or ultra-processed foods (such as cured meats). Additional gender indicators are also recommended to capture other dimensions of gender inequality that affect food security and to ensure that these aspects are targeted by food security projects. In addition to women’s time constraints discussed above, women’s participation in community leadership was scarce, further limiting their community-level decision-making about issues related to food security, and the project gender analysis documents women’s limited control over household resources, which jeopardizes their food security and that of their children.45 Finally, the decrease in indicators for emergency care-seeking could not be fully understood by community consultations as part of the evaluation. Therefore, future projects should further explore the facilitators and barriers for emergency care-seeking, assist communities with planning, and facilitate community monitoring and feedback about the quality of services. 4.2.2 MCHN: EQ2: Based on the evidence, which project outcomes are likely to be sustained? Continued role of mother leaders and nutrition promoters. During consultations with mother leaders and nutrition promoters, many expressed strong personal satisfaction with their roles as PAISANO volunteers and their aspiration to keep learning more about health. PAISANO’s FY18 quarterly reports documented the certification of mother leaders’ skills and experience to facilitate future volunteer or job opportunities, but data do not exist to determine how many volunteers have moved on to paid or volunteers work. Although many volunteers identified themselves as no longer active in the community, there were some exceptions. A group of volunteers in Huehuetenango gave examples of continuing to support health facilities, such as a volunteer who recently conducted home visits to provide key messages and support to a growth faltering child. Quite a few volunteers reported continuing to serve their communities on the COSAN, COCODE, health committee, or school committee, as a traditional birth attendant, or with other NGOs, even if they had not been active in their communities before volunteering with PAISANO. Overall, many volunteers indicated a willingness to provide help to MOH health staff, if asked. However, few MOH staff reported that a relationship had been established for activities with PAISANO volunteers after PAISANO’s exit. After reflecting on the value of volunteers’ assistance during interviews, several MOH staff pledged to contact former PAISANO volunteers to ask whether they could provide occasional help to health staff. Nearly all MOH staff interviewed expressed concern that because the MOH cannot offer an incentive to volunteers, they do not expect that volunteers who once received a subsidy would be willing to assist without one. Continued mechanisms for the referral system. Several mother leaders remarked that they will continue to refer cases of children with danger signs to health facilities, although no system exists for referral back to them for community follow-up. PAISANO staff interviews reinforced the hope that mother leaders will keep referring women or children with danger signs or suspected acute malnutrition, but recognized that referral for community follow-up will not occur. In one health district, health managers acknowledged PAISANO’s donation of a large stock of referral forms, which allows the district to refer patients from health posts to higher-level services. Several MOH staff said that it will be 45 Suggested themes for gender indicators include: (1) men’s time spent on unpaid household tasks (collecting water, firewood, cooking, cleaning, childcare); (2) women’s uptake of new agricultural technologies and practices; (3) women’s access to credit; (4) women’s decision-making about selling assets (such as animals); (5) women’s decision-making about use of household resources; (6) women’s participation in community leadership; and (7) women’s unmet contraceptive needs. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 37 more difficult for health staff to provide the same level of outreach without PAISANO’s support, but they will make home visits to children diagnosed with acute malnutrition. Sustainability of SBC promotion. With the conclusion PAISANO’s implementation, pregnant women and parents are unlikely to receive access to the range and quality of SBC activities that were carried out by volunteers and staff during PAISANO’s implementation, as the MOH model does not include the same priority on SBC (or the human resources and budget necessary to support it). Although the majority of volunteers had not defined their plans for continuing SBC promotion, several PAISANO volunteers from communities where PAISANO closed operations at least six months prior to the PE mentioned that they continue to coordinate with health posts. MOH staff in several districts assured interviewers that GMP sessions will continue to take place monthly, according to existing MOH protocols, with parallel health activities such as vaccinations and some health promotion. MOH managers reported that household visits to support behavior changes would be carried out for children with danger signs or suspected acute malnutrition, but that they face limitations on health workers’ time for health promotion Regarding continued use of PAISANO’s SBC materials, most MOH staff interviewed expressed appreciation for PAISANO’s closeout event held in each district to orient MOH personal on the use of PAISANO’s materials, prior to the transfer of SBC materials and anthropometric equipment. One district reported that donated PAISANO SBC materials had already been distributed to health facilities and were reportedly in use by first-line health staff and educators. Several other districts reported that they would distribute materials in their districts but had not yet devised a plan for distribution and use. In general, health staff lamented the end of PAISANO and the fact that volunteers would no longer make frequent home visits to provide health education and follow up on specific cases. End of food and cash assistance and impact on dietary diversity. Most project participants acknowledged that without the cash transfer, they will not be able to buy all the different foods they bought while receiving the transfer. But many said that, based on the nutrition education they received from PAISANO, they will prioritize some foods, such as Incaparina, because their children are now accustomed to drinking it and it is important for growth. Other participants (especially in Huehuetenango) said that Incaparina would be difficult to purchase due to its cost (Q10/$1.30 per bag), but they would make choices to substitute products that are less costly but also have benefits.46 Participants also said that they will buy small portions of foods only for their young child, if they cannot afford enough for the family. Participants considered their egg supply to be improved now that their chickens are contained in corrals, and in Huehuetenango, rabbits were mentioned as an easily available protein source. Many noted that improved home garden production will help their diets, if sufficient water is available. Participants from areas with water scarcity or excessive cold expressed the most concern about losing access to foods purchased with the cash transfer. A few government representatives shared their doubts that families can maintain dietary diversity without the cash transfer, but they admired PAISANO’s strategies to help families manage the cash for nutritional benefits while it was available. Motivation to continue receiving preventive health services. During the FGDs, project participants said that PAISANO’s key messages about seeking health services (such as GMP, supplementation during pregnancy, and care in the case of danger signs) are well-understood by mothers and that families will continue to seek health care. The endline survey did not indicate improved care-seeking for danger signs, but participants who were consulted expressed confidence that they will continue to go to GMP and preventive care. Sustained source of resources and capacity strengthening for continued volunteer services. Consultations with volunteers, project staff, and government partners did not mention a source of 46 One participant suggested that oatmeal or cream of wheat could be cheaper options, which are usually fortified with micronutrients in Guatemala, although they do not represent a “growth” or “protection” food. It was not clear whether other low-cost foods were identified for substitution from different food groups. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 38 continued capacity strengthening for PAISANO volunteers or resources for incentives, transportation, or teaching materials. PAISANO had the objective to develop volunteers as service providers who charge fees for their services, but the PE team understood that this objective was related to the agricultural sector, rather than the health sector, because this objective was not discussed for health in DIPs, the MTE, or other project documentation. Leveraging other USG and non-USG investments. PAISANO’s proposal and early DIPs describe the project’s plan to leverage and coordinate with other health projects and initiatives, including the Scaling Up Nutrition, the PEC, the Zero Hunger Initiative, and SESAN, among other entities. National coordination, however, proved problematic due to the volatile political and organizational context. As mentioned previously, the PEC was dismantled, and the Zero Hunger Initiative was put on hold during transition in national leadership. In terms of health coordination, PAISANO instead prioritized coordination in the departments of Quiche, Huehuetenango, and Quetzaltenango with MOH health management areas, health districts, and individual health facilities. Leadership transitions occurred frequently, so PAISANO’s strongest relationships were with individual health facilities for quarterly GMP sessions implemented jointly with PAISANO. Conclusions Most PAISANO MCHN volunteers are interested in continuing to serve their communities, but it is unlikely they will carry out the range of activities that they did during PAISANO’s implementation due to limited mechanisms for sustained training and support. Unlike agricultural volunteers, there is a not a clear market for continued service provision by health volunteers through a market-based solution. As a result, continued referrals and SBC targeted to pregnant women and young children are likely to be reduced, although participants report motivation to continue attending preventive and curative health services. PAISANO’s linkages are strongest at the health facility level, where some MOH health staff aspire to use project materials and assume a limited set of PAISANO’s activities, such as following up with the most serious cases. Families mentioned that it will be a challenge to maintain dietary diversity without the cash transfers, but many cited their motivation to prioritize nutritious foods and mentioned the benefits of improved animal and home garden production, as documented under SO2 sections. Recommendations It would be valuable for future projects to ensure that sustainability plans and progress toward benchmarks are vigilantly monitored. The project faced considerable sustainability constraints due to the changing MOH model and other challenges with staffing and funding, but field visits indicated that in many areas visited, sustainability steps had not been fully developed. In terms of health volunteers, it is recommended that MOH staff at each health facility (and higher management levels) meet regularly with health volunteers to fully explore and define the options for sustainable support and contact between volunteers and health facilities beyond the life of the project. In addition to transferring materials at the end of the project, it is recommended that projects support the MOH to define who will use the materials and optimally to adapt this process to the limits of staff time and MOH priorities, with time to field test and refine the approach. Opportunities exist for other USG projects and other entities to assume a role in facilitating the sustainability of PAISANO’s activities, and especially the corps of PAISANO volunteers, to maximize USG investments and enhance long-term food security outcomes. 4.2.3 MCHN: EQ3: In each technical sector, what are the strengths of and challenges of the effectiveness of the interventions’ implementation? Effective implementation methods and approaches for MCHN. Many project participants responded enthusiastically and with detailed lists when asked which of PAISANO’s approaches were most valuable for their family. Frequent responses included nearly all of PAISANO’s MCHN activities: recipe demonstrations and nutrition education on how to combine food groups (especially to include Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 39 “growth foods” and avoid junk food), support for home gardens and chicken-raising (including wire and feeders), hygiene education with follow-up home visits47 to check on the practices, GMP sessions to find out whether children were growing, and cash assistance to buy a variety of foods. In some areas, participants emphasized the value of rabbit-raising as a protein source at home and Ecofilters, which have resulted in less illness. Learning about the value of goat’s milk and giving their children goat’s milk were also frequently mentioned, although raising goats themselves was less commonly mentioned. In terms of overall approaches, participants and staff clearly appreciated PAISANO’s SBC focus on hands-on learning and skill-building, as well as the educational materials that were actively used (such as the recipes and flash cards). Participants mentioned that home visits were useful because they were obligated to try the behaviors because they would receive unscheduled visits from PAISANO staff or volunteers. MOH staff and other government partners said that PAISANO’s method for cash assistance with planning and spending support was highly effective for improving diets. Community norms or intra-family dynamics. PAISANO employed a gender strategy that integrated gender messages into all activities and encouraged men’s participation in all PAISANO activities, although there were challenges with men’s work schedules and entrenched social norms about male and female roles in a household. Staff described PAISANO’s gender curriculum, “Working Together,” as being focused on collaboration to improve the situation of their family, including a successful activity with men to list the men’s and women’s household duties and to solicit men’s commitment to take on additional roles. Participants and staff agreed that there was low coverage of men participating in gender activities, but those who did were often receptive. Men who participated described helping out more at home, and in separate focus groups, women noted that after participating, their male partners took on new household tasks (e.g., carrying water, cooking, caring for children or the woman when she is ill). During focus groups, several women said that their husbands agreed with spending money on healthy foods after their wives informed them which foods are important for children’s growth, implying that husbands accepted women’s information and opinion about the household budget. Staff and volunteers noted that PAISANO’s gender activities were slow to start and that having a well-defined gender strategy, ideally as the first step in implementation, should be a priority. Service fees as incentives for volunteers. PAISANO used “service fees” paid by project participants who received food or cash assistance to fund subsidies for project volunteers.48 Service fees were tied to project participants receiving food assistance or cash transfers (which were of higher value than the fees), and it should not be assumed that such fees represent community members’ willingness to pay fees to volunteers without receiving free inputs. Some volunteers considered the subsidy too small, especially mother leaders who faced transport costs to the communities where they worked. Staff reported that some nutrition promoters were dissatisfied when the subsidy changed from food to cash, because they felt that the value decreased. Also, the service fee mechanism relied on participating families to cover the volunteer costs, so when children “graduated” from PAISANO’s assistance, sometimes there were not enough new families to support the subsidy of the mother leader in that area. Without service fees to sustain their subsidies, volunteers’ positions ended, and the remaining volunteers or staff needed to assume extra territories. In theory, the service fees produced a source of funding that allowed PAISANO to set higher expectations for volunteer roles, but as a result, this level 47 Staff and volunteers mentioned that at first household members were reluctant to receive home visits, because they perceived these as monitoring the family. Therefore, PAISANO discussed adjustments to make the home visits more collaborative and friendly, and this resulted in a transition to families welcoming the home visits. 48 Mother leaders and agricultural leaders received Q500 monthly to cover three to four communities and were expected to work 98 hours per month. Nutrition promoters and agriculture promoters were responsible for only one community and were expected to work fewer hours per month. From May 2013 to February 2017, they received a bimonthly food ration of beans. In March 2017, the subsidy was switched to Q250. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 40 of volunteer work was too large to be sustained by a community member without an incentive. A less intensive volunteer role is more likely to be sustained, with or without an incentive.49 Community members’ willingness to pay for volunteers’ services, perhaps linked to sale of health-related goods or services, was not examined by the project but would be worth feasibility analysis and pilot testing. Effectiveness of targeting. PAISANO originally enrolled families with pregnant women or children 0– 30 months of age to receive MCHN interventions, including food assistance and other transfers. However, this targeting approach was not precisely aligned with the scientific rational that interventions during the 1,000-day period from pregnancy to 2 years of age confer the greatest long-term benefits for nutrition, health, and future earnings.50 Therefore, MTE recommendations included ending the duration of transfer benefits when the child reached 2 years of age, as well as a restriction on enrolling for a second pregnancy. The MTE also recommended the use of a poverty screening tool to determine eligibility for transfers of food, cash, seeds, animals, and other materials. Although families with pregnant women and children 0–23 months of age not meeting the new criteria were invited to participate in PAISANO without receiving transfers, few enrolled. The new targeting allowed PAISANO to more tightly focus assistance on the poorest families within the window of 1,000 days, but it limited PAISANO’s community reach and possibly its population-level impact on stunting. Changing targeting strategies at mid-term presented considerable programmatic implications for PAISANO, requiring the project to adjust project materials and strategies, inform community members and participants, and field test and apply the set of poverty criteria across the entire project area. Project staff mentioned frustration and discontentment expressed by some community members who were no longer eligible to participate due to the restriction on enrolling for a second pregnancy or due to a child aging out of assistance at 24 months of age, earlier than expected. Toward the end of PAISANO’s implementation period, a cut-off date was applied for accepting new pregnant women to ensure that participants would have a full year of participation, which also limited the total community population exposed to the project. As reported in the child nutrition section, this policy meant that prior to project completion, there were already children born whose mothers did not receive project interventions, including many children under 6 months of age who were included in the endline survey. End of eligibility for project assistance. According to ARRs and project staff, it was difficult to keep project participants active in PAISANO when they were no longer eligible for food or cash assistance, due to children aging out or not meeting other criteria. Project staff tried to persuade families to remain active until their children were at least 30 months of age by attending GMP, but in most project areas, staff noted that home visits and recipe demonstrations were limited to children under 24 months of age, so families had stopped participating when the cash transfer ended. An exception was in Huehuetenango, where participants described recipe demonstrations and community mobilization activities that were well-attended by community members of any age, because these activities were targeted to the wider community. In these cases, community members (beyond the 1,000-day MCHN population) were actively invited to participate in recipe demonstrations (such as preparation of rabbit), whereas PAISANO activities in other areas targeted only pregnant women and households with children under 2 years of age. Transition from food assistance to cash transfers. Although some project participants cited small logistical challenges transitioning from food assistance to cash transfers, participants expressed almost universal support for the change from food to cash, because it permitted more variety across their diet and made eating healthy foods more attractive to children. PAISANO used a checklist of locally available 49 Rogers, B.L., and Coates, J. 2015. Sustaining Development: A Synthesis of Results from a Four-Country Study of Sustainability and Exit Strategies among Development Food Assistance Projects. Washington, DC: FHI 360/Food and Nutrition Technical Assistance III Project (FANTA). 50 Victora, C.G. et al. 2008. Maternal and child undernutrition: consequences for adult health and human capital. The Lancet 371 (9609). pp. 340–357. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 41 nutritious foods to promote decision-making about spending the cash transfer on healthy foods, and participant households were expected to mark the selected foods on the checklist, which was verified by project staff prior to disbursing the cash transfer. To facilitate bulk price negotiation with the vendor and ensure accountability, groups of participants purchased the selected foods together, and staff conducted periodic follow-up visits to homes. Focus group participants were asked to what extent they considered PAISANO’s planning and purchase controls to be useful or overly controlled. Respondents unanimously—and often enthusiastically—expressed their agreement with the process applied, and they felt that it ensured that the funds would be spent only on nutritious foods. SESAN and MOH representatives concurred that the cash assistance had more value for the families because it was directed to improving the family diet. Quality of interventions and inputs from participants’ perspectives. Project participants expressed satisfaction with the quality of project interventions and particularly highlighted the value of project activities related to GMP, including the color-coded graphic, which they felt more clearly displays the nutritional status of the child, compared to the MOH growth chart, and home visits, which they felt ensured that their own (and other participants’ homes) maintained the practices taught by the project, such as covering food items, using the Ecofilter, keeping animals separate from living spaces, using the tippy tap to wash hands, and giving healthy snacks to children. They admired the quality of the recipe flipchart that was provided to each family and felt that it was useful to practice the recipes and then have the recipe guide for repeating the recipes. Many women could mention a child’s favorite recipe from the recipe flipchart. Participants were enthusiastic about the usefulness of receiving chicken wire, roofing materials for animal housing, and seeds for home gardens, although several noted challenges with goats that failed to reproduce or died, as well as one case of rabbits that died. Conclusions PAISANO’s SBC approach based on hands-on activities and close individual follow-up by volunteers and staff appears to have been effective at supporting behavior changes, particularly for dietary diversity and WASH, and food/cash transfers and productive activities appear to have addressed some of the families’ requirements for increased access to and availability of diverse foods. Cash assistance, rather than food assistance, was valued by participants as supporting greater dietary diversity and found to be associated with improvements in stunting and complementary feeding indicators. Targeting strategies implemented after the MTE concentrated PAISANO’s resources more tightly on the 1,000-day window of opportunity and on the poor but also presented programmatic challenges for implementation. Gender training had limited reach but was well-received by men and women exposed to activities. Recommendations PAISANO’s most successful activities and materials should be shared and replicated, as feasible, in the Western Highlands. Given that addressing gender inequality is crucial to addressing food insecurity and maximizing the benefits of food security projects, a well-defined gender strategy should be prioritized in project design and ideally as the first step in implementation. When changes to project targeting strategies are suggested after implementation has begun, it is important to assess whether the financial and time implications will be excessive, in order to determine whether the benefits of improved targeting are worth the investment of financial resources and effort. Efforts to include community members beyond the 1,000 days in community mobilization activities (in some areas) seem to have positive results for community buy-in and are recommended. Financing volunteer subsidies with service fees should be carefully considered to ensure stability of the funds over the course of the project, because losing volunteers mid-project is a lost opportunity for the volunteer and project outcomes. A larger concern is the temporary nature of subsidies that derive from project funding. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 42 4.2.4 MCHN: EQ 4: What are the key lessons learned and best practices that should inform future projects in the country? Synergy of approaches support dietary diversity. PAISANO’s successful experiences in increasing dietary diversity of households, women, and children 6–23 months of age present lessons that can inform future projects in the Western Highlands and beyond. A range of interventions supported the integration of different foods or food groups into diets (as shown in Table 4), and these approaches harmonize and reinforce each other so that families with increased egg production also attended recipe demonstrations during which they tasted recipes using eggs in ways that addressed taboos about feeding eggs to young children. When a range of strategies is available to promote dietary diversity, individual households can participate in interventions in accordance with the household’s potential resources (e.g., land, time, skills) and constraints (e.g., water, cold, poverty). Although PAISANO originally set out to introduce all interventions across the project communities, optimal results were found when interventions were tailored to the agro-ecological and cultural realities of the area.51 Practical, not theoretical SBC. Additional lessons learned can be gleaned from PAISANO’s hands￾on approach to teaching about and practicing improvements in complementary feeding. Recipe demonstrations offered opportunities for parents and children to taste new preparations and practice preparation, cash assistance (or home production) facilitated access to the food item, and home visits or other activities followed up on these experiences. Several of PAISANO’s communications materials were considered particularly useful and practical, such as the recipe flipchart and flash cards, because both were used actively for food preparation or games, rather than solely for information about complementary feeding (which can be complex, because it varies by age). As participants described their behavior changes, staff members were described as having respectful, collaborative attitudes toward participants, such as assisting with household tasks during home visits or suggesting that participants try only one bite or drop of a new food to overcome their fears. Effectiveness and preference for cash assistance. The majority of PAISANO participants, staff, and health partners expressed their belief that cash assistance empowered participants to practice the new nutrition knowledge and recipes learned from PAISANO through the purchase of a diversity of foods. Therefore, they viewed cash assistance as having greater utility to improve diets than a set of donated foods. An important element to the cash assistance was PAISANO’s systematic process for group planning and purchase of foods, as well as follow-up monitoring of purchases. Although highly regimented, project participants expressed their approval for the process, because it ensured that funds would be used for improving the family’s and child’s diets. Controls were eased up at the end of PAISANO’s implementation to give families greater liberty, but project reporting found that purchases of nutritious foods still predominated. An additional benefit was that vendors responded to the opportunity to sell a higher volume of diverse foods by bringing more products and offering volume discounts, indicating that demand for diverse foods will lead to greater market availability in this context. This successful experience represents a lesson learned for the context in the Western Highlands and for other areas with acceptable market access. It would be valuable to return to PAISANO communities well past project close-out to understand how families’ purchasing decisions were affected by the end of the cash transfer. Service fees as a subsidy for volunteers. PAISANO’s experience with funding volunteers’ subsidies by collecting monthly service fees from project participants who received food or cash assistance offers several lessons learned for other projects. As noted under EQ3, the funding source experienced variation over the course of the project as participants entered and left PAISANO when their children 51 Examples of agro-ecological and cultural constraints included poor goat reproduction due to a zone’s climate, communities with high or low interest in rabbit-raising, areas of water scarcity, and religious constraints against eating rabbit and goat, among others. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 43 aged out and when new eligibility criteria were applied for participation. The fluctuating amounts of service fees resulted in some volunteers losing their subsidies and therefore discontinuing service at different points in project implementation, creating transitions and turbulence for community coverage. The change in eligibility criteria was a MTE recommendation, so the negative effect on volunteer subsidies could not have been predicted during project design; however, it provides a consideration for future programming that considers employing service fees to fund a core function of project implementation (such as volunteer services). A larger concern is the temporary nature of these subsidies, which were found by the FFP-funded Exit and Sustainability Strategy to present a risk for the sustainability of service delivery.52 Therefore, it is most useful to consider using temporary service fees for temporary needs only—for example, to sustain service delivery only until an income-generating activity can support their services or another funding source begins. Volunteer incentives. PAISANO’s volunteers, and particularly mother leaders, took on important responsibilities that required a significant time commitment. Under these circumstances, it was logical to provide a subsidy to offset their time and opportunity costs for their time that could have been used for income-generating activities, as well as transport costs. Compensating volunteers’ significant workload with payment was helpful for ensuring that community-level agents would uphold their responsibilities and deliver results during project implementation. However, it does not result in promising options for volunteers’ sustained service delivery after the project has ended. Some volunteers expressed willingness to provide occasional support for individual families in need, or help MOH health staff, but understandably, none agreed to maintain a high level of service delivery without a new source of funding. Volunteer models that require a lower time commitment (and lower or no subsidies) may support sustainability more effectively in the Western Highlands, given the lack of an MOH model for training and support of community-level volunteers. In particular, support groups53 share responsibility among community participants and, in some contexts, have proven sustainable with minimal outside support. Volunteer empowerment and support for continued volunteer contributions. In addition to recounting their experiences with PAISANO’s training and activities, many PAISANO volunteers shared compelling accounts of personal transformation and increased self-worth that resulted from their role as a volunteer. Many volunteers hoped they would find ways of using their knowledge and skills, such as teaching, counseling, and public speaking. Although these project results are not captured as indicators, they are real effects of PAISANO’s investment of mentoring and supporting volunteers. Future projects could attempt to place volunteer empowerment as a more central approach to ensure that the project maximizes the potential of volunteers and assists them in finding a channel for their talents and interests, as they relate to the project objectives. Documentation of volunteers’ capacities and aspirations at project start to follow their progress over time would help projects better understand and respond to the opportunities, including by facilitating further study or support to charge fees for their services, perhaps in combination with sale of related products.54 Conclusions Lessons can be learned from PAISANO’s successful experiences in increasing dietary diversity and improving complementary feeding based on offering implementation approaches that synergize and complement each other and through effective SBC materials and methods. Use of cash assistance rather than food assistance offered benefits of reinforcing dietary diversity, and cash assistance was highly 52 Exit Strategy and Sustainability Strategy Summary Report 53 Annemijn E C Sondaal, Kirti M Tumbahangphe, Rishi Neupane, Dharma S Manandhar, Anthony Costello, Joanna Morrison. 2018. Sustainability of community-based women’s groups: reflections from a participatory intervention for newborn and maternal health in Nepal. Community Development Journal, bsy017, https://doi.org/10.1093/cdj/bsy017 54 As mentioned under EQ2, the PE team understands that PAISANO’s objective to assist volunteers to become self-sufficient providers of extension services was related to the agricultural sector, rather than the health sector, as this objective was not discussed for health in DIPs, MTE or other project documentation. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 44 preferred by project participants. In addition, PAISANO’s supportive process for planning and spending the cash transfers was highly effective for dietary diversity and accepted by participants. Volunteer subsidies funded by service fees provide examples of challenges for the stability of the funds, as well as repercussions for sustainability. Recommendations Ensuring that a range of interventions support and enhance the common goal of improved dietary diversity is an effective approach. These interventions should be practical, not theoretical, SBC activities, paired with home production and improved access to purchased foods, taking into account the opportunities and constraints related to agro-ecological and cultural factors. In the Western Highlands, cash assistance should be considered preferable to food assistance, provided that use of funds can be adequately supported and monitored. Using service fees as subsidies for volunteer activities should be considered with caution due to potential fluctuations and, more importantly, because they only last for the life of the project, presenting implications for sustainability. Empowerment and personal growth reported by volunteers are issues worth further documentation and support to capture these important results and maximize their benefits. 4.2.5 WASH: EQ1: To what extent has PAISANO met its defined goals, objectives, and outcomes? Findings Within the original design of PAISANO, WASH activities were limited to promotional messaging (SO2) as well as DDR-related WASH construction (SO3). MTE emphasized the close link between the lack of WASH services and children’s stunting and recommended the inclusion of WASH activities. However, due to the lack of additional funding, PAISANO used project funds for implementing activities at the community level as a complement to the behavior change and hygiene activities among households. Annex 5, Table A5.21 illustrates the baseline-endline comparison of sanitation facility, source of drinking water, and treatment of drinking water. At endline, improvements in household access to improved sources of drinking water, as well as improved sanitation services, were observed, but not in the availability of soap and water at hand washing stations. Table 6: Population-level WASH Indicators for PAISANO Intervention Communities Population-level Indicator for PAISANO Intervention Communities Baseline (%) Endline (%) LOA Target (%) Percentage of households in target areas practicing household water treatment technologies (boiling, filtering, chlorine) Chlorination 4.9 3.5 1 Filtration 1.2 23.3 25 Other 0.2 0.1 -- Boiling 96.2 85.4 74 Percentage of households using an improved drinking water source 21.5 41.8 -- Percentage of households using an improved sanitation facility 51.8 66.7 56 Percentage of households with soap and water at a handwashing station commonly used by family members. 77.1 80.3 83 PAISANO had important achievements in households that adopted appropriate hygiene behaviors. It was observed that environments inside the houses and yards were kept clean, mothers recognized the Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 45 critical moments for washing hands, the latrines were well managed, there was a strong acceptance of the use of Ecofilters for drinking water treatment, poultry were enclosed, and garbage was classified into organic and inorganic. At the community level, PAISANO strengthened the technical and administrative skills of the members of water committees, promoting chlorination and its monitoring. The project also established links with health districts for water quality monitoring and with the municipalities to support the provision of chlorine. Access to improved water sources. At endline, there was a statistically significant increase in the percentage of households using improved drinking water sources. Improved sources include piped water into dwelling or yards, public taps, public or private wells, and rainwater (Figure 12). Non-improved sources are surface water (river, dam, lake, ponds, stream, channel, irrigation cannel), tanker truck, and others. Figure 12: Baseline-Endline Comparison of Percentage of Households Using Improved Drinking Water Source 21.5% 41.8% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% 2013 Baseline 2018 Endline *** *** p<0.001 Data source: PAISANO Baseline and Endline PBS PAISANO did not make significant investments to improve households’ access to water services on a large scale. Some communities that prioritized water as a basic need were supported in improving their existing water supply systems, including activities such as replacement of pipelines, construction of distribution tanks, and installation of chlorinators. In Huehuetenango, PAISANO supported some communities in the construction of tanks for rainwater harvesting, an important and relevant activity considering the marked scarcity of water suffered by the communities of the municipality of Chiantla, but with limited reach compared to the total number of project participants. Both the linkage of the municipalities with project activities and the strengthening of the capacities of the community leaders to profile and negotiate funds for prioritized projects could have contributed indirectly to the improvement of the percentage of households with access to improved water sources. Access to improved sanitation facilities. There was also an increase in households using improved sanitation facilities, which include latrines and toilets that flush to piped sewer systems or septic tanks, as well as pit latrines with slab (Figure 13). Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 46 Figure 13: Baseline-Endline Comparison of Percentage of Households Using Improved Sanitation Facilities 51.8% 66.7% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% 2013 Baseline 2018 Endline *** *** p<0.001 Data source: PAISANO Baseline and Endline PBS Given that among the participant households the use of pit latrines with slab predominates, the increase in this indicator may be due to activities promoted by project, such as the application of the Participatory Hygiene and Sanitation Transformation (PHAST) methodology which promotes the improvement of latrine conditions, in addition to the use of ash, latrine cover, curtain over the entrance and a container for used paper. This methodology was focused on communities that were identified with higher rates of diarrhea and chronic malnutrition as well as deficiencies in sanitation facilities. PAISANO also encouraged linkages between participants and other institutions that could support them with sanitation facilities, such as Habitat in Huehuetenango. Handwashing. No change was observed in the percentage of households with soap and water at handwashing stations (Figure 14). The baseline data indicate that more than three-quarters of households have water and soap at a commonly used handwashing station. The project promoted the construction of tippy-taps located near the latrines using the resources of each household. Most participants agreed to the construction of tippy-taps and adopted appropriate criteria, such as installation of sumps to dispose wastewater correctly; however, in home visits, it was possible to verify that the tippy-taps were not being used and people continued using their pilas to wash their hands. Figure 14: Baseline-Endline Comparison of Percentage of Households with Water and Soap at a Commonly Used Handwashing Station 77.1% 80.3% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% 2013 Baseline 2018 Endline Difference from baseline to endline is not statistically significant Data source: PAISANO Baseline and Endline PBS Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 47 Other comparisons of WASH conditions among participant households. PBS data indicated that at baseline, most households boiled water prior drinking (96.2 percent). PAISANO promoted the use of filtering as a friendly alternative to the environment; therefore, at endline filtering had increased, from 1.2 percent to 23.3 percent (see Annex 5, Table A5.21). There was a decrease in the percentage of households that said water was not available for a day or more during the last two weeks (from 54 percent to 45.2 percent). This improvement could be promoted by the project's support to the rehabilitation of some community water supply systems; however, it should also be taken into account that baseline took place during the end of the dry season (April to June), and the endline took place between June and July, when the rainy season has usually started. Annex 5, Table A5.21 includes these and other changes in the WASH conditions among the participant households of the project. Selection criteria. PAISANO staff said that the communities selected to be supported with infrastructure projects and Food for Work (FFW) were identified through poverty criteria and by the level of interest in participating in the project activities. At the beginning of the project, the main criterion for the delivery of Ecofilter was households with children within the 1,000 days window. After MTE, PAISANO carried out the PPI study among participants to prioritize delivery inputs between households that fell below the poverty level, and those that did not comply with this requirement were not excluded from the project but received educational interventions only. Conclusions Although it cannot be determined to what extent the actions of PAISANO after the MTE had a direct and indirect contribution to the improvement in access to water and sanitation services among the participant communities of the project, PAISANO was successful in improving hygiene behaviors among participant households, as well as in strengthening the technical and administrative skills of the water committee members. Recommendations Future interventions should include actions aimed at improving the access of more households to piped water services and, taking into account the scarcity of surface water sources, encouraging the use of underground water in a sustainable manner because this is an abundant resource that is little used in the rural communities of Guatemala. 4.2.6 WASH: EQ2: Based on the evidence, which project outcomes are likely to be sustained? Sustainability factors. The application of hygiene practices observed in homes, the identification of critical moments for handwashing, and mothers’ affirmation that these actions have a direct benefit in the health of the family are the evidence that these practices will continue when the project ends. Both project staff as well as municipal authorities have observed significant changes in the hygiene behaviors of the participant households. In addition, the water committee members are currently carrying out activities related to preventive maintenance in their water systems. These activities are scheduled in clear, simple, and detailed plans. They also have tools and equipment (such as comparators to measure the concentration of chlorine), which they use correctly as a result of the technical strengthening received from the project. All these activities are now carried out without supervision by PAISANO, and therefore they are feasible to be continued in the future. However, these achievements can be greatly affected by the elections of new leaders that regularly take place in the communities every two years. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 48 Regarding Ecofilters, the evidence indicates that they will not be sustainable because the filtering unit must be changed every two years. Although the implementation strategy included encouraging families to save to be able to cover this future expense, none of the respondents acknowledged that they are making these savings. Many of them indicated that they see the benefit of using the filter because it is cheaper than the wood used to boil water and stated that they would look for ways to buy the spare part. Even so, the participants interviewed could not identify a local provider for filter replacements. In the case of communities that accepted water chlorination in their supply systems, only 50 percent are buying chlorine with contributions from service users. The rest are receiving it from municipalities or health districts with which PAISANO established continuity agreements for chlorine provision. Therefore, the continuation of this activity will depend on compliance with those agreements. Constructed and repaired assets. Rehabilitation of water supply systems is highly valued by communities, given that this is an essential service. In Guatemala, rural communities often have individual water supply systems and are directly responsible for keeping their projects in operation. They make economic contributions to carry out repairs only and contribute their work when necessary, so it is expected that they will continue doing this because they had been performing these services before PAISANO. Linkages with other service providers. PAISANO promoted the involvement of MOH districts in water quality monitoring activities as well as some municipalities for providing chlorine to almost a half of the communities that participated in the improvement of their water systems. In addition, PAISANO signed agreements with both actors so that they continue to provide these services after the project ends. Conclusions Good hygiene practices and safe water consumption will be maintained in the participating communities, as well as the improvement of preventive maintenance activities currently conducted by water committee members, although the latter may be affected by changes that may occur as a result of election processes in the communities. Activities that require investment in money from the users, such as chlorination and the purchase of spare parts for the Ecofilters, may be lost over time. Recommendations Projects should strengthen sustainability strategies aimed at those actions that are likely to be abandoned by their participants. For example, before initiating activities in a community, agreements approved by community assemblies could be signed in which there is a commitment to maintain a certain number of people who have been trained by the project. Because WASH infrastructure requires preventive maintenance, projects should promote the creation of community associations of WASH users. This allows establishing legal means for users to be obliged to provide adequate maintenance fees, but also allows them to demand quality services from those in charge of the services. Projects must go beyond solely involving government institutions in the activities implemented, because according to the law, municipalities are obliged to provide WASH services to their communities (Municipal Code) and the MOH is responsible for monitoring the quality of those services (Health Code). Projects must promote that each actor takes on the role required by law, as well as that the communities comply with the responsibilities that correspond to them, including payments for those services. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 49 Investments in WASH infrastructure must be designed to include the standards of the MOH and the Institute for Municipal Development,55 which include quality and comprehensiveness requirements, to provide sustainable services that generate impacts on the health of users. 4.2.7 WASH: EQ3: In the SO2 technical sector, what are the strengths and challenges of the effectiveness of the interventions’ implementation? Impact of interventions. The WASH TOC establishes that the impacts on health and nutrition in children are a product of a comprehensive intervention. PAISANO had a good approach for the promotion of hygiene habits within homes, but it was not as successful in supporting access to quality services in water and sanitation. Both the disinfection of water in the distribution systems and the improvement of the coverage of effective services (where quality, availability, and continuity are taken into account) were not included in the strategy that was implemented. PAISANO emphasized activities that were already consolidated in the communities, such as the promotion of disinfection of drinking water and handwashing stations, but it neglected the treatment of wastewater generated in households. Between baseline and endline, PBS found no change in the percentage of children under 5 years of age with diarrhea in the last two weeks (Figure 15). However, there was an increase in the percentage of children with diarrhea treated with oral rehydration therapy (ORT), which confirms the effectiveness of the project in improving the individual behaviors of people. Figure 15: Baseline-Endline Comparison of Percentage of Children under 5 Years of Age with Diarrhea in the Last Two Weeks and Children under 5 Years of Age with Diarrhea Treated with ORT 28.8% 53.3% 30.8% 63.4% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Percentage of children under age 5 with diarrhea in the last two weeks Percentage of children under age 5 with diarrhea treated with ORT 2013 Baseline 2018 Endline * ns ns=not significant, * p<0.05 Data source: PAISANO Baseline and Endline Those results should be analyzed in conjunction with other important factors that could affect them as well as seasonality. As mentioned previously, baseline and endline were carried out in different periods coinciding with the end of the dry season and the beginning of the rainy season of the year. According to MOH statistics, at the beginning of the rainy season, there is a peak in cases of waterborne diseases. This peak was occurring when the baseline was carried out, so it can be assumed that the seasonality did not affect the comparison with the endline results (Figure 16). 55 Ministerio de Salud Pública y Asistencia Social e Instituto de Fomento Municipal. “Guía de normas sanitarias para el diseño de sistemas rurales de abastecimiento de agua para consumo humano.” Guatemala, 2011. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 50 Figure 16: Waterborne diseases in Guatemala between Week 01/2012 and Week 26/2016 PBS baseline Data source: Department of Epidemiology, Ministry of Public Health and Social Assistance of Guatemala To examine the relationship between children under 5 years of age with diarrhea and the WASH status of their houses, the PBS team carried out a chi-squared test and determined that there was a strong link between the lack of services and WASH behaviors and the incidence of diarrhea (see Annex 5, Table A5.22). Targeting. Specific actions, such as provision of Ecofilters and the hygiene behavior change strategy, were targeted to families included in the 1,000 days criteria. According to the project staff, when the PPI was carried out, some households were determined as “not priority,” and therefore they were not provided with inputs after the research but were allowed to participate in education activities. Regarding communities that received support for rehabilitation of their water supply systems, PAISANO staff clarified that a prioritization was made between communities with higher rates of child malnutrition and communities that had expressed interest in improving the quality of water services. The staff also stated that in communities identified with low levels of sanitation infrastructure coverage, the PHAST strategy was implemented, although it did not achieve the number targeted. Acceptance by participants. Much of the acceptance of interventions and the involvement of people in educational processes is linked to an interest in receiving some type of tangible resource, such as filters and food or cash in the case of households or support for the improvement of water systems in the case of leaders. However, once people become involved in the interventions and became aware of their importance, participation was more spontaneous and proactive. Many of respondents expressed a complete conviction of the effectiveness of the recommendations of the project, such as the mothers who expressed that their families were not exposed to diseases because they consumed treated water. According to project staff and MOH representatives, carrying Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 51 out water quality tests in front of people and then discussing the results with them was an important reason for people to adopt good practices in their homes and accept the chlorination of water systems. The project’s message of learning by doing also motivated the water committee members to become fully involved in the training activities because they perceived that these were very valuable. On the other hand, the project staff indicated that they also faced strong cultural barriers, such as the belief that chlorine is harmful to health or will sterilize the population. Some communities interviewed stated that they believed that the use of chlorine was a strategy that came from developed countries to reduce the population. During household visits, it was observed that most families made good use of the latrine, kept it clean, and used and disposed of the cleaning paper correctly, but they did not keep the latrine covered, even though the project facilitators explained that they could use any resource for this purpose. It was also observed in some of the households that had built pens for their poultry that the chickens were kept in the yards and even inside the house due to the habit of allowing chickens to look for their own food, or due to other relatives that were not aware of the benefits of using the pens. Regarding gender, most WASH-related actions in the communities conform to traditional roles. For example, during the dry season, most recognized the need to carry water from a nearby source. Mothers said that they were responsible for this task because the “men work.” The mothers also said that men did not attend the hygiene talks because they were scheduled during times that men were working in the fields. However, many of the women in the Huehuetenango region stated that they also collaborated in agricultural work with the men, in addition to completing their household chores and without receiving support from the men at home. Conclusions At community level, the increase in access to WASH services did not have a significant impact on the reduction of diarrhea among children under 5 years of age. In general, WASH activities were well received by the participant groups. People were aware of the importance and benefits that WASH actions could represent in the health of families. In some cases, there are still strong cultural barriers that prevent the effectiveness of these interventions that must be taken into account in future WASH implementation activities. Recommendations Projects should identify and design behavioral change strategies based on cultural factors as an entry strategy in specific places to encourage the acceptance of WASH actions contained in the projects, which should include capacity studies related to intention to pay in order to obtain sustainable results. Projects should develop strategies (time schedules) for joint participation of mothers and fathers to WASH behavior change education meetings. Project should encourage young women to participate in water committees. 4.2.8 WASH: EQ4: What are key lessons learned and best practices that should inform future projects in the country? Considerations for future projects. Some practices and strategies addressed by the project should be contextualized. For example, the construction of tippy-taps was promoted even though most households had an outdoor sink basin (pila) with water and soap to wash their hands; as a result, many of the tippy-taps are not being used. On the other hand, the project did not put emphasis on the Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 52 correct disposal of the wastewater generated in homes56, which is an important factor in the chain of contamination within households. Given the impact that improved water and sanitation services have on the improvement of children’s nutrition within the 1,000-days window, a stronger strategy should be considered, and sufficient resources should be allocated to implement effective WASH actions. Lessons learned. Strong sensitization efforts must be made within the communities, because most believe that water is a free service. Therefore they do not prioritize their own investment to improve the availability of the service, often arguing that their economic situation does not allow them to make that expense, even though they pay much more for less essential services such as electricity, a cell phone, and, in some cases, cable television. It is important to conduct more intensive advocacy with the entities that are designated by law to provide water quality services in the communities, such as municipalities and the MOH, so that they assume the responsibilities that the law requires of them. Based on the lessons learned, how can potential unintended negative consequences be minimized and positive consequences be captured systematically? Payment for services. In addition to strengthening the provision of WASH services by the entities designated by the law, project participants must be sensitized to understand that quality services have a cost. This process should include research on participants’ capacity to pay, because currently the rates are based on the intention of payment. It is also necessary to strengthen accountability processes among community leaders, because many water committee members stated that people are reluctant to make economic contributions due to previous problems of embezzlement. Limited scope. WASH recommendations generated by the MTE were important and included in the project’s operational planning; however, the scope was limited, and, according to the project staff, the time was too short to be able to develop more consistent activities. The resources allocated to the WASH strategy were limited; for example, in the case of providing support for the rehabilitation of water supply systems, only the improvement of the operation was supported, not the extension of coverage for households or the improvement of service availability. As a result, this intervention had limited effects on the nutritional situation of children. Conclusions Given the positive impact that WASH interventions can generate on children’s health and nutrition, their inclusion in food security projects should include intensive activities for the population, and the necessary resources should be allocated for those activities. Acceptance of educational messages, both among participants and among community leaders, is due to effective behavior change and learning strategies through practice, and therefore they should continue to be addressed in the design of future projects. Recommendations Future nutritional food security projects should include WASH formally in their strategic objectives and results, given their importance for food utilization. They should address both community education and access to quality WASH services among participants. 56 According to the regulations for the design of rural aqueducts of the Municipal Development Institute –INFOM-, each person in a rural community must receive at least 90 liters a day. 12% is used for consumption and cooking, 18% for dishwashing, 30% for washing clothes and 40% for personal hygiene. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 53 4.3 SO3: IMPROVED RESILIENCE TO FOOD INSECURITY 4.3.1 EQ1: To what extent has PAISANO met its defined goals, objectives, and outcomes? Findings Guatemala is one of the most vulnerable countries to natural disasters around the world. Given that the populations that are affected by disasters generally experience significant setbacks in their levels of development, PAISANO initially focused on incorporating communities and municipalities into the National Coordination Body for Disaster Reduction (Coordinadora Nacional para la Reduccion de Desastres [CONRED]). The MTE determined that the strategy designed by PAISANO was focused on increasing community resilience57 to disasters but was not binding on food insecurity and chronic malnutrition problems, and therefore recommended working on strengthening governance processes in communities and linking them with municipal instances. Strengthening community governance. Based on the Guatemalan legal framework,58 PAISANO designed a strategy to strengthen the organizational and management capacities of the COCODEs and their various commissions. Taking into account the degree of vulnerability of each participant community (low, medium, or high), three intervention schemes were carried out. Those schemes refer to the intensity of the training process for community leaders. The first scheme was applied to all communities and included risk mapping and knowledge of legal framework of development councils. The second scheme also included updating the risk mapping, preparing risk reduction plans, and project profiling. The third included mentoring and accompaniment for negotiation of funding for prioritized investment in the COMUDE, NGOs, and government institutions. Those processes allowed the community leaders to better understand the roles and responsibilities of their positions in the COCODE or the commission to which they belong, according to what is established by law. The community mapping allowed them to identify the vulnerabilities that exist in their communities and how to face them through the elaboration of plans for integrated community risk reduction (PIRRs). The strengthening of their management capacities contributed to an effective management of the resources provided by the project, as well as profiling and negotiating funds for prioritized projects included in the PIRRs. An important aspect of this process is the promotion that PAISANO included for accountability. Linkage with the municipal instances. PAISANO worked with the Municipal Planning Bureaus, both to strengthen their support functions toward the communities, as well as to incorporate the prioritization of food security in the COMUDEs. In addition, PAISANO favored rapprochement between community leaders and municipal authorities so that the latter would financially support some prioritized projects as well as project activities, such as chlorination of water systems. Achievement of the project’s indicators. As shown in Table 6, at the end of FY5, some of the project’s goals had been achieved (projects supported with FFW, community vulnerability maps, and risk reduction plans); therefore, FY6 was focused on consolidating missing goals, as well as sustainability actions (signed agreements). 57 Community resilience refers to the capacity to overcome an adverse situation and continue development processes. 58 Decentralization Law and Development Councils Law Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 54 Table 7: Community Resilience Indicators for PAISANO Participants Reported in ARRs ARR Indicators for PAISANO Participants Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 LOA target Impact Indicator Number of communities that are implementing activities from their plans to improve resilience 38 43 -- 44 Percentage of the communities understanding the vulnerability maps 22% 38% -- 53% Number of communities that have completed vulnerability maps 174 187 -- 189 Number of communities that have completed community plans 65 90 118 165 -- 120 Number of people trained in disaster preparedness as a result of USG assistance 482 476 984 2372 -- 2946 Percentage of communities implementing projects with municipal resources 8% 19% -- 34% Number of agreements signed with the municipalities to improve development of the PAISANO intervention communities 5 6 -- 13 Strategic Objective 3: Community resilience through governance & disaster preparedness improved IR 3.1:Community preparedness for prevention and mitigation against adverse shocks increased Outcome Indicators Output Indicators IR 3.2: Community and municipality governance improved Output Indicators Data source: PAISANO Indicator Performance Tracking Table, FY 2017 Factors that promoted or inhibited the achievement of the project objectives. PAISANO had to face the fact that chronic child malnutrition is an invisible problem for society. At the same time, there is little knowledge of the structural causes of poverty; therefore, neither the population nor the government authorities give relevance to this aspect in their development plans. Moreover, community leaders are usually not interested in training processes, because they consider them to be a waste of time. PAISANO identified that the main motivation of the community leaders was to obtain financing to execute infrastructure projects, and the municipal authorities were interested in improving their relationships with the communities; therefore, project strategies were based on taking advantage of those aspects. Targeting strategies. PAISANO prioritized working with communities with high vulnerability and chronic malnutrition rates among children of more than 60 percent. The project identified communities with high vulnerability and that expressed interest in addressing their problems related to food security and worked with them to strengthen their capacities to develop project profiles and, according to the content of their community plans, provided support with FFW, including construction materials and tools to carry out rehabilitation of water systems, roads, and micro-irrigation. Community leaders who were involved in the third scheme of training were also strengthened in their project management capacities, prioritizing the contents of their community plans so they could present them to the COMUDE. Municipal authorities emphasized that the leaders who reached this level of strengthening were distinguished in COMUDE meetings due to their strategic vision and strong management capacity. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 55 At the same time, the project worked to strengthen Municipal Planning Offices in project management issues so they could provide support to the communities for presenting financial requests to government organizations and civil society institutions. Likewise, they promoted the discussion of chronic child malnutrition in the COMUDE meetings. However, the local and community authorities stated that there was little interest in this topic and that in most of the meetings the topic was addressed by MOH staff, but only through the presentation of data about health issues. Contribution of PAISANO to USAID’s efforts to reduce food insecurity. PAISANO has sensitized both the communities and the government authorities about the effects of chronic malnutrition in the development of the Guatemalan population, especially its effects on poverty. Because of this, some municipalities have assumed a commitment to continue supporting some project activities, as well as to strengthen the municipal bureaus that can follow up on the project’s efforts when it ends. Given that the COMUDE is an instance where government investment is prioritized, the effect that the project may have achieved among community leaders and municipal authorities will have a direct influence in addressing the problem of chronic malnutrition and food insecurity. Conclusions In general, community organization and governance were strengthened by improving the capacities of community leaders, both in the effective execution of their functions and in their relationship with government authorities and also in raising awareness of both parties regarding food security. It must be recognized that the scope was not large, because, as expressed by authorities and community leaders, prioritization of projects in the management bodies is still incipient. Despite these SO3 achievements, the evaluation team could not objectively determine whether these processes have brought an improved level of community resilience. Recommendations To achieve community resilience, it is important to analyze, in accordance with the TOC, whether it is sufficient to strengthen the organizational and management capacities of community leaders, in addition to linking them with government authorities, or whether it is necessary to address more in-depth and focused processes. 4.3.2 EQ2: Based on the evidence, which project outcomes are likely to be sustained? Quality and functionality of the processes. The strengthening of the organization and management capacities of the community leaders who participated in the second and third intervention models, as well as their relationship with the municipal dependencies, will be maintained when the project ends; however, these processes are vulnerable to being lost due to the changes in community authorities every two years, as well as the popular elections every four years. PAISANO developed a strategy for transferring responsibilities among community leaders who are subject to change; however, in the focus groups carried out for the PE, a significant difference could be observed in the performance and knowledge of the leaders who had been strengthened with the project compared to those who had recently assumed their new positions. Maintenance of constructed assets. In many cases, community infrastructure rehabilitation projects were well received by the population because they entailed support with materials and with FFW. Although PAISANO worked with leaders to develop maintenance plans, they may be followed up initially by members of COCODE or other community commissions, but over time they may be abandoned because were originally based on incentives. PAISANO promoted the installation of local surveillance and alert systems implemented in four communities to obtain information that could anticipate a nutritional emergency situation. These Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 56 systems are being operated by local people trained by the project. However, these systems may not be sustainable because it was not possible to identify partners with interest and ability to use the information generated, as well as support response actions. Community perception of the process. All the community leaders who participated in this study agreed that the participatory methodologies applied by the project were very effective. They pointed out that the best way to learn is through practice, and they felt very motivated because PAISANO staff served in a facilitation role only. Community leaders were the ones contributing ideas and knowledge for the development of community mapping and the formulation of community plans. Something that stood out is that through the cross visits between communities, community leaders were able to learn about the successful experiences of other communities, so they could replicate them. Community leaders also noted that the personal connections that PAISANO staff made with officials of institutions and government officials to present their project requests were very valuable and enabled them to understand the procedures for obtaining financing. Linkages with other service providers. Municipalities and some government agencies (such as the Chichicastenango Health Area) signed agreements to continue providing supplies and services, such as chlorine and water quality monitoring in priority communities. Some community leaders are currently negotiating funds for prioritized projects in their PIRRs. In San Juan Ostuncalco, the mayor said that the Municipal Office for Women and the Office of Economic Development will be supporting some of the health and food production activities initiated by the project. Conclusions Project staff, authorities, and community leaders agree that the capacity-building processes and the linkages generated with the project are quite solid and effective. Some project activities have signed agreements for follow-up and support from different actors; however, all these achievements can be weakened by the change of authorities structured in the law. PAISANO designed an innovative strategy for transition of authorities in their different levels of intervention, which could be effective, but this can only be verified with the passage of time. It must be recognized that the structures that greatly influence sustainability of activities are beyond the control of the project. Recommendations As recommended by the project staff, sustainability of the project activities is more likely to be achieved if these aspects are incorporated from the beginning of the project and not when the project is coming to an end. 4.3.3 EQ3: In the SO3 technical sector, what are the strengths and challenges of the effectiveness of the interventions’ implementation? The effectiveness and relevance of the technical interventions. PAISANO designed methodologies and adopted effective techniques according to each target group (“learning by doing,” or ERCA, acronym from Spanish that means Experience, Reflection, Conceptualizing, and Task). The participatory processes, sharing of experiences, combination of theory and practice in the training processes, and mentoring and accompaniment in the management processes were highly valued by the participants. Participatory development of vulnerability maps and integrated risk reduction plans sensitized community leaders about their vulnerabilities and how overcome them. Leaders who completed the third scheme continued profiling projects and negotiating economic resources in other government and private institutions. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 57 The process of linking the municipal authorities and representatives of institutions with leaders who were trained was also considered effective from both sides, who agreed that the project activities gave credibility to the institutions that are working in that area. In addition, FFW as well as project’s funds and inputs were used to implement prioritized projects included in PIRRs. This promoted the interest of the leaders to participate in the training processes, as well as to put into practice their project management skills. Factors in the implementation associated with effectiveness. Municipal authorities made reference to communities where other projects have encouraged participants to be dependent and not take active roles in efforts that encouraged their own development. Therefore they believed it was difficult to implement projects like PAISANO that mostly promote learning and behavior change. PAISANO addressed important factors to favor governance within communities, as issues of accountability within the processes of strengthening community leaders. This is relevant because it was identified that the reluctance of many people to pay for water services is due to previous leaders misusing the funds of the community. At the discretion of the project staff, contextual factors, such as culture, language, and education level, influence participants’ receptiveness to the activities addressed by the project. For example, a highly experienced institution was hired to develop training activities for water committees, but in some cases, the technicians in charge of facilitating the workshops did not speak the local language. Therefore, despite the fact that participants found the topics and methodologies interesting, they found it difficult to understand the messages. Some of the IPs also stated that they observed some project staff who did not have much identification with the communities, and therefore their meetings were not well attended, and in some cases training sessions had to be rescheduled because participants did not show up. Processes deemed more or less acceptable for target communities. It was common for community members to place low value on the training activities. Initially, PAISANO had limited participation during the first training sessions. The FFW was seen as an incentive to motivate the involvement of leaders in the training processes. Another factor that was alleged was that the schedules were not convenient due to the constraints of employment obligations. The project staff showed flexibility to adapt schedules, as far as possible. The leaders also stated that at first they perceived the training topics as boring, but the methodologies implemented by the project encouraged interest in the topics, so participation improved. Although gender issues were included in the training for community leaders, there was still a lack of women’s participation in the COCODEs and the water committees. On the other hand, in the COSANs, the majority of members were women. This demonstrates the strong division of labor between men and women in community organizations. One of the contributing factors for this that was mentioned was that communities in El Quiche have an organizational system in which all adult men have to be members of at least one community committee, which in effect excludes women. PAISANO also tried to influence compliance with the legal aspects related to the conformation of the COCODEs, such as their relationship with all existing committees and commissions in the communities; however, power disputes meant that in many cases this was not accepted. Project staff also reported that they made efforts to encourage compliance of community leaders in the legal aspects of accountability, such as the registration of committees and commissions with the Superintendency of Tax Administration and the Comptroller General Accounts of the Nation. However, there was great reluctance to accept these processes, and leaders said that legal procedures were cumbersome and generated fiscal obligations that may be subject to sanctions if they did not receive adequate follow-up. During the interviews, the majority of leaders stated that they have processes of Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 58 accountability in their communities, but they do not always comply with the registration requirements of the governmental entities that frame the law. The project took gender into account during the training sessions. When asked about this, several of the leaders recognized the importance of including women in the decision-making process, mentioning that their ideas and contributions enriched the discussions about the needs of the populations. Women participated in several of the committees that participated in the evaluation, but in a small proportion compared with that of men, and the segmentation of roles was also obvious, such as COSANs comprised entirely of women and committees formed for infrastructure management in which women did not participate. Several of the community leaders interviewed recognized the problem of chronic child malnutrition, but they were not yet addressing it as a key problem for the development of their communities, which was observed when they were asked about their management priorities or their interactions with the municipal authorities. Conclusions The design of the project implementation was effective in taking into account the characteristics of the target groups that were defined in the project. This made it possible to find effective solutions to the difficulties that staff encountered and to generate interest in activities and issues that were not yet fully accepted among residents and local authorities. The change in the strategy of implementation that resulted from the MTE improved the strategic focus of SO3, but it remaining time frame was inadequate to consolidate some important processes, especially in terms of raising awareness about effective actions linked to development. Recommendations Projects should dedicate enough time to generate awareness, both among participants and among authorities, to achieve effectiveness and sustainability of activities that render a significant impact on health and social development. Project should establish agreements with communities at the beginning of a project to favor equal participation of women and men in the commissions and committees formed or strengthened by the project. 4.3.4 EQ4: What are key lessons learned and best practices that should inform future projects in the country? Lessons, best practices, and approaches that should be considered in future projects. Some project partners thought that methodologies and strategies of the project are not completely aligned with those of the governing bodies at the national level, such as the MOH and SESAN. Some community leaders pointed out that it would be important to take their opinions into account when designing the projects so that their expectations and needs are included, although this may conflict with the objectives of the project. Sufficient time and effort must be devoted at the level of the communities and the authorities to sensitize them on the issue of food security and chronic child malnutrition, so that their involvement in the project is more profound and effective. The processes of participatory analysis and planning are valuable to contextualize and promote the processes of appropriation and identification of the participants and other actors with the project. The processes of linking communities and authorities require referencing the legal frameworks applicable to the parties, generating commitments, and following up on and strengthening the development processes that start from the community level through the national level. Exchanges of Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 59 experiences between communities with similar cultural and geographic characteristics foster interest by sharing their achievements, difficulties, and solutions. According to the project staff, if sustainability actions (as signed agreements, transference of responsibilities to government and municipalities instances, as well as activities to face turnover among leaders) were started at the beginning of the project, they could be better consolidated by the end of the project and would be much more effective. According to the project staff, the use of effective techniques, such as ERCA, in training processes for adults, as well as learning by doing methodologies, generated quality and sustainable results in the performance and appropriation of the participants. Identified risks. Given the high involvement of community leaders with government authorities, the project staff noted the risk of the politicians take advantage of project achievements, as happened in Huehuetenango, where some leaders were manipulated to use their positions and resources to support proselytizing campaigns. Conclusions Differences between target participants and strategies of implementation were mentioned by government institutions as impediments to effective coordination. The use of effective training techniques encouraged the acceptance and appropriation of the project topics. Project achievements can be used in a negative way in political campaigns. Recommendations Projects need to generate awareness of activities that will have a significant impact on participant health and social development. Projects should perform context analysis exercises before designing their operational strategies so they can understand the expectations of the future participants without neglecting the priorities of the donors. For improving community resilience, it is necessary to determine whether it is sufficient to strengthen community leaders’ organizational and management capacities, including linkages with government authorities, or whether it is necessary to address root causes, such as lack of essential services and job opportunities and deficiencies in education or health services. Interventions must include formal links with established national systems, such as accreditations to CONRED and SESAN, as well as compliance with applicable legal requirements, such as the registration of COCODEs and commissions at the Superintendence of Tax Administration and the Comptroller's Office of Accounts of the Nation. Good practices developed by the project should be replicated, especially those that include behavior change and capacity building because they have broken some barriers to the implementation of development projects. 4.4 CROSS-CUTTING ISSUES 4.4.1 Project Management Planning. The DIP was the main planning instrument used by the project. It served as the main guide for activity planning and included quarterly and annual targets to be reached under each SO. In addition to quarterly project planning, the regional coordinators interviewed said that they used the tool to conduct monthly staff meetings to review progress toward quarterly targets and make adjustments accordingly. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 60 The DIP was developed early in the project and served as a basis for an annual planning tool. Most managers and staff interviewed felt that the DIP planning was a useful process, giving them a path to achieving targets and helping them plan ahead and plan for the supplies that would be needed. One of the issues expressed by the PCI partner, was that Save the Children, based on their partnership experience with a previous food security project, was used to developing all the strategies, guidelines, and plans and making all the decisions, and expected PCI to agree. When PCI staff began to question some approaches and suggest different ways of implementing activities Save the Children was surprised and explained that being the prime, it was their responsibility to make all planning decisions. With time and particularly after the MTE, Save the Children realized that PCI had useful ideas and suggestions and began to ask their staff to participate in the planning of annual activities and development of field guidelines but always maintained the right to make final decisions and insisted that activities be implemented exactly as they had been planned. Most of the staff noted that although the DIP was a useful planning tool, it was also as noted as restrictive in terms of the activities they could conduct, particularly in the first years of the project. The staff were told to implement only the activities in the DIP, and they were required to implement the same activities for every community. This created some issues for the consortium partnership when, given the large number of families that are small coffee producers in Huehuetenango, PCI proposed including some technical assistance to help these families increase yields and quality of their coffee but were told that this was not part of their DIP mandate. However, during a visit by a USAID Guatemala officer, the officer realized that this was a need and requested that PCI incorporate this activity. At this time Save the Children did not have any objections, and through a PCI-ANACAFE partnership, more than 150 families received technical assistance on crop management, repair and maintenance of pulping machines, appropriate drying techniques, and access to better markets due to improved coffee bean quality. After the MTE, Save the Children realized that they needed to be flexible and implement activities that were appropriate to each community situation. This added flexibility allowed the project to not support home gardens where people did not have land or preferred to use their land for other purposes such as animal enclosures. In the case of PCI, it allowed them to implement cash crop production activities in appropriate areas and climates for potato, coffee, passion fruit, and papaya. Some respondents argued that the project planning could have been more flexible, noting that the annual plans usually had to be submitted in November of the year before, even though they would not be implemented until October of the following fiscal year. Another partnership planning challenge noted by PCI was related to the joint meetings that were to occur quarterly. The problem was that the dates for the meetings were changed frequently and with little notice, which required PCI to modify work plans and schedules and caused delays in implementation of activities. Logistics: The MTE team found that the commodity warehousing and conditions of the warehouses were good. They noted that the warehouses were clean and well ventilated, commodities were stacked properly and had pallets under them, non-food items were not stacked next to food items, and natural pest management tools were used. Records were well maintained with updated cards per stack, and warehouse managers prepared daily reports and physical counts at each delivery. The project delivered commodities from main warehouses to community distribution centers that were managed by the Comisiones Comunitarias de Salud Seguridad Alimentaria Nutrisional (Health and Food Security Community Committees). After the MTE, the project switched to a cash transfer system and stopped the commodity distribution. The evaluation team reviewed ordering procedures and purchases conducted by the project and found them to be generally acceptable. When interviewed, project staff and managers indicated that nearly all logistical items were ordered and procured in a timely manner. This included vehicles, fuel, per diems, and most supplies. The supplies were ordered by the technicians following and in accord with the Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 61 quarterly planning sessions. There were a few supply issues related to availability. Often there were delays in finding good quality animals to buy for the project participants such as rabbits and goats. The animals had to be brought from other areas of the country, which added to the delay. Another issue was the Ecofilter procurement. Initially the project procured the filters and provided training to the communities on how to use and maintain them, requiring that each family put aside 10 quetzals every month for filter replacement in two years. However, the MTE expressed concern that the filters were being given away and suggested that the project require that families buy them. At the time, Save the Children had already purchased 4,000 filters for distribution to participants. After this recommendation, the project and FFP went back and forth reviewing different plans for purchasing the filters. By early 2018, a decision had not been made, so at that point, FFP told the project to distribute the filters. This distribution was being done at the time of the PE, but obviously the project did not have time to train participants on the filters and set up a savings scheme for families to replace them in two years (see SO3 discussion for further details). Restructure of project organization. One of the factors that affected the overall management of the project was that as a result of the MTE recommendations, there were a number of project-related organizational changes. In addition, around this time, the Save the Children PAISANO leadership decided to restructure the project organization. At the beginning of the project, the regional field offices were organized so that the technicians (one or more for each SO in each region) reported to a regional supervisor who provided supervision and coaching. The supervisor then reported to the regional coordinator in a vertical system. The coordinator oversaw all the project activities and personnel in the office. After the restructuring, the technicians were placed under the supervision of subject specialists (SO specialists) located at headquarters, even though they were working in the regions. The specialists’ role was to design and oversee the implementation of field activities. The technicians were still supervised by the regional supervisors, but their supervisory role was only to determine whether the technicians were doing their job. If there were questions about the design of the job or methodological issues, then the supervisor reported this to the coordinator who then informed the appropriate SO specialist at headquarters. The SO specialist then redesigned the work and trained the technician on the new methodology. The reason given by the headquarters staff for this change was that they had wanted the technicians to be independent of the office structure. Several of the coordinators and technicians talked about this reorganization. The result was that the technicians had two supervisors and while some seemed to be managing it, others, especially some of the regional coordinators who were trying to manage all the activities, found it confusing. MTE recommendations and changes. In addition to the project’s restructuring, the MTE recommendations required several programmatic and organizational changes. In general, most people interviewed believed that the recommended changes were beneficial, improved some of the project design weaknesses, and helped re-focus the project. These included a more direct focus on WASH, much of which before had been seen as the responsibility of the Municipalities. Another change was to be more flexible in introducing community activities, taking into account appropriateness for community conditions instead of requiring that all activities be introduced in all the communities as in the original DIP (as discussed above). Another change was the reduction of the number of SBCC activities included in the SBCC package. The number of actions promoted was reduced from 19 to 5 (EBF, complementary feeding, handwashing, clean house, clean drinking water). There was also an increased emphasis on gender activities (see Section 4.4.2). A major change was the decision to replace Title II commodities with a cash transfer system (see SO2 section for discussion). This was a lengthy process involving education and follow-up with families to ensure that appropriate foods were selected (three food groups) and determining the markets with the best prices. This new system did not roll out until 2017. Another major change occurred as a result of an MTE observation about whether the project was working with the poorest populations in the regions and asking PAISANO to evaluate this. When asked about this, the headquarters staff stated that they Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 62 thought the project was working with the poorest sectors of the population but that it was difficult to establish poverty standards. As a result, they relied on the 1,000 days criterion (pregnant women, lactating women, and children under 2 years of age) promoted by FFP. However, in response to this MTE recommendation, the M&E unit conducted a PPI59 assessment of all the project communities to ensure that the project participants were in the poverty or extreme poverty range. They found that 85 percent of the families they worked with fit the PPI criteria for poverty. In discussing this with respondents in the regional offices, it was discovered that although this 85 percent figure was true for some areas, it was not for others. For example, Quetzaltenango reported that they lost between 25 and 40 percent of the families with whom they had previously worked. There was some disagreement among respondents regarding how accurate the PPI criteria were for assessing poverty, but at the same time, the respondents felt the analysis had validated that the project was already focused on the poorest sector. In addition to the poverty criteria, another recommendation affecting participant beneficiaries was to limit the participant numbers to one 1,000-day period per woman for ration or cash distribution. The result of these recommended decisions was that although no one was excluded from the project because of financial status, those who did not meet the poverty criteria or who had already participated in the project were no longer given rations or cash transfers. Any additional children were not included as participants, although the families were invited to continue participating in all educational activities. This change created a difficult operating atmosphere for the project. According to staff and community members interviewed, many families had to make special arrangements to attend project activities and because the food or cash was no longer offered as an incentive for attending meetings, they opted not to continue. Conclusion The DIP was a useful planning tool for project management and for reaching the programmed quarterly and annual targets. After the MTE, it became more useful by reducing the number community activities and permitting more flexible decision-making about which activities to promote in the different communities based on context. In general, all supply and logistical needs were met and organized by the Save the Children and PCI administrative offices. The MTE recommendations, together with Save the Children’s decision to restructure the overall project organization, brought about many changes to the project. In general, it was felt that the changes improved the project. However, because there were such a large number of changes that affected participants in the already large number of project communities, many of them were not instituted until the last year of the project. Now that the project has ended, it is questionable whether there has been enough time for the participants to really absorb the numerous behaviors, skills, and other improvements promoted by the project. Recommendations Future projects should use the first year of activity to develop their DIP because this will allow them time to evaluate and test appropriate implementation strategies that they plan to introduce. In response to team interviews with PAISANO staff who stated that the number and major nature of the changes required by the MTE recommendations made them very time-consuming and difficult to implement, future MTE evaluations should consider a collaborative, informal risk assessment by both the MTE team and IP of what the recommendations would entail. FFP staff stated that they conducted “a workshop after the MTE to discuss the recommendations and gave an opportunity to IPs to assess the feasibility of the recommendations given the remaining life of the project.” To strengthen this analysis, the recommended risk assessment could follow-up and ask for the IP to identify and list possible risks to implementing the proposed changes and to propose responses should those risks materialize. 59 The latest version of the PPI for Guatemala was created in October 2016 by Mark Schreiner of Microfinance Risk Management, L.L.C. Indicators in the PPI for Guatemala are based on data from Guatemala's 2014 Household Living Standards Survey. https://www.povertyindex.org/country/guatemala Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 63 4.4.2 Gender The gender activity was overseen by two SBCC and gender specialists. One worked with Save the Children and one with PCI. There were some differences in the gender approaches of the two organizations but they were able to coordinate on the strategy assessment and action plan. According to project staff, one of the reasons gender issues did not receive as much attention during the first half of the project was because the gender officers at Save the Children and PCI were also the SBCC officers, so during much of the initial half of the project they focused on developing SBCC strategies and materials to support and promote SO activities. The gender specialists and some of the project staff also indicated that, in addition to the project having other priorities, they thought the slow start to the gender-focused activities related to the lack of gender training and requisite skills for promoting this behavior change among staff. The specialists also identified a need to further work with project staff on their own gender biases, which, together with the lack of skills, may have accounted for their reluctance to introduce these activities earlier in the project. In 2014, the project conducted a rapid gender assessment, which documented how families make decisions and who makes them (men, women, mother-in-law), how families organize the division of labor, and who participates in different community events. Although the report did a good job of describing the gender situation in the target areas, as noted in the MTE, it did not provide an analysis for understanding gender dynamics that could guide project activities with respect to strengthening gender roles and sensitivity. According to the MTE and FG interviews, men’s involvement was limited because PAISANO was seen as a “women’s project.” The project had made an effort to involve men in the nutrition schools, but it was difficult to accommodate their schedules for participation. The MTE notes that at that point the project had not really assessed the effect of women’s participation in project interventions or developed strategies to improve women’s participation in decision-making. In response to the MTE, the project produced a Gender Action Plan60 with the intent to promote gender equity and equality to further project goals and objectives. The plan focused on actions that the project could take to involve men and women jointly in more of the SO activities. The theme was not so much women’s work versus men’s work as it was “working as a team” and sharing the work and household tasks. There was a strong effort to include both men and women in meetings and have fathers participate when women could not be available for meetings so they would learn and support the nutritional and child health messages being conveyed. After the Gender Action Plan was formulated, the project realized and decided that it needed to train all PAISANO staff so they could convey the gender message effectively. The project also recognized the need to target men because the assessment indicated that men were the decision-makers and, in addition to mothers-in-law, they were the main barrier to women’s participation in activities. The project developed a course for men61 consisting of four sessions emphasizing the theme of “better relationships between men and women.” They chose this theme because speaking about gender in the context of women’s rights did not resonate with the communities. Another key message was that instead of asking men to help their wives at home, it was more important to recognize that men and women as a team are responsible for the home and family. The men’s sessions were implemented in 2017 because it took some time to develop training materials, train staff, and determine how to organize the men’s groups. Part of the challenge was determining how to reach men and interest them in participating in a men’s group at a time that was convenient for them. The men’s course had sessions entitled “short meetings with men,” and the project realized that it already had men’s groups formed as part of the FFW activity (coffee, potato, papaya, etc., producers groups). Working with project supervisors, the team arranged to interrupt the meetings and give their sessions. Originally the gender team was planning to work with the 60 Save the Children. PAISANO Gender Action Plan (GAP). August 31, 2015 version. 61 Save the Children, Project Concern International. Lineamientos para implementar reuniones con grupos de hombres “Aquí entre hombres, hablemos de…” Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 64 technicians because they worked directly with the mother leaders and agriculture leaders, but the project leaders suggested that they work with the supervisors because they had the time and facilitation skills. One of the most impactful sessions was the “reloj exercise,” in which the men wrote down all the activities their wives did on a daily basis and compared them with the list of activities that the men did. This exercise made the men realize how much work the women did and encouraged them to take on more household responsibilities. The men interviewed by the evaluation team were enthusiastic about what they had learned about nutrition and how their relationships with their wives had changed through participation in the sessions. According to project staff and community leaders interviewed, the sessions were successful, particularly among younger men, but they indicated that this activity would have been more successful if it had started at the beginning of the project. Many of the staff stated that more time was needed for the communities to absorb the messages. Several of the participants as well as project staff interviewed commented that project interventions that helped women gain confidence included the women’s empowerment groups and the cash transfer program. Both these activities had women managing financial resources. The women’s empowerment group members had to learn basic accounting to keep track of the savings contributed and the loans as well as calculate interest to be charged or paid (see SO1 for more details). Then they had to figure out small business schemes in which to invest savings or withdraw loans and calculate how they would achieve a profit and pay back the loan. In all, the project formed and supported 120 women’s empowerment groups. Women receiving cash transfers had to learn how to manage the funds to purchase the variety of recommended foods at the best price. When interviewed and asked about what changes they had experienced, several of the women in these groups commented that the experience had given them more confidence and a sense of purpose in addition to learning bookkeeping skills. They explained that when the groups first formed, many of the women did not talk. Now that has changed and they all have a lot to share and discuss. It is difficult to measure the exact impact of the various gender strategies employed by the project. The quantitative PBS data indicate that during the project period, per capita expenditures increased among adult women by almost 20 percent and that the number of adult females living below the poverty line was reduced by almost half. At the same time, however, the percentage of women making decisions about their own health care decreased to 50 percent, which was 11 percent less than at baseline. This is confusing because women maintained a high level (95 percent) of making decisions about their child’s health care, indicating that they maintain their decision-making role in the home. When asked about this, respondents speculated that perhaps the project’s emphasis on joint decision-making between spouses about family health issues might have influenced women to discuss her health decisions with her spouse/partner instead of taking the decision by herself (see Section 4.2.1 for more discussion) But this does not explain why joint decision-making did not apply when making decisions about her child’s health. Another explanation was that at the end of 2014, the MOH closed the Community Coverage Extension Program. Because of this, except for a few health posts, there was no community-level service provision. Therefore, people who wanted health services had to invest in transport to distant locations and were unlikely to do this. PAISANO staff believed this factor contributed to the drop in women’s health￾seeking behavior during the second half of the project. In general, staff, leaders and men’s group participants were mixed about how far-reaching and sustainable the gender and relationship messages were. The consensus was that the communities needed more time to accept and integrate the new messages and behaviors being proposed. Those who participated liked them but did not think they had reached enough people and thought they needed more time. Conclusions In general, the project has made progress in improving gender awareness, particularly after the MTE, when more emphasis was placed on this area. The project developed Gender Action Plan and followed Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 65 up with training project staff and community leaders. The plan emphasized ways and discussion techniques for involving men in the ongoing educational activities. In addition, during the last year of the project, staff began working with men’s producers groups through the FFW activity, introducing a four-session training curriculum called “short sessions with men.” This was a successful activity focusing on family relationships and sharing household and family tasks. Other successful interventions included the organization and support of women’s savings and loan groups and the cash transfer program, which put women in a financial decision-making role. Despite improved poverty indicators for adult women noted in the PBS, there was a statistically significant decrease in the level of decision-making that women made with regard to their own health care. Because gender is such a culturally ingrained value with traditionally defined roles for men and women, it is concluded that even though important steps were taken for improving gender awareness, the project needed more time to integrate the new values into community consciousness. Recommendations Future projects should take into account the lessons learned from gender intervention experiences implemented by PAISANO. Such interventions include the strategy and materials for working with men’s groups, the savings and loan empowerment groups, and the cash transfer strategy. Future projects should implement the full scope of gender activities at the beginning of the project. This will entail early introduction of gender assessments and staff training regarding how to introduce gender inclusion activities into project implementation. It will also be necessary to assure that project staff are made aware of their own gender biases and introduced to mechanisms for addressing these as well as sensitizing others. Also, as noted in SO3, efforts should be made to include more women in the community commissions and committees. In addition to targeting men in gender-related activities, future projects should also target mothers-in￾law and grandmothers, who were also identified as influencers in women’s decision-making. 4.4.3 Monitoring and Evaluation The Save the Children M&E unit is composed of seven staff: one M&E specialist who manages the M&E for the project, two regional supervisors (Nebaj and Quetzaltenango), and four technicians (Nebaj, Quetzaltenango, Cunen, and Chichicastenango). PCI has five M&E staff: one coordinator, one quality monitor, one facilitator, and two data entry staff. These staff oversee the data collection process and conduct data quality assessments on a monthly basis. The project installed a routine monitoring system for the collection of annual monitoring indicators contained in the IPPT and established in the DIP plans. The indicator data are collected monthly by the different SO field technicians using data collection forms designed by the M&E team. The data are then verified by the M&E technicians and quality officers and entered into the management information system (MIS) quarterly. There is good coordination between the Save the Children and PCI partners for data collection, recording, and aggregation of data for reporting. The information is input quarterly into the web-based MIS, the automated information monitoring system (sistema automatizado de monitoreo de informacion [SAMI]) developed by Save the Children.62 SAMI is used as a registry of project participants and contains information on the participant and participant household as well as fingerprint and photographs for identification. The system is used to track performance data by participant, such as their health status and all SO activities that they take part in such as tracking height and weight of children by age, allowing PAISANO to document children’s growth. The results of home visits and documented activities are analyzed during regional monthly team 62 SAMI: a system developed for another Save the Children project in the eastern regions of the country. It is used to track project participants there identifying them with their fingerprints and photographs. The system was adjusted for use by the PAISANO project for the same purpose. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 66 meetings. The purpose is to determine what follow-up actions need to be taken and determine whether they need to re-define strategies used to address challenges or replicate positive actions. Up to the time of the MTE, the project had collected annual monitoring data through a participant survey measuring progress against annual DIP targets and indicators. The project had contracted a consulting firm that collected SO1 and SO2 data from participant families in 44 communities in the 3 project regions (Quiche, Huehuetenango, and Quetzaltenango). Due to budget constraints, the sample size was small. The annual surveys were discontinued after the MTE because FFP told the project that the data collected through routine monitoring in SAMI were sufficient. Subsequently, data included in the IPTT, annual DIP, and the SAPQ were completed through SAMI. The project conducted other kinds of monitoring activities, such as an SBCC survey of SO3 beneficiaries and non-beneficiaries to discover their opinions about project SBC activities. In 2016, PAISANO also conducted a post-food distribution survey among beneficiaries who had received food from the project. Results indicated that 39 percent said that in addition to food they also received nutritional support, 27 percent received agricultural assistance, and 25 percent participated in training activities. Despite its value, the MTE found that even though the M&E system was functional and had a clear structure as well as a defined process for data aggregation and reporting, it was underused, mainly because many of the staff did not understand how to use SAMI. To correct this and the fact that baseline findings had not been widely shared, the M&E staff conducted five presentations disseminating baseline findings and encouraging staff to reach non-participants as well as participants. It also held results review sessions with regional staff on the DIP and IPTT to strengthen the quality of quarterly data input and analysis and make sure the field staff understood how to use SAMI. This review process also strengthened the field teams’ ability to analyze data and interpret results for making programmatic decisions and follow-up. The evaluation team confirmed during interviews that many field respondents mentioned using the SAMI data during quarterly reviews with the M&E team to make decisions about how to follow up on activities. Some of the respondents said that initially they had problems with internet connectivity (also noted in the MTE), which slowed the data input to SAMI because it would require re-entering the data. There was an issue requiring the server to be operational during weekends so regional staff could input data that they did not have time to enter during weekdays. That was sorted out by Save the Children during the last half of the project. Some of the field coordinators, while noting that SAMI was useful for reporting, questioned whether the system was practical for making programmatic decisions. They used the example of tracking chronic malnutrition through growth monitoring sessions in which the levels fluctuated up and down from quarter to quarter. The problem was not analyzed, explaining that part of the issue was that the children measured varied during quarterly time periods. Some children dropped out and new ones were added, so it was hard to track the same child to determine whether his or her situation was improving. Staff suggested that monthly monitoring might have been more accurate. The evaluation team agrees with the MTE finding that the SAMI system is a best practice. The web-based system is flexible because it can be accessed anywhere and can quickly respond to user needs. As evidence of this, the project made several adjustments to SAMI when it introduced the cash transfer system in 2017. Save the Children contracted INSOFT, a Guatemalan software development company, to create specific programs to automate the cash transfer processes. This included training of M&E personnel who oversaw the SO field staff who managed the cash transfer process. The system has been used to support evidence-based management of project activities, and Save the Children has been promoting its utility to other organizations. According to project management staff, SESAN has adapted the program for its municipal programming, calling it SIMONE, and is demonstrating it to the mayor of Villanueva. USAID is also promoting it as a useful tool, and DHL has expressed interest in it as a logistics management tool. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 67 Conclusions The PAISANO M&E system is well coordinated with clearly defined responsibilities and structure for each project partner. The project has developed a well-organized process for data collection, aggregation, and reporting. The SAMI MIS system is considered a best practice due to its ease of use, versatility, and success in tracking project indicators. Although there are some reservations regarding how data are tracked for management decisions, most were satisfied with the M&E unit management and the utility of SAMI for tracking progress and making programming decisions. Recommendations FFP should promote the principle and process used by SAMI as an M&E tool with its other country projects. Future projects should ensure and adjust monitoring timelines to coincide with intervention activity dates. Final Performance Evaluation of the Food for Peace PAISANO Development Food Assistance Project in Guatemala – Final Evaluation Report 68 5. GENERAL CONCLUSION AND CHALLENGES The PAISANO project either met its targets or made good progress toward its key goals and objectives. Although most project staff and partners believed that the MTE recommendations were beneficial and on the right track to improve project implementation, the number of changes (cash transfers, WASH, gender plan, reorganizing participants through PPI criteria etc.) and the time required to implement them meant that many changes were not put into operation until the last year to year and a half of the project. Consequently, it is difficult to predict whether these changes and new practices will be sustained. Despite the important progress made by PAISANO, some important challenges remain in the Western Highlands:  The current MOH community health model does not include community volunteers, therefore, no mechanism is in place to train or continue support for volunteers developed by PAISANO. There is also no longer an active mechanism for community feedback on the quality of health services. In general, the government ministries lack the resources and capacity to deliver adequate services to meet demand (e.g., Ministerio de Salud Publica y Administracion Social [MSPAS], MAGA, SESAN).  The end of the cash assistance has been cited by project and participant respondents as a challenge for being able to afford the kinds of diverse foods that were accessed under PAISANO, even though they intend to continue prioritizing nutritious foods when they can afford them.  It is expected that water shortages for crop production will continue and become even more severe due to climate change. There is also an issue of poor water quality and resulting increases in diarrhea prevalence, which will require more investment in water and sanitation infrastructure.  EBF did not improve and appears to be decreasing in Guatemala. This will require further research and analysis and strategic project development.  There is an increasing prevalence of obesity and related chronic disease (e.g., diabetes) that will require more prevention, treatment and nutrition interventions.  To cope with the lack of employment and land resources, migration, including to the United States, is considered a viable option for food security.  Agriculture is likely to remain the most significant driver of economic development in the Western Highlands, but market-oriented opportunities are limited. Annex 1: Results Framework Goal: Food insecurity among vulnerable rural households in 3 departments of the Guatemala highlands reduced SO1: HOUSEHOLD ACCESS AND AVAILABILITY TO FOOD INCREASED IR 1.1: Household agricultural production diversified and increased IR 1.2: Women’s business and leadership opportunities increased IR 1.3: Access to markets increased SO2: MALNUTRITION AMONG GIRLS AND BOYS UNDER 5 YEARS REDUCED IR2.1: Use of quality MCHN preventive services increased IR 2.2: Use of improved health and nutrition practices at household level increased SO3: COMMUNITY RESILIENCE IMPROVED IR3.1: Community preparedness to manage and respond to adverse shocks increased IR 3.2: Community and municipality governance improved Cross-cutting SO: Status of women within target households and communities improved SO1: HOUSEHOLD ACCESS AND AVAILABILITY TO FOOD INCREASED IR1.1: Household agricultural production diversified and increased LR1.1.1: Use of improved agricultural production practices increased. - - - - - - - - - - Training of Technicians and community-based ag workers (LAs and PAs). Establishment of a LA certification program Pilot an entrepreneurial cost recovery scheme for LA service sustainability Gender and ag development training for staff and partners Farmer group development, including youth “4-S” groups Training in crop production, animal husbandry, horticulture and HH farming systems Pilot HH livelihood planning approach Establish Training and Reproduction Centers Farmer group training in appropriate harvest techniques and storage timing strategies Promotion of food drying and appropriate storage techniques IR1.2 Women’s business and leadership opportunities increased - - - Complete gender analysis at municipal level Leadership training of female association members Link CSSG female members to associations and cooperatives IR1.3: Access to markets increased LR1.3.1 Linkages between market actors with associations, communities and municipalities increased - - - - Complete value chain analysis on non-FtF cash crops Aggregation and processing centers (CAPs) established in each municipality Linkage to local, municipal and private stakeholders for market information Facilitate linkages with FtF and public and private sector stakeholders LR 1.3.2: : Marketing capacities of associations strengthened - Establish new associations or cooperatives - Governance and business training of associations - Facilitate formalization and business plans of associations - - - - - - - LR 1.1.2: Use of improved natural resources management practices increased Train farmer groups in agroforestry practices Link farmers to INAB conservation programs, PINFOR/PINPEP LR 1.1.3 Use of improved agricultural inputs increased Distribute seeds/vegetative materials Distribute small livestock Distribute irrigation and other ag production equipment Conduct seed fairs with/through COCODES Map input supply sources and disseminate findings to small holders LR 1.1.4 Access to savings and credit services increased - - Establish linkages between SO1 beneficiaries and MFIs banks Establish CSSG groups SO2: MALNUTRITION AMONG GIRLS AND BOYS UNDER 5 YEARS REDUCED IR2.1: Use of quality MCHN preventive services increased LR 2.1.1: Quality of community level MCHN services improved - - - - - - - - - Needs assessment of CHWs capacities Needs and resource assessment of health facilities capacities Training MOH facilities staff (men and women) Resource mapping to identify key stakeholders Training of CHWs (men and women) Linkages to/training of TBAs Support to CHWs to become self-sufficient extensionists Coordination meetings with MOH, COCODEs, COMUDES and other stakeholders Linkages made with CHNC and Scaling Up Nutrition (SUN) LR2.1.2: Referral systems between community and MOH services strengthened - - - Analysis of referral/counter referral system and develop/implementation strategy for strengthening using SYLVIA system. Linkage to and advocacy with other related MCHN programs Strengthen COCODEs to advocate for MCHN services IR2.2: Use of improved health and nutrition practices at household level increased LR 2.2.1: Knowledge of and attitudes towards key MCHN practices improved - - - - - - - - - - Distribute PM2A (1,000-day) ration GMP sessions and messaging for PM2A ration conditionalities Create household SBC improvement plans HPNs monitoring of SBC plan implementation progress SBC strategy development: Barrier Analysis and formative research on key practices; IEC materials adapted, as needed; findings/lessons shared with inter￾institutional working group Cross-training of SO1 and SO2 staff and community-level workers to promote effective delivery of key messages Nutrition schools/PD, cooking demos Form Peer Support Groups Form teen clubs for MCHN Community-wide FSN messaging via drama, song, etc. especially by adolescents LR 2.2.2 Knowledge of and attitudes towards household sanitation and hygiene practices improved - - Community education on Point of Use treatment SBC to promote key hygiene and household sanitation practices SO3: COMMUNITY RESILIENCE IMPROVED IR3.1: Community preparedness to manage and respond to adverse shocks increased LR 3.1.1: Community organization and planning for disaster risk-reduction improved - - - - - - - SILVYA: Implement Local Warning and Surveillance System Establish new and strengthen existing COMRED and COLRED for risk management and response Develop a list of core competencies with COMRED to certify “disaster resilient communities Implement community risk mapping and mitigation plans Facilitate the mapping of hazards and assets in each target municipality Facilitate COMUDE vulnerability profiles and DRR plans related to food insecurity and disasters within each of its communities Provide resources (technical support, supplies, and FFW) to support DRR plans LR3.1.2: Knowledge of key emergency response actions increased - - - - Complete Municipal resource map w/ COMUDES Facilitate development of economic development plans with COMUDEs Design and implement an awareness campaign on key threats associated with disasters Sentinel site reporting of key disaster indicators to MFEWS IR 3.2: Community and municipality governance improved LR 3.2.1: Governance and advocacy skills of COCODES/COMUDES strengthened - - Governance, leadership and gender awareness training for COCODE and COMUDE members Facilitate information sharing among COCODE and COMUDE to share better practices in governance Annex 2: Statement of Work April 27, 2018 1 Statement of Work Population-Based Final Evaluations of Food for Peace Development Food Assistance Projects SEGAMIL and PAISANO in Guatemala INTRODUCTION The final evaluation of the 2012 Guatemala Title II Food for Peace (FFP) projects is the second and final phase of a pre-post evaluation survey. The baseline study, conducted from April-June 2013, employed a mixed-method approach, and was designed to provide information on all four aspects of food security – availability, access, utilization and stability. The study investigated household food access, sanitation and hygiene, agriculture, household expenditures and assets, dietary diversity, and anthropometry among women and children. As with the baseline study, the Guatemala performance evaluation will also use a mixed-method approach, but will also utilize and integrated secondary data and project performance monitoring data. Methods will be chosen in order to generate the highest quality and the most credible and robust evidence possible to answer evaluation questions. BACKGROUND In fiscal year (FY) 2012, USAID/Food for Peace (FFP) awarded Catholic Relief Services (CRS) and Save the Children (SC) six-year Title II development food assistance programs (DFAP) in Guatemala with an overall investment of $79.3 million. The CRS Food Security Program Focused on the First 1,000 Days (SEGAMIL) aims to improve food security of 23,500 families in the San Marcos and Totonicapán departments, while the SC Western Program of Integrated Food Security and Food Actions (PAISANO) seeks to reduce the food insecurity of 26,500 families in Quiche, Quetzaltenango, and Huehuetenango departments. Both programs implement activities in three technical sectors – Agriculture/Natural Resource Management (Ag/NRM), Health/Nutrition (H/N), and Disaster Risk Reduction (DRR) – and have female empowerment as a cross-cutting focus. FFP partners implement in 18 of the 30 Feed the Future (FtF) priority areas in the Western Highlands region; PAISANO programs entirely within the zones of influence (ZOI), while approximately 60 percent of SEGAMIL’s programming is within the ZOI.1 CRS collaborates with the local non-governmental organization Caritas in San Marcos department and with ADIPO in Totonicapán department. They will improve food security among rural populations by (1) increasing food availability and access at the household level through sustainable agricultural production practices, sustainable natural resources management, diversification of income sources, and linkages between farmers’ groups and market processes; (2) reducing chronic malnutrition among children under five through activities such as improved health hygiene practices, nutrition and care practices, and increased household use of health services; and (3) strengthening local and municipal resilience systems through improved community and municipal management of risk and vulnerability to food insecurity. SC implements PAISANO in consortium with Project Concern International (PCI). SC is the prime and implements in Quiche and Quetzaltenango, where 20 percent of the communities participated in the prior multi-year assistance program PROMASA, while PCI implements in Huehuetenango. The program will reduce food insecurity among vulnerable rural populations by (1) increasing food 1 Santa Maria Chiquimula and Totonicapán in Totonicapán department and Comitancillo in San Marcos are not in the FtF zone. These communities were selected by CRS and ultimately approved by FFP based on extremely high levels of chronic malnutrition and poverty. April 27, 2018 2 access at the household (HH) level through increased and diversified agricultural production, fostering women’s business and leadership opportunities, and increasing access to markets; (2) reducing chronic malnutrition among children under five (CU5) by improving the use and quality of maternal and child health and nutrition care at the community level, and improved practices at the HH level; (3) improving resiliency at the community level via improved community and municipal governance and disaster response. Both projects are preceded by FFP-funded multi-year assistance programs (MYAPS) - SEGAMAYA (CRS) and PROMASA II (SC) and Asociacion Share de Guatemala (SHARE) - implemented in the Western Highlands from 2006-2011. EVALUATION PURPOSE & OBJECTIVES The overarching purpose of the final performance evaluation is to measure the development of outcomes of the SEGAMIL and PAISANO projects. Findings from the final evaluation will be used to determine the performance of the two DFAPs. Additionally, findings will help to inform the selection, design, and implementation of future USAID projects in the Western Highlands of Guatemala. EVALUATION QUESTIONS In the table below, FFP has identified the evaluation questions and provided the fundamental elements that should shape the Evaluation Team’s (ET) research. It is anticipated that the ET will use these, but it is not limited to working solely within this guidance. Q1: To what extent have the projects met their defined goals, purposes and outcomes? The ET will evaluate the contribution of SEGAMIL and PAISANO to USAID’s efforts to reduce food insecurity among vulnerable rural populations in five departments of the Western Highlands of Guatemala.2 The ET will support its determination to using both quantitative and qualitative methodologies when discussing the following: (1) project performance on indicators against targets set by both the partners and the key FFP indicators* Depth of Poverty, Stunting, and Underweight; (2) factors that promoted or inhibited the achievement of the project objectives, including, but not limited to the effectiveness of food-for-asset and/or cash-for-asset interventions; (3) plausibility of pathways and the determinants of achieving the key outcomes; (4) targeting strategies and their contribution to achieving project goals (especially with regard to gender and reaching the most vulnerable); (5) the appropriateness and effectiveness of interventions on the poorest individuals. * FFP’s global targets are: a minimum of 2 to 2.5 percentage point annual reduction of prevalence of stunting, a minimum of 3 to 4 percentage point annual reduction of prevalence of underweight, and a minimum of 4 percentage point annual reduction of depth-of-poverty. Q2: Based on the evidence, which project outcomes are likely to be sustained? The ET will evaluate the functionality and performance of systems and services established independently by the projects, as well as in collaboration with the private sector, Government of Guatemala, and non-governmental organizations and academic organizations to achieve project outcomes and sustainability. It will support its determination using both quantitative and qualitative methodologies that explore the following: (1) the quality of the processes, systems, and institutional arrangements developed and/or strengthened to sustain the necessary and critical services; (2) communities’ perceptions on the quality, frequency, effectiveness, and sustainability of the services provided by the project; (3) the likelihood that service providers will continue providing services after the project ends; (4) the motivation of the community and 2 2012 Food for Peace Country Specific Information: Guatemala http://pdf.usaid.gov/pdf_docs/pdacu283.pdf April 27, 2018 3 beneficiaries to demand and pay (or invest time) for the services; (5) whether the necessary resources and capacity strengthening will exist to sustain service providers; (6) the extent to which the projects leveraged other USG and non-USG investments to achieve sustained outcomes as identified in the theories of change; (7) evidence of enhanced linkages with other service providers. Q3: In each technical sector, what are the strengths of and challenges to the effectiveness of the interventions’ implementation and their acceptance in the target communities? The ET will evaluate the effectiveness and relevance of the technical interventions, including food-for-asset and/or cash-for-asset interventions, to achieve project outcomes, and discuss those findings in relation to the projects’ theories of change. It will support its determination using both quantitative and qualitative methodologies when discussing the following: (1) factors in the implementation and context associated with greater or lesser effectiveness in producing Outputs of higher or lower quality; (2) the interventions and implementation processes deemed more/less acceptable to members of the target communities. Q4: What key lessons learned and best practices should inform future projects in the country? During the course of its research, the ET should identify best practices, strengths, and challenges in the projects’ designs (including theories of change), supporting project achievements, as well as approaches that should be considered in designing future food and nutrition security projects and strengthening household and communities’ resilience capacities. The ET will support its determination using both quantitative and qualitative methodologies when discussing the following: (1) the unintended positive and/or negative consequences of the projects; (2) ways to minimize potential unintended negative consequences and systematically capture positive consequences. AUDIENCE & INTENDED USES The primary audience of the evaluation reports will be CRS and SC (and their sub partners), and USAID (FFP/Washington, USAID/Guatemala). The reports will also be shared with the Government of Guatemala. USAID will make extensive use of findings from the evaluation to make different presentations and bulletins as part of a wider dissemination of best practices and lessons learned. The evaluation recommendations may be used by FFP to refine proposal guidelines and project policy. PERFORMANCE EVALUATION METHODOLOGY The final evaluation will use a mixed-methods approach. The ET will begin with a desk review of project documents, validate its understanding of the projects via consultations with USAID and implementing partners, conduct a population-based household survey using all implementation villages as the sampling frame, and lead a purposively sampled qualitative study. It is preferred that, if possible, the quantitative and qualitative components are conducted sequentially so as to allow the quantitative data to inform the qualitative research. a) Desk Review The evaluation team should review the following documents to contextualize and refine the evaluation questions, as well as to gain an in-depth understanding about the project design, implementation, and the food security situation in the area. Partner annual monitoring data should be reviewed when preparing for qualitative research, considered in relation to the evaluation findings, and incorporated into the report in support of/contrast to select evaluation findings. While FFP recommends the below documents for pre-evaluation learning, the literature review should not be limited to the following: April 27, 2018 4 a) Project proposals b) Pipeline Resource Estimate Proposals (PREPs) c) Annual results reports (ARR), including Indicator Performance Tracking Tables (IPTT) for performance against targets d) Midterm review reports and corresponding action plans developed by the two projects e) Baseline Study for the Title II Development Food Assistance Projects in Guatemala, 2013 f) Guatemala Demographic Health Survey 2014/2015 g) 2015 Nutrition, Mortality, Food Security, and Livelihoods Survey (based on SMART methodology) h) Partner formative research and barrier analyses so as to better understand the context and if/how the studies influenced programming i) Monitoring data and reports j) Sustainability and Exit Strategy plans b) Quantitative Endline Survey FFP strongly prefers that tablets are used for quantitative data collection (e.g., computer-assisted personal interviews (CAPI)). The quantitative endline household survey will collect data on the same population-level impact and outcome indicators (presented below) that were collected during the baseline survey. Baseline data were collected between April 1 and June 30, and data collection for the final evaluation must occur during the same time period. The ET should use the same data collection instruments and parameters as the baseline study (level of statistical precision (95 percent confidence intervals), statistical power (80 percent), 6 percentage point reduction to estimate the sample size, Design Effect of 2, etc.)3. The quantitative endline survey design does not need to be identical to the baseline, however; if the projects reduced their target areas, for example, the sampling frame of 6,000 households used for the baseline will need to be adjusted. Data will be collected to support the calculation of the following indicators: 1 Prevalence of underweight children under five years of age 2 Prevalence of Poverty: Percent of people living on less than $1.25/day* 3 Mean Depth of Poverty* 4 Per capita expenditures (as a proxy for income) of USG-targeted beneficiaries* 5 Prevalence of stunted children under five years of age 6 Prevalence of underweight women (of reproductive age)* 7 Percentage of farmers who used at least [a project-defined minimum number of] sustainable agriculture (crop/livestock and/or NRM) practices and/or technologies in the past 12 months 8 Percentage of farmers who used improved storage practices in the past 12 months 9 Percentage of farmers who used financial services (savings, agricultural credit, and/or agricultural insurance) in the past 12 months 10 Percentage of farmers who practices the value chain activities promoted by the project in the past 12 months 11 Prevalence of households with moderate or severe hunger (Household Hunger Scale - HHS) 12 Average Household Dietary Diversity Score (HDDS) 13 Prevalence of children 6-23 months receiving a minimum acceptable diet (MAD) 3 Baseline Study for the Title II Development Food Assistance Projects in Guatemala http://pdf.usaid.gov/pdf_docs/pnaed236.pdf April 27, 2018 5 14 Women’s Dietary Diversity (Score): Mean number of food groups consumed by women of reproductive age (WDDS)* 15 Prevalence of exclusive breastfeeding of children under 6 months of age 16 Percentage of children under age five who had diarrhea in the prior two weeks* 17 Percent of children under five years old with diarrhea treated with Oral Rehydration Therapy (ORT)* 18 Percent of households using an improved drinking water source 19 Percent of households using an improved sanitation facility 20 Percent of households with soap and water at a handwashing station commonly used by family members 21 Percentage of pregnant women Seeking Health Services when danger signs are present 22 Percentage of mothers or caretakers Seeking Health Services when danger signs are present (childhood) 23 Percentage of mothers and caretakers seeking health services when neonatal danger signs are present 24 Percentage of pregnant women that recognize at least 2 pregnancy danger signs that indicate need of seeking Health Services 25 Percentage of women that recognize at least 2 neonatal (<28 days) danger signs that indicate need of seeking Health Services 26 Percentage of mothers that recognize at least 2 childhood illness signs that indicate need of seeking Health Services 27 Percentage of mothers of children 0-59 months reporting that she makes decisions either by herself or jointly with her partner regarding seeking health services for themselves 28 Percentage of mothers of children 0-59 months reporting that she makes decisions either by herself or jointly with her partner regarding seeking health services for their children’s health 29 Food Insecurity Experience Scale (FIES (12 month and 30 day recall)) *Partners were not required to set targets for contextual indicators. Note: Based on USAID’s strategic interests, USAID will add a few additional questions to predict sustainability and test poverty probability index (PPI) into the household questionnaire (and any corresponding indicators added). All quantitative data is owned by USAID and made available to the public barring rare exceptions. c) Consultations As a supplement to the desk review, and in preparation for qualitative research, consultations with FFP and partner staff in Washington, DC and Guatemala will allow the ET to corroborate its understanding of the design, approaches and interventions employed by each DFAP and acquired through the desk review. It is recommended that the ET engage with the staff at each organization prior to beginning fieldwork. Equally important to engaging pre-data collection is to reconnect post-data collection to “ground truth” findings with FFP/Guatemala and the partner staff. In the case of major disagreements, the program staff should provide evidence in support of the argument, and pending time constraints, the ET may revisit the field. d) Qualitative Study Qualitative methods will be used to collect information to answer evaluation questions and also to support the interpretation of the quantitative data. The ET will design the overall qualitative study approach and should consider a variety of primary data collection methods, such as semi-structured in-depth interviews, group discussions, key informant interviews, direct observations, and case studies (the ET may choose to use the most significant change methodology to identify a selective set of case studies). These methods - to the maximum extent possible - will ensure that if a different, April 27, 2018 6 well-qualified evaluator were to undertake the same evaluation, he or she would arrive at the same or similar findings and conclusions. The ET should decide on specific methodologies before traveling to Guatemala, and present those to FFP along with the number of interviews, FGDs, etc., per project, in the inception report. Following discussion and agreement, the ET will finalize the methods during the team meeting in-country. The evaluation team leader and members will be responsible for interviewing the direct, indirect and non-beneficiaries in their households and communities, as well as look for evidence of ongoing learning and activities (such as home gardens, etc.). The ET will also be responsible for interviewing relevant stakeholders for the evaluation and analyzing the qualitative data. Should the ET decide to hire additional researchers to complement the data collection effort, they cannot replace the evaluation team members’ role of collecting primary data using qualitative methods. Note: the intervention areas have large Mayan populations; the evaluators will need to take special care to communicate using the local language and in a culturally sensitive way to ensure the most accurate responses. The ET will contribute to the interpretation of the quantitative results using qualitative findings. In addition to the factors specifically identified earlier as essential to responding to the evaluation questions, during its research during the qualitative study the ET should also consider the efficacy of the following cross-cutting interests: project management; performance monitoring; strategies to improve gender equality at the participant and project management levels; environmental considerations; and conflict sensitivity (i.e., do no harm). Lastly, it is expected that the qualitative study will speak to lessons learned and best practices. The ET may find it useful to apply non-probability sampling methods to select a sub set of enumeration areas from the quantitative survey. In selecting interview sites, the evaluation team should strategically select large-enough-yet-manageable interview sites that generally represent the target area. As with the quantitative household survey, qualitative sampling should include both individuals who directly participated in the DFAP (beneficiaries) and those not specifically targeted with any intervention (non-beneficiaries). In addition, the qualitative team should interview USAID personnel, project staff, host Government officials, local government staff (including, but not limited to, municipalities, ministries, local and central authorities), community leaders and other knowledgeable people in the communities, community development councils (COCODES), and other agencies and individuals as appropriate. e) Data Analysis and Interpretation The evaluation team will statistically compare the endline data for each of the two strata (PAISANO and SEGAMIL) with that of the baseline for that stratum, and also for the overall country level, in order to detect changes (if any) for all key indicators. The ET will conduct descriptive and inferential analyses to describe the results, as well as various econometric analyses to identify the determinants of key outcomes and the magnitude and direction of changes. In advance of fieldwork, the evaluation team should develop a data analysis plan and submit to the COR for approval. When analyzing the data, however, the evaluation team should not limit itself to the data analysis plan; rather, the evaluation team should keep an open and curious mind to look for correlations between variables. In presenting the analysis, the evaluation team should be cognizant about the readers’ familiarity with the statistical presentation. FFP suggests avoiding jargons, but rather describe the statistical terms in a common language. Interpreting the results is as critical as the analysis. Oftentimes, it can be difficult for a reader to fully understand the key points and utility of the findings conveyed in a report. The analysis and April 27, 2018 7 interpretation should be presented in a “story telling format” so that the readers can see a human face as they read the report. While it is important for the reader to understand whether level of stunting is reduced in the area, it is equally important to understand the pathway; for example, how learning derived from project participation influenced people’s practices, which in turn resulted in positive changes in food security outcomes at the household and/or community level. Similarly, it is equally important for the readers to know some of the challenges participants faced that might have prevented them from reaping the full benefits of the projects. REPORT The ET will produce two reports in English, not to exceed 60-pages, for each DFAP. The draft reports will be shared within FFP, the USAID/Guatemala, and the partners for review and comment over a two-week period. The final report should include a table of contents, table of figures (as appropriate), acronyms, executive summary, introduction, purpose of the evaluation, research design and methodology, limitations, findings, conclusions, lessons learned, and recommendations. All evaluation questions should be answered, and the evaluation methodology should be explained in detail. To ensure a high quality deliverable, the reports should reflect a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the project, what did not, and why. Where noteworthy, the discussion should highlight and discuss the outcomes and impacts on males versus females. The report must integrate the quantitative analysis from the PBS and include a statistical comparison between baseline and endline results. Learning from the qualitative research will help to contextualize and interpret the quantitative data. The ET should also draw from partners’ annual monitoring data, where noteworthy, to substantiate and/or compare with evaluation findings. For example, if a qualitative finding is that beneficiaries report increased yields, the ET should compare/contrast that finding against data collected by the partners. Findings should be specific, concise, and supported by strong quantitative and/or qualitative evidence, and presented as analyzed facts/evidence/data, and not be based on anecdotes, hearsay or a compilation of people’s opinions. It should include analytical methods to include appropriate tests of differences; econometric analysis to evaluate the theories of changes and to explore the causal relation between the outcome and activities/variables based on the theoretical models; it is expected that the contractor will interpret the analytical findings. The report should disclose limitations to the evaluation, with particular attention to the limitations associated with the evaluation methodology, e.g., selection bias, recall bias, unobservable differences between comparator groups, etc. Recommendations should be supported by a specific set of findings, and be action-oriented, practical, and specific. It is expected that the final reports will address and incorporate feedback, as appropriate, from the reviewers. Should the ET disagree with any of the comments, it should raise this with the COR immediately for discussion. EVALUATION TEAM The Senior Evaluation Specialist will be responsible for designing and managing the evaluations and overseeing the work of the evaluation team members; coordinating with the implementing partners, USAID Mission and other stakeholders; coordinating with the endline PBS team; analyzing the findings and ensuring the quality of the report. As this is a mixed-method performance evaluation, in addition to the evaluation team, the endline survey will require extensive participation of the following personnel: Senior Survey Method Specialist, Data Analyst, Survey Coordinator, Data April 27, 2018 8 Management Specialist, Anthropometry Specialist, Country Operations Manager, and Survey Monitors. The PBS data collection team should be hired locally, if at all possible. The qualitative study will require in-country participation from the Evaluation Team Leaders and up to three subject matter specialists. As FFP projects are multi-sectoral, the evaluation team must possess expertise and field experience with food security and rural integrated programming, and demonstrate an in-depth knowledge of the following technical sectors and cross-cutting areas: agriculture and off farm livelihoods; nutrition; water, sanitation, and hygiene (WASH); gender, youth, resilience, and disaster risk management. The subject matter specialists must also possess experience and knowledge about the specific processes used by the projects (e.g., Care Groups, Farmer Field Schools, etc.). The qualitative study should also, to the extent possible and appropriate, include local representation. FIELD LOGISTICS The ET is responsible to arrange and pay for all logistics including anthropometric equipment, and transportation. It is strongly recommended that the ET consults the Mission on identifying a reputable car-rental firm. The Mission and partners should also be consulted on identifying interpretation services. The ET should request assistance from the partners on making introductions, as necessary, to local ministry representatives and community leaders. DELIVERABLES The ET shall produce the following deliverables during the evaluation and submit to the Contracting Officer’s Representative (COR). All draft documents should be submitted in Microsoft Word or Microsoft Excel, or in the rare occasion both .pdf and Word/Excel. The COR must review and approve all deliverables. Deliverable Timeline Work Plan • includes a brief synthesis and timeline for the Guatemala final evaluation, with the timeline including major activities throughout the study, including dates by which field guides and training materials will be completed. TBD PBS Enumerator Guide, Supervisor Manual, and Anthropometry Guide • provide revised detailed instructions on supervisor, enumerator and anthropometry trainings. Note that the PBS should use the supervisor, enumerator and anthropometry training guides developed for the baseline. Minor adjustments will be needed to accommodate the new indicators. TBD PBS Data Treatment and Analysis Plan • details how the data will be cleaned, weighted, and analyzed and must include: programming specifications and editing rules for cleaning data, data dictionary codebook, SPSS syntax and output for all data analyses and variable transformations into indicators; and • includes a descriptive, inferential, and econometric analyses plan. TBD PE Inception Report (~20 pages) • briefly synthesizes the literature review; • describes the qualitative evaluation methodology (including evaluation questions contextualized based on the literature review, sample site selection strategy and number of sites to be selected, number of interviews/discussions per project types of interviewees) and how qualitative data will be analyzed and coded; • introduces the evaluation team members and their roles; and TBD April 27, 2018 9 Deliverable Timeline • details how the qualitative information will be analyzed and integrated with quantitative. Final Evaluation Protocol (a combined protocol that discusses both the quantitative and qualitative components) (~30 pages) • describes the evaluation questions to be answered; • presents specific data collection methods by evaluation question; • identifies indicators to be collected; • introduces the local partner (data collection firm); • discusses the quantitative and qualitative analysis methods and plan; • presents PBS sample size, design and plan, survey design, questionnaire design, site selection plan for qualitative research; and • presents the fieldwork plan (including trainings and field support/supervision, data management, quality control, recording, analysis and reporting). TBD Pertinent Permissions and Approvals • demonstrate official approval from all relevant institutional review boards and from host country institutions to collect data, conduct the evaluation, and release data and reports, as required, as well as a statement affirming adherence to all requirements specified in USAID’s Scientific Research Policy. TBD PBS Quantitative Survey and Qualitative Instruments • include both English and Ngakarimojong versions of the household survey (note: the instrument must be back-translated to English via a second translator to ensure accurate translation. Following the pilot of the survey, any modifications based on field experience will again require translation and back translation to ensure accuracy). • describe the site selection methodology and factors used to select sample communities; • list sample communities; • discusses groups to be interviewed; and • explain criteria used to select respondents. TBD [preliminary estimates submitted in advance to support qualitative research] Draft Evaluation Reports (one for PAISANO and one for SEGAMIL) • will be no more than 50 pages per DFAP; • include as an annex tables for FFP indicator estimates, including comparisons between baseline/endline values. October 2018 In-country Briefings to USAID Guatemala and FFP Stakeholders in Guatemala • Two 60-minute presentations of the major findings of the baseline study to provide an opportunity for immediate stakeholder feedback that can be considered for revision (as appropriate and without compromising the validity or independence of the evaluation): o One presentation to USAID/Guatemala; o One presentation to FFP stakeholders in Guatemala, including the DFSA partners, donors, and Government of Guatemala (invited by USAID/Guatemala and the partners), via a 60-minute PowerPoint presentation. November 2018 Briefing to FFP/Washington • comprises a 60-minute PowerPoint presentation on the evaluation, major findings, lessons learned, and recommendations. November 2018 April 27, 2018 10 Deliverable Timeline Final Evaluation Reports • include items identified in the draft report as well as a three- to five-page executive summary of the purpose, background of the project, methods, findings, conclusions and recommendations, and the following annexes: the scope of work, tools used in conducting the evaluation (questionnaires, checklists, and discussion guides), and any substantially dissenting views by any Team member, USAID or the PVOs on any of the findings or recommendations; and • must be 508 compliant and uploaded to the Development Clearinghouse following COR approval. December 2018 Data (to be submitted at the time of the final report*) • include a separate electronic file of all quantitative data in an easily readable format that is organized and fully documented so as to facilitate use by those not fully familiar with the project or the evaluation; • provides cleaned data, sampling weights at each stage, final sampling weights, and all derived indicators; • includes a second final data set in CSV format that has been anonymized to protect individual confidentiality for use as a public data file in the USAID Open Data; and • include a separate file detailing GPS coordinates of households that participated in the PBS. *FFP may request data sets earlier for internal use only December 2018 Annex 3: Evaluation Protocol D Final Performance Evaluation of the Western Program for Integrated Food Security and Food Actions (Programa de Acciones Integradas de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) Guatemala Evaluation Protocol – FINAL Office of Food for Peace Contract #: GS-00F-189CA/7200AA18M00002 September 21, 2018 This publication was produced for review by the U.S. Agency for International Development. It was prepared by ICF Macro, Inc. Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol ii ACRONYMS COCODE Community Development Council (Consejo Comunitario de Desarrollo) COMUDE Municipal Development Committees (Committees Comités de desarrollo municipal CRS Catholic Relief Services DFAP Development Food Assistance Program EQ evaluation question FFP Office of Food for Peace FGD focus group discussion FY fiscal year IDI in-depth interview IP implementing partner IPTT Indicator Performance Tracking Table KII key informant interview MCHN maternal and child health and nutrition MTE mid-term evaluation NGO nongovernmental organization PAISANO Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integradas de Seguridad Alimentaria y Nutricional del Occidente) PBS population-based survey PCI Project Concern International PE performance evaluation PREP Pipeline Resource Estimate Proposal SEGAMIL Seguridad Alimentaria Enfocada en los Primeros 1,000 dias SO strategic objective USAID United States Agency for International Development USG U.S. Government WASH water, sanitation and hygiene Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol iii TABLE OF CONTENTS Acronyms............................................................................................................................................................................. ii 1. Background and Purpose of the Evaluation ....................................................................................................... 1 2. Research Questions and Sub-questions .............................................................................................................. 2 2.1 Evaluation Questions ...................................................................................................................................... 2 3. Methodology .............................................................................................................................................................. 7 3.1 Evaluation Design ............................................................................................................................................ 7 3.1.1 Overall Design .................................................................................................................................... 7 3.1.2 Overview of Data Sources .............................................................................................................. 8 3.2 Qualitative Data Collection Methods ....................................................................................................... 10 3.2.1 Types of Respondents .................................................................................................................... 10 3.2.2 Site and Respondent Selection ..................................................................................................... 11 3.2.3 Field Processes ................................................................................................................................. 14 3.2.4 Data Management Processes ........................................................................................................ 15 3.2.5 Informed Consent Procedures and Ethical Considerations .................................................. 17 3.2.6 Training Interpreters, including on Human Subjects Protocols ............................................ 17 3.2 Data Analysis .................................................................................................................................................. 18 3.2.1 Qualitative Data ............................................................................................................................... 18 3.2.2 Integration of PBS Data .................................................................................................................. 19 3.2.3 Integration of Findings from Document Review ...................................................................... 19 3.2.4 Proposed Report Outline .............................................................................................................. 20 Annex A: Qualitative Design Matrix ........................................................................................................................... 21 Annex B: Protocol Tools ............................................................................................................................................... 31 Annex C: Selected Sites ................................................................................................................................................. 80 Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 1 1. BACKGROUND AND PURPOSE OF THE EVALUATION The final performance evaluation (PE) of the 2012 Title II Office of Food for Peace (FFP) Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integradas de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) project, which is implemented by Save the Children, is the final phase of the project evaluation process. A baseline study was conducted between April and June 2013, and a population-based survey (PBS) was conducted at endline between June and July 2018. The baseline study used a mixed method approach including a PBS and qualitative data collection to provide information on all aspects of food security, including availability, access, use, and stability. The final evaluation includes a qualitative study to further understand the PBS endline results and related evaluation questions. This document is a protocol for the methodology to be used for the PE. Save the Children was awarded the PAISANO project by the United States Agency for International Development (USAID)/FFP in August 2012. The approximately $48 million program is implemented by Save the Children in partnership with Project Concern International (PCI). Save the Children was building on its experience of more than 10 years of implementing food security programs in Guatemala’s Western Highlands, and PCI had 37 years of implementing programs in the Western Highlands. The PAISANO project was conceived to build on lessons learned and strategies that had been successful in previous Save the Children and PCI projects. PAISANO’s goal has been to reduce food and nutrition insecurity among 26,500 vulnerable rural households in 13 municipalities in 3 departments of the Guatemala Western Highlands (Save the Children in 8 municipalities in Quetzaltenango and Quiche, and PCI in 5 municipalities in Huehuetenango). The project targets rural communities suffering from food insecurity and at risk of food insecurity, and communities with a high prevalence of chronic malnutrition. To achieve its goal, the project aimed to increase household access to food, reduce malnutrition among children under five years of age, and improve community resilience. The project also worked to ensure gender equality, mitigate environmental consequences, and influence behavior change. In fiscal year (FY) 2015, during its third year of the program, PAISANO underwent a mid-term evaluation (MTE). The MTE was led by FFP and conducted alongside the Catholic Relief Services (CRS) program, Seguridad Alimentaria Enfocada en los Primeros 1,000 Dias (SEGAMIL). The MTE report made several recommendations for the program. Among the most significant strategic objective 1 (SO1) and SO2 recommendations were the following: refining the beneficiary targeting strategy to better tailor and differentiate interventions to poor and extremely poor households; increasing consistency and quality of support; increasing linkages to agricultural value chains; placing a stronger emphasis on adult education and social and behavior change communication techniques; implementation of graduation elements and sustainability; formally incorporating water, sanitation, and hygiene (WASH) activities; and improving links with the country’s health systems. In SO3, the recommendations were focused on emphasizing work with municipal and community coordinating bodies, such as Municipal Development Committees (Committees Comités de desarrollo municipal [COMUDEs]) and Community Development Councils (Consejo Comunitario de Desarrollo [COCODEs]) for disaster risk reduction to improve governance and resilience, as well as better integrating youth into the program. Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 2 During FY 2016 and FY 2017, PAISANO made a number of adjustments to its program in response to the recommendations. It collaborated with the USAID Guatemala Mission, the Western Highlands Integrated Program, and other USAID projects on water, in particular regarding standardizing messages and revising materials for safe water consumption and water chlorination. In 2017, PAISANO rolled out a cash transfer program reaching 7,500 households. During both years, PAISANO improved access to nutritious foods through improved bean production, small livestock, and home gardens. To promote sustainability, the program coordinated with the Government of Guatemala and nongovernmental organizations (NGOs)1 to promote continued access to financial support for community projects. This review will focus on the PAISANO project from 2013 to 2018, noting its progress up to and after the MTE as well as any changes in interventions and activities. 2. RESEARCH QUESTIONS AND SUB-QUESTIONS 2.1 Evaluation Questions This section provides the evaluation questions (EQs) based on the scope of work for the final evaluation of the FFP Development Food Assistance Programs (DFAPs) in Guatemala. Sub-questions are included to help further contextualize the understanding of project achievements, strengths and weaknesses in project design and implementation, as well as lessons learned and best practices. Annex A provides the evaluation matrix with details on data sources and analytical approaches that will be used to addressed each question. Q1: To what extent has PAISANO met its defined goals, purposes, and outcomes? The evaluation team will assess: (1.1) Change in indicator values between baseline and endline and project performance on final indicator values against targets set by both the partners and the key FFP indicators that include poverty, stunting, and underweight. FFP’s global targets are: a minimum of 2 to 2.5 percentage point annual reduction of prevalence of stunting, a minimum of 3 to 4 percentage point annual reduction of prevalence of underweight, and a minimum of 4 percentage point annual reduction of depth-of-poverty. Sub-questions: a. As PAISANO transitioned from food assistance to cash assistance after the MTE, what evidence exists as to whether food or cash assistance was more effective for meeting objectives related to improved children’s diets and investment in productive assets? What were the lessons learned that can inform future food or cash assistance programming in Guatemala and beyond? b. What factors have inhibited or promoted real progress toward the achievement of targets for reducing the high rate of childhood malnutrition? Given a volatile political context, worsening economy, climate stresses, and decreases in government health 1 Municipalities; Municipal Development Committee; Municipal Commission for Health, Food and Nutrition Security (Comisión Municipal de Salud, Seguridad Alimentaria y Nutricional); Ministry of Health; Ministry of Agriculture (MAGA); Ministerio de Educacion (MINEDUC); Secretaría de Seguridad Alimentaria y Nutricional; CARITAS (charity); ECOFILTRO; Agua del Pueblo; Liga de la Leche Guatemala; Engineers without Borders; Escuela de Formacion Agricola - a MAGA academic center; Communities Leading Development project; Area Directorate of Health (Dirección del Area de Salud); among others. Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 3 funding and staffing, is there evidence that PAISANO’s interventions mitigated food insecurity for beneficiaries or non-beneficiaries? c. How effectively did PAISANO respond to these factors beyond its control to reduce their negative effects and pursue available opportunities? d. Which behavior change initiatives and activities have had the greatest impact on affecting change in nutritional status of pregnant and lactating women and children under two years of age? (1.2) Factors that promoted or inhibited the achievement of the project objectives, including, but not limited to, the effectiveness of food-for-asset and/or cash-for-asset interventions Sub-questions: a. Based on evidence, what are the plausible pathways and determinants for achievement of key outcomes based on project activities? b. What efforts were made in the design and implementation of the project to minimize women’s time burden for participation across the SOs? Was it sufficient? (1.3) Targeting strategies and their contribution to achieving project goals (especially with regard to gender and reaching the most vulnerable) Sub-questions: a. How do outcomes and targeting strategies vary across key beneficiary sub-groups prioritized by USAID (poorest households, women, children under five)? b. Why did many of the child nutrition and care-seeking behavior indicators (in addition to the household hunger score) not change, although expenditures as a proxy for income increased quite a bit? What did people spend additional income on? Did the household hunger score improve in households with adult female, no adult male but not in households with adult male and female because of specific differences in intra-household decision-making on expenditures, and if so, what differences? c. How effective was the project in targeting women and targeting the poorest, most vulnerable households? d. How effective was the application of poverty criteria to screen recipients with the highest need (implemented after the MTE)? Were criteria transparently applied? Were there negative effects to PAISANO’s overall effectiveness or child nutrition specifically? (1.4) Based on evidence from sub-questions 1–4, what has been the contribution of PAISANO to USAID’s efforts to reduce food insecurity among vulnerable rural populations in five departments of the Western Highlands of Guatemala? Q2: Based on the evidence, which project outcomes are likely to be sustained? The evaluation team will assess: (2.1) Quality and functionality of the processes, systems, and institutional arrangements developed and/or strengthened to sustain the necessary and critical services Sub-questions: a. What services are being provided and what fee-for-service models have been adopted by those trained as service providers in the agricultural sector? What is the evidence that providers are currently making a profit? What is the evidence that they will be able to access business development services to improve their business skills? What evidence is Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 4 there that providers are able to analyze market demand and grow their clienteles over time? b. What is the likelihood of private sector service providers that PAISANO has worked with continuing to provide services after the project has ended? c. How are service providers functioning, what extension model (fee for service, or other) will they use; is the business model economically viable? What capacity building support do they have? Are they linked to the government agriculture extension system? d. What assets constructed or repaired with food-for-asset efforts are deemed most valuable by community members? What is the evidence that communities will have the motivation and financial resources to maintain the assets? e. After subsidies have ended for mother leaders, do they intend to play a role in supporting optimal child health and nutrition practices? What continued role will nutrition promoters play? What financial or non-financial incentives will continue to motivate volunteers? f. Through what mechanisms will the referral system between the community and health facilities continue to function for acutely malnourished children and children/pregnant women with danger signs? What community groups/governmental systems/non￾governmental entities will support referral and counter-referral? g. Through what mechanisms, will elements of PAISANO’s social and behavior change promotion activities continue through ongoing community activities, Ministry of Health promotion, or other entities? h. What community-level system will continue to monitor and provide feedback on health services to ensure quality improvement and access for the community? (2.2) Communities’ perceptions on the quality, frequency, effectiveness, and sustainability of the services provided by the project Sub-questions: a. What are the perceptions of communities, particularly among mothers and fathers, on the effectiveness of PAISANO’s different approaches to promoting improved child health and nutrition? Which activities were considered most useful and to what end? b. Given end of PAISANO’s cash assistance, to what extent do project participants indicate an intent and ability to continue to diversify their diets (especially those of women and children)? If so, is this based on prioritizing spending (behavior change) or due to improved availability or access to diverse foods? (2.3) Motivation of the community and participants to demand and pay (or invest time) for the services (2.4) Whether the necessary resources and capacity strengthening will exist to sustain service providers (2.5) Extent to which the projects leveraged other U.S. Government (USG) and non-USG investments to achieve sustained outcomes as identified in the theories of change Sub-question: a. How have relationships with USG actors at national, regional, municipal, or local levels mutually contributed to their effectiveness? (of PAISANO and other actors) Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 5 (2.6) Evidence of enhanced linkages with other service providers Q3: In each technical sector, what are the strengths of and challenges of the effectiveness of the interventions’ implementation? The evaluation team will evaluate: (3.1) The effectiveness and relevance of the technical interventions, including food-for asset and cash-for-asset interventions, to achieve project outputs of greater or lesser quality, and discuss those findings in relation to the project’s theory of change. Sub-questions: a. In each of the project’s technical sectors (maternal and child health and nutrition; agriculture/livelihoods; and early warning systems/disaster risk/resilience), what were the most effective implementation methods and approaches? b. Did investments in improved availability of diverse foods (agriculture and animal raising) translate into improved diets for maternal and child health and nutrition populations? What were the most successful activities and products in terms of how well they were integrated into these diets? c. Are there specific WASH interventions and approaches that can be linked to decreases (or show great potential) that were achieved in reducing malnutrition among children under five? d. Have interventions across each SO changed community norms or intra-family dynamics in any positive or negative ways? What are the implications for each sector? For SO2 specifically, has gender training resulted in shared decision-making around spending on health and nutrition, particularly in the selection of diverse foods for women’s and children’s diets? What evidence is there to support this? e. Was it an effective strategy to use small service fees for food or cash assistance as subsidies for agriculture and nutrition volunteers? What were the advantages and disadvantages of this strategy, and what are the lessons learned? f. How did the end of eligibility for food or cash assistance (due to aging out, not meeting poverty criteria) impact project participants’ involvement in PAISANO’s activities? How did PAISANO manage this challenge and was it effective for keeping past recipients active in the project? (3.2) Factors in the implementation approach and methods, and the context associated with greater or lesser effectiveness in producing outputs of higher or lower quality Sub-questions: a. Who was targeted by and benefited from each project’s intervention activities, and how effective were the selected targeting approaches in achieving their respective goals? Did the targets change during the project period? b. How has the discontinuation of food rations provided by PAISANO affected the health and nutrition status of pregnant women and children under two? c. What were the implications of the transition from food to cash assistance in terms of the acceptability and perceived value of the transfer to beneficiaries? How effective has the creation of gender-based groups (women’s savings groups, mother care groups) been in promoting gender equity and equality, and what has been the effect of these groups on women’s participation in community organizations? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 6 d. How has the discontinuation of food rations impacted women’s participation in mother care groups, and health facility access by pregnant women and children under five? e. To what extent have improved production and storage practices and improved access to markets contributed to increasing household expenditures (as a proxy for income) and to reducing poverty? f. To what extent do community members consider that improved agricultural production or increased incomes have contributed to household dietary diversity or to improved nutritional outcomes of children and pregnant and lactating women? (3.3) The interventions and implementation processes deemed more or less acceptable to members of the target communities Sub-question: a. How were the quality of interventions and implementation outputs provided by the project perceived by the participants and their communities? Are there major differences in these perceptions of interventions and service delivery across key beneficiary sub-groups, and what are reasons for these differences in perceptions? Q4: What are the key lessons learned and best practices that should inform future projects in the country? (4.1) What are the lessons, best practices, and approaches that should be considered in designing future food and nutrition security projects? (4.2) What are the lessons, best practices, and approaches that should be considered in implementing and implementation approaches for future food and nutrition security projects? (4.3) What are the risks to the effective implementation of these approaches to support project achievements? (4.4) Based on the lessons learned, how can potential unintended negative consequences be minimized and positive consequences be captured systematically? Sub-questions: a. To what extent did the design of approaches reflect the factors that contribute to malnutrition in the Western Highlands? b. Was the approach for subsidizing maternal and child health and nutrition volunteers an effective model for producing the expected project outputs? And for ensuring continued benefits beyond the life of the project? c. Based on the transition from provision of food to cash transfers, which model is most effective for the particular context in the Western Highlands? d. Are there lessons learned in terms of how PAISANO managed the evolution of Ministry of Health leadership, funding, and models that could be useful for future projects operating in an uncertain conditions? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 7 3. METHODOLOGY 3.1 Evaluation Design 3.1.1 Overall Design The final evaluation will use a mixed-methods approach. The evaluation team will begin with a desk review of project documents, validate its understanding of the projects through consultations with USAID and implementing partners (IPs), conduct a population-based household survey using all implementation villages as the sampling frame, and lead a purposively sampled qualitative study. The quantitative and qualitative components were conducted sequentially to allow the quantitative data to inform the qualitative research. The qualitative component of the PAISANO PE will be organized to complement and expand on the quantitative results obtained through comparing the baseline and endline PBSs. The evaluation team will employ the following qualitative data collection methods: a desk review of performance reports and other project documents and data, key informant interviews (KIIs), focus group discussions (FGDs) with participants and non-participants, in-depth Interviews (IDs), and direct field observations in the sites to be visited. Aside from the desk review, these methods collect primary qualitative data that will provide additional context and depth of understanding for the quantitative PBS results and will help triangulate findings. KIIs, FGDs, and IDIs will collect data on three administrative levels—national, municipal, and community. They will serve as the primary source for answering evaluation questions that focus on beneficiary and stakeholder perceptions, motivations of government and service providers to maintain projects, understanding of project processes, explanations of beneficiary behavior changes and reasons for the changes, and other questions that are amenable to qualitative approaches. The evaluation team will develop a set of semi-structured data collection tools for the KIIs, FGDs, and IDIs, with each set consisting of instruments for each targeted respondent category. Limitations The primary limitations to this evaluation design are based in part on budgetary constraints, but mostly on the severe time constraints imposed by the IP's imminent cessation of activities and the subsequent unavailability of IP staff in the field after the first week of October 2018. Another limitation is that the sample of villages selected for data collection may not be fully representative of the target population covered by PAISANO. Although the final selection of sites was done independently and randomly by the PE team, based on specific objective selection criteria, the accessibility of sites and degree to which sites were representative were discussed with and verified by the IP. This was done to facilitate rapid development of the design and ensure feasibility of travel, given the time constraints faced by the PE, but it may have introduced a small amount of bias into the overall procedure. To address and mitigate these limitations, we limited the evaluation activities to eight communities in seven municipalities, as described in Section 3.2.2, which is the number of representative communities and municipalities that could be accessed in the time available. The team will strengthen the findings from interview responses by triangulating the findings relating to each SO through a comparison of responses for selected evaluation questions across the different regions and municipalities. Using a mixed method evaluation approach, the evaluation findings will be further strengthened by triangulating them with quantitative findings from the analysis of the 2018 PBS endline data, comparisons of the Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 8 baseline and endline data, and analysis of the life-of-activity indicator data comparing project targets to actual performance monitoring data from the Indicator Performance Tracking Tables (IPTTs) in the IP’s Annual Results Reports. The PE report will provide full disclosure on the limitations of the findings based on the methodology. 3.1.2 Overview of Data Sources Desk Review The team has conducted a desk review of project documents available from both the PAISANO project and USAID, and undertaken content analysis of information relevant to the EQs. The document review has allowed the team to view progress toward planned results, identify questions and issues to be pursued through the field work (see the Inception Report for questions and issues raised in document review), and will serve as a key resource for answering the EQs. The evaluation team has already received and reviewed documentation provided by FFP and PAISANO. The key documents include the following:                  Program Proposal Narrative Annual Results Reports and annexes FY 2018 quarterly reports and annexes Pipeline Resource Estimate Proposal (PREP) narratives and annexes PAISANO 2015 Mid-term Evaluation Report Program monitoring tools and data sets, including the IPTT and Performance Monitoring Plan PAISANO Results Framework Monitoring and Evaluation Plan PAISANO baseline and endline studies PREP sustainability and exit strategy and plans Pipeline and Resource Estimate Proposal (2013-2016) Save the Children/Guatemala Response to MTE Recommendations M/CIO EADS Analytical Brief on Agriculture in Guatemala Food and Agriculture Organization of the United Nations. La Agricultura Familiar alimenta a Guatemala y es base estructural del desarrollo rural integral y sostenible. Guatemala Demographic and Health Survey 2014-15 Nutrition Mortality, Food Security , and Livelihoods Survey based on Smart Methodology USAID/Guatemala monitoring reports As the desk review and field work will be completed in a relatively short timeframe, the team anticipates secondary data collection to be an ongoing process. To the greatest extent possible, findings from the document review will be integrated into the primary data collection process. The documents are being used to assist the evaluation team in preparing data collection instruments. Baseline and Endline PBS Data Quantitative baseline and endline data were collected through a PBS in the two DFAP areas in Guatemala (the CRS SEGAMIL project and Save the Children’s PAISANO project). The baseline sample included a total of 150 communities and 6,000 households (75 communities and 3,000 households in each of the two program areas). The baseline PBS was designed to collect data for 20 FFP indicators Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 9 related to food security, children’s nutritional status and feeding practices, women’s nutritional status and dietary diversity, WASH, agricultural practices and measurements of poverty. The PBS also included eight custom indicators to assess women’s health care awareness and decision-making for themselves and their children.2 In 2018, FFP awarded ICF Macro, Inc., a contract to conduct independent performance evaluations of the two DFAPs, which are expected to close out in September 2018. An endline PBS was conducted in the DFAP implementation areas as part of the PEs. The endline PBS was administered in 70 communities and 2,800 households across the two program areas (35 communities and 1,400 households for each of the two programs). The results of the endline PBS will be used to evaluate change over time in the indicators that were collected at baseline. The baseline and endline PBS data will also be used to conduct additional bivariate or multivariate analyses where appropriate; for example, to address EQs related to project achievement and targeting, and to empirically test hypothesized linkages along the project theory of change. Annex A describes how the PBS data will be used to address each evaluation question. Qualitative Data KIIs. Data will be collected through semi-structured interviews with key informants at national, municipal, and community levels, with respondent categories as specified in Section 3.2.1. Questions will be tailored for each respondent category to address program-specific elements of the four overarching evaluation question categories (e.g., project achievements, sustainability of outcomes, effectiveness of interventions, and unintended consequences and lessons learned). An interview guide with structured questions will be used to manage the interviews and organize responses. KIIs at the national level will be scheduled with PAISANO (Save the Children and PCI) senior staff, the USAID Guatemala FFP Officer, and relevant mission staff. The USAID mission and project staff will assist the evaluation team in identifying relevant stakeholders to be interviewed during an in-briefing upon arrival of the team in Guatemala. The evaluation team will also use Skype to receive an initial briefing from the PAISANO team. National-level KII respondents will include technical staff within the key government ministries of health and agriculture. At the municipal level, technical planning staff and health staff will be interviewed. At the community level, project beneficiaries and individuals involved in service delivery, such as mother leaders, agricultural leaders, agricultural promoters and nutrition promoters, and community group leaders, will be interviewed. KIIs will capture their perspectives on the effectiveness and efficiency of project interventions and targeted groups, quality of services provided, motivations and capacity to demand and sustain services, and the sustainability of project interventions and outcomes. FGDs. The evaluation team will facilitate FGD sessions within a purposive sampling frame at the community level and use semi-structured FGD guides with lead questions and probing prompts. The FGD guides will be tailored for each respondent category, as specified in Section 3.2.1. FGD sessions will be organized by gender and structured by key technical sector under the SO: maternal and child health and nutrition (MNCH) and WASH; agriculture and livelihoods; and early warning systems, 2 A complete description of the baseline study is provided in the “Baseline Study for the Title II Development Food Assistance Projects in Guatemala” report. Available at: https://www.usaid.gov/sites/default/files/documents/1866/Guatemala%20Baseline%20Study%20Report%2C%20March%202014.pdf Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 10 disaster risk reduction, and resilience. The cross-cutting SO on gender and sustainability of existing activities will be integrated into questions in the FGD guide, across each technical sector. In addition, the evaluation team will explore the effects of migration and remittances on families among project communities. FGDs will be conducted in groups of no more than 12 men or women. Similar to the community KIIs described above, the FGD instruments will be designed to address the four evaluation question categories outlined in Section 2. IDIs. In addition to large groups, the evaluation team will conduct a few IDIs with selected beneficiaries and participants, agricultural leaders and agricultural promoters, mother leaders, nutrition promoters, community leaders, and others to more thoroughly explore the four EQs and discover why certain behaviors were adopted, what the motivators or barriers were, if they noticed changes in the community related to project interventions, etc. The purpose of the IDI is to work with a small number of respondents and explore their opinions about the effectiveness of the interventions. An interview guide with structured questions will be used to manage the interviews and organize responses. Observation. The evaluation team will also conduct home visits with mother leaders, agricultural leaders and agricultural/nutrition promoters to conduct semi-structured interviews with households using the interview guides described above. The team will also observe child health care and WASH practices in the home as well as agricultural and livelihood practices around the home. 3.2 Qualitative Data Collection Methods 3.2.1 Types of Respondents Project beneficiaries and non-beneficiaries. These individuals will provide information on perceived benefits from participation in PAISANO activities and likelihood of continued actions and benefits after project close. The evaluation team will observe practices and infrastructure promoted by the project related to water and sanitation, agriculture/livestock, and disaster risk reduction/resilience. The evaluation team will also interview non-participants and non-beneficiaries. The purpose of interviewing this category is to see whether project messages and behaviors have been transferred to community individuals who, although not participating in program activities, have learned about these behaviors and decided to adopt them. The interviewers will try to ascertain why they adopted the behaviors or why they did not and in this way discover what the motivators and barriers are for improved behaviors and whether respondents think the practices are attributable to PAISANO interventions. Community volunteers, such as mother leaders, nutrition promoters, agricultural leaders, and others. These individuals will provide information on perceived effectiveness of project interventions and targeted groups, quality of services provided, motivations and capacity to demand and sustain services, and the sustainability of project interventions and outcomes. Municipal government staff, technical staff from government miniseries of health, agriculture/livestock, and hydraulics, food security staff and partner staff. These individuals will provide perspectives on PAISANO’s effectiveness and outcomes of project interventions, and their organization’s actions and commitments to assume project activities post-projects. Project staff: This includes headquarters staff, managers of regional offices, technical area specialists (nutrition, agriculture, disaster relief, livelihoods), and coordinators. The project Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 11 staff will be asked their perception of the effectiveness of project interventions for targeted groups, quality of services provided, motivations and capacity to demand and sustain services, and the sustainability of project interventions and outcomes. 3.2.2 Site and Respondent Selection Qualitative data collection will be conducted in a sample of communities that have participated in PAISANO activities. Communities were selected to ensure a representation of different ethnicities and languages, agro-ecological zones, and exposure to and participation in a variety of project interventions. Selection criteria include the following:      Program-specific parameters—communities participating in specific PAISANO interventions (such as agriculture, different types of animal production, mothers groups) Community characteristics—including ethnic composition (including language), demographic features (including outmigration), socio-economic status, market access, gender dynamics Agro-ecological/environmental features—agricultural zones/livelihoods zones, water availability/access Community engagement/performance—level of engagement or performance in PAISANO (weak, average, strong), engagement or performance including at least one community that received fewer interventions to compare with those that received multiple interventions Accessibility—distance, access, road conditions To select the communities to be included in the evaluation, the evaluation team decided that these should be representative of the geographic regions and linguistic communities intervened in by the PAISANO project. Eight communities, belonging to seven municipalities of the departments of Huehuetenango, El Quiché, and Quetzaltenango (see Annex C), were selected. In these communities, the K'iché, Ixil and Mam language communities are represented. Because the randomly selected sample where the PBS was conducted is statistically representative of the complete area where PAISANO worked, we selected sites from these communities. The evaluation team obtained community lists for the PBS endline survey with community population size, number of households, language spoken, and distance from the municipal head. The evaluation team then mapped the location of the communities that participated in the program to define a route that would make it feasible to choose a number of communities that met the criteria mentioned above, given the time required to travel and conduct the field work. The evaluation team then verified with Save the Children staff that this was a feasible route and communities were safe to visit. Save the Children staff pointed out the interventions they had carried out in each community. In communities that had experienced most of the program interventions (maternal and child health, WASH, and livelihoods), the evaluation team randomly selected among three or more possibilities. In the cases of communities with very specific activities (Training for Water Committees, Groups of Empowered Women, and Marketing Groups), it was not possible for the evaluation team to have more than one option, and these were purposively selected. Once the route was defined, the evaluation team included local visits to government organizations and other institutions with which the project worked and coordinated its activities (municipalities, health districts, Centro Integral de Producción y Capacitación Agroambiental (CIPCA), and the Highlands’ Goat Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 12 Production Center [Centro de Produccion Caprina del Altiplano] to understand their views about the program. The field visit plan also includes interviews with the staff who implemented the program. The evaluation team will interview staff from PAISANO offices in Quetzaltengo, Santa Cruz del Quiché, and Nebaj, to understand their opinions of the achievements and challenges that they had in executing the program, as well as the lessons learned for future interventions. The evaluation team also included interviews with regional institutions to better understand their coordination with the program and the sustainability opportunities for activities after the program ends. Save the Children staff provided a list of organizations they worked with in the proposed areas so that the evaluation team could carry out an institutional mapping of agencies and include them in the field visit schedule. The evaluation team has included visits and interviews with Secretaría de Seguridad Alimentaria y Nutricional (SESAN), the Ministry of Agriculture, Coordinadora Nacional para la Reducción de Desastres (CONRED), Water for People, AGEXPORT, Caritas, and World Vision in the plan. Through KIIs, FGDs, and IDIs, the PE will map the status and potential sustainability of the private service providers supported by PAISANO, including the fee-for-service providers established, trained, and, in some cases, equipped by the project. This mapping will include the following: the types of services provided, the degree to which women and youth are providing services, the fee structures and current profitability of the different types of providers, the capacity development providers received, the linkages providers have established and maintained with other public and private sector service providers, estimates by participants of the effectiveness and value of services provided, and assessments of whether the necessary resources and capacity strengthening will exist to sustain service providers and the likelihood that service provision will continue after the project ends. Respondent Selection After selection of the communities, respondents will be recruited at the community level based on the following criteria:    Beneficiaries. If multiple beneficiaries are available for interviews and FGDs, participation of beneficiaries will be prioritized based on most recent participation in project activities (which generally corresponds to those having the youngest children, because those with older children “aged out” of some activities).The evaluation team will also include non-beneficiary individuals in the groups or in separate groups where possible. Project volunteers. As much as possible, all project volunteers (including agriculture promoters, nutrition promoters, mother leaders, and agriculture leaders) in the visited communities will be included in KIIs or FDGs, prioritizing those with the longest participation as volunteers. Members of group organizations. Group leadership will be identified, if possible, or otherwise all available group members will be identified and prioritized by length of service. Examples for this group are the community leaders in the COCODEs, Comité de Seguridad Alimentaria Nutricional (COCOSAN), and local coordinators for disaster reduction (coordinadora local para la reducción de desastres [COLOREDs]). Because the evaluation team does not know the communities, the team will ask PAISANO’s regional sector technicians to assist with organizing community groups of beneficiaries and non-beneficiaries as Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 13 well as key informants, including project volunteers and members of organizations that the project works with, in accordance with the field visit schedule. Number of Respondents  FDGs. A total of 21 FGDs will be conducted. The FGDs will be gender-segregated, unless individuals pertain to a community group together (a water committee, for example). The FGDs will aim to include up to 12 participants per group discussion. Some focus groups will be highly focused on specific themes, such as the FGD for mother leaders and nutrition promoters on their actions and achievements with health, nutrition, and WASH promotion, including the role of food and cash assistance. Similar groups will be held with community volunteers or groups with a role in agriculture/livelihoods, water committees, or COCODE. A female beneficiary group will more broadly focus on maternal and child health and nutrition (MCHN) activities and agriculture/livestock activities. To more thoroughly explore certain themes, the evaluation team will conduct IDIs with selected individual participants.  KIIs. A total of 61 KIIs will be undertaken at the national, municipal, and community levels. Four KIIs will be conducted per community, totaling 40, plus 13 with municipal or regional officials. In addition, upon return from their field sites, approximately 8 KIIs will be conducted over a two-day period in Guatemala City among IP staff, USAID Guatemala FFP mission staff, Guatemalan government officials, and other pertinent stakeholders in Guatemala City.  Community and household observations. An estimated 18 community and household observations will be conducted in communities. These observations will involve a combination of the following: water and sanitation conditions related to practices, such as tippy taps, safe play spaces for child; agricultural fields or livestock systems, such as animal pens, etc.; and water and sanitation infrastructure, such a water tanks or latrines. A tentative list of target KIIs and FGDs by protocol category is presented in Table 1. Input on this list from the USAID mission and IPs is requested in the coming week to finalize the respondent list and KII and FGD data collection instruments. Table 1. Illustrative Sampling Frame for KIIs and FGDs Detailed breakdown of KIIs, FDGs, IDIs, and observations Total # Total respondents KII National Administrative Level  IPs (2), government ministries of health, agriculture/livestock, and hydraulics, SESAN, civil society organizations 8 8 Regional/Departmental/Municipal/Level  Regional government ministries of health (4 Area Directorate of Health/Dirección del Area de Salud), agriculture/livestock (3 Ministry of Agriculture), hydraulics (3), disaster risk reduction (3) 13 13 Community Level Maternal, child health, and nutrition and WASH  Community health workers and health center staff (nurse technicians)  Mother leaders  Nutrition promoters 13 13 Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 14 Detailed breakdown of KIIs, FDGs, IDIs, and observations Total # Total respondents  WASH committee representative Agriculture and livelihoods  Agricultural leaders  Para-veterinarians  Agriculture financial service providers and input providers 13 12 Early warning, disaster risk, resilience  Disaster warning systems (CONRED/COCODE) representative 4 4 Cross Cutting – Gender, Governance  Community Development Commission (COCODE) representative 5 5  Follow-up IDI with community health workers, mother leaders, nutrition promoters, WASH committee representatives, agriculture leaders, COCODE 5 5 KII TOTAL 61 FGD Community level M F  Mother leaders and nutrition promoters 3 6-20  Maternal, child health, and nutrition beneficiaries (who are also food/cash recipients) and non-beneficiaries 4 20-40  Water Committees 4 20-40  Agricultural leaders (and agriculture beneficiaries), agricultural promoters 2 4 30-45  COCODE/community leadership 4 20-40 FGD TOTAL 21 96-185 Observations Community level and household level  Maternal, child health, and nutrition water and sanitation practices and materials 6 6  Agriculture/livestock practices/systems 6 6  Water and sanitation infrastructure 6 6 Observations total 18 18 3.2.3 Field Processes Team responsibilities. The four-person evaluation team is composed of an experienced evaluation team lead, as well as three subject matter experts whose respective areas include agriculture and livelihoods, health and nutrition, water and sanitation behavior change and infrastructure, resilience/disaster risk reduction, community mobilization, governance, behavior change communications, and adult education. Team members will focus on their specific areas of expertise and will carry out data collection in pairs, when interviews, group meetings or observations present the opportunity to collect information across thematic areas. The evaluation team consists of the following members: Sandra Wilcox will assume overall responsibility for the evaluation, assuring that the scope of work is carefully followed, that high-quality information is collected and carefully analyzed, that the team maintains the objectivity and follows the proposed timeline (see below). She will take the lead on review of the monitoring and evaluation systems and management structure and functionality. Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 15 Kali Erickson, with expertise in maternal and child nutrition and health, will review activities around infant and young child feeding practices, maternal health and nutrition practices, growth monitoring, and behavioral change communication strategies to promote behaviors to reduce malnutrition. She will also review the use of food assistance and cash assistance promoted to improve household and maternal and child nutrition. John Russell, with expertise in agriculture and livelihoods, will review interventions related to agriculture and livelihoods promotion. He will review approaches designed and implemented by Save the Children and PCI and partners to transfer knowledge and practices related to agriculture and other livelihood strategies. Carlos Flores, with expertise in WASH, resilience, disaster risk reduction, and governance, will review interventions related to these areas. Team members will conduct interviews related to their field of expertise. In addition, each team member will examine the cross-cutting aspects of gender, social and behavior change communication, and staff and volunteer training and material development related to his or her area of expertise. Field work will be accompanied by Save the Children’s PAISANO staff, and in one area, PCI former staff members. Although PCI’s project activities concluded in June 2018, several PCI technical staff live in the area and will be contracted to accompany the evaluation team during the estimated three days covering the PCI intervention areas. In addition, translators will accompany each vehicle to translate between Spanish and the local language (Kiché, Ixil, Mam, and Poptí), depending on the area visited that day. Each subject matter expert will have a translator to assist with questions and facilitate group discussions when needed. The evaluation team will be supported by ICF staff, including internal quality assurance, quantitative data analysis (in preparing the required tables and graphics from the PBS data), and editorial support. The evaluation team will work closely with the ICF-led PBS team for comparative quantitative data analysis and data presentation. 3.2.4 Data Management Processes Quality control mechanisms. The evaluation team will take several steps to ensure that the qualitative data collected in the field (e.g., KIIs, FGDs, and Community Profiles) are of high quality and can be depended on to answer the EQs discussed previously. To ensure and improve the quality of data to be gathered, the qualitative evaluation protocol, describing the process for selection of respondents, time and process for asking questions, as well as the EQs themselves, will be reviewed by PAISANO to assure that it describes a feasible process, likely to be acceptable to community leaders, beneficiaries, project volunteers and project partners. The team lead ensures that the team follows the established schedule and all interviews take place. The team also anticipates recruiting at least one female interviewer per team to ensure access to and frank discussion with female beneficiaries. In addition, the evaluation team will implement a thorough team training and instrument pre-testing regimen prior to going into the field. The following approaches will also be used to maximize data quality:  The field team will rely on detailed hand-written notes that will be collected by team members, which will be reviewed by team members before departing from the community. Team meetings Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 16 will be held each evening spent in the field to discuss the quality of the data collected, to identify any potential problems or conflicts among the team members, and to share results and revisit challenges faced in the day’s data collection. Field notes for the day will be reviewed and enhanced during these meetings to ensure that information is not lost to interviewers and FGD facilitator recall. In addition, if time permits, the teams will take a break from conducting KIIs and FGDs, and will spend time synthesizing and coding the information contained in the field notes collected over the previous days.     The team will seek to address the “right answer” phenomenon. Beneficiaries, often feeling indebted for project inputs, are often reluctant to say anything that could be construed as negative about an FFP development food assistance project. Recognizing this phenomenon, the team will identify ways to skip over these initial responses and move quickly into details likely to more accurately capture the beneficiary experiences, as well as ask similar questions in another way to triangulate information. The team will explain interview confidentiality and request verbal consent prior to initiating any FGD or KII session. The team will also emphasize that responses will be used to inform and enrich future projects to partners, village leaders, and interviewees. The team will consistently seek to ensure privacy during interviews and discussions and will ask curious onlookers to leave surrounding areas. Each day, after data collection, the team will meet to review progress, discuss key findings and impressions, describe any difficulties or challenges encountered, and find solutions. The evaluation team will consult quantitative findings and project reports to triangulate any unusual or surprising findings, and request support from PAISANO staff to flag findings that contrast with existing information. Note-taking and photographs of observations. When evaluation team members work in teams for KII, one will take notes while the other asks questions. When there is only one team member conducting the interview in Spanish, that person will take his or her own notes or notes will be taken by an assistant. Wherever possible, the interviewer will be the same sex as the participant. FDGs will be conducted with two evaluation team members present, and notes will be taken by the evaluation team member who is not facilitating. In addition, FGDs will be recorded, as feasible, as a source for back-up information. If community members consent, team members will also use photography as a data collection tool when conducting community and household observations. Use of interpreters and translators. It is expected that some meetings with national staff will be carried out in English, and all other staff interviews, meetings, group discussions, and household interviews will be conducted in either Spanish directly by the evaluation team, or in the local language (Kiché, Ixil, Mam, Poptí) with the help of translators. Field translators will travel with the team during the field work to assist ET members and provide direct interpretations of proceedings. Translators will be hired independently, as feasible, or will be community members, but will not be PAISANO staff. Since translators will be hired in the field communities, the team will meet with them upon arrival and go over the tools with them so they understand the questions being asked. The translators will provide local language to Spanish translation and (if able) some note-taking. Most "training" would be on 1) the importance of accuracy of literal translation over "interpretation" and 2) getting a feel for the appropriate spoken interval to translate (i.e. every sentence is too short, every five minutes is too long, Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 17 etc.). The ET will also discuss issues of confidentiality and clarify any questions the translator/interpreter may have about the interview guides/questions. Role of project staff. During data collection, PAISANO staff will mainly assist the ET to locate and introduce potential interviewees. However, if needed due to cultural sensitivities, PAISANO staff will facilitate interviews when a member of the ET is able to sit with a translator and take notes as well as guide, contribute and/or verify the content of the interview. Typing Notes into electronic files. Interview notes will be typed into electronic files as soon as feasible and each file will be labeled with identifiers, including community name, type of data collection (KII, FDG, Obs.), date, main interviewer, and auxiliary interviewers, interpreter and others present. The electronic file will be shared for verification with others present, as feasible. Photographs will also be immediately labeled and entered into the project database, along with electronic recording files. Data will be in a password protected central location available only to the ET. 3.2.5 Informed Consent Procedures and Ethical Considerations The evaluation team will comply with USAID’s Scientific Research Policy by ensuing that informed consent is obtained from all KII and FGD participants. In addition, the evaluation team will seek and obtain Institutional Review Board (IRB) approval prior to beginning any qualitative field data collection. Informed consent. The evaluation team will obtain verbal informed consent from respondents before carrying out any data collection. A consent form will be used for all KIIs and FGDs that will be translated into Spanish along with other languages expected to be encountered in the field (e.g., Kiché, Ixil, Mam, Poptí), if deemed useful by the IP. For any translations, the consent form will be back-translated into English and will be piloted to ensure clarity before use in the field. Subjects will read or have the form read to them in Spanish or the relevant local language, where applicable, and respondents will be asked to sign the form, if literate. Careful attention will be paid to ensure that respondents understand that their responses will be used for research purposes and are expected to be made public without compromising their confidentiality and anonymity. Ethical approvals. Ethical approvals for conducting the qualitative research will be obtained by the evaluation team from ICF’s Institutional Review Board prior to the data collection. The evaluation team will strive to maintain high standards in methods, quality, and data security. All data collection activities will adhere to Institutional Review Board requirements and the professional standards of the American Evaluation Association. All data will be handled in such a manner as to protect the identities of informants in any situations in which their comments could potentially have a negative impact on their security or community status. The evaluation team will safeguard the confidentiality and anonymity of FGD and KII respondents. In all cases, and specifically where data are collected from individuals not acting in their professional capacity as representatives of an organization, the evaluation team will obtain verbal informed consent. 3.2.6 Training Interpreters, including on Human Subjects Protocols It is expected that the evaluation team will require at least one interpreter for each of the major local indigenous languages, as Spanish is spoken by all members of the ET. As the languages differ by region, a separate meeting will be held with translators and local IP staff upon arrival in each region with a different local language, to orient them about key aspects of research involving human subjects (such as confidentiality and the right to refuse participation in any aspect for any reason). Likewise, parameters of Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 18 PAISANO staff participation will be discussed and agreed upon, including the role of project staff to identify and organize interviews and when appropriate, help facilitate the discussions. 3.2 Data Analysis 3.2.1 Qualitative Data Analysis, Coding, Interpretation Methods Notes will be taken by team members during interviews and FGDs. In addition, for back-up, FGDs will be recorded, with consent of participants. Following initial creation of a “story line” describing the goals of the analysis, notes of FGDs will be coded manually, using a hybrid of both pre-set and emergent codes. Themes and categories of codes will correspond to the overall PE EQs, the goals and activities of the project and of each sector (agriculture/livelihoods, MCHN, disaster risk reduction, and WASH), the expected outcomes, and the perceived effectiveness of interventions.3 Each interview tool will be coded separately. Where feasible, coding will be done by more than one team member to reduce coding error. KII notes will be summarized using standard content analysis techniques.4 For data from the KIIs, FGDs, and IDIs, content analyses will summarize common trends and patterns to highlight project-, sector-, and gender-differentiated trends, and to identify examples of positive deviance and most significant change. Table 2 describes the category and range of codes that will be applied to each data collection instrument. Analysis of qualitative data will be integrated with PBS and document review findings as described below. Table 2. Category and Type of Codes Category Codes by type Sectors SO1 (agriculture/livelihoods), SO2 (MCHN/WASH), SO3 (resilience) Location Name of the implementation district Administrative level where the interview is conducted D (District), and C (community) Type of respondent TB (targeted beneficiary group [e.g., youth, pregnant or lactating women, adult women from poor households]) CBO (community-based organization [e.g. village peace committees, mother care groups, water user associations, male change agent group]) CL (community leaders [e.g. council leader, lead farmer, mother leader, nutrition promoter, agriculture promoter]) PS (private sector representative from a business or organized group [e.g.; commodity buyer, input supplier, service providers, agricultural extension agent, chamber of commerce]) PS (public sector representative or group [e.g., government officials, staff from a government funded health facility, police, judge]) Gender M, F Evaluation question number from the PE scope of work Strengths Challenges Other pertinent information 3 Saldaña, J. (2015). The Coding Manual for Qualitative Researchers. (3th Ed.). SAGE Publications. 4 Kippendorf, K. (2018). Content Analysis: An Introduction to Its Methodology (4th Ed.). SAGE Publications. Content analysis is a technique for systematically describing written, spoken or visual communication. It provides a quantitative (numerical) description. Many content analyses involve media—print (newspapers, magazines), television, video, movies, the Internet. Content refers to the meaning of the information. Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 19 Category Codes by type Uses the number of the actual question (e.g., 1.1, 1.2, 1.3) Team Organization for Analysis Under the overall guidance of the PAISANO PE team leader, the KII and FGD data will be analyzed by the technical specialist responsible for each sector: agriculture/livelihoods (SO1), MCHN (SO2), disaster risk reduction and governance (SO3), and WASH. To distribute the analytical workload evenly across the team, individual team members will be assigned by the team leader to be responsible for analyzing data from interviews of community leaders, FFP staff, project staff, and government representatives. 3.2.2 Integration of PBS Data PBS data can be used to determine whether findings from the PBS corroborate the qualitative finding for a given evaluation question. Findings from the qualitative data will also be used, as appropriate to the evaluation question, to interpret findings from the PBS. The baseline and endline PBS data will be used to assess change over time in the key FFP indicators and custom indicators. The baseline and endline PBS data will also be used, as appropriate to the evaluation question, to conduct additional disaggregations and bivariate or multivariate analyses to help strengthen findings established from the qualitative (or quantitative) data analysis, or to explain findings that cannot be corroborated. Annex A provides details on the how the PBS data will be integrated to address each question. The PBS data identified potential areas for further qualitative investigation that will be integrated with the EQs. For example:  Why did household hunger experiences remain unchanged despite improvements in household dietary diversity scores?  Why did the percentage of farmers using financial services not change?  Why did handwashing practices not change despite the increase in access to improved water sources and sanitation facilities?  Why were increases in exclusive breastfeeding not achieved despite the fact that breastfeeding is widely practiced? What level of support was provided by the programs for extended breastfeeding?  What contributed to the huge gains in children’s minimum acceptable diet? What provider and beneficiary experiences influenced behavior change around feeding of children?  Why is it that increased knowledge about pregnancy, neonatal and childhood illness danger signs did not result in increases in women who seek health care when the danger or illness signs are present?  Why is it that despite increased knowledge about pregnancy, neonatal and childhood illness danger signs, fewer women with children under 5 years of age are making health care decisions for themselves? 3.2.3 Integration of Findings from Document Review The purpose of the desk review of the project MTE, annual results reports, PREPs, and available secondary documentation was to identify key findings and explanatory factors from project reports and internal monitoring and evaluation data pertaining to the project goals and each of the EQs. Results from Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 20 the document review will be used to help define questions for the qualitative KIIs and FGDs, and to serve as guideposts for assessing and interpreting progress toward achieving the project’s desired outcomes. Examination of key documents will also be used to triangulate qualitative data provided by project participants and by KIIs as well as to provide explanatory context for both PBS and qualitative results. 3.2.4 Proposed Report Outline The PE report will be organized as follows: 1. Executive Summary, including theory of change, EQs, design and limitations, summary findings and conclusions, and key conclusions and recommendations 2. Evaluation Purpose and Questions 3. Project Background, including country context, program description, theory of change, baseline and mid-term context, and endline indicator relative to PAISANO targets 4. Evaluation Methodology, including data collection and analysis (with a brief summary of PBS methodology), and evaluation limitations and constraints 5. Findings and Conclusions, organized by the EQs listed above and using the analytical methods described 6. Recommendations 7. Annexes o o o o o o o o o o o Illustrations Tables Scope of Work Evaluation Questions Evaluation Design Matrix Qualitative Fieldwork Schedule Data Location Sites Respondent Types Interviewed by Location Data Collection Instruments PBS Methodology, Results Tables, and Supplemental Statistical Results References Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 21 ANNEX A: QUALITATIVE DESIGN MATRIX Evaluation Questions Type of Answers Needed Data Source(s) Collection Methods Data Analysis Methods Overall Project Achievements 1. To what extent has PAISANO met its goals, purposes, and desired outcomes; and what factors promoted or inhibited their achievement? Comparative, descriptive, normative -PAISANO documentation (e.g., proposal, annual and quarterly reports, IPTT monitoring, PREP reports, baseline and endline surveys) - Project participants, indirect beneficiaries -non-beneficiary community members - ALs, MLs, NPs, APs - PAISANO staff, managers - USAID FFP staff - USAID Guatemala staff - Key Government of Guatemala representatives at national, regional levels (Guatemala Ministry of Health, and Ministry of Agriculture), municipal staff - Local community leaders, councils, coordinating bodies - MTE report -private sector actors - Baseline and endline PBS - Desktop review Qualitative - KIIs using semi￾structured instruments specific to given respondent category - FGDs using semi￾structured instruments specific to given respondent category Quantitative: PBS survey - Content analysis of project participant and non￾participant responses (FGDs) and stakeholder responses (KIIs) to assess their views on extent to which activities have been achieved, and on what factors promoted or inhibited project activities and outcomes - Analysis of PAISANO Annual Results Reports, that describes achievements and factors related to performance progress to date - Analysis of mid-term evaluation describing findings and conclusions on achievements, factors related to progress and recommendations for project improvement. - Comparison of baseline and endline indicator estimates to assess change over time 1a. How do outcomes and targeting strategies vary across key beneficiary sub￾groups, as prioritized by USAID, (e.g., poorest households, women beneficiaries, children under 5), and why? Comparative, descriptive, normative Same as for EQ 1. Same as for EQ 1. As for EQ 1 analyses. Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 22 Evaluation Questions Type of Answers Needed Data Source(s) Collection Methods Data Analysis Methods Overall Project Achievements 1b. Based on available evidence, what are plausible pathways and determinants for achievement of key outcomes across project activities? Descriptive and normative As for EQ 1. As for EQ 1. - Document review using PAISANO project’s results framework theory of change as the reference point. Assess how well project implementation followed or deviated from the causal pathways in the RF. - Supplement analysis with data from KIIs and FGDs. Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 23 Evaluation Questions Type of Answers Needed Data Source(s) Collection Methods Data Analysis Methods Sustainability of Outcomes 2. Based on the evidence, what project activities and outcomes are likely to be sustained, and why? Descriptive - Direct and indirect project beneficiaries - ALs, MLs, NPs, APs - PAISANO documentation - Mid-term evaluation report - PAISANO staff - USAID FFP staff - USAID Guatemala staff -Key Government of Guatemala representatives at national and regional levels (Guatemala Ministry of Health, and Ministry of Agriculture) - Municipal and communal staff/ representatives - Staff at other NGOs and donors - Private sector actors - Local community leaders - PAISANO PBS baseline/endline data - IPTT baseline/endline data - Project documentation - KIIs using semi￾structured instruments specific to given respondent category. - FGDs using semi￾structured instruments specific to given respondent category. - PBS and IPTT data are collected independently -IDIs using semi structured instruments specific to respondent category - Content analysis of FGDs with direct and indirect beneficiaries to assess changes in practices and behaviors, as well as service utilization and satisfaction by direct and indirect beneficiaries. Discuss statistical comparison of baseline/endline data to assess direct/indirect beneficiaries’ changes and intent to sustain improved behaviors. - Content analysis of select KIIs to assess perceived changes in behavior, potential for other funding sources or mechanisms, and motivation to sustain some or all of the activities. - Analysis of project documents and select KIIs to assess the type of linkages and processes/systems in place between project structures and other organizations (national government, NGOs, private entities, funders, etc.) - Analysis of progress of PAISANO Sustainability Plans and Exit Strategies -Review of MTE reports that discuss the likelihood of sustainability based on project Sustainability Plans and Exit Strategies - Review of subsequent PAISANO Annual Reports to determine if PAISANO followed up on findings/conclusions/recommendations from MTE to ensure sustainability of activities and outcomes 2a. What processes, systems, and institutional arrangements (especially linkages and coordination with other USG and non￾USG investments and service providers) were made by the project or members of the target Descriptive - PAISANO project documentation - Mid-term evaluation report - Project staff - USAID FFP staff - USAID Guatemala staff - Key Government of Guatemala representatives at national and regional levels - Desk review - KIIs using semi￾structured instruments specific to given respondent category. - FGDs using semi￾structured instruments specific - Analysis of project documents and relevant KIIs (e.g., project staff, Ministry and Departmental staff, USAID staff, staff at other donors and NGOs) to assess the type, strength, and nature of processes, systems, and linkages in place, and their level of adequacy to sustain the project interventions. - Analysis of IP Sustainability Plans and Exit Strategies Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 24 Evaluation Questions Type of Answers Needed Data Source(s) Collection Methods Data Analysis Methods Sustainability of Outcomes population to sustain the necessary and critical services required to achieve and sustain project outcomes? (Guatemala Ministry of Health, and Ministry of Agriculture) - Municipal and communal staff / representatives - Staff at other NGOs and donors - Local community leaders to given respondent category. -Review of MTE report that discuss the likelihood of sustainability based on Sustainability Plans and Exit Strategies - Review of subsequent Annual Results Reports to determine if the project followed up on findings/conclusions/recommendations from MTE to ensure sustainability of activities and outcomes 2b. What is the level of motivation of the service providers to continue providing services after the project ends, and of the beneficiaries to receive and pay (or invest time) for these services? Descriptive - Direct and indirect project beneficiaries - ALs, MLs, NPs APs - Project documentation - Mid-Term Evaluation report - PAISANO staff - USAID FFP staff - USAID Guatemala staff - Key Government of Guatemala representatives at national and regional levels (Guatemala Ministry of Health, and Ministry of Agriculture) - Municipal and communal staff/ representatives - Staff at other NGOs and donors - Private sector service providers - Local community leaders - Desk review - KIIs using semi￾structured instruments specific to given respondent category. - FGDs using semi￾structured instruments specific to given respondent category. - IDIs using semi￾structured instruments specific to respondent category - Content analysis of FGDs with service providers and beneficiaries to assess motivation to invest (money and/or time) in providing and/or purchasing services, as well as their perception of the value of activities (crossover with 2.c.) - Content analysis of relevant KIIs (e.g., local community leaders, project staff, Ministry staff) to assess their motivation to continue services and fund them -Review of project documents to identify indications that beneficiaries are already investing time and/or money into project activities (e.g., cost-share, volunteer leader/ promoters, resumption of discontinued activity) - Analysis of PAISANO Sustainability Plans and Exit Strategies - Review of MTE report that discuss the likelihood of sustainability based on PAISANO Sustainability Plan and Exit Strategies - Review of subsequent PAISANO Annual Results Reports to determine if MTE findings/recommendations were followed up on to ensure sustainability of activities and outcomes Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 25 Evaluation Questions Type of Answers Needed Data Source(s) Collection Methods Data Analysis Methods Sustainability of Outcomes 2c. What are the community perceptions of the quality, effectiveness, and sustainability of implementation activities and services provided by the project? Descriptive - Direct and indirect project beneficiaries - ALs, MLs, NPs, APs - Project documentation - Local community leaders, councils, commissions - Communal representatives - Desk review - KIIs using semi￾structured instruments specific to given respondent category. - FGDs using semi￾structured instruments specific to given respondent category. - Content analysis of FGDs with beneficiaries (direct and indirect) and other project participants to assess motivation to invest (money and/or time) in providing and/or purchasing services, as well as their perception of the value of activities (crossover with 2.b.) - Content analysis of relevant KIIs (e.g., local community leaders, communal staff) to assess their motivation to continue services and fund them. -Review of project documents to identify indications that beneficiaries are already investing time and/or money into project activities (e.g., cost-share, volunteer leader/ promoters, resumption of discontinued activity). Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 26 Evaluation Questions Type of Answers Needed Data Source(s) Collection Methods Data Analysis Methods Effectiveness of Interventions 3. In each technical sector addressed by the project (MCHN; agriculture / livelihoods; early warning systems / disaster risk / resilience, WASH), what were the most effective implementation methods and approaches? Comparative, descriptive, normative - Direct and indirect project beneficiaries -ALs, MLs, NPs, APs - Project staff - USAID FFP staff - USAID Guatemala staff - Key Government of Guatemala representatives at national and regional levels (Guatemala Ministry of Health, and Ministry of Agriculture) - Municipal and communal staff/ representatives - Local community leaders - Project documentation (e.g., proposals, annual and quarterly reports) - Mid-Term Evaluation Report - PAISANO PBS baseline/endline data - IPTT baseline/endline data - KIIs using semi￾structured instruments specific to given respondent category. - FGDs using semi￾structured instruments specific to given respondent category. -Desk review of project documents, IPTT - PBS baseline and endline surveys - Content analysis of beneficiary responses (FGDs) and stakeholders responses (KIIs) to assess their views on effectiveness and of projects' implementation methods across the multiple technical sectors, as well as for specific project activities. - Analysis of project proposal, Annual Results Reports, and Mid-term Evaluation Report to understand changes in implementation approaches and activities in specific technical sectors. 3a. What are the strengths of and challenges of the projects’ overall implementation approach, management, communication, and collaboration? What steps were taken by the project to address challenges? Descriptive - PAISANO documentation - Mid-term evaluation report - Project staff - USAID FFP staff - USAID Guatemala staff - Key Government of Guatemala representatives at national and regional levels (Guatemala Ministry of Health, and Ministry of Agriculture) - Municipal and communal staff and representatives - Staff at other NGOs and - Desk review - KIIs using semi￾structured instruments specific to given respondent category - IDIs using semi structured instruments specific to given respondent category - Content analysis of relevant KIIs (e.g., project staff, government staff, USAID FFP staff, other NGO/donor staff implementing in same area) to assess their views on the project’s strengths and challenges. This includes program management partnership, how information management system was used for decision making, planning, coordination and communication mechanisms used by project - analysis of proposals describing implementation approach, management, communication and collaboration to be compared with annual reports to identify Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 27 Evaluation Questions Type of Answers Needed Data Source(s) Collection Methods Data Analysis Methods Effectiveness of Interventions donors implementing projects in same areas - Private sector service providers - Local community leaders strengths and challenges and steps the project took to address challenges - Review of mid-term evaluation reports that identify strengths, challenges and weaknesses of project implementation approach, management, communication, collaboration; compare with subsequent annual reports to determine if these strengths are still evident and what steps the project took to address challenges., how they have been overcome (and if so, how) - Compare results of these reviews with content analysis of KII data 3b. Who was targeted by and benefited from each of the project’s intervention activities, and how effective were the selected targeting approaches in achieving their respective goals? Did targets change over LOP? Comparative, descriptive - PAISANO documentation - Mid-term Evaluation Report - Project staff - USAID FFP staff - USAID Guatemala staff - Key Government of - Guatemala representatives at national and regional levels (Guatemala Ministry of Health, and Ministry of Agriculture) - Municipal and communal staff and representatives - Local community leaders -ALs, MLs, NPs APs - PBS baseline/endline data - IPTT baseline/endline data - KIIs using semi￾structured instruments specific to given respondent category - Desk review: project documents; baseline and end-line surveys - Analysis of project documents (e.g., proposals and progress reports, IPTTs) to understand logic and intent of targeting, as well as approaches selected and their relative effectiveness. - Content analysis of relevant KIIs (e.g., project staff, USAID staff) to understand effect and intent of targeting. - Comparison of findings related to targeting in the mid-term evaluations with those found in this round - Disaggregation of endline estimates of select FFP indicators by key target groups and comparisons with the baseline estimates. This will be performed for a select number of indicators as part of the multivariate modeling. Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 28 Evaluation Questions Type of Answers Needed Data Source(s) Collection Methods Data Analysis Methods Effectiveness of Interventions 3c. How are the quality, frequency, effectiveness, and suitability of the services provided by the project perceived by the beneficiaries and their communities? Are there major differences in these perceptions of service delivery across key beneficiary sub-groups, and what are reasons why? Comparative, descriptive - Direct and indirect project participants -ALs, MLs, NPs, APs - Previous evaluation reports - Project documentation - Private sector actors - USAID FFP staff - USAID Guatemala staff - Communal and municipal staff - Local community leaders - Desk review - KIIs using semi￾structured instruments specific to given respondent category. - FGDs using semi￾structured instruments specific to given respondent category. - Content analysis of FGDs with direct and indirect beneficiaries by select subgroups to assess their perception of the activities in which they were involved - Content analysis of relevant KIIs (e.g., local community leaders, private sector actors) to assess their perception of the activities in which they or their peers were involved -Comparison of findings related to perception of project activities in the mid-term evaluation with those found in this round Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 29 Evaluation Questions Type of Answers Needed Data Source(s) Collection Methods Data Analysis Methods Unintended Consequences 4. What are the positive or negative unintended consequences of each of the projects, if any, and what ways are there to minimize negative consequences and capture positive ones? Descriptive -Direct and indirect project beneficiaries - Project documentation - MTE evaluation report - Project staff - USAID FFP staff - USAID Guatemala staff - Key Government of - Guatemala representatives/ staff at national and regional levels (Guatemala Ministry of Health and Ministry of Agriculture) - Municipal and communal staff and representatives - Staff at other NGOs and donors - Private sector actors - Local community leaders -ALs, MLs, NPs, APs - PAISANO PBS baseline/endline data - KIIs using semi-structured instruments specific to given respondent category. - FGDs using semi￾structured instruments specific to given respondent category. - Desk review, baseline and endline surveys, monitoring processes - Content analysis of FGDs with direct and indirect beneficiaries, and relevant KIIs (e.g., project staff, partners, USAID FFP staff) to identify and assess their views on negative or positive unintended consequences - Content analysis of select KIIs for lessons learned, adaptive management in project implementation to address such consequences, and recommendations to minimize negative consequences (if identified) - Analysis of project documents to identify unintended consequences, and of previous evaluation reports to assess whether any previously identified unintended consequences remain relevant and how their magnitude may have evolved; compare analyses -If applicable, will develop additional statistical analysis of PBS data to identify certain types of unintended consequences and quantify them Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Final Evaluation Protocol 30 Evaluation Questions Type of Answers Needed Data Source(s) Collection Methods Data Analysis Methods Lessons Learned 5. What key lessons learned and best practices should inform future projects in the country? Descriptive, normative - Project documentation - MTE evaluation report - Project staff - USAID FFP staff - USAID Guatemala staff - Staff at Guatemala’s Ministries of Health and Agriculture - Municipal and communal staff/ representatives - Private sector actors - Local community leaders - Desk review - KIIs using semi-structured instruments specific to given respondent category - FGD / mini workshop with project staff and stakeholders to elicit lessons learned/ best practices -Evaluation team analysis of project progress and changes/ solutions in response to challenges - Content analysis of relevant KIIs (e.g., project staff, project partners, USAID FFP staff) to assess their views on lessons learned and best practices for future design of FFP programs - Analysis of project documents to identify lessons learned and review of mid-term evaluation reports to compare progress and management of previous lessons and best practices, as well as identifying those that seem to have not held over time AL= agriculture leader, ML= mother leader, NP= nutrition promoter, AP = agriculture promoter Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 31 ANNEX B: PROTOCOL TOOLS PAISANO SO1 Questions for Key Informant Interviews (community Leaders, FFP staff, project staff, government officials) 1) How effective has training of community–based agricultural extension workers (Agriculture Leaders, or LA) been in improving farmer access to needed support? 2) What are some examples / evidence of actual women’s empowerment through PAISANO’s agriculture and livelihood activities? 3) What education and/or training opportunities has PAISANO created to promote livelihood opportunities among youth? 4) Are there examples of fee-for-service providers trained by PAISANO and now operating profitably? 5) What impact has PAISANO had on farmer access to government and private input suppliers? 6) What impact have livestock “Training and Reproduction Centers” had on adoption of improved practice and market access among participating households? 7) To what extent have households participating in PAISANO’s SO1 been able to identify and promote natural resource management practices actually adopted by participants a. What have been some of the challenges and how has PAISANO responded to these challenges? b. What (if any) impact have SO1 activities had on the local environment (watershed)? 8) How effective have PAISANO been in linking participating farmers to markets and enabling profitable and sustainable engagement in value chains? a. What have been the primary challenges? b. How effective has training been in enabling market engagement among participating farmers? 9) How effective has PAISANO been in promoting engagement in non-agricultural productive activities? a. Which off-farm activities show the greatest potential for generating income? Why? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 32 FGDs with Community Leaders / Grupos focales con lideres Comunitarios (COCODE, COCOSAN, COLOREDs) 1. What roles do you have with the committee? (Write a list of group members of each committee and roles). - How long have you been functioning as a committee? - What is the size of each committee? All committees? / Cual es el tamano de cada comite? Todos los comites? - How do you become a member of the committee? How long do you stay in the position? (for COCODE members: ask if they were elected for a community assembly) - How many males and females on each committee? - How do you become a community leader? - Does the COCODE president participate in the monthly COMUDE meetings? - Does COMUDE discuss Nutrition and Food Security needs of communities in order to stimulate government investment? - Are any members of your committee part of the COCODE too? If yes, who? - Do the COLRED members have an accreditation that is endorsed by the CONRED National Secretariat? - Does the committee have mechanisms to ensure accountability of its activities? If yes, please explain 2. What do you think the PAISANO project has been trying to help your community achieve? (probe) 3. Have you received any training to support the work you do? - From who? - What training topics? - What is your opinion of the training (what was most useful? Not useful? Probe) - Did committee members participate regularly in training sessions? (if no, please explain why) 4. Does PAISANO monitor your work? - How often? What do you do during the visits? - What is your opinion about the monitoring? Most useful activities? Not useful? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 33 5. Have the members participated in gender training? How may? - What is your opinion of the training? (probe) - What did you learn? (probe) - As a result of this training are there more women participating in the committees? 6. How do you work with the promoters and mother leaders? Agriculture leaders? 7. Do you think there has been a change in the life of the community since the project began working with it? - If yes, how has it changed? (probe for positive/ negative changes) 8. Has there been a change in the number of women participating in key positions in the COCOSANES since the project began? 9. What do you consider to have been the major constraints for the project to fully accomplish all of its purpose? (agriculture, MNCH, WASH etc.)? 10. How have you and the project managed these obstacles? - In your opinion, have you been successful? Why? Why not? 11. Which of the current activities implemented by the project will continue after PAISANO ends? Why? (probe) What is the community doing to ensure that? 12. How will beneficiaries be selected to participate in activities now supported by the project after the project ends? - MCHN - Agriculture - Livelihood - WASH 13. Do members of your community migrate seasonally or longer? -If yes – seasonally or long term? - who migrates (men, women, families)? - Why do they migrate? -where do they usually migrate to? (please respond by category of migrants) Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 34 14. During the past 3 years have there been any natural disasters or emergencies that have affected the community? (drought, floods, epidemics, insecurity etc.)? Please describe the problem and why it happened. - How did the community manage the problem? Did the project help? - were any actions taken to prevent future problems like this? - does the community have a map of risks? - Does the community have a preparedness and response plan for emergencies? If yes, do you know what is included in it? 15. Some people do not participate in project activities. Why is that? 16. Are there other development or emergency projects operating in your community during the past 3 years? - if yes, What projects? - How long have they been here? - What services do they provide? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 35 Interview with PAISANO (SAVE and PCI) Staff 1. In your opinion, what has SAVE/ PCI accomplished with this project? 2. What is your role in the project? - How long have you worked with it? 3. How useful has the DIP workplan been for your job? 4. What is your opinion of the project planning process? - Did the planning process change after the Mid term review? If yes, How? 5. Do you think the Mid-term review and recommendations have been helpful for your work? Yes? no? - Why or why not? - Did the project make any important changes as a result of the recommendations? If yes, what were they? 6. How would you describe the gender integration or non-integration in the project interventions? - Were any changes in the strategy made after the MTE? - What have been the strengths and challenges? 7. How often did you receive supervision visits? What did you do during the visits? - Did your supervisor use a guide/ checklist during the visit? - Did you receive recommendations on how to improve your work? - Did you have time to discuss any issues you had with implementation of activities? 8. What is your opinion of the management of project logistics (procurement and distribution of equipment, supplies, vehicles etc.) and its impact on project implementation? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 36 9. Can you describe the way that the project collected, reported and used data at the different project levels? - Please give examples of how this data was used to make managerial or technical decisions about project activities? 10. Did you receive any technical assistance or other resources from the project? If yes, Please describe them. 11. Looking back over the past 6 years, what do you think is the most significant change in the health of mothers and children in the project area? What factors do you think contributed to this change? - How, if at all, did the project contribute to this change? 12. Do you think the changes brought about by the project will be maintained after the project ends? If yes, which ones and why? (probe) - Have any of the project activities been scaled up to benefit other communities? Please describe. 13. Now that the project’s ending, do you think the project participants/ beneficiaries will continue to request the same services that have been supported by the project? Will the MLs and ALs continue in their roles? What about the WASH activities? 14. Do you believe that there were constraints that inhibited PAISANO from fully accomplishing its objectives? If yes, what were they? 15. Looking back on the project, what do you see as important lessons learned and best practices accomplished by the project? 16. Did you notice if there were any unintended consequences as a result of project activities? If yes, what were they? 17. Is there any other comment or observation you would like to make about the project? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 37 Ag/Livelihoods Focus Group Discussions (beneficiaries) Questions (First, note number of participants, number of women and number of men participants) 1. How many of you directly participated in PAISANO agriculture or livelihood activities? Please raise your hand. [count raised hands, noting number for men and for women] 2. How many of you directly participated in the following activities (including training)? [count raised hands for each, noting number for men and for women] a. Household gardening b. Production of improved bean seeds c. Micro or drip irrigation d. Composting or use of organic fertilizers e. Seedling production and/or tree planting f. Goat management and production g. Poultry management and production (including eggs production) h. Rabbit management and production i. Post-harvest storage j. Milk or cheese production k. Other processing [note commodity and processing activity] l. Product aggregation and marketing m. Savings and lending group n. Youth entrepreneur group o. Handicraft production p. Other agriculture activity _____________________________ q. Other non-agriculture activity _________________________ 3. How many of you have not directly participated in PAISANO agriculture or livelihood activities but think you have indirectly benefitted from the project? [count raised hands, noting number for men and for women] For those of you who indirectly benefitted, which activities did you benefit from? [list] Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 38 The following questions are for both direct and indirect beneficiaries 4. Did any PAISANO agriculture or livelihood activities help you increase the amount of food your family had to eat or the number of different types of food your family had to eat?  yes  no If yes, which activities? [list] 5. Did any PAISANO agriculture or livelihood activities help you to market your agricultural production better?  yes  no If yes, which activities? [list] 6. Which agriculture or livelihood activities of the project were most beneficial to you? [list] For each activity cited, why were they especially beneficial to you? [list reasons, by activity] 7. Did any PAISANO agriculture or livelihood activities help you increase your income?  yes  no If so, which activities? [list] a. What are your current sources of income for you and your family, other than activities promoted by PAISANO? b. Do you know of other opportunities for you to increase your income that you are not able to do? If so what are they and why can’t you do them? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 39 8. Do you think there are things PAISANO could have done to help you improve your food security that it didn’t do?  yes  no If yes, what are they? [list] 9. Were there any agriculture or livelihood activities promoted by PAISANO that you thought were good but were too difficult or time-consuming or expensive for you to do?  yes  no If yes, which ones? [specify if they were too difficult or time-consuming or costly] 10. Are there activities that you started to do during the PAISANO project that you will find difficult or impossible to continue now that the project is finished?  yes  no If yes, which ones and why will they be difficult to continue? [list activities and reasons] 11. Do you know of other sources of support that can help you to continue in future the activities you began under PAISANO?  yes  no If yes, what are they? [list] Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 40 WASH Tool 1—Questions for Beneficiaries & Observation Checklist WASH Program Activity ___________________________________________________________________________________ Please identify which of the following activities the [PAISANO] program worked with your household or your community on? Please describe the nature of the support provided (financial, technical, educational)? Please describe if the community has self-led and/or self-financed any of these activities. [ PAISANO Activity] Activity Description ☐ Sanitation hardware/ latrines ☐ Handwashing hardware/ stations ☐ Improvement of water supply system ☐ Source-/system-level water quality ☐ Establishing/training village water committee ☐ Education on water quality/hygiene & health ☐ Education on diarrhea prevention/treatment ☐ Household hygiene & behaviors, select all that apply:  Handwashing  Drinking water treatment & storage  Food preparation  Household cleaning  Latrine use  Solid waste disposal  Animal confinement/ animal waste  Baby/children’s feces Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 41 Follow-on question: Have both you and your spouse been able to participate in these activities? If not, why? _____ _____________________________________________________________________________ ________________ Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 42 WASH Software—Household Sanitation & Hygiene Behaviors ___________________________________________________________________________________ Water Quality How do you evaluate if your water is ‘safe’ for drinking? Select all that apply: ☐ Smell ☐ Taste ☐ Visible appearance ☐ Water quality test (either beneficiary or program-led) ☐ Other (describe): _________________________________________________________ Follow-on questions:  Do you think that your drinking water is causing illness in your family? Please explain. _______________________________________________________________________ _______ What do you usually do to make your water safe to drink? ☐ Nothing or I do not treat it. Why_____________________________________________ ☐ Filter or strain it (Describe system): __________________________________________ ☐ Add bleach/chlorine (How often do you do this? How much do you add to XXL of water?): ________________________________________________________________ ☐ Boil (Describe process): ____________________________________________________ ☐ Solar disinfection (describe the system): _______________________________________ ☐ Let it stand or settle (without adding any chemical) Follow-on questions:  How do you store the water after you treat the water? In what container/apparatus? How long do you continue to use the same water? _____________________________________________ _______________________________________________________________________ _______  How much does it cost you (either in money or materials/firewood) to treat the water using this method? ______________________________________________________________________ Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 43  How much would you be able/willing to pay each month if you did not have to treat the water at home? ______________________________________________________________________  Are there times during the [year/month/week] when you are not able to properly treat the water? If yes, when? What do you do when this happens?  Does your household have an ECOFILTER? Yes ☐ No ☐ If yes: - How much did your pay for it? Q ________________ - How long have you had it? Less than 2 years ☐ More than 2 years - Do you know when the vessel needs to be replaced? Yes ☐ No ☐ Describe__________ - Do you know where you can buy the replacements? Yes ☐ No ☐ Describe___________ - Are you saving money for it? Yes ☐ No ☐ How much per month? Q__________________ Sanitation Behavior  Does everyone in your household (except young children) use the toilet/latrine when they are home? o If not, where do they go to the bathroom? o What about at nighttime?  Are there times during the [week/month/year] that you or your family members are not able to use this latrine? e.g., When your spouse has migrated to work? Does this impact their sanitation behavior when they are home? During the rainy season if the latrine floods? Children’s Sanitation. The last time [name of child] passed stools, what was done to dispose of the stool? ☐ Child used toilet/latrine ☐ Put/rinsed into toilet/latrine ☐ Put/rinsed into drain or ditch ☐ Thrown into garbage ☐ Buried ☐ Left in the open ☐ Other (describe): _______________________________________________________ At what age did/will your child begin to use the latrine/toilet? ______ years Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 44 Handle of household waste.  Where do you storage the waste? ___________________________________  Where do you dispose it? __________________________________________ WASH Hardware ___________________________________________________________________________________ Sanitation hardware. Which type of toilet/latrine do you use at home? ☐ None (Describe if open defecation or alt: ______________________________________) ☐ Unimproved (Describe: ___________________________________________________) ☐ Latrine, improved pit with slab (Describe): ____________________________________) ☐ Latrine, ventilated improved pit (Describe): ___________________________________) ☐ Latrine, improved pour-flush or alt (Describe: __________________________________ _______________________________________________________________________ ) ☐ Flush toilet (Describe capture/treatment: _____________________________________ _______________________________________________________________________ ) ☐ Other (Describe: _________________________________________________________) Distance from latrine to house? _______meters Distance from latrine to a drinking-water source? ____________meters Follow-on, qualitative questions (interviewer’s discretion):  How did you decide to install this type of latrine/toilet? Were you provided any technical support when making this decision?  Were you provided any subsidy for this latrine/toilet when it was installed? What did you pay or provide for the latrine construction?  When did you install this latrine/toilet? What maintenance does it require, and how often? Do you have any problems with the latrine during the rainy season?  How satisfied are you with this latrine/toilet? Does it meet the needs of your family? Do you have any improvements planned you would like to complete for the toilet/latrine? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 45  Has the PAISANO program worked with your family, or other families in the community, on sanitation hardware? What have they provided—technical support, financial support, education?  Do family members use paper or similar to clean after using the latrine? Yes ☐ No ☐ If yes: Where do them dispose waste paper? ___________________________________________  Do you share latrine with other households? Yes ☐ No ☐ How many?_______________________  Is latrine’s superstructure used for storing grains at a period of time during the year? Yes ☐ No ☐ How long ?___________________ months. Handwashing Hardware. Do you have a hand washing station [in your home/near latrine]? ☐ Yes. Describe:____________________________________________________________ ☐ No. Where do you go to wash your hands? _____________________________________ Do you know when people should wash their hands? ☐ After use the latrine/toilet ☐ Before breastfeeding or feed children ☐ Before cooking Other _______________________________ Follow-on questions: Where do you get water for hand washing and personal hygiene?____________________________________ Do family members use soap for hand-washing? Observe________________ Is there waste water around the hand-washing station? Observe___________ Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 46 Water Supply ___________________________________________________________________________________ Domestic drinking water source (select all used; note which are used for drinking/cooking/personal hygiene): ☐ Surface water- river, dam, lake, pond (Describe source): _________________________________________________________________ ☐ Spring (Describe shallow vs. protected): _________________________________________________________________ ☐ Rainwater (Describe roof and guttering/collection system): _________________________________________________________________ ☐ Community well (Describe: shallow/deep source): _________________________________________________________________ ☐ Piped water to yard or nearby plot (Describe location and no. households sharing): _________________________________________________________________ ☐ Piped water into dwelling (Describe if full piping, or single point): _________________________________________________________________ Follow-on questions:     How long does it take to go there, collect water, and return? __________ min Who usually goes to the water source to fetch the water for your household? (Describe gender and age) _______________________________________________________________________ Are there times during the year when your primary water source(s) is not available? When does this occur and for how long? Where do you go for water when this happens? _______________________________________________________________________ Do you use this water source for your productive needs (irrigation, livestock watering, coffee processing, etc.) as well? If so, can you approximate how much water you use for productive purposes each [day/week/month]? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 47 Water system operations & maintenance (qualitative):  Do you know who owns and is responsible for maintaining your water system? ______________ _______________________________________________________________________ _______ o Follow-on if the ‘village’ is identified as the owner: How does the village come together to make decisions regarding the water system? ________________________________ _________________________________________________________________ _______ o Follow-on if the ‘village water committee’ (or equivalent is identified): What are the responsibilities of this committee? How are decisions made regarding the water system? _________________________________________________________________ _________________________________________________________________ ______________  Has your water system been broken or non-functional in the previous two years? If so, for how long? By what process was the system repaired, and who paid for the repair? _______________________________________________________________________ _______ _______________________________________________________________________ _______  Did your household pay for construction (‘fondo revolvente’) of the water system? _______________________________________________________________________ _______ Follow-on questions if ‘yes’: o How much did your household pay for construction? Q _______ o Do you know how much the system cost to construct? Q __________  Do you pay for operations and maintenance (‘fondo privativo’) of the water system? _______________________________________________________________________ _______ Follow-on questions if ‘yes’: o How much does your household pay each month? Q _______ o Do all households pay the same amount? Is anyone exempted from paying for water? (note how many households in the village: ______) o When are fees collected? ___________________________________________________ o Do you know where the money goes and what it is used for? _________________________________________________________________ _______ o How much does your household pay for other services (electricity, cellphone, cable, etc)______________________________________________________________ _______ Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 48 o Why do you think people usually pay more for other services?__________________________________________________________ _______ Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 49 WASH Tool 2—Questions for the Water Committee members WASH Program Activity ___________________________________________________________________________________ Please identify which of the following activities the [PAISANO] program worked with members of the Water Committee? Please describe the nature of the support provided (financial, technical, educational)? Please describe if the community has self-led and/or self-financed any of these activities. [ PAISANO Activity] Activity Description ☐ Establishing of a Water Committee ☐ Training for WASH Committee members (select topics):  Treatment methods for water  Management and maintenance of water supply systems  Budgeting  Monitoring and testing of water quality  Social and behavior change for sustainability of water supply systems.  Others:_________________________ ________________________ ☐ Linkage with GO and NGO partners for sustainability of the community water system Which?________________________________ ☐ Testing of water quality ☐ Establishing of a sustainability plan for community water system. Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 50 Governance:  Were the Water Committee members elected by the community assembly? Yes ☐ When? (date)_________________________________________ No ☐ Why, and how were you elected? ________________________________________________  Was an act of constitution of the Water committee written in the COCODE minutes book? Yes ☐ No ☐ Why?__________________________________________________________________ _____  Are any of the Water Committee also a members of the COCODE? Yes ☐ No ☐  Does the Water Committee have a legal approval to collect fees from the community? Yes ☐ No ☐  Does the Water Committee have authorized receipts? Yes ☐ No ☐  Does the Water Committee have a mechanism for accountability? Yes ☐ No ☐  How many women are members of the Water Committee and what positions do they occupy? _______________________________________________________________________ _____________ If no one, explain why__________________________________________________________________ Training process  How many training sessions did you receive?__________________________________________________  How often were the training sessions given?__________________________________________________  Do you consider that the methodology used was adequate to foster learning? Yes ☐ No ☐ Why? _____________________________________________________________________ Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 51         What could be improved? _________________________________________________________________ The knowledge and skills received during the training process allowed members of Water Committee to assume the roles of each one? Yes ☐ How? _______________________________________________________________________ ___ No ☐ Why?__________________________________________________________________ _________ What factors do you think facilitated learning? ________________________________________________ _______________________________________________________________________ _______________ What factors should be improved? _________________________________________________________ _______________________________________________________________________ _______________ Do you consider that the training schedules were adequate? Yes ☐ No ☐ Why? _____________________________________________________________________ ___________________________________________________________ _____________ Was there continuous participation of all committee members? Yes ☐ No ☐ Why? _____________________________________________________________________ Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 52 Sustainability       Was an operation and maintenance fee established through a community assembly? Yes ☐ No ☐ What criteria were used to calculate the fee? _________________________________________________ _______________________________________________________________________ _______________ Does the fee enough for cover all the expenses relates to operation and maintenance? Yes ☐ No ☐ If not: Why do you think the community does not accept an adequate fee?_________________________ _______________________________________________________________________ _______________ Why do you think the community pays higher fees for other less important services (electricity, cellphone, cable) _______________________________________________________________________ _________ _______________________________________________________________________ _______________ Were Agreements established with other partners (GOs and NOGs) for the operation and maintenance of the system? Yes ☐ No ☐ Have you developed plans for effective operation and maintenance of the water system? Yes ☐ No ☐ Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 53 WASH Tool 3—Questions for Partner Staff 1) The ICF baseline study revealed that significant challenges exist in relation to both WASH access and utilization, especially in relation to water quality and hygiene practices. The Mid-term evaluation revealed that the PAISANO activities did not prioritize WASH as an important issue in order to reduce high rates of diarrhea in children under 5 years of age. What Did PAISANO program strategy change? Follow-on question: a) What activities have been added? 2) What have the partners done in formative research to determine the primary routes for fecal-oral contamination and the diarrhea rates? (E.g. helminth transmitted infections via soil or animal manure that is outside ). Follow-on question: a) How was the generated information incorporated into their WASH strategy and Theory of Change? 3) What were PAISANO targets for the following indicators in target beneficiary population by the end of the project: a) Access to Improved Drinking Water? b) Access to Improved Sanitation? c) Prevalence of handwashing stations? d) Diarrheal rates in children < 5 yr? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 54 4) What were the major accomplishments and weaknesses to date related to WASH (hardware and software) interventions and what implementation changes were made in response? Follow-up questions would be: a) Did the project use evidence-based behavior change theory of change? What are the barriers to behavior change? What has been successful in areas? 5) Were all expected WASH results likely to be achieved by the completion of the project, and if not, what changes in targeted results and/or implementation approaches would be adapted in future programs? Follow-up questions would be: a) Was the project reached the desired beneficiaries? If not, why? 6) What strategies were employed to influence the WASH behaviors of men, women and children? What were these strategies’ levels of successes in influencing the target population’s WASH behaviors? Follow-up question would be: a) How did PAISANO modify their approach to include men in SBCC and other programming, taking in count that a big quantity of them migrate for work? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 55 7) What was the quality of behavior change sessions, for example, interactive sessions, session length, topic selection, quality and use of visual aids, quality and use of flipcharts, participation and facilitator’s demonstrated knowledge of the topic? Follow-on question: a) Were any of these SBCC sessions reinforced in other places/times, e.g. at schools or health centers? 8) Which WASH activities were most effective in addressing assessed needs? Which activities were not successful and why? 9) What strategies were employed to engage local or regional government bodies in WASH access? Follow-up questions: a) How did PAISANO strengthen technical or regulatory capacity of these, in accordance with the government’s decentralization strategy? b) How did PAISANO engage them in water quality and long-term O&M of water systems? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 56 10) Which communication methods were preferred by women and men to receive the information they need about WASH activities? Were these formal or informal? a) If appropriate, did PAISANO adopt SBCC messaging to include menstrual hygiene management for young girls/women? 11) Did the program build, extend, or rehabilitate the community’s water system(s)5? What type of system? Was any of the systems non-functional since they were constructed? If so, for how long? How was the systems maintained? Who was benefiting from the water systems? Who was excluded? a) Are communities able to use the water systems for productive, as well as domestic uses? If so, at what level and for what purpose? If not, why? 12) How was water sources been maintained? What was the distribution of responsibilities between the community management structures and local/municipal government bodies? Were there examples of community-maintained water sources? What was the key factors behind the success of this model? What were the challenges in community-led management of water points? How can those challenges be addressed in the future? 5 ICF’s baseline assessment reported 48% access to improved drinking water, but the source was of unreliable supply, reducing overall access to 19.9%. Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 57 13) Were there households in the communities that do not receive this water supply? If yes, how did these households get water? Why didn’t these household receive water? 14) Is there a functioning water committee? Does the water committee have a savings account? Does the water committee collect monthly fees from households to maintain the water system? Are the monthly fees collected enough to pay for the operation and maintenance of the water system? 15) Where did the necessary funds come from for the water system’s maintenance or repair? What is the basis of the tariff structure? Are there differences between women and men in willingness and capacity to pay for water services? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 58 16) What was the level of staff training to promote health and hygiene interventions? Did they have adequate knowledge about behavior change and its links to program hierarchy? What additional training, if any, would be recommended for program staff? 17) What are the learned lessons that you observed during the execution of the Program activities? What should it be improved in order to get more effectiveness for WASH activities? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 59 FGD Guide for Mothers & Fathers (MCHN) Entrevistador(es):_________________________ Notas:_______________________________________ Traductor(es): ______________ Fecha: 9/_______ Ubicación________________ ID#: _______ 1. Are you familiar with the PAISANO project? (Describe project, if needed.) ¿Conoce el proyecto PAISANO? (Describa el proyecto se necesita.] Anote el numero de personas que dicen: Sí ____ o No______ 2. Por favor, puede indicar la edad de su hijo más pequeño? (Anote las edades:______________________________ 3. Can you explain what the PAISANO project sought to accomplish? ¿Puede explicar lo que el proyecto PAISANO quería lograr? 4. Please describe what kind of services or benefits you or your family have received from PAISANO? How long has it been since you last received these services or benefits? Por favor, describa qué tipo de servicios o benefícios usted o su familia ha recibido del proyecto PAISANO. ¿Cuanto tiempo ha pasado desde que recibió estos servicios o benefícios? 5. Please describe any ways the project negatively affected you or any member of your family. Por favor, explique en el caso de que el proyecto ha tenido algún efecto negativo para Ud. o su familia. 6. Which of PAISANO’s activities, if any, were most helpful for your family? In what ways were they helpful? Are there any ways that these activities (mentioned by the respondent) will continue to benefit your family in the future although PAISANO ends? How? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 60 7. If you participated in a nutrition school or received home visits, can you mention any ways that this changed how you take care of your children or take care of yourself? ¿Cuéntenos que…….] 8. Of the behaviors learned from PAISANO about caring for children or yourself, which practices do you believe you will continue, and why? [¿De los comportamientos que Ud. aprendió de PAISANO para cuidar a sus hijos o si misma, cuales practicas piensa seguir practicando y porque? ¿Cuales no?] 9. What types of PAISANO’s activities taught you about the value of consuming eggs, goat milk and rabbit? Are any of these foods consumed more in your community since the beginning of PAISANO? Which foods? Since PAISANO started, are any consumed more by your family? Why or why not? 10. At what age do you consider it acceptable to feed a child eggs? Rabbit meat? Goat milk? Does your youngest child consume any of these foods? Why or why not? If yes, where do you get them from? 11. What do mothers and fathers do in this community when they notice that their child has danger signs? What medical assistance is available, what distance, and what schedule is it available? Why do some parents not seek medical help? 12. How has PAISANO promoted women’s participation in community activities? If there have been changes in women’s participation, please describe the change. Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 61 13. How has PAISANO promoted different ways of husbands and wives coordinating together in their families? (For reference only: shared decision making about spending & health, sharing tasks). If there have been any changes for husbands and wives in your community, please describe the change. 14. Did your family receive food and/or cash assistance as a participant in PAISANO? (Specifiy if food, cash or both). Why do you think that PAISANO provided this support to you? How did this support impact your family? 15. In your opinion, is it preferable to receive a transfer of food or of cash? Do you have any comments about the advantages or disadvantages of either food or cash assistance? Did you face any challenges with the process for receiving the cash transfer? 16. ONLY participants who received cash: What did you typically spend the cash transfer for? If you bought food, were they foods you normally buy or new foods? (Probe: which foods) What other items did you purchase? Will you continue to purchase these items in the future? 17. At any point did you stop receiving some benefits or services from PAISANO? Why? (Probe if due to child’s age or change in criteria). How did PAISANO communicate this change? Which benefits/services did you stop receiving? Which benefits/services could you continue to receive from PAISANO? Did you stop participating in any of PAISANO’s activities? What are the reasons? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 62 18. Please mention the most important changes you see in the health of pregnant women and children in this community since PAISANO began. What factors contributed to these changes? How did PAISANO contribute to this change? Por favor, mencione los cambios más significativos en la salud de madres y niños en su comunidad. ¿Cuáles son algunos factores que han contribuido? ¿Como considera Ud. que PAISANO haya contribuido a este cambio? 19. When PAISANO ends, which of these changes in the health of pregnant women and children are likely to continue and which ones are not likely to continue? (Probe: activities of volunteers, referral system, families’ health practices). 20. How effective do you believe that PAISANO has been in your community at reducing child malnutrition? What activities have been the most effective? ¿Desde su punto de vista, que tan efectivo ha sido el proyecto en lograr sus propósito de reducir la desnutricion infantil? ¿Cuales actividades had sido mas exitosas? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 63 KII Guide with MCHN Participant (mothers/fathers) Entrevistador(es):____________________________ Ages of children in HH:_________________________ Traductor(es): ______________ Fecha: 9/_______ Código de ubicación________________ ID#: _______ 1. Are you familiar with the PAISANO project? (Describe the project, if the name is not familiar.) [¿Conoce el proyecto PAISANO? (Describa el proyecto si el nombre no es familiar.] Sí ______ No______ 2. Can you explain what the PAISANO project sought to accomplish? ¿Puede explicar lo que el proyecto PAISANO quería lograr? 3. Please describe what kind of services or benefits you or your family have received from PAISANO? How long has it been since you received these services or benefits? Por favor, describa qué tipo de servicios o benefícios usted o su familia ha recibido del proyecto PAISANO. ¿Cuanto tiempo ha pasado desde que recibió estos servicios o benefícios? 4. Please describe any ways the project negatively affected you or any member of your family. Por favor, explique en el caso de que el proyecto ha tenido algún efecto negativo para Ud. o su familia. 5. Which of PAISANO’s activities, if any, were most helpful for your family? In what ways were they helpful? Are there any ways that these activities (mentioned by the respondent) will continue to benefit your family in the future although PAISANO ends? How? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 64 6. If you participated in a nutrition school or received home visits, can you mention any ways that this changed how you take care of your children? ¿Cuéntenos que……. ] 7. Of the behaviors learned from PAISANO about caring for children or yourself, which practices do you believe you will continue, and why? [¿De los comportamientos que Ud. aprendió de PAISANO para cuidar a sus hijos o si misma, cuales practicas piensa seguir practicando y porque? ¿Cuales no?] 8. What types of PAISANO’s activities taught you about the value of consuming eggs, goat milk and rabbit? Are any of these foods consumed more in your community since the beginning of PAISANO? Which foods? Are any consumed more by your family since PAISANO started? Why or why? 9. At what age do you consider it acceptable to feed a child eggs? Rabbit meat? Goat milk? Does your youngest child consume any of these foods? Why or why not? If yes, where do you get them from? 10. What do mothers and fathers do when they notice that their child has danger signs? What medical assistance is available, and how far away, and what schedule is it available? Why do some parents not seek assistance? 11. How has PAISANO promoted women’s participation in community activities? If there have been changes in women’s participation, please describe the change. Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 65 12. Did your family receive food and/or cash assistance as a participant in PAISANO? (Specifiy if food, cash or both). Why do you think that PAISANO provided this support to you? How did this support impact your family? 13. In your opinion, is it preferable to receive a transfer of food or of cash? Do you have any comments about the advantages or disadvantages of either food or cash assistance? 14. ONLY if you received cash, did you face any challenges with the process for receiving the cash transfer? Did you need to open a bank account? What did you typically spend the cash transfer for? If you bought food, were they foods you normally buy or new foods? (Probe: which foods) What other items did you purchase? Will you continue to purchase these items in the future? 15. At any point did you stop receiving some benefits or services from PAISANO? Why? (Probe if due to child’s age or change in criteria). How did PAISANO communicate this change? Which benefits/services did you stop receiving? Which benefits/services could you continue to receive from PAISANO? Did you stop participating in any of PAISANO’s activities? What are the reasons? 16. Please mention the most important changes you see in the health of pregnant women and children in this community since PAISANO began. What factors contributed to these changes? How did PAISANO contribute to this change? Por favor, mencione los cambios más significativos en la salud de madres y niños en su comunidad. ¿Cuáles son algunos factores que han contribuido? ¿Como considera Ud. que PAISANO haya contribuido a este cambio? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 66 17. When PAISANO ends, which of these changes in the health of pregnant women and children are likely to continue and which ones are not likely to continue? (Probe: activities of volunteers, referral system, families’ health practices). 18. How effective do you believe that PAISANO has been in your community at reducing child malnutrition? What activities have been the most effective? ¿Desde su punto de vista, que tan efectivo ha sido el proyecto en lograr sus propósito de reducir la desnutricion infantil? ¿Cuales actividades had sido mas exitosas? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 67 FDG with Mothers Leaders or Nutrition Promoters (MCHN) Entrevistador(es):____________________________ Número de participantes:___________________________ Traductor(es): ______________ Fecha: 9/_______ Ubicación________________________ ID#: _______ 1. What is or was your role with PAISANO? Note number of ML: _____ NP:______ 2. Are you currently active as a volunteer? Note number of Active:____ No longer active:____ How long have you/did you perform your volunteer role? 3. What are the reasons that motivated you to be a volunteer with PAISANO? 4. What are types the activities you carried out as a volunteer with PAISANO? Where did you work and with whom? Which community groups did you coordinate with to carry out your role? How did you coordinate? 5. Please tell us how you used what you learned from the project to carry out your volunteer activities. Specifically, can you tell us about the training and materials you received from PAISANO to do your job? a. Assess training: Frequency? How effective was training? How could it have been improved? b. Materials: how useful were they for your job, are they understandable, do mothers like them? How often did you use them? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 68 6. How did your activities (e.g. census of pregnant women, referral and counter referral, assessing sick children or growth monitoring, home visits, etc.) contribute to your community? What challenges did you face in carrying out these activities? 7. Which of your activities, if any, were most helpful for the communities you worked for? ____ In what ways were they helpful? Are there any ways that these activities (mentioned by the respondent) will continue to benefit your community in the future although PAISANO ends? How? 8. Which of your activities, if any, do you intend to continue after PAISANO ends? Are there any people or organizations that will assist you? Which ones and how? Will any organization continue to train or financially support you? 9. Since PAISANO began, what changes do you see in the way mothers and fathers feed children under age two? Considering promotion of behavior change, as well as agriculture, animal raising or economic activities, which of PAISANO’s activities do you think was most effective for helping parents improve child feeding? Were any activities less effective? 10. What types of PAISANO’s activities taught about the value of consuming eggs, goat milk and rabbit? Are any of these foods consumed more by families since the beginning of PAISANO? Which foods? Are any consumed more by young children PAISANO started? Why or why? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 69 11. What were the main challenges faced for the promotion of exclusive breastfeeding by PAISANO? How did PAISANO’s activities address these challenges? Was it effective? 12. How did PAISANO promoted women’s participation in community activities? If there have been changes in women’s participation, please describe the changes. 13. In terms of gender roles, how did PAISANO promote different ways of husbands and wives working together in their families? (For reference only, do not read: shared decision making about spending & health, sharing tasks). If there have been any changes for husbands and wives’ gender roles in your community, please describe the change. 14. During your time as a volunteer, were your volunteer activities affected by changes within the MOH health? How? In what ways do you collaborate with MOH health workers? How do you think MOH health workers view your work as a volunteer? 15. What do you think are the most significant changes in the health of pregnant women and children in this community since PAISANO began? What factors do you think contributed to these changes? Por favor, mencione los cambios más significativos en la salud de madres y niños en su comunidade desde que empezo PAISANO. ¿Cuáles son algunos factores que han contribuido? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 70 16. How (if at all) did the project contribute to this change? ¿Como considera Ud. que PAISANO haya contribuido a este cambio? 17. When PAISANO ends, which of these changes in the health of pregnant women and children are likely to continue and which ones are not likely to continue? (Probe: activities of volunteers, referral system, families’ health practices). 18. What challenges did PAISANO face in accomplishing its plans? What did it do to manage the challenges? 19. How effective do you believe that PAISANO has been in your community at reducing child malnutrition? What activities have been the most effective? ¿Desde su punto de vista, que tan efectivo ha sido el proyecto en lograr sus propósito de reducir la desnutricion infantil? ¿Cuales actividades had sido mas exitosas? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 71 FDG with Mothers Leaders or Nutrition Promoters (MCHN) Entrevistador(es):____________________________ Número de participantes:___________________________ Traductor(es): ______________ Fecha: 9/_______ Ubicación________________________ ID#: _______ 1. What is or was your role with PAISANO? Note number of ML: _____ NP:______ 2. Are you currently active as a volunteer? Note number of Active:____ No longer active:____ How long have you/did you perform your volunteer role? 3. What are the reasons that motivated you to be a volunteer with PAISANO? 4. What are types the activities you carried out as a volunteer with PAISANO? Where did you work and with whom? Which community groups did you coordinate with to carry out your role? How did you coordinate? 5. Please tell us how you used what you learned from the project to carry out your volunteer activities. Specifically, can you tell us about the training and materials you received from PAISANO to do your job? a. Assess training: Frequency? How effective was training? How could it have been improved? b. Materials: how useful were they for your job, are they understandable, do mothers like them? How often did you use them? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 72 6. How did your activities (e.g. census of pregnant women, referral and counter referral, assessing sick children or growth monitoring, home visits, etc.) contribute to your community? What challenges did you face in carrying out these activities? 7. Which of your activities, if any, were most helpful for the communities you worked for? ____ In what ways were they helpful? Are there any ways that these activities (mentioned by the respondent) will continue to benefit your community in the future although PAISANO ends? How? 8. Which of your activities, if any, do you intend to continue after PAISANO ends? Are there any people or organizations that will assist you? Which ones and how? Will any organization continue to train or financially support you? 9. Since PAISANO began, what changes do you see in the way mothers and fathers feed children under age two? Considering promotion of behavior change, as well as agriculture, animal raising or economic activities, which of PAISANO’s activities do you think was most effective for helping parents improve child feeding? Were any activities less effective? 10. What types of PAISANO’s activities taught about the value of consuming eggs, goat milk and rabbit? Are any of these foods consumed more by families since the beginning of PAISANO? Which foods? Are any consumed more by young children PAISANO started? Why or why? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 73 11. What were the main challenges faced for the promotion of exclusive breastfeeding by PAISANO? How did PAISANO’s activities address these challenges? Was it effective? 12. How did PAISANO promoted women’s participation in community activities? If there have been changes in women’s participation, please describe the changes. 13. In terms of gender roles, how did PAISANO promote different ways of husbands and wives working together in their families? (For reference only, do not read: shared decision making about spending & health, sharing tasks). If there have been any changes for husbands and wives’ gender roles in your community, please describe the change. 14. During your time as a volunteer, were your volunteer activities affected by changes within the MOH health? How? In what ways do you collaborate with MOH health workers? How do you think MOH health workers view your work as a volunteer? 15. What do you think are the most significant changes in the health of pregnant women and children in this community since PAISANO began? What factors do you think contributed to these changes? Por favor, mencione los cambios más significativos en la salud de madres y niños en su comunidade desde que empezo PAISANO. ¿Cuáles son algunos factores que han contribuido? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 74 16. How (if at all) did the project contribute to this change? ¿Como considera Ud. que PAISANO haya contribuido a este cambio? 17. When PAISANO ends, which of these changes in the health of pregnant women and children are likely to continue and which ones are not likely to continue? (Probe: activities of volunteers, referral system, families’ health practices). 18. What challenges did PAISANO face in accomplishing its plans? What did it do to manage the challenges? 19. How effective do you believe that PAISANO has been in your community at reducing child malnutrition? What activities have been the most effective? ¿Desde su punto de vista, que tan efectivo ha sido el proyecto en lograr sus propósito de reducir la desnutricion infantil? ¿Cuales actividades had sido mas exitosas? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 75 KII with PAISANO Health/Nutrition Staff Notas:_________________________________ Entrevistador: _________________ Fecha: 9/________ Código de ubicación________________ ID#: _______ 1. Please describe your role(s) with PAISANO and length of time. 2. Which of PAISANO’s activities you are most familiar with? 3. In your role(s) with PAISANO, which community/governmental groups, MOH services or NGOs did you coordinate with? Which issues/topics did you coordinate? How frequently? What was the goal of this coordination? How effective was the coordination? 4. To what extent did the training you received from PAISANO help you or your staff to carry out your job? Which types of training were most, or least effective? (Probe: examples) 5. What materials PAISANO produced were most useful for carrying out your job or your staff’s job? How were they used? Which materials were less useful? 6. Please mention PAISANO’s activities that you consider more effective at improving child health and nutrition, and why. (Probe examples). Do you expect that these strategies or activities will be continued? By whom? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 76 7. Please mention any of PAISANO’s activities that you consider less effective at improving health and nutrition, and what challenges these activities faced. 8. How effective do you think PAISANO has been at strengthening the referral and counter-referral system in the communities? Please explain. What do you expect will happen to the referral system after PAISANO closes? Why? 9. PAISANO used a range of strategies to help families improve children’s diets. Which strategies do you think were most and least effective? (If needed, probe agricultural production, animal raising, food or cash assistance, recipe demonstrations, home visits, radio, etc.). 10. Regarding PAISANO’s promotion of animal source foods for children under age two, which foods did families include into children’s diets more? Why were some promoted foods not fed to children? (If needed, probe cost or acceptability of feeding children rabbit, goat milk, eggs, chicken). 11. Comparing PAISANO’s support to some families to raise rabbits, goats, or chickens, which animal activity was most useful for improving children’s diets? Why? 12. What challenges did PAISANO faced in encouraging men and women to participate? How did the project try to overcome these challenges? (If not mentioned, probe how the project addressed women’s time burden). Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 77 13. In your view, were PAISANO’s activities affected by changes in the health system (e.g. end of PEC, new MOH model)? Are you aware how PAISANO adapted its activities to manage these changes? If so, which of PAISANO’s adaptations were beneficial and which were not? 14. Which community health activities supported by PAISANO do you think should continue after PAISANO ends? Which activities do you expect to continue? Will the activities change? (Probe: training of volunteers, GMP, health education, home visits, referrals and follow up). 15. Which community groups, government bodies or NGOs do you expect will continue PAISANO’s health and nutrition activities? Which ones? (Probe: volunteer training, referral system, health promotion, financial or other incentives). 16. Do you have any other comments about your participation in PAISANO? Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 78 KII with PAISANO MCHN partners (e.g. MOH, SESAN, Caritas) 1. How familiar are you with the PAISANO project? Can you explain what this project seeks to accomplish? (Describe the project, if the name is not familiar.) 2. What is/has been your involvement or coordination with the project? Please describe your role and how for long. 3. What kind of information has PAISANO shared with your organization? Was sharing regular? How was information used? 4. What kind of materials has PAISANO provided to you/your organization? (Specify). How are they used? How useful do you consider these materials? 5. If you or your staff have been trained by PAISANO, what topics were covered? To what extent did the training help you or your staff carry out your job? (Probe for examples). 6. Please mention any of PAISANO’s activities that you consider effective at improving health and nutrition, and why. (Probe examples). Do you expect that these strategies or activities will be continued? By whom? 7. Please mention any of PAISANO’s activities that you consider less effective at improving health and nutrition, and what challenges these activities faced. Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 79 8. How effective do you think PAISANO has been at strengthening the referral and counter-referral system in the communities where it works? What do you expect will happen to the referral system after PAISANO closes? Why? 9. In your view, were PAISANO’s activities affected by changes in the health system (e.g. end of PEC, new MOH model)? Are you aware how PAISANO adapted its activities to manage these changes? If so, which of PAISANO’s adaptations were beneficial and which were not? 10.Which community health activities supported by PAISANO do you think should continue after PAISANO ends? Which activities do you expect to continue? Will the activities change? (Probe: training of volunteers, GMP, health education, home visits, referrals and follow up). What support do you/your organization plan to provide to continue activities and which ones? (Probe: volunteer training, resources, financial or other incentives). Final Performance Evaluation of Western Program of Integrated Food Security and Food Actions (Programa de Acciones Integrados de Seguridad Alimentaria y Nutricional del Occidente [PAISANO]) — Draft Evaluation Protocol 80 ANNEX C: SELECTED SITES Date Mayan Language Community Municipality Department Participants / Stakeholders Activity Program MCH Livelihoods WASH Resilience Marketing Savings S/19 K’iché Xepol Chichicastenango El Quiché Participants, COCODE￾COSAN leaders X X X X S/20 K’iché Chiul Cunén El Quiché Participants, COCODE￾COSAN leaders, agricultural leaders X X X X S/21 K’iché La Hacienda Cunén El Quiché Mother leaders, marketing groups, COCODE-WASH leaders X X X X S/22 Ixil Xix Chajul El Quiché COCODE-WASH leaders, participants X X S/24 Ixil La Pista Nebaj El Quiché COCODE-WASH leaders, participants X X X S/25 Spanish Ojo de Agua Chiantla Huehuetenango Mother leaders, agricultural leaders, participants, Empowered women X X X X S/26 Mam Duraznales Concepción Chiquirichapa Quetzaltenangp Participants, COCODE￾COSAN leaders X X X S/27 Mam Monrovia Sn. Juan Ostuncalco Quetzalltenango Agricultural leaders, COCODE-WASH leaders X X Annex 4: Tabular Summary of Indicators and Change from Baseline to Endline Table A4.1. FFP Guatemala Endline Indicators - PAISANO Project Area Indicator Value Confidence Interval Number of Records Weighted Population Standard Deviation Standard Lower Upper Error DEFT FOOD SECURITY INDICATORS Average Household Dietary Diversity Score (HDDS) 7.0 6.8 7.2 1,402 27,348 1.8 0.10 2.0 Prevalence of households with moderate or severe hunger (HHS) 6.0 4.4 7.5 1,534 29,815 23.7 0.77 1.3 Male and female adults 6.0 4.4 7.6 1,324 25,894 23.7 0.80 1.2 Adult female, no adult male 5.7 2.4 9.1 192 3,610 23.7 1.64 1.0 Adult male, no adult female NA NA NA 16 269 NA NA NA Child, no adults NA NA NA 2 42 NA NA NA Prevalence of moderate and severe food insecurity in the population, based on the Food Insecurity Experience Scale (FIES) [30 day recall] 31.0 26.7 35.2 1,534 29,815 36.9 2.16 2.4 Male and female adults 30.8 26.6 34.9 1,324 25,894 36.8 2.13 2.2 Adult female, no adult male 32.4 25.5 39.4 192 3,610 38.1 3.54 1.3 Adult male, no adult female NA NA NA 16 269 NA NA NA Child, no adults NA NA NA 2 42 NA NA NA POVERTY INDICATORS Per capita expenditures (as a proxy for income) of USG-assisted areas $2.54 $2.35 $2.74 8,902 173,968 $1.44 0.10 2.6 Male and female adults $2.49 $2.29 $2.68 8,159 159,987 $1.31 0.09 2.6 Adult female, no adult male $3.21 $2.79 $3.63 690 13,055 $2.70 0.21 1.1 Adult male, no adult female NA NA NA 46 797 NA NA NA Child, no adults NA NA NA 7 128 NA NA NA Prevalence of poverty: Percent of people living on less than $1.25/day 14.7 9.8 19.6 8,902 173,968 35.4 2.39 2.7 Male and female adults 14.9 9.9 19.9 8,159 159,987 34.4 2.47 2.6 Adult female, no adult male 11.2 5.1 17.2 690 13,055 40.8 2.97 1.0 Adult male, no adult female NA NA NA 46 797 NA NA NA Child, no adults NA NA NA 7 128 NA NA NA Mean depth of poverty 2.6 1.7 3.4 8,902 173,968 9.0 0.44 1.9 Male and female adults 2.5 1.6 3.4 8,159 159,987 8.1 0.46 2.1 Adult female, no adult male 2.2 0.4 4.1 690 13,055 14.1 0.91 0.9 Adult male, no adult female NA NA NA 46 797 NA NA NA Child, no adults NA NA NA 7 128 NA NA NA WASH INDICATORS Percentage of households using an improved source of drinking water 41.8 33.3 50.3 1,536 29,868 49.3 4.17 3.3 Percentage of households using improved sanitation facilities 66.7 61.1 72.2 1,536 29,868 47.2 2.73 2.3 Percentage of households with soap and water at a handwashing station commonly used by family members 80.3 74.8 85.7 1,536 29,868 39.8 2.68 2.6 AGRICULTURAL INDICATORS Percentage of farmers who used financial services in the past 12 months 21.1 16.2 25.9 1,418 48,746 40.8 2.37 2.2 Percentage of farmers who practiced value chain activities promoted by the project in the past 12 months 28.3 22.9 33.8 1,418 48,746 45.1 2.68 2.2 Percentage of farmers who used three sustainable agricultural practices in the past 12 months 60.5 53.9 67.1 1,418 48,746 48.9 3.22 2.5 Percentage of farmers who used at least two sustainable crop practices in the past 12 months 65.5 57.8 73.2 1,418 48,746 47.6 3.78 3.0 Percentage of farmers who used at least two sustainable livestock practices in the past 12 months 14.9 11.0 18.8 1,418 48,746 35.6 1.90 2.0 Percentage of farmers who used at least two sustainable NRM practices in the past 12 months 1.0 0.3 1.7 1,418 48,746 9.9 0.34 1.3 Percentage of farmers who used improved storage practices in the past 12 months 18.9 14.4 23.4 1,418 48,746 39.2 2.22 2.1 WOMEN'S HEALTH AND NUTRITION INDICATORS Prevalence of underweight women 1.4 0.7 2.1 1,987 39,447 11.6 0.34 1.3 Women’s Dietary Diversity Score (WDDS) 4.8 4.6 5.0 2,166 44,945 1.5 0.09 2.6 CHILDREN'S HEALTH AND NUTRITION INDICATORS Prevalence of underweight children under 5 years of age (Total) 25.8 22.4 29.2 1,265 25,554 43.8 1.66 1.3 Indicators, 95 Percent Confidence Intervals and Base Population [Guatemala, 2018] Page 1 Lower Upper Table A4.1. FFP Guatemala Endline Indicators - PAISANO Project Area Indicators, 95 Percent Confidence Intervals and Base Population [Guatemala, 2018] Indicator Value Confidence Interval Number of Records Weighted Population Standard Deviation Standard Error DEFT Male 25.7 21.6 29.8 619 12,384 43.9 2.02 1.1 Female 25.9 21.3 30.5 646 13,170 43.6 2.24 1.3 Prevalence of stunted children under 5 years of age (Total) 69.7 65.5 73.9 1,263 25,525 46.0 2.06 1.6 Male 71.0 66.9 75.1 618 12,369 45.6 2.02 1.1 Female 68.5 62.4 74.5 645 13,156 46.2 2.97 1.6 Prevalence of wasted children under 5 years of age (Total) 1.4 0.6 2.1 1,264 25,536 11.6 0.38 1.2 Male 1.2 0.3 2.1 619 12,384 11.0 0.43 1.0 Female 1.5 0.5 2.5 645 13,152 12.1 0.50 1.1 Percentage of children under age 5 with diarrhea in the last two weeks (Total) 30.8 27.0 34.6 1,283 25,843 46.2 1.86 1.4 Male 29.4 23.4 35.5 630 12,534 45.9 2.97 1.6 Female 32.1 27.8 36.5 653 13,309 46.4 2.14 1.2 Percentage of children under age 5 with diarrhea treated with ORT (Total) 63.4 56.5 70.2 406 7,969 48.2 3.36 1.4 Male 69.7 60.0 79.3 195 3,691 47.3 4.74 1.4 Female 57.9 50.2 65.7 211 4,278 49.0 3.80 1.1 Prevalence of exclusive breast-feeding of children under six months of age 66.4 57.7 75.0 155 3,042 47.4 4.23 1.1 Male 72.1 60.8 83.3 76 1,442 46.4 5.52 1.0 Female 61.2 49.5 73.0 79 1,600 48.4 5.75 1.1 Prevalence of children 6-23 months of age receiving a minimum acceptable diet (MAD) 47.7 43.1 52.3 395 7,936 50.0 2.25 0.9 Male 44.0 37.8 50.2 196 3,830 50.5 3.05 0.8 Female 51.2 43.9 58.4 199 4,106 49.3 3.57 1.0 CUSTOM INDICATORS Percentage of women with a child under 5 years who recognize two or more pregnancy danger signs 71.2 66.1 76.3 847 19,092 45.3 2.49 1.6 Percentage of women with a child under 5 years who recognize two or more neonatal danger signs 63.0 56.9 69.2 847 19,092 48.3 3.01 1.8 Percentage of women with a child under 5 years who recognize two or more childhood illness signs 66.5 62.2 70.8 847 19,092 47.2 2.12 1.3 Percentage of women with a child under 5 years seeking health care when pregnancy danger signs are present 69.5 59.3 79.6 212 4,724 46.2 4.99 1.6 Percentage of women with a child under 5 years seeking health care when neonatal danger signs are present 66.1 53.9 78.3 194 4,275 47.4 5.98 1.8 Percentage of women with a child under 5 years seeking health care when childhood illness signs are present 80.1 73.5 86.7 353 7,751 40.0 3.25 1.5 Percentage of women with a child under 5 years making decisions about health care for themselves 50.4 43.6 57.1 756 17,093 50.0 3.32 1.8 Percentage of women with a child under 5 years making decisions about health care for their child 95.8 92.7 98.9 756 17,093 20.1 1.52 2.1 NA : Not available Page 2 Table A4.2. Comparison of Baseline and Endline Indicators - PAISANO Project Area [Guatemala 2013, 2018] 2013 Baseline 2018 Endline Raw Difference (Endline - Baseline) Significance Level1 FOOD SECURITY INDICATORS Average Household Dietary Diversity Score (HDDS) 6.1 7.0 0.8 *** 2,648 1,402 Prevalence of households with moderate or severe hunger (HHS) 7.6 6.0 -1.6 ns 2,793 1,534 Male and female adults 7.1 6.0 -1.1 ns 2,521 1,324 Adult female, no adult male 12.9 5.7 -7.2 * 245 192 Adult male, no adult female NA NA NA NA 24 16 Child, no adults NA NA NA NA 3 2 POVERTY INDICATORS Per capita expenditures (as a proxy for income) of USG-assisted areas $1.89 $2.54 $0.66 *** 17,529 8,902 Male and female adults $1.84 $2.49 $0.64 *** 16,561 8,159 Adult female, no adult male $2.71 $3.21 $0.51 * 898 690 Adult male, no adult female NA NA NA NA 62 46 Child, no adults NA NA NA NA 8 7 Prevalence of poverty: Percent of people living on less than $1.25/day2 44.2 14.7 -29.4 *** 17,529 8,902 Male and female adults 45.5 14.9 -30.6 *** 16,561 8,159 Adult female, no adult male 20.1 11.2 -8.9 † 898 690 Adult male, no adult female NA NA NA NA 62 46 Child, no adults NA NA NA NA 8 7 Mean depth of poverty (expressed as percent of poverty line) 11.8 2.6 -9.2 *** 17,529 8,902 Male and female adults 12.2 2.5 -9.7 *** 16,561 8,159 Adult female, no adult male 4.4 2.2 -2.2 ns 898 690 Adult male, no adult female NA NA NA NA 62 46 Child, no adults NA NA NA NA 8 7 WASH INDICATORS Percentage of households using an improved source of drinking water 21.5 41.8 20.3 *** 2,795 1,536 Percentage of households using improved sanitation facilities 51.8 66.7 14.9 *** 2,793 1,536 Percentage of households with soap and water at a handwashing station commonly used by family members 77.1 80.3 3.1 ns 2,768 1,536 AGRICULTURAL INDICATORS Percentage of farmers who used financial services in the past 12 months 20.2 21.1 0.8 ns 2,546 1,418 Percentage of farmers who practiced value chain activities promoted by the project in the past 12 months 15.9 28.3 12.4 *** 2,553 1,418 Percentage of farmers who used three sustainable agricultural practices in the past 12 months 50.2 60.5 10.3 * 2,573 1,418 Percentage of farmers who used at least two sustainable crop practices in the past 12 months 58.8 65.5 6.7 ns 5,546 1,418 Percentage of farmers who used at least two sustainable livestock practices in the past 12 months 9.7 14.9 5.2 * 5,546 1,418 Percentage of farmers who used at least two sustainable NRM practices in the past 12 months 0.8 1.0 0.2 ns 5,546 1,418 Percentage of farmers who used improved storage practices in the past 12 months 8.7 18.9 10.3 *** 2,558 1,418 WOMEN'S HEALTH AND NUTRITION INDICATORS Prevalence of underweight women 2.2 1.4 -0.8 ns 2,202 1,987 Women’s Dietary Diversity Score (WDDS) 3.8 4.8 1.0 *** 2,564 2,166 CHILDREN'S HEALTH AND NUTRITION INDICATORS Prevalence of underweight children under 5 years of age (Total) 26.4 25.8 -0.6 ns 2,578 1,265 Male 27.3 25.7 -1.6 ns 1,281 619 Female 25.5 25.9 0.4 ns 1,297 646 Prevalence of stunted children under 5 years of age (Total) 74.7 69.7 -5.0 * 2,578 1,263 Male 77.1 71.0 -6.1 * 1,281 618 Female 72.3 68.5 -3.8 ns 1,297 645 Number of Observations Baseline Endline Page 3 2013 Baseline 2018 Endline Raw Difference (Endline - Baseline) Significance Level1 Table A4.2. Comparison of Baseline and Endline Indicators - PAISANO Project Area [Guatemala 2013, 2018] Number of Observations Baseline Endline Prevalence of wasted children under 5 years of age (Total) 1.0 1.4 0.4 ns 2,578 1,264 Male 0.8 1.2 0.4 ns 1,281 619 Female 1.2 1.5 0.3 ns 1,297 645 Percentage of children under age 5 with diarrhea in the last two weeks (Total) 28.8 30.8 2.0 ns 2,623 1,283 Male 29.0 29.4 0.5 ns 1,306 630 Female 28.7 32.1 3.5 ns 1,317 653 Percentage of children under age 5 with diarrhea treated with ORT (Total) 53.3 63.4 10.1 * 736 406 Male 55.4 69.7 14.2 * 373 195 Female 51.2 57.9 6.8 ns 363 211 Prevalence of exclusive breast-feeding of children under six months of age 66.1 66.4 0.3 ns 298 155 Male 69.4 72.1 2.7 ns 170 76 Female 61.6 61.2 -0.4 ns 128 79 Prevalence of children 6-23 months of age receiving a minimum acceptable diet (MAD) 22.1 47.7 25.6 *** 842 395 Male 22.0 44.0 22.0 *** 429 196 Female 22.3 51.2 28.8 *** 413 199 CUSTOM INDICATORS Percentage of women with a child under 5 years who recognize two or more pregnancy danger signs 37.4 71.2 33.8 *** 1,708 847 Percentage of women with a child under 5 years who recognize two or more neonatal danger signs 26.6 63.0 36.4 *** 1,707 847 Percentage of women with a child under 5 years who recognize two or more childhood illness signs 31.7 66.5 34.8 *** 1,707 847 Percentage of women with a child under 5 years seeking health care when pregnancy danger signs are present 64.0 69.5 5.4 ns 310 212 Percentage of women with a child under 5 years seeking health care when neonatal danger signs are present 76.4 66.1 -10.2 ns 283 194 Percentage of women with a child under 5 years seeking health care when childhood illness signs are present 85.0 80.1 -4.9 ns 644 353 Percentage of women with a child under 5 years making decisions about health care for themselves 61.5 50.4 -11.1 ** 1,543 756 Percentage of women with a child under 5 years making decisions about health care for their child 97.0 95.8 -1.2 ns 1,543 756 1 ns = not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 2 Expressed in constant 2010 USD NA : Not available Page 4 Annex 5: Descriptive and Bivariate Analyses Tables 2013 Baseline 2018 Endline Total population 188,255 173,968 Male 90,394 81,050 Female 97,861 92,918 Women of reproductive age (15-49 years) 45,384 44,945 Children ages 0-59 months 27,264 25,843 Males ages 0-59 months 13,570 12,534 Females ages 0-59 months 13,694 13,309 Children ages 0-5 months 3,158 3,042 Males ages 0-5 months 1,789 1,442 Females ages 0-5 months 1,369 1,600 Children ages 6-23 months 9,262 7,936 Males ages 6-23 months 4,757 3,830 Females ages 6-23 months 4,506 4,106 Total households 29,956 29,868 Male and Female Adults 27,224 25,920 Adult Female no Adult Male 2,457 3,637 Adult Male no Adult Female 243 269 Child No Adults 32 42 Average household size 6.3 5.8 Percentage of households with children 0-59 months 61.2 57.2 Percentage of households with a child 6-23 months 30.4 24.4 Percentage of households with a child 0-5 months 10.4 9.4 Household headship (% male) 85.1 78.8 Education level of head of household No formal education* 45.0 42.6 Pre-primary 0.3 0.4 Primary* 49.8 50.1 Lower Secondary1 2.6 2.8 Upper Secondary2 1.7 3.3 Higher 0.6 0.8 2 Equivalent to "Diversificado" in Guatemala. Table A5.1a. Population and Household Characteristics in the Project Area, SC-PAISANO [Guatemala, 2013, 2018] Source: FFP Baseline (2013) and Endline (2018) PBS in Guatemala 1 Equivalent to "Básico" in Guatemala. Percent Number Regular participation in project activities Yes 46.0 677 No 54.0 858 Type of project activities in which households participated on a regular basis1 Received food rations 93.7 632 Attended nutrition training/meetings 92.9 626 Attended agriculture training/meetings 68.2 455 Received cash transfers 39.4 256 Other activities 29.0 196 Number of project activities in which households participated on a regular basis Did not participate in any activities 54.5 865 1 1.8 29 2 6.9 111 3 18.5 269 4 13.9 198 5 4.4 63 1 Percentages add up to more than 100 because households can participate in more than one type of activity. Percentage is proportion of all respondents who answered "Yes" to regularly participating in project activities. A total of 12 households in the SC-PAISANO project areas did not report information on project participation. Table A5.1b. Project participation Percentage of households reporting regular participation in project activities, overall and by activity type [Guatemala 2018] SC-PAISANO Baseline 2013 (BL) Endline Comparison by Participation Status (Sig.) All Households (I) All Households (II) Direct Participants (III) Indirect participants (IV) All Households (V) Direct Participants (VI) Indirect participants (VII) Direct vs. Indirect Participation (VIII) Household dietary diversity score (HDDS) 6.1 7.0 7.0 7.0 *** *** *** ns Per capita consumption expenditures (as a proxy for income) of USG￾assisted areas (constant 2010 USD at 2005 PPP) $1.89 $2.54 $2.29 $2.80 *** *** *** *** Prevalence of poverty: Percent of people living on less than $1.25/day (%) 44.2 14.7 17.2 11.9 *** *** *** * Mean depth of poverty (%) 11.8 2.6 2.9 1.9 *** *** *** ns Women's dietary diversity score (WDDS) 3.8 4.8 4.9 4.7 *** *** *** * Prevalence of underweight children under five years of age (%) 26.4 25.8 26.6 24.4 ns ns ns ns Prevalence of stunting children under five years of age (%) 74.7 69.7 72.0 65.4 * ns ** * Prevalence of children 6-23 months with a minimum acceptable diet (MAD) (%) 22.1 47.7 49.7 44.2 *** *** *** ns Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 NOTE: The household survey asked respondents whether they participated regularly in any of the project activities. Respondents who answered "Yes" are considered direct participants. Respondents who answered "No" or "Don't know" are considered indirect participants. A total of 12 households in the SC-PAISANO project areas did not report information on project participation. Households with missing information on project participation are not included in the BL-EL comparison by participation status (Columns VI and VII) nor in the endline comparison by participation status (Column VIII). Edline 2018 (EL) BL-EL Comparison (Sig.) Table A5.2 Comparison of Baseline and Endline Indicators by Project Participation Status - Save the Children PAISANO Program Guatemala FY 2012 FFP Development Food Assistance Programs - Select Indicators 2013 Baseline 2018 Endline Sig. Cereals 99.5 98.2 ** Roots and tubers 40.7 50.6 ** Vegetables 65.2 70.9 † Fruits 35.3 54.7 *** Meat, poultry and organ meat 24.3 33.5 ** Eggs 44.3 63.4 *** Fish and seafood 5.4 3.6 ns Pulses/legumes/nuts 63.7 73.2 ** Milk and milk products 8.7 18.9 *** Oils/fats 54.5 65.0 *** Sugar/honey 94.7 92.9 ns Miscellaneous (tea, coffee, condiments, etc.…) 75.3 71.0 † Number of households 2,797 1,402 Table A5.3. Baseline-Endline Comparison of Household Dietary Diversity Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 NOTE: Only includes households that reported that yesterday was not an unusual or special day. Food groups consumed by households [Guatemala 2013, 2018] SC-PAISANO Table A5.4. Baseline-Endline Comparison of Use of Financial Services and Value Chain Activities Percentage of farmers by access to financial services and value chain activities [Guatemala 2013, 2018] 2013 Baseline 2018 Endline Sig. Financial services Saved money 14.2 3.9 *** Obtained agricultural credit 6.5 1.3 *** Had agricultural insurance 2.8 18 *** No financial services 79.9 78.9 ns Value chain activities Market-oriented production 42.2 55.8 ** Calculation of cost of production for the market 10.1 19.7 *** Keeping production records 1.6 4.4 *** Developing production and sales plans 10.3 19.2 *** No value chain activities 57.8 44.2 ** Number of farmers 2,576 1,418 SC-PAISANO Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 NOTE: Each household is associated with one farmer's record. In households with multiple farmers, responses were consolidated into one record per household. If more than one farmer provided a response, the response of the first farmer (usually the household head) was considered. Table A5.5. Baseline-Endline Comparison of Use of Sustainable Agricultural Practices Percentage of farmers using sustainable agricultural practices by type of practice [Guatemala 2013, 2018] 2013 Baseline 2018 Endline Sig. Crops1 Use of improved seeds for beans 2.4 4.2 * Use of soil conservation practices 40.8 34.7 ns Use of organic fertilization 50.9 56.2 ns Household garden production 45.4 48.4 ns Management of fruit trees 35.3 49.2 ** Production of native herbs 31.6 30.8 ns Use of at least two sustainable crop practices 58.8 65.5 ns No sustainable crop practices 19.1 16.7 ns Livestock2 Improved (roof, wall, water, and food) livestock facilities for rabbits, goats, pigs and poultry 28.4 40.1 ** Vaccination of poultry 26.1 24.3 ns Management of goats (3 of 4 categories: deworming, vaccination, hoof removal, vitamin supplementation) 0.3 1.0 * Use of at least two sustainable livestock practices 9.7 14.9 * No sustainable livestock practices 55.2 49.6 ns Natural resource managemet3 Agroforestry 3.5 3.6 ns Natural forest regeneration 1.6 4.0 ** Reforestation - new plantations 9.9 11.6 ns Reforesting of water resources 0.6 0.6 ns Management of forests and woods 2.8 0.9 * Production of forest plants 0.2 0.3 ns Use of at least two sustainable NRM practices 0.8 1.0 ns No sustainable NRM practices 82.5 80.7 ns Number of farmers 2,576 1,418 SC-PAISANO Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 NOTE: Multiple responses allowed so totals may add up to more than 100 percent. Each household is associated with one farmer's record. In households with multiple farmers, responses were consolidated into one record per household. If more than one farmer provided a response, the response of the first farmer (usually the household head) was considered. 1 Includes use of improved seeds for corn, soil conservation practices, use of organic fertilization, household garden production and production of native herbs, management of fruit trees and use of improved seeds for beans. 2 Includes improved livestock facilities (roof, wall, water and food) for rabbits, goats, pigs and poultry; vaccination of poultry, and management of goats (3 of 4: deworming, vaccination, hoof removal, vitamin supplementation). 3 Includes agro-forestry, reforestation, management of forest plantations, management of natural reforestation and production of plants. Constant 2010 USD Number of households2 Sig. Use of financial services No $2.42 1,101 ** Yes $2.90 317 Use of value chain activities No $2.43 1,019 ** Yes $2.75 399 Use of at least two sustainable crop practices No $2.36 510 * Yes $2.60 908 Use of at least two sustainable livestock practices No $2.45 1,213 *** Yes $2.91 205 Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 1 Each household is associated with one farmer's record. In households with multiple farmers, responses were consolidated into one record per household. If more than one farmer provided a response, the response of the first farmer (usually the household head) was considered. 2 The sample size in the table refers to the number of households, however the percentage of people living under the $1.25/day is a weighted average where the sampling weights take into account the number of members per household. Daily per capita consumption expenditures in constant 2010 USD (2005 PPP) by household use of agricultural services and practices [Guatemala 2018] Table A5.6. Daily Per Capita Consumption Expenditures by Household Agriculture Status at Endline SC-PAISANO Percent Number of households2 Sig. Use of financial services No 15.2 1,101 ns Yes 13.1 317 Use of value chain activities No 14.7 1,019 ns Yes 15.1 399 Use of at least two sustainable crop practices No 17.8 510 ns Yes 13.2 908 Use of at least two sustainable livestock practices No 15.9 1,213 * Yes 8.4 205 Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 1 Each household is associated with one farmer's record. In households with multiple farmers, responses were consolidated into one record per household. If more than one farmer provided a response, the response of the first farmer (usually the household head) was considered. 2 The sample size in the table refers to the number of households, however the percentage of people living under the $1.25/day is a weighted average where the sampling weights take into account the number of members per household. Table A5.7. Prevalence of Poverty by Household Agriculture Status at Endline Percent of people on less than $1.25 per day (2005 PPP) by household use of agricultural services and practices [Guatemala 2018]1 SC-PAISANO Percent Number of households2 Sig. Use of financial services No 2.6 1,101 ns Yes 2.1 317 Use of value chain activities No 2.4 1,019 ns Yes 2.6 399 Use of at least two sustainable crop practices No 2.9 510 ns Yes 2.2 908 Use of at least two sustainable livestock practices No 2.7 1,213 ** Yes 1.1 205 Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 1 Each household is associated with one farmer's record. In households with multiple farmers, responses were consolidated into one record per household. If more than one farmer provided a response, the response of the first farmer (usually the household head) was considered. 2 The sample size in the table refers to the number of households, however the percentage of people living under the $1.25/day is a weighted average where the sampling weights take into account the number of members per household. Table A5.8. Depth of poverty by Household Agriculture Status at Endline Mean depth of poverty expressed as percent of the $1.25 per day poverty line (2005 PPP) by household use of agricultural services and practices [Guatemala 2018]1 SC-PAISANO Table A5.9. Baseline-Endline Comparison on Women's Nutritional Status Height and BMI levels of non-pregnant women 15-49 years of age [Guatemala 2013 2018] 2013 Baseline 2018 Endline Sig. Percent less than 145 cm 51.2 47.7 ns Mean Body Mass Index (BMI) 24.6 25.5 *** Normal (%) 18.5-24.9 (total normal) 58.6 50.8 *** Underweight (%) <18.5 (total underweight) 2.2 1.4 ns 17.0-18.4 (mildly underweight) 1.6 1.3 ns <17 (moderately and severely underweight) 0.5 0.0 ** Overweight/obese (%) ≥25 (total overweight or obese) 39.2 47.8 *** 25.0-29.9 (overweight) 29.7 34.2 * ≥30.0 (obese) 9.5 13.6 * Number of non-pregnant women 15-49 2,568 1,987 SC-PAISANO Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 Percent Number of women Sig. Participation status1 Indirect 48.2 1,063 ns Direct 47.3 922 Total 47.8 1,985 Food rations2 No 58.6 60 ns Yes 46.5 862 Total 47.3 922 Nutrition training/meetings2 No 38.9 79 ns Yes 48.0 843 Total 47.3 922 Agriculture training/meetings2 No 44.1 296 ns Yes 48.7 626 Total 47.3 922 Cash transfers2 No 45.4 559 ns Yes 50.0 363 Total 47.3 922 Other activities2 No 45.6 653 ns Yes 51.2 269 Total 47.3 922 Prevalence of women's overweight or obesity by project participation status [Guatemala 2018] Table A5.10. Prevalence of Overweight of Obese Women 15-49 years by Project Participation Status at Endline SC-PAISANO Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 1 Direct participants refer to households that self-reported regularly participating in project activities. Indirect participants refer to households that self-reported not regularly participating in project activities. 2 Includes only women living in households that self-reported regularly participating in project activities. Table A5.11. Baseline-Endline Comparison of Women's Dietary Diversity Food groups consumed by women 15-49 [Guatemala 2013, 2018] 2013 Baseline 2018 Endline Sig. Grains, roots and tubers 98.6 99.4 * Legumes and nuts 54.9 68.8 *** Dairy products (milk, yogurt, cheese) 9.7 21.5 *** Organ meat 5.4 4.2 ns Eggs 43.1 63.3 *** Flesh foods and misc. small animal protein 31.3 39.8 ** Vitamin A dark green leafy vegetables 55.3 65.1 ** Other Vitamin A rich vegetables and fruits 36.3 44.9 * Other fruits and vegetables 48.6 74.5 *** Number of women 15-49 2,564 2,166 SC-PAISANO Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 2013 Baseline 2018 Endline Sig. Prevalence of stunted children 74.7 69.7 * <6 43.9 46.2 ns 6-11 64.7 66.3 ns 12-23 79.6 70.8 † 24-35 84.9 80.5 ns 36-47 82.4 72.1 * 48-59 75.2 70.5 ns Number of children 2,578 1,236 Prevalence of underweight children 26.4 25.8 ns <6 6.6 18.3 *** 6-11 17.9 24.3 ns 12-23 31.5 29.1 ns 24-35 37.1 27.3 * 36-47 28.1 21.7 ns 48-59 25.0 30.1 ns Number of children 2,578 1,265 Prevalence of wasted children 1.0 1.4 ns <6 0.5 1.1 ns 6-11 1.7 2.2 ns 12-23 1.9 1.3 ns 24-35 1.4 1.6 ns 36-47 0.3 0.6 ns 48-59 0.1 1.7 † Number of children 2,578 1,264 Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 1 The prevalence of stunting is based on children with valid height for age measurements, the prevalence of underweight is based on children with valid weight for age measurements, and the prevalence of wasting is based on children with valid weight for height measurements. SC-PAISANO Table A5.12. Baseline-Endline Comparison of the Malnutrition Status of Children Under Five Prevalence of stunted, underweight and wasted children by age in months [Guatemala 2013, 2018]1 2013 Baseline 2018 Endline Sig. Prevalence of stunted children 0-23 months 66.9 62.9 ns Number of children 1,143 540 Prevalence of underweight children 0-23 months 21.8 24.9 ns Number of children 1,143 542 Table A5.13. Baseline-Endline Comparison of the Malnutrition Status of Children Under Two Prevalence of stunted and underweight children 0-23 months [Guatemala 2013, 2018] SC-PAISANO Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 2013 Baseline 2018 Endline Sig. Prevalence of stunted children 12-59 months 80.5 73.7 * Number of children 2,009 978 Prevalence of stunted children 12-59 months [Guatemala 2013, 2018] SC-PAISANO Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 Table A5.14. Baseline-Endline Comparison of the Prevalence of Stunting Among Children 12-59 months 2013 Baseline 2018 Endline Sig. Prevalence of underweight children 6-47 months 29.8 26.0 ns Number of children 1,821 880 Prevalence of underweight children 6-47 months [Guatemala 2013, 2018] SC-PAISANO Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 Table A5.15. Baseline-Endline Comparison of the Prevalence of Underweight Among Children 6-47 months Percent Number of children Sig. Participation status1 Indirect 65.4 475 * Direct 72.0 788 Total 69.7 1263 Food rations2 No 75.8 36 ns Yes 71.9 752 Total 72.0 788 Nutrition training/meetings2 No 70.9 54 ns Yes 72.1 734 Total 72.0 788 Agriculture training/meetings2 No 75.0 249 ns Yes 70.7 539 Total 72.0 788 Cash transfers2 No 75.7 462 * Yes 67.3 326 Total 72.0 788 Other activities2 No 71.6 564 ns Yes 73.2 224 Total 72.0 788 Table A5.16. Prevalence of Stunting by Project Participation Status at Endline Prevalence of stunted children under five by project participation status [Guatemala 2018] Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 1 Direct participants refer to households that self-reported regularly participating in project activities. Indirect participants refer to households that self-reported not regularly participating in project activities. 2 Includes only children living in households that self-reported regularly participating in project activities. SC-PAISANO Not breastfeeding <2 0.0 2-3 0.0 4-5 0.0 6-8 0.4 9-11 3.2 12-17 7.7 18-23 33.6 Exclusively breastfed <2 80.0 2-3 65.5 4-5 55.1 6-8 9.8 9-11 2.1 12-17 0.0 18-23 0.0 Breastfed and plain water only <2 4.8 2-3 14.8 4-5 9.4 6-8 1.4 9-11 3.2 12-17 0.0 18-23 0.0 Breastfed and non-milk liquids <2 0.0 2-3 3.9 4-5 17.3 6-8 7.3 9-11 5.0 12-17 0.6 18-23 0.0 Breastfed and other milk <2 11.7 2-3 14.5 4-5 4.3 6-8 2.1 9-11 0.0 12-17 0.6 18-23 0.0 Breastfed and complementary foods <2 3.4 2-3 1.4 4-5 13.8 6-8 79.0 9-11 86.5 12-17 91.1 18-23 66.4 Number of children 550 Table A5.17. Baseline-Endline Comparison of Breastfeeding Status Breastfeeding status for children 0-23 months by age in months [Guatemala 2018] NOTE: The results for these subgroup analyses are based on small sample sizes and may be unreliable. 2018 Endline Breastfed children 6-8 months Percent with minimum meal frequency (2 or more) 62.0 Percent with minimum dietary diversity (4 or more) 32.3 Grains, roots, and tubers 82.8 Legumes and nuts 30.4 Dairy products (milk, yogurt, cheese) 20.6 Flesh foods (meat, fish, poultry, and liver/organ meats) 7.2 Eggs 31.6 Vitamin A-rich fruits and vegetables 46.1 Other fruits and vegetables 40.3 Number of children 63 Breastfed children 9-23 months Percent with minimum meal frequency (3 or more) 74.2 Percent with minimum dietary diversity (4 or more) 69.5 Grains, roots, and tubers 96.7 Legumes and nuts 56.1 Dairy products (milk, yogurt, cheese) 35.8 Flesh foods (meat, fish, poultry, and liver/organ meats) 31.5 Eggs 61.5 Vitamin A-rich fruits and vegetables 75.3 Other fruits and vegetables 61.3 Number of children 275 Non-breastfed children 6-23 months Percent with minimum meal frequency (4 or more + 2 milk) 36.5 Percent with minimum dietary diversity (4 or more) 62.8 Grains, roots, and tubers 100.0 Legumes and nuts 56.7 Dairy products (milk, yogurt, cheese) 54.3 Flesh foods (meat, fish, poultry, and liver/organ meats) 35.7 Eggs 73.9 Vitamin A-rich fruits and vegetables 70.1 Other fruits and vegetables 74.0 Number of children 57 Table A5.18a. Baseline-Endline Comparison of Minimum Acceptable Diet (MAD) Components of MAD indicator for children 6-23 months [Guatemala 2018] NOTE: The results for these subgroup analyses are based on small sample sizes and may be unreliable. 2018 Endline 2013 Baseline 2018 Endline Sig. Percent with minimum meal frequency 60.7 67.1 † Percent with minimum dietary diversity 28.9 62.6 *** Grains, roots, and tubers 90.4 94.9 ** Legumes and nuts 36.9 52.0 *** Dairy products (milk, yogurt, cheese) 15.6 35.8 *** Flesh foods (meat, fish, poultry, and liver/organ meats) 15.5 28.2 *** Eggs 34.2 58.3 ** Vitamin A-rich fruits and vegetables 48.0 69.8 *** Other fruits and vegetables 30.5 59.6 *** Number of children 6-23 months 842 395 Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 Table A5.18b. Baseline-Endline Comparison of Minimum Acceptable Diet (MAD) Components of MAD indicator for children 6-23 months [Guatemala 2013 2018] SC-PAISANO Percent Number of children Sig. Participation status1 Indirect 44.2 158 ns Direct 49.7 237 Total 47.7 395 Food rations2 No 61.2 15 ns Yes 48.8 222 Total 49.7 237 Nutrition training/meetings2 No 51.7 18 ns Yes 49.5 219 Total 49.7 237 Agriculture training/meetings2 No 54.9 78 ns Yes 47.2 159 Total 49.7 237 Cash transfers2 No 41.2 123 * Yes 58.4 114 Total 49.7 237 Other activities2 48.7 176 ns No 52.5 61 Yes 49.7 237 Total SC-PAISANO Prevalence of children 6-23 months of age receiving a MAD by project participation status [Guatemala 2018] Table A5.19. Prevalence of Minimum Acceptable Diet (MAD) by Project Participation Status at Endline Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 1 Direct participants refer to households that self-reported regularly participating in project activities. Indirect participants refer to households that self-reported not regularly participating in project activities. 2 Includes only children living in households that self-reported regularly participating in project activities. 2013 Baseline 2018 Endline Sig. Percentage of women with a child under five making decisions about healthcare for themselves1 61.5 50.4 ** Woman decides alone 11.0 20.9 ** Husband/partner decides alone 36.3 41.5 ns Woman and husband/partner decide together 50.8 35.7 *** Sometimes the woman decides and sometimes the husband/partner decides 0.8 0.4 ns Other 1.0 1.5 ns Number of women with a child under five 1,543 756 Table A5.20. Baseline-Endline Comparison of Women's Participation in Decisions Regarding Own Medical Attention [Guatemala 2013, 2018] SC-PAISANO Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 1 This includes the following: (1) woman decides alone; (20 woman and husband/partner decide together; or (3) sometimes the woman decided and sometimes the husband/partner decides. 2013 Baseline 2018 Endline Sig. Improved, not shared sanitation facility Flush to piped sewer system 10.7 18.3 ns Flush to septic tank 3.6 6.5 * Pit latrine with slab 37.5 41.9 ns Improved, shared sanitation facility Flush to piped sewer system 1.1 2.4 ns Flush to septic tank 0.4 0.7 ns Pit latrine with slab 4.9 5.8 ns Non-improved sanitation facility Open pit 35.5 21.6 *** No facility 5.7 2.5 * Other 0.7 0.4 ns Improved source of drinking water Piped water into dwelling 16.2 27.1 * Piped water into yard 22.2 50.7 *** Public tap 3.8 5.2 ns Private well 3.0 4.8 ns Public well 8.1 1.8 *** Rainwater 0.7 3.2 ns Non-improved source of drinking water Surface water (river/dam/ lake/ponds /stream/canal/irrigation channel) 45.5 6.1 *** Tanker truck 0.0 0.1 ns Other 0.6 1.1 ns Water availability Water is generally available from this source (% Yes) 68.1 65.5 ns Water is not available a day or more during the last two weeks (% Yes) 54.0 45.2 † Water treatment prior to drinking1 Boil 96.2 85.4 Filter 1.2 23.3 Bleach/chlorine added 4.9 3.5 Purchase purified water 0.1 1.2 Other 0.2 0.1 No treatment 2.5 1.9 Number of households 2,797 1,536 Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 1 Multiple responses allowed so totals may add up to more than 100 percent. Table A5.21. Baseline-Endline Comparison of Household Sanitation and Drinking Water Sanitation facility, source of drinking water, and treatment of drinking water [Guatemala 2013, 2018] SC-PAISANO Percent Number of children Sig. Improved water source Household does not use an improved water source 31.7 752 ns Household uses an improved water source 29.6 531 Improved sanitation facility Household does not use an improved sanitation facility 30.9 459 ns Household uses an improved sanitation facility 30.8 824 Handwashing station with water and soap or another cleansing agent Household does not have a handwashing station with water and soap or another cleansing agent 30.7 293 ns Household has a handwashing station with water and soap or another cleansing agent 30.9 990 All households 30.2 1,283 Table A5.22. Diarrhea and household water, hygiene and sanitation (WASH) at Endline SC-PAISANO Sig. = significance level of p-value for test of differences. ns not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 NOTE: Chi squared tests were used to examine the statistical significance of the relationship between the prevalence of diarrhea and household WASH status. Prevalence of diarrhea among children under five by household WASH status [Guatemala, 2018] Annex 6: Methodology and Summary of Multivariate Regression Analysis 1 PAISANO FINAL PERFORMANCE EVALUATION REPORT ANNEX 6 SUMMARY OF METHODOLOGY AND RESULTS OF SUPPLEMENTAL MULTIVARIATE ANALYSES This annex provides the methodology and summary of results of the supplemental multivariate analyses that the evaluation team conducted to explore the factors that promoted or inhibited the achievement of the PAISANO goal and objectives. Multivariate analyses controlling for key socioeconomic and project-specific factors as covariates was used to understand the factors that are associated with daily per capita consumption expenditures, and the prevalence of stunting and underweight among children under five. These indicators were selected for additional analyses to help inform learning on key outcomes and impact indicators. Daily per capita consumption expenditures increased markedly over the life of the award from $1.89 at 2013 baseline to $2.54 at 2018 endline. The prevalence of stunting declined since baseline from 74.7% to 69.7%, but at endline close to one-quarter of children under five were underweight despite improvements in dietary diversity, household water and sanitation practices, and poverty levels. The following sections describe the data used, provide variable definitions, and describe the statistical methods, results and conclusions. Daily per capitation consumption expenditures Data used Data used in these analyses come from population-based household surveys (PBS) implemented at project baseline (BL) (2013) and endline (EL) (2018) and utilized for the 2018 performance evaluation of the project. The surveys collected standard information on household and respondent characteristics; food security and poverty; agricultural practices; children’s health and nutrition; and women’s health and nutrition. The analyses are restricted to cases with nonmissing information on the dependent and explanatory variables. The final sample size for the analyses of daily per capita consumption expenditures was 3,894 households. Variable definitions Dependent variables The main outcome of interest is daily per capita consumption expenditures. The Data Treatment Analysis Plan and details on the methodology for calculating of daily per capita consumption expenditures. To deal with normality, daily per capita consumption expenditures were transformed by taking its natural logarithm. Explanatory variables The analyses controlled for individual, household and project-level characteristics that can influence daily per capita consumption expenditures. The selection of covariates is based the availability of data collected at baseline and endline. Variables with statistically significant bivariate associations with the outcome are excluded from the multivariate analysis if post-estimation tests indicated multicollinearity. Bivariate analyses of the daily per capita consumption expenditures were conducted for each of the following household agricultural practices that were promoted by the project: acquiring agricultural credit in cash or in kind, making savings, use of value chain activities (any two), use of at least two sustainable crop practices, use of at least two sustainable livestock practices, and use of at least two 2 sustainable NRM practices. Household food security status was considered since households that experience hunger may be less likely to participate in income generating activities. However, this variable may be endogenous since lower daily per capita consumption expenditures may contribute to a higher prevalence of hunger. Department dummies are added to capture variations in agro-ecological zones and other unobserved regional factors such as distance to the market and differences in the implementation of activities between departments. To examine and control for potential differences in effects for direct and indirect participants, a dummy is included for households in which any member participated in the project activities on a regular basis. The designation of the household as a direct participant is based on the household survey respondent’s reply and is not verified using project documents. Additionally, the analyses explored the association of daily per capita consumption expenditures with the type of intervention which could include food rations, agriculture trainings/meetings, nutrition training/meetings, receipt of cash transfers, and other activities. The number of interventions received by each household was also explored. The benefits of increased consumption expenditures may be higher for households participating in a higher number of interventions. There is little variation in participation by type of intervention since most direct participants generally participated in all interventions, except for cash transfer which were introduced following the midterm evaluation. Cases that comprise the baseline sample are coded “0” for participation variables, since no project activities were implemented at the time of the baseline survey. The following household sociodemographic characteristics were considered: number of children 0-4, number of children 5-17, number of adult females, number of adult males, household gendered type1, and the age, sex and educational attainment of the household head. Because the projects targeted children under the age of two, households with a higher number of children under the age of five are likely to benefit from project interventions compared to households where children have aged out of the under two age brackets Households with working age males and male-headed households, compared to adult female only households or female-headed households, may be more likely to engage in market-oriented production or income generating opportunities and/or may be more likely to access credit to purchase productivity-enhancing agricultural inputs and equipment, make investments in household water and sanitation infrastructure, or purchase food items that cannot be home grown. Alternatively, because the Women’s Empowerment Groups through the savings and loan groups aimed to increase women’s use of credit and making savings, households with a participating adult female may have been more likely to pursue practices that contribute to increases in daily per capita consumption expenditures. The results of the bivariate analyses are presented in Annex 7 Tables A7.1 Table A7.5. For the purposes of parsimony only variables whose bivariate association was found to be statistically significant were included in the multivariate model. Statistical Methods The analyses used ordinary least squares (OLS) regression model to analyze the correlates of daily per capita consumption expenditures. The analyses accounts for the two-stage stratified cluster sampling design. All analyses were conducted using STATA 14. The r-squared of the models is reported as a measure of goodness of fit. Additional post-estimation tests were conducted to check for the influence of outliers and the model was re-run excluding influential cases (i.e. cases with high residuals and leverage). 1 Gendered household type was omitted from the analyses since it is highly collinear with other covariates. 3 Results Annex 7, Table A7.6 illustrates the results of the OLS regression of the natural logarithm of daily per capita consumption expenditures. The results are reported as coefficients with the 95% confidence interval. The baseline model (Model 1) controls for the survey round variable only and indicates that daily per capita consumption expenditures are higher at endline compared to baseline. Improvement in daily per capita consumption expenditure between BL and EL can be partially explained by households’ use of agricultural practices promoted by the project. Acquiring agricultural credit, making savings, using at least two value chain activities, at least two sustainable crop practices, and at least two sustainable livestock practices are associated with higher daily per capita consumption expenditures (Model 2). These associations remain statistically significant net of all other factors in the final model (Model 7). Experiencing hunger is associated with lower daily per capita consumption expenditures. Participation in the project activities is associated with lower daily per capita consumption expenditures (Model 5). Model 6 includes receipt of cash transfers instead of the overall participation variable to explore the net effects of this intervention. The analysis focused on cash transfers since this intervention was introduced following the midterm evaluation and was therefore less universal than the other intervention of food rations, and nutrition and agriculture training and meetings. Model 6 also illustrates a negative association of cash transfers with daily per capita consumption expenditures. The final model (Model 7) controls for a host of household sociodemographic characteristics that could influence daily per capita consumption expenditures. The r-squared indicates that the final model explains 33.3 percent of the variance in daily per capita consumption expenditures. We performed post estimation diagnostics to identify influential cases (i.e. with high residuals or leverage) and re-ran the analyses excluding those influential cases. Cook's D was used to identify cases (households) with high residuals and leverage using the conventional cut-off point of 4/n where n is the number of observations (in this case 4/3,894). A total of 154 households from the baseline sample and 63 households from the endline sample were flagged as influential observations and were excluded from this analysis. The results of the analysis excluding influential cases are provided in Annex 7, Table A7.7 and are consistent with the findings reported in Annex 7, Table A7.6. Discussion and conclusions The objective of the additional multivariate analyses was to better understand the factors associated with improvements in daily per capita consumption expenditures. The results underscore the positive association of participation in agricultural interventions promoted by the project with improvements in consumption expenditures. Specifically, acquiring agricultural credit, making savings, using at least two value chain activities, at least two sustainable crop practices, and at least two sustainable livestock practices promoted by the project is associated with an increase in daily per capita consumption net of other factors. Household agricultural status may be contributing to an increase in daily per capita consumption expenditures either directly through the consumption of more diverse home-grown foods and/or indirectly by increased income from higher yields and profits. It is not unexpected that the association between cash transfers (or the project participation variable) is negative since the project targeted the poorest of the poor. Inclusion of household sociodemographic characteristics increases the goodness of the fit of the model from about 16 percent to 33 percent. The size and composition of the household can affect daily per capita consumption expenditures by influencing the availability of labor to participate in income generating activities, and indirectly by determining eligibility for participation in project interventions since the project targeted households with pregnant and lactating women, children under two, and the poorest of the poor. 4 The results of these supplemental analyses are intended to identify potential correlates that are associated with daily per capita consumption expenditures that may support the future design of project activities and/or beneficiary targeting based on household demographic and socioeconomic factors. However, the pre-post design of the data collection does not allow statements to be made about attribution relating to project impact. PREVALENCE OF STUNTING AND UNDERWEIGHT AMONG CHILDREN UNDER FIVE Data Used in the Analysis Data used in these analyses come from population-based household surveys (PBS) implemented at project baseline (BL) (2013) and endline (EL) (2018) and utilized for the 2018 performance evaluation of the project. The surveys collected standard information on household and respondent characteristics; food security and poverty; agricultural practices; children’s health and nutrition; and women’s health and nutrition. The analyses are restricted to cases with nonmissing information on the dependent and explanatory variables. The final sample size for the analyses of children’s stunting and underweight is 3,104 children. Definitions of Variables Dependent variables The main outcomes of interest are the prevalence of stunting and underweight among children under five. The baseline and endline surveys collected weight and height measurements for children under five. Data are collected for all children under. The World Health Organization (WHO) child growth standards (WHO Multicentre Growth Reference Study Group 2006) were used to compute z-scores for height-for-age (HAZ) and weight-for-age (WAZ). Each z-score is calculated by comparing the child’s height/length or weight with the median value of the WHO 2006 reference population. The difference is divided by the standard deviation of the reference population. datasets will be cleaned by flagging cases with z-scores beyond specified lower or upper cutoffs and excluding them from the computation of all indicators. Cases with height-for-age z-scores that are less than -6 standard deviations (SD) from the median or greater than +6 SD above the median were flagged and excluded from the calculation of the prevalence of stunting. Cases with weight-for-age z-scores that are less than -6 SD from the median or greater than +5 SD above the median were flagged and excluded from the calculation of the prevalence of underweight. Children with HAZ or WAZ below – 2 SD from the median are considered moderately-to-severely stunted or underweight, respectively. Explanatory variables The analyses controlled for individual, household and project-level characteristics that can influence the prevalence of stunting and underweight. The selection of covariates is based on the availability of data collected at baseline and endline. The analysis of children’s stunting and underweight included the child’s age, sex, and whether the child experienced diarrhea in the two weeks prior to the survey. The bivariate analyses also explored the association of child nutritional outcomes with the caregiver’s educational level and age to capture differences in basic literacy and experience with caregiving since in some case children may be looked after by older siblings rather than grandmothers or adult women. Children’s dietary diversity was not included since this would result in a reduction in sample size because information on food groups 5 consumed and meal frequency were collected only for children 6 -23 months. Alternatively, household consumption of the food groups promoted by the project were included in the analyses -namely, the growth foods (plant and animal-based protein) and protection foods (fruits and vegetables), as well as milk and milk products. Knowledge of childhood illness danger signs by a mother of a child under the age of five in the household was also explored. Children are more likely to receive treatment if their mother, or another mother living in the same household, can correctly identify childhood illness danger signs that require medical treatment. The analyses considered the following household sociodemographic characteristics: number of children 0-4 other than the child, number of children 5-17, number of adult females, number of adult males, household gendered type, and the age, sex and educational attainment of the household head. Because the project targeted children under the age of two, households with a higher number of children under the age of five are likely to benefit from project interventions compared to households where children have aged out of the under two age brackets. The size and composition of the household may also affect how often children are fed, what they are fed, and how much. Households with working age males and male-headed households, compared to adult female only households or female-headed households, may be more likely to engage in market-oriented production or income generating opportunities and/or may be more likely to access credit to purchase productivity-enhancing agricultural inputs and equipment, make investments in household water and sanitation infrastructure, or purchase food items that cannot be home grown. Alternatively, because the Women’s Empowerment Groups through the savings and loan groups aimed to increase women’s use of credit and making savings, households with a participating adult female may have been more likely to pursue practices that contribute to increases in daily per capita consumption expenditures. Households’ water, sanitation and hygiene status can influence children’s nutritional status since the use of unhygienic practices can lead to diarrheal disease and loss of important minerals and vitamins and contribute to weight loss. Variables explored for inclusion in the multivariate analyses included whether the household used an improved drinking water source, an improved sanitation facility, and whether the household has soap and water at a commonly used handwashing station. Household socioeconomic status was included by daily per capita consumption expenditures. Since daily per capita consumption expenditures is skewed it was transformed by taking its natural logarithm to avoid the influence of outliers on the outcome. The analyses also explored the association of children’s stunting and underweight with several key activities promoted by the DFAPs, and undertaken by the primary farmer in the household, that aimed to increase household food security through increased food production, food availability, and economic resources: acquiring agricultural credit, making savings, use of at least two value chain activities; use of at least two sustainable crop practices; use of at least two sustainable livestock practices; and use of at least two sustainable natural resource management practices. The underlying aim for the inclusion of such variables is to better understand their potential role in fostering improvements in food security and children’s nutritional status. Department dummies are added to capture variations in agro-ecological zones and other unobserved regional factors such as distance to the market or health services and differences in the implementation of activities between departments. To examine and control for potential differences in effects for direct and indirect participants, a dummy is included for households in which any member participated in the project activities on a regular basis. The designation of the household as a direct participant is based on the household survey respondent’s reply and is not verified using project documents. Additionally, the analyses explored the association of children’s underweight and stunting with the type of intervention 6 which could include food rations, agriculture trainings/meetings, nutrition training/meetings, receipt of cash transfers, and other activities. The number of interventions received by each household was also explored. The benefits to children’s nutritional outcomes may be higher for households participating in a higher number of interventions. However, there is little variation in participation by type of intervention since most direct participants generally participated in all interventions except for cash transfer which were introduced following the midterm evaluation. Cases that comprise the baseline sample are coded “0” for participation variables, since no project activities were implemented at the time of the baseline survey. The results of the bivariate analyses are presented in Annex 7, Table A7.1 and Table A7.2. For the purposes of parsimony only variables whose bivariate association was found to be statistically significant were included in the multivariate model and some variables were excluded after post-estimation tests indicated multicollinearity. Statistical Methods The analyses used logistic regression models to analyze the correlates of the prevalence of stunting and underweight. The analyses accounts for the two-stage stratified cluster sampling design. All analyses were conducted using STATA 14. We ran post-estimation model specification to assess goodness of fit and model specification. Results Stunting Annex 7, Table A7.3 illustrates the results of the multivariate logistic regression of the prevalence of stunting. The results are reported as odds ratios with the 95% confidence interval. The baseline mode (Model 1) indicated lower prevalence of stunting at endline compared to baseline. Model 2 illustrates making savings is associated with lower odds of stunting. Use of at least two value chain activities is also associated with lower odds of stunting but this association is marginally statistically significant (model 2). Children living in households that consume meat, poultry or organ meat have lower odds of stunting compared to children who live in households that do not consume this food group, but this association is marginally statistically significant (Models 2 and 3). On the other hand the odds of stunting are higher for children living in households that consume fish and seafood compared to children who live in households that do not consume this food group. Living in a household with water and soap at a commonly used handwashing station is associated with lower odds of stunting (Models 4- 7), but this association is marginally statistically significant in the final model (Model 8). Model 6 indicates that an increase in daily per capita consumption expenditures is associated with lower odds of stunting and this association remains statistically significant across all models. The odds of stunting are about 1.4 times higher for children living in households that participated in the project compared to children in households that are indirect project participants (Model 7). The association between receipt of cash transfers and stunting is statistically non-significant (Models 8 – 9). Underweight Annex 7, Table A7.4 illustrates the results of the multivariate logistic regression of the prevalence of underweight. The results are reported as odds ratios with the 95% confidence interval. The final model indicates that an increase in daily per capita consumption expenditures is associated with lower odds of underweight (Models 3 and 4) but this association washes out in the final model (Model 5) which controls for the child’s characteristics and the sociodemographic characteristics of the household. 7 Discussion and Conclusions The objective of the additional multivariate analyses was to better understand the factors associated with improvements in the prevalence of stunting among children under five and the factors that impeded the achievement of improvements in children’s underweight. The results of the multivariate analysis of children’s stunting and underweight underscore the role of daily per capita consumption expenditures in contributing to a reduction in the odds of malnutrition. The results do not indicate a statistically significant effect of household agriculture status. However, multivariate analysis of daily per capita consumption expenditures indicates that household participation in agricultural interventions promoted by the project is associated with improvements in consumption expenditures. Specifically, acquiring credit, making savings, and use of value chain activities, or at least two sustainable crop practices or at least two sustainable livestock practices promoted by the project is associated with an increase in daily per capita consumption (see Annex 7, Table A7.6 for the results of the multivariate analyses of daily per capita consumption expenditures). Household agricultural status may be contributing to a reduction in vulnerability to children’s malnutrition, either through the consumption of more diverse home-grown foods and/or by increased income from higher yields and profits. It is not unexpected that the association between cash transfers and stunting washes out in the multivariate analyses since daily per capita consumption expenditures are associated with project participation – households that participated in the project experienced improvements in daily per capita consumption expenditures overtime compared to household that did not. The project aimed to reduce malnutrition by increasing use of improved health and nutrition practices. The results of the multivariate analyses generally did not indicate a statistically significant association between consumption of food groups promoted by the project and children’s stunting or underweight. This may partially be examined by the fact that these variables measure household-level consumption and not the food groups consumed by the child. The higher odds of stunting among children who consume fish and seafood compared to the odds of those who do not may be explained by the fact that fish and seafood are likely to be consumed by poorer households and the prevalence od stunting tends to be higher in poorer households. The results of these supplemental are intended to identify potential correlates that are associated with child malnutrition which may support the future design of project activities and/or beneficiary targeting based on household demographic and socioeconomic factors. However, the pre-post design of the data collection does not allow statements to be made about attribution relating to project impact. These analyses leveraged data available at baseline and endline to include a range of factors that are expected to be associated with the stunting and underweight while excluding variables that exhibited multicollinearity. However, some key variables are not included. Postestimation diagnostics of the stunting and underweight models indicated goodness of fit but also pointed to possible model specification. Children’s malnutrition is often closely associated with the mother’s health (e.g. weight of the mother during pregnancy) and the child’s birth weight but it was not possible to include such variables in these analyses as the survey instrument did not collect this information. Annex 7: Multivariate Tables VARIABLE % N Sig % N Sig Child's characteristics Child's sex Male 74.0 1,517 † 26.9 1,517 ns Female 70.7 1,587 26.0 1,587 Child's age in years < 6 months 43.7 342 *** 11.6 342 *** 6-11 65.1 348 19.4 348 12-23 75.5 678 29.5 678 24-35 83.3 608 33.2 608 36-47 78.3 560 26.0 560 48-59 72.5 567 29.4 567 Prevalence of diarrhea Child did not experience diarrhea in the past seven days 72.7 2,193 ns 25.3 2,193 * Child experienced diarrhea in the past seven days 71.4 911 29.3 911 Caregiver's characteristics Educational attainment Never attended school 77.1 1,062 *** 30.8 1,062 ** Primary or less 72.5 1,717 25.7 1,717 Secondary, vocational, or higher 56.6 322 17.4 322 Caregiver's age (years) 15-19 69.6 1,102 ns 24.7 1,102 ns 20-24 73.8 748 28.8 748 25-29 72.4 541 25.1 541 30/34 71.8 414 29.0 414 35-39 79.9 222 25.6 222 40-49 75.8 59 23.3 59 50 + 75.5 18 31.5 18 Mother's health seeking knowledge Percentage of women with a child under 5 years who recognize two or more childhood illness signs Does not recognize two or more childhood illness signs 73.6 1,732 ns 27.0 1,732 ns Recognizes two or more childhood illness signs 71.0 1,372 25.9 1,372 Household sociodemographic characteristics Household head sex Male 72.7 2,758 ns 27.1 2,758 † Female 69.2 346 21.8 346 Household head age (years) 18-24 69.0 226 ns 29.6 226 ns 25-34 75.2 975 29.0 975 35-44 70.7 860 24.5 860 45-54 73.0 541 25.7 541 55-64 70.4 323 27.4 323 65+ 70.0 177 19.4 177 Household head educational attainment Never attended school 74.6 1,283 *** 27.2 1,283 ns Primary or less 72.5 1,635 26.5 1,635 Secondary, vocational/technical or higher 55.5 186 20.3 186 Number of adult males None One adult male Two or more adult males Number of adult females None One adult female Two or more adult females Gendered household type Adult male and female 72.4 2,959 ns 26.3 2,959 ns Adult female only 71.2 132 29.5 132 Adult male only 62.5 13 23.4 13 Number of adult males 0 71.2 132 ns 29.5 132 ns 1 73.2 1,875 27.5 1,875 2 71.0 679 24.8 679 3 71.2 293 25.3 293 4 68.9 99 18.7 99 5 or more 74.9 26 14.7 26 Number of adult females 1 72.2 1,696 ns 27.2 1,696 ns 2 72.6 767 27.1 767 3 70.0 420 24.9 420 4 81.4 143 22.9 143 5 61.6 54 10.8 54 6 or more 82.9 24 27.3 24 Number of children 0 to 4 other than the child None 65.3 1,237 *** 23.1 1,237 * 1 77.1 1,285 28.0 1,285 2 77.4 445 31.6 445 3 77.5 101 28.1 101 4 or more 86.8 36 25.0 36 Number of children 5-17 0-1 64.2 532 *** 24.7 532 ns 1 69.1 589 24.0 589 2 70.8 537 26.3 537 3 75.9 548 27.8 548 Table A7.1. Results of bivariate analyses of the prevalence of moderate-to-severe stunting and moderate-to-severe underweight among children under five, SC￾PAISANO [Guatemala 2013, 2018] Stunting Underweight VARIABLE % N Sig % N Sig Table A7.1. Results of bivariate analyses of the prevalence of moderate-to-severe stunting and moderate-to-severe underweight among children under five, SC￾PAISANO [Guatemala 2013, 2018] Stunting Underweight 4 78.5 419 29.9 419 5 78.5 270 28.8 270 6 or more 77.4 209 24.3 209 Household water, sanitation, and hygiene (WASH) status Use of improved water source Household does not use an improved water source 74.0 2,196 † 27.0 2,196 ns Household uses an improved water source 68.9 908 25.2 908 Use of improved sanitation facility Household does not use an improved sanitation facility 74.1 1,431 † 27.0 1,431 ns Household uses an improved sanitation facility 70.9 1,673 26.0 1,673 Water and soap at a commonly used handwashing station Household does not have water and soap at a commonly used handwashing station 77.2 748 ** 26.9 748 ns Household has water and soap at a commonly used handwashing station 70.8 2,356 26.3 2,356 Household food security status Household did not experience moderate to severe hunger 71.5 2,898 *** 26.2 2,898 ns Household experienced moderate to severe hunger 84.5 206 30.1 206 Household consumption of diverse food groups Cereals No 84.6 36 † 30.3 36 ns Yes 72.2 3,068 26.4 3,068 Roots and tubers No 72.7 1,730 ns 27.0 1,730 ns Yes 71.9 1,374 25.7 1,374 Vegetables No 70.8 1,016 ns 30.5 1,016 * Yes 73.0 2,088 24.6 2,088 Fruits No 73.7 1,880 ns 27.5 1,880 ns Yes 70.4 1,224 24.9 1,224 Meat, poultry, and organ meat No 72.3 2,282 ns 27.0 2,282 ns Yes 72.3 822 25.0 822 Eggs No 75.3 1,574 ** 28.1 1,574 ns Yes 69.7 1,530 25.0 1,530 Fish and seafood No 71.8 2,928 * 25.9 2,928 * Yes 81.8 176 35.0 176 Pulses/legumes/nuts No 74.7 997 † 24.8 997 ns Yes 71.3 2,107 27.2 2,107 Milk and milk products No 73.0 2,783 † 26.3 2,783 ns Yes 67.3 321 27.6 321 Oils/facts No 74.4 1,301 † 27.6 1,301 ns Yes 71.0 1,803 25.7 1,803 Sugar/honey No 72.5 192 ns 24.3 192 ns Yes 72.3 2,912 26.6 2,912 Miscellaneous No 73.1 885 ns 25.4 885 ns Yes 72.0 2,219 26.8 2,219 Household agriculture status Use of credit Did not use credit 72.1 2,934 ns 26.3 2,934 ns Used credit 77.6 170 28.5 170 Use of savings Did not make any savings 72.9 2,834 † 26.6 2,834 ns Made savings 65.6 270 24.8 270 Use of value chain activities (any two) Did not use value chain activities 73.8 2,542 * 26.8 2,542 ns Used value chain activities 66.5 562 25.0 562 Use of at least two sustainable crop practices Did not use at least two sustainable crop practices 72.3 1,325 ns 27.1 1,325 ns Used at least two sustainable crop practices 72.3 1,779 26.0 1,779 Use of at least two sustainable livestock practices Did not use at least two sustainable livestock practices 72.5 2,756 ns 26.6 2,756 ns Used at least two sustainable livestock practices 70.8 348 25.1 348 Use of at least two sustainable NRM practices Did not use at least two sustainable NRM practices 72.3 3,088 ns 26.4 3,088 ns Used at least two sustainable NRM practices 70.4 16 31.4 16 Household poverty status ($1.25/day at 2005 PPP) Household is at or above the poverty line 70.2 1,901 * 26.7 1,901 ns Household is below the poverty line 76.5 1,203 25.9 1,203 Project-level factors Project participation Household did not participate in at least one project activity 72.8 2,476 ns 26.3 2,476 ns Household participated in at least one project activity 71.3 628 26.8 628 Type of participation Food rations Did not receive food rations 72.8 2,504 ns 26.6 2,504 ns Received food rations 71.2 600 26.2 600 VARIABLE % N Sig % N Sig Table A7.1. Results of bivariate analyses of the prevalence of moderate-to-severe stunting and moderate-to-severe underweight among children under five, SC￾PAISANO [Guatemala 2013, 2018] Stunting Underweight Nutrition training/meetings Did not participate in nutrition trainings/meetings 72.6 2,506 ns 26.4 2,506 ns Participated in nutrition trainings/meetings 71.7 598 26.5 598 Agriculture training/meetings Did not participate in agriculture trainings/meetings 72.9 2,667 ns 26.5 2,667 ns Participated in agriculture trainings/meetings 70.4 437 26.2 437 Cash transfers Did not receive cash transfers 73.2 2,847 † 26.7 2,847 ns Received cash transfers 66.9 257 24.9 257 Other activities Did not participate in any other project activities 72.3 2,931 ns 26.6 2,931 ns Participated in other activities 72.8 173 24.9 173 Department Quiche 75.9 1,677 *** 27.4 1,677 ns Huehuetenango 63.9 845 24.3 845 Quetzaltenango 73.1 582 26.6 582 Survey round 2013 75.0 2,149 * 26.7 2,149 ns 2018 69.2 955 26.1 955 All children under five 72.3 3,104 26.4 3,104 *** p<0.001, ** p<0.01, * p<0.05, + p<0.10 Not stunted Stunted Sig. Not underweight Underweight Sig. Household dietary diversity score 6.6 6.4 * 6.5 6.4 ns Average daily per capita consumption expenditures (constant 2010 USD) $2.24 $1.91 *** $2.04 $1.90 *** Number of project activities in which household participated 1.1 1.0 ns 1.1 1.0 ns *** p<0.001, ** p<0.01, * p<0.05, + p<0.10 Table A7.2. Relationship between prevalence of children's stunting and underweight and select household characteristics, SC￾PAISANO [Guatemala 2013, 2018] Stunting Underweight VARIABLE OR 95% CI OR 95% CI OR 95% CI OR 95% CI OR 95% CI OR 95% CI OR 95% CI OR 95% CI OR 95% CI Survey round (ref.:2013) 2018 0.748* 0.567 - 0.988 0.747* 0.567 - 0.985 0.804 0.614 - 1.054 0.832 0.632 - 1.095 0.926 0.712 - 1.204 0.754+ 0.543 - 1.046 0.945 0.737 - 1.212 0.948 0.745 - 1.207 0.972 0.747 - 1.265 Household agriculture status Made savings in the last 12 mos. (ref.: No) 0.691* 0.489 - 0.978 0.728+ 0.514 - 1.031 0.739+ 0.521 - 1.049 0.758 0.535 - 1.073 0.755 0.530 - 1.075 0.756 0.536 - 1.068 0.771 0.549 - 1.082 0.738+ 0.519 - 1.050 Used (at least two) value chain activities (ref.: No) 0.791+ 0.602 - 1.038 0.794 0.599 - 1.051 0.811 0.612 - 1.074 0.834 0.629 - 1.106 0.822 0.623 - 1.083 0.838 0.634 - 1.108 0.818 0.630 - 1.062 0.756+ 0.562 - 1.017 Household consumption of food groups promoted by the project Vegetables 1.129 0.929 - 1.372 1.139 0.938 - 1.383 1.138 0.931 - 1.391 1.136 0.927 - 1.392 1.140 0.932 - 1.395 1.105 0.901 - 1.354 1.106 0.873 - 1.401 Fruits 0.957 0.779 - 1.176 0.971 0.789 - 1.196 1.057 0.860 - 1.298 1.054 0.856 - 1.298 1.056 0.860 - 1.298 0.998 0.809 - 1.231 0.990 0.784 - 1.252 Meat, poultry, and organ meat 1.058 0.843 - 1.327 1.065 0.853 - 1.330 1.157 0.935 - 1.433 1.156 0.935 - 1.428 1.159 0.936 - 1.437 1.140 0.916 - 1.419 1.224 0.961 - 1.558 Eggs 0.825+ 0.667 - 1.020 0.821+ 0.663 - 1.015 0.859 0.698 - 1.059 0.854 0.691 - 1.056 0.861 0.700 - 1.059 0.871 0.699 - 1.087 0.863 0.686 - 1.085 Fish 1.699* 1.076 - 2.682 1.777* 1.129 - 2.797 1.819* 1.141 - 2.897 1.832* 1.155 - 2.905 1.811* 1.136 - 2.886 1.647* 1.024 - 2.650 1.843** 1.195 - 2.842 Legumes 0.918 0.750 - 1.123 0.925 0.755 - 1.133 0.916 0.751 - 1.119 0.909 0.746 - 1.108 0.918 0.751 - 1.123 0.942 0.774 - 1.146 0.907 0.734 - 1.120 Milk and milk products 0.821 0.605 - 1.115 0.855 0.626 - 1.169 0.949 0.700 - 1.287 0.947 0.693 - 1.294 0.952 0.704 - 1.288 0.931 0.692 - 1.252 0.906 0.650 - 1.262 Household WASH status Household used an improved water source (ref.: No) 0.873 0.664 - 1.150 0.857 0.657 - 1.118 0.864 0.665 - 1.123 0.857 0.657 - 1.117 0.879 0.678 - 1.139 0.881 0.674 - 1.153 Household used an improved sanitation facility (ref.: No) 0.918 0.770 - 1.095 0.948 0.792 - 1.135 0.964 0.804 - 1.156 0.949 0.792 - 1.136 0.884 0.738 - 1.060 0.935 0.768 - 1.138 Household has water and soap at a commonly used handwashing station (ref.: No) 0.737* 0.580 - 0.938 0.773* 0.610 - 0.978 0.758* 0.594 - 0.968 0.777* 0.614 - 0.982 0.753* 0.603 - 0.940 0.790+ 0.624 - 1.000 Household poverty status Natural logarithm of daily per capita consumption expenditures in constant 2010 USD 0.611*** 0.499 - 0.748 0.621*** 0.505 - 0.763 0.611*** 0.499 - 0.748 0.608*** 0.504 - 0.734 0.725** 0.570 - 0.921 Project-level factors Participated in at least one activity regularly (ref.: did not participate in any project activities) 1.356* 1.041 - 1.766 Received cash transfers (ref.: did not participate in any project activities or participated but did not receive cash transfers) 0.928 0.666 - 1.291 1.029 0.746 - 1.420 0.806 0.543 - 1.195 Department (ref.: Quiche) Huehuetenango 0.552*** 0.415 - 0.734 0.556*** 0.413 - 0.749 Quetzaltenango 0.975 0.711 - 1.339 1.019 0.715 - 1.453 Child's characteristics Child's sex (ref.: male) Female 0.777* 0.641 - 0.942 Child's age in years (ref. less than 6 months) 6-11 2.759*** 1.821 - 4.181 12-23 5.376*** 3.536 - 8.175 24-35 9.112*** 5.793 - 14.332 36-47 5.910*** 4.021 - 8.687 48-59 4.041*** 2.773 - 5.888 Caregiver's educational level (ref.: None) Primary or less 0.891 0.718 - 1.105 Secondary, vocational, or higher 0.508*** 0.355 - 0.726 Household head educational level (ref.: None) Primary or less 1.053 0.829 - 1.338 Secondary, vocational, or higher 0.593* 0.380 - 0.926 Number of children under five other than child (ref.: none) One 1.785*** 1.452 - 2.194 Two 1.855*** 1.327 - 2.594 Three 2.154* 1.043 - 4.449 Four or more 2.238 0.701 - 7.147 Constant 3.003*** 2.593 - 3.478 3.259*** 2.787 - 3.811 3.412*** 2.705 - 4.304 4.533*** 3.321 - 6.187 4.960*** 3.582 - 6.868 5.012*** 3.592 - 6.995 4.919*** 3.533 - 6.850 6.288*** 4.302 - 9.190 1.273 0.791 - 2.050 N 3,104 3,104 3,104 3,104 3,104 3,104 3,104 3,104 3,100 *** p<0.001, ** p<0.01, * p<0.05, + p<0.10 Model 6 Model 7 Model 8 Model 9 Table A7.3. Results of logistic regression of the prevalence of moderate-to-severe stunting, SC-PAISANO [Guatemala 2013, 2018] Model 1 Model 2 Model 3 Model 4 Model 5 VARIABLE OR 95% CI OR 95% CI OR 95% CI OR 95% CI OR 95% CI Survey round (ref.:2013) 2018 0.968 0.770 - 1.218 1.027 0.806 - 1.308 1.075 0.849 - 1.361 0.995 0.701 - 1.411 1.097 0.756 - 1.593 Household consumption of food groups promoted by the project Vegetables 0.733** 0.592 - 0.906 0.732** 0.591 - 0.906 0.723** 0.582 - 0.896 0.722** 0.575 - 0.906 Fruits 0.910 0.694 - 1.193 0.944 0.715 - 1.245 0.925 0.700 - 1.223 0.923 0.704 - 1.209 Meat, poultry, and organ meat 0.927 0.720 - 1.193 0.958 0.744 - 1.234 0.949 0.732 - 1.231 0.990 0.755 - 1.300 Eggs 0.840+ 0.689 - 1.023 0.856 0.704 - 1.040 0.863 0.707 - 1.053 0.863 0.705 - 1.058 Fish 1.630* 1.117 - 2.378 1.641* 1.123 - 2.399 1.610* 1.094 - 2.370 1.704** 1.140 - 2.547 Legumes 1.179+ 0.994 - 1.399 1.179+ 0.998 - 1.393 1.198* 1.010 - 1.421 1.209* 1.019 - 1.435 Milk and milk products 1.130 0.884 - 1.445 1.176 0.921 - 1.502 1.166 0.913 - 1.490 1.185 0.918 - 1.529 Household poverty status Natural logarithm of daily per capita consumption expenditures in constant 2010 USD 0.828* 0.688 - 0.998 0.815* 0.667 - 0.997 0.862 0.687 - 1.082 Department (ref.: Quiche) Huehuetenango 0.830 0.612 - 1.126 0.804 0.595 - 1.084 Quetzaltenango 1.074 0.862 - 1.337 1.100 0.894 - 1.355 Project-level factors Participated regularly in project activities (ref.: No) 1.146 0.776 - 1.692 1.015 0.677 - 1.522 Child's characteristics Child's sex (ref.: male) Female 0.902 0.739 - 1.100 Child's age in years (ref. less than 6 months) 6-11 1.820* 1.129 - 2.933 12-23 3.241*** 2.116 - 4.964 24-35 3.776*** 2.401 - 5.939 36-47 2.572*** 1.562 - 4.233 48-59 3.073*** 1.973 - 4.785 Child had diarrhea (ref.: No) 1.167 0.943 - 1.445 Household sociodemographic characteristics Caregiver's educational level (ref.: None) Primary or less 0.812 0.605 - 1.089 Secondary, vocational, or higher 0.514** 0.316 - 0.835 Female-headed household (ref.: male-headed) 0.786 0.552 - 1.120 Constant 0.365*** 0.319 - 0.417 0.433*** 0.335 - 0.559 0.452*** 0.350 - 0.583 0.471*** 0.348 - 0.637 0.204*** 0.123 - 0.336 N 3,104 3,104 3,104 3,104 3,100 *** p<0.001, ** p<0.01, * p<0.05, + p<0.10 Model 5 Table A7.4. Results of logistic regression of the prevalence of moderate-to-severe underweight, SC-PAISANO [Guatemala 2013, 2018] Model 1 Model 2 Model 3 Model 4 Table A7.5. Results of bivariate analyses of daily per capita consumption expenditures, SC-PAISANO [Guatemala 2013, 2018] Baseline 2013 Endline 2018 BL-EL sample constant 2010 USD Sig. constant 2010 USD Sig. constant 2010 USD Sig. Household sociodemographic characteristics Household head sex Male $1.82 *** $2.49 + $2.13 *** Female $2.26 $2.72 $2.54 Household head age 18-24 $2.08 + $2.69 ns $2.38 * 25-34 $1.90 $2.46 $2.16 35-44 $1.78 $2.46 $2.10 45-54 $1.80 $2.48 $2.13 55-64 $2.04 $2.73 $2.37 65+ $1.89 $2.62 $2.32 Household head educational level Never attended school $1.71 *** $2.24 *** $1.96 *** Primary or less $1.94 $2.70 $2.31 Secondary, vocational/technical or higher $2.68 $3.25 $3.00 Number of adult males 0 $2.69 *** $3.16 ** $2.97 *** 1 $1.95 $2.56 $2.25 2 $1.78 $2.50 $2.13 3 $1.60 $2.19 $1.87 4 $1.32 $2.15 $1.70 5 or more $1.62 $2.46 $2.06 Number of adult females 1 $2.01 *** $2.63 + $2.31 *** 2 $1.84 $2.51 $2.18 3 $1.68 $2.32 $2.00 4 $1.46 $2.28 $1.79 5 $1.62 $2.82 $2.20 6 or more $1.47 $2.20 $1.99 Number of children under five None $2.28 *** $3.05 *** $2.66 *** 1 $1.82 $2.44 $2.11 2 $1.54 $2.10 $1.81 3 $1.37 $1.98 $1.70 4 or more $1.33 $1.94 $1.59 Number of children 5-17 0-1 $2.83 *** $3.14 *** $2.99 *** 1 $2.28 $2.96 $2.68 2 $2.04 $2.73 $2.41 3 $1.71 $2.40 $2.06 4 $1.55 $1.94 $1.72 5 $1.45 $1.76 $1.57 6 or more $1.34 $2.00 $1.58 Gendered household type Adult male and female $1.82 *** $2.48 *** $2.14 *** Adult female only $2.68 $3.16 $2.97 Adult male only $2.80 $3.44 $3.12 Child Only $3.97 … $3.97 Household agriculture status Use of credit Did not use credit $1.85 * $2.53 ns $2.19 ns Used credit $2.09 $2.54 $2.17 Use of savings Table A7.5. Results of bivariate analyses of daily per capita consumption expenditures, SC-PAISANO [Guatemala 2013, 2018] Baseline 2013 Endline 2018 BL-EL sample constant 2010 USD Sig. constant 2010 USD Sig. constant 2010 USD Sig. Did not make any savings $1.83 * $2.51 ns $2.18 ns Made savings $2.13 $2.84 $2.30 Use of value chain activities (any two) Did not use value chain activities $1.80 *** $2.44 ** $2.08 *** Used value chain activities $2.23 $2.76 $2.57 Use of at least two sustainable crop practices Did not use at least two sustainable crop practices $1.76 * $2.36 * $2.01 *** Used at least two sustainable crop practices $1.95 $2.61 $2.30 Use of at least two sustainable livestock practices Did not use at least two sustainable livestock practices $1.82 ** $2.46 *** $2.12 *** Used at least two sustainable livestock practices $2.29 $2.91 $2.66 Use of at least two sustainable NRM practices Did not use at least two sustainable NRM practices $1.86 ns $2.52 ns $2.18 ns Used at least two sustainable NRM practices $1.83 $3.13 $2.61 Prevalence of moderate-to-severe hunger None To Some $1.88 ns $2.54 * $2.20 * Moderate To Severe $1.65 $2.26 $1.91 Project-level factors Household participated in at least one project activity regularly $1.86 $2.77 *** $2.15 ns Household did not participate in at least one project activity n/a $2.29 $2.29 Household participated in at least one project activity Type of intervention Food rations Did not receive food rations $1.86 $2.79 *** $2.17 ns Received food rations n/a $2.24 $2.24 Nutrition training/meetings Did not participate in nutrition trainings/meetings $1.86 $2.76 *** $2.16 ns Participated in nutrition trainings/meetings n/a $2.26 $2.26 Agriculture training/meetings Did not participate in agriculture trainings/meetings $1.86 $2.68 *** $2.17 ns Participated in agriculture trainings/meetings n/a $2.25 $2.25 Cash transfers Did not receive cash transfers $1.86 $2.58 *** $2.17 * Received cash transfers n/a $2.33 $2.33 Other activities Did not participate in any other project activities $1.86 $2.57 ** $2.18 ns Participated in other activities n/a $2.25 $2.25 Number of project activities in which HH regularly participated 0 $1.86 $2.78 *** $2.16 ns 1 n/a $2.24 $2.24 2 n/a $2.36 $2.36 3 n/a $2.34 $2.34 4 n/a $2.16 $2.16 5 n/a $2.24 $2.24 Department Quiche $1.66 *** $2.44 ** $2.04 ** Huehuetenango $2.00 $2.38 $2.19 Quetzaltenango $2.27 $2.97 $2.60 Total $1.86 $2.53 $2.19 *** p<0.001, ** p<0.01, * p<0.05, + p<0.10 Table A7.6. Results of OLS regression of the natural logarithm of daily per capita consumption expenditures (2010 constant USD at 2005 PPP), SC-PAISANO [Guatemala 2013, 2018] VARIABLES Coef. 95% CI Coef. 95% CI Coef. 95% CI Coef. 95% CI Coef. 95% CI Coef. 95% CI Coef. 95% CI Survey round (ref.:2013) 2018 0.347*** 0.254 - 0.439 0.327*** 0.233 - 0.420 0.324*** 0.231 - 0.417 0.331*** 0.242 - 0.420 0.414*** 0.322 - 0.506 0.352*** 0.262 - 0.443 0.288*** 0.208 - 0.367 Household agriculture status Acquired agricultural credit (in cash or in-kind) in the last 12 mos. (ref.: No) 0.067 -0.023 - 0.156 0.065 -0.025 - 0.156 0.069 -0.021 - 0.159 0.070 -0.017 - 0.158 0.068 -0.021 - 0.158 0.089* 0.020 - 0.159 Made savings in the last 12 mos. (ref.: No) 0.101* 0.022 - 0.179 0.095* 0.018 - 0.171 0.117** 0.046 - 0.187 0.118** 0.048 - 0.187 0.116** 0.045 - 0.186 0.081* 0.006 - 0.156 Used (at least two) value chain activities(ref.: No) 0.101*** 0.048 - 0.153 0.098*** 0.047 - 0.150 0.094*** 0.048 - 0.140 0.097*** 0.053 - 0.142 0.098*** 0.053 - 0.142 0.071** 0.026 - 0.116 Used at least two sustainable crop practices (ref.: No) 0.068* 0.014 - 0.122 0.065* 0.011 - 0.119 0.049* 0.001 - 0.097 0.054* 0.006 - 0.102 0.052* 0.003 - 0.100 0.055* 0.010 - 0.100 Used at least two sustainable livestock practices (ref.: No) 0.186*** 0.126 - 0.245 0.185*** 0.125 - 0.246 0.182*** 0.125 - 0.239 0.193*** 0.137 - 0.248 0.190*** 0.133 - 0.247 0.191*** 0.146 - 0.236 Household food security status Household experienced hunger (ref.: did not experience hunger) -0.131+ -0.263 - 0.001 -0.124+ -0.250 - 0.003 -0.122+ -0.250 - 0.005 -0.124+ -0.252 - 0.003 -0.119* -0.238 - -0.000 Department (ref.: Quiche) Huehuetenango 0.031 -0.089 - 0.152 0.039 -0.078 - 0.156 0.038 -0.082 - 0.158 -0.048 -0.153 - 0.057 Quetzaltenango 0.244*** 0.168 - 0.320 0.233*** 0.160 - 0.305 0.245*** 0.171 - 0.319 0.152*** 0.074 - 0.230 Project-level factors Participated in at least one activity regularly (ref.: did not participate in any project activities) -0.165*** -0.227 - -0.104 Received cash transfers (ref.: did not participate in any project activities or participated but did not receive cash transfers) -0.112*** -0.163 - -0.060 -0.058* -0.112 - -0.004 Household sociodemographic characteristics Household head age (ref,: 15-24) 25-34 0.038 -0.038 - 0.114 35-44 0.085+ -0.006 - 0.175 45-54 0.122* 0.019 - 0.225 55-64 0.189*** 0.086 - 0.292 65+ 0.090+ -0.008 - 0.188 Female-headed household (ref.: Male) 0.104*** 0.050 - 0.157 Household head educational level (ref.: none) Primary or less 0.136*** 0.091 - 0.180 Secondary, vocational/technical or higher 0.305*** 0.197 - 0.412 Number of adult males -0.040*** -0.063 - -0.018 Number of adult females -0.021+ -0.046 - 0.004 Number of children under five -0.096*** -0.116 - -0.076 Number of children 5-17 -0.085*** -0.100 - -0.071 Constant 0.458*** 0.401 - 0.516 0.372*** 0.299 - 0.445 0.384*** 0.318 - 0.450 0.334*** 0.268 - 0.399 0.330*** 0.266 - 0.394 0.330*** 0.265 - 0.395 0.656*** 0.540 - 0.772 N 3,894 3,894 3,891 3,891 3,891 3,891 3,891 R-squared 0.100 0.132 0.135 0.164 0.174 0.167 0.333 *** p<0.001, ** p<0.01, * p<0.05, + p<0.10 Model 1 Model 2 Model 3 Model 4 Model 5 Model 6 Model 7 VARIABLES Coef. 95% CI Survey round (ref.:2013) 2018 0.290*** 0.225 - 0.356 Household agriculture status Acquired agricultural credit (in cash or in-kind) in the last 12 mos. (ref.: No) 0.075* 0.009 - 0.142 Made savings in the last 12 mos. (ref.: No) 0.100** 0.038 - 0.162 Used (at least two) value chain activities(ref.: No) 0.062** 0.023 - 0.102 Used at least two sustainable crop practices (ref.: No) 0.056** 0.019 - 0.094 Used at least two sustainable livestock practices (ref.: No) 0.186*** 0.147 - 0.225 Household food security status -0.102*** -0.154 - -0.050 Household experienced hunger (ref.: did not experience hunger) Department (ref.: Quiche) -0.025 -0.101 - 0.052 Huehuetenango 0.144*** 0.082 - 0.207 Quetzaltenango Project-level factors -0.087*** -0.133 - -0.041 Received cash transfers (ref.: did not participate in any project activities or participated but did not receive cash transfers) Household sociodemographic characteristics Household head age (ref,: 15-24) 0.042 -0.023 - 0.107 25-34 0.096** 0.024 - 0.168 35-44 0.142** 0.056 - 0.228 45-54 0.186*** 0.104 - 0.268 55-64 0.120** 0.037 - 0.203 65+ 0.104*** 0.057 - 0.152 Female-headed household (ref.: Male) Household head educational level (ref.: none) 0.141*** 0.105 - 0.178 Primary or less 0.282*** 0.188 - 0.375 Secondary, vocational/technical or higher -0.042*** -0.060 - -0.023 Number of adult males -0.045*** -0.066 - -0.025 Number of adult females -0.108*** -0.126 - -0.091 Number of children under five -0.095*** -0.106 - -0.083 Number of children 5-17 Constant 0.714*** 0.620 - 0.808 N 3,677 R-squared 0.434 *** p<0.001, ** p<0.01, * p<0.05, + p<0.10 Table A7.7. Results of OLS regression of the natural logarithm of daily per capita consumption expenditures (2010 constant USD at 2005 PPP) excluding influential cases, SC-PAISANO [Guatemala 2013, 2018] NOTE: Model excludes influential observations that can affect the estimation of the coefficients. Cook's D was used as a measure of influential cases to identify cases (households) with high residuals and leverage using the convention cut-off point 4/n where n is the number of observations (in this case 4/3,894). A total of 154 households from the baseline sample and 63 households from the endline sample were flagged as influential observations and were excluded from this analysis.