EVALUATION Long-term Impact Evaluation of the Malawi Wellness and Agriculture for Life Advancement Program Expanding the Reach of Impact Evaluation (ERIE) Consortium This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Ariel BenYishay, Kristen Velyvis, Katherine Nolan Lila Kumar Khatiwada, Carrie Dolan, Danice Brown Guzman, Tom Purekal October 2019 2 Table of Contents Executive summary....................................................................................................................... 3 Results summary................................................................................................................................................................................................... 4 Overall conclusions............................................................................................................................................................................................... 8 Acknowledgements....................................................................................................................... 9 Glossary .......................................................................................................................................... 9 List of tables and figures ............................................................................................................. 10 Introduction................................................................................................................................. 11 Overview of WALA project............................................................................................................................................................................... 11 Theory of change................................................................................................................................................................................................ 12 Goals of the long-term impact evaluation .................................................................................................................................................... 13 Overview of research questions...................................................................................................................................................................... 13 Data collection methodology and plan ......................................................................................................................................................... 15 Qualitative methods........................................................................................................................................................................................... 18 Quantitative methods........................................................................................................................................................................................ 21 Potential threats to the evaluation................................................................................................................................................................. 22 Results .......................................................................................................................................... 25 Research Question 1: To what extent have WALA interventions and related activities continued to take place after the program’s completion?...................................................................................................................................................................................... 25 Research Question 2: What was the impact of WALA on resiliency?................................................................................................. 50 Research Question 3: What was the impact of WALA on child nutrition? ........................................................................................ 56 Research Question 4: Did improvements in resiliency due to WALA allow communities to maintain child nutrition gains even in the face of environmental disruption?............................................................................................................................................ 61 Research Question 5: Did gains in child nutrition due to WALA lead to subsequent improvements in health and education at older ages? ...................................................................................................................................................................................................... 63 Conclusions .................................................................................................................................. 65 References .................................................................................................................................... 67 Appendix: Shocks and stresses in Southern Malawi ................................................................ 69 Abstract Are people better off once development projects end? Are households and communities more self-reliant after projects have spent hundreds of thousands—or millions of dollars? To improve development outcomes and justify future spending, government agencies and philanthropic organizations need to know if the resources they provide are actually transforming people’s lives. The Expanding the Research of Impact Evaluation (ERIE) team worked with the USAID Office of Food for Peace on a long-term follow up of The Wellness and Agriculture for Life Advancement (WALA) program in Malawi, which was implemented from 2009-2014, at a cost of $80,900,000. Researchers from ERIE were asked to study the program 4 years after it ended to see how people’s lives had changed, if they were more resilient in the face of severe drought and flood, and whether or not the program had sustainable impacts on recipients. In order to assess WALA’s long-term impacts, the ERIE team created a counterfactual that did not exist before - identifying a group of villages to compare to the treated WALA villages. Using this methodology, ERIE researchers were able to understand WALA’s long-term, sustainable impacts on households and communities. Specifically, they were able to understand the impact of WALA on household resiliency and child nutrition, the extent to which WALA interventions and related activities continued once the project ended, whether or not child nutrition gains were maintained in the face of environmental disruptions, and if gains in child nutrition led to other improvements in health and education at older ages. WALA Award number: P-A-09-00001-00 3 Executive summary The Wellness and Agriculture for Life Advancement (WALA) project, which operated from 2009-2014, sought to improve the food security and vulnerability of 214,974 chronically food insecure households in eight districts in Southern Malawi (Nsanje, Chikwawa, Thyolo, Mulanje, Zomba, Machinga, Chiradzulu, and Balaka). This project was funded by the Office of Food for Peace (FFP) of the United States Agency for International Development (USAID). WALA was implemented by a consortium of organizations that included Catholic Relief Services (CRS)/Malawi, ACDI/VOCA, Africare, Chikwawa Catholic Diocese (CCD), Emmanuel International (EI), Project Concern International (PCI), Save the Children (SAVE), Total Land Care (TLC), and World Vision International (WVI). Project activities included maternal and child health projects; nutrition, agriculture, natural resource management, irrigation, and economic activity treatments; and disaster risk reduction activities. Sustainability of the project relied on household members’ continued motivation to use the practices taught and supported under WALA, as well as continued access to public and private services. As initial trends looked promising, USAID commissioned an evaluation to assess the long-term impacts of WALA. At endline, the project reported a substantial reduction in child undernutrition prevalence rates among project beneficiaries, as well as in communities’ need for food aid during crises. These changes took place within WALA-treated villages, but with no comparison group identified, it was not clear whether these changes could be attributed to WALA activities. Moreover, after WALA activities ended in 2014, it was possible that some gains had faded while others had been sustained. In order to assess WALA’s long-term impacts, the Expanding the Reach of Impact Evaluation (ERIE) team created a rigorously identified group of comparison villages to compare to the treated WALA villages. These comparison villages were identified by using data from the 2010 Demographic and Health Survey (DHS) and a variety of spatial covariate datasets that reflect pre-WALA conditions. The team combined these data to generate an interpolated gridded geographic surface of the percentage of stunted under-5 children at baseline in 2009. We used this surface to match each WALA village with a comparison village that had similar stunting rates at baseline. The final evaluation sample then included 100 village pairs randomly selected from the full set (with 100 WALA villages and 100 comparison villages). Both quantitative and qualitative methods were utilized to collect follow-up data from WALA villages and comparison villages. The quantitative survey was administered in 2018 to 21 households in each of the 200 selected villages across the eight WALA districts. A subset of the sample involved revisiting households interviewed in 2013 as part of the project endline survey, while the remainder were randomly selected households. The household survey questionnaire covered health, nutrition, agriculture, income, and development project participation. It also included an under-5 child nutrition questionnaire, which included questions on maternal health and child nutrition. The enumeration team also collected the anthropometric measurements of children aged 6-59 months. Since the comparison group was initially selected based on comparability in predicted baseline stunting rates, it was possible that residual prediction error could potentially have resulted in a comparison group that was different from the treatment group in ways that were correlated with long-term outcomes. If this was the case, causal attribution based on comparisons between these groups could be biased. However, the research team confirmed that the treatment and comparison groups were indeed comparable by looking at a variety of demographic indicators in the household survey that were unlikely 4 to change quickly. An additional threat to the evaluation included potential contamination in the comparison group from other development activities. Should other development activities specifically target comparison group villages, it could attenuate any differences in true treatment status across the groups and thus hinder the research team’s ability to causally attribute results to these treatments. In order to address this potential threat, the household survey collected information about household participation in development activities in the past ten years and today. The data shows that households in WALA villages were 7.5% more likely to report having participated in at least one development activity than those in the comparison group. Given that WALA engaged a subset of households within villages and that a number of technical reasons shaded this estimate downwards, this suggests that the comparison group indeed reflects the counterfactual conditions for credible causal impact estimation. Qualitative data collection was completed in 14 villages across four of the eight districts. Two districts were served by WALA follow-on projects (UBALE and Njira) while the other two districts did not have either follow-on project. The researchers selected villages that were surveyed by the quantitative team and that were nominated by implementers as locations where the WALA project was well￾implemented, the community was receptive, and interventions were perceived to be most successful in terms of adoption and outcomes. On top of this, researchers also looked at villages with the largest drop in stunting rates during the project. From this selection criteria, the qualitative team selected 14 villages where they conducted a total of 28 focus group discussions and 28 key informant interviews. The team also interviewed a total of eight implementing partners and eight government officials. During the interviews, the discussion covered a wide range of topics including perspectives on WALA activities in general and their sustainability, household resiliency, coping mechanisms, livelihoods in the face of shocks, diversification of income sources, and reliance on direct food aid. Results summary 1.To what extent have WALA interventions and related activities continued to take place after the program’s completion? We assess the continuation of WALA activities in both the full quantitative sample of 200 villages and the 14 particularly successful villages in which we carried out qualitative interviews. In our full sample, while there had been quantitative differences in participation in agricultural, livestock, and environmental activities between WALA and comparison villages over the past decade, there are no remaining differences in participation in these activities currently. Even in the particularly successful villages in our qualitative subsample, nine out of the ten agricultural and natural resource activities were either discontinued or had mixed results for sustainability. In the qualitative subsample, only village savings and loans (VSLs) were sustained to a substantial degree after WALA left; however, we find no evidence of differential participation in VSLs in the full quantitative sample. Assessing the primary barriers to sustaining WALA activities using qualitative interviews and surveys, we identified the following: (1) resources generated by the community were not enough to replace external resources which were no longer available; (2) technical and managerial capacity strengthening did not materialize into maintaining interventions as anticipated; and (3) communities were not linking to governmental organizations or other entities for support. Moreover, the shocks and stresses experienced by communities after WALA ended were large and sustained, which may have pushed communities beyond any increase in coping 5 capacity attained through the project. Shocks included pest infestations, droughts, floods and windstorms. These threats caused many of the activities WALA promoted to be discontinued and/or diminished their potential positive impacts enormously, which in turn affected the ability of treatment communities to sustain positive outcomes—even as some of the activities were themselves designed to make communities more resilient to these shocks. WALA promoted at least eight types of health and nutrition activities. In the full quantitative sample, we again observe important differences in participation in health and food-related activities over the past decade between WALA and comparison villages, but there are no remaining differences in participation currently. Digging into the particularly successful qualitative subsample, we identified two types of activities that were discontinued (child nutrition and maternal and child health and nutrition rations), two types that were sustained (sanitation and latrines), and four types that had mixed results for sustainability after WALA ended. Linkages with local leaders and governmental agencies appear to have had the largest positive effect on the sustainability of WALA health and nutrition activities. Community chiefs and other government agencies enforced regulations against open defecation and encouraged community members to follow them, which contributed to the sustainability of the sanitation and latrine activities WALA promoted. Threats to the sustainability of these health and nutrition activities were quite similar to those constraining agricultural and environmental activities. One of the main successes reported through qualitative interviews was the promotion of village banks, or VSL (village savings and loans) programs. This increased households’ access to cash in the form of both savings and microloans. VSLs were frequently mentioned as a support that played a critical role for households facing shocks after WALA ended. However, these results are unique to the qualitative subsample that was purposely selected for successful implementation of program activities. The results from the full quantitative sample show no statistical difference between VSL participation in WALA and comparison villages. 5.6% of households in comparison villages participated in VSL programs in the past 10 years compared to 6.0% in WALA villages. Today, 2.9% of households still participate in VSL programs in comparison villages while 2.3% still participate in WALA villages. 2.What was the impact of WALA on resiliency? Southern Malawi has experienced a remarkable set of shocks and stresses over the past decade, including droughts, floods, high winds, market failures, crop pests, and diseases. WALA sought to build households’ and communities’ ability to handle these shocks through a focus on strengthening effective local institutions, providing extension services, and strengthening financial services that promote saving and lending within local communities. The quantitative survey assessed households’ overall perceptions of their own resilience through questions about their shock preparedness and recovery. According to the quantitative survey data, households in WALA-participating villages report worse recovery from recent shocks than households in comparison villages. The share of households reporting only partial recovery is significantly higher in WALA-participating villages than in comparison villages, where more households are more likely to experience complete recovery. This is despite the fact that the frequency and types of shocks reported in WALA and comparison villages are quite similar. When asked about the change in their resilience relative to conditions 10 years ago, households in WALA villages more frequently cite static or worsening resilience than do those in comparison villages. This result is true across a variety of empirical specifications, control variables, and robustness checks. 6 WALA promoted conservation agriculture practices to address adverse climatic changes affecting agriculture. In qualitative discussions, participants in these activities reported that practices such as crop rotation and mulching enhanced crop production, reduced soil erosion, helped conserve soil moisture, and protected crops from being washed away by storms. These skills were designed to help farmers adapt to shocks such as floods and droughts by reducing their impact. WALA also promoted practices to improve the nutrient content of soil, such as the application of manure fertilizer enriched soil to increase yields. Other WALA-promoted practices included dam construction for water conservation; promotion of new crop varieties that mature more quickly; crop diversification; and training sessions on food management and budgeting. While initially successful, these agricultural practices were overpowered by shocks including droughts, floods, winds, pests, and market failures. Irrigation systems and WALA’s increased access to markets programs also failed after WALA ended. Because the shocks villages experienced were so large, frequent, and widespread, they dampened the impact of the previously successful village savings and loans groups. WALA-implemented activities did not have many linkages to outside areas and government responsiveness was slow or nonexistent, leaving households’ living conditions unstable and their resilience capacities overrun. WALA introduced new ideas and practices, many of which participants considered useful. People adopted different strategies that were provided by WALA in order to develop resilience during shocks. A minority of households who are using the technologies provided by WALA have increased their resilience. However, the majority did not end up more resilient. Evidence has shown that resilience has been particularly hard to achieve when the shocks overwhelm coping strategies. These results have been difficult to explain and merit further research. 3.What was the impact of WALA on child nutrition? At baseline in 2009, 42.4% (95% CI of 39.7%-45.2%) of children aged 6-59 months in WALA villages were identified as stunted, which dropped to 37.1% (34.9%-39.3%) at endline in 2013. The 2018 follow￾up survey found that the stunting rate was 34.2% (31.9%-36.5%). The trends for the percentage of underweight children was similar. At baseline in 2009, 17.6% (15.6%-19.6%) of children 0-59 months were considered underweight. At endline in 2013, this number had decreased to 11.3% (9.9%-12.7%) and stayed consistent at follow-up in 2018, remaining at 11.8% (9.7%-13.8%). These numbers align with the national 2015/2016 DHS survey that reported 37% of children under 5 years old were stunted and 12% were underweight. Since the team was able to return to WALA households at endline, this created the opportunity to develop a panel dataset for those treated households. Since no comparison villages were surveyed at endline, we are unable to create the same panel dataset for comparison locations. The data showed that there are similar stunting rates between children aged 0-59 months in the revisited WALA households at endline (36%) and their younger siblings that were 0-59 months at follow-up in 2018 (37%). The data shows a similar pattern for underweight rates at endline (14%) and follow-up (16%). The similarity in results when examining village-wide and within-household dynamics further suggest a consistent pattern of static malnutrition and anthropometrics post-WALA. However, these results are within WALA-treated villages only, and do not account for dynamics that may have taken place across Malawi as a whole. After comparing both stunting and underweight status across WALA and comparison villages, we find that there is no statistically significant difference between the groups: the average stunting rate at baseline in the comparison group was equal to the that in the 7 treatment villages, and at follow-up in the comparison villages is 32.6% (lower than the 34.2% rate in WALA villages). After adding controls for UBALE/Njira presence in each village, adjusting for district￾specific fixed effects, and adding controls for a sophisticated set of non-linear adjustments for baseline stunting rates, there is still no significant difference in stunting rates between WALA and comparison villages (these results are detailed in columns 1, 2, and 3 in Table 14). We find similar evidence for underweight status. The average percent of underweight children in WALA villages at follow-up is 11.8% and the percent of underweight children in comparison villages is 12%, with no statistically significant difference between the two groups of villages. For full treatment effect information, refer to Table 14. We are confident that any impacts of WALA on underweight status are quite small. Taken together, these results indicate that WALA has not induced long-term changes in village-wide childhood undernutrition and anthropometrics. The research team also analyzed whether there were discernable impacts from WALA on households’ dietary and food security outcomes. Analysis of WALA impacts on dietary diversity, as measured by the Household Dietary Diversity Score (HDDS), and on the degree of food insecurity experienced by each household shows that, if anything, food security appears to have worsened in WALA villages rather than improved. This result is consistent across specifications and controls and is statistically significant at the 95% confidence level. 4.Did improvements in resiliency due to WALA allow communities to maintain child nutrition gains even in the face of environmental disruption? Even in the face of the numerous shocks reported above, it is still possible that WALA engendered actual resilience in terms of childhood nutrition. If different portions of our sample of WALA-treated villages experienced particularly pronounced shocks that the project helped them weather, our null average treatment effects on childhood malnutrition may obscure important gains in these hard-hit areas. Weather data from the Climatic Research Unit (CRU) joined to the household survey data revealed that both WALA and comparison villages were hit equally hard by the reported weather shocks. With both WALA and comparison villages struggling equally in the face of floods and droughts, it is possible to evaluate whether or not WALA households were better equipped to handle these shocks compared to those villages that did not benefit from the program. After comparing the outcomes between WALA and comparison villages, the quantitative data shows that WALA households’ children were no better off than comparison villages in terms of underweight and stunting rates. The quantitative data collection shows that any improvements in household resiliency from WALA at endline were unable to overcome the environmental disruptions that occurred between the end of the project in 2014 and today. 5.Did gains in child nutrition due to WALA lead to subsequent improvements in health and education at older ages? Despite not finding any significant differences between WALA and comparison villages in stunting and underweight rates among young children at follow-up, it is nonetheless possible that cohorts of children who were young during WALA's implementation and thus may have benefitted from the program continue to experience some long-lasting gains. We focus on educational outcomes for these children, both because anthropometric measures at older ages are not reliable indicators of malnutrition, and 8 because educational outcomes are both directly important and potentially influenced by earlier malnutrition. The analysis revealed that WALA children were 3% more likely to have attended school than children of the same age in comparison villages, with the difference statistically significant at the 90% confidence level. However, there were no statistically significant differences between the two groups in literacy, grade completed, or current health status. These results indicate some limited gains for children who were young during WALA implementation, suggesting some limited, longer-lasting impacts from WALA's interventions. Overall conclusions Previous performance evaluations and case studies of WALA activities indicated that the program successfully introduced a variety of new ideas and practices around sustainable, productive agriculture, public health, and other themes. However, in both the qualitative sample and our larger quantitative sample of villages that received household surveys, we find only limited evidence of sustained use of these practices and no differences in longer-term adoption. We also observe no differences in village￾wide child nutrition conditions between WALA villages and comparison sites. We find very slight traces of some potential benefits from (temporary) participation in WALA in early childhood, as children who were young during WALA's implementation are now slightly more likely to have attended school than those in comparison villages. Potential explanations for why WALA-promoted practices were not sustained include particular program features and, more broadly, shocks faced by villages in southern Malawi that outstripped communities' resilience and overwhelmed individual coping strategies. It is possible that WALA￾promoted practices that focused on intra-village strategies were too narrow and small-scale to adequately protect households. The analysis also revealed that households in WALA villages today have higher rates of food insecurity and weaker resilience than comparison villages. While the data does not allow us to definitively distinguish between potential explanations, it is possible that households expended extra effort to switch to WALA-promoted practices, and the severity of the shocks wiped out any gains from those investments. This left households worse off and more vulnerable than they would have been otherwise. It is also possible that households were more exposed than they had been previously because they shifted into these new practices. Neither of these cases suggests that agricultural practices promoted under WALA were inherently flawed; instead, these practices need to be adjusted to cope with the risk of major shocks and complemented by much broader insurance and social protection schemes that extended beyond village or even district scales. 9 Acknowledgements This publication was produced for the United States Agency for International Development. It was prepared by the Notre Dame Initiative for Global Development, AidData at William & Mary, and Mathematica. We acknowledge the invaluable financial support, discussions, and feedback of the USAID Evaluation and Impact Assessment, Food for Peace, and Malawi mission offices. The 2018 follow-up round of data were very capably collected by Wadonda Consult Ltd under the leadership of Ephraim Wadonda Chirwa, who sadly passed away after the data collection. Other contributing Wadonda team members included Peter M. Mvula, John Phuka, and Miriam M. Matita. Lloyd Banward at Tango International graciously provided details and access to prior rounds of data. We would also like to acknowledge the survey respondents for the contribution of their time and knowledge. Glossary ADB – Asian Development Bank CA – Conservation agriculture CBO – Community-based organization CCFLS – Community complimentary feeding and learning sessions CRU – Climatic Research Unit DHS – Demographic and Health Survey ERIE – Expanding the Reach of Impact Evaluation FGDs – Focus group discussions GADM – Database of Global Administrative Areas GVH – Group village headman HDDS – Household Dietary Diversity Score KIIs – Key informant interviews LSMS – Living Standards Measurement Surveys MCHN – Maternal and child health and nutrition PSPs – Private Service providers TA – Traditional Authority UBALE – United in Building and Advancing Life Expectations WALA – Wellness and Agriculture for Life Advancement USAID – United States Agency for International Development VCPC – Village civil protection committee VSL – Village Savings and Loans 10 List of tables and figures Figure 1. WALA interpolated surface of child stunting rates with WALA and comparison villages...........17 Table 1. Geospatial data sources................................................................................................................................. 18 Table 2. Qualitative interviews.....................................................................................................................................19 Table 3. Demographic comparisons across WALA and comparison group households...............................22 Table 4. Frequency of main development activities................................................................................................. 24 Table 5. Sustainability of WALA activities in agriculture, agribusiness, environment and natural resource management, and disaster risk reduction in the villages sampled for qualitative research as most successful and engaged in WALA ................................................................................................................................27 Table 6. Statistical comparisons for agricultural practices between WALA and comparison villages........30 Table 7. Statistical comparisons for conservation agricultural practices between WALA and comparison villages.................................................................................................................................................................................31 Table 8. Statistical comparisons for natural resource management practices between WALA and comparison villages..........................................................................................................................................................32 Table 9. Factors facilitating and hindering the sustainability of WALA activities in agriculture, agribusiness, environment and natural resource management, and disaster risk reduction .........................35 Table 10. Sustainability of WALA activities in health and nutrition ....................................................................42 Table 11. Factors facilitating and hindering the sustainability of WALA activities in health and nutrition 47 Table 12. Household recovery from the most recent shock ...............................................................................51 Table 13. Household shock preparedness relative to 10 years ago....................................................................52 Figure 2. Trends in WALA villages in childhood anthropometrics .....................................................................57 Figure 3. Comparing children in 2013 to younger siblings in 2018 in WALA villages....................................58 Table 14. Stunting and underweight treatment effects...........................................................................................58 Table 15. Food insecurity and HDDS with controls...............................................................................................59 Table 16. Growing season precipitation rates..........................................................................................................62 Table 17. Educational results........................................................................................................................................63 Table 18. Reported shocks from 2014-2018 in WALA implementation areas................................................69 Figure 4. Reported frequency of shocks from 2014-2018 in WALA implementation areas and comparison areas.............................................................................................................................................................70 Figure 5. Timeline of Shocks in Southern Malawi from 2014 - 2018 ..................................................................71 11 Introduction The Expanding the Reach of Impact Evaluation (ERIE) consortium has partnered with USAID’s U.S. Global Development Lab, the USAID Malawi Mission, and the Office of Food for Peace (FFP) to assess the long-term impacts of the Wellness and Agriculture for Life Advancement (WALA) project that was carried out from 2009 to 2014. ERIE partners on this evaluation include faculty and staff from AidData at William & Mary, the Notre Dame Initiative for Global Development (NDIGD), and Mathematica. The research team has combined quantitative and qualitative data collection to evaluate the lasting effects of the WALA project. This Evaluation Report lays out the context, theory of change, motivation for the evaluation, research questions, data sources, analytical methods, evaluation risks and challenges, analysis results for each research question, and policy conclusions. Overview of WALA project The goal of the Wellness and Agriculture for Life Advancement (WALA) project was to improve the food security of 214,974 chronically food insecure households in 39 Traditional Authorities (TAs) in eight districts in southern Malawi by 2014 through various strategic activities. This project was funded by the Office of Food for Peace (FFP) of United States Agency for International Development (USAID). WALA was implemented by a consortium of organizations that included Catholic Relief Services (CRS)/Malawi, ACDI/VOCA, Africare, Chikwawa Catholic Diocese (CCD), Emmanuel International (EI), Project Concern International (PCI), Save the Children (SAVE), Total Land Care (TLC), and World Vision International (WVI). The project concluded in 2014 and the performance evaluation results showed positive changes on a number of outcome indicators. The project had the following strategic objectives: • Maternal and child health and nutrition (SO1): 170,724 vulnerable households have improved maternal and child health, and nutrition status. • Agriculture, natural resource management, irrigation, and economic activity (SO2): 147,500 smallholder farming households have improved livelihood status. • Disaster risk reduction (SO3): 273 targeted communities have improved capacity to withstand shocks and stresses. The performance evaluation report of WALA cited “impressive improvements in child undernutrition prevalence rates” among the beneficiaries. In addition, case studies such as those done on the Lingoni site demonstrated that some WALA communities may have been more resilient, requiring less food aid during crises. Examining WALA’s outcomes four years after the project's completion, we now aim to understand if these and other impacts were sustained. ERIE’s long-term retrospective approach can identify if the food security situation of households has changed as a result of the WALA activity, and if child nutrition and household dietary diversity gains were sustained. Moreover, by adding a rigorously identified comparison group, the present evaluation can assess the extent to which WALA activities caused improvements in child undernutrition and community resilience. 12 The sustainability of outcomes after project completion is a key goal for many donors, but as many as 40% of all new projects are not sustained beyond the first few years following cessation of external funding (Bamberger and Cheema 1990; ADB 2010). An ADB study (2010) of 491 projects showed key sustainability issues varied by sector. For example, in the water supply and sanitation sector, constraints to sustainability included lack of revenue-generating powers. For the health and social protection sector, a key factor facilitating sustainability was the institutionalization of new structures and systems. For the agriculture and natural resources sector, constraints for projects in this study were limited implementation of project activities and inadequate programs for maintenance. In addition, we still know relatively little about the sustainability of gains in many areas, including food security efforts. The present study of the WALA project adds to this body of literature while providing evidence of the impact of the activity. This evaluation offers the first impact estimates of WALA's interventions based on (rigorously identified) comparison sites. We selected these comparison sites by using data from a DHS round conducted just before the project's implementation to model stunting rates across Southern Malawi (adding a variety of other geospatial data and using cutting-edge geo-computational techniques to do so). Together with an array of robustness checks, this approach provides us with reliable long-term impact estimates from WALA's interventions. Ideally, we would be able to separately identify impacts at the project's 2014 end, and its impacts in the longer-term, tracing out the time path for such impacts. Because no data was collected in comparison sites at the end of WALA, we are unable to do so. Thus, we cannot discern whether the project generated shorter-term impacts that were not sustained, or whether it created no meaningful gains even during its implementation. Theory of change WALA aimed to sustainably improve food security and reduce vulnerability among chronically food insecure households and communities through demonstration of optimal health and nutrition behaviors during critical periods; adoption of sustainable agriculture, environment and natural resource management practices; and establishment of mechanisms to mitigate and respond to shocks. Adoption of gender equitable and inclusive decision-making processes and demonstration of these behaviors and practices aimed to: (a) prevent undernutrition of children under 2; (b) permit households to increase income and build assets through increased production, diversification and market linkages, and increased and diversified income sources; and (c) contribute to increased risk management to improve household and community resilience to shocks and reduce natural resource degradation. The WALA project worked closely with different Government departments and Ministries to get their buy-in and strengthen capacity. It also used market-based approaches and strengthened community capacity to continue many of the necessary services and infrastructures built, repaired, or improved by WALA. Sustainability of the outcomes produced by WALA would largely depend on household members’ continued access to quality services and motivation to maintain the practices; systems to facilitate service delivery and equitable access to public and private services and inputs; incentives for and capacity of the public and private service providers to continue to provide high quality services; and a strong demand for these services. 13 Goals of the long-term impact evaluation The post-project evaluation included a detailed look at the following: • Evaluate the key food security and nutritional outcomes against the outcomes measured in the performance evaluation of WALA and assess whether the outcomes can be attributed to the sustainability strategies implemented by WALA. • Assess whether WALA target communities have continued, scaled up, or wound down activities promoted by WALA. Identify which activities were and were not sustained, and the factors that contributed to either status. Of particular interest is the uptake of WALA-introduced interventions (e.g. practices, techniques, basic service provision, latrines, etc.) among indirect beneficiaries within intervention communities. • Evaluate if the continuation, functionality, and role of various community-based institutions and mechanisms developed or strengthened by WALA worked as a conduit of community- level changes, capacity building of public and private service providers, and as a catalyst to promote social accountability and justice. • Identify unintended consequences (both positive and negative), best practices, and key elements that were essential to sustaining the activities and practices. • Understand how well the WALA communities absorbed the effects of weather shocks and adapted their strategies to maintain their food and nutrition security. Did household resilience capacities help households manage and/or recover from the recent 2015-2016 drought? Based on the project objectives, USAID Washington’s and the USAID Malawi Mission’s interests in long￾term impacts of the WALA interventions, and ERIE consortium members’ suggestions, the following research questions were explored through an impact evaluation. Overview of research questions 1.To what extent have WALA interventions and related activities continued to take place after the program’s completion? To understand the program’s long-term impacts, we assessed the continuation, scaling or cessation of WALA interventions and related activities either through externally supported interventions or independently, as well as the presence and activities of community-based organizations. We measured these in both treatment and comparison villages, while accounting for multiple sources of engagement, including USAID-funded follow-on projects United in Building and Advancing Life Expectations (UBALE) and Njira, government ministry activities, and other projects with similar sets of interventions.1 Interventions and activities considered included: 1 These follow-on interventions are not the focus of the current evaluation, whose aim is to attribute potential impacts to the WALA interventions. 14 • Agricultural, agribusiness, and natural resource management interventions • Disaster risk reduction activities • Nutrition programs • Health and sanitation • Village savings and loans • Watershed management and irrigation • Access and availability of food aid 2.What was the impact of WALA on resiliency? In the face of environmental disruptions, such as the El Niño-induced drought during the 2015-16 growing season, are WALA households and communities better able to demonstrate the following: • Improved coping mechanisms or responses • Access to or function of resilience capacities • Maintenance of sustainable livelihoods in the face of shocks • Continued natural resource management • Diversification of income sources, into livelihoods with different risk factors • Reduced need for direct food aid • Maintenance of healthy nutrition status In answering this question, we considered communities’ differential exposure to shocks and differential outcomes and the connections of these outcomes to community and household self- reliance. 3.What was the impact of WALA on child nutrition? By the end of WALA in 2014, beneficiary villages had experienced a 5.3 percentage point reduction in stunting since the baseline, a 6.3 percentage point reduction in underweight status, and an improvement in household dietary diversity. We therefore sought to address the following questions: • Had this lower level of stunting and underweight status in beneficiary villages been maintained in the four years since the project’s completion? Had it continued to decline, stabilized, or reverted to higher levels? • What changes in stunting and underweight status do we observe in comparison villages that did 15 not benefit from WALA over this time period? What can we conclude about WALA’s causal impacts? • Do we observe current differences in household dietary diversity between treated and comparison villages? • To what degree do overall food availability, feeding practices, or other factors account for the impacts we observe? 4.Did improvements in resiliency due to WALA allow communities to maintain child nutrition gains even in the face of environmental disruption? We aimed to assess whether we observe sustained child nutrition impacts even in WALA communities that experienced particularly difficult drought conditions (relative to comparison communities that also experienced these conditions). 5.Did gains in child nutrition due to WALA lead to subsequent improvements in health and education at older ages? We sought to trace any impacts on health and education outcomes among children who were under 5 years old during WALA implementation and are now between 4 and 9 years old. Data collection methodology and plan The ERIE research team employed both quantitative and qualitative methods to answer the above research questions. Addressing questions two, four, and five depended primarily on quantitative analysis, while tackling questions one and three made use of both quantitative and qualitative methods. Our team used a combination of existing data sources and new data collection to address these questions, as described below. Existing data We made use of a number of existing microdata sources in our evaluation, including: • The 2013 endline household survey microdata collected by Tango International, including child stunting and underweight status, households’ participation in WALA activities, and other characteristics. • The 2010 and 2004 Demographic & Health Survey (DHS)2 nationally-representative microdata, including child stunting and underweight status and the cluster GPS coordinates provided by DHS. These data will be vital to constructing our comparison group (as discussed in the 2 Because we rely on the DHS for pre-program baseline data, we do not use the 2016 DHS round. 16 following section) and adjusting for any differential pre-trends thatmay have occurred. • Spatiotemporal covariate datasets (described below in Table 1), for constructing our comparison group of villages and in our machine learning models predicting maize production. • Satellite imagery from the Landsat program,3 from which we utilized crop greenness measures. • The 2010-2011 and 2016-2017 Living Standards Measurement Surveys (LSMS) microdata, including maize production and geospatial features.4 • Precipitation levels from the Climatic Research Unit (CRU) data5 collected from weather stations around the world, to produce a global surface at 0.5 decimal degree resolution. In addition, we made use of WALA’s Baseline Report6 and Endline Report7. New data collection To understand the current rates of child stunting and underweight status, we collected an additional round of household survey data within the 8 districts in which WALA worked. We combined this survey with qualitative research, including focus group discussions (FGDs) and key informant interviews (KIIs), most notably to examine the influence of WALA on resiliency. Our survey sample was composed of both WALA villages and comparison villages, selected based on the similarity in their predicted baseline child stunting rates. Our qualitative research was conducted in a purposively sampled subset of the survey sample, as described below. Comparison group construction To construct this matched sample, we used the 2010 DHS round data to reflect pre-WALA conditions, as well as a variety of spatial covariate datasets (see the list of data sources included in Table 1 below) to generate an interpolated gridded surface of the percentage of stunted under-5 children at baseline. These covariates were chosen because they have previously demonstrated utility in predicting child stunting in Malawi.8 The team used an Empirical Bayesian Kriging (EBK) regression prediction approach within ArcGIS Pro to generate a spatially continuous child stunting layer for Malawi (c.f. Gething et al 2015). EBK Regression Prediction is a geostatistical interpolation method that combines kriging with regression analysis to make predictions that are more accurate than either regression or kriging can achieve on their own. In EBK Regression Prediction, explanatory variables are transformed into principle components prior to modeling, solving the problem of multicollinearity and ensuring stability without the loss of accuracy.9 3 Landsat 7. URL: https://landsat.usgs.gov/landsat-7 (visited March 2019). 4 National Statistics Office (NSO). Malawi Fourth Integrated Household Survey 2016-2017. ID Number: MWI_2016_IHS-IV_v02_M. Available at: http://microdata.worldbank.org/index.php/catalog/2936/related_materials 5 Harris et al. 2014. Updated high‐resolution grids of monthly climatic observations – the CRU TS3.10 Dataset. doi:10.1002/joc.3711 6 Consortium Administration & Technical Capacity Hub,WALA. 2010. “Wellness & Agriculture for LifeAdvancement: Baseline Survey Report”. 7 TANGO International. 2014. “End-line Survey Report CRS MALAWI WALA PROGRAM 2009-2015”. 8 Spatial Data Repository. The Demographic and Health Surveys Program. Modeled Surfaces. MW2015_DHS_CNNUTSCHA2_MS_v01. ICF International. Funded by USAID. Available from spatialdata.dhsprogram.com. Accessed 12APRIL19 9 Empirical Bayesian Kriging – ArcGIS Pro. https://pro.arcgis.com/en/pro-app/help/analysis/geostatistical-analyst/what-is-ebk-regression-prediction- .htm 17 We then overlaid the sample of 240 WALA villages in which the 2013 endline survey was conducted, thus obtaining estimates of baseline (2009) rates in this sample. For each of these villages, we then identified the nearest matches in predicted percentage of stunted under-5 children at baseline from the pool of non-WALA villages. This matching was done district-by-district to ensure comparison villages were sufficiently similar in other respects to WALA villages. We thus obtained 240 village pairs of WALA-treated and comparison areas. For more information on the creation of the interpolated surface, see “WALA2018_Interpolation_Narrative.pdf” in the annex. Figure 1. WALA interpolated surface of child stunting rates with WALA and comparison villages We also aimed to assess the extent of spillovers from WALA villages to nearby comparison villages. The geographic distribution of WALA and comparison villages was sufficiently dense such that some comparison group villages were near a number of WALA villages, while others were quite far from any WALA villages (discussed in further detail below). 18 Table 1. Geospatial data sources Covariate Description Data source Travel times The amount of travel time it takes to reach a settlement of 50,000 or more Andrew Nelson, European Union GEM Unit, 2000 Temperature Average temperature for months January to December in degrees Celsius WorldClim Version 2, 2009 Rain Average rainfall Climate Hazards Group InfraRed Precipitation with Station data 2.0, 2010 Nighttime lights Average radiance of the cells whose centroid falls within a radius of 10km (rural) or 2km (urban) Version 1 VIIRS Day/Night Band Nighttime Lights, 2015 Enhanced Vegetation Index Calculated by measuring the density of green leaves in the near-infrared and visible bands Vegetation Index and Phenology (VIP) Phenology EVI-2 Yearly Global 0.05 Deg CMG V004, 2010 Evapotranspiration Average potential evapotranspiration CGIAR-CSI Global Aridity and Global PET Database, 2009 Elevation Topography-based elevation estimate Nasa SRTM Aridity Average aridity index of the cells whose centroid falls within a radius of 10km (rural) or 2km (urban). Aridity is calculated by dividing the actual evapotranspiration by the potential evapotranspiration. CGIAR-CSI Global-Aridity and Global-PET Database, based on 1960-1990 climate data All population Average number of people in the cells whose centroid falls within a radius of 10 km (for rural points) or 2km for urban points World Pop, 2010 Qualitative methods Sample selection There were three stages of sampling for the qualitative sample. The first stage was at the district level. We selected four districts that represented the different types of districts where WALA was implemented. We sought two districts where UBALE or Njira were serving villages that had been 19 served by WALA, and two districts where UBALE and Njira were not implemented. This strategy was used so that we could better understand the experiences of those who received follow-up from USAID and those who did not. The districts selected were Zomba and Chiradzulu (WALA-only districts), Chikwawa (served by UBALE), and Machinga (served by Njira). Following district sampling, we selected villages for qualitative sampling from the villages where the quantitative survey was to be conducted. We selected three villages per district where USAID had followed up and four villages per district where USAID had not, for a total of 14 villages. We contacted the implementers who worked in each district and requested nominations of villages where the program was well-implemented, the community was receptive, and where they perceived the interventions most successful in terms of adoption and nascent outcomes. We also looked at stunting data from 2010 and 2016 and targeted villages that had the largest drops in stunting between those two time periods. The goal was to study the villages where WALA implementation was well-done and large changes in stunting had occurred, in the hopes of learning about the interventions that worked well and how they worked under best-case scenarios. This allowed us to learn about the potential of these interventions when implemented as well as possible and identify lessons of success. This strategy was designed to maximize the chances of locating positive deviance, which would further allow us to identify feasible mechanisms to leverage WALA activities most effectively. Within each selected district, we conducted an average of 18 interviews. In each district we interviewed two implementing partner staff members who worked on WALA, selecting when possible those who were most involved with implementation, most knowledgeable about WALA, and had sustained contact with WALA villages since the conclusion of WALA activities. We also sampled two government agents per district who had worked with the WALA project. These district- and local-level officials were selected based on their involvement with and knowledge about the implementation of the WALA activities and the villages in which the activities were undertaken. We also sought participants who could provide historical perspectives regarding WALA and other programs implemented in WALA villages since WALA ended. Table 2. Qualitative interviews Participant types Type of guide Number of interviews / focus groups Implementing Partners Key informant interviews 8 (2 per district) Government Officials Key informant interviews 8 (2 per district) Community Members Focus group discussions 28 Groups (2 per village, one all- male and one all-female) Resilient households Key informant interviews 28 (2 per village) Total 72 interviews and focus groups using 4 interview guides 20 Within each selected village, we conducted two focus groups, one with men and one with women, all of whom participated in at least one WALA intervention. In Chikwawa and Machinga, the two districts with USAID there was an added criterion that focus group members also were involved in at least one Njira or UBALE activity. In each selected village, we also conducted two key informant interviews with members of resilient households. These households were identified through conversations with village heads and one or two other village members, preferably those who had been in some leadership position in the WALA activities. The criteria for resilient households were that they: (1) were households within the village; (2) had participated in WALA; (3) had food throughout most of the year; (4) had diversified income sources; and (5) were able to bounce back in the event of a shock. In each village, two of the most resilient households were chosen and one interview was conducted per household. Each interview could be conducted with one member of the household or several members, either together or consecutively. Data collection occurred between November 2 and December 2, 2018. Topics The topics for discussion with WALA implementers included the activities that were implemented in the villages they worked in, and whether they or others were still supporting the communities with these activities. We also asked them about their perspectives on the factors that have supported the continuation or expansion of these activities or that have led to their discontinuation and their perspectives on the effectiveness of linkages established during the project. For district and local government officials who were involved with WALA, topics covered in interviews included their perspectives on the impact of WALA on their own and other government officials’ capacity strengthening, their integration in the project, and their knowledge of other similar projects implemented in WALA villages. We also asked about their perspectives on community members’ motivation to maintain the practices, equitable access to public and private services and inputs, incentives for and capacity of the public and private service providers to continue to provide high-quality services, and community member demand for these services. At the village level, topics that were covered with WALA participants included resilience of communities and households. In particular, we focused on coping mechanisms, livelihoods in the face of shocks, natural resource management, diversification of income sources, and reliance on direct food aid. We also asked how the communities selected define and experience resilience and to link these findings with risk reduction activities and interventions. Finally, in interviews with members of resilient households, we covered topics including factors that helped their households attain or maintain their pathway of resilience and general factors that lead some groups to cope with shocks while others collapse or fall deeper into vulnerability. The interviews addressed the following information about each resilient household: the livelihood/economic activity; sources of income for the household; perceptions of why the household coped better than most; and types of interventions that helped the household and would help other households to build their resilience. 21 Quantitative methods Household survey sampling We collected household survey data in 100 village pairs (200 villages), with 21 households interviewed in each village. To sample the 100 village pairs from the up to 240 potential pairs, the team used stratified random sampling. The strata variables included: (1) whether the Traditional Authority (TA) where the village is located is served by UBALE/Njira; (2) the district; and (3) whether the village was an original project location or a location that was added in the second and third year of the program. Additionally, half of the sampled endline villages were matched to comparison villages within the same while the other half were matched to a comparison village in another TA with a similar percentage of stunted under-5 children. In order to identify the comparison pool, we selected villages from the geonames10 location list, since it was the most comprehensive location list with coordinates available. Unfortunately, some of the locations selected were geographic points instead of populated villages. In these instances, the survey team found the closest village within the same TA that did not have WALA activity to survey. The survey firm was unable to survey some of the selected villages due to flooded roads, inaccessible terrain, and other factors. These villages were replaced by the ERIE team with another location either within the same TA or in a TA in the same district with the closest stunting rate. Within each sampled village, we constructed household samples via a combination of revisits to the 2013 endline survey sample (which included 12 households per village) and an additional 9 randomly sampled households. This sampling was be stratified such that at least 80% of sampled households would have children aged 0-9 living in them and 20% of the sample would have no children under 9. Our sample thus also had a reasonable share of households with older children who were under 5 during WALA and for whom we can observe longer-term impacts as well. This sampling design allowed us to estimate separate effects for households with older children who previously benefitted from WALA, as well as households with only younger children who may nonetheless benefit from ongoing activities or other spillovers. Enumerators selected respondents using the random walk sampling methodology. This random walk methodology was used to select both households with children under 9 and households with no children under 9 years old. Each enumerator was assigned a specific random walk pattern to obtain the correct percentage of each type of household. In a small number of villages, after consultation with the survey firm (who has significant experience working in Malawi villages), it was determined that for villages with less than 75 people, enumerators utilized a snowball sampling procedure in order to find households with no children under 9 since this type of household proved to be challenging to locate in each village.11 For specific details on the random walk methodology and snowball sampling procedures, see “WALA2018_RandomWalkProcedures.pdf”. Within each sampled household, we interviewed the head of household as well as the mother(s) of all children under 5 years old, following similar protocols to those in the endline survey. In addition to the 10 Geonames. URL: http://geonames.nga.mil/gns/html/namefiles.html (visited September 2018). 11 In villages with fewer than 75 residents, our sampling rate was very high, and the team was likely to sample a large share of households with no children under 9 years old. It was necessary to utilize snowball sampling to locate this type of household since a large share of them would be surveyed. This only took place in a very small subset of villages. 22 household surveys, in each village, all children aged 0-5 were weighed and measured by a trained anthropometric team and a child questionnaire was administered to each child’s mother. Topics The survey focused on child nutrition and anthropometrics, as well as household demographics, livelihood sources, agricultural production, resiliency/coping mechanisms, and participation in WALA and other development programs. For the complete surveys see the “WALA2018_Followup_HHSurvey _Final.pdf” in the annex. Anthropometrics were captured by collecting weight and length (or height) of under-5 children in sampled households, coupled with age, sex, and recent nutritional practices for these children. The overall health and educational status of children who are 5-9 years old (and would have been under 5 during the 2013 endline survey) were also captured. The survey also included questions regarding household shock exposure and coping mechanisms for recent shocks and adverse events. For the complete child survey, see “WALA2018_Followup_ChildSurvey_Final.pdf” in the annex. We were thus able to define stunting for each under-5 child based on whether the child’s height- for-age is two standard deviations below the mean (i.e., whether height-for-age z-score < -2). Similarly, we were able to define each child’s underweight status based on their weight-for-age z- score. Potential threats to the evaluation As described in the household survey sampling section above, the comparison group was initially selected based on comparability in predicted baseline stunting rates. Despite the successful implementation of the spatial interpolation model, it is nonetheless possible that residual prediction error could potentially have resulted in a comparison group that was different from the treatment group in ways that were correlated with long-term outcomes. If this were the case, causal attribution based on comparisons between these groups would be biased. We therefore use data from our survey, on a variety of demographics that are likely to change only slowly, to confirm the comparability of treatment and comparison villages. These comparisons are laid out below in Table 3 and confirm that, although the team specifically matched villages based on predicted baseline stunting rates, the two groups are in fact quite comparable in a variety of other demographics. Table 3. Demographic comparisons across WALA and comparison group households Control (C) Treatment (T) T-test difference (C-T) Variable N Mean/SE N Mean/SE Head of household: female 2100 0.518 2098 0.540 -0.022 [0.011] [0.011] Household owns land 2100 0.974 2098 0.975 0 [0.003] [0.003] Mean age of head of household 2100 24.091 2098 25.023 -0.932 [0.458] [0.475] 23 Type of shelter: formal home 2100 0.529 2098 0.507 0.021 [0.011] [0.011] Type of shelter: traditional 2100 0.469 2098 0.490 -0.021 [0.011] [0.011] Type of shelter: temp structure/shack 2100 0.001 2098 0.002 0 [0.001] [0.001] Type of shelter: flat 2100 0.000 2098 0.000 0 [0.000] [0.000] Type of shelter: single room 2100 0.000 2098 0.000 0 [0.000] [0.000] Floor: sand 2100 0.021 2098 0.018 0.004 [0.003] [0.003] Floor: mud 2100 0.809 2098 0.826 -0.017 [0.009] [0.008] Floor: cement 2100 0.169 2098 0.155 0.014 [0.008] [0.008] Floor: wood 2100 0.000 2098 0.001 -0.001 [0.000] [0.001] Floor: tile 2100 0.000 2098 0.000 0 [0.000] [0.000] Floor: other 2100 0.000 2098 0.000 0 [0.000] [0.000] Roof of dwelling: none 2100 0.003 2098 0.004 -0.001 [0.001] [0.001] Roof of dwelling: thatch 2100 0.459 2098 0.468 -0.009 [0.011] [0.011] Roof of dwelling: rustic mat 2100 0.000 2098 0.002 -0.001 [0.000] [0.001] Roof of dwelling: wood planks 2100 0.000 2098 0.000 0 [0.000] [0.000] Roof of dwelling: iron sheets 2100 0.536 2098 0.525 0.01 [0.011] [0.011] Roof of dwelling: cement fiber 2100 0.001 2098 0.000 0.001 [0.001] [0.000] Roof of dwelling: ceramic tiles 2100 0.001 2098 0.000 0.001 [0.001] [0.000] Roof of dwelling: other 2100 0.000 2098 0.000 0 [0.000] [0.000] Read and write in Chichewa 2906 0.317 3082 0.341 -0.023* [0.009] [0.009] Read and write in English 2906 0.088 3082 0.087 0.001 [0.005] [0.005] Age when they started school 2528 5.649 2772 5.762 -0.113 [0.046] [0.085] Attended school last year 2530 0.821 2776 0.835 -0.014 [0.008] [0.007] The value displayed for t-tests are the differences in the means across the groups. ***, **, and * indicate significance at the 1, 5, and 10 percent critical level. 24 Another potential threat to the evaluation includes potential contamination in the comparison group from other development activities. We use an intent-to-treat approach to estimate treatment effects, accounting for potential non-compliance by measuring treatment status in both groups and adjusting for these less-than-perfect treatment differences between groups. If other development activities specifically targeted comparison group villages (at higher rates), this would attenuate any differences in true treatment status across the groups and thus our ability to causally attribute results to these treatments. We know that UBALE and Njira were both active in a subset of the districts in which WALA took place, and that other development activities were also quite prevalent during this timeframe and in these geographies. To address this potential threat to the evaluation, we carefully measured exposure to various development activities over time. The household survey asked numerous questions about household’s participation in 44 different development activities. The questions include asking about their participation in these development activities during the last 10 years and their participation in them today. After analyzing the household’s participation in any development activity over past 10 years, we find that households in WALA villages are 7.4% more likely to report having participated in at least one activity than those in the comparison group. This is likely to be an underestimate of the true treatment intensity during the WALA implementation period, as respondents are likely to underreport participation. Table 4 below describes the prevalence of different activity types. Moreover, after adjusting for this potential contamination in our regression analysis (as well as participation in UBALE and Njira), the primary results (discussed below) remain robust. Table 4. Frequency of main development activities Past 10 years Still participates Comparison group mean Treatment Effect Comparison group mean Treatment Effect Health 41.6% 5.7%*** 22.3% -0.8% Agriculture 48.0% 6.9%** 27.4% -0.7% Livestock 25.7% 4.7%* 12.7% 4.1%** Food 10.3% 3.2%** 4.2% 1.5%* Environment 18.4% 4.7%*** 9.4% 0.2% Village Savings & Loans 5.6% 0.3% 2.9% -0.6% Village Committees 11.5% 2.2%* 5.1% 0.8% All types of development activities 70.4% 7.4%*** 45.0% 0.2% The value displayed for t-tests are the differences in the means across the groups. These results control for district fixed effects, UBALE/Njira participation, and baseline predicted stunting controls. ***, **, and * indicate significance at the 1, 5, and 10 percent confidence level. 25 Results Research Question 1: To what extent have WALA interventions and related activities continued to take place after the program’s completion? Key findings: • Even in the particularly successful villages selected for the qualitative study, nine out of ten agriculture and environmental management activities were either discontinued or had mixed results for sustainability. • In the villages selected for the qualitative study, six out of eight health and nutrition activities were discontinued or had mixed results for sustainability after WALA ended. • In the wider, quantitative sample, there are no differences in participation in agricultural, livestock, environmental, health or food-related activities between WALA and comparison villages. • The shocks experienced by households after WALA ended were large and persistent, including pest infestations and droughts. These shocks caused many of the activities WALA promoted to be discontinued and/or diminished their potential positive impacts enormously. • Other factors hindering sustainability were a lack of sustained resources, a lack of continued technical and managerial capacity building, and a lack of vertical linkages to governmental organizations and other entities. WALA implemented a number of interventions in the areas of agriculture, environment and natural resource management, health, nutrition, and disaster risk reduction. To understand the program’s long￾term influence, we assessed the continuation, scaling or cessation of WALA interventions and related activities, as well as the presence and activities of community-based organizations. According to Rogers and Coates (2016), three factors that are critical to the sustainability of development project activities, outcomes, and impacts are “(1) a sustained source of resources; (2) sustained technical and managerial capacity; and (3) sustained motivation (of beneficiaries and service providers).” They go on to note that linkages to governmental organizations and/or other entities are a fourth factor central to sustainability in many circumstances. In the context of southern Malawi, we also consider the risk to the sustainability of activities from stresses and shocks experienced by households and communities, and therefore look at the dramatic effects on sustainability from shocks experienced in the four years after WALA ended. (See the Results section for Research Question 2 below for a description of the main shocks and stresses experienced in Southern Malawi between 2014 and 2018.) In this section, we identify the WALA activities that were sustained and those that were not four years after the end of the project in the 14 villages sampled for the qualitative study as well as in the larger quantitative sample. We look at the findings from the smaller set of villages in the Machinga, Chikwawa, 26 Zomba, and Chiradzulu districts that were purposively selected as the most successful to learn about the factors that can support the sustainability of these activities. We also examine the barriers to success, even in the best-case scenarios, to shed light on the optimal sustainability outcomes we can expect for these interventions in this type of context. To identify these factors and how they worked, we analyzed our qualitative data and then used a sustainability framework to classify sustainability facilitators and barriers according to whether they were due to resources, capacities, motivations, or linkages, based the Rogers and Coates (2016) factors for sustainability. We also look at the effect of shocks on sustainability. We then look at the results of the quantitative survey in the larger set of villages that includes the full range of contexts in which WALA interventions were implemented. Their success is naturally attenuated when not looking at the hand￾picked best-case scenarios. We start with sustainability for agriculture, agribusiness, environment and natural resource management, and disaster risk reduction interventions, followed by a section on the sustainability of health and nutrition interventions. Sustainability of agriculture, agribusiness, environment and natural resource management, and disaster risk reduction interventions In our first areas of focus—agriculture, agribusiness, environment and natural resource management, and disaster risk reduction—project documents and qualitative interviews (with implementers, government officials, and WALA participants in the 14 villages sampled for qualitative research) show that ten main activity types were implemented during WALA in these villages (Table 5). This is a substantial subset of the full list of WALA activities implemented overall. 27 Table 5. Sustainability of WALA activities in agriculture, agribusiness, environment and natural resource management, and disaster risk reduction in the villages sampled for qualitative research as most successful and engaged in WALA Note: Other activities implemented by WALA (but not identified as implemented in the 14 villages sampled for the qualitative study) included promotion of community based organizations (CBOs), faith- based organizations, or other groups; support to change the legal status of CBOs; weather-related early warning and response systems; and disaster risk management plans at the village level. In summary, participants in the 14 villages in which we collected qualitative data, as well as implementers and government officials working in these districts, reported that WALA introduced a number of new ideas and activities. Participants received agriculture training in drought-resistant crops like sorghum and sweet potatoes, and how to plant early maturing varieties of crops such as maize. New seeds and agricultural methods were also introduced, such as the Sasakawa farming method.12 Respondents reported WALA promoted conservation agriculture (CA) practices including new soil and water conservation techniques such as pit planting, contour trenches, box ridges, and manure making. They also taught communities to use watershed management, such as planting trees along riverbanks, to reduce the impact of extreme storms and drought, and promoted watershed development groups. Interviewees reported that WALA introduced irrigation farming using underground PVC pipes, water 12 The Sasakawa farming method uses closer ridge spacing (75 cm apart) and single seeds (25 cm apart), instead of the traditional method of planting multiple seeds 50 cm apart along ridges. Activities Discontinued, sustained, or mixed Agriculture training (new seeds/methods, intercropping, management of insects/disease, post-harvest management, manure making, producer groups) Mixed Conservation agriculture practices (soil management practices that minimize erosion or maintain soil cover, including crop rotation, pit planting, contour ridges or trenches, marker ridges, stone bunds, water absorption or check dams, and Vetiver grass) Mixed Irrigation Discontinued Agribusiness (group marketing, identifying new contacts for marketing, connections between communities and external organizations) Discontinued Natural resource management (tree planting, fertilizer trees, forestry groups) Mixed Food/assets for work Mixed Infrastructure (dams, fuel efficient cookstoves) Mixed Savings and lending groups or activities (VSLs) Sustained Private Service Providers, Community Animal Health Workers (Paravets), Agriculture Service Providers Mixed Village Civil Protection Committees (VCPCs) Discontinued 28 pumps, and portable treadle pumps and worked with water users’ associations. WALA helped form Agribusiness Groups which grew and sold chilies, pigeon peas, and others crops to national and international markets through linkages supported by WALA. WALA also introduced natural resource management activities by promoting tree planting, fertilizer trees (nitrogen-fixing trees and shrubs such as Faidherbia albida (Msangu), Cajanus cajan (pigeon peas) and other leguminous crops), and the creation of forestry groups. In this group of focus areas, WALA also introduced village savings and loans (VSL) groups to facilitate savings, lending, and investment. WALA also promoted food/assets for work activities by offering paid work opportunities such as constructing dams, planting trees, and building irrigation infrastructure. Participants received bags of maize and other food in exchange for working on these projects. Participants also reported receiving training to make energy-saving cookstoves from WALA. Private Service Providers, including VSL promoters and para-veterinarians (paravets), were linked to communities by WALA in order to support new activities. WALA connected communities to government extension workers to treat livestock. Finally, to improve communities’ preparedness for natural disasters, WALA supported Village Civil Protection Committees (VCPCs) and provided training and support so they could act quickly in case of natural disasters. Almost all of these activities were embraced and adopted to at least some extent during WALA. In the qualitative data collection, participants reported both positive and negative experiences with all activities, but overall participants reported that WALA introduced useful techniques, which were adopted by at least some participants in all villages sampled. Community members in WALA-promoted activities reported they participated because they saw immediate benefits or the potential for benefits. However, since the end of WALA in 2014, the extent to which WALA interventions and related activities continued to take place diminished. In Table 5, we identify the agriculture, agribusiness, environment and natural resource management, and disaster risk reduction activities that were sustained and those that were not. 1.Discontinued Of the ten types of agriculture, agribusiness, environment and natural resource management, and disaster risk reduction activities that WALA implemented in the 14 villages in which we collected qualitative data, three types of activities were largely discontinued after the completion of the project. These were irrigation systems, agribusiness activities, and Village Civil Protection Committees (VCPCs). Farmers who were in irrigation groups reported that few farmers still participate in these groups. They reported this was mainly due their inability to maintain or replace equipment due to the lack of resources and trained technicians, lack of water to irrigate during droughts, and low prices for crops grown under irrigation schemes, which demotivated participants to continue investing in equipment. Agribusiness groups and activities were also discontinued in most of these communities; only one bee keeping group continues to exist, and one agribusiness network. Low prices for cash crops and linkages that fell through with national and international markets were the main reasons. Market linkages created during WALA were not sustained or promised prices did not materialize, and participants did not have the capacity to create new linkages outside their local areas. The activities of VCPCs have likewise largely been discontinued. Only two VCPCs were reported to still exist after WALA ended. In one community, the chairperson of the VCPC during WALA reported he still files reports with the government when there is a disaster, and there were reports that the 29 government may provide some assistance in response. However, most VCPC activities supported by WALA were not sustained. 2.Mixed results Six of the ten activity types had slightly better results with a mix of sustained and discontinued practices. These included agriculture practices, conservation agriculture practices, natural resource management, the outcomes and impacts of food/assets for work activities, infrastructure, and private service providers. For the agriculture practices that were promoted, many were shown to be successful in increasing farmers’ yields, and were therefore appreciated, adopted and continued. One example is early maturing varieties of crops such as rice and maize, which provide viable options in an environment of changing climate. One farmer in Machinga echoed the words of many in reporting, Now we are seeing a change. People were clinging to old rice seeds that take long to mature, but since this group [WALA] came all those seeds have disappeared such that our children do not know them. We are just planting early maturing varieties. Whether there is little or enough water, we still eat. (FG7_F1) Other practices have been continued by some farmers, while discontinued by others. An example is Sasakawa farming, which has produced higher yields for farmers and has continued in some communities. However, it was dropped by many farmers after the fall armyworm attack, since Sasakawa required additional fertilizer, which was hard to afford. Manure fertilizer making is another example. Some farmers are continuing to use manure fertilizer because they have found it useful. However, shocks have led to many farmers selling livestock as a coping mechanism. The lack of manure from livestock together with a lack of money to buy fertilizer (which was mixed with manure and then used on crops) has contributed to most farmers discontinuing manure making and groups disbanding. Finally, many new agriculture practices have been discontinued almost completely. Management of insects and pests has been unsuccessful for the most part because farmers do not have enough money to buy the necessary chemicals. As one farmer noted, “there is no money to use to buy the chemicals to spray so when the insects attack there is nothing to do.” (FG4_F1) New crops such as soybeans were introduced, but a lack of market linkages made the crops impossible to sell. Sesame was also introduced but was affected by worms and drought. Looking at our quantitative data on sustainability of agricultural practices, we see in Table 6 below that there was very little statistical difference after WALA between the wider group of WALA and comparison villages in the use of improved seeds, intercropping, and beekeeping, three of the agriculture activities WALA promoted. Only intercropping is more widely used in the wider group of WALA villages than the comparison village. This suggests that the few agriculture practices continued by farmers in the qualitative sample villages—those with the most enthusiasm and best implementation— show a little more sustainability that the larger group, but that the mixed sustainability findings are consistent. 30 Table 6. Statistical comparisons for agricultural practices between WALA and comparison villages Agriculture practices Comparison group mean Treatment effect Improved seeds 53.6% 2.8% Intercropping 52.7% 4.4%* Beekeeping 1.3% 0.3% These results control for district fixed effects, UBALE/Njira participation, and baseline predicted stunting controls. ***, **, and * indicate significance at the 1, 5, and 10 percent confidence level. For the conservation agriculture (CA) practices that were promoted, such as crop rotation, zero tillage, and check dams, most were adopted during WALA and perceived to contribute to higher yields, reducing soil erosion, conserving soil moisture, and protecting crops from being washed away by storms. Many individuals in the 14 communities included in the qualitative study continued to practice CA techniques after WALA, such as contour ridges, contour bunds, mulching, and pit planting. In one community, the contour bunds created during WALA were still there even after heavy rains and continue to be a resource in farming. A group of community members in another village is reportedly training others in watershed management techniques learned during WALA. In yet another village, community members are training each other in contour ridges and estimate that virtually every plot of land has them. As one farmer there said, Farming is going on well because some of the things that we were trained in—like reducing distance between the ridges—are still being done even though WALA left. There were some people who had not yet reduced the distance between the ridges, but through WALA’s training and other people training them, most of us now have reduced the distance between the ridges. (FG8_M6) However, the qualitative data suggest that more than half of all participants have discontinued the conservation agriculture practices, and almost all the groups working on these activities have disbanded. Part of the reason may be that farmers find some of these practices “too much work.” Participants explained that check dams, for example, were difficult to build and that without WALA’s support they were only continuing to maintain them rather than building new ones. One woman explained, “we stopped constructing the check dams because when…[WALA] came to introduce [check dams]…we were working in the rivers, piling up stones maybe even above my height, so after completion of the task we stopped there.” (FG13_F1) Environmental context also matters. For example, WALA promoted Vetiver grass that was helpful in reducing soil erosion due to rain, but which did nothing during extended drought. One woman explained, “when it comes to floods, we were taught to plant Vetiver. The Vetiver acts as a band and prevents water run-off. But when it comes to drought it is difficult to use the capacities because rains didn’t fall.” (FG9_F2) Table 7 shows our quantitative data on the sustainability of six conservation agricultural practices in the wider quantitative sample of WALA villages. For six of the practices—contour ridges, contour trenches, marker ridges, stone bunds, water absorption, and check dams—there was no statistical difference in 31 the use of these practices between WALA villages and comparison villages, after WALA had left. For box ridges and Vetiver grass use in watershed management, we find lower use of these practices in WALA villages than comparison villages. This suggests that the few practices that were continued under the best-case scenarios did not translate into impacts in the wider group of villages. Table 7. Statistical comparisons for conservation agricultural practices between WALA and comparison villages Conservation agricultural practice Comparison group mean Treatment effect Contour ridges 93.9% -0.5% Contour trenches 27.5% 1.6% Marker ridges 59.8% -0.2% Stone bunds (contour bunds) 18.7% -1.8% Water absorption 14.5% -1.0% Check dams 8.0% 1.3% Box ridges 58.1% -7.8%** Vetiver grass 19.0% -4.4%* These results control for district fixed effects, UBALE/Njira participation, and baseline predicted stunting controls. ***, **, and * indicate significance at the 1, 5, and 10 percent confidence level. The sustainability of natural resource management activities was also mixed in our qualitative sample. Promoted practices included tree planting, cultivation of fertilizer trees, and capacity building of government-created forestry groups. After WALA, in at least two villages, forestry groups are continuing to plant trees with the support of the government forestry official. One community member noted, Right now…we are planting reeds in the river through the Forestry department but continuing the skills that we were given by our parent [WALA], so these activities are ongoing with the support from other organizations and departments. (FG6_M2) In another two villages, group members continue to sow seedlings in nurseries for community members to plant trees on their own property. And in a fifth village, some community members are continuing to plant fertilizer trees, including at least one group who pools money to purchase trees. In addition, some individuals are continuing to plant trees on their land as they are able, including households involved in afforestation who remain motivated to continue planting trees. As one farmer noted, “We are still sowing seeds and planting trees during the rainy season. In that way we are conserving natural resources in our community.” (FG4_M1) 32 On the other hand, not all natural resource management activities continued after the completion of WALA. Some communities planted fertilizer trees during WALA, but reported that they were killed by termites and not replanted. Other community members were reportedly paid with food to plant trees during WALA; however, likely as expected, after the food for work program ended, most of those participants stopped planting trees. Looking at our quantitative data in Table 8, we see that after WALA ended there was no statistical difference between the wider sample of WALA and comparison villages in the practice of planting leguminous or nitrogen-fixing fertilizer trees, our only quantitative natural resource management indicator. Again, the mixed results in the villages with the best chance of succeeding do not seem to have translated to the wider group of villages. Table 8. Statistical comparisons for natural resource management practices between WALA and comparison villages Natural resource management practice Comparison group mean Treatment effect Fertilizer trees 12.9% 0.9% Composite index of any agricultural practice promoted under WALA+ 93.8% -1.1% + This index reflects whether a household used any of the agricultural practices promoted by WALA. These results control for district fixed effects, UBALE/Njira participation, and baseline predicted stunting controls. ***, **, and * indicate significance at the 1, 5, and 10 percent confidence level. There were also mixed results for sustainability of the outcomes and impacts of food/assets for work activities. Activities that were included in these schemes included planting trees, building infrastructure such as irrigation systems and dams, and building energy-efficient stoves. In return, respondents reported that WALA paid community members with bags of maize or other food. Several participants in focus groups mentioned learning how to make stoves from WALA, but reports were mixed on whether the stoves are still being used; some users remained committed because the stoves saved money on fuel, while others returned to traditional cooking methods. One focus group participant who continued to use the fuel-efficient stoves explained: “In the past before WALA came, we were using fireplace stones to support pots; they told us that the fireplace stones use a lot of firewood. WALA was promoting the use of a stove which was using less firewood so from that time up to now, most of us use less firewood.” (FG12_F1) Other outcomes and impacts of food/assets for work activities also had minimal sustainability. As mentioned, once participants no longer received food for the forestry work undertaken during WALA, they stopped planting trees (although the status of the trees that were planted is unclear). Only some of the infrastructure dams that WALA paid community members to build were fully completed; the rest were left partially completed during WALA. The completed dams have had mixed results. During drought completed dams are left without sufficient water, and therefore respondents reported they do not function as intended. However, in some cases these dams have been converted for use in 33 aquaculture. When there is sufficient water, this provides community members with immediate income, and thus those dams continue to be used for this purpose. Finally, WALA interventions promoting private service providers (PSPs) also had mixed results when it comes to sustainability. In the 14 villages chosen for the qualitative study, some VSL promoters and village bank PSPs have continued with this work, and are paid by community members, especially where VSL groups continue to be active. A few paravets trained by WALA also continue working as PSPs. One paravet in a village in Machinga noted that since WALA trained him, farmers in his community continue to request his services even though WALA has ended. Up to now, those farmers who know that I can assist with livestock issues report to me. When they have done that, I link them with government trained workers who come to administer injections and vaccines to their livestock. That is happening now, but in the past, we didn’t know what to do. In the past when chickens have been attacked by Newcastle disease, we were just accepting the situation, we didn’t know that it could be prevented. (FG7_M1) However, most PSPs, paravets, and agriculture service providers have discontinued service. In some cases, lead farmers have continued working as volunteers, but this appears to be rare as well. 3.Sustained Only one of the ten WALA activity types was sustained in the 14 villages sampled for the qualitative study after the WALA project ended—VSLs. Although not successful in every village, VSL groups by and large remained a source of savings and loans after WALA left, and many villages have increased the numbers of VSL groups. In two of the 14 villages we sampled, VSL groups had been discontinued. Reasons given included inactive community agents trained by WALA and the resultant lack of support and monitoring, and lack of continued capacity building. Some members also dropped out due to the inability to pay loans or a lack of money to contribute or save, which has been linked to shocks. Others have left the groups because they did not want to pay a fee to a PSP, or because they feared assets being seized. However, in general, VSL members remained enthusiastic due to the benefits they experience from the savings and loans. VSLs have given them access to support education, home improvements, business capital, and agricultural inputs. Linkages between communities have helped provide peer training on VSL groups, and VSL promoters who were trained to support VSLs have remained available in some areas and have provided refresher trainings on technical and managerial capacities, and motivation for organizers and members. Although they lack linkages to formal financial institutions that could augment the resources they have to work with, VSL groups as they are have been sustained and, in some places, expanded in number in the villages best placed to succeed. The quantitative data, on the other hand, shows that there is no significant difference between VSL participation in the past 10 years and VSL participation currently in WALA villages and comparison villages. 5.6% of comparison households participated in VSL groups at some point over the last 10 years compared to 6% in WALA villages. Currently, 2.9% of comparison households participate in VSL groups and 2.3% of WALA households currently participate. Thus, in the wider group of WALA villages there are no increases in participation in VSL groups in treatment villages due to WALA and no overall difference compared to comparison villages. 34 Factors facilitating and hindering sustainability of agricultural, natural resources management, and risk reduction activities We now take a systematic look at the supportive factors and barriers to sustainability different activities faced. Table 9 summarizes our findings using a sustainability framework to classify facilitators and barriers according to whether they were resources, capacities, motivations, or linkages. This analysis is based on qualitative data in the 14 villages chosen as examples of the best- case scenarios for WALA implementation and community engagement. As such, one can surmise that many of the facilitators would not be as strong in the larger sample of all WALA villages, and some of the barriers even more difficult to surmount. This analysis gives us an idea of what factors could affect sustainability of these types of activities if implemented again. Many of these factors would have been more difficult to detect, if not looking for positive deviance in the qualitative study. 35 Table 9. Factors facilitating and hindering the sustainability of WALA activities in agriculture, agribusiness, environment and natural resource management, and disaster risk reduction Evaluation Framework Which factors facilitated (✓) or hindered (✗) the sustainability of the following program activities? Factors Activities Resources Capacities Motivations Linkages Agriculture training Mixed Mixed Mixed Mixed Conservation agriculture practices Mixed Mixed Mixed Irrigation ✗ ✗ ✗ Agribusiness ✗ ✗ ✗ Mixed Natural resource management Mixed ✓ ✓ ✓ Food/assets for work ✓ ✓ Mixed Infrastructure ✗ ✗ ✓ ✓ Savings and lending groups or activities (VSLs) ✓ Mixed Mixed Mixed Private Service Providers, Community Animal Health Workers (Paravets), Agriculture Service Providers ✗ ✓ Mixed Mixed Village Civil Protection Committee Mixed ✓ Mixed Note: “✗” indicates a barrier to sustainability, “✓” indicates a facilitator of sustainability, and “Mixed” indicates the presence of both. 36 1.Sustained source of resources The first factor that affects sustainability is a sustained source of resources. This factor facilitated the sustainability of two types of activities among the ten—VSLs and food/assets for work. VSLs remained a source of savings, loans, and investments for community members. In addition, through self- financing, VSLs were able to provide themselves with a sustained source of money that supported their sustainability. VSL groups were reported to be extremely useful by participants, and they are the only group created or promoted by WALA that continues to exist in most of the 14 qualitatively sampled villages. In addition, a minority of participants interviewed reported that dams funded through food/assets for work during WALA were still there after the end of WALA and continue to serve as income-creating resources. However, more often than not, not being able to access sustained sources of resources post-WALA was a barrier to the sustainability of WALA-promoted activities both in the continuation and expansion of activities. There were at least four main ways that resource constraints affected sustainability. 1. First, participants did not have the resources to fix or replace materials needed for activities as they were damaged post-WALA. One example was irrigation systems WALA constructed. Participants unequivocally stated that irrigation helped them to harvest and survive during food scarcity and drought. However, these systems included underground PVC pipes that cracked, leaked or broke over time, and community members were unable to afford to fix them as they were damaged, resulting in the sub-optimal use or no use at all of irrigation systems. 2. Secondly, promoters and others who provided work for the activities no longer received compensation to continue their activities. Promoters, for example, were paid during WALA to do things like train VSL group leaders and find markets for agribusiness products but were no longer paid to do those types of tasks once WALA ended. Therefore, their efforts were not sustained. Communities were also paid for activities, such as planting trees. Once WALA ended, community members no longer felt incentivized or motivated to continue these activities. 3. Thirdly, communities could no longer access inputs for activities that required consistent inputs or continual renewal. One example of this was the need to purchase fertilizer to mix with manure for one of the conservation agriculture (CA) activities. Even if the resulting fertilizer is less expensive, money was still needed to purchase the chemical fertilizer. Another example was high transportation costs to move agriculture products to markets. Without sustained resources, transportation costs prevented farmers from selling in some markets. Even better seed varieties saw limited sustainability, since they needed to be purchased each year. Activities that were not completed by the end of WALA were abandoned, such as irrigation, where communities did not receive all the required equipment, and infrastructure such as dams that were not completed. Once WALA ended, those activities did as well. 4. Fourthly, competition now existed for the limited natural resources made available through the project. In the case of irrigation, water resources were not sufficient for all farmers attached to the irrigation systems. This led to conflicts between some community members, which was an unexpected negative consequence of this activity. The systems require users to work together to maintain and manage them, so that all users benefit. However, the systems pitted farmers against each other for valuable access to water, limiting their motivation to work together. 37 2.Sustained technical and managerial capacity The second factor that affects the sustainability of development projects is sustained technical and managerial capacity so that participants and service providers can operate independently of the project. This factor had mixed effects on the sustainability of WALA activities. First, it facilitated sustainability in two main ways. 1. WALA trained several types of service providers, so they were able to continue to provide services once the project was over. The first type were service providers to support VSL programs. This provided technical and managerial capacity among service providers who were then available to support VSLs in some of the intervention areas. A second type were paravets who were trained to support farmers in keeping livestock healthy and in making linkages to government extension workers and local veterinarians. Another type was agriculture service providers. Communities were now no longer receiving these technical inputs and either did not have the resources or the motivation to pay for these technical inputs on their own. 2. The second way WALA facilitated sustainability through technical and managerial capacity was by training participants in the technical aspects of the activities they promoted. For example, they provided training for farmers in manure making, the Sasakawa farming method, ridge spacing, watershed techniques, and the like. Training of community members and the creation of links between communities also helped provide peer training for villages where WALA could not work, for example helping to spread VSLs. Interview respondents also noted that communities were trained in what to do during disasters, for example if a storm blew roofs off houses. Training provided by WALA has enabled some households to continue natural resource management on their own, even after WALA ended: “The seedlings that I [have] now… I just found them [from a trading center] so I wanted to continue what WALA trained me in; now I have a tree nursery” [RH_F12]. However, the lack of sustained technical and managerial capacity strengthening was also a hindrance to sustainability in a number of ways. First, not having follow-up training for community participants meant that community members who misunderstood their initial training could not be corrected; second, community members who faced changing conditions, such as shifting weather patterns, could not adapt; and third, community members who were not trained in some aspects of the activities they tried to sustain did not know how to deal with them. This hindrance manifested itself among lead farmers who, according to one agricultural extension and development coordinator, misunderstood a training in agriculture practices, and thus their “measurements and dimensions, even depth and mixing of soil and manure were all wrong.” (GO_M5) Among VSL groups, when shocks occurred and members could not repay loans, some groups seized assets on a large enough scale that other members became frightened of the groups, and the VSLs lost membership. With more support and continual learning, other options might have been put into place to maintain the integrity of the rules governing the groups without rigidly following them to the detriment of the institutions themselves. Gaps in technical and managerial training were also felt in irrigation activities, where there were no trained technicians to maintain pipes and treadle pumps, thereby inhibiting their sustainability. Lack of training on how to secure markets also had detrimental effects on the sustainability of agribusiness activities, and lack of continued training support left VCPCs stagnant. 38 3.Sustained motivation of beneficiaries and service providers that does not rely on project inputs The third factor that affects the sustainability of development projects is sustained motivation on the part of project participants and service providers to continue engaging in the activities after the project ends. In the 14 villages included in the qualitative study, we saw that incentives from the outcomes and impacts of the activities provided motivation for participants to sustain their engagement with the activities. Many of the agriculture activities, such as making manure fertilizer, mulching, and reducing space between contour ridges, were perceived as contributing to higher yields and reducing soil erosion. Water harvesting through check dams and ditches was reported as ensuring that crops were not washed away, as they have been in the past, and hence improving crop yields. Tree planting was also seen to reduce the impacts of storms, prevent water from running in fields, and protect the soil, and thus households reported they remained motivated to continue planting trees on their own. Monetary benefits also motivated sustainability of activities, such as the continued use of dams built by WALA, because they provided immediate income through fisheries. Some users of energy saving stoves remained committed because the stoves saved money on fuel. VSL members largely remained enthusiastic because they benefited from the savings and loans. Motivation to continue participating in VSLs was also provided by VSL promoters, who received payment from communities in order to facilitate the groups, which was their own motivation. This factor—benefits providing motivation— seems to have positively affected sustainability more than any other. The data show that households tended to continue WALA-promoted initiatives out of their own interest. While in some cases benefits kept participants and service providers incentivized to continue activities promoted by WALA, in others the lack of benefits had the opposite effect. For example, agriculture techniques that did not yield impacts within one crop cycle were reported to be discontinued. Farmers involved in agribusiness activities said they lacked motivation to continue some practices when the prices they received for their products—such as chilies and pigeon peas— were much lower than WALA had promised. Farmers involved in irrigation schemes also reported being demotivated to continue investing in equipment when they experienced low prices for their irrigated crops. Lack of benefits for agribusiness participants who had opened formal financial accounts with banks was also demotivating, and farmers reported closing accounts because banks did not make agriculture loans and the cost of transportation to banks was high. In addition, we saw demotivation of lead farmers, service providers, and promoters who did not continue the work they were trained for under WALA when they were not paid for that work. The lack of monitoring and support from any organization and inactive community agents trained by WALA after 2014 was blamed by participants for some VSL groups lacking direction and eventually being dissolved. Motivations can also pull both ways. Some participants noted that pit planting was helpful in conserving the moisture in the land, but other participants in the activity found the activity too much work. In such cases, each farmer had to weigh the benefits and the costs. 39 4.Linkages to governmental organizations and/or other entities A fourth factor that is essential to consider when looking at sustainability of development projects is linkages, including vertical linkages, such as from a community agribusiness group to district or regional markets, and/or horizontal linkages, such as among local groups or villages. Appropriate linkages are important for the sustainability of most technical sector interventions. Most of the linkages reported in the qualitative data were vertical. Some of these linkages were direct with governmental units, while others were established by private service providers (PSPs) or promoters. For example, some members of farmers groups reported that they remained directly linked to government extension workers in the forestry department and agriculture department. Farmers in Machinga explained WALA’s role in this sustained contact: WALA was able to get in touch with agriculture officials and they briefed them on their programs and linked them [the government officials] with [us]. Without WALA establishing that link between us and the government extension workers…we could not have been in touch with them because they could have just been waiting on us to approach them whenever we had a problem. (FG6_M1) A few members of Village Civil Protection Committees (VCPCs) also reported some sustained connections between their communities and the government. Community members using water collected behind dams for fishing also reported being connected to fisheries in Domasi, which could be private or could involve the national government. On the other hand, some linkages were sustained through promoters or PSPs. For example, some paravets provided linkages between community members and government workers and private labs. Some VSLs were reported to be linked to VSL promoters, who can refresh technical and managerial capacities and motivate organizers and members. Linkages with follow-up projects also helped sustain activities. In some communities where UBALE is being implemented in Chikwawa, participants reported that their learnings from WALA were smoothing the way for similar work with UBALE. These reported linkages are important, but they did not seem to be as widespread or as strong as necessary. In addition, other linkages were missing. For example, some farmers reported they did not have linkages to lead farmers and community agents, who stopped working to support farmers once WALA ended. Participants in agribusiness reported they received some seeds that were introduced specifically to sell at market, like soybeans, but that the market linkages never materialized or the prices offered were much lower than promised. Others reported they did not have linkages to markets outside their local area. In addition, some respondents reported that during WALA, implementing staff did not involve government extension workers enough to hand over activities. At least one community reported their VCPC did have linkages with the government, but they were not supportive. VSLs also did not have linkages to institutions that could augment their resources. Furthermore, many VSLs were not registered. As informal entities, they have fewer chances of being recognized by institutions and linked. 40 The effects of shocks and stresses on sustainability of agricultural, natural resources management, and risk reduction activities Southern Malawi experienced an extraordinary set of shocks and stresses during the four years after WALA, including pest infestation, droughts, and floods. (See the Results section for Research Question 2 below for a description of the main shocks that occurred during this time.) Shocks and stresses can have dramatic and long-term negative impacts on households and communities, especially in areas where subsistence farmers live. Our survey data shows that between 37% and 42% of WALA households reported that they had not even partially recovered from the most recent shock they had experienced. These events are the threats that likely had the most profound effect on the sustainability of WALA activities. Below, we consider how shocks and stresses experienced by households and communities during the four years after WALA ended affected the sustainability of WALA agricultural and natural resource management activities. Respondents in the 14 villages selected for the qualitative study reported how and why these shocks caused WALA-promoted activities to be discontinued. The most commonly mentioned type of shock—experienced by more than 60% of households in the wider survey population—was a biological shock, including agricultural insects, pests, and diseases, which inflicted damage and destroyed crops in the areas of the intervention. Perhaps the most devastating of these pests was the fall armyworm. As one participant in the 14 selected villages for the qualitative study noted, “last year these pests came… They say that it is a new armyworm [fall armyworm]. They have brought terrible misery.” (RH_M2) Another participant described their destruction: They attacked every field in those areas. The chemicals that the … Agriculture Office was giving people after they reported the problem were not enough for the whole area. Even rice was destroyed by the fall armyworms. The chemicals were not enough for all of us …to save our food. The food [crops] got destroyed. Up to today we are facing hunger. (FG6_M12) The second most commonly mentioned shock—reported by more than half of the survey population— was a series of meteorological shocks, including severe droughts and floods. Droughts, perhaps the most devastating meteorological shock, were reported to have started in earnest after the WALA project ended in 2014 and continued up through the data collection period. As one focus group participant noted, “In 2015, we faced drought, and all these years up to now, we have been facing it.” (FG10_F5) The prolonged and persistent droughts had widespread effects. For example, one government official in the Zomba district noted that in 2017, several hundred households were affected by drought in the surrounding area. “Out of the 2,100 [households], the number of households that were … affected by the drought might be around 500.” (GO_F1) After WALA ended, meteorological shocks such as droughts and floods, along with pests and diseases, reduced the ability of farmers to benefit from most of the agriculture and conservation agriculture practices promoted by WALA in many ways. These shocks lowered crop yields and caused crops to fail. The droughts affected irrigation schemes promoted by WALA, which either did not have any water or did not have enough water for all farmers involved in the systems. Fishing behind WALA-built dams was diminished during drought due to lack of sufficient water as well. In addition, farmers sold livestock in response to the shocks, which reduced the number of farmers who could continue manure fertilizer making. 41 The effects of the weather shocks and pests seem to have overwhelmed the benefits of WALA’s activities. Participants reported that during WALA, yields improved. However, in the face of the shocks experienced after WALA, even when participants continued to follow WALA’s teachings on new agricultural methods, they were unable to counteract the extent of the damage. A focus group participant in Zomba captured a sentiment heard frequently: “We used to have more maize before the drought [and] fall armyworms problems came. We follow all the [WALA] practices but drought and fall armyworms are drawing us back.” (FG1_F2) Another woman in Chikwawa shared some of the effects of the shocks: “There has come kodikodi [mealy bug]. The whole plant dries up. Sesame and cotton farming have both gone down...Last year a lot of people grew sesame, but no one harvested anything.” (FG14_F5) Another serious shock that farmers experienced was market failure. Farmers in agribusiness groups began growing chilies, pigeon peas, and other crops during WALA to sell to national and international markets. However, market prices for these products plummeted to the point that the price did not cover the production costs. For example, farmers grew pigeon peas knowing that previous prices ranged from Mk350 to Mk500 per kilo. However, after WALA left, the price dropped as low as Mk20 per kilo. This shock caused most farmers to discontinue growing these crops for market. As one government official said, “Most farmers have been disappointed this year, and some have said they will not plant pigeon peas [or] chili in this growing season [2018/19].” (GO_F1) Market problems were not limited to selling produce but also obtaining inputs like fertilizers. For example, fertilizer became more difficult to obtain. A focus group participant described the four years after WALA ended this way: “With WALA leaving, coupled with the fact that the fertilizer under the government coupon system only targets two or three people per village, things are difficult for us.” (FG7_M7) Difficulty getting fertilizer negatively affected the sustainability of the manure fertilizer activity and other agriculture interventions promoted by WALA. Due to the many shocks and stresses faced by the communities where WALA was implemented, farmers had less money than they might have otherwise. This made continuing some of the WALA activities difficult as well. The reduced income and savings caused by droughts, pests, and market problems meant farmers could not buy inputs to continue activities or pay private service providers. VSLs were affected as farmers did not have the income they expected to repay loans, and this reduced overall group payouts. In addition, in some areas VSL groups decreased in size because of sustained drought, as farmers did not have additional money to save. Sustainability of nutrition and health interventions WALA introduced a number of new ideas and activities in health and nutrition, according to project documents and qualitative data from implementers, government officials, and WALA participants. Table 10 below contains a list of the eight main health and nutrition activities reported in the 14 villages sampled for the qualitative study. This is a substantial subset of the full list of WALA nutrition and health activities implemented overall. Participants in the sampled villages, as well as implementers and government officials working in the four districts sampled for the qualitative study, reported that WALA taught communities important concepts in health and nutrition, including diet diversity and child nutrition. WALA introduced the idea of kitchen gardens and through community education and the formation of care groups also promoted health-seeking behavior, particularly for pregnant women, as well as messages around nutrition and immunization. Care groups coordinated with the health surveillance assistant to conduct growth monitoring exercises and nutritional assessments for children, conduct sanitation training sessions, and promote breastfeeding and visits to the health center for 42 pregnant and nursing mothers. WALA also provided MCHN food rations for pregnant women, children under 5, and breastfeeding women. WALA introduced new ideas about keeping communities clean and hygiene in general, including the importance of sweeping and digging garbage pits. Communities reported that WALA introduced hand washing stations, called tippy taps, as well as latrines and drop hole covers. Respondents reported that during WALA, these activities were embraced and adopted in the communities sampled. Participants described mainly positive experiences and said they participated in the activities enthusiastically because they saw the benefits or potential benefits. WALA also provided valuable inputs, such as materials for kitchen gardens and cooking demonstrations, which permitted and perhaps encouraged the adoption of these effective new techniques. However, since the end of WALA in 2014, the extent to which WALA health and nutrition interventions and related activities continued to take place diminished. In Table 10, we identify the health and nutrition activities that were sustained in the 14 villages sampled for the qualitative study and those that were not. Table 10. Sustainability of WALA activities in health and nutrition Activities Discontinued, sustained, or mixed Kitchen gardens Mixed Care groups (safe motherhood groups, support groups, and other groups focused on health, sanitation and nutrition; and promoting CBOs) Mixed Health-seeking behavior Mixed Child nutrition (cooking demonstrations, CCFLS) Discontinued MCHN food rations (pregnant, under-5s, breastfeeding women) Discontinued Keeping communities clean (sanitation) Sustained Tippy Taps (hand washing facilities) Mixed Latrines (drop hole covers) Sustained Note: Other activities implemented by WALA, but not identified as implemented in the 14 villages sampled for the qualitative study, included connections created between the communities and external organizations or entities. 1.Discontinued Of the eight types of health and nutrition activities reported in the 14 villages we sampled, two types have mainly been discontinued since the completion of the project. These are child nutrition activities and MCHN food rations. For the first, WALA taught communities about child nutrition through conducting or promoting cooking demonstrations and community complimentary feeding and learning sessions. Community members reported that they found these beneficial in reducing child malnutrition. 43 In particular, members reported that learning how to cook highly nutritious foods involving local staples, such as pumpkin porridge, reduced child hunger. Many communities reported participating in the nutrition demonstrations organized by WALA. A focus group participant explained, “They were teaching people about dietary mix of foods. They were also mostly teaching women how to prepare food to give to a malnourished child.” (FG8_M1) Another focus group participant reported that in her Chikwawa community, There are now fewer deaths of pregnant women and the number of malnourished children has been reduced due to the training in the six food groups. Before WALA came, when they were talking about the six food groups, I thought it meant six different types of foods, one to six not knowing that we had all the groups right in our area. What we didn’t know was the know-how to use them. But up to now, those activities are continuing and women in the villages are able to know how to prepare the six food groups mixing different food types. (FG12_F1) Respondents reported that during WALA there was enough rain in most communities to grow new crops like sweet potatoes to make nutritious porridge for children. In addition, WALA provided food aid. Both of these factors, combined with child nutrition activities, were reported to help reduce hunger and malnutrition. However, after WALA, there was sustained drought, and since WALA ended most community members have been unable to apply nutritional learnings because there was simply not enough food, the dietary diversity was not available, and community members lacked access to recommended foods. A man in Chiradzulu describes the situation after WALA left: The main focus for the [WALA nutrition] groups was teaching them how to prepare porridge made out of pumpkins. So because of the drought that has occurred in these recent years, even pumpkins were not available. They were also trained how to cook sweet potato porridge but that was not even available—the sweet potatoes, due to the drought that I have talked about. People did not grow sweet potatoes. So the group activities have not happened in the past three years because of drought. (FG8_M1) Since WALA ended, activity participants also lack support, encouragement, and resources to participate in or hold cooking demonstrations, and in most villages, it was reported that the nutrition groups have disbanded. Some individuals reported that they continue to use the learnings from these demonstrations in their own households using local ingredients. However, in other villages, even this has not been sustained. In some communities in Chikwawa, UBALE is continuing to train leaders in spreading nutrition messages, and in some communities UBALE or Millennium Promise is continuing to support learning and demonstration groups. However, the activities themselves have not continued independently of WALA, because community members either do not have access to the food or are unable or unwilling to make monetary contributions for the purchase of ingredients for cooking demonstrations. The second type of activity that was discontinued since WALA ended was the distribution of MCHN food rations to pregnant and breastfeeding women and children under 5 years old. In some communities follow-up projects have provided support, but according to one resilient household member, the number of beneficiaries is smaller than it was during WALA. 44 2.Mixed results Four of the eight health and nutrition activities mentioned by WALA participants had mixed results, with some aspects being sustained and the rest discontinued. These included kitchen gardens, care groups, health-seeking behavior activities, and tippy taps. Many communities reported participating in kitchen gardens when promoted by WALA. In some villages, kitchen gardens are reportedly still helpful to individual households. For these gardeners, they remain motivated because the gardens provide easy sources of vegetables for home consumption and help them save money, because they do not need to purchase these vegetables at the market. As one focus group participant in Zomba explained, Because before the coming of WALA we did not know how to make kitchen gardens so that we can grow vegetables like pumpkin leaves, WALA taught us to have kitchen gardens. If you have no money to buy relish, you just go in your garden and get the vegetables. Right now, many homes have kitchen gardens. WALA taught us this. (FG2_F9) However, most community members did not maintain the kitchen gardens long-term. Many kitchen garden fences, used to protect the gardens from animals, were attacked by termites and/or destroyed by storms. In addition, some gardeners had water sources distant from their gardens, which made maintaining them unsustainable in the long-term, or they did not have access to water at all during periods of drought. WALA also supplied seeds and equipment such as watering cans during the project. Once WALA was over, some farmers no longer had access to the seeds. For the second type of activity, WALA supported care groups, support groups, safe motherhood groups, and other groups focused on health, sanitation, and nutrition. WALA also promoted CBOs within communities. These groups were active in providing valuable services in their communities, from establishing nurseries to care for young children and visiting lactating mothers to teach about exclusive breastfeeding, to encouraging women to deliver at health centers and hospitals and tracking ART defaulters to promote child protection. These services were very well received during WALA, and respondents expressed their appreciation of the information that they learned and services they received and provided. Participants also reported being enthusiastic about involving male community members in care group activities for women and children. Since WALA ended, the sustainability of these activities has been mixed. Of the activities that have been sustained or expanded, support groups that serve as community forums and track ART defaulters have expanded in many communities. Four community-based organizations in the 14 villages sampled for the qualitative study continue to exist. Some child protection groups continue, and some activities of care groups are continuing, even if they are extremely reduced. Finally, some respondents reported that individually they try to follow the recommendations they learned from various groups. However, in general, most care groups have disbanded or contracted since WALA ended. A lack of resources seems to be part of the reason. Community members reported they were unable to make contributions at nurseries to cover costs that were previously covered by WALA or purchase airtime to communicate with health personnel. Some of the health promoters who had been hired to create care groups left, and local volunteers were not able to sustain activities. One woman reported the health group in her community was “like sheep without a leader.” (FG5_F13) The third type of activity with mixed sustainability was health-seeking behavior. WALA provided training, particularly to community volunteers, to encourage pregnant women to deliver in health centers and to take their newborns for checkups. WALA provided phones to volunteers to 45 communicate with health centers and send reports. One volunteer explains the work they did. We were mostly regarded as community volunteers…we were trained about… assisting pregnant women and children.…Whenever we came across a pregnant woman, we were explaining to them the advantages of giving birth in a health center or hospital…When a child has been born we also monitor the child...If the mother is not taking the child for growth monitoring, we discuss with the mother so that she should be going with the child to the health center for growth monitoring. (FG12_M5) When asked about the WALA activity with the biggest impact, many respondents mentioned health￾seeking behavior activities. Some community members reported that they thought maternal mortality went down and many reported that they individually still used the knowledge they learned during the trainings—most notably, having husbands accompany their wives to checkups. However, many volunteers no longer provide the services they did during WALA. Working outside of any structure, motivation waned. Another barrier after WALA was simply the inability to purchase airtime to communicate with health personnel and phones provided by WALA for safe motherhood groups having been damaged. The fourth and final activity that had mixed sustainability results was handwashing stations. WALA trained communities in the importance of handwashing and promoted the use of tippy taps for this purpose. While most participants in interviews understood the benefits of handwashing, particularly after defecating and before preparing food or washing dishes in order to avoid spreading disease, it appears that not all households could continue their use due to lack of water. Tippy taps have been sustained in some households that have access to water and a number were observed by the evaluation team. However, use of those in households that have scant water resources have been discontinued. Some community members reported that they thought tippy taps had contributed to a reduction in diarrhea, but without continued encouragement, motivation was also a barrier. 3.Sustained Only two of the eight health and nutrition activities were sustained after the WALA project ended. They were efforts to keep communities clean and activities focused on latrines. Community members reported that WALA taught lessons on the importance of particular hygiene and sanitation activities such as sweeping, smearing homes with mud, and laying clothes to dry on racks. WALA promoted these activities through community leaders who would take responsibility for clusters of 10 households near their own. Leaders met and were trained in sanitation and in turn trained the ten households assigned to them. One cluster leader described the change she saw during WALA. “During that time, people did not have handwashing facilities, but after we trained them, people made handwashing facilities and others constructed toilets. This has helped us a lot.” (FG12_F6) Community members reported appreciating these activities and the results. As one woman in Chiradzulu explained, The coming of WALA helped our village a lot. Many people did not have toilets. The coming in of WALA ensured that every household should have a toilet, the tippy taps, a cover for the toilet, a drying rack, a rubbish pit, a clothes line. So the coming in of WALA has helped our village to be bright. (FG8_F6) Community members reported that many of these new ideas about keeping communities clean through sanitation and hygiene activities continue to exist. In another village in Chiradzulu, a focus group 46 participant explained that they were able to continue these activities because their health surveillance assistant continued to promote community cleanliness and hygiene after WALA ended. She noted, “although WALA ended, the extension workers were still promoting the activities, and we worked very hard when working with them.” (FG9_F) In another Chiradzulu village, a man described the continuing activities as follows: “After introducing the tippy tap, they [WALA] taught us to smooth our household floors, our kitchens have to be hygienic, everything has to be clean. We dig garbage pits where we throw the garbage, and women also smooth the toilet floors.” (FG9_M) Promotion of latrines is the other activity that has been sustained. Communities learned how to construct pit latrines through WALA and, in most cases, it appears that the latrines are still being used. As one focus group participant noted, “Since the WALA project came, that is when we started the building of pit latrine[s]… It started that time up to now.” (FG10_F4) Many participants reported that WALA had trained communities on the importance of latrines in reducing the spread of disease. Perhaps due in part to this training, some participants reported a movement within their villages to enforce the use of toilets rather than open defecation, which has contributed to participants’ motivation to use latrines. As one focus group participant noted, Sanitation is continuing because of cholera. Last year no one suffered from cholera in our village because our advisors have worked hard together with chiefs. Everybody has a toilet. No one goes to the bush. … [The TA chief] said that anyone who goes into the bush, we should get their name. (FG14_F) Communities reported that they are still building pit latrines, especially because of the enforcement from local chiefs and other government agencies. Factors facilitating and hindering sustainability of nutrition and health interventions Below, we take a systematic look at the supportive factors and barriers to sustainability that different activities faced. Table 11 summarizes our findings using a sustainability framework to classify facilitators and barriers according to whether they were resources, capacities, motivations, or linkages. After detailed findings, we also discuss the effects of shocks and stresses on the sustainability of health and nutrition activities. 47 Table 11. Factors facilitating and hindering the sustainability of WALA activities in health and nutrition Evaluation Framework Which factors facilitated (✓) or hindered (✗) the sustainability of the following program activities? Factors Activities Resources Capacities Motivations Linkages Kitchen gardens ✗ ✓ Care groups ✗ Mixed ✓ ✗ Health seeking behavior ✗ Mixed Mixed Child nutrition ✗ ✓ Mixed MCHN food rations ✗ Keeping communities clean ✓ ✓ Tippy taps ✗ ✓ Mixed Latrines ✓ ✓ ✓ Note: “✗” indicates a barrier to sustainability, “✓” indicates a facilitator of sustainability, and “Mixed” indicates the presence of both. 1.Sustained source of resources The first factor that affects sustainability is a sustained source of resources. A sustained source of resources did not play a role in facilitating the sustainability of any of the activities in the health and nutrition group. Instead, the lack of sustained resources post-WALA was a barrier to the sustainability of six of the eight WALA-promoted health and nutrition activities, as participants did not have the inputs needed for activities. The most obvious example of this is a lack of MCHN rations and local foods needed for child nutrition. Once WALA ended, most community members no longer had access to MCHN rations (although a smaller number did receive rations from another project) and droughts restricted access to local food sources, making WALA-promoted child nutrition practices difficult or impossible to continue. Another example was the lack of resources needed to run cooking demonstrations, pay volunteers, or purchase airtime for phones. These inputs were provided by WALA during the project but were ones which community members could not afford after the project ended. Cooking demonstrations, nurseries run by volunteers, and communications with health personnel were therefore not sustained after the project ended. Another example was the lack of water. This affected the sustainability of tippy taps and kitchen gardens in places where water sources were distant or unreliable. 48 2.Sustained technical and managerial capacity The second factor that affects the sustainability of development projects is sustained technical and managerial capacity, so that participants and service providers can operate independently of the project. This factor played a mixed role in the sustainability of most of WALA’s health and nutrition activities. On one hand, WALA provided a great deal of well-appreciated training for community members, leaving them with the technical and managerial capacities to continue activities as during the project. For example, community volunteers were trained in teaching lactating mothers about exclusive breastfeeding and growth monitoring; pregnant women and their husbands were trained to go to health centers and hospitals for deliveries and checkups; communities were trained to use local ingredients for nutritious meals; and communities were taught about hygiene and community sanitation and how to build pit latrines. However, community members did not have access to continued training, which limits the usefulness of these capacities in the long run. First, community members who were not trained during WALA missed out; for example, women who were not pregnant during WALA but later became pregnant would not have received training in health-seeking behavior, such as going to health centers for deliveries and checkups. Second, since many of the activities disbanded and practices were only continued by some individuals in their own households, the opportunity to share or reinforce capacities was diminished. Third, misinformation, misremembering information, or outdated strategies cannot be corrected or updated for changing circumstances without additional training. 3.Sustained motivation of beneficiaries and service providers that does not rely on project inputs The third factor that affects the sustainability of development projects is sustained motivation on the part of project participants and service providers to continue engaging in the activities after the project ends. For the suite of health and nutrition activities, we did not see any motivation for service providers to continue engaging in the activities after the project ended. For participants, we found sustained motivation to be both a facilitator of and a barrier to sustainability. It acted as a facilitator in two main categories: where benefits were experienced by individuals, and where benefits were perceived for the community. Individual benefits that reportedly motivated participants included easy sources of vegetables for the small group of kitchen gardeners who continued them and an appreciation of increased involvement of husbands in prenatal and baby checkups by women. More substantially, respondents reported that they thought the health and nutrition activities promoted by WALA led to benefits for the community, such as a perceived reduction in maternal deaths because women were delivering at health centers and hospitals, perceived reductions in child malnutrition and child deaths due to nutrition learnings, and perceived reductions in diarrhea and cholera due to new sanitation practices such as tippy taps. However, those motivations did not appear to be strong enough alone to sustain these activities in the long term. The lack of sustained motivation was a barrier to sustainability after WALA ended when project activities relied on continued encouragement and monitoring by WALA implementation staff. Respondents reported that the absence of these roles reduced volunteers’ motivation to continue work, such as organizing or participating in cooking demonstrations, and reduced community members’ motivation to continue the use of tippy taps and other promoted practices. Perhaps a combination of a 49 lack of monitoring and a lack of resources for airtime has also led some community volunteers to use remaining phones provided by WALA for personal use rather than WALA-promoted activities. 4.Linkages to governmental organizations and / or other entities A fourth factor important for the sustainability of development projects is linkages, including vertical linkages, such as from a community care group to district or national health institutions, and/or horizontal linkages, such as among local groups or committees. Appropriate linkages are important for the sustainability of most technical sector interventions. Unfortunately, there were very few linkages established or sustained in conjunction with WALA’s health and nutrition activities. Respondents did report that a linkage with health surveillance assistants was helping to sustain community sanitation and hygiene activities. There were also reports that local chiefs and other government agencies were enforcing regulations around open defecation, which seems to be facilitating sustainability of the latrine activities. However, care groups that were linked to government health centers and health surveillance assistants have not been sustained. The lack of substantial vertical or horizontal linkages to support WALA health and nutrition activities was certainly a barrier to their sustainability. The effects of shocks and stresses on sustainability of nutrition and health interventions Shocks and stresses experienced by households and communities during the four years after WALA ended did affect sustainability of WALA activities. Respondents noted a number of ways in which shocks negatively impacted WALA-promoted activities. Droughts and pest attacks negatively affected health and nutrition activities through the resultant lack of food. Respondents reported that nutrition cooking demonstrations and community complimentary feeding and learning sessions could not be sustained because the drought reduced dietary diversity and households’ access to recommended foods. “Each one of us [group leaders] had his or her own household to visit, and after three days we did the growth monitoring to find out how children have changed. But because of shock which is the drought we stopped.” [RH_F1] There was also not enough food for activities like community-based childcare centers, so parents did not want to send their children there. The lack of water from droughts also created a barrier to the sustainability of kitchen gardens, since drinking water had to be prioritized. Lack of water also stopped tippy tap use in some communities. The severity of the shocks experienced in this four-year period was extreme and overwhelmed much of the work WALA had done in nutrition and health. 50 Research Question 2: What was the impact of WALA on resiliency? Key findings: • Survey data show that household respondents’ perceptions of their recovery from their most recent shock is worse in WALA villages than in comparison villages. • Even in the top-performing villages selected for the qualitative study, most of the activities aimed at increasing absorptive, adaptive, and transformative capacities were not sustained by the majority of participants after the project ended. Weather and pest shocks also overcame any new resilience capacities of households that participated in WALA. One of the goals of WALA was to build the resilience capacities of WALA households and communities through the activities implemented. USAID defines resilience as “the ability of people, households, communities, countries and systems to mitigate, adapt to and recover from shocks and stresses in a manner that reduces chronic vulnerability and facilitates inclusive growth” (USAID 2013). In this section, we describe the effect WALA had on household and community resiliency capacities to face shocks. (Southern Malawi has experienced a remarkable set of shocks and stresses over the past decade. In the appendix, we describe the main shocks and stresses faced by WALA communities and households in the four years since WALA ended, using findings from both our qualitative and quantitative research. The most common of these shocks included crop pests and diseases (including fall armyworm infestations) and both droughts and flooding conditions.) Resilience capacities of households and communities to withstand shocks are multidimensional and can encompass a range of abilities, including economic, social, technological, infrastructure, and institutional. Correspondingly, WALA used a multi-sectoral approach. For example, WALA’s focus on agriculture practices was supported by its work to strengthen the provision of extension services and the creation of village saving and lending organizations (VSLs) within local communities. Building different resilience capacities can strengthen households and communities and reduce their vulnerability to shocks and stresses in three main ways: by helping them absorb, adapt, and/or transform in the face of disruptions. Absorptive capacity is the capacity of households or communities to bounce back after a shock, which involves anticipating, planning, coping and recovering from shocks (Oxfam 2017). This capacity would include households’ assets, availability of informal safety nets, access to cash and savings, preparedness and mitigation plans, availability of humanitarian assistance, and the like. During shocks, households might use one or more of these capacities in order to absorb and bounce back from the shock. Adaptive capacity is the ability of a household or community to make appropriate changes in order to better manage or adjust to a changing situation, such as climate change or increasing periods of dryness. This would include changes in management of land, soil and water, and enhanced and inclusive access to productive resources including credit, markets, livestock, social networks, and livelihood sources. Transformative capacity is the capacity to make fundamental changes in the structure that causes or increases vulnerability and risk as well as how risk is shared within societies. Transformative capacity 51 provides the enabling environment that allows households or communities to absorb or adapt to shocks and stresses in ways that do not have negative impacts on their wellbeing (Bene 2018). Transformative capacity includes communal natural resources, basic agriculture and livestock services, collective actions, availability of formal safety nets, networking with external people/institutions, and the like. All three types of capacities allow for the possibility that households and communities can be both vulnerable and resilient (FSIN 2014). One of the goals of WALA was to develop the resilience capacities of both households and communities in program areas by providing knowledge, skillsets, and technologies to improve their abilities to absorb shocks and stresses, adapt to them, and transform to reduce their impact. Below, we examine households’ overall perceptions of their own resilience to the shocks they have faced in the four years after WALA ended using household survey responses. We then explore how WALA’s activities impacted the three types of resilience capacities for communities and households, based on focus group and key informant interviews. Perceptions of resilience Our survey data show respondents’ perceptions of their recovery from recent shocks in both the WALA villages and comparison villages. In Table 12 below, we provide regression-adjusted results on respondents’ ratings of their own household’s perceived recovery from each recent shock. These results indicate that households in WALA-participating villages actually report worse recovery from recent shocks than do those in comparison villages. This is true across a variety of empirical specifications, control variables, and robustness checks. The share of households reporting only partial recovery is significantly higher in WALA-participating villages than in comparison villages, where more households are more likely to experience complete recovery. This is despite the fact that the frequency and types of shocks reported in WALA and comparison villages are similar. Table 12. Household recovery from the most recent shock +District controls +UBALE/Njira control +Baseline stunting controls Comparison group mean Household recovered or partly recovered from shock Treatment effects -2.9% -2.9% 3.3%* 67.7% [-1.3] [-1.3] [-2.0] i p-value in brackets ***, **, and * indicate significance at the 1, 5, and 10 percent critical level. Controls are listed in the left-most column. The full control set includes fixed effects for district, UBALE/Njira participation, housing characteristics, and each of 22 shock types, as well as a polynomial in baseline stunting rates. 52 Moreover, when asked about the change in their resilience relative to conditions 10 years ago, households in WALA villages less frequently cite any improvement (although these differences are not statistically distinguishable). Table 13 below documents that the frequency of household respondents reporting no change in preparedness was higher in WALA villages, although this difference is not statistically different. Table 13. Household shock preparedness relative to 10 years ago Comparison group mean Treatment effect Respondents better off today than 10 years ago 22.2% 0.0% Respondents report there is no difference today than 10 years ago 45.4% 3.7% Respondents worse off today than 10 years ago 32.5% -0.9% ***, **, and * indicate significance at the 1, 5, and 10 percent critical level. These results control for district fixed effects, UBALE/Njira participation, and baseline predicted stunting controls. Below, we explore through qualitative data why we do not see an improvement in the perception of resiliency in villages where WALA was implemented. Absorptive capacities Several of WALA’s strategies were designed to build absorptive capacities of households and communities. One of the main successes in the 14 villages selected for the qualitative study was promoting village savings and loans (VSLs). This increased households’ access to cash as savings and as loans. This was frequently mentioned by participants as a support that played a critical role for households facing shocks after WALA left. Respondents reported that they used money saved or borrowed to help absorb and recover from shocks. As one woman explained, “through village banks you are able to borrow money… to buy food.” (FG4_F3) Others reportedly borrowed money from village banks to buy livestock as a form of insurance should hard times come. One farmer explained how his household uses this strategy: “When we have four livestock, we sell two livestock to obtain food so that children eat.” (FG14_M2). WALA also promoted a number of nutrition activities to build participants’ coping strategies when dealing with food insecurity, including dietary diversification training and the importance of using locally available foods to increase nutrient intake. Households at large regard nsima, a cornmeal porridge, as a staple food due its high calories. Foods such as sweet potatoes, rice, and cassava are considered less preferred foods, but were encouraged for nutritional reasons and to diversify food sources, in cases of shock. During droughts, some households reportedly ate sweet potatoes and cassava. As one woman explained, “we have to eat food that is available in the household; maybe we can eat the sweet potatoes in the morning, in the afternoon or in the evening. This helps to manage food.” (FG10_F6) 53 Despite these designs, in general the absorptive capacities of households were not affected by WALA activities, because most of the activities were not sustained after the end of WALA. The severity of the shocks and stresses were such that they overcame any new absorptive capacities of households that participated in WALA and made households’ efforts to build those capacities collapse as well. For example, even in the 14 top-performing villages selected for qualitative study, when droughts came in 2014-2018, dietary diversification was not an option for many households to deal with food scarcity, since the drought meant that fewer or no types of food were available. Adaptive capacities Most of WALA’s activities were designed to help households change the way they managed land, soil, water, and other resources and enhance their access to productive resources such as credit and markets, in order to be better able to manage or adjust to changing situations. For example, in the 14 villages selected for the qualitative study, VSLs created the opportunity for households to borrow money to diversify their livelihoods in the face of shocks. In one example, a number of women obtained loans from their VSLs during lean seasons so their husbands could travel into Mozambique to buy maize. Some of the maize was consumed by the households, while some was sold to repay the loans. Loans were used by respondents to start small-scale businesses, including buying and re-selling livestock, crops, and clothing, and selling fritters or other food. One government official reported that this strategy helped farming households deal with shocks to crop production, reporting that “the main use is through engaging in small scale businesses, they use earnings to buy food.” (GO_F2) Another set of capacities targeted by WALA included providing training to change or improve budgeting and food management. This was designed to help households cope with food insecurity by determining appropriate food portions, increasing their acceptance of eating foods other than maize as a main meal, and developing household budgets. WALA also worked to strengthen the adoption of improved farming practices and promoted conservation agriculture to address adverse climatic changes affecting agriculture. Participants in these activities reported that practices such as crop rotation and mulching enhanced crop production, reduced soil erosion, helped conserve soil moisture, and protected crops from being washed away by storms. These skills were designed to help farmers adapt to shocks such as floods and droughts by reducing the impact of shocks. For example, mulching helps retain moisture for growing crops when rains are not sufficient. Pit planting is another method. As one respondent reported, In line with conservation farming, in trying to retain some of the moisture, some people have now started practicing conservation farming. Maize planted on ridges wilts faster than the one that has been pit planted. Now for those practicing this, they plant maize as if they are planting groundnuts. So that at least the residual moisture should be used up by the maize. All this because of drought. (FG6_M12) WALA promoted new crops that were more resistant to drought, such as sweet potato, and introduced new crop varieties of rice, maize, and cassava that mature more quickly in responses to drier weather so households could remain food secure in a changing climate. Crop diversification was promoted to reduce food insecurity in periods when maize yields were low by giving farmers access to harvests from other crops even if one crop failed. A government official lauded this strategy, saying, “even though the yields for maize may be lower than expected, the availability of [other] foods and the level of hunger is totally different from someone who did not grow other crops such as sweet potatoes, 54 tomatoes and cassava.” (GO_F2). One resilient household member who adopted some of these practices reported, “I did not require food aid.” (RH_M5) Other households who adopted these practices also reported being more food secure. Another farmer who adopted crop diversification and was asked if he was still practicing it responded, “yes, and I still have food.” (FG13_M5) Agriculture practices promoted by WALA were heralded for assisting households to obtain higher yields and help meet their food requirements during the project. Especially for resilient households—who participated in WALA and were reported to have food throughout most of the year, to have diversified income sources, and to be able to bounce back in the event of a shock—these practices increased food security. As one resilient household member in Chiradzulu explained, “after learning what WALA taught us [I] still follow it up to now. My household has food. I never lack food. I don’t buy food now because I am following what WALA taught.” (RH_M7) Another resilient household member in the district also reported, “I had enough to eat [after using Sasakawa]. Some people were even coming here to borrow bags of maize.” (RH_M5) However, even in the top-performing villages selected for study, most of the activities aimed at increasing adaptive capacities were not sustained by the majority of participants after the project ended. Weather and pest shocks severely mitigated the impact of WALA’s agricultural and home management interventions if they were continued. For instance, farmers in Machinga explained that while new crops introduced by WALA reduced their hunger, they did not eliminate it. “The sweet potatoes—the children’s stomachs were full because of that. Though the sweet potatoes sustained us, we did not have enough food…because the plots with moisture are few [due to drought].” (FG5_M8) Transformative capacities A few of WALA’s activities were designed to make fundamental changes to the structures that cause or increase vulnerability to shocks. Four of these activities included training private service providers (PSPs) to provide basic agriculture and livestock services; linking community members to external business to market new crops; conservation agriculture (CA) techniques such as tree and Vetiver grass planting; and implementing irrigation infrastructure. The promotion of irrigation farming was a transformative approach to help improve resiliency by creating infrastructure to make farming more successful in more situations. In the few places it still exists, its potential is exciting. As one farmer in Zomba who still uses irrigation reported, “to ensure we harvest more and also to become more food secure…we depend on irrigation farming because it sustains us.” (FG3_M8) A government official reported on farmers still using WALA-promoted irrigation, stating, “it is mostly tomatoes that are grown as cash crops, which they sell at the markets and then they are able to buy food.” (GO_F2) However, not all households in WALA communities had access to irrigation; shocks in many communities were severe enough that the irrigation did not work anyway; and participants reported that most of the irrigation systems ultimately failed after WALA ended. This lack of sustainability ultimately undermined this resiliency capacity. Creating access to markets was another transformative capacity that WALA tried to improve but which was ultimately not successful. Soil and water management were promoted by WALA through tree and Vetiver grass planting along riverbanks to protect houses from heavy winds and flooding and through digging trenches to direct rain and flood water. However, most of these activities and others were not sustained in the 14 selected villages for the qualitative study or in the full sample as villages, as reported above; therefore, they did not have the hoped-for impact on resiliency. 55 Shocks overwhelmed coping strategies As noted previously, after WALA ended, communities faced numerous, widespread, and varied shocks and stresses. Interview participants reported that geophysical and climate-related shocks have been recurring every year since 2015, and that the recent droughts had been longer and more severe than in the past. Combined with flooding and windstorms, the results have been devastating. One woman in Zomba described the recent years thus: “After the drought, rainfall came, and the maize that survived from the drought grew. And thereafter, the heavy winds came and destroyed the rest of the maize crops.” (FG1_F8) These shocks have been accompanied by pest and insect attacks and market failures. The majority of the people in these communities are poor farmers and the impacts of these shocks were tremendous. When households face more than one shock in a relatively short period, it becomes disproportionately more difficult to cope with and recover. Given the large parts of the populations affected, local social capital is highly strained as a resource. Building resiliency to these shocks has been hampered by their severity, number, and spread. A normal adaptive capacity in the face of shocks would be relying on bonding social capital, that is, connections with others nearby who can provide support in the form of cash, food or non-food items in cases of emergency. However, with shocks affecting the majority of the community, even this capacity is diminished. Respondents to the qualitative discussions reported that currently most members of their communities need food aid. The adaptive and transformative capacities WALA attempted to build often failed in the face of the shocks they were meant to manage or avoid. Another common adaptive and transformative capacity in cases of shocks is depending on bridging social capital, that is, connections with outside institutions or people in order to get cash, food, or non-food in-kind support. As explained in the sustainability analysis above of the activities WALA implemented, linkages outside local areas were rare. Government responsiveness would be another capacity important for providing help with shocks of this magnitude. However, in this situation, the government has not been able to act promptly. As one implementer said, “before we came in, when these people have faced a shock, it took a long time for the government to respond, or sometime the government [is] not coming to their rescue at all.” (IP_M3) Amid these shocks and stresses, households’ living conditions became unstable and their resilience capacities were overrun. The impacts of the shocks, therefore, have been widespread hunger in the communities. Almost all respondents in focus groups said that they needed food aid, as there was no food to eat. These data show that resiliency has been particularly hard to achieve. 56 Research Question 3: What was the impact of WALA on child nutrition? Key findings: • Anthropometric data show that child stunting and underweight rates remained constant between the endline (2013) and follow-up (2018) rounds in WALA villages. • We observe very similar rates of child stunting and undernutrition in comparison villages, indicating no impacts that are attributable to WALA. • Food insecurity appears slightly worse in WALA villages than in comparison villages, and dietary diversity is not any better in the WALA villages than in the comparison group. WALA set out to “improve the food security of 214,974 chronically food insecure households in 39 Traditional Authorities within five livelihood zones in Southern Malawi by 2014”.13 One of the measures of success was the reduction in child stunting and underweight rates determined by anthropometric measurements of children aged 6 months to 5 years old. Stunting in early childhood can have negative impacts on adulthood including “poor cognition and educational performance, low adult wages, lost productivity and, when accompanied by excessive weight gain later in childhood, an increased risk of nutrition-related chronic diseases.”14 Height, weight, age, and presence of edema were measured and used to create height-for-age and weight-for-age scores for each eligible child.15 Children who had a height-for-age normalized z-score under -2 were considered stunted while children who had a weight-for-age normalized z-score under -2 were considered underweight. At baseline, 42.4%16 (95% CI of 39.7% - 45.2%) of children aged 6-59 months were identified as stunted. At endline, stunting had dropped to 37.1% (34.9% - 39.3%). In the 2018 follow-up survey, the stunting rate was 34.2% (31.9% - 36.5%). The underweight trends are similar over this timeframe. At baseline, 17.6% (15.6% - 19.6%) of children 0-59 months were considered underweight. At endline, this number had decreased to 11.3% (9.9% - 12.7%) and stayed consistent at follow-up at 11.8% (9.7% - 13.8%). These numbers align with the national 2015/2016 DHS survey that reported 37% of children under 5 years old were stunted and 12% were underweight. 13 Wellness and Agriculture for Life Advancement PL480 Project. Accessed at: https://www.globalwaters.org/HowWeWork/Activities/wellness￾and-agriculture-life-advancement-pl480-project 14 WHO. 2018. Stunting in a nutshell. Accessed at: https://www.who.int/nutrition/healthygrowthproj_stunted_videos/en/ 15 Z-scores were calculated using the Stata command zscore06 and were generated using the WHO Anthro Analyzer. Both scores were comparable. 16 The endline survey team was unable to validate baseline anthropometric indicators and therefore estimated their own indicators from the baseline data to create their comparisons. Thisreport utilizes the indicators calculated in the endline report for both the endline and baseline measurements. 57 Figure 2. Trends in WALA villages in childhood anthropometrics The survey team was able to return to WALA households surveyed at endline, which created the opportunity to develop a panel dataset for those households. (We do not have the same ability to compare households in comparison villages between endline and follow-up, since no comparison villages were surveyed at endline). With this panel dataset, the research team was able to look at any differences between children aged 0-59 months in WALA households at endline and their younger siblings that were 0-59 months at follow-up. There are similar stunting rates between both groups. At endline, stunting rates among children 6-59 months was 36% and at follow-up it was 37%. Similarly, underweight rates of children aged 0-59 months were 14% at endline and at follow-up they were 16% among their younger siblings aged 0-59 months in those specific households. The similarity in results when examining village-wide and within-household dynamics further suggest a consistent pattern of static malnutrition and anthropometrics post-WALA. 42.4% 37.1% 34.2% 17.6% 11.3% 11.8% 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% 35.0% 40.0% 45.0% Baseline (2009) Endline (2013) Follow-up (2018) Stunting Underweight 58 Figure 3. Comparing children in 2013 to younger siblings in 2018 in WALA villages However, the data described to this point reflect only the treatment group and thus do not distinguish between changes in the WALA villages and broader trends happening more generally in Southern Malawi. We therefore compare these anthropometric measures among children in WALA villages and those in our comparison group. We find that the rates of both stunting and underweight status are not statistically different between these groups. Table 14 below presents these results. The average stunting rate in the comparison villages is 32.6% (lower than the 34.2% rate in WALA villages). We then adjust for district-specific fixed effects, finding that stunting impacts to due WALA villages are 0.5%, but are not statistically different from zero. Adding controls for UBALE/Njira presence in each village and for a sophisticated set of non￾linear adjustments for baseline stunting rates (results in columns 2 and 3 of Table 14) continues to indicate no significant differences in stunting due to WALA. Thus, we find no average treatment effects attributable to WALA in terms of child stunting. Table 14. Stunting and underweight treatment effects + District controls + UBALE/Njira control + Baseline stunting controls Comparison group mean Stunting rates Treatment effects 0.5% 0.6% 0.1% 32.6% [0.8] [0.8] [0.9] Underweight rates Treatment effects -0.5% -0.2% -0.6% 12.0% [0.7] [0.9] [0.7] i p-value in brackets 36.0% 37.0% 14.0% 16.0% 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% 35.0% 40.0% Endline (2013) Follow-up (2018) Stunting rates Underweight rates 59 The control variables are listed on the left-most columns. Column 1 includes only district fixed effects; column 2 adds a fixed effect for UBALE/Njira participation; and column 3 adds a polynomial in baseline predicted stunting and housing characteristic fixed effects. ***, **, and * indicate significance at the 1, 5, and 10 percent confidence level. It is important to note that these null results are actually quite precise. Our evaluation sample size and design were powered to detect impacts of only 2.5 percentage points (roughly half of the change in stunting between 2009 and 2013). Moreover, our regression-based standard errors are consistent with minimum detectable effects of 3.6 percentage points. Thus, even under our most optimistic results, we can conclude that there are no WALA impacts larger than 3 percentage points. Given that the mean stunting rate in the comparison villages is 32.6%, we are confident that any treatment effects would be no larger than 1/10 of such stunting prevalence. We then examine WALA effects on underweight status, finding nearly identical mean underweight rates in our comparison and treated villages. Adjusting for district-specific unobservables (col 1), UBALE/Njira interventions (col 2) and baseline stunting rates (col 3), we find very small reductions in underweight status that are at most 0.6%. Again, our regression-based standard errors provide minimum detectable effects of 2.9 percentage points. Thus, we are confident that any drops in underweight status due to WALA are quite small. Taken together, these results indicate that WALA has not induced long-term changes in village-wide childhood undernutrition and anthropometrics. Because we do not have measures of these anthropometrics among our comparison group in 2013, we cannot distinguish between several potential dynamics and interpretations of our results. It is possible that during the WALA program, participating villages experienced a reduction in stunting and underweight status that other villages (such as those in our comparison group) did not, but that the latter villages "caught up" once the program ended. Alternatively (and potentially more likely), both WALA and comparison villages similarly benefitted from broader trends happening in Malawi, in which drops in childhood undernutrition between 2009 and 2013 have slowed in more recent years. However, irrespective of these dynamics, our primary results described above mean that we observe no long-term impacts of WALA on childhood anthropometric outcomes. We next explore whether there were discernable impacts on households’ dietary and food security outcomes that may not have led to detectable impacts on child anthropometrics. To do so, we assess WALA impacts on dietary diversity, as measured by the Household Dietary Diversity Score (HDDS). In Table 15 below, we show that the comparison group average score was 5.6, with the treatment group experiencing nearly identical scores, irrespective of the particular set of controls used. Moreover, we also analyze WALA impacts on the degree of food insecurity experienced by each household, as measured by the number of months of the preceding year in which the household experienced food insecurity. The comparison group mean is approximately 3.2 months each year and households in WALA villages on average experience an additional one-third of a month of insecure food access. This result is consistent across specifications and controls and is statistically significant at the 95% confidence level. It indicates that, if anything, food security appears to be worse in WALA villages than in the comparison group. Table 15. Food insecurity and HDDS with controls +District controls +UBALE/ Njira control +Baseline stunting Comparison group mean 60 controls Household Dietary Diversity Score Treatment effects -0.1 -0.1 -0.1 5.6 [-1.0] [-1.0] [-1.0] Food insecurity (number of months) Treatment effects 0.4* 0.4* 0.4* 3.2 [2.3] [2.3] [2.2] i p-value in brackets The control variables are listed on the left-most columns. Column 1 includes only district fixed effects; column 2 adds a fixed effect for UBALE/Njira participation; and column 3 add a polynomial in baseline predicted stunting and housing characteristic fixed effects. ***, **, and * indicate significance at the 1, 5, and 10 percent confidence level. 61 Research Question 4: Did improvements in resiliency due to WALA allow communities to maintain child nutrition gains even in the face of environmental disruption? Key findings: • WALA and comparison villages experienced quite similar shocks. Resiliency (measured under Research Question 2) is not meaningfully better in WALA villages. • As a result, we do not observe that child nutrition outcomes were differentially sustained in WALA villages experiencing more severe shocks relative to the outcomes in comparison villages experiencing such shocks. The preceding sections document varied results on multiple measures of resiliency, generally indicating little consistent evidence of improvements in resilience (and, in fact, documenting worsening self￾perceived resilience due to WALA). It is nonetheless possible that WALA engendered actual resilience in terms of childhood nutrition, even in the face of severe environmental shocks. Moreover, if different portions of our sample experienced particularly pronounced shocks that WALA helped them weather, our null average treatment effects on childhood malnutrition may obscure important gains in these hard-hit areas. We therefore seek to test whether child anthropometrics impacts are consistent across villages with relatively worse environmental shocks post-WALA. Consistent with the self-reported shocks discussed above, various weather data reveal that Southern Malawi faced several significant weather shocks between the end of WALA in 2014 and the follow-up survey in 2018. In 2015, households faced a severe flood that affected the entire southern region. Blantyre, Nsanji, Chikwawa, Mulanje, and Zomba districts were the hardest hit. The following year, households faced a drought that led to crop failure in those southern districts.17 Again, the data shows that both WALA and comparison villages were hit equally hard by these back to back climatic shocks. The research team joined weather data from the Climatic Research Unit (CRU) to household data to order to analyze how much precipitation each village received during the 2014–2018 window and whether there was any difference between WALA and comparison villages. The CRU data is collected from weather stations and sensors around the world and is used to produce a global surface at 0.5 decimal degree resolution.17 By joining precipitation levels and looking specifically at levels during the growing seasons, the CRU data confirmed that both WALA and comparison villages were hit equally hard by these weather shocks. Furthermore, this similarity is consistent with the similarity in shocks reported in the household surveys. 17 World Bank. 2016.Malawi drought 2015-2016: post-disaster needs assessment(PDNA)(English).Washington,D.C.:World Bank Group. http://documents.worldbank.org/curated/en/640011479881661626/Malawi-drought-2015-2016-post-disaster-needs- assessment-PDNA 62 Table 16. Growing season precipitation rates Variable Mean Standard deviation 25th percentile 75th percentile 2008-09 growing season monthly precipitation in mm 121.6 15.3 105.4 131.4 2009-10 growing season monthly precipitation in mm 106.2 13.6 96.4 116.1 2010-11 growing season monthly precipitation in mm 124.0 16.3 112.2 134.9 2011-12 growing season monthly precipitation in mm 129.7 15.1 119.3 138.6 2012-13 growing season monthly precipitation in mm 118.7 16.1 107.9 128.8 2013-14 growing season monthly precipitation in mm 112.6 13.5 106.5 121.6 2014-15 growing season monthly precipitation in mm 98.4 20.9 86.7 110.8 2015-16 growing season monthly precipitation in mm 82.2 15.3 66.4 90.0 With both WALA and comparison villages struggling equally in the face of the flood and drought, it is possible to evaluate whether or not WALA households were better equipped to handle these shocks compared to those villages that did not benefit from the program. After comparing the outcomes between WALA and comparison villages facing these weather shocks, the quantitative data shows that WALA households’ children were no better off than comparison villages in terms of underweight and stunting rates. Throughout these analyses, we find little consistent evidence of gains in child anthropometrics in WALA villages experiencing particularly pronounced droughts or flooding. Outcomes in both WALA and comparison villages are quantitatively quite similar under these varying conditions. The quantitative data collection shows that any improvements in resiliency from WALA seen at endline was unable to help communities overcome the environmental disruptions seen between the end of the project and today. 63 Research Question 5: Did gains in child nutrition due to WALA lead to subsequent improvements in health and education at older ages? Key findings: • Cohorts of children who were under 5 during the WALA program appear to be slightly better off in terms of their education than similar cohorts in comparison villages. • These positive impacts are limited to attending school, and do not extend to grade completion, literacy, or other measures. Despite not finding any significant differences between WALA and comparison villages in stunting and underweight rates among young children at follow-up, it is nonetheless possible that cohorts of children who were young during WALA's implementation and thus may have benefitted from the program continue to experience some long-lasting gains themselves. Indeed, while we lack data on comparison sites at the project endline to assess whether there were any impacts on child anthropometrics during the project, examining these longer-term outcomes among such cohorts can help trace out such early￾life impacts. We consider a wide age window for such children, assessing outcomes for children who were 0-8 years old when WALA was active, and are thus 4-12 years old during the follow-up survey. We focus on educational outcomes for these children, both because anthropometric measures at these older ages are not reliable indicators of malnutrition and because educational outcomes are both important directly and potentially influenced by earlier malnutrition. Table 17 below shows impacts on literacy, school attendance, highest class and qualification achieved, and whether or not they were still in school. Table 17. Educational results Variable Comparison group mean Treatment effect Attended school 87.2% 2.9%* Able to read and write in Chichewa 31.7% 2.3% Able to read and write in English 8.8% -0.3% Attended school last year 82.1% 1.0% Age child started in school 5.6 years old 0.1 Highest grade completed 2.0 [standard 2] 0.0 * Indicates significance at 90% confidence levels. These results control for district fixed effects, UBALE/Njira participation, and baseline predicted stunting controls Note: The 2015-2016 DHS reports that 75% of women and 83% of men in Malawi are able to read in their local language. The WALA follow-up survey asked about both reading and writing in one question, and so the results cannot be directly compared to the DHS literacy outcomes. 64 After analyzing all of these factors, WALA children were 3% more likely to have attended school than children of the same age in comparison villages, with the difference statistically significant at 90% confidence. However, there were no statistically significant differences between the two groups in literacy or grade completed. There were also no differences in current health status. Taken together, these results indicate some limited gains for children who were young during WALA implementation, suggesting some limited, longer-lasting impacts from WALA's interventions. 65 Conclusions Previous performance evaluations and case studies of WALA activities indicated that the program successfully introduced a variety of new ideas and practices around sustainable, productive agriculture, public health, and other themes. In qualitative interviews and focus groups in 14 of the most successful WALA villages, we find residents viewed many of these practices as useful, even four years after the program's completion. However, in these and our larger survey sample of households, we find only limited evidence of sustained use of these practices. In fact, in our full sample, we find no differential longer-term adoption of the WALA-promoted practices. In the 14 villages selected for the qualitative study, which were chosen as examples of the most likely places for interventions to be sustained, only three of 18 activity types were largely sustained, and none were expanded. The remainder were either entirely discontinued or had mixed results for sustainability. Without additional training, assistance from outside organizations or the government for motivation and resources, and support from groups working together to sustain activities, the impact of these activities will likely remain small and diminish over time. In the full sample of WALA villages, sustainability was even harder to find, which is not surprising. The full sample of villages on average did not have the high standards of implementation found in the 14 purposively selected ones, nor the community enthusiasm. Factors supporting sustainability—such as motivation of participants and quality of capacity building—were thus likely weaker in the full sample. As a result, we observe no differences in village-wide child nutrition conditions between WALA villages and comparison sites, including in terms of child anthropometrics and household dietary diversity. We find very slight traces of some of the potential benefits from the (temporary) participation in WALA during early childhood, as children who were young during WALA's implementation are now slightly more likely to have attended school than children of the same ages in comparison villages. Why did WALA's promise not materialize in the longer term? Our qualitative data collection highlighted three primary challenges: (a) lack of planning for sustained resources to support activities after the end of the project; (b) lack of continued technical and managerial capacity building to refresh and upgrade skills and bring more people up to speed; and (c) lack of linkages to governmental organizations and other entities external to the local context. More broadly, however, it is clear that the extent of shocks faced by villages in Southern Malawi further threatened the sustainability of many of the activities WALA promoted causing them to be discontinued and/or diminishing their potential positive impacts enormously. The shocks therefore outstripped communities' resilience. Both qualitative and quantitative evidence consistently showed that the crop pests, droughts, and other weather and biophysical shocks overwhelmed coping strategies. Given that the scale of these shocks extended beyond individuals and affected entire villages (and even districts), WALA-promoted practices that focused on within-village strategies were likely too narrow and small-scale to adequately protect households. Insuring against such shocks requires regional and potentially even national-scale responses and planning. Perhaps most troubling are the higher rates of food insecurity and weaker perceived resilience now exhibited by households in WALA villages relative to those in comparison villages. Households in WALA villages more frequently report longer spells of food insecurity, worse recovery from shocks, and weaker capacity to cope with shocks relative to their abilities a decade ago. There are several potential explanations for these results, although our data do not allow us to definitively distinguish between them. It is possible that, following WALA recommendations, households expended additional effort or resources to pursue new practices, including growing new cash crops and implementing new 66 agricultural or natural resource management techniques. The severity of the shocks wiped out any gains from these investments, leaving the households effectively worse off and more vulnerable, or perceiving themselves to be. Similarly, it is possible that by shifting households into new practices that were not sufficiently protected against major risks, these households were left more exposed than they had been previously (or were more exposed than households following more traditional practices and arrangements in comparison villages). Neither of these cases suggests that agricultural practices promoted under WALA were inherently flawed; instead, these practices needed to be implemented with sustainability planning in mind and adjusted to cope with the risk of major shocks. They should also be complemented by much broader transformational insurance and social protection schemes that extended beyond village or even district scales. We thus draw several lessons regarding sustainability from the analysis: 1. Ensuring that incentives are aligned with activities: Of the factors that can contribute to sustainability, the motivation of beneficiaries to continue activities, due to the personal benefits experienced had the largest positive influence on the sustainability of the agriculture and natural resource management activities. 2. Working with government and village leaders and focusing on bylaws and regulations: Linkages with local leaders and governmental agencies appear to have had the largest positive influence on the sustainability of WALA health and nutrition activities. These factors kept motivation high and created enforcement mechanisms to sustain practices. 3. Lack of sustained resources and lack of continued technical and managerial capacity building were large barriers to sustainability of agriculture and health and nutrition activities during the four years after WALA ended. In addition, lack of linkages to governmental organizations and other entities was a barrier for some agriculture interventions. 4. When working in an area prone to shocks, ensuring continued resources, capacity building, and linkages is essential to keep activities on track after shocks and ensure sustainable progress. Threats to sustainability caused many of the activities WALA promoted to be discontinued and/or diminished their potential positive impacts enormously. 67 References Béné, C. 2018. Concept Note: Measuring Resilience Across and Between Scales and How to Do It. Produced by TANGO International as part of the Resilience Evaluation, Analysis and Learning (REAL) Associate Award. Climate Risks in Food For Peace Geographies: Malawi. 2019. https://www.climatelinks.org/sites/default/files/asset/document/2019_USAID_ATLAS_Malawi_FFP_CRP. pdf Consortium Administration & Technical Capacity Hub, WALA. 2010. “Wellness & Agriculture for Life Advancement: Baseline Survey Report”. National Statistical Office (NSO) [Malawi] and ICF. 2017. 2015-16 Malawi Demographic and Health Survey Key Findings. Zomba, Malawi, and Rockville, Maryland, USA. NSO and ICF. Geonames. URL: http://geonames.nga.mil/gns/html/namefiles.html (visited September 2019). FAO. 2016. http://www.fao.org/3/a-i5665e.pdf Food Security Information Network (FSIN). 2014. A Common Analytical Model for Resilience Measurement: Causal Framework and Methodological Options. Resilience Measurement Technical Working Group. Technical Series No. 2. Frankenberger, T., Mueller, M., Spangler, T., & Alexander S. 2013. Community Resilience: Conceptual Framework and Measurement Feed the Future Learning Agenda. Rockville, MD: Westat. Gething, Peter, Andy Tatem, Tom Bird, and Clara R. Burgert-Brucker. 2015. Creating Spatial Interpolation Surfaces with DHS Data DHS Spatial Analysis Reports No. 11. Rockville, Maryland, USA: ICF International. Harris et al. 2014. Updated high‐resolution grids of monthly climatic observations – the CRU TS3.10 Dataset. doi:10.1002/joc.3711 Holmes, R. and Costella, C. with Bailey, M., Kruczkiewicz, A., Poulter, R. Sharp, K. and Scott, L. 2017. Towards A Shock Sensitive Social Protection System for Malawi. London: ODI and The Hague: Red Cross Climate Centre. International Food Policy Research Institute (IFPRI) Blog: Issue Post, FEBRUARY 26, 2018, BY KIMBERLY KEETON. Malawi's new reality: Fall armyworm is here to stay. http://www.ifpri.org/blog/malawis-new-reality-fall-armyworm-here-stay Landsat 7. URL: https://landsat.usgs.gov/landsat-7 (visited March 2019). National Statistics Office (NSO). Malawi Fourth Integrated Household Survey 2016-2017. ID Number: MWI_2016_IHS-IV_v02_M. Available at: http://microdata.worldbank.org/index.php/catalog/2936/related_materials 68 National Statistics Office (NSO). Malawi Third Integrated Household Survey 2010-2011. ID Number: MWI_2010_IHS-III_v01_M. Available at: http://microdata.worldbank.org/index.php/catalog/1003 Oxfam. 2017. The Future is a Choice: Absorb, adapt, transform resilience capacities. Available online at: https://oxfamilibrary.openrepository.com/bitstream/handle/10546/620178/gd-resilience- capacities￾absorb-adapt-transform-250117-en.pdf?sequence=4&isAllowed=y Rogers, Beatrice Lorge and Coates, Jennifer. 2016. Sustaining Development: A Synthesis of Results from a Four-Country Study of Sustainability and Exit Strategies among Development Food Assistance Projects—Executive Summary. Washington, DC: FHI 360/Food and Nutrition Technical Assistance III Project (FANTA). Spatial Data Repository. The Demographic and Health Surveys Program. Modeled Surfaces. MW2015_DHS_CNNUTSCHA2_MS_v01. ICF International. Funded by USAID. Available from spatialdata.dhsprogram.com. Accessed 12 April 2019 TANGO International. 2014. “End-line Survey Report CRS MALAWI WALA PROGRAM 2009-2015”. United Nations Malawi: FAO, partners team up against Fall Armyworm outbreak in Malawi. 2017. Available online at: https://mw.one.un.org/fao-partners-team-up-against-fall-armyworm-outbreak- in￾malawi/ USAID. (2013). The Resilience Agenda: Measuring Resilience in USAID. Washington, DC: USAID. Available at: https://www.usaid.gov/sites/default/files/documents/1866/Technical%20Note_Measuring%20Resilien c e%20in%20USAID_June%202013.pdf Wellness and Agriculture for Life Advancement PL480 Project. Available at: https://www.globalwaters.org/HowWeWork/Activities/wellness-and-agriculture-life-advancement- pl480- project WHO. 2018. Stunting in a nutshell. Available at: https://www.who.int/nutrition/healthygrowthproj_stunted_videos/en/ World Bank. 2016. Malawi drought 2015-2016: post-disaster needs assessment (PDNA) (English). Washington, D.C.: World Bank Group. http://documents.worldbank.org/curated/en/640011479881661626/Malawi-drought-2015-2016-post￾disaster-needs-assessment-PDNA 69 Appendix: Shocks and stresses in Southern Malawi Shocks and stresses take many forms, including meteorological, biological, human induced, technological, and market-based, and can have dramatic and long-term negative impacts on households and communities, especially in areas where subsistence farmers live. Southern Malawi has experienced a remarkable set of shocks and stresses over the past decade. In this appendix we describe the main shocks and stresses faced by WALA communities and households in the four years since WALA ended, using findings from both our qualitative and quantitative research. The main shocks and stresses experienced since WALA ended as reported by respondents in our qualitative study can be found in Table 18, by type. Table 18. Reported shocks from 2014-2018 in WALA implementation areas Type of shocks Specific shocks experienced Geophysical / meteorological Drought Flood Wind Biological Insect/pest attacks on agriculture Animal diseases External or market factors Market failure Illness or accident of household member High costs of agricultural inputs The frequency with which households reported such shocks in our survey is reflected in Figure 4. The most prevalent shocks included crop pests and droughts. 70 Figure 4. Reported frequency of shocks from 2014-2018 in WALA implementation areas and comparison areas Source: 2018 household survey. Based on data from the historical record, the main shocks to hit Southern Malawi between 2014 and 2018 by year are listed below, along with brief indicators of their severity and magnitude. 71 Figure 5. Timeline of Shocks in Southern Malawi from 2014 - 2018 Year <2013 2014 2015 2016 2017 2018 Shocks Consecutive years of dry spells Fall armyworm infestations & heavy rainstorms Record rainfall (a 1- in-500-year) & floods Strongest El Niño event in 35 years Fall armyworm infestation Fall armyworm infestation Severity and Magnitude Much of southern Malawi declared state of disaster Affected more than 1 million people and caused $335 million in damage Extensive drought and crop failure, left 39% of population in need of food aid 70% of country’s districts declared disaster areas. In 2 months, the pests destroyed crops of ~140,000 families. Fall armyworm damaged 10 percent of the maize harvest and impacted more than 1 million families. Percentage of Malawi population needing humanitarian food/cash assistance 8% 18% 40%* 2014-15 lean season 2015-16 lean season 2016-17 lean season *The highest in the southern Africa region and in Malawi’s history (Holmes et al 2017) In qualitative interviews, participants reported the dramatic severity of many of these incidents in the villages in which WALA intervened. We report on these shocks and how they were experienced below. 1. Geophysical/meteorological shocks Droughts The main form of meteorological shock experienced in WALA villages in the four years after WALA ended was drought. Droughts were reported by interview participants to have started in 2015 and were 72 still occurring in late 2018, when data collection was completed. The widespread impact of the droughts was immense. Our survey data show over half of households were affected by droughts in the WALA area over 2015-18. For many, the persistent and prolonged droughts were unusually severe. One focus group participant in the Chiradzulu district reported: Since I married here that river had never dried before…but for these past few years it is sad that the river even dries up. … It has been thirty years now [since I married into this village]. (FG11_M3) The magnitude of the impact was expressed in another focus group discussion in that village: Moderator: So you are saying everyone—each household in this community—was affected with the drought? Participant: Yes….We have about 200 something households. (FG10_F6) The droughts dried rivers and drew moisture from the soil. In addition, with no water in the rivers to use, residents with irrigation systems were often unable to irrigate. In many cases, people also depend on rivers for watering their animals—thus drying rivers also created a risk of community members losing their livestock to the drought. Floods Flooding was also widespread in the four years after WALA ended. While the droughts led to field crop failures, flooding destroyed and washed away housing structures, livestock, and additional crops. For example, in a community in Zomba, more than 25 houses were destroyed by heavy rains and flood in 2015. In Chikwawa, a focus group participant described the damage: “Flooding destroyed a lot of things. Plates, pots were washed away. Some houses were destroyed by floods.” (FG14_F9) Impacts were reported for agriculture as well. In Zomba a participant reported, “the floods washed away the soil where we plant our crops during the dry season, and it ended up in the lake; as a result, other areas are not used for farming.” (FG4_M2). In another focus group, a woman added her experience. “It started when the maize crops were producing, and you could also find fresh maize kernels. Some of the maize crops were [lodged and] covered with mud.” (FG1_F5) And in Chiradzulu a participant noted, “instead of the water just going to the river it also washed away crops.” (FG8_M6) High winds Strong winds during storms added their own destructive power during this time, destroying housing structures and crops. Heavy winds started in 2015 and recurred in 2017 and 2018 in communities in all four qualitative study districts. The heavy winds blew roofs off houses and struck down trees. Many of the trees fell on the top of homes and destroyed the structures. One focus group member noted, “out of 50 households most households’ roofs had blown off.” (FG9_F4) In summary, interview participants reported that geophysical and climate-related shocks have been re￾occurring each and every year since 2015. The participants reported that the recent droughts had been longer and more severe that in the past. Combined with flooding and windstorms, the results have been devastating. One woman in Zomba shared her story this way: “After the drought, rainfall came, and the maize that survived from the drought grew. And thereafter, the heavy winds came and destroyed the rest of the maize crops.” (FG1_F8) The damage from each shock was difficult for households to recover 73 from, and the widespread nature of these meteorological shocks made it such that neighbors were all dealing with crises at the same time. 2.Biological shocks Insect/pest attacks in agriculture Insects, pests, and diseases are major sources of crop damage in the tropics, and this was the most commonly reported shock by household survey respondents. More than 60% of all households experienced some version of crop pest or disease. Respondents in all four qualitative study districts reported that major pest attacks occurred after WALA. Some of the main pests included fall armyworms, maize stalk borers, mabobo (a type of grasshopper), and cotton mealybugs. The fall armyworms were reported to have wreaked havoc since their first occurrence in December 2016 (IFPRI 2018) and continued to destroy crops such as maize and rice. Fall armyworms attack maize once it starts to produce, eating the top of maize stalks and destroying the crop. In a focus group discussion with men in Chiradzulu the participants described how destructive these insects are and the continuing threat. Said one farmer, “they eat everything! Beans, maize...” (FG9_M) Given the scope of the infestations, government extension agents were not able to provide enough pesticides (such as Cypermethrin) for all farmers. As one farmer explained, “the government knows about this shock. We went there to inform them about the issue. They said, ‘We will do something,’ but lied to us and did not give us the pesticides.” (FG13_M) Farmers also found local treatments such as a mixture of ashes and soil ineffective. As a result, many farmers’ crops were destroyed. Livestock disease Keeping livestock is one strategy to absorb shocks in rural areas. When there are shocks, people sell livestock to obtain cash or slaughter them for consumption. Fewer than 10% of surveyed households mentioned livestock diseases affecting them in the four years after the program ended, but community members in the WALA areas did report that disease outbreaks killed livestock—one of their diversification strategies. A farmer in Chiradzulu reported that she lost all five of her goats to an unknown disease, while three other farmers lost goats to foot and mouth disease. In Machinga, a farmer lost all her goats due to rinderpest. In all four districts, respondents reported pigs, goats, cattle, and chickens were affected by diseases. While WALA reportedly helped community members obtain medicine to combat livestock disease, after the program concluded the farmers were unable to access medicine: “WALA was even bringing medicines for pigs, goats, [and] chickens, but after they left, we have nowhere to go.” (FG8_F9) 3.External or market shocks Market problems While WALA communities did not face widespread market failures, after WALA ended imperfect price information, plummeting prices, unreliable market linkages, and lack of markets for WALA-promoted crops were stresses for farmers who grew those crops. Participants who grew WALA promoted crops such as chilies and pigeon peas reported that in the four years after WALA ended, they faced markets where prices did not cover the costs of production. One member of a resilient household explained the economy of crops such as pigeon peas saying, “before, we benefited. After [WALA], [we do not.] We 74 are selling the products at lower [than production] prices.” (RH_F11)18 Several communities reported cases of companies that had agreed to buy agriculture products from WALA participants, but later reneged. An agribusiness group member in Zomba explained, “for chilies, the [production] group is no longer there due to lack of market dependability. WALA was able to identify markets for us, but the buyers were crooked. So people, after growing chilies, took to them to Nanyumbu to sell, but none were sold.” (FG1_M1). In Machinga, a farmers’ group explained that while a company bought chilies at the agreed-upon price for the first two years, in the third year that company began growing its own chili crops, and farmers were unable to find another company that would buy their chilies for more than a third of the previous price. After soybeans were introduced and grown, they were also impossible to sell after WALA left, due to a lack of markets. High costs of agricultural inputs Market challenges were not only limited to selling produce, but also obtaining inputs like fertilizers that provide precious nutrients for crops. Fertilizers are expensive to obtain on the market in general. In the quantitative surveys, 7.7% of comparison households and 10.2% of treatment households reported facing unusually high prices for agricultural inputs. The Malawian government has a coupon system for fertilizer that provides discounted or subsidized prices for farmers. However, there are a limited number of coupons distributed in each village, and thus not all farmers can benefit from this program. 18 While 3% of surveyed households in WALA villages and 4.2% of surveyed households in comparison villages reported low prices for their agricultural outputs, these reports do not focus solely on crops promoted by WALA. (The difference is not statistically significant different between the comparison and WALA villages.) Annex 1: Scope of Work (April 2018) Context The goal of the Wellness and Agriculture for Life Advancement (WALA) project was to improve the food security of 214,974 chronically food insecure households in 39 Traditional Authorities (TAs) in eight districts in southern Malawi by 2014 through various strategic activities. The project concluded in 2014 and the performance evaluation results showed positive impacts on a number of outcome indicators. The project had the following strategic objectives: ● maternal and child health and nutrition (SO1); ● agriculture, natural resource management, irrigation, and economic activity (SO2); and ● disaster risk reduction (SO3). The final evaluation report of WALA activity cites “impressive improvements in child undernutrition prevalence rates” among the beneficiaries. In addition, case studies such as the Lingoni site demonstrate that some WALA communities may have been more resilient, requiring less food aid during crises. Examining the activity outcomes three years after its completion can aid in the understanding if these and other impacts were sustained. ERIE’s long-term retrospective approach can demonstrate how the food security situation of households has changed as a result of the WALA activity, and if child nutrition gains sustained. The sustainability of outcomes after project completion is a key goal for many donors, but as many as 40% of all new projects are not sustained beyond the first few years following cessation of external funding (Bamberger and Cheema 1990; ADB 2010). Moreover, we still know relatively little about the sustainability of gains in many areas, including food security efforts. A sustainability study of the WALA project can add to this body of literature while providing evidence of the impact of the activity. WALA Theory of Change WALA aimed to sustainably improve food security and vulnerability among chronically food insecure households and communities through demonstration of optimal health and nutrition behaviors during critical periods (1,000 days); adoption of sustainable agriculture, environment and natural resource management practices; and establishment of mechanisms to mitigate and respond to shocks. Adoption of gender equitable and inclusive decision-making processes and demonstration of these behaviors and practices aimed to (a) prevent undernutrition of children under 2, (b) permit households to increase income and build assets through increased production, diversification and market linkages, increased and diversified income sources, and (c) contribute to increased risk management to improve household and community resilience to shocks and reduce natural resource degradation. 1 The WALA program worked closely with different Government departments and Ministries to get their buy-in and strengthen capacity. It also used market-based approaches and strengthened community capacity to continue many of the necessary services and infrastructures built/repaired/ improved by WALA. Sustainability of the outcomes produced by WALA would largely depend on household members’ continued access to quality services, motivation to maintain the practices; systems to facilitate the service delivery, equitable access to public and private services and inputs; incentives for and capacity of the public and private service providers to continue to provide high quality services; and, a strong demand for these services. Motivation for Long-Term Impact Evaluation USAID has expressed the following interest for the post-project evaluation: ● Evaluate the key food security and nutritional outcomes against the outcomes measured in the quantitative performance evaluation of WALA and assess whether the outcomes can be attributed to the sustainability strategies implemented by WALA. ● Assess whether WALA target communities have continued activities (e.g. practices, techniques, irrigation schemes, other infrastructures, and latrines) promoted by WALA, their evolution, and the effectiveness of linkages established during the project. Identify which activities were and were not sustained, and the factors that contributed to either status. Of particular interest is the uptake of WALA introduced interventions (e.g. practices, techniques, basic service provision, latrines, etc.) among indirect beneficiaries within intervention communities. ● Evaluate if the continuation, functionality, and the role of various community based institutions and mechanisms developed or strengthened by WALA worked as a conduit of community-level changes, capacity building of public and private service providers, and as a catalyst to promote social accountability and justice. ● Identify unintended consequences (both positive and negative), best practices, and key elements that were essential to sustaining the activities and practices. ● Understand how well the WALA communities absorbed the effects of a drought (as a result of El Nino), and adapted their strategies to maintain their food and nutrition security? Did household resilience capacities, many of which were strengthened by the WALA project, help households manage and/or recover from the drought? Based on the project objectives, USAID Washington and the mission’s interest in long term impact of WALA intervention and ERIE consortium members’ suggestions, the following research questions can be explored through an impact evaluation. Research questions 1. What was the impact of WALA on child nutrition? 2 At the end of the WALA activity, a 5.3 percentage point reduction in stunting was observed among beneficiaries. We therefore aim to understand: ● Has this lower level of stunting in beneficiary villages been maintained in the three years since the project’s completion? Has it continued to decline, stabilized, or reverted to higher levels? ● What changes in stunting do we observe in comparison villages that did not benefit from WALA over this time period? What can we conclude about WALA’s causal impacts? 2. What was the impact of WALA on resiliency? In the face of the environmental disruptions such as El Nino, are WALA communities better able to demonstrate the following: ● Improved coping mechanisms ● Maintenance of sustainable livelihoods in the face of shocks ● Continued natural resource management ● Diversification of income sources, into livelihoods with different risk factors ● Reduced need for direct food aid 3. Did improvements in resiliency due to WALA allow communities to maintain child nutrition gains even in the face of environmental disruption? Do we observe sustained child nutrition impacts even in WALA communities that experienced particularly difficult drought conditions (relative to comparison communities that also experienced these conditions)? Methods The ERIE research team will employ both quantitative and qualitative methods to answer the above research questions. Addressing questions #1 and #3 will depend primarily on quantitative analysis, while tackling question #2 will additionally make use of qualitative methods. Quantitative Analysis: Our team will collect an additional round of survey data, linked to several rounds of surveys already collected by Tango and the Demographic & Surveys (DHS), to analyze impacts on nutrition outcomes. We will: ● Select comparison villages within the same 8 districts (and potentially within the same TAs as WALA beneficiary villages) using village GPS data and baseline conditions baseline conditions from the DHS (collected in 2004 and 2010). While the DHS is not a panel at the village level, we can create modeled baselines for comparison villages in the follow-up sample using baseline data for nearby comparison villages in the earlier samples. ● Conduct one round of household survey data collection in beneficiary and comparison villages, measuring child stunting and other nutritional outcomes 3 ● Use the follow-up data to compare changes in WALA and comparison villages, statistically assessing the program’s impacts ● Merge in data on rainfall and agricultural conditions from satellite imagery, models, and in-country sources, to explore how the program's treatment effects have varied with exposure to droughts and agricultural production. Qualitative Analysis: Our team will address research question #2 through Focus Group Discussions (FGDs) and Key Informant Interviews (KIIs). Qualitative methods will be used to assess the sustainability of the outcomes of the WALA project. Particularly, the qualitative method explores which outcomes and services have continued beyond the end of the project (and why), which have not (and why not); whether the sustainability of outcomes and services can be attributed to the strategies, institutions, and systems developed by WALA to sustain these outcomes; and to identify the factors that either facilitated or inhibited sustainability. The findings should advance our understanding of how sustainability ‘happens’ – both during and after a project period – providing insights into project quality, cost-effectiveness, and sustainability of project models and approaches. USAID/FFP, USAID/Malawi, implementing partners (IPs), government officials and other key stakeholders in Malawi will use this information to inform the design of sustainability strategies in future FFP programming in Malawi and elsewhere. 4 Addendum: Tools and instruments used for data collection (see linked document) -Quantitative survey portion -Qualitative question guides