Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report Final: February 18, 2019 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by ICF Macro, Inc. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report Submitted by: ICF Macro, Inc. 530 Gaither Rd., Suite 500 Rockville, MD 20850 Contract number: GS-00F-189CA/7200AA18M00002 Prepared by: Catherine Longley, Team Leader Monica Woldt, Nutrition and Health Specialist Ina Schonberg, Food and Nutrition Security Specialist Nkwenge Priscilla Kimbugwe, WASH Specialist Ramu Bishwakarma, Lead Analyst Benita O'Colmain, Senior Survey Methods Specialist Cover photo credit: Ina Schonberg February 18, 2019 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report i CONTENTS Acronyms............................................................................................................................................................................ vi Acknowledgments.......................................................................................................................................................... viii Executive Summary ..........................................................................................................................................................ix 1. Introduction.................................................................................................................................................................1 Project Background..........................................................................................................................................1 Theory of Change ............................................................................................................................................. 2 Context ............................................................................................................................................................... 3 2. Evaluation Overview..................................................................................................................................................6 2.1 Evaluation Purpose ...........................................................................................................................................6 2.2 Evaluation Questions.......................................................................................................................................6 3. Evaluation Methods and Limitations......................................................................................................................7 3.1 Data Collection Methods................................................................................................................................7 3.2 Quantitative Data Collection Methods and Analysis: PBS ......................................................................7 3.2.1 Sampling .................................................................................................................................................7 3.2.2 Data Analysis.........................................................................................................................................8 3.3 Qualitative Data Collection Methods and Analysis..................................................................................8 3.3.1 Field Sites Visited.................................................................................................................................9 3.3.2 Field Data Collection Methods.........................................................................................................9 3.3.3 Data Analysis.......................................................................................................................................10 3.4 Limitations of the Evaluation Methodology ..............................................................................................10 4. Evaluation Findings...................................................................................................................................................12 4.1 EQ1: To what extent did the project meet its defined goals, objectives, and outcomes?............12 4.1.1 Achievement on Key Poverty and Food Security Indicators..................................................12 4.1.2 Achievement on Nutrition Indicators...........................................................................................14 4.1.3 Factors Affecting Achievement.......................................................................................................17 4.1.4 Targeting Strategies...........................................................................................................................19 4.1.5 Contribution to Reducing Food Insecurity..................................................................................20 4.2 EQ2 and EQ3. For each technical sector, what are the strengths of and challenges in the project design (including theories of change) that influence the effectiveness of the interventions? In each technical sector, what are the strengths of and challenges to the effectiveness of the interventions’ implementation?.............................................................................................................................20 4.2.1 Crop Agriculture ...............................................................................................................................20 4.2.2 Livestock..............................................................................................................................................25 4.2.3 Market Development........................................................................................................................27 4.2.4 Savings and Loans...............................................................................................................................31 4.2.5 Nutrition..............................................................................................................................................32 4.2.6 Food Distribution ..............................................................................................................................40 4.2.7 Health ...................................................................................................................................................44 4.2.8 WASH ..................................................................................................................................................48 Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report ii 4.3 EQ4. Which project outcomes are likely to be sustained? ..................................................................52 4.3.1 Adoption of Improved Smallholder Farm Management Practices .........................................52 4.3.2 Adoption of Improved Smallholder Livestock Management Practices .................................53 4.3.3 Increased Linkages to Markets.......................................................................................................54 4.3.4 Increased Access to Credit.............................................................................................................54 4.3.5 Improved Health and Nutrition Practices at the Household Level.......................................55 4.3.6 Improved Service Delivery for Prevention and Treatment of Maternal and Child Illnesses and Malnutrition ..................................................................................................................56 5. Lessons Learned, Best Practices, and Recommendations..............................................................................58 5.1 Lessons Learned and Best Practices...........................................................................................................58 5.1.1 Crop Agriculture ...............................................................................................................................58 5.1.2 Livestock..............................................................................................................................................58 5.1.3 Market Development........................................................................................................................59 5.1.4 Savings and Loans...............................................................................................................................59 5.1.5 Nutrition..............................................................................................................................................60 5.1.6 Food Distribution ..............................................................................................................................62 5.1.7 Health ...................................................................................................................................................63 5.1.8 WASH ..................................................................................................................................................64 5.2 Recommendations..........................................................................................................................................66 5.2.1 General.................................................................................................................................................66 5.2.2 Livelihoods and Food Security (SO1) ...........................................................................................66 5.2.3 Health and Nutrition (SO2)............................................................................................................67 6. References.................................................................................................................................................................69 Annexes Annex 1: RWANU Results Framework Annex 2: Evaluation Statement of Work Annex 3: Evaluation Matrix Annex 4: Consent Form Annex 5: Data Collection Instruments Annex 6: Selection of Village Sites Annex 7: Descriptions of Technical Interventions and Their Sustainability Annex 7A: Crop-Based Agricultural and Marketing Interventions Annex 7B: Livestock Interventions Annex 7C: Market Development and VSLAs Annex 7D: Household-Level Nutrition and Health Interventions Annex 7E: Food Distribution Annex 7F: Health Service Delivery Annex 7G: WASH Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report iii Annex 8: Sources of Information Annex 8A: Project Documents Reviewed Annex 8B: Fieldwork Itinerary Annex 8C: Summary of Key Informants Annex 8D: Summary of FGD and IDI Participants in Village Sites Annex 8E: Sample Household Profiles Annex 8F: PBS Indicator Results Tables for RWANU Annex 8G: Use of Sustainable Agricultural Practices Annex 8H: Multiple Regression Results, Stunting, Wasting, and Women’s Underweight Annex 8I: IPTT Tables Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report iv FIGURES Figure 1: Summarized Results Framework ..................................................................................................................1 Figure 2: Pathway to Food Security...............................................................................................................................2 TABLES Table 1: Sample Sizes for Qualitative Data Collection Methods............................................................................9 Table 2: Depth of Poverty Baseline/Endline Results at the Population Level....................................................12 Table 3: Key Indicator Baseline/Endline Results and Distribution across Per Capita Expenditure Quintiles.............................................................................................................................................................................13 Table 4: Stunting, Underweight, and Wasting in Children at Baseline/Endline.................................................14 Table 5: Women’s Underweight and Dietary Diversity at Baseline/Endline.....................................................16 Table 6: Contextual Factors Cited in Project Documentation as Inhibiting Achievement of Project Objectives..........................................................................................................................................................................17 Table 7: PBS Agricultural Indicator Results (BL-EL Comparison)........................................................................21 Table 8: Underweight among Children Ages 6–23 Months, Total and by Sex, by Year................................35 Table 9: WASH Indicators at Baseline/Endline.........................................................................................................48 Table 10: Selected WASH Indicator Trends for Southern Karamoja, 2013–2017..........................................49 Table 6.1: Villages, Sub-county Towns, and District Towns Visited by the Evaluation Team.................... 125 Table 6.2: PBS-sampled villages (sample frame) and sampling process............................................................ 125 Table 7D.1: Best Practice in MCG Compared to MCG Implementation Under the RWANU Project . 134 Table 7D.2: Positive Nutrition and Health Outcomes of RWANU Project Mentioned During FGDs and KIIs................................................................................................................................................................ 136 Table 7D.3: Underweight Among Children Ages 6–23 Months, Total and by Sex, by Year...................... 137 Table 7E.1: RWANU Ration Type and Composition .......................................................................................... 142 Table 7E.2: RWANU Food Rations Compared to USAID Preventing Malnutrition in Children Under Age 2 Approach (PM2A) Guidance............................................................................................................. 143 Table 7E.3: RWANU Food Distribution: Length of Activity Target, Achievement, and Percentage of Target Achieved............................................................................................................................................................ 143 Table 7E.4: Stunting and Wasting among Children Ages 6–59 Months in Food Security and Nutrition Assessments in Karamoja, June 2016, July 2017, January 2018, and July 2018 .............................................. 144 Table 8F.1: FFP Uganda Endline Indicators - RWANU Project Area............................................................... 160 Table 8F.2: Comparison of Baseline and Endline Indicators - ACDI/VOCA RWANU Program.............. 163 Table 8F.3: FFP Uganda Endline Indicators - Comparison Across Project Areas......................................... 166 Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report v Table 8F.4: Population and Household Characteristics in the RWANU Project Area.............................. 168 Table 8G.1: Percentage of Farmers by Type of Agricultural Practice.............................................................. 169 Table 8H.1: Results of Logistic Regression of the Prevalence of Moderate-to-Severe Stunting, ACDI RWANU [Uganda 2013, 2018]..................................................................................................................... 171 Table 8H.2: Results of Logistic Regression of the Prevalence of Moderate-to-Severe Wasting, ACDI RWANU [Uganda 2013, 2018]..................................................................................................................... 172 Table 8H.3: Results of Logistic Regression of the Prevalence of Women's Underweight (15-49 years), ACDI RWANU [Uganda 2013, 2018].......................................................................................... 173 Table 8H.4: Results of Logistic Regression of the Nutritional Status Among Children and Women by Household's Goat Ownership Status (Pooled BL and EL datasets), ACDI RWANU [Uganda 2013, 2018].................................................................................................................................................... 174 Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report vi ACRONYMS CAHW community-based animal health worker CLTS Community-Led Total Sanitation CRS Catholic Relief Services DFAP Development Food Assistance Program DFID UK Department for International Development DFSA Development Food Security Activity DHMT district health management team DHS Demographic and Health Survey DRR disaster risk reduction EQ evaluation question FAO Food and Agriculture Organization of the United Nations FEWS NET Famine Early Warning Systems Network FFP Office of Food for Peace FGD focus group discussion FSNA Food Security and Nutrition Assessment FTG farmer training group HDDS household dietary diversity score IDI in-depth interview IMAM Integrated Management of Acute Malnutrition IPTT Indicator Performance Tracking Table IR intermediate result IYCF infant and young child feeding KII key informant interview LOA life of activity MCA male change agent MCG mother care group MSC Microfinance Support Centre MUAC mid-upper arm circumference NGO nongovernmental organization NRM natural resources management ODF open defecation free PBS population-based survey Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report vii PLW pregnant and lactating women PM2A preventing malnutrition in children under the age of 2 approach PPP purchasing power parity RWANU Resiliency through Wealth, Agriculture, and Nutrition in Karamoja SACCO Savings and Credit Cooperative SO strategic objective UBOS Uganda Bureau of Statistics UNICEF United Nations Children’s Fund USAID United States Agency for International Development VHT village health team VSLA village savings and loan association WASH water, sanitation, and hygiene WFP World Food Programme WLG women’s livestock group WLIA women’s livelihood innovation award YLIA youth livelihood innovation award Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report viii ACKNOWLEDGMENTS The authors wish to thank the many individuals and groups who took the time to meet with us and share their insights about the RWANU project and Karamoja. We wish to acknowledge the invaluable inputs to both the fieldwork and the preliminary analysis by the three Research Assistants, Francesca Natee, Catherine Sagal, and Ikondere Julius Amunyo. We are also grateful to Lawrence Achia, who worked as the Field Guide, scheduling interviews with key informants and arranging for village-based discussions. Additional logistical arrangements were very efficiently made by Aidah Nakitende at IRC. We are also grateful to our drivers, Patrick, Bashil, and John. At ICF, we thank Jasbir Kaur, Benita O’Colmain, Ramu Bishwakarma, Jennifer Yourkavitch, Sujata Ram, Cindy Young-Turner, and Jo Ann Ruckel for their oversight and inputs to the report itself. Comments on earlier drafts were gratefully received from reviewers at USAID, ACDI/VOCA, and key former RWANU staff from the implementing partners. Overall guidance and support were provided by Mara Mordini and Arif Rashid. Any errors or misrepresentations are unintentional and remain the responsibility of the authors. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report ix EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS This evaluation is an external final performance evaluation of the Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) project. RWANU was one of two Food for Peace (FFP)-funded Development Food Assistance Programs (DFAPs) in Karamoja during the period 2012–2018. The purpose of the evaluation is to provide robust evidence-based findings, conclusions, and recommendations to determine the effectiveness of RWANU, influence the implementation of the two current Development Food Security Activities (DFSAs) in Karamoja, and inform and shape future United States Agency for International Development (USAID) projects. Key evaluation questions (EQs) were as follows: (1) To what extent did the project meet its defined goals, objectives and outcomes? (2) What are the strengths and challenges in the project design that influence the effectiveness of the interventions? (3) What are the strengths of and challenges to the effectiveness of the interventions’ implementation? (4) Which project outcomes are likely to be sustained? PROJECT BACKGROUND The RWANU project was a five-year, $50 million, Title II DFAP funded by USAID FFP. It was implemented in Uganda across 16 sub-counties of the four districts of Southern Karamoja (Napak, Nakapiripirit, Moroto, Amudat) from August 2012 through August 2017 by a consortium of three partners: ACDI/VOCA (lead implementer), Concern Worldwide, and Welthungerhilfe. The overall project goal was “Reduced food insecurity among vulnerable people in Southern Karamoja.” The project had two strategic objectives (SOs): SO1—Improved availability and access to food; and SO2—Reduced malnutrition in pregnant and lactating mothers and children under two. Cross-cutting issues related to gender, conflict mitigation, natural resources management, and disaster risk reduction. EVALUATION DESIGN, METHODS AND LIMITATIONS The evaluation used a mixed-methods approach. Quantitative data were collected by baseline and endline population-based surveys (PBSs) implemented in the RWANU project area in 2013 and 2018 respectively. Qualitative data were collected by the evaluation team in 2018 through (1) the review of major project documents, (2) key informant interviews (KIIs), (3) semi-structured in-depth interviews (IDIs), (4) focus group discussions (FGDs) with both participants and non-participants, (5) direct field observations in the five village sites visited, and (6) profiles of selected individual participant households (compiled through IDIs). The purpose of the qualitative component was to illustrate, triangulate, and explain the quantitative results, and gather detailed information relating to the project design and implementation, as well as the sustainability of outcomes and lessons learned. Methodological limitations included: (1) Timing of baseline and endline surveys—baseline data were collected at the beginning of the typical lean season, whereas endline data were collected at the end of the typical lean season. Results for some indicators, particularly the food security indicators and anthropometric indicators, may have been affected by this difference in timing and reflect conditions at their worst at endline. (2) Identification and sampling of project participants in the PBS— Respondents verbally confirmed whether they participated in RWANU, but no formal verification was made. The endline PBS included interviews with 1,228 households in the RWANU project area, of which 420 (35 percent) claimed to be project participants, and 808 claimed to be non-participants (65 percent). It is likely that some respondents may have denied their earlier participation in RWANU, leading to some participants being identified as non-participants in the PBS. Whether through Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report x mis-identification or sampling methods, the proportion of self-reported participants in the endline survey is relatively low. Because the PBS results are generalizable to the full population in the project area rather than just project participants the analysis included multivariate regression models in which self￾reported project participation was included to determine whether participation was a contributing factor for the outcome net of the effects of other variables. (3) Limited field sites visited—The five villages visited by the evaluation team provided an appropriate mix of agro-ecologies and project interventions, but they could not fully capture the complexity of the project area. Additional insights were gained through district-level and institutional interviews as well as secondary documented evidence. FINDINGS, CONCLUSIONS, LESSONS LEARNED, AND BEST PRACTICES Findings Conclusions Lessons & Best Practices EQ1. To what extent did the project meet its defined goals, objectives and outcomes? Endline PBS results for most indicators showed little change at population level. Positive changes greatest for households that received assistance under both SO1 and SO2. Project document reported just 11 percent of participant households received both SO1 and SO2 assistance. Challenges: leadership changes; shifts in vision/direction; inappropriateness of some approaches; the Karamoja context (recurrent drought, etc.); too many activities with too few participants, thinly spread. Clear targeting strategy was lacking for SO1. Apart from those who were also young mothers (targeted by SO2), many of the poorest and most vulnerable were excluded, e.g., elderly, disabled, ill (HIV-affected) – these groups were not targeted by USAID’s design The number of youth specifically targeted was relatively small (28 groups of approx. 12 members each). RWANU hired educated young adults as staff who benefitted from training and work experience. Overall, project targets were ambitious, and the project did not meet them. Systemic change in the Karamoja context takes time and expectations were too high. Key factors that inhibited the achievement of project objectives:  Poor project design, including the lack of a clear conceptual framework, limited scale of some activities, and limited attention to context;  Lack of strong, consistent leadership combined with contradictory and inconsistent advice;  Lack of clarity about targeting for SO1;  Challenges in targeting households for multiple types of assistance. RWANU was not designed to target the most vulnerable (as perceived by the local population). Youth were not explicitly targeted in large numbers. The Karamojong youth ‘bulge’ represents a missed opportunity. Projects require longer time frames, e.g. 7-10 years rather than 5 years. Project design should include contingency plans and budget for ‘bad’ years caused by drought and/or flooding. A clear and well-justified conceptual framework is essential in ensuring project coherence and consistency, regardless of potential changes in leadership / advisors. Clear conceptual framework can help to provide clarity in who should be targeted for different activities and ensure sufficient scale to achieve impact. Project design must be appropriate to context – this requires a good understanding of the context itself. ‘Layering’ of interventions at the household level is more likely to lead to positive changes. EQ2. For each technical sector, what are the strengths of and challenges in the project design (including theories of change that influence the effectiveness of the interventions? EQ3. In each technical sector, what are the strengths of and challenges to the effectiveness of the interventions’ implementation? General: Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report xi Findings Conclusions Lessons & Best Practices Inter-agency coordination and collaboration with government was cited by both project staff and government officials as a challenge in some sectors. Traditional and govt structures were not well utilized by the project. The selection of SO1 project participants was not done consistently and did not always involve community participation. Migration of community members reduced availability to consistently participate in various activities. Alcohol consumption is widespread. Male change agents addressed some associated problems (violence), but underlying causes of alcoholism not directly addressed. Inter-agency coordination and collaboration with government structures is essential to ensure close linkage and limit potential for duplication and working at crossed purposes. Lack of project integration with community structures inhibited impact; limited coordination with local government structures. SO1 targeting lacked clarity and community buy-in. Proactive strategies to deal with migration/movement issues were not evident in program design, which in turn impacted achievement of objectives. Insufficient attention was given to the problem of alcohol abuse. Village savings associations may have increased beer availability. Funding and mechanisms to support government-led inter￾agency coordination and collaboration essential for future projects. Ensure effective strategies to work closely with traditional and government structures. Projects require a specific strategy to deal with migration/movement issues to ensure impact intervention success. Greater attention needed to address both the underlying causes and consequences of alcohol abuse. Crop Agriculture: Recent livelihood shifts include increase in crop-based agriculture. Interventions increased agricultural knowledge and land under cultivation. PBS data showed 16 percentage point increase in uptake of improved agricultural practices, significantly correlated with participation in agricultural training. Approx. one-third of new crop varieties provided were inappropriate; some seed distributed late. Five crop varieties introduced by RWANU were still being grown. Not all farmers able to purchase seeds from agro-input dealers due to cost and distance. Veg production was included in SO1 and SO2 activities but used different approaches and targeted different farmers. Farmers located close to trading centers and access to water for irrigation were able to engage in commercial vegetable production. Farmer training, opening of new land, and introduction of new crops was relevant and effective for the uptake of improved agricultural practices. Lack of change in PBS data for crop diversity was surprising. Seed distribution / varietal selections proved problematic, though some successes in adoption of new varieties. Greater emphasis on seed saving needed. Joint design / integration of agriculture and nutrition objectives was lacking. Block farm / bulking approaches were not effective. Appropriateness of efforts to promote commercial crop production is questionable in drought- and flood-prone environment of Karamoja, with the exception of commercial veg production in some locations. Greater attention needed on crop diversification; more effort needed to work with national and international agricultural research centers as well as private seed companies; on￾farm varietal trials would allow farmers to test and identify appropriate varieties. Training in seed selection and seed preservation needed. Integration of agriculture with nutrition necessary to maximize outcomes for both sectors. Agricultural interventions must be better adapted to local context, including disaster￾prone conditions that are prevalent in Karamoja, seasonal migration patterns, socio￾economic differentiation among farmers, etc. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report xii Findings Conclusions Lessons & Best Practices Block farm approach and bulking was not successful; groups and bulking centers no longer operational. Seasonal migration (generally by the poorest farmers) was a challenge to group activities. Recurrent drought and floods as well as pests affected agricultural production throughout the project period; only one ‘good’ year. Seasonal migration and socio￾economic differentiation among farmers were not taken into consideration in design or implementation. Drought, flood and pests are widespread and recurrent, making crop production risky and unreliable. Livestock: Relatively small part of the project, despite the importance of livestock in Karamoja: livestock are more resilient to drought than crops; pastoralism is the preferred livelihood strategy; milk is highly nutritious and supports an integrated approach. Women’s livestock groups (WLGs) still active; goats have multiplied and continue to provide milk, cash and meat; goats individually owned; WLG’s savings association helps to fund animal health needs. Focus on Community Animal Health Workers (CAHWs) appropriate; no effort was made to increase demand for CAHW services beyond WLGs; link between CAHWs and WLGs continues to work well. Govt vet officers closely involved in WLGs and liaison with CAHWs. PBS data show that percentage of farmers using improved livestock management more than doubled. Livestock components of project were appropriate and successful; participant numbers too small to achieve meaningful outcomes. Jealousies arose due to limited numbers of participants /communities targeted to receive goats and oxen. WLGs adopted improved management practices, but unlikely that population-level increase in improved livestock management practices was related to RWANU due to relatively small scale of implementation. Positive gender-related outcomes were achieved by targeting women for the goats and combining this with gender awareness training, Good collaboration with sub￾county and district veterinary officers. Greater emphasis on livestock would have been appropriate. Good example of integration among project components. Savings associations for the WLGs are helpful in managing the money needed to pay for animal treatment. CAHWs training should be combined with activities to increase the demand for CAHW services (i.e. training on improved practices for livestock owners). Galla goats are appropriate and have high milk production. Best practice example in gender empowerment. Best practice example in involving local government technical specialists. Markets: Included support to 10 bulking centers, 12 horticulture groups, 24 honey groups, 102 block farms; 10 agro-input dealers; and 36 business grants (i.e., women’s livelihood innovation awards, WLIAs; youth livelihood innovation awards, YLIAs). Business training was good, but few WLIA/YLIA businesses still operational. Agro-input dealers still in business; successful dealers were Design of the market interventions lacked coherence, scale and integration Critical understanding of local market realities was missing or only developed over time. Limitations to agriculture-based market development (e.g., recurrent drought, flood, pests; remoteness and poor infrastructure) proved greater than The design of market development interventions must be appropriate to the context and must consider socio-economic differentiation. Challenges of infrastructure and transport must be taken into account; producers located close to market towns or the Kenyan border have an advantage. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report xiii Findings Conclusions Lessons & Best Practices those with expert knowledge / experience in agriculture. Documentation reported that poorer households were excluded, whereas better-off households best-placed to promote market development were not targeted. Honey purchases by honey company declined after project ended due to prohibitive transport costs. Company trained artisans in construction of hives; bee-keepers reportedly purchased them. Agribusiness Association ceased to function; leadership weak and members did not pay their contributions. Honey Association not fully established by project end. anticipated, leading to the failure of the block farms and bulking centers. Lack of sustainability of market development interventions limited their outcomes. Strong leadership and financial benefits to members are critical for continued success of both the business groups and business￾oriented associations. Access to improved hives was increased by training local artisans to make them. Coordination and advocacy among government and donor￾funded projects is necessary so that the efforts of one are not undermined by the other. Criteria for selection of individuals to become agro￾input dealers should ensure they have technical knowledge of agriculture and livestock￾keeping. Associations take time to establish and must have strong leadership and viable business plans. Savings and Loans: Consensus from all interviewee types was that Village Savings and Loans Associations (VSLAs) were the most successful and effective part of the RWANU project. VSLAs created a savings culture and allowed access to savings and loans to meet emergency needs and education costs; to invest in agricultural production; and for income generation, incl. beer brewing. PBS results showed a significant decrease in access to agricultural credit. Poor record-keeping in some VSLAs addressed by voluntary Field Agents; some former RWANU extension workers still support their groups. Greater engagement between husband and wife in household decision￾making. Gender training helped ensure that women retained control over their money. All groups met by the evaluation team continued with the VSLA activities; levels of contributions varied in some groups; some groups had reduced or suspended contributions on seasonal basis. VSLAs are effective in increasing access to credit for both consumption and productive purposes - for both men and women. Creation of savings culture is potentially transformational. PBS result is perplexing, likely due to focus on “agricultural” credit. Opportunities for income generation especially important for men due to lack of livestock. Additional support from voluntary Field Agents or extension workers both worked well. Greater spillover could have been achieved by making the specially￾designed savings box (with three locks) available through local blacksmiths and traders. VSLAs can contribute to women’s empowerment with appropriate gender training. Flexibility in the contribution amount allowed groups and individual members to contribute what was affordable / sustainable. VSLAs should be scaled up so that more people can benefit. Training should be offered to local blacksmiths so that – if feasible - affordable savings boxes can be made available. Options for income-generating activities suitable for men, women and youth should be incorporated into the VSLA approach. Encourage flexibility in contributions among individuals and according to season. Register VSLAs with sub-county offices and support VSLA awareness-creation opportunities among government officers so that VSLAs can benefit from other forms of government support. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report xiv Findings Conclusions Lessons & Best Practices Nutrition: MCG approach was appropriate for Karamoja; but areas for improvement were identified. Child stunting and wasting did not change between baseline and endline, and underweight increased significantly. Exclusive breastfeeding among children under 6 months of age improved significantly, especially for boys, but IYCF practices among children 6-23 months of age did not improve. Women’s underweight nearly doubled from baseline (27 percent) to endline (46 percent). Prevalence of diarrhea in the last two weeks increased significantly for girls under 5 years. Studies in Karamoja show that women’s high workloads; poor IYCF practices, water and sanitation, and purchasing power; and high fertility/lack of child spacing were perceived reasons for poor child nutritional status. Meal frequency among children 6–23 months of age was significantly greater in households with goats (2.8) compared to those without (2.3). Women’s dietary diversity was significantly higher in households with goats (3.1) compared to those without (2.6). WLG members valued the couples training prior to receiving the goats. Women widely regarded keyhole gardens as beneficial, though heavy labor was needed for their construction and maintenance. MCA approach resulted in positive changes in men and household dynamics; but areas for improvement were identified. The effectiveness of the IMAM Surge approach appeared to have been mixed; there is lack of data to clearly indicate its effectiveness. The MCG approach can be strengthened by: ensuring the quality and frequency of training and monthly contacts; providing adequate time for mother caregivers to absorb and apply a focused number of new practices; tailoring messages to life cycle needs; and allowing time for staff to reflect on potential adjustments to improve impact. The social and behavior change approach can be strengthened by promoting behaviors at the community level. There is a need for increased project focus on improving women’s nutritional status, decreasing women’s workload, as well as improving IYCF practices for children, water and sanitation, income generation, and child spacing/family planning. The goat distribution was popular and has good potential to improve dietary diversity, meal frequency, and child and maternal nutritional status. Keyhole gardens showed mixed results; less labor-intensive designs should be explored. The MCA approach can be expanded and improved. There is potential for the IMAM Surge approach to be effective in Karamoja; it should continue to be supported and monitored. Social and behavior change interventions implemented through MCGs should focus on a limited number of behaviors for adoption. Sufficient time is needed for review, reflection, and overcoming obstacles. Messages should be aligned with life cycle needs. Social and behavior change approaches require actions at the community level in addition to the individual level. MUAC tapes to detect and refer underweight women should be considered; special studies to better understand barriers to adoption of IYCF practices overall and for boys versus girls, and WASH and family planning practices; and monitoring women’s workload and access to income, its impact on nutrition outcomes, and necessary project adaptations. Goat distribution has good potential to improve project impact and should be considered for expansion, with attention to sanitation. Kitchen garden design must consider women’s time, labor, and resource constraints. The MCA approach should include older men as MCAs, and more training topics. MCG and MCA groups should reinforce one another. Best Practices include: MCGs as a platform; special studies into poor outcomes and uptake of recommendations; couples training to prevent and manage potential conflict. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report xv Findings Conclusions Lessons & Best Practices Food Distribution: KIIs and FGDs revealed concerns regarding participant dependency on food distribution. Some project staff estimated the SO1-SO2 overlap at household level as approximately 30 percent, rather than the targeted 75 percent. There was some perception that food distribution targeting encouraged women to become pregnant; but no data to verify this. Some mothers said that they received different rations and ration sizes than mothers in other communities, causing much resentment. The RWANU food ration generally aligned with the PM2A guidance for protein and kilocalorie intake for children 6–23 months of age but appears low for the kilocalorie and protein needs of pregnant and lactating women (PLW) and the household ration for the lean season. Sharing of food rations was common. Songs and drama effectively shared messages at food distribution sites. Food distribution was also combined with health outreach. Lack of household-level overlap across SO1 and SO2 activities encouraged dependency on food rations. Unintended negative consequences of the targeting method for food distribution may have included increased fertility or decreased child spacing; more data are needed to verify this. There is a strong local preference for the same categories of participants (e.g., pregnant women, lactating women, young children, and households) to receive the same food ration. Given the deteriorating food security situation and the extent of ration sharing, the ration should have been evaluated and adjusted to meet the kilocalorie and protein gap for PLW and young children. Interactive educational sessions and outreach services at food distribution sites were appreciated by participants. Integrated services helped participants save time and increased their access to health services. Overlap and layering of project interventions can help achieve outcomes and decrease dependence on food rations. Monitoring and quantifying potential unintended negative consequences of food rations allows for adjustments to avert negative consequences. Project participants in respective participant categories should receive the same food ration or package of services to prevent conflict. Periodic analysis of the food ration size and content is warranted during project implementation. Integration of health services and food distribution can improve participant access to health services and save time for all concerned. Interactive educational sessions may result in improved learning. Integrated outreach, food distribution and family planning are a best practice. Health: RWANU health interventions were greatly valued and contributed significantly to improved health service coverage, quality, and service use among PLW during the time of project implementation. Some health staff (4 of 13) said that the community scorecard was a best practice that empowered community members to speak freely about health issues and resulted in improved health service quality. ANC attendance significantly increased from baseline and endline. One former RWANU staff member said the project lacked adequate attention to underweight in PLW and malaria in children. The approaches used to improve health service quality, coverage, and use by pregnant women and children under two were generally appropriate and effective during project implementation. Support for outreach, equipment, and nutrition training were the most effective interventions to improve quality and coverage. The community scorecard was considered successful in increasing dialogue between community members and health facility staff. More attention is needed to support health facilities to address underweight in PLW and prevention and treatment of malaria Support for outreach, equipment provision, training, and the application of the community scorecard were essential in helping health facilities reach coverage targets and improve health service quality. Project design requires a strong health system strengthening component if an effective health systems project is not already operating in the area. Districts and health facilities need support to ensure timely and effective delivery of essential drugs and therapeutic Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report xvi Findings Conclusions Lessons & Best Practices All interviewed district health staff and one former RWANU staff member said that the project was inadequate in strengthening health systems. A 2017 assessment found some critical bottlenecks to efficient health service delivery, including drug stock-outs. Great variation was observed in WASH facilities and practices at health centers; considerable improvements are needed in some. in children, as well as strong health systems strengthening, and advocacy and support to improve health facility monitoring and logistics to ensure drug supplies. Most but not all health facilities recognized the need to maintain their WASH facilities as a model for community clients. More work is needed to ensure provision and maintenance of improved water and sanitation facilities and hygiene. food for acutely malnourished children. WASH in health facilities must be exemplary. Support for WASH in health facilities should be an integral component of project activities. WASH: A significant WASH component was not initially included in project design. The design of WASH activities was influenced by studies and assessments. Project data showed that 75.1 percent of respondents knew at least three critical moments for hand washing. PBS data showed that access to a handwashing station with soap and water decreased from 4 percent at baseline to 1.2 percent at endline. Tippy taps promoted by the project proved to be unsustainable. Affordability and distance to market were barriers to soap use. Access to traditional pit latrines increased, from 11.3 percent at baseline to 18.7 percent at endline. Cultural norms influence the use of latrines. The quality of latrines is generally poor. Latrines are commonly damaged by collapsing soils and frequent flooding. WASH facilitation was done well, but the main challenge was the limited subsequent follow-up. Insufficient efforts were made to work traditional village leaders. Incentives such as soap, hoes and axes were not shared. Endline PBS data showed that 36.9 percent of households had access to an improved drinking water source, with no statistically significant change from baseline. Systems for borehole maintenance and repair were generally weak. Formative research is necessary to develop appropriate strategies for the design of WASH interventions. The project was effective in raising awareness of WASH aspects. WASH practices improved slightly among project participants, but not to the extent needed to show any positive differences in the FFP indicators. Follow-up required for behavior change was not consistent. Tippy taps were not appropriate. The provision of subsides and incentives was not effective in the long term. The frequent need to re-build latrines leads to demotivation and fatigue. Access to improved water sources was not effective in the long term, mainly because the systems put in place for the subsequent maintenance and repair of boreholes proved to be ineffective. Recognition of the central role of WASH in Karamoja is vital in addressing health and nutrition. Innovative, appropriate, affordable, and sustainable technologies for latrines, hand cleansing agents, and handwashing facilities are needed. In the absence of more durable latrines, realistic targets should include plans for re-building latrines every few years. Gender and cultural issues need to be understood and addressed appropriately; women construct latrines, but this is gradually changing; male and female in-laws cannot use the same latrine. Behavior change requires continuous motivation and frequent reminders and follow￾up from various actors, including local chiefs. This should be included in the initial design and adequately budgeted. Capacity for the management and repair of borehole pumps needs to be strengthened. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report xvii Findings Conclusions Lessons & Best Practices EQ4. Which project outcomes are likely to be sustained? General: Local actors indicated a lack of engagement in the early part of the project and inability to continue many project activities. Exit reported by many to have been abrupt. Employing local (Karamojong) staff required considerable capacity development investments and was widely considered as a plus due to understanding of language, culture, context and contributing to local capacity development. Formal registration of groups at sub￾county offices allowed government staff to provide additional support through other projects Clear exit plan from the beginning would have improved buy-in and assignment of institutional responsibilities and capacity building – to foster shared ownership from the start. Use of local staff supported project acceptance at all levels. Staff training and their own application of knowledge for personal businesses has allowed many (especially youth) to be seen as local role models. Potential for continued support from new projects / programs if groups have been registered at sub￾county level. Exit / transitional planning should be part of core program design and engage local stakeholders from the start. Local capacity development through local staff, volunteers and partners is an important positive side-effect in building a cadre of development professionals. Best practice is to coordinate with and involve sub-county officials and to formally register groups with local government offices for continued support. Improved farm management practices adopted: Agricultural training led to a significant increase in the uptake of improved practices. Both agro-input dealers and former RWANU extension workers continue to provide advice to farmers when asked. Not all farmers can afford to purchase inputs. The knowledge that farmers gained will be retained and developed. Continued adoption of improved farm management practices is likely to be sustained only for those farmers who seek advice and/or have access to agro-input dealers or extension workers. For the adoption of improved farm management practices to be sustainable, the capacity of farmers to learn, innovate and share must be enhanced, and links to potential sources of new technologies/practices must be strengthened. Improved livestock management practices adopted: Many WLGs still active and still using CAHW services. Vet drug shop owner reported demand for vet services and that people are able to pay for drugs. General perception was that service provision would continue. Trend for increasing livestock ownership following earlier losses. Increasing demand for vet services most likely relates to increasing livestock ownership. High likelihood that outcome will be sustained, provided that other projects continue to support CAHWs and entrepreneurs continue to make quality vet drugs available more widely and with effective regulatory controls. Continued need to promote and support improved livestock management practices, service providers, and regulatory controls for animal medicines. Increased linkages to markets: Linkages created between agro-input dealers and suppliers. Contacts between farmers and traders was said to set the foundation for potential future commercial growth. Honey producers challenged by transport; most successful producers are near Kenya border. Internal and external trade is increasing due to security. Free distribution of seed by government projects presents a major challenge to private sector input markets. Networks and linkages between suppliers and traders are an important aspect of sustainable linkages to market. ‘Helping’ producers by providing transport to market does not create sustainable market linkages. Free distribution of seed and other inputs undermines efforts to promote private sector input supply system. Challenges of infrastructure and cost of transport must be taken into consideration for market linkages to be sustainable. Greater aid coordination and advocacy needed on appropriate mechanisms for agro-input supply / distribution. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report xviii Findings Conclusions Lessons & Best Practices Increased access to credit: VSLAs still active; some expanded their membership; some new VSLAs created; lack of savings boxes prevented other new VSLAs. Very few VSLAs have taken credit from Microfinance Support Centre (MSC) due to interest rate. Planned link with Savings and Credit Cooperatives (SACCOs) was not sustained due to unsustainability of SACCOs themselves. VSLAs themselves have proved to provide a sustainable source of savings and credit to VSLA members, with some spillover. Potential for greater spillover if savings boxes locally available. Linkages to formal credit suppliers was not successful in increasing access to credit. If feasible, training should be offered to local blacksmiths so that affordable savings boxes can be made available. Alternative formal institutions must be identified and established for linkage with VSLAs, and/or MSC should develop products more appropriate to VSLAs. Improved health and nutrition practices at household level: The project lacked strategies to sustain the roles of MCGs. Some VHTs were trying to support MCGs in an ad hoc manner, but there was no formal VHT role for MCGs after the project ended. Some lead mothers still met but this was because they were a VSLA. Some but not all MCGs were registered with the sub-county office. There was confusion among some MCG members and government staff regarding whether MCGs had been registered. Lead mother or VHT screening of children with MUAC tapes was the nutrition-related activity most consistently implemented after RWANU project closure. Mothers were no longer motivated to practice the behaviors learned during the RWANU project. MCAs were still active, though more data are needed to determine the extent to which this was the case. The project lacked a strong plan to sustain the roles of established community structures (e.g. MCGs) and did not effectively communicate strategies for sustainability with participants, partners, or other stakeholders. VSLAs foment community-level group cohesion and motivated some MCGs to continue meeting, but not necessarily to continue nutrition- and health-related activities. Community-level screening of children with MUAC tapes was sustainably implemented in some communities. Sustained community-level adoption of promoted nutrition and health behaviors was limited. Collaboration with partners and stakeholders is needed to design and test mechanisms to motivate and sustain community structures and behavior adoption after project closure. VHTs can potentially play a vital role in sustaining MCGs; their role should be discussed with key stakeholders from project inception. VSLAs can potentially form part of a sustainability strategy for community groups. Community-level screening of children with MUAC tapes has potential for sustainable implementation and should be supported for expansion and continued quality improvement. Improved service delivery for health and nutrition: District health staff said that the project ended abruptly and lacked an exit strategy. About half (6 of 13) health staff indicated that the RWANU equipment was still present and functional in the health facilities. There was no The project did not have a well￾defined and communicated sustainability and exit strategy. Equipment received under the RWANU project is likely to continue to be used, but lack of resources for maintenance is a constraint. A well-defined sustainability and exit strategy for health systems strengthening is essential. This must be communicated with key stakeholders at project inception, and then monitored and adjusted during implementation. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report xix Findings Conclusions Lessons & Best Practices provision for maintenance of donated equipment, such as weighing scales. There was a lack of joint supportive supervision to health facilities and outreach activities. District-level monthly coordination meetings were helpful, but the district did not follow through on action plans about half of the time. Maintaining health care coverage and quality under the current resource constraints is a challenge for the DHMTs and health facilities. DHMT ownership, commitment, and capacity to sustain project activities is limited. Senior-level project staff participation in health sector planning and priority setting can encourage local ownership. Equipment is highly valued by districts and health facilities, but maintenance of donated equipment must be ensured. Support for DHMT is necessary to improve its capacity and motivation to participate in supportive supervision and follow through with action plans. RECOMMENDATIONS General The Karamoja context presents an extremely complex and challenging context. Project design, timeframes, targets, and implementation approaches must be tailored accordingly. With under￾developed markets, low capacity of formal institutions, and changing livelihood patterns, projects require longer timeframes (e.g., 7–10 years) to achieve substantial impacts. A comprehensive conceptual framework is essential for project coherence, scale and targeting clarity, and to collaborate and communicate with partners and stakeholders. Contingency plans and budgets for “bad” years caused by drought or flooding must be built in to the project design. A close working relationship with both government and traditional structures is essential to ensure coordination, uptake of improved practices and behaviors, and long-term sustainability. Funding and mechanisms are needed to support government-led cross-agency coordination structures, as well as collaborative studies and assessments to build consensus across different agencies regarding intervention approaches. Local stakeholders must be engaged from the start in the design and planning of exit strategies, which must be developed as a part of program design. Monitoring, evaluation, and learning systems must be able to track participating households and how different household types benefit (e.g., based on participation in different project activities as well as relative wealth and food security status). Large-scale population-level outcomes should not be anticipated where local institutions, government structures, and market systems are weak or poorly developed. Livelihoods and Food Security (SO1) In the Karamoja context, the concept of resilience needs to be central to project design, implementation, and monitoring. An appropriate balance between crop-based and livestock-based interventions must be determined according to the resilience of specific livelihood strategies in relation to variations in the local agro-ecological context, as well as people’s aspirations for the future. Appropriate climate-smart agriculture approaches should be incorporated into agricultural interventions, including risk reduction, mitigation, and management strategies. A much closer integration of livelihoods, food security, and nutrition sectors is necessary. The success of the VSLAs needs to be expanded by identifying and promoting a range of diverse opportunities for income generation suitable for men, women, and youth. Male engagement (e.g. through the male change agent approach) is needed to prevent conflict over women’s increased access to income. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report xx Market systems development should adopt inclusive or pro-poor approaches and must be based on a sound understanding of socio-economic differentiation and the various constraints and risks that exist within the Karamoja context (e.g., remoteness, poor transport infrastructure, aid handouts, high-risk crop production). Health and Nutrition (SO2) It is necessary to implement social and behavior change approaches that involve both community engagement for wider community consensus and support for behavior change adoption and individual-level behavior change approaches. Promoting a limited number of top-priority behaviors and practices and doing so through cross-agency/entity campaigns will create momentum and wider uptake. Including adolescent girls as a target group for behavior change is essential for their adoption of improved health and nutrition practices. Conducting in-depth studies will assist project staff to better understand and promote effective behavior change strategies, including studies on: motivators and inhibitors to sustained activity by MCGs and MCAs after project closure; adoption of IYCF practices; long-term impacts of the goat distribution; facilitators and barriers to the adoption of family planning and WASH practices; and sharing of food rations/food from vouchers. Application of these study findings to new DFSAs can assist the projects to overcome barriers and achieve targets. The current DFSA has a unique opportunity to test and refine varied modalities (e.g., cash transfer, voucher, or other) to improve nutrient intake for vulnerable women and children, learning from and exchanging information and experiences with institutions or organizations in other countries that have tested these modalities in settings similar to Karamoja, and building on studies conducted under RWANU (e.g., barrier analysis on consumption of animal source foods). A well-defined sustainability strategy for community structures, including formal linkages to existing structures, such as the VHTs and the sub-county, in collaboration and communication with district, sub-county, and health facility staff and community leaders and elders, is necessary to motivate and sustain community structures and their functions, like the MCGs and MCAs, after project closure, as well as continued adoption of critical practices. Continued support to the DHMT and health facilities in health systems strengthening, outreach, equipment, and training is needed. DFID has a Karamoja nutrition program (2017–2021) and a malaria program (2017–2022), in addition to other health programs, such as family planning. DFID’s programs have a health system strengthening approach. USAID partners should collaborate with DFID partners to define a common approach and understanding to health systems strengthening and the roles and responsibilities of each organization. Health facilities should be supported to address underweight in PLW and prevention and treatment of malaria in children, and advocacy and support should be provided to improve health facility staffing, absenteeism, and monitoring and logistics to ensure drug and therapeutic nutrition supplies. The implementation of the community scorecard or a similar tool with proven effectiveness should be supported, along with its sustainable use by incorporation into the Ministry of Health system. Projects need to include an exit strategy for the health component in the project design, communicate it clearly with all partners and stakeholders, discuss it on a regular basis with partners, and adapt it as needed, ensuring that it includes strengthening the government system so that government staff have the capacity to conduct activities after project closure. Projects should also work closely with the DHMT to provide supportive supervision to health facilities and work to increase DHMT ownership of interventions. An intensive, coordinated, and multi-partner approach to WASH is needed in Karamoja. The Ministry of Health and local government structures require support to lead and coordinate such a Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report xxi campaign, with support from the Ministry of Water and Environment (responsible for water and sanitation in public places) and the Ministry of Education and Sport (responsible for school sanitation). The institutional level needs political, administrative, and traditional structures involved to create the enabling environment to support effective WASH coordination, implementation, and follow-up. WASH-related district ordinances and local by-laws have been effective in Uganda, promoted and enforced by sub-county officers, VHTs, and local leaders to ensure ownership, sustainability, and accountability. Lessons and best practices must be adopted for effective behavior change approaches, including follow-up and motivation by local traditional leaders. Consistent WASH messaging should be incorporated into a wide range of government sectoral programs, interventions, and institutions to create a widespread movement to promote and reinforce appropriate WASH behaviors. There is need to strengthen public-private partnerships for innovative sanitation technologies and maintenance of WASH facilities. Implementing agencies must work closely with government structures and the private sector to identify and make available appropriate, low-cost WASH technology options (e.g., cleansing agents, handwashing facilities, latrines, and borehole pumps) suitable for the Karamojong environmental and socio-cultural context, together with the necessary associated systems for effective marketing, maintenance, and repair. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 1 1. INTRODUCTION PROJECT BACKGROUND The Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) project was a five-year, $50 million, Title II Development Food Assistance Program (DFAP) funded by the United States Agency for International Development (USAID) Office of Food for Peace (FFP). It was implemented in Uganda across 16 sub-counties of the 4 districts of Southern Karamoja (Napak, Nakapiripirit, Moroto, and Amudat) from August 2012 through August 2017 by a consortium of three partners: ACDI/VOCA (lead implementer), Concern Worldwide, and Welthungerhilfe. Rwanu means “future” in the Ngakarimojong language. The overall project goal was “Reduced food insecurity among vulnerable people in Southern Karamoja.” RWANU aimed to reduce food insecurity among vulnerable people by improving access to food and reducing malnutrition, as outlined in the 2016 results framework shown in Figure 1. 1 Figure 1: Summarized Results Framework Goal: Reduced food insecurity among vulnerable people in Southern Karamoja SO1: Improved access to food for men and women IR 1.1: Improved smallholder farm management practices adopted by men and women IR 1.2: Improved smallholder livestock management practices adopted by men and women IR 1.3: Increased linkages to markets IR 1.4: Access to credit increased SO2: Reduced malnutrition in pregnant and lactating mothers and children under age two IR 2.1: Improved health and nutrition practices at the household level IR 2.2: Improved service delivery for prevention and treatment of maternal and child illnesses and malnutrition Cross-cutting: gender, conflict mitigation, natural resource management, and disaster risk reduction Key: SO=strategic objective, IR=intermediate result Source: RWANU Monitoring and Evaluation Plan (2016) Under the first strategic objective (SO1) (improved availability and access to food for men and women), interventions were to be tailored to the different livelihood zones and centered on enhancing on- and off-farm productivity through the adoption of improved practices and technologies, developing sustainable relationships between participants and public and private stakeholders, and linking smallholder farmers to profitable domestic markets. According to the project proposal, activities were to be designed to reduce the risks associated with rain-fed agriculture; identify and promote low-risk, 1 The results framework presented here added IR 1.4 to the original detailed version presented in Annex 1. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 2 higher-return commodities; and increase livestock ownership while at the same time drawing households into market systems. Many of these activities involved building the capacity of farmer groups and small producer groups through the provision of free or subsidized inputs and training in technical skills, savings mobilization, basic business skills, and marketing. The project also worked toward the establishment of a sustainable private sector animal drug supply system and an agro-input dealer network. Under SO2 (reduced malnutrition in pregnant and lactating mothers and children under 2 years of age), interventions were to be focused on curative and preventive health care, expansion of clinical health services, and improvement of household and community responses to health and nutrition challenges. Nutrition activities were to include the promotion of proper infant and young child feeding (IYCF) practices and improved consumption of micronutrients and macronutrients by pregnant and lactating women (PLW), together with emphasis on equitable intra-household food distribution. Activities were to be primarily implemented through the mother care group (MCG) approach, male change agents (MCAs), livelihoods trainings, opinion leaders, social and behavior change activities, and food distribution for all PLW and children 6–23 months of age. 2 Improved service delivery was to involve assessment of and support to Ministry of Health centers and outreach services, and the training of health workers at various levels. The cross-cutting objective focused on gender equity, conflict mitigation, natural resources management (NRM), and disaster risk reduction (DRR). Gender equity was promoted by including both men and women in project activities, facilitating women’s participation without overburdening them, and ensuring that both men and women engage in remunerative production for the market. NRM and DRR activities were to be designed to support the existing regional early warning system, introduce innovative NRM technologies, and carry out DRR activities. Conflict was to be mitigated through proportionate targeting of participant ethnic groups, stakeholder engagement, and community dialogues. THEORY OF CHANGE At the time of proposal development, most FFP DFAP results frameworks followed a standard format with relatively similar multi-sectoral components, adapted to the local context and food security profiles. The project’s vision, as presented through its “pathway to food security” (Figure 2), outlined parallel approaches for SO1 (to stabilize consumption and generate income and assets) and SO2 (on maternal child health and nutrition). Targeting was key to ensure a layering of multiple project components to increase project impact on priority households and communities. The intention stated repeatedly in project documents was to have a 75 percent overlap between participants in SO1 and SO2 activities. Figure 2: Pathway to Food Security Source: Project proposal 2 Participants living farther than 5 kilometers from a health facility received a food ration from the RWANU project, and participants living within 5 kilometers of a health facility received a World Food Program ration distributed through the health center. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 3 The proposal recognized staple crop cultivation as a high-risk, low-return activity, so the stated approach was to stabilize consumption and protect assets by reducing the inherent risks of rain-fed crop agriculture. The implicit assumption was that these risks could be reduced by crop and varietal diversification, improved production practices, and market linkages. As such, the proposal described “push” strategies (e.g., provision of productive inputs, training, provision of food for consumption by mothers and young children, infrastructure rehabilitation) and “pull” strategies (e.g., the pull of the market and returns on investments) that together create an economic “pathway out of poverty,” reinforced by concurrent maternal and child health activities. The project’s vision lacked any detail as to how these push and pull strategies would lead to the progression of the intended changes, or the assumptions that were made. As this report will show, the relative emphasis between “push” and “pull” strategies was left open to interpretation and changed over the project’s lifetime according to the influence of successive chiefs of party and advisors.3 The proposal appeared to assume that all households would benefit from the pull of the market, yet it is well known that market development does not always help the poorest and that specific efforts are needed to ensure inclusivity (e.g., Markelova and Meinzen-Dick, 2009; Campbell, n.d.). Both the proposal and the project itself failed to recognize any differences in the comparative levels of poverty among the population and their relative abilities to participate in market development. CONTEXT Karamoja is typically classified as one of the world’s poorest areas. Baseline figures in 2012 estimated that more than 95 percent of the population in Southern Karamoja were living on less than $1.25 per day. A history of conflict in the region, combined with drought, loss of cattle, and the collapse of health, education, and water services culminated in an acute emergency in 1980 when more than 50,000 people died. Since then, international agencies and the government have provided food, emergency, and development assistance at varying levels on an annual basis. It is generally acknowledged that a “culture of dependency” continues to influence Karamojong communities. Although “progress” is being made, changing attitudes and behaviors happens slowly. In the 1980s and 1990s, Karamoja continued to be affected by widespread cattle rustling, and violence was exacerbated by high levels of small arms and instability in nearby regions. The government’s disarmament campaign of 2008–2013 eventually brought peace to the region, although not without extraordinary challenges for the population. Livestock mobility was severely restricted, leading to overgrazing, disease, and widespread mortality. It is estimated that 75 percent of cattle, 68 percent of goats, and 65 percent of sheep were lost from 2008 to 2013 (Food and Agriculture Organization of the United Nations [FAO], 2014, cited by Cullis, 2017). At the time of RWANU’s project design, the Government of Uganda and USAID were supporting an effort to increase crop production in the region and were reluctant to invest in livestock, given the risk of another round of cattle disease, raiding, and general insecurity. Efforts to promote settled agriculture instead of more resilient livestock-based systems have been controversial, as in other pastoral areas of East Africa (Cullis, 2018; Famine Early Warning Systems Network [FEWS NET], 2016; Krätli, 2010; Levine, 2010). With the loss of their livestock and government support for “resettlement,” significant shifts in livelihood strategies took place. Many households were encouraged to relocate to the green belt4 to take up crop production. Since 2010, there has been a general drift toward increased reliance on crops in traditional pastoral areas,5 an increase in livestock holdings in the agricultural zone, and increasing 3 It is unfortunate that there was no mid-term evaluation as this may have been able to promote greater clarity in the overall project approach. 4 The green belt is an area suitable for mixed farming (both crops and livestock) situated in western Karamoja, stretching across parts of Napak and Nakapiripirit Districts. 5 In contrast to other predominantly pastoral groups in the Greater Horn of Africa, the Karamojong have traditionally practiced opportunistic, seasonal cropping of mainly sorghum (Cullis, 2018; Caravani, 2016). Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 4 trade due to improved access to markets and emerging employment opportunities, especially in and around towns (FEWS NET & FAO, 2013). Formal and informal wage labor and employment trends have been growing in importance, especially in peri-urban areas and areas of expanding agricultural production. Crop production provides the most common source of casual wage labor; the most important employers are farmers who cultivate land areas of about three acres or more. At peak times during the agricultural season (February–August), some men and women migrate to the more productive agricultural areas where farms tend to be larger (e.g., Namalu Sub-county, Nakapiripirit District) (Mueller & Bbosa, 2015). Migration or short-term movements create a complex scenario, as labor participation varies daily, monthly, and seasonally. Wage labor provides critical income for household needs but is proving insufficient for poorer households to re-invest in livestock, leaving them less resilient than livestock holders in the face of the highly variable rainfall that is experienced in the region (Iyer & Mosebo, 2017). Highly variable crop production and the importance of wage labor are such that more than 76 percent of households in Karamoja (and 87 percent in Moroto District) rely on markets for 50 percent or more of their food (Government of Uganda et. al., 2016). Changes in livelihoods have also led to shifts in gender relations in Karamoja; although women generally have less control than men over productive resources, they are playing an increasing role in livelihoods diversification, sometimes becoming the primary household providers (Kaari et al, 2016), including increased responsibility for crop cultivation and petty trade activities. Karamoja experiences recurrent droughts, floods, and unpredictable rain patterns. FEWS NET’s long-term climate trend analysis noted that both spring and summer rains have decreased in Uganda in the last 25 years, and that since the 1980s, temperatures have been rising.6 Karamoja’s single rainy season runs typically from February/March–October/November, with a peak in April/May, and a history of high variability. Climate models suggest that this variability will increase, along with increasing temperatures. During the project period, farm production and outputs were affected by adverse weather events.7 Some analysts emphasize the relative vulnerability of crop-based livelihoods in dryland regions compared to traditional agro-pastoral livelihood systems. Others emphasize human influences as more important risk management factors, such as conflict, livelihood strategic choices, and development planning in the face of episodic but inescapable adverse weather events. 8 South Karamoja’s soil types also pose a particular challenge, having very limited water retention capacity, cracking during the dry season and becoming waterlogged during the wet season, often resulting in flooding. These poor-quality soils, which produce low yields at the best of times, make agricultural production especially vulnerable to temperature increases and drought and flood cycles in a climate change scenario (USAID, 2017). Food assistance has been a substantial support to the region during emergency periods; in 2016, 48 percent of households in Karamoja received some level of food aid. Although continued food assistance promotes a culture of dependency, support provided through various interventions (including RWANU) contributes to greater, better quality, and more diverse food consumption, resulting in better 6 FEWS NET notes that “the magnitude of observed warming, especially since the early 1980s, is large and unprecedented within the past 110 years, representing a large (2+ standard deviation) change from the climatic norm.” (FEWS NET Climate Trend Analysis 2012) 7 In 2014, farmers experienced early excess rains (waterlogging and seedling damage), followed by extended dry spell in May and June, followed by excess rain. In 2015, the onset of the first planting season was late, and farmers in Napak lost their first crop due to drought; meanwhile, farmers waiting to plant late in the season did not as rain did not materialize. In 2016 prolonged drought and erratic rainfall (including localized flooding) continued to affect the quality and quantity of crop yield, along with a high incidence of pests and diseases. During 2017, dry spells affected lowland horticulture which fell by 60 percent, reduced honey production, reduced VSL savings, and led to an increase in population movement in search of food and pasture for livestock. ACDI-VOCA Annual Results Reports 2013-2017 8 IDS Changes in Drylands of East Africa (2016); and IRC, Karamoja Drought Risk Assessment (2011) https://www.researchgate.net/publication/275887293_Karamoja_Uganda_Drought_Risk_Assessment_Is_drought_to_Blame_fo r_Chronic_Food_Insecurity/download Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 5 food security and nutrition outcomes among those who participated in at least one development program (Government of Uganda et al., 2016). An example of the deteriorating food security situation from December 2014 to March 2016 involved a reduction in the number of children without access to animal milk; assisted by development programs, the number of children with access to animal milk increased from 30 percent to 70 percent. This illustrates the underlying fragility of the situation faced during the project period as well as the mitigating role of projects like RWANU on the unfavorable harvests, high food prices, and other stressors. (Government of Uganda et al., 2016). The large number of government initiatives, international donors, and international nongovernmental organization (NGO) projects operational in Karamoja are both complementary and sometimes overlapping, despite increased efforts to coordinate with and through district offices and NGO coordinating committees. In 2018, $75 million was tracked through 9 donors and 41 active projects, in addition to government initiatives. As a result of similar efforts in recent years, changes are being registered. Attention by the national government is reflected in an Office of the Prime Minister proceedings report indicating that development across Karamoja improved between 2012 and 2016: for example, the population below the poverty line decreased, from 75 percent to 51 percent; access to improved water supply increased, from 78 percent to 92 percent; births in health centers increased, from 27 percent to 71 percent; households owning livestock increased, from 28 percent to 54 percent; and households reporting improved living standards increased, from 78 percent to 85 percent. But in the same period, global acute malnutrition increased, from 11.7 percent to 13.8 percent, and the rate of children in school decreased, from 50 percent to 37 percent, reflecting continuing serious food insecurity in the region.9 9 Office of the Prime Minister, “Proceedings of the 9th Karamoja Policy Committee meeting and annual review of KIDP2 - Dec. 7, 2017.” https://www.karamojaresilience.org/publications/item/proceedings-of-the-9th-karamoja-policy-committee-meeting-and￾annual-review-of-kidp2-moroto-district-council-hall Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 6 2. EVALUATION OVERVIEW 2.1 EVALUATION PURPOSE This evaluation is an external final performance evaluation of the RWANU project. RWANU was the first of two FFP-funded DFAPs in Karamoja during the period 2012–2018. RWANU was implemented in four districts of Southern Karamoja (Napak, Nakapiripirit, Moroto, Amudat). The second DFAP was the Growth, Health, and Governance Program, led by Mercy Corps and implemented in the three districts of Northern Karamoja (Kaabong, Kotido, and Abim). As of November 2018, and slightly more than one year since the RWANU project closed, two new Development Food Security Activities (DFSAs) are starting up in Karamoja: the Nuyok Project led by Catholic Relief Services (CRS) in the western districts of Abim, Nakapiripirit, and Napak; and the Apolou Project led by Mercy Corps in the eastern districts of Kaabong, Kotido, Moroto, and Amudat. Each of these new activities includes parts of the former RWANU project area. The purpose of the evaluation is to provide robust, evidence-based findings, conclusions, and recommendations to determine the effectiveness of RWANU, influence the implementation of current DFSAs, and inform and shape future USAID projects. The evaluation assesses implementation and progress toward objectives. The statement of work for the evaluation is presented in Annex 2. The primary target audience of this evaluation are FFP staff in Washington, DC, and Kampala, and ACDI/VOCA and its partners, Concern Worldwide, and Welthungerhilfe. The secondary target audience includes the Government of Uganda and the implementing partners of the current DFSAs in Southern Karamoja, CRS, and Mercy Corps. Findings from the evaluation will be used by FFP in different presentations and bulletins as part of a wider dissemination of best practices and lessons learned. As such, the lessons may be used by implementing partners of other DFSAs. The evaluation recommendations may also be used by FFP to refine proposal guidelines and project policy. 2.2 EVALUATION QUESTIONS The main evaluation questions (EQs) are listed below, with the various sub-questions presented in Annex 2: EQ1. To what extent did the project meet its defined goals, objectives, and outcomes? EQ2. For each technical sector, what are the strengths of and challenges in the project design (including theories of change or TOCs) that influence the effectiveness of the interventions? EQ3. In each technical sector, what are the strengths of and challenges to the effectiveness of the interventions’ implementation? EQ4. Which project outcomes are likely to be sustained? EQ5. What are the key lessons learned and best practices that should inform future projects in the country? Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 7 3. EVALUATION METHODS AND LIMITATIONS 3.1 DATA COLLECTION METHODS The evaluation used a mixed-methods approach that included the analysis of quantitative and qualitative data to address the evaluation questions, which are summarized in the evaluation matrix in Annex 3. Quantitative data were collected by baseline and endline population-based surveys (PBSs) implemented in the RWANU project area in 2013 (baseline) and 2018 (endline), as described in Section 3.2. The quantitative data were complemented by qualitative data collected by the evaluation team in 2018. The purpose of the qualitative component was to illustrate, triangulate, and explain the quantitative results, and gather detailed information relating to the project design and implementation, as well as the sustainability of outcomes and lessons learned. 3.2 QUANTITATIVE DATA COLLECTION METHODS AND ANALYSIS: PBS The quantitative results for the performance evaluation use the data collected through a baseline and endline population-based household survey administered in all four districts where the RWANU project was implemented. The baseline PBS was conducted from February 25 to April 30, 2013 (at the beginning of the typical lean season in Karamoja, which runs from March to June), and the endline PBS was conducted from June 7 to July 6, 2018 (at the end of the typical lean season). The primary objective of the baseline PBS was to assess the status of FFP and custom indicators prior to project implementation; the primary objective of the endline PBS was to assess the change in these same indicators at the end of the DFAP project cycle by undertaking statistical tests of differences between baseline and endline measures. This pre-post design allows for the measurement of change in indicators over time at the population level. This design does not allow statements to be made about attribution or causation relating to project impact. The indicators are related to food security, poverty, women’s and children’s health and nutritional status (including anthropometry), water, sanitation and hygiene practices, agricultural practices, and gender (access to and decisions on credit, control over use of income, and group membership). Custom indicators were added by the programs to assess women’s antenatal care, access to any kind of sanitation facility, mobility in the project area, number of crops produced, use of private sector veterinary care, and use of farmer-managed natural regeneration practices. The endline PBS used the same data collection instrument as the baseline PBS.10 3.2.1 Sampling The household samples selected for the baseline and endline PBSs are based on a multi-stage clustered sample designed to power a test of differences between the baseline and endline estimates for the FFP child stunting indicator, a key indicator for FFP. The sample size is derived based on a formula to detect a difference in the prevalence of stunting from baseline to endline of 6 percentage points with 95 percent confidence and 80 percent power. Based on this calculation, the baseline PBS included 80 villages and 2,400 households, and the endline PBS included 43 villages and 1,290 households.11 These 10 The ICF team supplemented the questionnaire to include questions for indicators to be included in the baseline study for the newly funded DFSAs in the Karamoja region; however, only the 2013 baseline study modules were used to calculate the endline indicators. 11 Because the input parameters were somewhat different for the two time points, the sample size requirement for endline was updated from a baseline value of 1,557 children per project to an endline value of 988 children per project, given that this combination of sample sizes for baseline (1,577) and endline (988) ensured an overall power of 80 percent for the two time Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 8 sample size calculations included a pre-inflation factor to take into account anticipated household level non-response as well as an adjustment factor to take into account the number of households that need to be contacted to achieve the required sample size of children under 5 years of age. A detailed description of the endline sampling plan is provided in the “Uganda Endline PBS Protocol,” and a detailed description of the baseline sampling plan can be found in Annex 1 of the baseline study report.12 3.2.2 Data Analysis The baseline and endline PBS data were used to assess change over time for all FFP indicators. The data were also used, as appropriate to the evaluation question, to conduct descriptive and bivariate analyses to help strengthen findings established from the quantitative and qualitative data. Multivariate analyses on the pooled datasets (combining both the baseline and endline PBS) controlling for key socioeconomic and project-specific factors as covariates explored the factors that are associated with the prevalence of stunting and wasting among children under 5 years of age, the prevalence of underweight in women of reproductive age and daily per capita expenditures. The multivariate models controlled for several confounding factors in addition to project participation status and two different time points (baseline and endline) to empirically explore hypothesized relationships between indicators as per the project theory of change and to determine whether project participation is associated with change over time net of other factors. The designation of participating versus non-participating project households was based on household respondents’ answers to a set of questions about exposure to the project interventions. One respondent per household was asked, “Have you or someone from your household participated in RWANU project activities?” Respondents who answered “Yes” were project participants. Respondents who answered “No” were considered non-participants. Respondents who reported they or someone in their household participated in the project were asked about the type of assistance received. Project assistance could include food rations, cash transfers, nutrition training or meetings, agriculture-related training or meetings, or other types of assistance. The responses were not validated by the projects, so it was not possible to determine definitively that the respondents accurately reported the source and type of project assistance. 3.3 QUALITATIVE DATA COLLECTION METHODS AND ANALYSIS Qualitative data were collected through (1) the review of major project documents, including monitoring and assessment reports; (2) key informant interviews (KIIs); (3) semi-structured in-depth interviews (IDIs); (4) focus group discussions (FGDs) with both participants and non-participants; (5) direct field observations in the sites visited; and (6) profiles of selected individual participant households (compiled through IDIs). This section describes these approaches. The evaluation team comprised four experienced subject matter experts, encompassing expertise in rural livelihoods, food security, nutrition, public health, water, sanitation, and hygiene (WASH), and other related areas. Each team member also had considerable experience in evaluation and qualitative data collection and analysis. For the in-country fieldwork, the team was supported by a Field Guide (a points. In principle, the sample size of required children should be identical for the baseline and endline PBSs, because the computation is made at the time of the baseline to ensure that the pair of sample sizes provides an overall power of 80 percent. The actual realized sample size of children from the baseline PBS (2,668 children per program) greatly exceeded the target sample size for the baseline PBS (1,557 children per program). This implied that the overall power of the statistical test would likely be considerably greater than 80 percent had the same sample size used at baseline also been used at endline, because the power of the statistical test is determined by the actual realized sample size of children at the two time points. Therefore, it was possible to reduce the sample size at endline relative to what it was at baseline and still maintain 80 percent power overall. 12 A complete description of the baseline study is provided in the “Baseline Study for the Title II Development Food Assistance Projects in Uganda” report, available at: https://www.usaid.gov/documents/1866/uganda-baseline-study-report-march-2014 Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 9 former senior RWANU staff member) and three Field Assistants. The evaluation team was supported by ICF staff who provided quantitative data analysis, internal quality assurance, and editorial support. The initial desk review of documentation was presented in the Inception Report, and additional relevant documents were subsequently reviewed by the evaluation team, as necessary. The project documents reviewed by the evaluation team are shown in Annex 8A and include the project proposal, Annual Results Reports, Pipeline Resource Estimate Proposals, Indicator Performance Tracking Tables (IPTTs), as well as various analyses, studies, assessments, and monitoring reports. 3.3.1 Field Sites Visited The evaluation team visited five project villages, plus four associated sub-county towns and two associated district towns, as shown in the fieldwork itinerary in Annex 8B (October 6–18, 2018). Data were also collected through KIIs in Moroto (the regional capital for Karamoja), in Kampala, and through phone and internet calls. The two fieldwork districts were selected according to those with the most project participants (Napak and Nakapiripirit). In these two districts, villages were selected from among the PBS-sampled villages, stratified according to: (1) livelihood zones (agro-pastoral and agricultural), and (2) the typical range of project activities implemented (based on information provided by ACDI/VOCA). Villages that were being mapped for the ongoing CRS-led Nuyok project were excluded from the sample frame. Two villages13 (one per agro-ecological zone) were selected in Nakapiripirit District, and three villages14 (one in the agricultural zone, two in the agro-pastoral zone) were selected in Napak District. Annex 6 shows the sample frame and the selection of the village sites. In each village, visits were made to nearby gardens, WASH facilities, and, if possible, the homes of selected profile households.15 For each village and sub-county, a visit was made to the local project￾supported health facility. In sub-county and district towns, visits were made to agro-input shops that had been supported by the project. 3.3.2 Field Data Collection Methods KIIs were conducted with former implementing partner staff, stakeholders, and project participants at national (Kampala), district, sub-county, and village levels (Table 1 and Annex 8C). Most KIIs were conducted in English; translation was required for those conducted at the village level. Key informants at village level included the village chief, village health team (VHT) members, MCAs, and lead farmers. Table 1: Sample Sizes for Qualitative Data Collection Methods Key Informant Type KIIs* IDIs FGDs Remote Kampala District HQ Sub-county HQ Village USAID/FFP staff 3 Implementing partner staff 2 7 13 1 1 Government stakeholders 8 18 Private sector stakeholders 3 3 Participants 13 7 17 Non-participants 3 5 TOTAL 72 10 22 * The numbers refer to the number of key informants. See Annex 8C for additional information about the key informants interviewed. 13 Kopedur Village (Moruita Sub-county) and Natirae Village (Lolachat Sub-county) 14 Naoi Village and Loitakwa Village (Lopeei Sub-county) and Morusapir (Iriiri Sub-county) 15 In each village visted by the evaluation team, two households were selected for in-depth interviews (IDIs), as described in Section 3.3.2. The IDIs were written up as household profiles (see Annex 8E). Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 10 In each village, IDIs were conducted with representatives from two case study participant households, selected according to relative wealth and food security status: one “average” household and one vulnerable, chronically food insecure household. In most villages, the vulnerable household identified for the IDI had not directly participated in the project, so these interviews were not as in-depth as had been anticipated. 16 The purpose of the IDIs was to be able to compare the effects of the project on households of different wealth and food security status. See Annex 8E for examples of two household profiles. FGDs were conducted at village level with representatives from different types of participant groups, including farmer training groups, block farm groups, village savings and loan associations (VSLAs), 17 women’s livestock groups, lead mothers, and mothers (caregivers) who had participated in the health and nutrition training. FGDs were also conducted with non-participants. The guidelines and checklists used for the various interviews and discussions are presented in Annex 5. The additional technical questions were incorporated into the interviews and discussions, as appropriate. 3.3.3 Data Analysis Notes from each interview and FGD were typed into a standardized format and then compiled according to the EQs for formal analysis. Evaluation team members and the Field Assistants participated in two preliminary analysis workshops during the fieldwork period. The first workshop focused on the different technical sectors and the respective intervention approaches and methods, systematically identifying strengths, challenges, appropriateness, effectiveness, and sustainability in relation to each sector. Gaps and additional questions were highlighted to ensure that these were addressed in the subsequent fieldwork. The second workshop involved the triangulation of findings from different sources to identify key themes and start to draw preliminary findings and conclusions in relation to the EQs. Output from the two analysis workshops was used to draft preliminary findings presented to USAID Mission staff in Kampala at the end of the fieldwork period. Formal analysis took place after the electronic fieldnotes had been compiled according to the EQs and coded by sector and subject matter. Fieldwork findings were internally triangulated according to different methods (e.g., KIIs, IDIs, FGDs), locations, sites, and informant types, and these were subsequently triangulated and substantiated with findings from different sources (e.g., qualitative data, quantitative PBS data, IPTT data, project documentation). Kkey themes and preliminary conclusions emerging from the preliminary analysis were substantiated and elaborated with reference to the fieldnotes, quantitative data, project documentation, and other literature. 3.4 LIMITATIONS OF THE EVALUATION METHODOLOGY Timing of the evaluation. Data collection for the evaluation took place approximately one year after the closure of the project. The project ended in August 2017, quantitative data were collected in June 2018, and qualitative data were collected in October 2018. The compilation of qualitative data from the implementing partners was challenging because the project expired in 2017 and both ACDI/VOCA and Concern Worldwide had closed their in-country offices, dispersing staff and making it difficult to access data and documents and obtain information from those who had implemented the project. Planning and organizing the fieldwork was also a challenge. This limitation was mitigated by hiring a former senior RWANU staff member to assist the evaluation team as a Field Guide and in contacting former project staff, partners, and participants to contribute to the field data collection exercise. 16 Village chiefs had been asked to identify these households prior to the evaluation team’s visit. It is possible that the message conveyed to the chiefs was not understood as it had been intended, but the clear finding from the exercise was that RWANU did not specifically target those who are locally perceived to be “vulnerable.” 17 In most cases, VSLAs overlapped with farmer training groups and block farm groups. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 11 Timing of baseline and endline household surveys. Baseline data were collected over eight weeks during the months of March and April 2013, at the beginning of the typical lean season. The endline data were collected over four weeks from June 7 to July 6, 2018, during the height of the rainy season and at the end of the typical lean season. Although both timepoints were during the lean season, results for some indicators, particularly the food security indicators and anthropometric indicators, such as underweight and wasting, may be affected by this difference in timing and reflect conditions at their worst at endline. The comparison of some food security and nutrition indicators between baseline and endline must take into consideration these differences in the timing of the data collection. Identification and sampling of project participants in the PBS. Respondents verbally confirmed whether they participated in RWANU, although no formal verification was made. The endline PBS included interviews with 1,228 households in the RWANU project area, of which 420 (35 percent) claimed to be project participants, and 808 claimed to be non-participants (65 percent). Former project staff reported a fairly widespread practice of understating project participation (both previous and current) among the communities in the project area. This is related to the perception of the population that previous or current participation in a project or activity will diminish chances of being targeted and included in another project. Communities were aware of the start of the new successor USAID projects across the sub-region, and it is likely that some respondents may have denied their earlier participation in RWANU, leading to some participants being identified as non-participants in the PBS. Whether through mis-identification or sampling methods, the proportion of self-reported participants in the endline survey is relatively low (35 percent). Because the PBS results are generalizable to the full population in the project area rather than just project participants, the analysis included multivariate regression models in which self-reported project participation was included to determine whether participation was a contributing factor for the outcome net of the effects of other variables. Reference is also made to the project monitoring data (which refers to project participants only) where relevant. Limited field sites visited. It was possible to visit five village sites for the qualitative data collection in the time allowed. The RWANU project operated in an exceptionally complex environment in terms of rapidly changing contextual factors. The four districts in the RWANU project are categorized into two agro-ecological zones, but in reality, they have much greater variability. Even at the community level, there are differences in the mix and proportion of livelihood assets and corresponding strategies across different wealth groups. The five villages visited by the evaluation team provided an appropriate mix related to project design, but they did not fully capture the complexity of experience in the evolving context of localized and extended migration, economic evolution and change, security improvements, evolving livelihood strategies, and risks experienced at household and community levels. Insights were gained through district-level and institutional KIIs, but not all experiences were fully captured through direct qualitative interviews. Broader insights were obtained through secondary documented evidence. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 12 4. EVALUATION FINDINGS 4.1 EQ1: TO WHAT EXTENT DID THE PROJECT MEET ITS DEFINED GOALS, OBJECTIVES, AND OUTCOMES? 4.1.1 Achievement on Key Poverty and Food Security Indicators Three key poverty and one primary food security impact indicators for the RWANU project area were measured at the population level. Targets for the participant population were established (adjusted in some cases) and outlined in the RWANU IPTT (Annex 8I), submitted to FFP annually. Improvements were noted in the mean depth of poverty (poverty) and small improvements in per capita expenditure. There was little or no change in prevalence of poverty (low significance18 level) or mean household dietary diversity score (not statistically significant), which is a measure of food security. (See Annex 8F for detailed PBS indicator results). Overall poverty prevalence. At endline, 92.9 percent of the population was below the poverty line of USD $1.25/day (based on purchasing power parity (PPP) 2005). This decrease of 2.7 points is significant at a p-level of 0.01 (i.e., 99 percent confidence). Mean depth of poverty is a more sensitive and relevant measure of poverty representing the average shortfall in per capita expenditures below the poverty line for all individuals in the target population. At baseline, the mean depth of poverty was 66.7 percent (of the USD $1.25/day PPP 2005). It decreased to 60.9 percent at endline, a decrease of 5.8 percentage points (Table 2). Exceeding the (revised) target and representing an 8.6 percent improvement from baseline, it represented just USD $0.07 per day (or USD $25 per year).19 Note that targets in all tables refer to targets for project participants only, not the whole population.20 A statistically significant decrease of 6.5 percentage points in the mean depth of poverty was also found among households in which both male and female adults were present. There was no significant change for more vulnerable households in which there was only a female adult present (Annex 8F). Table 2: Depth of Poverty Baseline/Endline Results at the Population Level Mean Depth of Poverty (Percent of USD $1.25/day) Baseline 2013 Target Endline 2018 Difference Achieved Shortfall from poverty line implied expenditure level 66.7% ($.42) 61.9% ($.48) 60.9%* ($.49) - 5.8 pts ($.07) * Statistically significant, p<0.01 Daily per capita expenditures. Daily per capita expenditures increased by USD $0.16 over five years in the project region (Table 3). Regression analysis found that when participation variables were included, the increment in per capita expenditures between baseline and endline was diluted, indicating that non-project factors were also influential. These could include greater population mobility resulting from improved security, the impact of infrastructure development in the region, increased investment by government and a range of projects, as well as shifting livelihood strategies with a growing cash-based economy. The regression analysis showed that socio-economic factors correlated with higher expenditure included education (at secondary or higher-level), smaller household size, and households using agriculture financial services and practicing sustainable agriculture. It also showed that reported 18 In this report, “significant” refers to statistical significance, which identifies whether the survey result obtained is reliable and not due to chance. Sufficient sample sizes relative to the indicator definition determines significance. 19 Preliminary analysis indicated that the changes were greatest for those households that participated in SO1 and SO2 and food rations. 20 PBS data analysis did not allow for the disaggregation of participants and non-participants, so it was not possible to use the PBS data to determine whether the targets were achieved. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 13 project participation that correlated with higher expenditure included nutrition training or “other activities.” Disaggregation of daily per capita expenditures by expenditure quintiles of endline data provided a more nuanced picture of consumption distributions across different groups. For example, the average daily per capita expenditures among the poorest quintile was only USD $0.19 (PPP 2005), compared to USD $1.90 (PPP 2005) for the richest quintile of the population. Thus, the degree of poverty varied widely, and likely affected which households had the time and resources to take advantage of RWANU training and support, with the poorer households having fewer productive assets and less time to attend meetings and training sessions.21 Regression results showed higher per capita expenditures in Nakapiripirit and Amudat than in Napak, and lowest average expenditures in Moroto District. This finding is consistent with general observation during the qualitative survey regarding general level of economic activity and resources in the respective districts. Table 3: Key Indicator Baseline/Endline Results and Distribution across Per Capita Expenditure Quintiles Indicator Baseline 2013 Target Endline 2018 Endline Quintile 1 Endline Quintile 2 Endline Quintile 3 Endline Quintile 4 Endline Quintile 5 Daily per capita expenditure (USD) $0.52 $0.94 $0.69* $0.19 $0.36* $0.46* $0.64* $1.90* Household dietary diversity score 2.7 3.5 2.9 2.0 2.7** 3.0 3.2** 3.0** * Statistically significant ** Statistically significant compared to the poorest quintile Household dietary diversity score (HDDS). The HDDS measures household access to 12 food groups and ranges from 0 to 12. The average score was 2.7 at baseline with no statistically significant improvement at endline (Table 3). The range of scores at endline of average HDDS across expenditure quintiles provides further insight into the differences in household socio-economic status. The difference between average HDDS for households in each consumption quintile (except for the third quintile) compared to households in the poorest quintile was statistically significant. These results reflect the persistent high level of poverty in the project area and are challenging to interpret and generalize, given the variability of household profiles (including movement), livelihood strategies, and food access within communities, and are not captured by specific survey data. Conclusion—Poverty and Food Security Indicators Moderate positive changes were found for poverty indicators, to which the RWANU project was likely to have contributed. Full attribution could not be established, given the focus on a PBS, rather than extensively sampling for participants and non-participants in key activities. It is reasonable, however, to project that the large economic investment of the RWANU project in many spheres, such as health, agriculture, trainings, procurement and provisioning, transport, staff, and food assistance, contributed to increases in general expenditures (income proxy). Qualitative interview discussions indicated that a household’s food security status will typically vary from year to year, according to weather conditions, notably the presence and severity of drought or flood, and the prevalence of crop and livestock pests and diseases affecting agricultural production. Furthermore, given that cash is only a part of the Karamojong asset base, livestock-based animal and 21 A RWANU assessment found that RWANU groups established under SO1 excluded the poorest community members (Mueller & Bbosa, n.d.) Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 14 agricultural production play a significant role in direct home consumption. These may be more available to the upper two or three wealth quintiles, mirroring or increasing the per capita consumption differences by consumption quintile as shown in Table 3. 4.1.2 Achievement on Nutrition Indicators Stunting in children under 5. Survey results indicate no significant changes—either overall or by the sex of the child—in the prevalence of stunting between baseline and endline. Project targets were not met. Similar to the finding at baseline, at endline the prevalence of stunting among boys was higher than that of girls (Table 4). Overall, the prevalence of stunting continues to be very high according to criteria from the United Nations Children’s Fund (UNICEF), the World Health Organization, and the World Bank (UNICEF et al., 2018). Data from food security and nutrition assessments conducted by UNICEF and WFP in Karamoja in May of 2013 and July of 2018 also show that there was no change in the prevalence of child stunting, for example, the prevalence of stunting among children 6-59 months was 40 percent in Napak in 2013 and in 2018 (UNICEF et al 2013; UNICEF et al 2018). 22 Table 4: Stunting, Underweight, and Wasting in Children at Baseline/Endline Stunting in Children Baseline 2013 Target Endline 2018 Difference (Baseline to Endline) Stunting in Population (children under 5 years) 38.0% 28% 36.1% -1.9 pts Boys 42.6% 33% 39.4% -3.2 pts Girls 33.7% 24% 33.1% -0.6 pts Wasting in Population (children under 5 years) 11.2% 12.1% +0.9 pts Boys 12.9% 12.8% -0.1 pts Girls 9.6% 11.5% +1.9 pts Underweight in Population (children under 5 years)** 20.5% 16% 29.3% +8.8 pts Boys** 23.1% 17% 34.2% +11.1 pts Girls* 17.9% 15% 24.8% +6.9 pts * Change statistically significant: p<0.05 ** Change statistically significant: p<0.01 Multiple regression models show no association between project duration and child stunting in the RWANU project area, confirming that there was no significance difference in the prevalence of stunting between the baseline and the endline (Annex 8H). The prevalence of stunting among children in households that participated in combined interventions of food rations, agriculture, and nutrition training, in general, was not different from the prevalence of stunting among children in households that did not participate in such activities. Households that practiced the minimum sustainable agriculture practices tended to have a lower prevalence of child stunting regardless of socio-economic and geographic conditions; such households may have improved food production or income that could be used to improve children’s dietary intake. These households were also part of the VSLAs, with access to savings and loans that allow women to improve children’s dietary intake or feeding frequency without having to spend time away from the child on income-generating tasks, such as collecting firewood, which can increase workload and decrease the time available for childcare. In addition, children who experienced diarrhea in the two weeks before the survey, on average, tended to have a higher prevalence of stunting, regardless of their households’ project participation status and socio-economic. An analysis of factors that may have affected project outcomes and impact related to stunting is presented in Section 4.2.5. 22 Findings regarding the prevalence of stunting were similar for Nakapiripirit (39 percent in 2013; 40 percent in 2018) and Amudat (25 percent in 2013; 26 percent in 2018). Stunting in Moroto was 44 percent in 2013 and 38 percent in 2018. Note the food security and nutrition assessments by UNICEF and WFP measure children 6-59 months, compared to the RWANU baseline and endline, which measured children under 5 years of age. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 15 Among households with similar socio-economic conditions and project participation status, female children on average were less likely to be stunted than male children, which has been seen in analyses of anthropometric data in sub-Saharan Africa (Wamani et al., 2007). The reasons for this finding are unclear. Households with members with secondary or higher-level education tended to have lower prevalence of child stunting, compared to households with no education, net of households’ socio-economic and geographic conditions and project participation status. Wasting in children under 5 years of age. The prevalence of wasting among children under 5 years of age in the endline survey remained stubbornly high (12.1 percent) compared to the baseline (11.2 percent); this change was not statistically significant.23 Multiple regression analyses indicated no significant association between project duration and prevalence of child wasting, regardless of project participation and other indicators of socio-economic status. Children in households that had participated in the following three interventions simultaneously—received food rations and agriculture and nutrition training—tended to have a lower prevalence of wasting, compared to children in households that had not participated in all three activities together, net of households’ socio-economic status. This finding suggests that participation in all three activities resulted in long-term benefits in relation to child weight relative to height (see regression analysis results in Annex 8H). It is interesting that there were no similar findings in relation to child stunting. Factors that may have affected project effectiveness are discussed in Section 4.2.5. The prevalence of underweight for children under 5 years of age substantially increased during the project period, from 20.5 percent at baseline to 29.3 percent at endline, an increase of 8.8 percentage points (Table 4). For boys, 34.2 percent were underweight at endline, an 11.1 percentage point increase from baseline, and for girls, 24.8 percent were underweight at endline, a 6.9 percentage point increase from baseline. The increase in the prevalence of underweight is statistically significant. Underweight reflects both chronic (past) and acute (present) malnutrition, although it does not distinguish between the two. Both underweight and wasting indicators suggested deteriorating short- and medium-term nutritional status of children under 5 years of age over the course of the project. Further analysis of the findings related to underweight are discussed in Section 4.2.5. 24 It is important to note that during the endline survey there was heavy rainfall, flooding, and incidents of cholera in some areas, which, in combination with sub-optimal health care services and limited access to outreach services, would have likely increased the prevalence of diarrhea, and subsequently, the prevalence of wasting and underweight. The prevalence of diarrhea significantly increased for girls under age 5 years from baseline (23.0 percent) to endline (28.5 percent). There was no difference in treatment with oral rehydration therapy between baseline and endline overall (84.0 percent at endline), or for boys or girls. The anthropometric indicators reflect both child health status and dietary consumption. The latter requires household access to diverse foods, which, as noted above, is a challenge in the Karamoja context due primarily to poverty. In addition, baseline data were collected during the beginning of the lean season, and endline at the end of the lean season, which may have also affected the findings in terms of prevalence of wasting and underweight among young children. The prevalence of underweight among women increased dramatically during the project period to 45.5 percent for the general population, exceeding the baseline by 18.7 percentage points (Table 5). 23 A prevalence of wasting of 10–14 percent is considered “high” according to UNICEF, the World Health Organization, and the World Bank (UNICEF et al., 2018). Note that data from food security and nutrition assessments conducted by UNICEF and WFP in 2013 and 2018 also show that wasting in Karamoja was about the same, 12.5 percent among children 6-59 months of age in May of 2013 and 11 percent in July of 2018 (UNICEF et al 2013; UNICEF et al 2018). 24 Data from food security and nutrition assessments conducted by UNICEF and WFP in 2013 and 2018 show that underweight among children 6-59 months remained about the same, for example, Napak (29 percent in 2013; 28 percent in 2018), Nakapiripirit (31 percent in 2013; 29 percent in 2018); and Amudat (20 percent in 2013; 19 percent in 2018). Moroto had an underweight prevalence of 39 percent in 2013 and 30 percent in 2018. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 16 Table 5: Women’s Underweight and Dietary Diversity at Baseline/Endline Population Women Baseline 2013 Target Endline 2018 Difference Monitoring 2017 Underweight 26.8% 14% 45.5%*** +18.7 pts N/A Women’s Dietary Diversity Score (WDDS) 2.6 3.0 2.7 +0.1 pts 3.1 N/A=not available *** Statistically significant p<0.001 Large and significant increases in the average prevalence of underweight women of reproductive age strongly corroborates FEWS NET and RWANU project reports, indicating an unstable food security situation over the project period. The women’s underweight and children’s underweight data combine to create a powerful picture of nutritional stress. With these results, it is apparent that the targeted reductions were unrealistic. The reversal of progress is unusual and indicates greater vulnerability, rather than increased resiliency. The 2016 Uganda Demographic and Health Survey (DHS) also shows that 36 percent of women 15–49 years of age in Karamoja were underweight, the highest prevalence in Uganda and four times higher than the average for Uganda. It is noteworthy that during food shortages, women may go without food to ensure availability of food for other household members, including young children (Mathys et al., 2017). Multiple regression results indicate a positive association between the prevalence of women’s underweight and project duration, confirming the significant difference between the baseline and endline prevalence for women’s underweight. Women in households that participated in project activities other than food rations, nutrition training, or agriculture training tended to have a lower prevalence of underweight. This result is similar to that for child stunting and may be related to activities, such as VSLAs, that may have provided women with savings and loans that resulted in improved access to income generation or food. Women 15–20 years of age were more likely to be underweight, compared to women 21–25 years of age. In the 2016 Uganda DHS, the prevalence of underweight was also highest among women 15–19 years of age. The Uganda DHS also indicated that 24 percent of women 15–19 years of age in Karamoja have begun bearing children, and the high prevalence of underweight in this age group could be related to childbearing, but it could also relate to lack of control over resources and lower hierarchy of teenage girls in the household, compared to men and older women. Finally, women in households that practiced sustainable agricultural practices were more likely to be underweight than households that did not, net of socio-economic and project participation status. This may be a result of greater targeting of vulnerable households—or greater vulnerability of farming compared to livestock￾focused household livelihood strategies—given the high input of women’s labor in agriculture. It is interesting that households that practiced the minimum sustainable agriculture practices tended to have a lower prevalence of child stunting (see regression analysis results in Annex 8H). A further analysis of factors that may have influenced women’s underweight is presented in Section 4.2.5. Women’s dietary diversity score (WDDS, average food groups). 25 The average score of women’s dietary diversity was low and showed no statistically significant change between baseline (2.6) and endline (2.7) [Table 5]. This may be related to limited agricultural production levels, despite increases in land under production and application of at least three agricultural practices promoted by the project. The hoped-for increases in income, milk production, and crop diversity did not appear to be sufficient for this indicator to affect population targets. The percentage of women and men reporting increased 25 The Women’s Dietary Diversity Score (WDDS) is an indicator of change in the micronutrient adequacy of women’s diets, an important dimension of diet quality. WDDS is the average number of nine nutrient-rich food groups consumed by women of reproductive age (15–49 years of age) the previous day or night. The WDDS was replaced with the Minimum Dietary Diversity Indicator for Women in 2014. For more information see: http://a4nh.cgiar.org/2015/09/22/get-to-know-the-new-indicator-for￾measuring-womens-dietary-diversity/ Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 17 access to and control over income increased substantially. 26 Qualitative interviews indicated more crop diversity in kitchen and backyard gardens; however, the amounts and levels of consumption across households appeared to be very limited, and women reported having destroyed keyhole gardens in hopes of being selected for a follow-on project. Analysis also showed a lower women’s dietary diversity (2.4) in the lowest wealth quintile at endline, a statistically significant result. 4.1.3 Factors Affecting Achievement Several important external and internal factors combined in a complex way with program design, targeting, and implementation to influence the effectiveness of the interventions. Although some project outcomes were met, the impact objectives, as described in Section 4.1.1, and a variety of project outcomes, as described later in the report, were not. Local context—relatively unique physical, economic, cultural, and social features of the local context are often referred to by stakeholders as the “Karamoja context.” The project was ostensibly designed to take many of these features into account, but program design and implementation gaps arose and significantly influenced SO1, SO2, and cross-cutting aspects of resiliency and gender. These are touched upon in Section 4.2, with a few examples provided in Table 6. Weather variability was cited in RWANU reports as a primary factor in the fluctuations in production and other negative results in annual monitoring and endline survey data. FEWS NET data confirms several challenging years during which rainfall was erratic in terms of either too much, too little, or at the wrong times. Intra￾household sharing of food rations in Karamoja proved culturally difficult to avoid and was almost universal, with some inter-household sharing. Table 6: Contextual Factors Cited in Project Documentation as Inhibiting Achievement of Project Objectives Contextual Factors SO1 SO2 Cross￾cutting Varied agro-ecology of region (physical) X X Dynamics of post-conflict livelihoods (cultural-economic) X X Limited local market infrastructure (economic) X X Migration within and across sub-districts (economic, cultural) X X X Complex gender relations (cultural-social) X X X Increasing alcohol consumption (economic, cultural-social) X X X Limited traditional leadership engagement by RWANU (cultural) X X X Youth issues—exclusion, education, roles, early marriage (cultural-social) X X X Source: Project annual and quarterly reports and qualitative interviews, triangulated with other reports from Karamoja Resilience Center Section 4.2 describes the project design and related implementation challenges for each technical sector. Overarching challenges with the project design included the lack of a clearly defined conceptual framework, limited scale of some activities, and limited attention to key features of the local context (e.g., agro-ecology, weather variability, post-conflict livelihoods and infrastructure, and socio-economic differentiation). Given the complex and evolving context of Karamoja, many of the initial RWANU interventions appeared to be somewhat exploratory, and the first two years involved extensive start-up time and studies. Considerable investments were made in training and capacity development as part of a facilitative approach to help shift from short-term emergency programming approaches toward longer-term developmental approaches. Qualitative feedback revealed that the training and capacity development (among project participants as well as staff) was highly needed at the time of project initiation. 26 Although income levels did not change substantially during the project period, female control over and use of income increased from 58.5 percent to 79.2 percent, and male control increased from 53.2 percent to 83.0 percent. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 18 Cross-cutting challenges affected program implementation. A high rate of turnover in RWANU project management led to frequent changes in emphasis and priority setting, impacting project focus. Several key informants identified lack of consistent leadership—there were five chiefs of party over the project period—combined with contradictory and disruptive influences from changing ACDI-VOCA and USAID project advisors (Mission, FFP, Bureaus). In addition, scale, reach, and balance factors impacted effectiveness. Under SO2, RWANU achieved 154 percent of target of participants, with significant expansion of enrollment and the target population, initially due to adjustments to the program to include nutrition programs for children under 5 years of age rather than under 2 years of age, to address lower population density and sync with the government approach. These adjustments created additional burdens on the program, such as expanding MCG activities, and increased the amount of supervision and outreach required. A large number of discrete SO1 interventions and activities did not include the mother caregivers who participated in the SO2 activities. The project seemed to have too many activities, too thinly spread, for too short a timeframe. Systems approaches and enabling environment support were initiated in both SO1 and SO2. Aspects of these, such as mid-upper arm circumference (MUAC) screening, group registration, input dealers, community-based animal health workers (CAHWs), knowledge creation, and health service demand creation were helpful, they had limited impact and sustainability. Given the limited scope of interventions and the underlying variable agro-climatic conditions, lack of cost competitiveness, and previously low level of productive market infrastructure (e.g., limited roads, cost of transport for farmers, warehousing, limited traction and animal health services, few input suppliers, limited farmer experience), it was unrealistic to think that food access would be significantly improved through enhanced production or income generation over the two to three-year duration of the core project. Food sharing was a key reason that food assistance rations were not effective in consistently and sufficiently improving diets of PLW and children 6–23 months of age. Rations were highly valued as a resource transfer, however, and some KIIs implied that it served as an incentive for mothers to engage in training and other MCG-encouraged activities. During 2012 when RWANU was initiated, post-conflict pressures were high, livestock losses were profound, and the government encouraged sedentary livelihoods with a greater emphasis on agriculture and market-oriented approaches. RWANU’s design responded to this environment and over time tested and experimented with several approaches to strengthen livelihoods and incomes. Sources of resilience typically include social capital, financial inclusion, self-efficacy, women’s empowerment and gender equity, diversification of livelihood risk, sustainability of natural resources, and access to markets.27 Despite the inclusion of the word “resiliency” in the project title, RWANU’s focus on resilience was implicit rather than explicit. Project interventions supported actions in all these areas, and KIIs indicated important progress in access to savings and women’s empowerment. Significant effort was also placed on diversifying agricultural livelihoods and linkage to grain markets, although the latter efforts were hampered by suitability, as noted in Sections 4.2.1 and 4.2.3. Livelihood risks faced by community members in different wealth groups did not appear to be sufficiently taken into account by the project. It was evident from KIIs that strategic approaches to diversification differed considerably across wealth groups, and risk and opportunity calculations differed not only by community but by household. In hindsight, greater consideration should have been given to improving the health and productivity of livestock (both cattle and small ruminants) and to a wider enabling environment, which is important to creating sustainable and effectively integrated livelihood strategies. For participating households, VSLAs did have a significant impact on creating a savings culture 27 USAID Center for Resilience, Resilience Evidence Forum Report, April 2018. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 19 and provided a solution for urgent food, health, and education needs. VSLAs had a more selective and lesser impact on diversifying and increasing income-earning activities. Scope and scale of livelihoods interventions and numbers of participants (e.g., block farms, value chain activities, goats, VSLAs) were inadequate in scope and scale to lead to mid-level outcomes in some instances. In others, anticipated mid-level outcomes were necessary but ultimately not sufficient to achieve higher outcomes and impact. Small-scale activities had some demonstration value but no significant impact and require either more concentration of effort or greater scale. The project design failed to proactively consider the degree and differences in wealth and capacity within communities, because “everyone was considered poor.” Lack of attention to wealth and capacity differences limited program outcomes for the more vulnerable. Livelihoods strategies for Karamojong households are evolving in response to new opportunities created by greater security, greater economic investment, and incentives by government and development projects in the region. However, food security remains tenuous, and better-off households are in a stronger position to take risks and make investments. High costs of production, erratic rainfall, and challenging social conditions continue to limit progress to increasing food production and income. Short-term migration takes place for different purposes for different community members and creates considerable complexity. Limited community-level food availability and access to food during the lean season falls hardest on the poorer households. 4.1.4 Targeting Strategies The targeting approach lacked clarity and involved substantial changes in the first few years of implementation. For SO1, the methods used to identify and select farmer training group membership varied across locations and appeared to be very random; the most marginal community members, such as landless or quasi-landless households with high dependence on wage incomes, may have been excluded almost entirely (Mueller and Bbosa, n.d.). SO2 targeted PLW with children under 2 years of age. SO2 targeting was informed by a census28 that was regularly updated throughout the project lifespan. As mentioned above, the expansion of participants in SO2 to include children under 5 years of age likely created burdens on the program. The project achieved just 11 percent “overlap” by 2017, though the original target was 75 percent. Criteria for participation in various components was variable and did not explicitly support optimal impact on households with young children. Key informants revealed that monitoring data and methods used to measure the desired layering or overlap of activities were a constant source of confusion for project staff. Vulnerable groups—Although PLW and young children are vulnerable and were the vast majority of program participants (more than 140,000 enrolled in MCGs), the most food insecure (e.g., the elderly, disabled, and chronically ill) were not targeted at all by the program. Also, focus group interviews indicated that time constraints or migration for wage labor sometimes prevented the poorest community members from being registered or participating fully in activities. Youth were generally marginalized—Youth were not explicitly targeted until late in the project, 29 and KIIs in some villages reported that youth were excluded from most SO1 activities, such as agriculture and VSLAs. Some young mothers were included in MCGs but may have transitioned between households, staying home until the first or second child was born, and the bride price was paid and afterward joining the husband’s household, reducing opportunities to be consistently included in the 28 The census was undertaken in the first year of the project and was particularly time consuming, causing delays to the start of implementation activities. The census was updated on a monthly basis as newly pregnant women were identified (Annual Results Report, 2014). 29 The evaluation team was informed that youth as a demographic was not significantly emphasized by the donor until after RWANU had been awarded and implementation had started. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 20 program. Findings in the nutrition sector (Section 4.2.5) suggest that adolescent girls should have been targeted. 4.1.5 Contribution to Reducing Food Insecurity Based on the findings outlined above, the project made a limited contribution to USAID’s broader objectives for improving food and nutrition security among chronically food insecure households. The program did lay a good foundation for future efforts, however, by introducing new ideas, practices, and linkages. It served as a catalyst for engaging wide numbers of people and set them up for engagement in new ways, with selected examples of success in women’s empowerment, savings, hygiene, selected farming, and child feeding and care practices. The most wide-scale set of activities under RWANU included trainings to improve knowledge and skills, but in many instances, it was not enough to change the range of behaviors and practices necessary to achieve outcomes and sustainably improve food and nutrition security, or to substantially increase incomes. There is a real opportunity to build on RWANU’s mobilizing work and lessons learned. Systemic change in the Karamoja context takes time. For RWANU, the timeframe for impact in the transitional Karamoja context—socially and economically complex, post-conflict, and environmentally stressed—was unrealistic. Many of the project impact and outcome indicators were not well suited, especially those attempting population-level change during the relatively short project period. Expectations of impact, even on the target population, not to mention on the population base, were too high. RWANU was not designed to support the poorer households and most vulnerable community members, and trickle-down impacts were not evident. Youth are at risk in multiple ways, and although RWANU hired educated young adults as staff, included some young mothers, and experimented with a few youth grants, KIIs indicated that many youth in the communities visited did not have an avenue for participation in productive or livelihood activities. “Layering” of interventions at the household level leads to positive changes, but the project was not successful in targeting households for multiple types of assistance. A clear targeting strategy was lacking for SO1. Layering requires good coordination among project partners and stakeholders, and the internal financial, management, and information structures were not as strong as needed. The lack of effective layering hindered overall achievement. 4.2 EQ2 AND EQ3. FOR EACH TECHNICAL SECTOR, WHAT ARE THE STRENGTHS OF AND CHALLENGES IN THE PROJECT DESIGN (INCLUDING THEORIES OF CHANGE) THAT INFLUENCE THE EFFECTIVENESS OF THE INTERVENTIONS? IN EACH TECHNICAL SECTOR, WHAT ARE THE STRENGTHS OF AND CHALLENGES TO THE EFFECTIVENESS OF THE INTERVENTIONS’ IMPLEMENTATION? 4.2.1 Crop Agriculture Under SO1, farmers were trained through lead farmers and farmer training groups (FTGs), and some FTGs were subsequently selected to become block farming groups or marketing groups to encourage commercial farming. Seed was given to group members, and improved agricultural practices were promoted using demonstration plots. Lead farmers and the FTGs were supported by farm extension workers (RWANU employees)30 who also supervised the demonstration plots and trained the groups to become VSLAs. Vegetable production was promoted under both SO1 and SO2, but by different staff and using different approaches and targeting different participants. 30 Farm extension workers were primarily educated youth from the local communities who were trained by RWANU. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 21 Training for extension workers and lead farmers was highly rated (including study visits, such as agricultural shows, and World Food Day events). The FTGs, especially lead farmers, appreciated the support from the extension workers. One lead farmer who had been trained in gender issues reported that this had had an impact on current decision-making practices between husbands and wives. In all farmer FGDs conducted by the evaluation team, both men and women were present, and most female participants were clearly comfortable in asserting their views rather than automatically deferring to the men. For three out of the five village sites visited, and two out of the three extension workers interviewed, there were challenges in implementing group activities due to the seasonal migration of group members to work as casual laborers in the green belt. In at least two villages visited, some of the non-participants interviewed would have liked to join the FTG but were not present at the time of group formation.31 In one village visited, young men who were not FTG members had nevertheless learned from the group trainings and demonstration plot, suggesting that there was some spillover of agricultural training to non-participants. This spillover appears to be confirmed by the PBS results at population level (Table 7). Table 7: PBS Agricultural Indicator Results (BL-EL Comparison) Indicator Baseline 2013 Endline 2018 Raw Difference (EL–BL) Significance Level Number of Farmers BL EL Percentage of farmers who used at least three sustainable agricultural practices in the past 12 months32 19.0 35.0 16.0 *** 3,080 1,667 Male 22.9 41.2 18.3 *** 1,418 773 Female 15.5 29.8 14.3 *** 1,662 894 Percentage of farmers who used improved storage practices in the past 12 months33 52.2 47.1 -5.1 ns 2,915 1,667 Male 50.2 44.5 -5.7 ns 1,343 773 Female 54.1 49.2 -4.9 ns 1,572 894 Average number of crops produced per farmer in the past 12 months 2.5 2.0 -0.5 *** 3,075 1,662 Percentage of farmers adopting farmer-managed natural regeneration practices in the past 12 months 15.9 30.6 14.8 *** 3,061 1,667 BL=baseline, EL=endline ns=not significant, † p<0.1, * p<0.05, ** p<0.01, *** p<0.001 Source: BL and EL PBS. See also Annex 8F. 31 In one case, it was reported that these non-participants had migrated to undertake seasonal agricultural labor at the time of group formation. 32 See Annex 8G for further details on this indicator. The three sustainable agricultural practices used for this indicator refer to improved agricultural practices promoted by RWANU; not all of the sustainable agricultural practice listed in Annex 8G are necessarily “improved.” 33 The way in which the survey asked about storage practices made it impossible to distinguish traditional storage practices from improved storage practices. RWANU promoted the improvement of granaries by the addition of rat guards and metal bases, but the questionnaire simply captured “granary,” not whether it was “improved” or “traditional.” Also, the use of PICS bags could not be compared at endline because it was not included at baseline. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 22 PBS results at population level show a significant increase in male and female farmers’ use of three sustainable agricultural practices in the past 12 months. Multivariate regression analysis shows that both project duration (time) and participation in agriculture-related training variables were significantly positively associated with the use of selected improved agricultural practices promoted by the RWANU project (Annex 8G). The significant positive association between households’ participation in agriculture training and improved agricultural practice even after controlling for the effect of project duration suggests that households that participated in such trainings were indeed more likely to practice project-promoted agricultural practices than those that did not participate in such trainings. Results for specific agricultural practices at baseline and endline are shown in Annex 8G (Table 8G.1). 34 The figures show that, overall, sustainable agricultural practices were more commonly applied than NRM practices by farmers in the RWANU project area in 2018. The apparent decrease in the average number of crops produced per farmer is surprising; in all villages visited, FGDs reported an increase in both crop and varietal diversity, particularly for vegetables. However, KIIs indicated that crop planting and production outcomes varied, based on whether diverse free seed was made available (and whether the seed was appropriate and distributed on time) or whether farmers purchased seeds. It is worth noting that 2017 annual monitoring data showed that the project successfully reached its targets for promoting and applying specific technologies or management practices (Annex 8I). Of the 18,814 farmers employing RWANU technologies or practices, 8,800 farmers were reached with improved seed varieties (148 percent of target); 15,243 farmers received support for the fall army worm infestation (7309% of target); 15,423 farmers were supported in post-harvest handling and storage activities (119 percent of target); and 12,847 farmers were supported in climate mitigation efforts (117 percent of target). Activities deprioritized due to limited demand included irrigation (4 farmers, or 8 percent of target) and horticulture (4 farmers, or 3 percent of target). By comparison, only 42,859 households participated in 2017 activities, reduced from around 148,000 in prior years when full food distribution and MCGs were in operation. FTG members received seed for different crops and crop varieties (both improved and local) through free seed distributions.35 At the start of the project, RWANU partnered with the Nabuin Agricultural Research Development Institute, located in Nakapiripirit District, to identify and test appropriate crops and crop varieties36 for seed distribution; however, the partnership failed after two years due to conflicting priorities. Across the five villages visited, farmers reported that approximately one-third of the varieties of seed provided were unsuitable to local growing conditions or did not meet local preferences. The evaluation team found that approximately five varieties of crops37 introduced by RWANU were still being grown in the villages visited. Farmers and former staff also reported that seed was distributed late, especially in the early part of the project, due to lengthy approval processes. The time needed for approval was subsequently factored in the planning process, although some farmers reported that the third-year seed distribution was also late. Toward the end of the project, it was expected that farmers would purchase seed from agro-input dealers, but not all farmers visited by the evaluation team were able to do so. Some were not aware of the presence of agro-input dealers, some farmers were located too far from agro-input shops, and others could not afford to buy seed or were not convinced of the advantages of improved seed. In one FGD, 7 out of 19 farmers reported 34 There was a small increase in the percentage of farmers planting in rows, from 15.2 percent at baseline to 20.5 percent at endline; and intercropping almost doubled from 17.5 percent at baseline to 30.4 percent at endline (Table 8G.1, Annex 8G). 35 In the fourth year of the project (2016), seed vouchers were implemented through agro-input dealers, for which farmers paid 30 percent of the cost of the seed. 36 Without a long history of diversified crop production in Karamoja, it was not known which crops and varieties were most suitable to the local agro-ecological conditions. The Nabuin Agricultural Research Development Institute carried out on-station and on-farm agronomy trials; planned trials for integrated pest management were not implemented. 37 Identification of the varieties was not verified, but these were thought to be serenut (groundnut), Longe 5 (maize), serena (sorghum), a local bean variety, and a local cowpea variety. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 23 purchasing seed from agro-input dealers, but this was exclusively vegetable seed that was not available from other sources.38 Farmers felt that the seed supplied by agro-input dealers is expensive and for this reason they prefer to purchase seed from grain traders.39 Horticulture was promoted among horticulture production groups, focusing on training in nursery bed management, pest and disease control, and drip irrigation. It was also promoted among MCGs, primarily through the introduction of keyhole gardens.40 Feedback from interviews and FGDs suggested that the nursery beds were appropriate, but not drip irrigation. Feedback about the keyhole gardens was mixed; in some places, they were considered to be too labor-intensive to be viable, and in other locations, several farmers were still maintaining them. The keyhole gardens were intended for seasonal vegetable production for household consumption only, not for large-scale production. Where they were still in use, women appreciated the convenience of having a source of vegetables close to the home. Some women reported planting a wider variety of vegetables in their plots after being introduced to them with the keyhole gardens. RWANU supported selected FTGs to establish block farms of 10 acres to promote commercial production and improved practices.41 Some groups42 were given a pair of oxen and a plow, and others used hired tractors43 to open up new farm land. Block farmers contributed 30 percent of the cost of inputs, such as tractor hire and seed. Comparison with the baseline data (Table 8G.1, Annex 8G) shows that the use of ox plows roughly doubled, from 23.1 percent at baseline to 41.1 percent at endline. The use of tractors remained low at 3.2 percent, most likely due to the lack of tractors available and the cost of hiring them. Evidence from the farmer FGDs and extension workers and project documentation confirm that the block farm approach was generally not successful. The size of the block farm was considered to be too small for a group-based approach, production was generally too low to warrant group storage and marketing,44 and farmers did not like working as a group.45 None of the five block farm groups met by the evaluation team had continued to work as a farming group. The provision of tractors and ox plowing was appreciated because it increased the land available for farming, but this land had since reverted to individual use. Among the villages visited, one FGD reported that the oxen had been purchased from the group by an individual farmer in the village; another FGD reported that the group retained ownership, but one person took responsibility for caring for the oxen, and the money earned from ox plow services and spent on the cost of care was managed by the VSLA comprised of the former block farm members. The only report of an existing block farm group was in Karita, where markets are better developed due to their proximity to Kenya. 38 Vegetable seeds (e.g., onion, kale, carrot, tomato, eggplant) are notoriously difficult to harvest and save from one year to the next and are commonly purchased by African farmers. 39 When asked about quality, the farmers replied that they did not see any difference between the quality of seed from grain traders and that from agro-input dealers. 40 A keyhole garden is a circular, raised garden with a section cut out for access to a composting basket in the middle in which everyday kitchen and garden waste is composted. More than 10,000 keyhole gardens had been established through the MCG platform by 2015 (ARR 2015). The design of the keyhole garden observed by the evaluation team had been modified in that it did not have the keyhole (i.e., the cut-out section and central composting basket); it was essentially a circular raised bed located near the house. 41 An additional purpose mentioned in one report was to test whether farmers could work together. 42 Sixty groups from 2015 and 42 groups from 2014 43 Due to the lack of availability of tractors in Karamoja Region, RWANU had to hire tractors from Soroti and Mbale. It was reported that entrepreneurs in Karamoja are now starting to buy tractors to rent out to farmers. 44 Recurrent drought and floods affected farming throughout the project: during the project period, good harvests occurred only in 2014 and in 2016 in some areas; beans were badly affected by pests in 2014, and maize was affected by fall army worm in 2016. 45 The 2016 Annual Results Report noted conflict in block farm groups due to uneven division of labor and sharing of tasks. Other challenges included farmers needing to manage both individual farms and block farms, with farmers prioritizing individual farms. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 24 Conclusions Overall, it appears that the project used a blueprint design in the agricultural sector without sufficiently tailoring it to the specificities of the local context (e.g., drought- and flood-prone environment in which crop production is highly unreliable, seasonal migration for farming and farm labor) and without sufficient integration in the design between agriculture and nutrition. More could have been done to integrate the crop-based agriculture and nutrition approaches (e.g., through targeting and the ways in which vegetables were promoted). Given the changing livelihoods and gender dynamic in Karamoja and the relatively recent increase in the uptake of crop agriculture (as described in Section 1.3), the link with agricultural research and the need to train farmers in crop production are justified, as is the integration of gender issues into the agricultural training. Greater efforts should have been made to support and work with the Nabuin Agricultural Research Development Institute and other agricultural research organizations. Farmers gained considerable knowledge through the FTGs and lead farmers, and the 16 percentage point increase in the uptake of improved agricultural practices was significantly correlated with participation in agricultural training. The agricultural training approach through the FTG/lead farmer/extension worker model was appropriate, but challenges were noted due to the seasonal migration of farmers. One unforeseen positive consequence of the agricultural training was that some of the extension workers, notably those located in the more productive areas, were able to apply what they had learned and invest their salaries to establish successful agriculture-based enterprises in the green belt, providing a role model to local youth. This path had been encouraged by one of the former management-level staff. Crop diversification and especially vegetable production was appropriate, and—based on the feedback from farmers—many of the agronomic, pest management, post-harvest, and processing and preparation practices promoted were appropriate. The provision of tractors and ox plowing was effective in opening up new land for farming; private entrepreneurs are now making tractors available in Karamoja. Seed distribution and varietal selections proved problematic due to late seed delivery and inappropriate varieties in some cases; however, the fact that farmers are still growing some of the varieties that were provided shows some success. Many farmers cannot afford to purchase improved seed or do not have an agro-input shop close enough; greater emphasis should have been placed on seed saving by farmers. Among the cross-cutting issues, gender was well-integrated into the farmer training approach and contributed to changes at the household level. The effects of NRM and DRR were more limited, with considerably less uptake of NRM practices. Given the recurrent nature of natural disasters, more could have been done to promote NRM and DRR, such as climate-smart agriculture. A RWANU study on climate-smart agriculture was undertaken in 2017, but this was too late to have any impact on implementation.46 Despite the increased uptake of crop-based production, the poor soils, recurrent drought and floods, and frequent pest and disease infestation, together with a forecast for increased incidence of extreme weather (USAID, 2017), are such that the potential for reliable commercial crop production is limited and restricted mainly to the green belt zone. Even in the green belt zone, commercial crop production is highly risky and is suitable for a relatively few better-off farmers. The project failed to take account of socio-economic differentiation among farmers. Agriculture-based economic growth has been described as “incongruent with the local context” (FEWS NET, 2016: 2). The promotion of commercial agriculture (e.g., through block farms) in agro-pastoral areas was inappropriate. The block farming approach was also not successful, mainly due to the group-based approach and low levels of agricultural production in Karamoja. Crop production is regularly affected by recurrent drought, floods, and pests; these are normal, rather than unusual, events. 46 The study debriefing suggests that one of the reasons for undertaking the study was to identify lessons for a potential RWANU follow-on project (i.e., to inform the proposal-writing process). Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 25 4.2.2 Livestock Activities included under the livestock sector refer mainly to goat distribution and support to 211 women’s livestock groups (WLGs) and CAHW training. In addition to these two main activities, a pair of work oxen for plowing was given to each of 102 block farming groups, and training was provided for beekeeping and honey production. Traditionally the people in the Karamoja region are wild honey gatherers (Ondoga, 2010),47 suggesting that beekeeping was appropriate, although this component of the project was relatively small, with just 24 groups established, each with approximately 20 members. None of the FTGs interviewed mentioned any livestock-related training, suggesting that the focus of the FTGs was on crop-based agriculture, although one of the government officers interviewed reported that FTGs were taught about fodder production. Compared to the number of FTGs, thought to total approximately 650 groups, the overall number of WLGs (211) was much smaller. Despite this, PBS results for the adoption of sustainable livestock practices show considerable increases, with at least a doubling of the percentage of farmers using animal shelters, vaccinations and deworming, and purchasing animal medicines48 (Annex 8G). It is not known whether these increases are linked to the RWANU project, but this seems unlikely, given the relatively small numbers of livestock groups and CAHWs supported and the apparent focus of the FTGs on crop￾based agriculture rather than livestock-based practices. It is far more likely that the population-level increase in these practices emerged out of the experience of widespread livestock losses due to the spread of disease that resulted from having to keep livestock in the protected kraals during the disarmament campaign (Cullis, 2018). Livestock was reported to be a very small component of the project.49 This in itself is noteworthy, given that the cultural and social values of the people of Karamoja are predominantly pastoral, for men in particular, and livestock are more resilient to drought than crops. There were large-scale livestock losses as a result of conflict and the disarmament campaign, and because cattle-raiding was one of the main sources of conflict, agencies were reluctant to promote restocking in the period immediately following disarmament due to the fear that more livestock might lead to new raids. Small stock, such as goats, however, were deemed appropriate, particularly because animal-sourced foods, notably milk and meat, are an important aspect of nutrition and allow for a more integrated project approach (i.e., contributing to intermediate results [IRs] 1.2 and 2.1). As such, the goat component of the project was highly relevant, as confirmed by participants who noted increased household nutrition (see also Section 4.2.5) and financial resources to cover essential household and emergency needs. WLG members were selected from among the mother caregivers who were also supported under SO2 (i.e., pregnant and lactating women with young children). All government veterinarians and livestock specialists interviewed agreed that the focus on CAHWs was appropriate, and this also built on earlier development efforts by other agencies. RWANU supported a total of 134 CAHWs (RWANU Annual Survey, 2017), selected from among existing CAHWs who had already received some training and were based outside main trading centers. Both WLG members and CAHWs reported that the link between CAHWs and WLGs worked well to promote the health and productivity of goats. The implementing partner reported that much more could have been done to link 47 Ondoga, J.J. (2010 August). Opportunities for alternative livelihoods in Karamoja. Report to FAO. http://www.fao.org/fileadmin/user_upload/drought/docs/1_Opportunities%20for%20Alternative%20Livelihoods%20in%20Karam oja.pdf (accessed October 9, 2018) 48 Unfortunately, it is not known whether these purchases were from the CAHWs, the agro-vet shops supported by the project, or from informal market traders. 49 The key informant reported that the livestock component amounted to just 3 percent of the total budget. No financial information was made available to the evaluation team, so it is not possible to provide any verified figures on the budget or expenditure for livestock-related activities. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 26 with other projects (e.g., rebuilding herds, rangeland management, and CAHW coordination) and ensure animal well-being (e.g., vaccination infrastructure and fodder production). The main positive unforeseen pathway emerging from the livestock sector for both goats and beekeeping relates to gender dynamics and women’s empowerment. A former RWANU staff member found that women turned out to be the best beekeepers, an activity previously only undertaken by men, proving that women could keep bees successfully. Although women in Karamoja traditionally have not owned livestock, the WLGs allowed them to own goats and benefit not only nutritionally but also financially. The sale of young goats by WLG members provided an important source of income that allowed for medical treatment for household members, postnatal care, and purchase of food when needed. Not only did this support both SO1 and SO2, but it also empowered women. Another unforeseen pathway stemmed from the exchange visits that took place as part of the beekeeping training, which supported peaceful relationships among groups that had previously been enemies. On the negative side, the cost of goats and the small budget allocation was such that the benefits were limited to relatively few women, and there was dissatisfaction among many communities that RWANU did not provide the same inputs to all target villages, as evident in all villages visited by the evaluation team. It was also reported that men were marrying new wives and married women were stopping their family planning in the hope of getting pregnant and qualifying for a goat. Although the evaluation team heard many anecdotes from different sources,50 there is no firm evidence to prove that this was the case. There was a one-year delay in starting up the goat/WLG component due to a decision to hold all activities until the population census51 had been completed, which took three months longer than expected, plus the need for special approval on the animal medicines required for preventative treatments,52 as well as an outbreak of foot and mouth disease. The census was used to identify villages with the highest numbers of PLWs and the individuals in villages who qualified for goats (i.e., those with young children). Small groups of 10–15 members were formed, and materials for goat shelter construction were provided, as well as fodder tree seedlings. The project provided goats to 2,242 women in 211 groups (RWANU Annual Review Report, 2017). Five young female goats53 were given to each member; individual ownership increased the sense of responsibility (Lepillez, 2016). In addition, each group received one buck for breeding. Local does, Galla does, Galla bucks, and Toggenburg cross-bred bucks were distributed.54 Women were trained in herd management, buying and marketing of goats, planning and budgeting, and group savings and credit. Each WLG was linked to a local CAHW who had been trained by the project for health monitoring and veterinary services. The members of the WLG met by the evaluation team reported that they still use the services of the CAHW and that they contribute to buy medicines as a group to treat any sick animals. Each of the eight women in the FGD had successfully cared for her goats and, after four years, each now had between 7 and 12 goats (average 9.5), having sold some of the young goats to meet basic and emergency needs. The WLGs were successful in changing norms around goat ownership and increasing women’s ability to 50 This was also reported by Hopwood et al. (2017) in relation to World Food Programme food aid, which targeted PLW. 51 The household census undertaken for all targeted project villages to identify participants for SO2 activities in the project area was not in the original Implementation Plan; although costly, it was useful in forming groups and identifying participants. 52 Because animals must be given preventative treatment before distribution, the delay in the approval of the drugs led to a delay in the procurement of the goats. 53 Five goats were economically viable for milk production; previous experience had found that two goats was not enough. The goats provided were quite young and had to be looked after for about a year before they started kidding and producing milk. A goat kids once a year and produces milk for about one month or slightly more after each kid. The evaluation team found that group members tended to pair up and share milk between them when one woman’s goats had milk but the other woman’s goats did not. By sharing the milk from 10 goats, the pair could each have milk for about 6–7 months of the year. 54 Galla goats were introduced due to low adaptability of Toggenburg crosses bucks in the drier areas; 40 percent of kids born to Toggenburg crosses bucks died, compared to 4 percent of kids born to Galla bucks. Galla are a high milk yielding breed common in northern Kenya and southern Ethiopia (Annual Results Report 2014). Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 27 own goats (Lepillez, 2016); the women interviewed by the evaluation team stated that this was made possible by the gender training provided by the project, which included training for men. The training and inputs were appreciated by the CAHWs and government veterinary officers alike. All five of the veterinary officers interviewed by the evaluation team both at sub-county and district levels reported that they were closely involved in the project through links with the CAHWs and the WLGs, training courses, and quality assurance. The RWANU field officers responsible for livestock usually met with the government veterinary officer at the sub-county level whenever they made field visits; this was appreciated by the sub-county officers because it helped them keep abreast of development activities for overall coordination purposes. Under the beekeeping component, 24 groups were trained, and individual group members were supplied with improved beekeeping equipment. Where possible, the residential training sessions included one day of gender training and one day of conflict training. Toward the end of the project, efforts were made to establish a beekeepers’ association to support ongoing training and marketing needs, but it was not possible to complete the training required for the association members. Conclusions Despite initial problems with breed selection, the goat component appears to have been very successful for the relatively small number of participants in that household financial resources increased and nutrition improved. The gender element of both the goat groups and beekeeping was successful in proving that women can own goats and keep bees and have control over decision-making and finances. The goat component had good involvement from the sub-county and district veterinary officers, and the link with CAHWs worked well. WLG members appear to have adopted improved livestock management practices. PBS data show that the percentage of farmers using key improved livestock management practices more than doubled, but it is unlikely that this population-level change was related to RWANU because of the relatively small scale of implementation of the livestock component. The strength of the goat component was its relevance to the local context and also to both IR 1.2 and IR 2.1. The main challenge was the limited budget in relation to the cost of goats and the limited numbers of participants, resulting in jealousies both within villages and between villages, as well as the limited activities implemented. Support to the CAHWs may have increased the availability and quality of improved livestock services, but little was done to promote the demand for such services beyond the WLGs. For the cross-cutting objectives, positive gender-related outcomes were achieved by targeting women for goat ownership and beekeeping, and beekeeping training contributed to conflict mitigation between previously warring groups. 4.2.3 Market Development The market development sector includes the following interventions: marketing of honey and vegetables, 55 bulking and marketing of grain crops, support to private sector agro-vet input dealers, and grants made through the women’s livelihood innovation award (WLIA) and youth livelihood innovation award (YLIA). Each intervention was implemented on a relatively small scale, encompassing 24 beekeeping groups, 12 horticulture groups, and 102 block farming groups (approximately 2,000 individual farmers); 36 innovation awards for women and youth (totaling 465 individuals); and 10 individual agro-vet input dealers. As mentioned under EQ1, increased emphasis was given to market interventions toward the end of the project, with the addition of activities to support private sector input supply and the innovation awards. 55 RWANU supported a value chain approach to honey, but the support to vegetable production and marketing appears not to have taken a value chain approach, despite its inclusion in the project proposal document. It is perhaps for this reason that the 2016 market assessment did not include it among the five market interventions assessed (Kayobyo et al., 2016). Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 28 Market development provides the “pull” strategy in the RWANU project’s “pathway out of poverty” (see Section 1.2), and the project proposal recognized that linking households to markets and benefiting from the pull of the private sector would be a longer-term process. By implication, this necessitates the long-term sustainability of the interventions. At the start of the project, a market assessment undertaken in 2013 focused on nine value chains56 to assess the potential improvements in farmer productivity and chain efficiency and the business environment. Although the report highlights a number of so-called drought-resistant crops and varieties, it failed to recognize the degree to which drought and floods are a constraint to crop production in many parts of Karamoja, especially in relation to the gross margin calculations, which appear to have been based on crop yields expected in “good” years. Potential returns on investment may have looked good on paper, but in years of drought, flood, or pests, which tend to be the norm rather than the exception, a farmer will recognize losses. The market assessment recommendations clearly influenced project design (e.g., by focusing on maize, beans, sorghum, groundnuts, and goats), although not all recommendations were implemented (e.g., milk market development). YLIA/WLIA were introduced in RWANU’s fourth year; 28 YLIAs and 8 WLIAs were supported, totaling 465 individual members (166 men, 275 women) (Ishara, 2017). Some applicants were already RWANU project participants, but applicants did not necessarily have to be participating in other RWANU activities to apply for the awards, thus reducing the likelihood of “layering”; grants were awarded on the strength of the group’s business application. The evaluation team met with one very successful WLIA and heard mixed reports of other YLIAs/WLIAs from former staff, government officials, and project participants. It is thought that relatively few are still operational. In 2016, a Market System Interventions and Resilience Assessment found “strong evidence that each intervention area was associated with improved household and market system resilience and nutrition,” although the interventions also showed “varied levels of likely sustainability.” 57 The qualitative fieldwork undertaken by the evaluation team revealed the following findings relating to sustainability:   Former RWANU staff reported that beekeepers from at least one-third of the RWANU groups continued to produce honey, but the honey marketing company brought in by RWANU reported that it was not able to buy as much honey from RWANU producers as it had during the project due to transport constraints and the likelihood that Pokot beekeepers near the border were selling their honey to Kenyan buyers.58 In two of the five villages visited, vegetables continued to be marketed in nearby trading centers. In one case, they were being marketed on a group basis; in two other locations, key informants and non-participants reported that vegetable marketing had been adopted by non-participants. 56 The nine value chains included maize, beans, sorghum, green grams, groundnuts, onions, sukuma wiki, cowpeas leaves, and goats. It appears that the milk value chain was also assessed because it was recommended that support should be given to promote the marketing of milk and milk products. 57 The assessment examined the extent to which specific interventions had contributed to (1) improved household and market system resilience, and (2) improved household food security and nutrition. The interventions assessed included the following: (1) savings and credit associations, (2) crop and livestock inputs and services development, (3) bulk grain marketing and block farming, (4) honey value chain strengthening, and (5) business development support to youth and women. Although the report makes references to vegetable production and marketing, this was not explicitly reviewed as an intervention in itself, but rather as different components of other interventions. This study was the only RWANU project document reviewed by the evaluation team that considered the concept of resilience in any detail. It is also interesting in that it considered the impact of market interventions not only on food security but also on nutrition, claiming that those involved in market system interventions show improved nutrition outcomes. 58 In general, Karamoja honey does not appear to be very competitive at a national level in Uganda due to high transport costs. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 29   There is thought to be only one block farm group out of a total of 102 that were established that continues to function as a group.59 The evaluation team found no evidence of any bulking centers that continued to be operational out of a total of 10 that had reportedly been established, although this may have been due to the limited geographical coverage of the fieldwork. In one case, a bulking center that had been planned was reported to have never actually been established; in other cases, the block farm groups did not harvest enough to warrant the use of the bulking centers. Agro-input services were still being provided by agro-input dealers and CAHWs supported by RWANU. Not all farmers could afford to buy improved seed from agro-input dealers, and not all farmers had access to agro-input dealers. The RWANU project failed to account for labor markets in its design and did not differentiate project participants according to their relative socio-economic status (Mueller & Bbosa, n.d.). The labor market assessment60 was undertaken too late to allow for any change in the project design. In at least one instance, land opened up for the block farm group later reverted to individual use by the landowner, thus increasing productive assets for the relatively better-off farmers. This is positive in terms of increasing land under production, but it is only through the consequent increased opportunity for casual wage labor that poorer households can then benefit. Another negative consequence of opening up new land for commercial crop cultivation by the block farm groups was that, in some cases, this exacerbated existing land disputes. The honey marketing company (Golden Bees) helped with training in beekeeping and also trained 15 local artisans from all 4 districts in the construction of improved hives, contributing to the local availability of hives. Honey producers relied on the apiculture specialist and RWANU vehicles for transporting their honey to the marketing company’s shop in Moroto, but this transport option was no longer available after the project ended, and as a result, honey sales by project-supported producers to the marketing company dropped. Although it was not possible for the evaluation team to meet with any honey producers, the apiculture specialist reported that members from at least eight beekeeping groups were still producing honey, suggesting that honey production was profitable for these individuals. Although a beekeepers’ association was formed prior to project closure, there was insufficient time for adequate training, and the association failed to get established. Despite the apparent failure of the block farming groups and bulking centers, linkages between buyers and farmers were reported in project documentation and confirmed by KIIs with sub-county and district officials. Literature suggests that certain “basics” need to be put in place to enable farmer groups to access markets effectively; these include improving rural infrastructure, providing extension services, making credit markets accessible to the poor, and making relevant market information available (Markelova & Meinzen-Dick, 2009). These basics are all lacking in southern Karamoja. The same authors warn that market development does not always help the poorest, who may not have the minimum asset threshold needed to participate in market exchanges. Inclusive market systems development is one approach that can potentially improve USAID’s programming by creating market systems that are competitive, inclusive, and resilient (Campbell, n.d.). The starting point for inclusive market systems development is an understanding of the local context and peculiarities of the existing systems. Although the RWANU project undertook an initial value chain assessment, this failed to fully consider the drought-prone context of Karamoja or the potential for poorer households to participate in specific 59 Three out of the four block farm groups that the evaluation team met gave up their block farming activities after losing their crops to floods or drought; in one case, the group members decided to apply what they had learned to their individual plots rather than work as a group. 60 The Labor Market Assessment was undertaken in late 2015/early 2016 and highlighted the importance of wage labor for food security among poor households, and that the most important employers are medium- to large-scale farmers. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 30 value chains. The subsequent market systems and resilience assessment and the labor market assessment were undertaken too late to influence project implementation or direction. Efforts to promote a private seed sector appeared to be more successful than bulk grain marketing, but similarly failed to understand the context of local seed systems, notably the frequency of free seed distributions as well as the limited potential for poor farmers to purchase seed of different crops. The importance of farmer seed saving appears to have been overlooked; none of the farmers met by the evaluation team reported to have been trained in seed preservation or seed-saving. The need for such training was also mentioned in the market assessment report, which noted that some farmers thought they had to buy seed every year for open-pollinated maize varieties. 61 From the perspective of the input dealers, the training was done very well, with an important lesson emerging from the initial selection of individuals, as reported by a government official and confirmed by the agro-input dealers met. Three types of individuals were selected to receive support to become agro-input dealers: (1) small-scale business people, (2) those who had been small agro-input dealers before, and (3) those with expert knowledge or experience in agricultural production. Only the last two were successful; the small-scale business people lacked the technical knowledge necessary for quality assurance of the inputs provided. All three of the agro-vet input dealers interviewed appreciated the training and support from RWANU, and all were still in business. One had recently expanded by constructing a store and opening a second shop; he had also benefitted from additional support from Mercy Corps. None of these three dealers relied solely on the agro-vet shops for their livelihoods; each had additional income sources, such as commercial farming and soda sales. Members of the Karamoja Agribusiness Association reported that the association was no longer functional. Like the beekeepers’ association, there was insufficient time available for the association to become well-established. Other reported constraints included weak leadership and the failure of members to pay their contributions. Some Karamoja Agribusiness Association members continued to collaborate in radio advertisements to publicize the seeds available at planting time. The WLIA/YLIA assessment report highlighted the members’ appreciation of the training provided. The types of businesses supported included cereal banking, grinding mill, poultry, piggery, goat rearing, cattle trading, tailoring, bakery, and hair salon (Ishara, 2017). The one group interviewed by the evaluation team was engaged in agricultural production and second-hand clothes sales. The group had a very energetic leader and committed members who had clearly benefited (e.g., by being able to pay school fees and construct new houses); those who were not committed had left the group, and the membership of others who lacked commitment (mainly men) was to be cancelled. Conclusions Overall, the design of the market interventions lacked coherence, comprising several different interventions, each implemented on a small scale and lacking sufficient integration with one another or the broader existing systems. Critical understanding of local market realities in the variety of agricultural product and variable-district contexts was missing or only developed over time. Although analyses and assessments provided insight into various contextual factors limiting agriculture-based market development (e.g., recurrent drought, flood, pests; remoteness and poor infrastructure), hindsight has shown that these limitations proved greater than anticipated, leading to the failure of the block farms and bulking centers. Although the project proposal noted that market development is a long-term process, the lack of sustainability of many of RWANU’s market development interventions limited their impacts. RWANU failed to take account of labor markets and the socio-economic differences among project participants; many poorer households may have been excluded, and the better-off households, such as those with alternative forms of income and some capital to invest (as in the case of some former 61 Seed can be saved by farmers for open-pollinated maize varieties; it is only hybrid varieties for which it is necessary to purchase fresh seed each year. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 31 RWANU extension workers), who were best-placed to promote market development, were not necessarily targeted. The training for agro-input dealers and WLIA/YLIA awardees was very good. Strong leadership and financial benefits to the members are critical for continued success of both the business groups and business-oriented associations. In general, effective associations take time to establish and must have a viable business plan. 4.2.4 Savings and Loans The overwhelming consensus from all FGDs and interviews relating to SO1 was that VSLAs were the most successful and effective aspect of the RWANU project. The VSLAs created a savings culture among project participants and allowed access to savings and loans for various purposes, including meeting emergency needs (e.g., hospital costs, medication, and, at times, food); covering education costs; purchasing household items (e.g., roofing, saucepans, and other household items); investing in agricultural production (e.g., poultry, goat-rearing, purchasing seeds, paying for laborers); and creating opportunities for income generation (e.g., beer brewing, petty trade). Given the broader livelihoods shifts described above for Karamoja, opportunities for income generation are especially important now that men are spending more time at home due to the loss of livestock. A former RWANU staff member reported that women’s income-generating activities with loans and savings from VSLAs were initially focused almost entirely on beer brewing and that efforts had to be made to diversify their investments. Unfortunately, the PBS data do not reflect the success of the VSLAs,62 most likely because the questionnaire specifically asked about “agricultural credit,” whereas the VSLA savings and loans are used for a variety of purposes, often to meet emergency needs. All SO1 groups and some SO2 groups established by the RWANU project were also trained in savings and loans; it is estimated that there may have been 1,400 VSLAs for project participants. It was also reported that non-participants formed their own VSLAs (Annual Results Report 2016), and this was confirmed by the fieldwork. With such a high number of VSLAs, plus spillover among non-participants, the apparent decrease in access to credit shown by the PBS results is perplexing. One of the challenges in implementing the VSLAs was poor record-keeping due to low literacy levels. This was addressed by the introduction of educated voluntary field agents who were trained and provided with a bicycle to assist VSLAs in record-keeping. These field agents are still supporting the VSLAs and are also VSLA members. Some of the former RWANU extension workers are also still helping the VSLAs that they formerly supervised by assisting in the annual share-outs, when the money saved over the course of a 9- to 12-month period is distributed according to the contributions of the group members. Links between the VSLAs and formal credit institutions were made, but these proved not to be sustainable, as discussed in Section 4.3.4. All VSLA groups met by the evaluation team had continued with their VSLA activities after the RWANU project had ended, although the levels of savings contributions varied. In some cases, all group members contributed the same amount; in other cases, the contributions were based on what an individual could afford. Some groups had reduced contributions or suspended contributions on a seasonal basis, to be resumed at a later date. In one village visited, non-participants had worked with the VSLA/FTG leader to establish their own VSLA by sharing the toolkit of the project-supported VSLA; in another case, non-participants wanted to form a VSLA but lacked the necessary toolkit. Male and female VSLA members reported greater dialogue and engagement between husbands and wives in household decision-making. Gender training was incorporated into the VSLA training to help ensure that women 62 The percentage of farmers who reported to have used financial services (savings, agricultural credit or agricultural insurance) in the past 12 months decreased significantly, from 27.1 percent at baseline to 9.0 percent at endline (Annex 8F, Table 8F.2). Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 32 retained decision-making control over their savings and loans. Women reported that they had been empowered by their engagement with cash. Conclusions The VSLAs were both appropriate and effective in increasing access to savings and credit by men and women for both consumption and productive purposes. VSLA members needed additional awareness￾raising or training in diverse options for income-generating activities in which to invest. The VSLA model was adapted to suit the local context by the introduction of voluntary field agents where necessary to support bookkeeping in groups with low literacy and by having flexibility in the level of members’ contributions according to what was affordable and sustainable. Greater spillover could have been achieved by making the specially designed savings box (with three locks) available through local blacksmiths and traders. 4.2.5 Nutrition The RWANU project design and results framework for the household-level nutrition and health component included the following: (1) establishing MCGs in targeted communities; (2) promoting positive health-seeking behaviors; (3) promoting consumption of diverse, nutritious food; and (4) promoting safe water practices, improved sanitation practices, and safe hygiene behaviors. The support was intended to result in improved health and nutrition practices at the household level. This section reviews the strengths and challenges of the design and implementation of the household-level nutrition and health component, except for WASH, which is addressed in Section 4.2.8. See Annex 7D for a list of the technical interventions.63 Interviewed community members, district and facility health staff, and RWANU project staff considered the technical interventions as generally relevant to achieve the project outcomes, although the design of the keyhole kitchen gardens was seen as requiring adjustments to improve their effectiveness, which is discussed below. MCGs. An MCG is a group of 10–15 community-based volunteer lead mothers who meet regularly with project staff for training and supervision, and then hold meetings and visit regularly with 10–15 neighbors, sharing what they have learned and facilitating behavior change at the household level. The total number of MCGs by the end of the project was 345, just short of the life of activity (LOA) target of 350 and comprised 3,499 lead mothers (ACDI/VOCA 2017b).64 More than half of the interviewed health staff (7 of 13) shared that the MCG approach was very good for Karamoja and one interviewed parish chief indicated that the MCGs were one of the most successful aspects of the RWANU project. Eight of nine former RWANU health staff interviewed stated that the establishment of the MCGs was very effective because the MCGs empowered women by increasing knowledge and building confidence, penetrated villages and households in remote areas, raised awareness on health-related issues, and served as role models in the community. In addition, three of 63 Technical interventions included the MCG approach, integrated management of actue malnutrition (IMAM) SURGE approach, MCA approach, drama groups, goat distribution, and keyhole kitchen gardens. Drama groups were mentioned during FGDs and KIIs in reference to food distribution sites and health facility activities, and in this report findings for the drama groups are primarily discussed in the sections on food distribution and health facility activities. 64 In March 2015, Samaritan’s Purse completed a three-year UK Department for International Development-funded Maternal and Child Health project in Napak. The project included training women using the MCG approach, with 219 MCGs comprising 2,130 leader mothers and grandmothers and 35,597 mothers. In April 2015, RWANU started the incorporation and restructuring process of the Napak MCGs to align them with the RWANU MCG inclusion criteria, which included pregnant and lactating mothers and women with children under 5 (ACDI/VOCA. 2015). RWANU staff shared that Samaritan’s Purse had rolled out its own maternal and child nutrition curriculum, which covered all the optimal practices, and the transition from the MCGs being under Samaritan’s Purse to RWANU went smoothly; after about two months of training the Napak groups were performing well. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 33 five interviewed VHT members said that the collaboration of the lead mothers and the VHTs on follow￾up and monitoring visits to households motivated the household members to adopt promoted practices, such as breastfeeding, personal hygiene, and food preparation. MCGs were also important for educating mothers on the importance of antenatal services and immunizations, which resulted in increased facility￾level deliveries and improved immunization in isolated communities. One of five FGDs with mother caregivers also shared that the monitoring and follow-up visits by the lead mothers motivated them to access health services, such as antenatal care, child immunizations, facility-level births, and family planning. Endline survey results showed that there was a statistically significant increase in the percentage of mothers of children 0–23 months of age who attended four or more antenatal care visits with their youngest child—75 percent at baseline and 84 percent at endline, although the project was just shy of achieving the LOA target of 85 percent.65 Two former RWANU staff shared that it could be difficult to find women at home during planting and harvest season, and migration made it difficult to consistently learn and adopt health and nutrition messages. Project reports indicated that staff attempted to address this issue with more intense training activities during periods of the year not occupied by planting and harvesting. Table 7D.1 in Annex 7D shows best practices in MCGs compared to MCG implementation under the RWANU project. Project implementation of MCGs followed the majority of best practices, but with improvements to implementation, they could have been more impactful. Suggested improvements include the following: (1) ensure the frequency of contacts for mother caregivers to at least two per month through the group sessions or home visits, (2) expand the number of health promoters to increase the opportunities for supervision of the mother leaders by the health promoters during their sessions with mother caregivers and home visits, (3) tailor messages to the life cycle needs of each mother caregiver during home visits and monitor delivery to ensure timely messages and actions to adopt promoted behaviors. Three of nine former RWANU staff shared that the MCG approach and its effectiveness to positively influence behavior change could have been strengthened by: (1) developing mechanisms to continually motivate mother caregivers to apply behaviors; (2) improving community mobilization for wider participation and ownership of activities and promoted behaviors; (3) targeting adolescent girls; and (4) ensuring the quality of MCG training, providing adequate time for mother caregivers to absorb and apply the information and a focused number of new practices, and allowing time for staff to reflect on implementation and potential adjustments to improve impact. MCG lead mother and mother caregiver involvement in other RWANU project activities. Only 16 of 93 lead mothers and mother caregivers interviewed in 10 FGDs shared that they were also involved in the RWANU farmers’ group, and an additional group of mother caregivers (10) said that they were part of a RWANU VSLA, which was beneficial to the women and their families. Women shared that they sold part of their harvest from their involvement with the farmers’ group to pay for health needs or children’s school supplies and used loans from the VSLA to start small businesses, such as raising chickens or making brew. Although there are limited data available from participants regarding their involvement in both SO1 and SO2 activities, the findings shared here demonstrate that participants found involvement in both SO1 and SO2 activities beneficial. This is an important consideration for current DFSA projects. MCG incentives. Although one of five interviewed VHT members said that the incentives (e.g., watering cans, jerry cans, hoes, saucepans, axes, and machetes) that lead mothers received helped motivate them, two of five FGDs of lead mothers felt that the inputs they received were insufficient for the lead mothers because there were not enough for all group members, which caused divisions in the 65 Note that the IPTT indicator for which the target was set is slightly different from the baseline/endline indicator. The IPTT indicator is: “Percentage of mothers with children aged 0-12 months who had four or more antenatal visits when they were pregnant with their youngest child.” Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 34 group, and they also that they did not receive them often enough. Two of nine interviewed former RWANU staff said that the incentives raised group expectations, and later when the project did not provide more incentives, the groups were demotivated. Also, the incentives were intended to be shared with mother caregivers, but they were not always shared, and only two of five FGDs with lead mothers indicated that they shared the inputs with mother caregivers. In at least 4 of 10 FGDs with lead mothers or mother caregivers, FGD participants emphasized the need to treat participants equally. Although incentives for lead mothers and their groups may initially help motivate lead mothers, inconsistent application over groups and time can lead to conflicts in the groups. It is important for projects to identify sustainable ways to support groups and motivate them. MCG training on food preservation. One of five FGDs of mother caregivers shared that they did not learn food preservation methods like solar drying from RWANU. They said that preserving food helps to increase consumption of diverse food in different seasons. RWANU quarterly reports for 2017 discuss training and demonstrations of solar drying for horticultural farmers, but there does not appear to be a link to MCGs and the keyhole gardens. The information shared by mother caregivers suggests that it would be useful to provide training on food preservation. Community-level opinions on RWANU nutrition and health outcomes. In all five FGDs with mother caregivers, two of five FGDs with lead mothers, and two of five KIIs with VHT members, respondents felt that the RWANU project had resulted in some positive outcomes (Annex 7D, Table 7D.2). When asked about MCG activities, about half of the responses from FGDs with lead mothers and mother caregivers focused primarily around WASH, and in two or three cases, it was necessary to specifically ask the group about nutrition activities. Inquiries during FGDs and KIIs did not shed light on why this may be the case; however, the IYCF module was the first training module that the lead mothers and mother caregivers received (third quarter of 2014), with subsequent training on WASH, the fourth module in the series, in the first quarter of 2016. Although several RWANU staff indicated that key points of modules were periodically reviewed with lead mothers, it is not clear to what extent this occurred with specific modules and how effective this was, or whether health promoters or lead mothers tended to focus more on certain topics, such as WASH, compared to IYCF messages and behaviors. This could also be related to a lack of timing of message tailored to the life cycle needs of each mother caregiver. Child nutritional status. Few respondents (two of five FGDs with mother caregivers and one KII with a VHT member) commented on child nutritional status outcomes, and responses showed mixed results. One group of mother caregivers felt that there were fewer children with acute malnutrition compared to before RWANU project implementation. One group of mother caregivers said that there was no difference in child nutritional status before and after RWANU project implementation because the mothers in the community did not receive the RWANU food ration. One VHT member said that child nutritional status had improved during the RWANU project implementation, but now that the project was over and that the food rations provided by Andre Foods International are less than those received under the RWANU project, the number of cases of acute malnutrition has increased.66 One VHT member also commented that women have improved knowledge of child feeding but that only some have adopted the new child feeding practices because they cannot afford the additional food. Six of 10 FGDs with lead mothers or mother caregivers revealed that lack of money for food prevents mothers from increasing frequency of feeding and dietary diversity for young children. Two interviewed former RWANU staff recommended that training for mothers be holistic, not just focused on nutrition, 66 Andre Foods International is a registered national NGO in Uganda and has partnered with the World Food Programme in Karamoja to support food ration distribution and nutrition and health promotion, among other activities. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 35 but on education and livelihoods, because mothers need a constant income to be able to afford a variety of food to improve their nutrition and that of their families. A Concern Worldwide contextual analysis of nutrition in Karamoja in 2017 found that the perceived causes of malnutrition among community members were low number of meals per day, large family size/narrow child spacing, and alcoholism as an aggravating factor (Stallkamp, 2017). The biggest challenges to overcoming malnutrition were lack of money and income-earning opportunities, long distance to markets, and high workload of women, along with lack of agricultural inputs and land, erratic rainfall, poor access to water and sanitation, and sharing of food rations. See Annex 7D for additional information regarding women’s workload and implications for project implementation and outcomes. An additional contributor to poor child nutritional status could be poor maternal nutritional status. Maternal underweight almost doubled among women between the baseline and endline survey. According to the 2016 DHS, the percentage of infants born with low birth weight in Karamoja was 9.5 percent (among the 76 percent of births with a reported birth weight), and 17 percent of mothers considered their infant “very small” or “smaller than average” (Uganda Bureau of Statistics [UBOS] et al., 2018).67 Low birth weight is associated with childhood stunting (Black et al., 2013). Table 8 shows project monitoring data of the proportion of children 6–23 months of age, total and by sex, who were underweight by fiscal year. Annual monitoring data were collected in September, except for 2017, when data were collected in the May/June lean period due to the project closure. Worsening underweight among children 6–23 months of age from 2014 to 2016 may be related to a high level of food insecurity, as reported in the July 2016 food security and nutrition assessment (UNICEF et al., 2016), including increased weather and rainfall variation that led to poor harvests over the prior three consecutive growing seasons, and an increase in staple food prices. Annex 7D provides an analysis of the prevalence of underweight among boys and girls. Table 8: Underweight among Children Ages 6–23 Months, Total and by Sex, by Year Age Group and Sex 2014 2015 2016 2017 6–23 Months, Total 22.7% 25.6% 27.3% 32.1% 6–23 Months, Boys 22.3% 29.9% 28.5% 40.7% 6–23 Months, Girls 23.0% 20.5% 26.0% 23.7% Source: ACDI/VOCA 2017a Overall, there is a need for increased focus on improving incomes, reducing women’s workload, and improving women’s nutritional status, and a better understanding of care and feeding for boys and girls to achieve improvements in child nutritional status. Infant and young child feeding—exclusive breastfeeding. The percentage of children under 6 months of age who were exclusively breastfed increased significantly (59 percent at baseline, compared to 76 percent at endline), and specifically for boys under 6 months of age (56 percent at baseline, compared to 86 percent at endline) but not for girls under 6 months of age (61 percent at baseline, compared to 68 percent at endline). FGDs and KIIs did not reveal reasons why prevalence of exclusive breastfeeding between baseline and endline increased significantly among boys under 6 months of age but not among girls. Mother caregivers said that they feed their boys and girls the same way. Former RWANU staff were surprised by the exclusive breastfeeding results and could not think of any reason for this outcome. The RWANU project did achieve the LOA target (69 percent) for exclusive breastfeeding among children under 6 months of age. Due to poor 2014 monitoring results on the prevalence of exclusive breastfeeding, RWANU project staff conducted a barrier analysis and applied the findings to increase the emphasis on exclusive breastfeeding during project implementation. Annex 7D 67 Low birth weight was defined as the percentage of births with a reported birth weight below 2.5 kilograms, regardless of gestational age. The sample was all live births in the 5 years before the survey that had a reported birth weight from either a written record or the mother’s report. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 36 provides details on the RWANU training on exclusive breastfeeding, the monitoring results, the barrier analysis findings, and changes to project implementation in response to the findings. IYCF—complementary feeding. FGDs with lead mothers, mother caregivers, and non-participants, and KIIs with district nutrition staff and former RWANU staff revealed several poor IYCF practices, including feeding children the same food, and at the same frequency, as adults—just twice daily, morning and evening, especially when the mother is working outside the home (e.g., agricultural day labor, breaking stones for quarry, collecting firewood, or making charcoal). During the hungry season there may be just one meal a day. Mothers usually take children under 2 years of age with them but leave children 2 years of age and older at home with an older sibling or grandparent, who may or may not be able to feed them. One lead mother shared that she prepared a maize and sugar porridge as a liquid for her 1-year-old child, which indicates that there are issues with consistency and nutrient density of food for young children, as well as diversity and frequency. During the dry season when there is little water and little food, children are fed residue, the roughage that is left over after brewing the local alcohol, which is available for free. Children are also given the local brew (alcohol), including boys and girls under 2 years of age. The endline survey findings showed that there was no change in the percentage of children 6–23 months of age receiving a minimum acceptable diet (7.0 percent at baseline and 7.7 percent at endline).68 It should be noted that the endline survey was conducted one year after the last MCG sessions, and the lack of behavior change messages/motivation and project support most likely played a part in the poor outcomes of some indicators, such as minimum acceptable diet. The RWANU annual monitoring survey among project participants found that the percentage of children 6–23 months of age receiving a minimum acceptable diet was 7.7 percent in 2014, 25 percent in 2015, 22 percent in 2016, and 13 percent in 2017 (ACDI/VOCA 2017a). An IYCF module post-test survey conducted by RWANU staff found that among mother caregivers who participated in the MCGs, only half (53 percent) of the caregivers of children 6–23 months of age reported that they gave meals to their child three times a day, and 40 percent fed their child a variety of foods. Mother caregivers indicated that the primary barrier to improving dietary diversity was that foods were either not locally available or were too expensive. The 2015 Concern Worldwide post-module survey report recommended strengthening the linkage between RWANU SO1 and SO2 to address the challenges of achieving dietary diversity at the household level. Staff shared that creating and strengthening the latter linkages was a challenge during project implementation. In 2016 RWANU staff conducted a barrier analysis study on the use of animal source foods by PLW and children 6–59 months of age. The findings and implications are in Annex 7D. Knowledge of IYCF and health messages. RWANU final annual monitoring data showed that the percentage of caregivers who knew at least four of six IYCF practices and three of eight maternal and child health practices was 87 percent, just shy of achieving the 90 percent LOA target. This shows that knowledge of IYCF practices was good, but the findings above indicate that much more needs to be done to overcome barriers to practice. Underweight among women. The prevalence of underweight among women 15–49 years of age increased significantly (27 percent at baseline and 46 percent at endline), and there was no significant change in women’s dietary diversity score (2.6 percent at baseline and 2.7 percent at endline). The increase in women's underweight was also seen in the June 2017 UNICEF/World Food Programme (WFP)/FAO Food Security and Nutrition Assessment (FSNA) (FSNA, UNICEF, et al., 2017), in which the highest proportions of underweight women were found in the districts of Nakapiripirit (47 percent), 68 Minimum acceptable diet is an indicator for assessing IYCF practices and measures the proportion of children 6–23 months of age who receive a minimum acceptable diet (apart from breastmilk). It is the number of breastfed children 6–23 months of age who had at least the minimum dietary diversity and the minimum meal frequency during the previous day divided by the number of breastfed children 6–23 months of age, AND the number of non-breastfed children 6–23 months of age who received at least two milk feedings and had at least the minimum dietary diversity not including milk feeds and the minimum meal frequency during the previous day divided by the number of non-breastfed children 6–23 months of age. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 37 Napak (47 percent), and Moroto (43 percent). These findings are similar to the FSNA findings from June 2016, when Moroto (38 percent) and Napak (41 percent) had the highest proportion of underweight women (UNCEF et al., 2016). The prevalence of underweight among women in Nakapiripirit increased from 28 percent to 47 percent in the FSNA from 2016 to 2017, which coincides with the time when the RWANU food rations were no longer distributed. The poor outcome for women is due in part to a lack of project focus on assessing, identifying, and treating women who were underweight. In addition, the heavy rains during the endline survey could have increased the incidence of diarrheal disease, which, in combination with women's limited access to outreach services and sub-optimal health care services during the time of the endline survey, could have increased women’s underweight. Other factors that could have increased women’s underweight include poor access to diverse foods due to poor linkages with the agricultural SO of the project, and heavy workload, such as agricultural day labor, breaking stones for quarry, collecting firewood, or making charcoal—all very labor-intensive tasks that bring in small amounts of income for women to support their households. In addition, during the time of project implementation, the food ration for PLW could have included more energy and protein to fill the women’s nutrient gap, which is discussed further below. One FGD with women also said that lack of funds was a constraint to improving the diets of PLW. There is a need for an increased focus on improving women’s diets, preventing and treating women’s underweight, improving income generation, and implementing labor-saving technologies for women. Integrated Management of Acute Malnutrition (IMAM) Surge approach. About 80 percent of lead mothers were trained in 2013–2014 to screen and identify acute malnutrition in children 6–59 months of age using the MUAC tape, to assess for nutritional edema, and to refer suspected acute malnutrition cases to the VHTs. The effectiveness of the IMAM Surge approach appeared to have been mixed, but there is lack of data to clearly indicate the extent of its effectiveness. During RWANU project implementation in 2014, targets to identify children with acute malnutrition and refer them to therapeutic feeding programs were exceeded (168 percent), with 2,638 children admitted to therapeutic feeding programs in the project districts (ACDI/VOCA, 2014b). In November 2015, the project indicated that there were 1,511 children identified with severe acute malnutrition, and 51 percent were not enrolled in a therapeutic treatment program. Subsequent reports did not indicate whether the approach achieved targets (ACDI/VOCA, 2016). Staff noted that the lead mothers needed continuous mentoring on the use of the MUAC tapes to improve accuracy while taking measurements (Concern Worldwide, 2015). The Concern Worldwide SURGE team and RWANU responded by increasing training and follow-up for lead mothers, VHTs, health facility staff, and mothers. Recent research has shown that mothers with low literacy and numeracy in developing countries can successfully identify children with severe acute malnutrition and moderate acute malnutrition using MUAC tapes (Blackwell et al., 2015; Grant et al., 2018). There is potential for the approach to be effective, and it should continue to be supported in Karamoja and its use documented and shared with the wider development community. Implementers should also develop strong links with development professionals and researchers using the approach in other settings to establish a network for learning and adaptation. In terms of feedback from participants, three of five interviewed VHT members said that MCGs played an important role in sensitizing and creating awareness among mothers through monthly monitoring of children under 5 years of age with MUAC tapes and monitoring of pregnant women. Two of five FGDs of lead mothers said that their training on use of the MUAC tapes to identify malnourished children was a benefit to the community and to them all. Lead mothers said that they measured children’s MUAC twice per month and referred children with acute malnutrition to the VHT, who then checked the MUAC measurement and referred the parents and child to the health facility. MCAs. This was a pilot program introduced in Year 2 of the RWANU project to involve men in promoting joint decision-making and positive household-level health and nutrition behavior change. During both the FGD with lead mothers and with mother caregivers in one project community, FGD participants shared that the MCA approach resulted in positive changes in men and household dynamics. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 38 The lead mothers said that the RWANU project encouraged men to attend some of the lead mother trainings, which helped contribute to men attending antenatal services with their wives and constructing latrines. Mother caregivers noted less domestic violence; more joint decision-making in the household; more freedom for women to sell food, such as vegetables; and more assistance from men with cash for the household and household chores, such as cooking, bathing children, washing children’s clothes, and buying food. The mother caregivers felt that the couples’ communication training they received was useful and that the MCAs and messages helped bring unity to families. One of two interviewed MCAs said that his MCA work brought happiness to his family, reduced fights, and brought peace at home, and now he bathes his children, and if his wife is not at home, he brings water, firewood, cooks, or goes to the health center with the child. Another interviewed MCA noted that he has seen some small progress in changing men’s behaviors, especially regarding some men assisting with weeding and cultivating. MCAs explained that after being trained they discussed the ideas and behaviors with their close friends and over time slowly convinced their friends, who gradually tried some of the new practices. Then the MCAs encouraged their friends to convince a few of their neighbors to try the ideas, and in this way the new ideas and practices expanded. Key findings from a Concern Worldwide survey on the MCA approach in August 2016 are described in Annex 7D. The findings showed that MCAs were serving as positive role models, but there was still much more effort needed to positively influence the adoption of behaviors among men that the MCAs were working to influence. The report recommended supporting MCAs as they engage with other men to achieve the same levels of behavior change that they themselves had achieved. FGDs and KIIs also revealed that the MCA intervention could have had a greater impact if it had higher coverage, elders and community leaders were adequately engaged, training covered more topics, and MCGs and MCAs trained together to support each other. Although the MCA approach resulted in some improvements in men taking on more responsibility and reducing conflict in the home, there is room for the approach to be expanded and improved. Goat distribution. The RWANU project provided 2,267 goats to approximately 420 women, achieving the LOA target of 2,100 goats distributed. In an FGD with lead mothers in a community in which goats were distributed, an interview with a village chief, and an FGD with sub-county officials, FGD members and key informants shared that they thought that one of RWANU’s greatest impacts was improved child nutrition through the goat distribution. An analysis of pooled baseline and endline data (see Table 8H.4, Annex 8H) found that meal frequency, or the average number of meals in the last 24 hours, among children 6–23 months of age was significantly greater in households with goats (2.8), compared to those without goats (2.3), and women’s dietary diversity was significantly higher in households with goats (3.1), compared to those without goats (2.6). However, there was no difference in the nutritional status of children 6–23 months of age or women’s underweight in households with goats compared to those without goats. It is important for projects to monitor hygiene and sanitation among families that receive goats to ensure that children are not in contact with goat feces. It is not clear whether the sanitation and hygiene aspects of goat distribution were monitored by the project or specific guidance was provided to mother caregivers regarding this topic. One of five FGDs with lead mothers reported that the goat distribution was the most appreciated RWANU project activity. One FGD with a women’s goat group shared that through the goats that the women received through the RWANU project, they have been able to have goat milk for six to seven months of the year, which is boiled and given to children older than 6 months of age, mixed with porridge for children’s complementary food, or consumed by PLW if there is a sufficient quantity.69 If the 69 An important consideration is to ensure that infants under 6 months of age are exclusively breastfed, and from 6–12 months of age, ideally the infant should be breastfed and provided appropriate complementary food, and the goat milk should be prioritized for the lactating mother. Small quantities of goat milk could be boiled and added to porridge for infants 6–12 months Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 39 women are milking a few goats at once they will use some milk as fresh milk, some for soured milk, and some for butter. The women said that they are now able to afford to buy meat when before they had the goats they could not. Women have sold the kidded goats, not the original goats, to pay for medical treatment, food, household necessities, and items for a newborn baby. The women and their husbands received couples training before being given the goats, so they discussed and made joint decisions about the goats. Five of 10 FGDs of lead mothers or mother caregivers commented that they found it unacceptable that other groups of mothers received goats and that they did not. Overall, the goat distribution has very good potential to improve dietary diversity, meal frequency, and child and maternal nutritional status. Goat distribution should be considered for expansion, with careful monitoring of outcomes and impact. Keyhole kitchen gardens. Project reports indicated that by the end of the project, more than 10,000 keyhole gardens were constructed. Most (7 of 10) FGDs with lead mothers and mother caregivers valued the usefulness and benefits of the keyhole gardens. During two of five FGDs with mother caregivers, participants shared that they were attracted to the MCG in part because of the keyhole gardens, and one lead mother shared that she thought it was the most successful aspect of the RWANU project. Participants shared that the keyhole gardens were used to grow vegetables, such as greens, cowpeas, eggplant, and onion. The vegetables were used for consumption or sale. The keyhole gardens made access to vegetables convenient and saved money and time otherwise spent purchasing the vegetables in the market. A basin of bundled vegetables could sell for UGX 3,000 to 5,000, and the funds could be used to purchase salt, silver fish, or other foods to improve meal frequency and diversity, used to pay medical bills, or placed in a VSLA. One FGD of non-participants said that the keyhole gardens belonging to participants allowed the non-participants to conveniently purchase vegetables without having to go to the market. Five of the seven FGDs appear to still be using the keyhole gardens, but two FGDs in one community said that the keyhole gardens were destroyed by floods and they did not have funds to purchase the seeds, which in the past were provided by the RWANU project. Three of five interviewed VHT members also shared that the vegetables grown in the keyhole gardens helped meet nutrient needs and contributed to food security; one VHT had built a keyhole garden and was still using it, and another VHT said that some men established their own keyhole gardens, noting that women were earning money from them. However, interviews with sub-county officials, a few lead mothers, and some former RWANU staff indicated that the challenges with the keyhole gardens were the heavy labor needed for their construction, access to water, and time and effort for their appropriate maintenance. Sub-county officials felt that less labor intensive and appropriate methods could be implemented. A former RWANU staff member shared that although the gardens were challenging to construct, women overcame the challenge by working together to build them. Although the keyhole gardens were appreciated, they did have challenges, and it would be worthwhile to consider more appropriate options for women to grow vegetables for home use or sale. Other health priorities. One of five FGDs with mother caregivers commented that they felt that malaria was their biggest problem, and one interviewed former RWANU staff member felt that malaria prevention and treatment was not adequately addressed by the project. Malaria can increase mortality, as well as increase anemia. Conclusions The household-level nutrition and health component’s technical interventions, including the MCG approach, the IMAM SURGE approach, the MCA approach, and goat distribution, were generally appropriate, but each can be improved. Although many project participants, staff, and partners felt that of age (Pan American Health Organization & WHO, 2004). Undiluted, boiled goat milk may be given to infants starting at 12 months of age. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 40 the MCG approach was very good, the approach and its effectiveness to positively influence behavior change could have been strengthened by continually motivating mother caregivers to sustainably adopt behaviors, improving community mobilization, targeting adolescent girls, aligning interventions with life cycle needs, ensuring the quality of the MCG training, time for behavior adoption, and time for staff to reflect and improve upon interventions. The IMAM SURGE approach implemented with the lead mothers showed mixed results in identifying and referring children with severe acute malnutrition and moderate acute malnutrition to health facilities. However, recent research demonstrates the potential of the approach, and new projects should adopt the approach, along with quality improvement methods, and monitor it carefully to ensure quality implementation and strengthening of community-level capacity. Although the MCA approach resulted in some improvements in men taking on more responsibility and reducing conflict in the home, there is room for the approach to be expanded and improved by including more training topics, including older men and elders as well as young men, and having the MCG and MCA groups meet and reinforce one another. The goat distribution was popular and has good potential to improve dietary diversity, meal frequency, and child and maternal nutritional status. The goat distribution should be considered for expansion, with careful monitoring of outcomes and impact, and learning applied to adapt the program as needed during implementation. The keyhole garden intervention, as designed, showed mixed results, and it is necessary to explore designs that provide better outputs and benefits with less time and labor. The RWANU project technical intervention approaches and methods in household-level nutrition and health were generally effective in improving mothers’ knowledge of nutrition but did not result in sustained behavior change or improvement in maternal or child nutritional status. Integrated programming, in which participants benefit from not only nutrition and health interventions, but also from interventions, such as the farmers’ group or the VSLA, seem to result in economic benefits for project participants. Integrated programming is critically important for current development food security activities to incorporate in their project designs. Although incentives for lead mothers and their groups may initially help motivate lead mothers, inconsistent application over groups and time can lead to conflicts within the groups. It is important for projects to identify more sustainable ways to support and motivate groups. Overall, there is a need for increased project focus on improving incomes, reducing women’s workload, improving women’s nutritional status, improving child spacing/family planning, and a better understanding of care and feeding for young children, including boys versus girls, to achieve improvements in child nutritional status. There appear to be two consistent barriers to adoption of improved IYCF practices: (1) lack of funds to purchase recommended foods to improve dietary diversity and meal frequency, and (2) inadequate linkages between agriculture and nutrition to positively influence adoption of improved dietary intake. There is also a need for improving income generation for women, and implementing labor-saving technologies for women, while ensuring that men accept and support women’s initiatives. In addition, incorporation of malaria prevention and prompt treatment is important to consider for both child and maternal health. 4.2.6 Food Distribution Alignment of food package with preventing malnutrition in children under the age of 2 approach (PM2A) guidance. The RWANU food ration package is described in Annex 7 and Table 7E.1. Using relevant figures for Karamoja, Table 7E.2 (Annex 7E) shows the energy in kilocalories and protein in grams per day in the RWANU food rations for PLW, children 6–23 months of age, and the household ration, compared to the food ration guidance provided in PM2A (FANTA-2 2010; ACDI/VOCA 2012). The RWANU food ration generally aligned with the PM2A guidance for protein and kilocalorie intake for children 6–23 months of age but appears low for the kilocalorie and protein needs of PLW and the household ration for the lean season. The RWANU project proposal indicated Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 41 that the household ration was not designed to meet the entire calorie and protein gap estimated during the lean season, but that the ration would be periodically evaluated to ensure that the individual and household rations were as small as possible and still met biological needs. The RWANU project estimated the kilocalorie gap as the average of the national gap for Uganda and the gap for Karamoja because the Bellmon analysis indicated that the food security situation in Karamoja had improved since the Karamoja-specific calorie gap was estimated; however, as mentioned above, the FSNA for 2016 indicated that the food security situation had deteriorated, so further analysis of the appropriateness of the ration size would have been warranted during the time of project implementation (USAID, 2011). 70 Achievement of targets for ration provision. Food distribution began in the second year of the project (2013) and ended in December 2016; no new participants were added to the food distribution component of the program after May 2016 (ACDI/VOCA. 2016 [Q3 report]). Participants located within five kilometers of a health facility received a WFP maternal child health and nutrition ration for PLW and children 6–23 months of age, which did not include a household ration during the lean period, and the composition of the WFP ration during the dry season may have varied depending on the availability of commodities by WFP. Various FGDs with mother caregivers provided slightly differing figures for the composition and amount of rations received, perhaps because it had been at least 22 months since they had received the last ration. Table 7E.3 (Annex 7E) shows the length of activity target, achievement, and percentage of target achieved for provision of PM2A rations to children 6–23 months of age, PLW, and individuals provided the protective household ration during the lean season. The target was almost met (96.1 percent) for PLW, but it was not met for young children or individuals receiving the protective ration.71 This occurred because initial participant numbers in the proposal were overestimated, which became apparent during the project census. Conditionality of food ration provision. Originally, the provision of the food ration was conditional on attendance at MCG household caregiver meetings, child vaccination against measles, vitamin A supplementation, and attendance at three health facility antenatal visits. In 2014, RWANU piloted the conditionality for food distribution eligibility, which was found challenging due to the varying time between the four conditions, data collection, and matching eligibility for ration distribution. In the 2015 Pipeline Resource Estimate Proposal, RWANU indicated that it did not enforce the ration conditionality requirements but continued to encourage the actions at health centers and Ministry of Health outreach at food distribution points. Sharing of food rations. Food distribution was relevant, given women’s and children’s poor nutritional status in Karamoja, but the plausibility of food rations contributing as intended to reducing malnutrition in pregnant and lactating mothers and children under 5 years of age was limited because of sharing of food rations. Three of five FGDs with non-participants revealed that they received some food rations from relatives or friends participating in food distribution. Three of five FGDs with mother caregivers indicated that food rations were shared with family members, including the children of co-wives in polygamous households. In two case studies of vulnerable elderly women in two communities, the women said that relatives or friends who received the food rations would at times share some of the ration with them, and an FGD with lead mothers and interviews with a VHT member and a local council member from three different communities also showed that sharing of food rations was common. The RWANU project conducted a qualitative assessment of the food distribution project component in 2014, which also found that participants reported sharing food rations with other 70 The Bellmon analysis is a market analysis of key commodities and logistics mechanisms to assess the feasibility and appropriateness of monetization of Title II commodities for USAID FFP programming. The Uganda Bellmon was based on a desk study and field work conducted during April to June 2011. 71 Project monitoring data show that 65 percent of the target number of male children were reached, and 67 percent of the target number of female children were reached; 43 percent of the target number of individuals receiving the protective ration were reached (Table 7E.3, Annex 7E). Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 42 household members, including children, men, and visitors, regardless of the intended participants (ACDI/VOCA 2014 [Q2 report]). RWANU reports indicated that rations lasted two to three weeks, rather than one month, due to sharing (ACDI/VOCA 2014 [Q2 report]). Two FGDs with lead mothers and mother caregivers also said the ration could last two to three weeks, but a VHT member, a local council member, and two former RWANU staff said that the food rations could last only one to two weeks due to sharing, depending on the season. A contextual nutrition analysis conducted by Concern Worldwide found that community members said that those who do not share become victims of theft (Stallkamp, 2017). The food rations could have been slightly increased to better cover the calorie gap and compensate for sharing. At the same time the project needed to implement more sustainable solutions, such as increasing household incomes and access to improved quantity and quality of food. Positive consequences of food rations. Aside from increasing availability of food in the household, a positive consequence of the food rations cited by one of five FGDs with non-participants was that before the RWANU project, mothers would give young children brew as a means of providing them with food, but the food rations, as well as the training on food preparation, helped meet children’s nutritional needs so mothers did not have to provide their child with brew. Unintended negative consequences of food rations. A possible unintended negative consequence of the food distribution intervention might have been an increase in pregnancy to obtain the food rations; however, there are no data regarding this and only anecdotal information. Two district and health facility staff, one VHT, one FGD with mother caregivers, and four former RWANU staff indicated that they perceived that the food distribution intervention encouraged women to become pregnant; however, two interviewed individuals indicated that verification with data would be needed, because it was not clear whether this was an issue. In Year 2, to respond to these concerns raised early in the project cycle, the RWANU project included a package of integrated outreach services at food distribution points, which included a collaboration with Marie Stopes International to provide family planning services. Another potential unintended negative consequence may have been encouraging dependency on food rations and unintentionally discouraging crop production. Two interviewed former RWANU staff, one district health staff, and one FGD with non-participants shared concerns regarding food rations creating further dependency on food distribution and discouraging the production of food crops. Two former RWANU staff shared that food distribution has been conducted in Karamoja for more than 40 years, so it is deeply entrenched. The lack of household-level project component overlap noted above, particularly regarding crop production, marketing, and income generation, encourages the dependency on food rations, rather than alleviating it. Perceived effectiveness of food rations. Two of three FGDs with mother caregivers in communities that received RWANU food rations said that the rations helped improve the nutrition and health status of the children and themselves, and one of the FGDs, along with the VHT in the same community, said that due to receiving the rations it was no longer necessary for the participants to migrate to the green belt during the hungry season. The mother caregivers said that children ate three meals a day when they were receiving the ration. A VHT and an MCA commented that the best and most successful RWANU intervention was the food distribution, and two FGDs with non-participants indicated that the food rations helped women and children meet their food needs and become healthier. During a case study, a mother caregiver shared that the food rations helped reduce food insecurity, her child was able to eat regular meals, the mother had a reduced workload because she did not have to collect firewood to sell to obtain food, and her child was healthier than when there was no ration. A Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 43 district health staff member also reported that the food distribution was successful in reducing emergency levels of global acute malnutrition. RWANU staff felt that the rations might have kept the nutrition situation from getting much worse. Table 7E.4 in Annex 7E shows data from the food security and nutrition assessments conducted by UNICEF and WFP in June 2016, two years before the RWANU endline survey and six months before RWANU food distribution ended; in June 2017, one year before the RWANU endline survey and six months after the last RWANU food distribution; in January 2018, six months before the RWANU endline survey; and in July 2018, at the same time as the RWANU endline survey. The food security and nutrition assessments included anthropometric data for children 6–59 months of age, so the results are not comparable to the RWANU endline survey, which included anthropometric data for children under 5 years of age, but the information provides a rough comparison, showing that the prevalence of stunting does appear to have been slightly higher in the project districts of Nakapiripirit, Napak, and Moroto, compared to the endline value for the project area, and that food distribution may have kept the prevalence of stunting from getting much worse. However, this is impossible to say definitively without data from appropriate comparison or control groups during the corresponding time periods. The prevalence of stunting among children under 5 years of age in Karamoja was 35 percent in the 2016 DHS (UBOS et al., 2018). The prevalence of stunting in the RWANU endline survey and in the UNICEF and WFP food security and nutrition assessments remains classified as very high. Health services and education during ration distribution. One of five FGDs with mother caregivers said that during food distribution, health promoters sensitized the mother caregivers on health and nutrition topics, including IYCF, personal hygiene, child spacing, breastfeeding, and food hygiene, and gave food demonstrations and encouraged mothers to include eggs or sunflower seeds in the porridge. In addition, health staff at one facility shared that food distribution was combined with outreach, such as child immunizations. A former RWANU staff member shared that health education in group sessions at food distribution sites was better received if promoters used interactive approaches to present messages, such as songs and counseling cards, and minimized lecturing. The usefulness of drama groups at food distribution sites was also mentioned by health facility staff and a case study household. One of five FGDs with mother caregivers and two FGDs with non-participants shared that during food distribution, mothers’ cleanliness was inspected, and if mother caregivers did not have clean clothes, hair, or hands, they were not given a food ration. One former RWANU staff interviewed said that food distribution activities acted as community entry points during the implementation of the RWANU project and served as a source for participation in other project components. Annex 7E provides a brief description of the integration of health service provision and health messaging at food distribution points. Participant problems with food rations. Few respondents reported problems with food rations. One of five FGDs with lead mothers reported that some mothers stopped receiving the ration, and others received it up to the child’s second birthday. During one case study of a vulnerable household, the caregiver also shared that her child received food rations for two months and then she no longer received the rations and she was uncertain why. However, none of the other FGDs with lead mothers or mother caregivers in communities that received RWANU rations reported problems, and a case study household reported having a very positive experience, sharing that the ration was received each month and community mobilizers informed her two days prior to the distribution day. Challenges with logistics of food distribution. A few challenges mentioned by former RWANU staff included poor timing of food distribution with educational sessions due to delays in the arrival of the food at the distribution site, additional mobilization necessary by ACDI/VOCA staff to remind participants of the distribution, and some participants receiving rations from both RWANU and WFP; however, ACDI/VOCA and WFP worked to coordinate distribution days to ensure that it would be difficult to obtain rations from both entities. Various mobilization methods were used, including using Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 44 churches, radio, local council members, and RWANU staff to announce food distribution dates. Registration was updated by the RWANU monitoring and evaluation team, which received information from health promoters, who were informed by lead mothers. Acceptability of food rations. FGDs among mother caregivers in the three communities that received RWANU food rations found them to be very good. FGDs among mother caregivers in the two communities that had not received the RWANU food distribution, but instead had received the food distribution from the health facility, were dissatisfied with the health facility’s rations because the ration amount was less than the ration package received by the communities supported with food rations from RWANU, and less consistently available. The mothers said that they received different rations and ration sizes than the mothers in other communities, and they did not understand why they received differing project benefits. Both latter communities recommended that in future projects all participants be treated equally. Integrated program implementation to decrease dependency on food rations. USAID carried out a portfolio review of the RWANU program in 2013 and discussed sustainability of project impact and the methods used to achieve sustainability, including food distribution. According to 2013 ACDI/VOCA quarterly reports, senior RWANU project staff agreed that livelihoods programs needed to overlap with food distribution activities to promote sustainability and prevent dependence. At the end of the project, however, interviewed staff estimated that the overlap between SO1 and SO2 at the household level was approximately 30 percent, 72 rather than the targeted 75 percent. Conclusions The food distribution component of the RWANU project was highly valued by project participants but was not successful in helping the project achieve outcomes and impact targets on maternal and child nutritional status. Given the deteriorating food security situation in the project area and the extent of food sharing that occurred, the ration should have been evaluated and adjusted to better meet the kilocalorie and protein gap for pregnant and lactating women and young children. Lack of household￾level overlap in project interventions across the agriculture and marketing SO1 and the nutrition and health SO2 encouraged dependency on food rations. The project needed to more effectively integrate SO1 and SO2 activities to sustainably improve incomes and access to adequate quality, diverse foods to improve nutritional status of women and children. Project participants and health staff considered the interactive educational sessions and outreach services at food distribution sites as very good project approaches that were appreciated by participants, and the integrated services helped participants save time and increased their access to health services. Some households benefited from the food distribution by allowing them to stay in their communities rather than migrate in search of food or work, and among some households, the provision of food rations for young child consumption may have displaced dangerous behaviors, such as feeding brew to young children. However, there may have been unintended negative consequences of food distribution, including increased fertility or decreased child spacing during the time that the project component was implemented, but more data are needed to verify whether this was the case. Project participants strongly voiced their preference that categories of participants, for example, pregnant women, lactating women, young children, and households, receive the same food ration composition in each category. 4.2.7 Health Project design and technical interventions. The RWANU project design and results framework for the health component included the provision of support to health facilities in maternal and child health, including support for outreach services and to strengthen growth monitoring and promotion at 72 As mentioned in Section 4.1, the “overlap” was a constant source of confusion among project staff, with different understanding as to how it should be measured and different calculations as to the overlap figure attained. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 45 facilities and during outreach. It also included community-based screening and referral of children with acute malnutrition through the Concern Worldwide IMAM SURGE approach. The overall health support was intended to result in improved prevention and treatment of maternal and child illness. Annex 7F provides a detailed list of the technical interventions. Relevance of technical interventions. The technical interventions were very relevant to achieve the project outcomes. At the time the RWANU project was being designed, 42 percent of women 15–49 years of age in Karamoja cited serious problems accessing health services due to distance to a health facility (UBOS and ICF, 2012). Only 27 percent of live births were delivered in a health facility (ibid), 50 percent of health centers in Moroto and Napak lacked basic maternity equipment for a delivery (Wilunda et al., 2015), and 2.6 percent of children under 5 years of age had severe acute malnutrition (weight-for-height z-score <-3 SD). The prevalence of global acute malnutrition was 7.1 percent for children under 5 years of age (weight-for-height z-score <-2 SD); the level of global acute malnutrition was categorized as “medium” in terms of severity (range is very low, <2.5 percent, to very high, ≥15 percent) (UNICEF et al., 2018). Gaps in RWANU health interventions. All interviewed district health staff and one former RWANU staff member said that the project was inadequate in strengthening health systems and needed to focus on ways to improve the sustainability of activities, coverage, and outcomes after project closure. In addition, one unintended consequence of outreach shared by one district health staff was that health staff remaining at the health facility were overwhelmed with work when their colleagues were on outreach visits. One former RWANU staff member also mentioned the following gaps in RWANU health interventions:    Lack of adequate attention to malaria prevention, which affects child health and anemia; 69 percent of children 6–59 months of age in Karamoja had malaria, and 68 percent of children 6–59 months of age in Karamoja were anemic, hemoglobin <11g/d—the highest prevalence of malaria (UBOS and ICF, 2018) and the second highest prevalence of anemia in Uganda (UBOS & ICF, 2012). Inadequate attention to assessing, identifying, and treating women who were underweight; women’s underweight was almost twice as high at endline (46 percent), compared to baseline (27 percent) Inadequate staffing at health facilities, which affects feasibility to achieve targets; project targets for antenatal care visits and immunizations were not met, although coverage for each increased between baseline and endline. Perceptions regarding effectiveness of RWANU approach and methods. The RWANU project health interventions were considered of great value by all interviewed district and health facility staff (13 individuals), and they were seen as a significant contributor to improved health service coverage, quality, and service use among pregnant women and women with young children during the time of project implementation. District- and facility-level health staff considered the RWANU staff approach and methods effective during project implementation, in terms of improving health service quality and coverage and health service use by pregnant women and for children under 2 years of age. Staff perceptions regarding health service coverage and use are supported in part by the endline survey results, which showed that there was a statistically significant increase in the percentage of mothers of children 0–23 months of age who attended four or more antenatal care visits with their youngest child (75 percent at baseline and 84 percent at endline), although the project was just shy of achieving the LOA target of 85 percent.73 Project annual monitoring data showed that at the end of the project, 83 percent of children 12–23 months of age were fully immunized, but the project did not 73 Note that the IPTT indicator for which the target was set is slightly different from the baseline/endline indicator. The IPTT indicator is: “Percentage of mothers with children ages 0-12 months who had four or more antenatal visits when they were pregnant with their youngest child.” Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 46 achieve the end of project target of 90 percent. All five interviewed district health staff from Napak, Nakapiripirit, and Nabilatuk, and eight interviewed health staff from four health clinics in these districts, shared that RWANU project support for outreach activities, equipment, and staff training and coaching helped them achieve their coverage targets for antenatal care and child immunization and for difficult to reach and underserved populations. Factors associated with greater effectiveness of health interventions. All five interviewed district health staff and all eight interviewed health facility staff from four health facilities said that the factors in the implementation approach and the context that they associated with greater effectiveness in producing outputs during the time of the project included the following:    RWANU support for outreaches, such as fuel and vehicle maintenance, staff allowances, and in some cases, project vehicles Equipment for health facilities, including mattresses, delivery kits, and equipment for the maternity ward, weighing scales, microscopes, and solar panels Training provided to health facility management teams and staff74 Three-quarters of interviewed health staff (10 of 13) shared that the nutrition training that RWANU staff provided helped them learn how to conduct growth monitoring and promotion, measure MUAC, assess and manage malnourished children, and appropriately use Ministry of Health forms to capture the data. More than half of the interviewed health staff (7 of 13) also shared that the MCG model was very good for Karamoja because it was a good mechanism to reach community members and served as part of the referral pathway for malnourished children under the VHT. One district official recommended that MCG activities be captured in VHT reports. About half of interviewed health staff (6 of 13) thought that the RWANU-supported drama group educational sessions at health facilities, outreaches, and food distribution helped effectively convey health messages and, for example, reduced stigma for those with HIV and tuberculosis. A third of interviewed health staff (4 of 13) said that the community scorecard was a best practice because it empowered community members to speak freely about health issues and resulted in improved health service quality. Annex 7F provides information about the findings of a RWANU community scorecard impact assessment survey. Overall, the community scorecard was considered successful and is a best practice that should be replicated in the new development food security activities. A little more than a third of interviewed health staff (5 of 13) said that the quarterly joint Ministry of Health supervision and mentoring helped identify gaps and ideas and skills to fill gaps, such as correct completion of nutrition registers and referrals for malnourished children and pregnant mothers. However, one of five interviewed RWANU staff felt that much more supervision was needed, including of VHTs, and that given the demands on district health management teams (DHMTs) and, in some cases, the lack of functional DHMTs, there was a lack of joint supportive supervision to health facilities and outreach activities. There was also a need to ensure participation of district nutrition and maternal and child health focal points, because without the senior-level district management supervision, health facilities were not being optimally managed. District-level monthly coordination meetings were helpful, but about half of the time, the district did not follow through on action plans. Although support for supervision was considered good, more supervision and strengthening of DHMTs was recommended to improve quality of service delivery. Factors associated with lesser effectiveness of health interventions. Few interviewed district and health facility staff shared factors in the implementation approach and the context that they associated with lesser effectiveness in producing outputs. One district staff indicated that the RWANU project resulted in positive changes in access and use of health service and behavior change adoption, 74 The information in these bullets comes from open-ended question about the successes and challenges of the RWANU project and follow-up questions to probe more deeply about successes and challenges that were mentioned. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 47 but that one cannot expect the same level of change as in other areas of Uganda given the Karamoja context, including the recent history of insecurity and lack of education and high illiteracy, and that constant motivation for behavior change is needed for sustainable impact. Three-quarters of interviewed health facility staff (six of eight) said that the health facility suggestion box introduced by RWANU staff was either not used very much because of low community-level literacy or, in one case, because the health facility staff did not have the key to unlock the box. The 2017 community scorecard impact assessment also found some critical bottlenecks to efficient health service delivery, including understaffing, lack of staff accommodation, absenteeism, and drug stockouts. These areas will require improvement in new development food security activities. Community acceptance of RWANU project health interventions. Lead mothers in two of five FGDs shared that through RWANU-supported outreach services, immunization coverage, use of antenatal care services, and health facility births increased, compared to coverage and health facility use prior to the RWANU project. Mothers in two of five MCGs said that they valued what they learned through antenatal care visits at health facilities, the care and good attitude of health facility staff, and the availability of birthing kits and family planning at the health facilities. One of two interviewed parish chiefs said that the RWANU project successes included interventions in health, such as the supplies provided to health facilities. Overall, the RWANU health interventions appear to be well received and appreciated by project participants. WASH in health facilities. Three of four health facilities had a borehole, and staff contribute UGX 5,000 and 8,000 per year for its operation and maintenance. One health facility had piped water provided through UNICEF support. One health facility’s borehole was broken for six months and just repaired in September 2018. Two of four health facilities had well-maintained and hygienic latrines that allowed for privacy. In one health facility, the latrines were filthy, and there was evidence of open defecation in the latrine area. Some of the health facilities assigned individuals for latrine cleaning and others did not. Health center handwashing facilities were provided by the RWANU project or UNICEF, but soap was rarely provided due to the lack of primary health care budget, and in one health facility, the RWANU-provided handwashing facility was not replaced after it fell apart. Overall, work is needed to increase access to improved water sources and improved sanitation facilities in health facilities, and to improve hygiene practices. Conclusions The RWANU project approach and methods to improve health service quality, coverage, and health services used by pregnant women and for children under 2 years of age were generally appropriate and effective during project implementation. Support for outreach, equipment, and nutrition training were the most effective interventions to improve quality and coverage, and, together with MCG activities, motivated increased use of health services. The community scorecard was considered successful in increasing dialogue between community members and health facility staff about health service access and quality and is a best practice that should be replicated in new DFSAs. The RWANU￾supported drama groups also appear to have been effective in some areas for transmitting health and nutrition messages. The health interventions were well received and appreciated by project participants. Focus areas requiring improvement in new DFSAs include health systems strengthening and adequate mechanisms to ensure sustainability of health interventions and outcomes. Support for supervision was considered good under RWANU, but more supervision is recommended to improve quality of service delivery, as well as strengthening of DHMTs and their participation in supervision visits. Most but not all the health facilities recognized the need to maintain their WASH facilities in a manner to serve as a model for community clients using the health facility services, but more work is needed to ensure that health facilities have access to improved water and sanitation facilities and take measures to ensure hygiene. More attention is also needed to support health facilities to address underweight in PLW and Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 48 prevention and treatment of malaria in children, as well as advocacy and support to improve health facility staffing, absenteeism, and monitoring and logistics to ensure drug supplies. 4.2.8 WASH A significant WASH component was not initially included in the project design; given the context at the time,75 it was decided to focus attention on the community engagement required to establish the MCGs and support a “light” WASH approach through the MCGs by promoting handwashing and fecal disposal through the “dig and bury” method.76 After the MCGs had been established, increased focus on WASH became possible. By the third year of the project, there had been some savings in the RWANU budget, which was used to expand the WASH component by promoting the construction and use of latrines and to rehabilitate 24 boreholes for 64 communities in 3 out of the 4 project districts. RWANU did not implement WASH activities in Napak District because another UK Department for International Development (DFID)-funded project was already implementing WASH activities in Napak. The design of WASH activities was influenced by various studies undertaken by Concern Worldwide staff and specialist consultants, as described in Annex 7G. The need for WASH interventions is clearly illustrated by the baseline data (2013), which indicated that access to a handwashing station with soap and water was not widespread (4 percent), and diarrheal diseases were common, especially among young children (affecting 23 percent of children in the two weeks prior to the survey)77 (Table 9). Table 9: WASH Indicators at Baseline/Endline WASH Indicators Baseline 2013 Target for Project Participants Endline 2018 Difference (BL to EL) Percentage of households using an improved source of drinking water 41.9% 86% 36.9% -5.0 pts (ns) Percentage of households using improved sanitation facilities 4.0% 25% 1.2% -.2.8 pts *** Percentage of households with soap and water at handwashing station commonly used by family members 4.0% 6% 1.2% -2.8 pts ** Households with access to a sanitation facility (not necessarily improved) 11.3% N/A 18.7% +7.4 pts † Percentage of children under 5 years of age with diarrhea in prior two weeks 22.9% 20% 27.2% +4.3 pts (ns) Percentage of households practicing open defecation 87.9% 64% 81.0% -6.9 pts † BL=baseline, EL=endline, N/A=not available ns = not significant, † p<0.1, * p<0.05, ** p<0.01, *** p<0.001 At population level,78 WASH indicators decreased or registered no change, except for household access to any sanitation facility (whether improved or unimproved and typically a traditional pit latrine), which increased by 7.4 percentage points, though this was marginally significant (p<0.1). Regression analysis of 75 Former staff involved at the project design stage had been advised by other development actors that Karamoja was a “graveyard” of hygiene and sanitation and numerous approaches had ended in failure. At the time of project initiation, access to communities was extremely limited for security reasons, and there was a genuine concern that MCG formation would not be possible in this context, given the need for considerable community engagement time. Only after community access had improved, the RWANU teams were fully established, and communities were accepting MCGs and community collective action, was it possible for RWANU to be more ambitious in WASH activities. 76 Although lower on the sanitation ladder, the “dig and bury” method would—if implemented—yield results. 77 It should be noted that there was heavy rainfall and associated flooding during the endline data collection in June 2018; this is likely to have had an influence on the diarrhea indicator. There were also incidents of cholera in some areas. 78 Given that the expanded WASH component was not implemented in Napak District, it might be more accurate to exclude Napak from the overall sample. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 49 the PBS data showed no significant association between households that reported regularly participating in project activities and access to any sanitation facility, implying that there is no evidence to suggest that households participating in regular project activities had better access to sanitation facilities than those who did not participate. RWANU monitoring data indicated that by 2017, 93.6 percent of project households were using an improved drinking water source, exceeding the target of 86 percent for project participants; 8.8 percent were using improved sanitation facilities, and 3.5 percent had access to handwashing stations. The differences in project monitoring data and population-based data point to the potential for impact on project participants that would be fully reflected in a participant-based survey. Although the quantitative indicators above show little improvement and some decline, both the qualitative and project monitoring data suggest there was an increase in WASH knowledge as well as perceived improvements in some WASH practices among project participants based on local norms. Knowledge of handwashing. Project monitoring data for 2017 indicated that 75.1 percent of respondents knew at least three of the five critical moments for hand washing. Similar findings emerged from the qualitative data; FGDs conducted by the evaluation team revealed that mothers have knowledge of at least three out of the five critical handwashing times—before preparing food, before feeding the baby, and after latrine use. Despite this knowledge, the application of measurable WASH practices remained very low or even decreased at the population level. Handwashing practices and personal hygiene. PBS data indicated that handwashing79 decreased from baseline (4 percent) to endline (1.2 percent). Government figures show a large increase for Nakapiripirit, negligible improvements in Amudat and Napak, and a decrease in Moroto (Table 10). However, there was widespread consensus among the FGDs and KIIs that the project was successful in improving personal hygiene in many communities.80 For example, one of the health center staff reported that people, especially women, are now “smart, clean, and presentable,” which was not common previously. The apparent disparity between the quantitative and qualitative findings is explained by the fact that there are no quantitative indicators for personal hygiene, and that the improvements described by the qualitative findings did not necessarily meet the standards expected of the quantitative indicators. This is illustrated by handwashing practices; although the majority of the FGD participants washed their hands in some way, only half reported washing their hands with soap or ash; the other half said that they used water only. All FGDs and interviewees cited affordability and distance to market as barriers to the use of soap. Five KIIs with district health officers, project staff, community leaders, local council chiefs (LC-1) and VHTs reported that water access has improved, but the tippy taps promoted by the project were not appropriate (see explanation below). Table 10: Selected WASH Indicator Trends for Southern Karamoja, 2013–2017 District 2013 2014 2015 2016 2017 Amudat % handwashing 3.3 NA 3.1 11.9 4.0 % latrine coverage 5.2 5.4 10.3 17.1 21.0 Moroto % handwashing 8.0 NA 2.2 0.0 1.7 % latrine coverage 8.4 3.6 2.2 2.3 15.4 Napak % handwashing 10.5 NA 13.1 5.8 11.0 % latrine coverage 22.0 28.0 13.0 22.0 24.7 Nakapiripirit % handwashing NA NA 10.0 7.0 24.6 % latrine coverage 8.7 15.5 22.3 31.4 34.3 NA=not available Source: Ministry of Water and Environment sector performance reports 2013, 2014, 2015, 2016, 2017 79 Handwashing is measured by the physical presence of soap and water at a handwashing station used by the household. 80 Qualitative indicators of personal hygiene relate to the cleanliness of a person’s overall appearance, notably their fingernails, teeth, body, clothes, and also include their smell. Key informants were generally appreciative of the transformation in the cleanliness of Karamoja people compared to the past. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 50 Access to latrines. Access to pit latrines (whether improved or not) increased at the population level, from 11.3 percent at baseline to 18.7 percent at endline. Government data show that latrine coverage was 24.7 percent in Napak District and 34.3 percent in Nakapiripirit District in 2017 (Table 10), and district officials attributed this to the RWANU project. Discussions with all 6 FGDs and 29 KIIs reported that there was increased understanding about the importance and benefits of having a latrine.81 This was observed in Loitakwa village, Napak District, where latrine construction was ongoing in 80 households, the majority of the village, and largely influenced by the actions of VHTs and the community leaders in response to high diarrhea incidences. The evaluation team noted that although communities had put in a lot of effort in constructing latrines, the aspect of quality remained a challenge. The poor quality of latrines is attributed to inadequate prioritization for investment in latrines and widespread poverty among communities, combined with the nature of collapsing soils and frequent flooding during the rainy seasons. For example, three out of six FGDs revealed that latrines constructed in 2014 had been washed away by flooding. The key FFP indicator is “access to improved82 latrines,” which decreased from 4 percent at baseline to 1.2 percent at endline. The apparent decrease in this indicator may be explained by the survey sampling method (the PBS endline sample contained a majority [65 percent] of non-participant households); and the possibility that some latrines may have been washed away by flooding. 2017 project monitoring data recorded access to improved latrines as 8.8 percent. The poor quality and poor siting of the latrines constructed may have resulted in some improved latrines being destroyed or washed away by heavy rains and flooding. Again, it is worth highlighting differences in the measurement and expectations for WASH indicators. Government data, which measure access to any latrine, whether improved or not, show an overall increase from 2013 to 2017 for all districts (Table 10). In Karamoja, the overriding aim is to reduce the practice of open defecation, whether through “dig and bury” or the use of latrines. The RWANU project was able to declare 7 out of 64 villages “open defecation free” (ODF) during 2016–2017, and this is widely considered to be a significant achievement. There was a high prevalence of open defecation at baseline (89.7 percent), and the project failed to reach its target of 64 percent; behavior change and the construction of latrines by community members both proved to be extremely challenging. A much more intensive approach involving necessary follow-up visits as well as the support from local traditional leaders would have been needed to reach this target. For those villages that were declared ODF, it was acknowledged that the lack of continued follow-up and monitoring would cause some to revert to open defecation status. Even where latrines exist, cultural norms influence their use: among the Karamojong, a mother-in-law and son-in-law or father-in-law and daughter-in-law cannot use the same latrine. One-third of KIIs and two-thirds of FGDs suggested that communities are aware of the health risks relating to open defecation and that mindsets are slowly changing. Access to improved water sources. PBS data for households with access to an improved drinking water source83 showed no significant change from baseline (41.9 percent) to endline (36.9 percent); 81 Such awareness is unlikely to have existed among Karamojong transhumant pastoralist communities of the past, for whom latrine construction was not a priority because they moved from place to place in search of pasture. This cultural heritage may help explain some of the challenges in promoting changes in both attitudes and behaviors. 82 According to the WHO/UNICEF Joint Monitoring Program, a latrine must have washable floors/squat holes to qualify as “improved.” Traditional pit latrines tend to have dirt floors with squat holes made from logs and cow dung and are therefore unimproved. 83 Improved drinking water sources, as defined by the WHO Joint Monitoring Programme (JMP) for Water Supply and Sanitation are sources that are protected by the nature of their construction or through an active intervention from outside contamination, in particular, contamination from fecal matter. These sources include water piped into the dwelling, plot, or yard; a public tap or standpipe; a tube well or borehole; a protected dug well; a protected spring; or rainwater collection. An improved drinking water source must have water available year-round without experiencing interruptions of a day or longer in a two-week period. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 51 project monitoring data show that the participant-based target of 86 percent was achieved by 2017. Although the Annual Results Report for 2017 stated that hand pump mechanics assisted the communities with repairs in case of breakdowns, the qualitative data revealed that it took a long time (generally between 6 and 12 months for the villages visited) for broken borehole pumps to be repaired due to the bureaucracy involved,84 combined with an apparent lack of linkage between the water user committees and the pump mechanics.85 Findings from interviews suggested that none of the 24 rehabilitated boreholes are currently functional.86 Both the evaluation team and project monitoring found that the systems for borehole maintenance and repair are generally weak: there is low collection of user fees, low capacity among mechanic associations, and low availability of spare parts for repair (Mercy Corps, 2016). District officials further reported that most of the boreholes produced salty water due to the presence of minerals in the Karamoja region. Given the size of the project area and the level of need for borehole rehabilitation, 24 boreholes were considered to be a small number for the targeted communities under the project.87 Implementation approaches. In terms of implementation, the RWANU project mainly embraced the Community-Led Total Sanitation (CLTS) approach, but also used home improvement campaigns and the Participatory Hygiene and Sanitation Transformation (PHAST) approach (see Annex 7G for additional details). CLTS/PHAST facilitation at the village level was done by sub-county level health assistants, supported by community development officers, both of whom had been trained by RWANU project staff in CLTS and PHAST approaches. The facilitation was reported to have been done well, but the main challenge was the limited subsequent CLTS/PHAST follow-up. In addition, insufficient efforts were made to work with local community-based traditional leaders, who hold the influence needed to promote and support WASH activities. The project promoted tippy taps, which are low-cost, simple handwashing facilities, usually installed near the latrine. Part of the reason for the low PBS results for handwashing was because the tippy taps proved to be inappropriate and, therefore, unsustainable for various reasons that were reported by the FGD participants, including affordability, vandalization and misuse by children, theft, and destruction by the sun, livestock, and termites. Incentives, such as soap and jerrycans, were provided to encourage handwashing. However, key informants explained that the jerrycans are highly valued for grain storage and other purposes and consequently were not used as tippy taps. Because the plastic containers are so valued, they were reportedly often stolen when left at a tippy tap handwashing station. In addition to motivation and incentives (as described above in relation to CLTS/PHAST and soap and jerrycans), conditionality and subsidies were also part of the approach. The distribution of food rations was supposedly conditional on the adoption of personal hygiene practices, although there were no reports of food rations being withheld due to lack of personal hygiene. KIIs and FGDs reported that the monthly distribution involved a “health parade” to allow staff to undertake a visual check of both mothers and children to ensure personal hygiene practices by the food recipients. Awareness-raising sessions at the distribution centers were used to provide information and disseminate personal hygiene messages. The project provided hoes and axes to subsidize the cost of latrine construction, and these were given to the lead mothers with the expectation that they would be shared among the MCGs. Four out of six FGDs and six key informants reported that the tools were not shared, however. In other groups, lead mothers and their caregiver group members rotated the tools among themselves during 84 After the need for repair has been reported to the district level, if the district water officer has the necessary budget available for the repairs, there are bureaucratic processes involved in obtaining approval to spend the budget and procure the necessary spare parts, which are generally not readily available within Karamoja but must be purchased from Kampala or Mbale. 85 The water use committee members interviewed did not appear to know about the pump mechanic or how they should contact the mechanic; they instead reported faults to the sub-county office, which then informed the district water officer. 86 Note that it was not possible for the evaluation team to travel to each of the districts to be able to verify this. 87 The literature further states that boreholes tend to be “poorly distributed as a result of weak coordination and planning among development partners and government: while most communities do not have sufficient boreholes, other boreholes serve a smaller-than-recommended population” (Mercy Corps, 2016: 26). Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 52 latrine construction. The evaluation team learned that in the Karamoja region, it is mainly women who construct latrines, although 4 FGDs and 11 KIIs reported that this is gradually changing, and men are also becoming involved, partly due to the introduction of the MCA model. Conclusions Given the time and effort involved in promoting behavior change, combined with the importance of WASH for nutrition outcomes, it is unfortunate that it was not possible to implement the expanded WASH component more intensively from the start of the project, but the decision to begin with a “light” WASH approach was justified by the situation at the time. Promoting behavior change is complex and must take into account local cultural perceptions regarding the potential acceptability of particular practices and how best to motivate people to adopt them. As such, it is appropriate that the project undertook the formative research necessary to develop strategies for the design of WASH interventions. The project was effective in raising awareness of WASH aspects, as evidenced by the knowledge about WASH practices cited above, especially in relation to personal hygiene as evidenced by the qualitative data, and to home improvement hygiene as evidenced by the widespread construction and use of raised utensil racks. Personal hygiene was a prerequisite to receiving food rations and was seen to be more common than other practices. One reason for this might be that it represents a small, doable action, illustrating that step-wise change is indeed possible. Some WASH practices appear to have improved at the population level (e.g., the 7.4 percentage point increase in access to latrines and qualitative findings on personal hygiene), but not to the extent needed to show any significant positive differences in the FFP indicators. An increase in knowledge is a necessary first step to behavior change, but the widespread adoption of WASH practices has yet to take place in the project area. The lack of significant change in the WASH indicators was partly due to contextual factors (as summarized below), and also due to aspects of the approaches that were found to be inappropriate or not well-implemented. For example, the shaming aspect of the CLTS approach is not appropriate in Karamoja; the provision of subsidies and incentives was also not effective in the long term, often leaving group members demoralized by seemingly unfair distribution and unable to take ownership of their sanitation facilities; tippy taps were not appropriate; and the follow-up needed as part of the CLTS/PHAST approaches was not consistent. Both environmental context and cultural norms had repercussions on attainment and sustainability of project results. Collapsing soils and frequent flooding hindered construction and maintenance of latrines, leading to demotivation and fatigue among communities, despite their initial willingness. Innovative, locally appropriate technologies are needed for the construction of sustainable latrines in Karamoja. Access to improved water sources was not effective in the long term, mainly because the project did not create effective linkages with hand pump mechanics to attend to timely repairs at the grassroots level. The number of boreholes rehabilitated was relatively small in relation to the needs of communities, and systems put in place for the subsequent maintenance and repair of boreholes proved to be ineffective. 4.3 EQ4. WHICH PROJECT OUTCOMES ARE LIKELY TO BE SUSTAINED? 4.3.1 Adoption of Improved Smallholder Farm Management Practices The PBS data show a significant increase in the percentage of farmers using improved farm management practices; qualitative data also suggested that non-participants had learned and applied these practices, but surprisingly, only one FGD reported that seed of improved crop varieties provided by RWANU had been saved, shared, and adopted by other farmers. These findings suggest that non-participant farmers have continued to adopt improved practices after the end of the project. However, there were also Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 53 several instances in which seed of improved varieties was reported to have been “lost” due to drought or floods. If farmers wanted to re-acquire such varieties, they should be able to purchase seed from the agro-input dealers supported by the project. Participants in one FGD reported that they had contacted the agro-input dealer about a pest problem and had learned about new practices from a local agro-input dealer who had been supported by RWANU. The two agro-input dealers interviewed reported that there is demand for their services, and one, with support from another NGO project, was significantly expanding his business with additional inventory, a new shop, and additional storage. In general, however, there are simply too few agro-input dealers to make seed and other inputs widely available for purchase; the continued free distribution of seed also hinders their business. Former RWANU extension workers reported that they no longer provide regular extension or outreach services, but occasionally they have been called by farmers from their former groups who need advice on specific problems. If the former extension worker cannot help, then they contact the local government Production Officer. This suggests that there is demand for agricultural extension and improved technologies, as would be expected among a population that has relatively recently taken up diversified crop-based agriculture and where new land is still being opened up for cultivation, often with oxen and plows that had been provided by the RWANU project. Many FGDs and interviewees at all levels reported that the project ended very abruptly and left a gap, although the sub-county staff reported that, in some cases, the formal registration of groups has allowed them to target RWANU groups for subsequent government support. Both the former extension workers and government officers mentioned that their personal connections to group members has allowed them to promote links with other projects. The findings show that there is clearly a demand for knowledge about improved agricultural practices, but existing sources of private and government extension or agro-input services are limited in their reach. Free seed distribution hinders the establishment and sales of private service providers, but there is a recurring demand for seed due to the frequency of drought and floods. The continued adoption of improved farm management practices is likely to be sustained only for those farmers who take the initiative to seek advice (e.g., from other farmers) and have access to the services of agro-input dealers who can remain in business or to the services of former RWANU extension workers who remain in the local area. The knowledge that farmers have gained will be retained and applied where appropriate, but the likelihood of continued learning is limited; the project did not explicitly instill a culture of active learning or the capacity to innovate among farmers. On a positive note, the government is now targeting former RWANU groups for new projects or programs because they were registered at the sub-county level, and individual government officers are linking new projects and inputs to former RWANU groups and lead farmers because they know them personally. 4.3.2 Adoption of Improved Smallholder Livestock Management Practices Information provided by government veterinary specialists and the CAHW suggests that many of the WLGs still exist. For example, the CAHW met by the evaluation team currently supports two WLGs; the women call him when necessary, provided that they can afford to pay for his services. Despite the relatively small number of WLGs established by the project, there appears to be widespread demand for improved livestock management practices, as evidenced by the apparent increase in the percentage of livestock owners using animal shelters, practicing deworming and vaccination, and purchasing livestock medicines. WLG members, the CAHW, and a veterinary inputs dealer reported that there is demand for veterinary services and that livestock owners are able to pay for drugs. The veterinary inputs dealer reported that of the drugs sold, approximately 60 percent are for cattle, 30 percent are for sheep and goats, and 10 percent are for chickens. The trend for increasing livestock ownership (Aklilu, 2017) suggests that demand for veterinary services and medicines will increase, although there is also a need for more effective regulatory controls for the sale of animal medicines. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 54 4.3.3 Increased Linkages to Markets With the increase in security since demobilization, trade is generally increasing between sub-districts and districts. Cross-border trade is also taking place, with crops and honey going to Kenya, and cattle, sheep, and goats going to South Sudan (Aklilu, 2017). Key informants reported that RWANU played a contributory role in system development by bringing groups of farmers together with traders and input dealers to improve networks, helping increase the interest in crop-based production, and increasing production through opening new land. It was suggested by one District Officer that these networks are setting a foundation for potential future commercial growth for sorghum, maize, and cassava. RWANU also successfully created networks between agro-input dealers and various seed companies and agro-input suppliers in Kampala, Lira, and elsewhere for the supply of seed and other inputs. The continued free distribution of seed by government projects (e.g., Operation Wealth Creation), however, presents a major challenge to the private sector supply of seed and other inputs by agro-input dealers; farmers are unlikely to purchase seeds if they know that they will receive them for free. Members from at least 6 of the 24 beekeeping groups are still producing honey; those in Karita Sub-county were reported to be most successful because they are able to sell their honey across the border in Kenya. Not many of the beekeepers trained by RWANU are selling honey to Golden Bees because of transport constraints, suggesting that viable markets will not develop until the transport infrastructure in Karamoja has improved considerably. In addition, due to the remoteness of Karamoja in Uganda, honey from Karamoja may not be able to compete with honey sourced from other regions of the country. 4.3.4 Increased Access to Credit The VSLAs have proved to be very popular, and some have expanded their membership (e.g., from 20 to 25 members in the case of 3 groups met by the evaluation team). There is also evidence that new groups have been formed for non-participants and others would develop if they could access the specially designed savings boxes, suggesting that the VSLAs themselves are sustainable. VSLAs have clearly increased access to savings for all members, but not necessarily all have benefitted from increased access to credit. Formal registration of VSLAs with sub-county offices has allowed government officers to provide support to some groups through other projects, but links with formal credit institutions have not led to sustainable access to formal credit sources. RWANU established a link with the Microfinance Support Centre (MSC), a government-owned company that provides credit and business development services. Although MSC had developed a financial product tailored to VSLAs—a one-year loan of Sh. 5–50 million with 13 percent per annum interest—relatively few VSLAs were reported to have taken advantage of this product. Of the three VSLAs assessed by MSC in 2016, only one took out a loan. A former RWANU employee who is a member of a VSLA that had taken out an MSC loan revealed that the MSC interest rate is too high to be feasible for the VSLAs. MSC itself has relatively little experience in working directly with VSLAs; its main client institutions tend to be small- and medium-scale enterprises, micro-financial institutions, cooperative unions, and Savings and Credit Cooperatives (SACCOs). Because MSC lacks the staff to provide outreach services, it must rely on other organizations and projects to create linkages with village-based groups. RWANU supported 30 VSLAs to convert into three SACCOs, which are legal entities registered with the Ministry of Trade and Cooperatives and which, in theory, can access credit from formal institutions, such as MSC. The MSC key informant, however, stated that SACCOs are now in decline and the government is no longer promoting them due to fundamental challenges and lack of trust in their management structure. It seems unlikely that the SACCO model will provide a sustainable means for VSLAs to access credit. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 55 4.3.5 Improved Health and Nutrition Practices at the Household Level The RWANU project lacked a strong sustainability plan and strategies to sustain the roles of established community structures, specifically MCGs, and did not effectively communicate strategies for sustainability with participants, partners, and other key stakeholders. The project resulted in limited sustainability of the MCGs and limited sustained adoption of promoted nutrition and health behaviors at the community level. For the MCGs that said they were active, VSLAs motivated them to continue meeting but not necessarily to continue nutrition- and health-related activities. All five FGDs with lead mothers indicated that mothers were either no longer motivated to practice the behaviors learned during the RWANU project because they were not receiving food rations, although the absence of an MCG to encourage continued practice of behaviors could be an additional factor. Lead mother FGDs did indicate that there were limited practices that some mother caregivers were willing to continue, such as family planning and handwashing, although endline findings do not support this statement regarding handwashing. As noted in Section 4.2.5, five of seven FGDs with lead mothers and mother caregivers appeared to still be using their keyhole gardens. Few participants in FGDs had latrines, and few mentioned appropriate treatment and storage of water at the household level. Both MCAs approached during the qualitative data collection were still active; however, without more data it is difficult to indicate the extent to which both MCAs and MCGs were still functioning. Two of five FGDs of lead mothers indicated that they were still meeting and that what was bringing them together after the closure of the RWANU project was being part of a VSLA. A study of MCG sustainability would be useful to ascertain the facilitators and barriers for MCGs to continue to be active, including the impact of their registration at the sub-county level. Findings would inform project interventions for improving their sustainability. It is noteworthy that there was confusion among some MCG members and government officials regarding whether MCGs were registered with the sub-county—one group of lead mothers said they had been registered, another said they had not, one district health staff said MCGs had not been registered, and health staff at a facility said they had been registered (see Annex 7D for more details). Studies are also needed to better understand the reasons why some nutrition and health practices appear to have been sustainably adopted and others were not adopted or were no longer practiced after RWANU project closure. Three of five interviewed VHTs said that the RWANU project lacked a sustainability plan and strategies to sustain the roles of established community structures like MCGs, and one VHT suggested strengthening community structures like MCGs through saving groups. Two of these VHTs also recommended that traditional structures, such as elders and clan leaders, participate in decision-making for community development plans, including health and nutrition and in participant selection, because they are respected and influential. Two of nine former RWANU staff also commented that there was a lack of a clear sustainability plan for the RWANU-established community structures in the Ministry of Health system or local government structures, and lack of monitoring and follow-up to continue to motivate community groups to remain active. New DFSAs require detailed sustainability plans for each project component, and the plans should be discussed with all key stakeholders and monitored throughout the course of the project, with plan elements implemented on an ongoing basis throughout project implementation, not just at the end of the project. The plans, updated plans, and progress on plans should be shared and discussed at least annually, if not more frequently, with the USAID Mission. Although the VHTs continued to be active in all villages visited by the evaluation team, the focus that VHTs can place on WASH may be limited by the competing demands of government, United Nations, and NGO projects and programs. Better coordination and strategies are needed to prevent overburdening the VHTs and ensure that communities receive the continued support they need to adopt improved hygiene practices, either through the VHTs, MCGs, MCAs, or other community structures. Translating increased WASH awareness and knowledge into practice continues to be a challenge. One exception was found in Loitakwa Village, in Napak District, where the VHT reported that 80 households had recently been mobilized to build latrines. This initiative was stimulated by Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 56 realization of the link between malnutrition and the high incidence of diarrhea in the area, combined with the efforts of local leaders. The role of local leaders in the continued promotion of WASH activities is essential. Some of the drama groups established by the RWANU project to promote WASH messages reportedly continued to perform at community functions if they were paid for their services, but it was unclear the extent to which the groups were paid. Some drama groups remained active in part because they had also established themselves as VSLAs. It would be worthwhile to investigate the extent to which, as VSLAs, the drama groups continued to focus on sharing WASH messages through performances, or if their focus was now on the VSLA. MUAC screening by either lead mothers or VHTs, as part of the IMAM SURGE approach, appears to be the nutrition-related community-level activity most consistently sustained after RWANU project closure, at least among those communities visited during the qualitative field work. It would be useful to have consistent monitoring data on the approach over a broad area and an extended time period to better understand the successes and challenges in implementation. The VHTs appear to have no formal role or connection to MCGs in their communities, and links or support appears to be ad hoc. Two of five interviewed VHT members in two communities said that they were trying to support MCGs, but a third VHT in another community indicated that there was no formal role for VHTs with the MCGs after the RWANU project ended. There is a need to formalize VHT support for MCGs. Despite some challenges with the implementation of the SURGE approach and its sustainability, it should continue to be implemented, documented, and improved upon in Karamoja, given recent studies that demonstrate its effectiveness in low-literacy and low-resource settings (Blackwell et al., 2015; Grant et al., 2018). The continued high levels of global acute malnutrition in Karamoja (10 percent) (UNICEF et al., 2018) demonstrate the need for the approach. Goat distribution appears to have provided sustained benefits to participant households after project closure. As with the MCGs and adoption of promoted behaviors, it would be useful to conduct a study on the women’s groups that received goats to determine the extent of their sustainability and the long-term impacts on dietary diversity and feeding frequency for both young children and the mother caregivers who received goats. Study findings would help to strengthen interventions in new DFSAs. Despite the provision of food rations, there were no sustained positive outcomes in maternal or child nutritional status. New DFSAs will use cash transfer and vouchers instead of food rations, but strong interventions are needed to sustainably increase incomes and access to affordable, nutrient-rich foods for women and young children, and to improve knowledge and practice. Overall, continued promotion and adoption of improved WASH practices was not sustained in the project area, and endline results demonstrate poor adoption of the use of improved water sources, improved sanitation, and handwashing. Community structures, such as VHTs, MCGs, MCAs, and drama groups, need continued support to promote the uptake of WASH practices, and support for WASH practices from community leaders and traditional leaders is essential. Intensive approaches are needed to improve sanitation, including frequent community follow-up visits, support from local traditional leaders, and innovative, locally appropriate technologies for the construction of sustainable latrines. There is also a need to increase the number of borehole technicians, expand access to spare borehole parts, and improve linkages between water user committees and technicians to improve timely, quality repairs for broken down boreholes. 4.3.6 Improved Service Delivery for Prevention and Treatment of Maternal and Child Illnesses and Malnutrition Four of five district health staff said that the project ended abruptly and lacked an exit strategy (see Annex 7F for details). Staff commented that an exit strategy is needed from the beginning, including strengthening the government system so that government staff have the capacity to conduct activities Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 57 after project closure. One district staff member expected that RWANU staff would be co-located with health facility staff to facilitate learning and skills acquisition. Another district staff member recommended that future projects should use the last year of implementation to test the capacity of the health system to sustain what has been put in place and determine what support is still needed to strengthen the system, and that subsequent partners should then be able to build upon prior work, rather than repeat the same type of support. A third district health staff member recommended that future projects plan to leave behind a vehicle to enable outreach and transport for supplies, such as therapeutic food for children with severe acute malnutrition; be designed to directly support Ministry of Health work plans; and conduct joint planning and priority setting meetings between project decision￾makers and Ministry of Health staff to promote a sense of ownership by the Ministry of Health. A fourth district staff member recommended improving Ministry of Health ownership of community structures, such as the MCGs, by having the health unit in charge and the health assistant manage and supervise the MCGs; and by improving support for nutrition in the district development plans. Sustainable implementation of outreach activities is limited to those communities where health facilities have received support from other partners, and these are a small fraction of all outreach, leaving a gap in health service access in difficult-to-reach areas (see Annex 7F for details). There is no evidence that the community scorecard is still being used by the DHMTs or health facilities. Maintaining health care coverage and quality under the current resource constraints is a challenge for the DHMTs and health facilities. Equipment received under the RWANU project is likely to continue to be used to support health facility activities to the extent that equipment functionality can be maintained, but lack of resources for maintenance is a constraint. DHMT ownership, commitment, and capacity to sustain project activities is limited, due in part to lack of a well-defined and communicated sustainability and exit strategy from project inception. A district staff also shared that senior-level project staff participation in planning and priority setting can encourage local ownership. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 58 5. LESSONS LEARNED, BEST PRACTICES, AND RECOMMENDATIONS 5.1 LESSONS LEARNED AND BEST PRACTICES 5.1.1 Crop Agriculture Lessons learned—Project design:            It is essential that projects have a good understanding of the local development agro-ecological and socio-economic context, including the nature of drought and flooding, seasonal migration patterns, and socio-economic differentiation among farmers, so that agricultural interventions are appropriate to the local context. Agriculture should be better integrated with nutrition to maximize outcomes for both sectors. Greater attention should have been given to crop diversification through working with national and international agricultural research centers as well as private seed companies. National agricultural research stations lack capacity and require support. Prior to widespread seed distribution and provisioning, on-farm varietal trials should be implemented to allow farmers to test and identify the varieties that they prefer and that are adapted to local conditions. Lessons learned—Project implementation: Farmers in Karamoja have limited experience in crop-based agriculture. Training in seed selection and seed preservation should be provided in addition to the agricultural practices promoted by the RWANU project. For greater sustainability in the adoption of improved farm management practices, the capacity of farmers to learn, innovate, and share should be increased, and links to potential sources of new technologies and practices should be strengthened. Best practice: Coordination with sub-county officials and ensuring that they are aware of project groups and activities (e.g., through monitoring visits) is a best practice. Formally registering groups with local government offices increases the likelihood of continued support from other projects. 5.1.2 Livestock Lessons learned—Project design: Women are both competent and accepted to own and manage goats and to have control in decision-making over the sale of offspring and the use of finances. They are also capable of beekeeping and honey production. Social implications of targeting mechanisms should be considered in resource transfers, such as goat distribution for mothers of young children to avoid the possibility of early marriages and abandonment of family planning practices. The training of CAHWs has the potential to increase the availability of improved livestock services, but this must be combined with activities to increase the demand for such services (e.g., training on improved practices for livestock owners). Lessons learned—Project implementation: VSLAs for the WLGs are helpful in managing the money needed to pay for the treatment of sick animals and for regular deworming and vaccinations. Galla goats are an appropriate breed for the Karamoja region and have high milk production. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 59 Best practices:               The livestock component benefits from close involvement of local government technical specialists. Individual goat ownership (rather than group ownership) helps increase the sense of individual responsibility; group membership and savings help participants organize treatment for their animals and share milk among members. Gender training for WLG members and their husbands allows women to retain control over goats and related decision-making. Beekeeping training with integration of cross-cutting issues, especially gender and conflict, appeared to support multi-sectoral impacts. 5.1.3 Market Development Lesson learned—Project design: The design of market development interventions must be based on a good understanding of the local context (including agro-ecological potential, physical infrastructure, and labor markets), and, based on socio-economic differentiation and proximity to trading centers, must recognize the different market-related needs of the population. Lessons learned—Project implementation: The challenges of infrastructure and the cost of transport must be taken into consideration to ensure the sustainability of market linkages; producers located close to market towns or the Kenyan border have an advantage. In general, effective associations take time to establish and must have strong leadership and viable business plans. Coordination and advocacy among government and donor-funded projects is necessary to agree on appropriate mechanisms for agro-input supply so that the efforts of one are not undermined by the other. Best practice: Ensuring that criteria for the selection of individuals to become agro-input dealers include technical knowledge of agriculture and livestock-keeping, whether gained from personal experience or formal training, is a best practice. 5.1.4 Savings and Loans Lessons learned—Project design: Awareness-raising for options for income-generating activities suitable for men and women should be incorporated into the VSLA approach to avoid too much emphasis on beer-brewing. The success of the VSLAs suggests that they should be scaled up so that more people can benefit (e.g., through more groups), and more training should be offered to local blacksmiths so that affordable savings boxes can be made available. Alternative formal institutions and/or loan products must be identified and established for linkage with VSLAs because SACCOs and MSC proved not to be effective. Lessons learned—Project implementation: VSLAs have proved to provide a sustainable source of credit to VSLA members. SACCOs are not sustainable and therefore are not a good link for VSLAs; the high level of interest charged by MSC makes it unattractive to many VSLAs; loan products with lower interest rates are needed. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 60 Best practices:  Flexibility in contributions among individuals and according to season should be encouraged.  VSLAs should register with sub-county offices and VSLA awareness-creation opportunities should be supported among government officers so that VSLAs can benefit from other forms of government support. 5.1.5 Nutrition Lessons learned—Project design:        Project design should include holistic, integrated approaches that help participants overcome financial constraints to behavior change adoption. Specifically, project design and implementation should ensure that participants benefit from project interventions that increase income through acquiring skills and knowledge. Participants should be able to improve knowledge and adopt appropriate behaviors that improve maternal and child health and nutritional status without adding undue labor burdens, especially on mother caregivers, who need adequate time and options for childcare and self-care. Couples counseling is an integral activity to include in project design to prevent conflicts between men and women over changes in women’s access to and control over income. The IMAM SURGE approach included lead mothers screening children using MUAC tapes to detect children with severe acute malnutrition and moderate acute malnutrition and refer them for treatment. A high proportion of women, especially adolescents, suffer from underweight in Karamoja, and a similar effort using MUAC tapes to detect and refer for treatment women with underweight should be considered in project design. Adolescent girls are at a critical stage in their life cycle that demands attention to ensure their optimal nutritional status, health, and well-being. Given that there is a relatively high prevalence of underweight among adolescent girls in Uganda, project design should include appropriate activities to address the nutrition and health needs of adolescent girls. The goat distribution activity has good potential to improve dietary diversity, meal frequency, and child and maternal nutritional status, and an expansion of the goat distribution activity should be considered in project design. Careful monitoring of outcomes and impact is required to ensure that a greater number of vulnerable households benefit. Special attention should be given to hygiene and sanitation related to children’s exposure to goat feces. The design of kitchen gardens requires consideration of women’s time, labor, and resource constraints, as well as agro-climatic conditions, such as drought and flooding. Participants and key stakeholders should work together to adapt designs to specific, community-level conditions and participant needs. The time, effort, and resources placed on kitchen gardens must be balanced by outputs that make kitchen gardens worthwhile to participants. All of the latter should be monitored and discussed with participants on an ongoing basis, and adaptations should be made to fit participant needs and expectations. Lessons learned—Project implementation: Social and behavior change approaches require not only activities to support individual behavior change, but also social and behavior change at the community level, including community mobilization for wider community participation and ownership of project interventions and promoted health, nutrition, and WASH practices. The individual-level social and behavior change interventions implemented as part of the MCGs require quality training with a limited number of key behaviors for adoption, and sufficient time to review, reflect, and implement new behaviors and strategies for overcoming obstacles to behavior adoption. To support improved adoption, key messages and behaviors need to be timely, relevant, and aligned with mother caregiver life cycle needs. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 61            Increasing the number of project health promoters would decrease the ratio of health promoters to lead mothers and increase opportunities for more supervision of lead mothers during sessions with mother caregivers and home visits. Project staff need adequate time to reflect on and discuss project processes, outputs and outcomes, and options for modifying interventions to better reach targets and improve impact. Time constraints due to income-generation needs and during planting and harvesting season limited attendance by some mothers to MCG meetings. The project staff demonstrated adaptability by altering the MCG approach, including more focused training during times that were more convenient for women. To avoid conflicts, and in coordination with local leaders and elders, participant incentives should be carefully planned to ensure equitable distribution of project benefits. Although incentives for lead mothers and their groups may initially help motivate lead mothers, inconsistent application to groups over time can lead to conflicts within the groups. It is important for projects to identify sustainable ways to support groups and motivate them, without causing conflicts and dissatisfaction. Local food preservation has been practiced by mother caregivers in the project area in the past. Opportunities to build on this knowledge with new and improved food preservation techniques, such as solar drying for MCGs, should be offered. Evidence shows that women’s workload in the Karamoja context can keep women from having sufficient time to provide appropriate care for young children. To allow time for mother caregivers to provide appropriate child feeding and care practices, women’s workload should be monitored during project implementation, and project activities adapted to diminish women’s workload through labor-saving technologies while teaching mothers and other caregivers appropriate IYCF practices, ensuring quality childcare options, and promoting appropriate child spacing/family planning. Malaria is a serious problem in the project area and requires attention for malaria prevention and treatment among both children and pregnant women, either through DFSAs or linking to projects that address malaria programming. In some project areas, family migration in search of labor kept families from benefiting from project interventions. Considering their unique situations, strategies should be included for reaching migrating families with nutrition, health, and behavior change messages. The MCA approach resulted in some improvements in men taking on more responsibility and reducing conflict in the home, and there is room for the approach to be improved, including the addition of more training topics. Cultural norms regarding the relationship between younger and older men in Karamoja may have limited the effectiveness of the MCA approach, and including elders and older men as MCAs may also strengthen the approach. The MCG and MCA approaches should be designed to work together, and participants should be trained together to improve project outcomes. Lessons learned—Project sustainability: Collaboration with partners is needed to design mechanisms to motivate and sustain community structures and their functions, like the MCGs and MCAs, after project closure. A well-defined sustainability strategy for community structures, including formal linkages to existing structures, such as the VHTs and the sub-county, in collaboration and communication with district, sub-county, and health facility staff and community leaders and elders, is needed, as well as ways to motivate continued adoption of critical practices. VSLAs foment group cohesion and motivate groups to keep meeting; they, or similar mechanisms, can be important components of a sustainability strategy for community groups.           Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 62 VHTs have the potential to play a vital role in sustainably supporting MCGs after project closure, and their role should be considered and discussed with key stakeholders from project inception and design. Projects need to fully address the challenge of motivating mother caregivers to sustainably continue to apply behaviors over time after project closure. Appropriate mechanisms need to be established to ensure motivation of participants to consistently practice desired behaviors. The specific mechanisms need to be designed, tested, and adapted as needed throughout project implementation. Best practices: The use of MCGs is a best practice. The MCGs serve as a platform for working with women in the community and are considered a useful mechanism by the Karamoja Ministry of Health staff and community leaders. Recent research shows that the care group model is a cost-effective approach, and there is strong evidence that use of care groups can reduce childhood malnutrition, the prevalence of diarrhea, and child morality (Perry et al., 2015). In the case of RWANU, the behavior change interventions and messages transmitted through the MCG approach, including links to SO1, can be improved, as indicated in the lessons learned above. Conducting special studies to investigate the reasons behind poor monitoring survey outcomes, and implementing recommendations coming out of these special studies, is a best practice that was implemented by the RWANU project. The IMAM SURGE approach, with lead mothers screening children using MUAC tapes in the community, is a best practice. There is room for improving the approach as it was implemented under RWANU, but the approach should not be discarded. New DFSAs should build on the approach and improve it. Couples training to prevent and manage potential conflict between men and women and husbands and wives with the introduction of new project activities—and especially with income-generating activities—is a best practice and should be continued, with adaptations and improvements as needed, in new DFSAs. The adaptability of the health and nutrition components of the project to ensure ongoing learning and better understanding of the context and local conditions was a best practice. For example, changing the MCG cycle length from four to six weeks, promoting linkages with IMAM and treatment programs by training lead mothers to assess MUAC and refer children to the VHTs, and adjusting the initial counseling card design from drawings to photographs that the communities could better relate to demonstrated successful attempts to improve the project. Formal registration of community groups with the sub-county was a best practice, but it should be conducted earlier in the project cycle, communicated clearly with MCGs and partners, and groups should be supported to take full advantage of the registration during project implementation as a part of strengthening sustainability. Distribution of goats for vulnerable households to improve dietary diversity and frequency and have a source for cash in emergencies was a best practice and should be expanded, adapted, and improved upon as needed in new DFSAs, with attention to proper hygiene and sanitation so children do not come into contact with goat feces. 5.1.6 Food Distribution Lesson learned—Project design: Overlap and layering of project interventions for households—across agriculture, marketing, nutrition, and health—can help households strengthen the skills and capacities that will enable them to increase incomes and access to improved diets and in turn help women and children to be healthy and grow well and decrease dependence on food rations. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 63 Lessons learned—Project implementation:            Closely monitoring and quantifying, in collaboration with partners, potential unintended negative consequences, such as an increase in the number of women and girls getting pregnant to access project benefits, such as food rations, is useful to clearly understand the relationship between the project design and unintended outcomes and allow for appropriate, timely, and effective adjustments to avert negative consequences. Project participants in respective participant categories should receive the same food ration or package of services to prevent conflicts and assist in achieving project outcomes and impact. Interactive educational sessions will better engage participants and may result in improved recall of the educational sessions and their content. Periodic analysis of the food ration size and content is warranted during project implementation to ensure the ration continues to fill the participant nutrient gap when the food security situation changes or deteriorates over time, or when sharing of rations prevents targeted participants from benefiting from ration provision because of cultural pressure for sharing and potential negative repercussions when donated food is not shared. Food ration distribution that is conditional on mothers and their children demonstrating appropriate hygiene can help support adoption of improved hygiene practices during project implementation. The integration of health services and food distribution can improve participant access to health services, save time for both participants and health staff, and reduce workload among community volunteers at the village level. Best practice: Integrated outreach in conjunction with food distribution services is a best practice, including family planning services. 5.1.7 Health Lessons learned—Project design: Various factors that affect the health and nutritional status of women and children need to be adequately addressed in the project design phase, including interventions to address malaria and anemia in children, and underweight in PLW. Addressing additional factors that affect women’s and children’s health and nutrition requires appropriate resource allocation and collaboration and coordination with district and health facility staff. Understaffing at health facilities limits the potential to achieve project targets. Implementing partner and donor advocacy is needed with health authorities to positively influence resource allocation and staffing levels. The project design requires a strong health system strengthening component based on a health systems assessment of supplies, equipment, logistics, staffing, and training needs, if a separate health systems project is not already operating in the area. A strong health systems strengthening component requires an appropriate budget and coordination and collaboration with district and health facility staff. Lessons learned—Project implementation: RWANU project support for outreaches, equipment for health facilities, training provided to health facility management teams and staff, and the application of the community scorecard were essential in helping health facilities reach coverage targets and improve the quality of health services. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 64              Outreach activities can place an increased burden of work on health facility staff who remain at the health center during the outreach activity, and measures need to be discussed and taken to decrease this burden. Drug stockouts are a growing problem, and districts and health facilities need support to ensure timely and effective logistics systems to prevent stockouts of essential drugs and therapeutic food for acutely malnourished children. Supervision and mentoring of health facility staff in facilities and at outreaches, and of VHTs, in collaboration with DHMTs, is extremely important and should be a priority for projects. WASH in health facilities must be exemplary if communities are to be expected to practice appropriate WASH behaviors at the health facility and replicate them in their homes. Support for WASH in health facilities should be an integral component of project WASH activities. Lessons learned—Project sustainability: A well-defined sustainability and exit strategy for the health systems strengthening component of the project is essential. It must be communicated with key stakeholders at the beginning of the project, monitored, and adjusted during project implementation to ensure sustainability of activities, coverage, and outcomes after project closure. Equipment donation is highly valued by districts and health facilities, but measures should be taken to help the Ministry of Health ensure maintenance of donated equipment, such as weighing scales. This is challenging given DHMT resource constraints, and advocacy will be needed for support from district nutrition coordination committees and district leadership. Support for the DHMT is necessary to improve its capacity and motivation to participate in supportive supervision and follow through with agreed upon action plans. This helps the project achieve health objectives and targets and improves district ownership of results. Senior-level project decision-makers should participate in joint planning and priority-setting meetings with district-level health staff and health facility staff to encourage local ownership of activities. Best practices: Integrated outreaches are a best practice because they help health facilities efficiently meet coverage targets and provide isolated and vulnerable populations with essential preventive and curative health services. The community scorecard, or similar intervention, is a best practice because it empowered community members to speak freely about health issues and resulted in improved health service quality. Best practices for WASH in health facilities include access to piped water or a functional borehole on the health facility premises, separate latrines for men and women that are cleaned daily, and handwashing stations with soap near the latrines. 5.1.8 WASH Lessons learned—Project design: Future integrated projects in Karamoja should prioritize the WASH component in the initial design by ensuring realistic budget allocations. The recognition of the central role of WASH in Karamoja is vital in addressing health and nutrition needs among targeted communities. There is a need to identify innovative, affordable, and sustainable WASH technology options for latrines, hand cleansing agents, and handwashing facilities that are appropriate to the Karamoja context in terms of soils and weather patterns, availability of local materials and technicians, and cultural preferences.           Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 65 The gender and cultural issues relating to the construction and use of latrines need to be studied further and addressed appropriately to improve the quality and usage of latrines. It is mainly women who construct latrines in Karamoja, although this is gradually changing, and men are also becoming involved. For cultural reasons, male and female in-laws cannot use the same latrine. Karamoja has an extremely low level of good WASH practices. Behavior change takes time, and expectations and targets must be realistic, based on the low starting point, the widespread dependency syndrome that influences attitudes to behavior change, socio-cultural factors, and the physical and ecological environment. Unless more durable latrine materials and designs can be found, realistic targets should include community-based plans for re-building latrines every few years. Lessons learned—Project implementation: The shaming aspect of the CLTS approach is inappropriate in Karamoja; agencies that use the CLTS approach for WASH must use alternative methods to promote the proper disposal of feces. Creating knowledge is relatively straightforward; promoting change in practice is considerably more challenging. Continuous motivation (e.g., through income-generating activities, role models, exemplary leadership, and internal and external learning visits), sensitization and frequent reminders and follow-up from various actors (e.g., VHTs, local leaders, sub-county officers, local chiefs, health staff, and implementing agency staff) are essential to promote behavior change; this should be included in the initial design and adequately budgeted. This is especially important in attaining and maintaining ODF status. Capacity for the management and repair of borehole pumps needs to be strengthened, ensuring that the necessary linkages between local water user committees and pump mechanics are functional, spare parts are available, and district-level maintenance systems are effective. Water quality analysis needs to be done to assess whether boreholes meet the recommended parameters of World Health Organization guidelines before they are commissioned. In situations in which the nature of ecology proves to be salty, there is a need to adopt the use of polyvinyl chloride pipes and stainless-steel pipes that do not react to common chemicals. Lesson learned—Project Sustainability: Community structures, such as VHTs, MCGs, MCAs, and drama groups need continued support, including from community leaders, to promote the uptake of WASH practices, both during project implementation and after the project closes. Best practices: The ability of the project to incorporate an expanded WASH component shows good adaptability in response to the need for intervention changes to maximize positive outcomes. Barrier analyses and other formative studies, as well as pre- and post-training surveys, are all helpful in ensuring that the right issues are being addressed. This is important in providing an appropriate roadmap for planning, implementing, and monitoring WASH activities. The Ministry of Health should develop and implement best practice guidelines for WASH facilities and behaviors at health centers. WASH in health facilities should be exemplary if communities are to be expected to practice appropriate WASH behaviors at the health facility and replicate them in their homes. Support for WASH in health facilities should be an integral component of project WASH activities. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 66 5.2 RECOMMENDATIONS 5.2.1 General The Karamoja context presents an extremely complex and challenging context for donor-funded aid interventions and must be fully understood if development interventions are to be successful. This involves a comprehensive and detailed understanding of various dimensions (agro-ecological, political￾historical, socio-economic, and cultural) so that project design, timeframes, targets, and implementation approaches can be tailored accordingly. In Karamoja’s post-conflict context with under-developed markets, low capacity of formal institutions, and changing livelihood patterns, projects require longer timeframes (e.g., 7–10 years rather than 5 years) to achieve substantial impacts. Features of the socio￾cultural context must be factored into project design, for example, socio-economic differentiation, changing livelihood strategies, migration and population movement, gender relations, alcohol consumption, and the dependency syndrome. The complexity and changing nature of livelihood practices, markets, government and other donor interventions, social change, and weather events make it essential to use a comprehensive conceptual framework to ensure project coherence and consistency, regardless of potential changes in leadership and advisors. A clear conceptual framework with attention to risks and roles of other actors helps provide clarity in who should be targeted for different activities and ensure sufficient scale to achieve impact. The development and periodic review of conceptual frameworks are useful tools in collaborating and communicating with partners and stakeholders. Contingency plans and budgets for “bad” years caused by drought or flooding must be built in to the project design. A close working relationship with both government and traditional structures is essential to ensure coordination, uptake of improved practices and behaviors, and long-term sustainability. Funding and mechanisms are needed to support government-led cross-agency coordination structures, as well as collaborative studies and assessments to build consensus across different agencies regarding intervention approaches. Future projects should work more closely with traditional and government structures to increase uptake and sustainability and should include local actors in workshop and program planning, both for inputs and communication, understanding, and increased effectiveness. Local stakeholders must be engaged from the start in the design and planning of exit strategies, which must be developed as a part of core program design. To gain greater insight and learning from implemented strategies, approaches, and activities, projects should ensure that monitoring, evaluation, and learning systems across the life of the project can characterize households to understand which types of households participated in which activities, and how different household types (e.g., based on project participation and relative wealth and food security status) benefitted in term of adoption, outcomes, and impacts. Realistic targets must take into account transitional contexts. For example, large-scale population-level outcomes should not be anticipated where local institutions, government structures, and market systems are weak or poorly developed. Examining results through PBSs is an effective means of identifying wider impacts of a project and a wider program portfolio, but this can be insufficiently granular if activities have limited spill-over effects or the distinctions between participants and non-participants is unclear or not analyzable. 5.2.2 Livelihoods and Food Security (SO1) In the Karamoja context, the concept of resilience needs to be central to project design, implementation, and monitoring. An appropriate balance between crop-based and livestock-based interventions (including related market development interventions) must be determined according to a better understanding of the resilience of specific livelihood strategies in relation to variations in the local agro-ecological context, as well as people’s aspirations for the future. Appropriate climate-smart agriculture approaches should be incorporated into agricultural interventions, including efforts to minimize farmers’ costs and risks. Explicit acknowledgement of and engagement in risk reduction, Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 67 mitigation, and management strategies at the project, community, and household levels would likely improve project performance. Future projects should take advantage of project learning and the resources of the Karamoja Resilience Support Unit and its extensive studies of the Karamoja region, which provide detailed insights and both practical and policy recommendations. A much closer integration of livelihoods, food security, and nutrition sectors is necessary to ensure that households have access to the increased income needed for improved production and enhanced nutrition, and that the production, preservation, and use of nutritious foods are promoted through agricultural interventions. The success of the VSLAs in creating a savings culture now needs to be expanded by identifying and promoting a range of diverse opportunities for income generation suitable for men and women, including youth. It is essential to ensure male engagement with discrete and focused strategies to support program outcomes, such as through the MCA approach, and to prevent conflict over women’s increased access to income. Market systems development should adopt inclusive or pro-poor approaches and must be based on a sound understanding of socio-economic differentiation and the various constraints and risks that exist within the Karamoja context (e.g., remoteness, poor transport infrastructure, aid handouts, high￾risk crop production). 5.2.3 Health and Nutrition (SO2) It is necessary to implement social and behavior change approaches that involve both community engagement for wider community consensus and support for behavior change adoption and individual￾level behavior change approaches. Promoting a limited number of top-priority behaviors and practices and doing so through cross-agency/entity campaigns will create momentum and wider uptake. Including adolescent girls as a target group for behavior change is essential for their adoption of improved health and nutrition practices. Conducting in-depth studies will assist project staff to better understand and promote effective behavior change strategies, including studies on: (1) motivators and inhibitors to sustained activity by MCGs and MCAs after RWANU project closure; (2) adoption of IYCF practices, considering cultural beliefs and social and economic constraints, particularly those that may be related to care and feeding of boys versus girls; (3) long-term impacts of the goat distribution on dietary diversity, feeding frequency, and nutritional status for young children of mother caregivers who received goats, as well as any negative outcomes (e.g., potential illness due to exposure to goat feces); (4) facilitators and barriers to the adoption of family planning practices; (5) facilitators and barriers to the adoption of WASH practices; and (6) sharing of food rations or food from vouchers among family, extended family, and other individuals, and intrahousehold food allocation, to determine the best way to ensure that critical foods reach PLW, adolescents, and young children, including food obtained through vouchers. Application of these study findings to new DFSAs can assist the projects to overcome barriers and achieve targets. Pay special attention to barriers related to women’s workload and time for childcare, as well as the impact of project activities on the latter, documenting impacts that increase or decrease women’s work and time for childcare, and taking appropriate measures as needed to prevent negative repercussions will enhance project outcomes and impact. The current DFSA has a unique opportunity to test and refine varied modalities (e.g., cash transfer, voucher, or other) to improve nutrient intake for vulnerable women and children, learning from and exchanging information and experiences with institutions or organizations in other countries that have tested these modalities in settings similar to Karamoja, and building on studies conducted under RWANU (e.g., barrier analysis on consumption of animal source foods). A well-defined sustainability strategy for community structures, including formal linkages to existing structures, such as the VHTs and the sub-county, in collaboration and communication with district, sub￾county, and health facility staff and community leaders and elders, is necessary to motivate and sustain Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 68 community structures and their functions, like the MCGs and MCAs, after project closure, as well as continued adoption of critical practices. Continued support to the DHMT and health facilities in health systems strengthening, outreach, equipment, and training is needed if a health system strengthening project with a focus on nutrition and maternal and child health is not already functioning in the project area. DFID has a Karamoja nutrition program (2017–2021) and a malaria program (2017–2022), in addition to other health programs, such as family planning. DFID’s programs have a health system strengthening approach. USAID partners should collaborate with DFID partners to define a common approach and understanding to health systems strengthening and the roles and responsibilities of each organization. Developing strong coordination and collaboration between health systems strengthening projects and the community nutrition and maternal and child health activities implemented by DFSAs is of critical importance. In addition to the types of health services assisted under RWANU, health facilities should be supported to address underweight in PLW and prevention and treatment of malaria in children, and advocacy and support should be provided to improve health facility staffing, absenteeism, and monitoring and logistics to ensure drug and therapeutic nutrition supplies. The implementation of the community scorecard or a similar tool with proven effectiveness should be supported, along with its sustainable use by incorporation into the Ministry of Health system. Projects need to include an exit strategy for the health component in the project design, communicate it clearly with all partners and stakeholders, discuss it regularly with partners, and adapt it as needed, ensuring that it includes strengthening the government system so government staff have the capacity to conduct activities after project closure. Projects should also work closely with the DHMT to provide supportive supervision to health facilities and work to increase DHMT ownership of interventions. An intensive, coordinated, and multi-partner approach to WASH is needed in Karamoja if positive change is to be realized. The Ministry of Health and local government structures require support to lead and coordinate such a campaign, with strong support from the Ministry of Water and Environment (responsible for water and sanitation in public places) and the Ministry of Education and Sport (responsible for school sanitation). At the institutional level, political, administrative, and traditional structures must be involved to create the necessary enabling environment to support effective WASH coordination, implementation and follow-up. WASH-related district ordinances and local by-laws have been seen to be effective in other districts,88 promoted and enforced by sub-county officers, VHTs, and local leaders to ensure ownership, sustainability, and accountability. Lessons and best practices (as detailed above) must be adopted for effective behavior change approaches, including the involvement of local traditional leaders for necessary motivation and follow-up. Consistent WASH messaging needs to be incorporated into a wide range of government sectoral programs, interventions, and institutions (e.g., youth empowerment program, women’s entrepreneurship program, social protection program, wealth creation program, schools, and health facilities) to create a widespread movement to promote and reinforce appropriate WASH behaviors. There is need to strengthen public-private partnerships for innovative sanitation technologies and maintenance of WASH facilities. Implementing agencies must work closely with government structures and the private sector to identify and make available appropriate, low-cost WASH technology options (e.g., cleansing agents, handwashing facilities, latrines, and borehole pumps) suitable for the Karamojong environmental and socio-cultural context, together with the necessary associated systems for effective marketing,89 maintenance, and repair. 88 In West Nile, for example, the Uganda Sanitation Fund has supported the establishment of effective WASH-related by-laws in Moyo, Nebbi, and Arua Districts. Although the CLTS approach does not promote the use of by-laws, there is clearly a need for additional efforts in this regard. 89 There is currently no policy on sanitation marketing in Uganda; efforts to promote and create public-private partnerships must be supported and strengthened. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 69 6. REFERENCES ACDI/VOCA. 2012. Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU). Project Proposal. Partial Submission, Revision 3, August 13, 2012. ACDI/VOCA. 2013. Resiliency through Wealth, Agriculture and Nutrition (RWANU). FY 14 Quarterly Report: October–December 2014. ACDI/VOCA. 2013a. A Cost of the Diet analysis in Karamoja, Uganda, July 2013. ACDI/VOCA. 2013b. PREP: Pipeline Resource Estimate Proposal Implementation Year 2 (August 10, 2013 – August 09, 2014), August 2013. ACDI/VOCA. 2014. Resiliency through Wealth, Agriculture and Nutrition (RWANU). FY 14 Quarterly Report: January–March 2014. ACDI/VOCA. 2014. Resiliency through Wealth, Agriculture and Nutrition (RWANU). FY 14 Quarterly Report: April–June 2014. ACDI/VOCA. 2014a. PREP: Pipeline Resource Estimate Proposal Implementation Year 3 (August 10, 2014 – September 30, 2015). ACDI/VOCA. 2014b. Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Annual Results Report, Fiscal Year 2014, Submitted November 3, 2014. ACDI/VOCA. 2015. Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU), Annual Results Report, Fiscal Year 2015, Submitted November 2, 2015. ACDI/VOCA. 2016. Resiliency through Wealth, Agriculture and Nutrition (RWANU). FY 16 Quarterly Report: April–June 2016. ACDI/VOCA. 2016. Resiliency through Wealth, Agriculture and Nutrition (RWANU). FY 15 Quarterly Report: October–December 2015. ACDI/VOCA. 2017. Resiliency through Wealth, Agriculture and Nutrition (RWANU). FY 17 Annual Results Report. ACDI/VOCA. 2017a. Indicator Performance Tracking Table (IPTT). Final project IPTT, 2017. ACDI/VOCA. 2017b. Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU), Annual Results Report, Fiscal Year 2017, Submitted November 28, 2017. ACDI/VOCA. 2017c. Resiliency through Wealth, Agriculture and Nutrition (RWANU). FY 17 Quarterly Report: October 1–December 31, 2016. Submitted January 30, 2017. Aklilu, Y., 2017. Livestock Trade in Karamoja, Uganda: An Update of Market Dynamics and Trends. Karamoja Resilience Support Unit, USAID/Uganda, Kampala. Black, R. E.; et al. 2013. “Maternal and Child Undernutrition and Overweight in Low-Income and Middle￾Income Countries.” The Lancet. 382 (9890): 427–51. Blackwell N, Myatt M, Allafort-Duverger T, Balogoun A, Ibrahim A, and Briend A. 2015. Mothers Understand and Can do it (MUAC): A comparison of mothers and community health workers determining mid-upper arm circumference in 103 children aged from 6 months to 5 years. Archives of Public Health 73:26 Campbell, Ruth (no date) ‘A Framework for Inclusive Market System Development’ Leveraging Economic Opportunities (LEO) Brief for USAID. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 70 Concern Worldwide. 2015. LQAS Survey Report Infant and Young Child Feeding, Quality Assessment of Module1: IYCF implementation, March 2015. Concern Worldwide. 2016. Post Male Change Agent Training Survey Report, Concern S02-SBCC Team, 16 November 2016. Cullis, A. 2018. Agricultural Development in Karamoja, Uganda: Recent Trends in Livestock and Crop Systems, and Resilience Impacts. Karamoja Resilience Support Unit, USAID/Uganda, UK aid, and Irish Aid, Kampala. Eriksson et al 2010. Boys Live Dangerously in the Womb. Am J Hum Biol., 22(3): 330-335. FANTA-2. 2010. Title II Technical Reference Materials. TRM-01: Preventing Malnutrition in Children Under 2 Approach (PM2A): A Food-Assisted Approach. Revised November 2010. Washington, D.C.: Food and Nutrition Technical Assistance II Project (FANTA-2), FHI 360. FAO. 2001. Chapter 4. Energy requirements of children and adolescents. In Human Energy Requirements. Report of a Joint FAO/WHO/UNU Expert Consultation Rome, 17–24 October 2001. FAO Food and Nutrition Technical Support Series. Accessed December 26, 2018 at: http://www.fao.org/docrep/007/y5686e/y5686e06.htm. FAO. 2018. Resilience Analysis in Karamoja. FAO Resilience Analysis Report No. 10. FEWS NET, 2012. Climate Trend Analysis: Informing Climate Change Adaptation Series FEWS NET, 2013. Uganda—Karamoja Region Livelihood Zones and Descriptions. Available at http://fews.net/east-africa/uganda/livelihood-description/december-2013 FEWS NET, 2016. Karamoja, Uganda Enhanced Market Analysis. Chemonics International Inc. for USAID. Gelsdorf, K., Maxwell, D., and Mazurana, D. 2012. Livelihoods, basic services and social protection in Northern Uganda and Karamoja. Secure Livelihoods Research Consortium Working Paper 4, Feinstein International Center. Government of Uganda (GOU), WFP & UNICEF, 2016, Food Security & Nutrition Assessment: Karamoja, Uganda. Grant A, Njiru J, Okoth E, Awino I, Briend A, Murage S, Abdirahman S, and Myatt M. 2018. Comparing performance of mothers using simplified mid-upper arm circumference (MUAC) classification devices with an improved MUAC insertion tape in Isiolo County, Kenya. Archives of Public Health (2018) 76:11. Hopwood, Julian, Holly Porter and Nangiro Saum 2017. ‘Resilient patriarchy: public authority and women’s (in) security in Karamoja, Uganda’, Disasters 2018, 42(S1): S140−S158. ICF International. 2014. Baseline Study for the Title II Development Food Assistance Programs in Uganda. Contract #: AID-OAA-M-12-00009. USAID/FFP. Ishara, A. 2017. Grants Impact Assessment of WLIA & YLIA Grants Groups. Iyer, P. and Mosebo, M. (2017), Looking for Work: Wage Labor, Employment, and Migration in Karamoja, Uganda, Karamoja Resilience Support Unit, USAID/Uganda, Kampala Kaari, E., Maganya, G., Wangoolo, J., Kamau, C., and Karungi, C. 2016. Final Evaluation of the DFID Funded, “Enhancing Resilience in Karamoja Programme (ERKP)”-UG16NF065. Final Evaluation Report. WFP Uganda Country Office. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 71 Krätli, S. 2010. Preface: Karamoja with the Rest of 'The Rest of Uganda', Nomadic Peoples, Vol. 14, No. 2, Special Issue: Disarmament as Development for Karamoja: A New Chapter in Dismantling Pastoral Production in Uganda, pp. 3-23 Kayobyo, G., Okech, M., Bbosa, S., and Schweisguth, M.A. 2016. RWANU Market System Interventions and Resilience Assessment. ACDI/VOCA. Lepillez, K, 2016. Livestock Activity Gender Impact Assessment. RWANU. Levine, S. 2010. “What to do about Karamoja?” Why pastoralism is not the problem but the solution: A food security analysis of Karamoja. FAO/ECHO. Markelova, Helen and Ruth Meinzen-Dick, 2009. Collective Action for Smallholder Market Access. CGIAR Systemwide Program on Collective Action and Property Rights (CAPRi) Policy Brief Number (6 April 2009). www.capri.cgiar.org/ Mathys E, Cashin K, Sethuraman K. 2017. USAID Office of Food for Peace Food Security Desk Review for Karamoja, Uganda. Washington, DC: FHI 360/FANTA. Mercy Corps, 2016. Karamoja Strategic Resilience Assessment (August, 2016). Mueller, B., and Bbosa, S. n.d. Wage Labor for Food Security in Southern Karamoja: A Labor Market Assessment for the RWANU Program Office of the Prime Minister (OPM), 2017. “Proceedings of the 9th Karamoja Policy Committee meeting and annual review of KIDP2 - Dec. 7th, 2017”. https://www.karamojaresilience.org/publications/item/proceedings-of-the-9th-karamoja-policy￾committee-meeting-and-annual-review-of-kidp2-moroto-district-council-hall PAHO and WHO. 2004. Guiding Principles for Complementary Feeding of the Breastfed Child. Accessed November 14, 2018 at http://www.who.int/nutrition/publications/guiding_principles_compfeeding_breastfed.pdf Perry H, Morrow M, Davis T, Borger S, Weiss J, DeCoster M, Ricca J, and Ernst P. 2015. CareGroupsII: A Summary of the Child Survival Outcomes Achieved Using Volunteer Community Health Workers in Resource-Constrained Settings. Global Health: Science and Practice 2015, Volume 3, Number 3. Stallkamp G. 2016. Report on Barrier Analysis survey on Exclusive Breastfeeding. Location: Selected sites in four districts in South Karamoja (RWANU project implementation area). May 2016. Stallkamp G. 2017. Contextual Analysis of Nutrition in Karamoja. PowerPoint presentation. Concern Worldwide’s Karamoja Programme Dissemination Event. February 7, 2017. Svedberg P. 1988. Undernutrition in Sub-Saharan Africa: Is there a sex bias? Institute for International Economic Studies University of Stockholm S-106 91 Stockholm and World Institute for Development Economics Research United Nations University SF-00100 Helsinki. September 1988. The Technical and Operational Performance Support (TOPS) Technical and Operational Performance Support Program. 2016. Care Groups: A Reference Guide for Practitioners. Washington, DC: The Technical and Operational Performance Support Program. Tumilowicz A, Habicht JP, Pelto G and Pelletier DL. 2015. Gender perceptions predict sex differences in growth patterns of indigenous Guatemalan infants and young children. Am J Clin Nutr 2015; 102:1249– 58. Uganda Bureau of Statistics (UBOS) and ICF International. 2012. Uganda Demographic and Health Survey 2011. Kampala, Uganda: UBOS and Calverton, Maryland: ICF International. Final Performance Evaluation of Resiliency through Wealth, Agriculture, and Nutrition in Karamoja (RWANU) Final Evaluation Report 72 Uganda Bureau of Statistics (UBOS) and ICF. 2018. Uganda Demographic and Health Survey 2016. Kampala, Uganda and Rockville, Maryland, USA: UBOS and ICF. Uganda Bureau of Statistics (UBOS) 2017. Uganda National Household Survey 2016/2017. Kampala: UBOS. UNICEF and WFP. 2013. Karamoja Food Security and Nutrition Assessment, Karamoja Uganda, May 2013. UNICEF and WFP. 2016. Karamoja Food Security and Nutrition Assessment, Karamoja Uganda, July 2016. UNICEF, FAO, WFP. 2017. Karamoja Food Security and Nutrition Assessment in Karamoja Subregion, June 2017. UNICEF and WFP. 2018a. Food Security and Nutrition Assessment for Karamoja Sub-region, January 2018. UNICEF and WFP. 2018b. Food Security and Nutrition Assessment for Karamoja Sub-region, July 2018. UNICEF, WHO, and World Bank. 2018. Levels and Trends in Child Malnutrition. UNICEF/WHO/World Bank Group Joint Child Malnutrition Estimates. Key findings of the 2018 edition. USAID. 2011. USAID Office of Food for Peace Uganda Bellmon Estimation. July 2011. USAID. 2012. Building Resilience to Recurrent Crisis: Policy and Program Guidance. USAID. 2016. USAID Office of Food for Peace (FFP) 2016–2025 Food Assistance and Food Security Strategy. FFP, Bureau for Democracy, Conflict and Humanitarian Assistance. USAID. 2017. Climate Risk Screen for Food Security: Karamoja Region, Uganda. Climate Risk Profile. Climate Change Adaptation Thought Leadership and Assessments (ATLAS) Task Order No. AID-OAA￾I-14-00013. USAID Center for Resilience, 2018. Resilience Evidence Forum Report, April 2018. www.usaid.gov/sites/default/files/documents/1867/0717118_Resilience.pdf https://www.usaid.gov/sites/default/files/documents/1866/170130_Karamoja_Food_Security_Climate_Scr eening.pdf Wamani et al 2007. Boys are more stunted than girls in Sub-Saharan Africa: a meta-analysis of 16 demographic and health surveys Henry Wamani, BMC Pediatrics, 7:17. Wilunda et al. 2015. Availability, utilization and quality of maternal and neonatal health care services in Karamoja region, Uganda: A health facility-based survey. Reproductive Health, 12:30. https://doi.org/10.1186/s12978-015-0018-7 ANNEX 1: RWANU RESULTS FRAMEWORK (presented in project proposal) Project Goal: Reduced Food Insecurity among Vulnerable People in South Karamoja Objective 1: Improved availability and access to food Objective 2: Reduced malnutrition in pregnant and lactating mothers and children under 5 IR 1.1: Improved smallholder farm management practices adopted IR 1.2: Improved smallholder livestock management practices adopted IR 2.1: Improved health and nutrition practices at the household level IR 2.2: Improved prevention and treatment of maternal and child illness IR 1.3: Increased linkages to markets Cross-Cutting Gender IR: Intra-household relationships improved Additional Cross-Cutting Issues: (1) Conflict mitigation, (2) Disaster risk reduction/natural resource management IMPACT OUTCOME SO 1 Activities 1.1  Provide training on FaaB, improved crop production, farmer-managed natural regeneration, post-harvest handling  Collaborate with NabuZARDI on improved seeds and planting materials  Construct/rehabilitate water management structures and train water user associations I.2  Establish and train women’s goat rearing groups  Train and provide technical assistance to Community Animal Health Workers 1.3  Provide training and technical assistance on honey and horticulture  Strengthen capacity to group market  Establish and train producer savings and credit groups SO 2 Activities 2.1  Establish Mother Care Groups in targeted communities  Promote positive health seeking behaviors  Promote consumption of diverse, nutritious food  Promote safe water, improved sanitation practices, and hygiene behaviors 2.2  Provide support to MoH maternal and child health centers  Provide support to MoH outreach services  Support community-based screening/referral of acutely malnourished children and pilot growth monitoring Cross-Cutting Activities Conflict mitigation  Involve traditional leaders  Engage in community dialogue to address theft issue  Ensure balanced targeting of beneficiary ethnic groups Gender  Evaluate women’s and men’s labor roles and allocation of time  Train men and women in the importance of positive health and nutrition behaviors  Identify men as change agents to promote joint decision-making and other positive empowerment behaviors INPUT/ OUPUT Disaster risk reduction / Natural resource management  Strengthen linkages among established district EWS, local government, and beneficiaries  Develop appropriate risk reduction measures through hazard and vulnerability analysis  Improve natural resource management  Improve resiliency at the household level ANNEX 2: EVALUATION STATEMENT OF WORK Statement of Work Population-Based Final Evaluations of Food for Peace Development Food Assistance Projects GHG and RWANU in Uganda INTRODUCTION The final evaluation of the 2012 Uganda Title II Food for Peace (FFP) projects is the second and final phase of a pre-post evaluation strategy. The baseline study, conducted from mid-February through April 2013, employed a mixed-method approach, and was designed to provide information on all four aspects of food security – availability, access, utilization and stability. The study investigated household food access, sanitation and hygiene, agriculture, household expenditures and assets, dietary diversity, and anthropometry among women and children. As with the baseline study, the Uganda final evaluation will also use a mixed method approach but will also utilize and integrate secondary data and project final monitoring data. Methods will be chosen in order to generate the highest quality and the most credible and robust evidence possible to answer evaluation questions. BACKGROUND In fiscal year (FY) 2012, USAID/Food for Peace (FFP) awarded Mercy Corps (MC) and ACDI/VOCA five-year Title II development food assistance projects (DFAP) in Uganda with an overall investment of over $100 million. The MC Northern Karamoja Growth, Health and Governance Program (GHG) aimed to improve food security of households in Kaabong, Kotido, and Abim, while the ACDI Resilience through Wealth, Agriculture and Nutrition (RWANU) sought to reduce the food insecurity of households in Napak, Moroto, Nakapiripirit and Amudat regions. GHG offered a range of economic, health, and governance initiatives to solidify the gains from increased security and establish a foundation for broader self-sufficiency. Targeted food aid for pregnant and lactating women and children under the age of two was used to transition vulnerable populations from decades of food aid by filling nutrition deficits in highly food insecure households. GHG expanded in its penultimate year to conduct a livestock sector pilot program to identify opportunities for livestock production and marketing in Karamoja’s post-conflict development. The project had three strategic objectives (SO): SO1: Pro-Poor Market Development (Livelihoods strengthened) – focused on building local capacity to provide vital products and services on a commercially sustainable basis to vulnerable households; SO2: Nutritional Status among Children under Five Improved – focused on improving local public and private health care, promoting improved household food consumption, and improving water infrastructure and sanitation and hygiene behaviors; SO3: Reduced Incidences of Conflict – focused on helping local conflict mitigation structures adapt to the current conflict dynamic, while supporting traditional authority structures and male and female youth to play more constructive roles in improving security. In the implementation of GHG, MC collaborated with numerous non-governmental and academic partners: World Vision (WV), Whave, Kaabong Peace and Development Agency (KAPDA), Abim Women Organized Together In Development (AWOTID), and Tufts University’s Feinstein International Center (FIC). MC implemented all economic programming and was responsible for ensuring that a gender sensitive approach underlay all activities. WV led supplementary feeding activities, and in collaboration with MC and WHAVE, conducted community-level public health initiatives, behavior change, water, sanitation, and hygiene programming. KAPDA and AWOTID concentrated on conflict management and governance activities, working primarily through local formal and informal authority structures to bolster local systems for conflict reduction and government service improvement. Lastly, Tufts University’s FIC conducted annual impact evaluations using its community-focused Participatory Impact Assessment methodology to create a strong understanding of the impact of the consortium’s work, and to inform implementation. The GHG consortium used a facilitative approach to implementation, pushing local actors to take the lead in sustainably providing products. ACDI/VOCA implemented RWANU in consortium with Welthungerhilfe (WHH) and Concern Worldwide (Concern). ACDI/VOCA was responsible for overall program management, as well as led activities related to crop production, alternative livelihoods, increasing resiliency through the promotion of group savings, improving market linkages, and managing food distribution. WHH carried out the technical training and input provisions related to livestock, while Concern implemented the nutrition, health, hygiene and health outreach component. ACDI/VOCA and partners collaborated to ensure integration across activities and promotion of gender equity and women’s empowerment, environmental stewardship, and conflict mitigation. Two SOs guided implementation: SO1: Improved access to food for men and women: Interventions were tailored to the three different livelihood zones, and activities sought to reduce the risks associated with rain-fed agriculture; identify and promote low-risk, higher-return commodities; and increase livestock ownership while drawing households into market systems. RWANU also focused on building the capacity of farmer groups and small producer groups through training in technical skills, savings mobilization, basic business skills and marketing. SO2: Reduced malnutrition in pregnant and lactating women and children under five: Activities focused on preventing malnutrition during the first 1,000 days of life through a package of curative and preventative health care, behavior change activities, and improved consumption of micro- and macronutrients. RWANU implemented activities around proper infant and young child feeding practices and the nutritional needs of pregnant and lactating women and children under two, while promoting equitable intra-household food distribution. RWANU implemented these activities through the Mother Care Group approach, male change agents, livelihood trainings, opinion leaders and social behavior change activities. EVALUATION PURPOSE & QUESTIONS The overarching purpose of the final evaluation is to measure the development outcomes of the GHG and RWANU projects. In the table below, FFP has identified the evaluation questions and provided the fundamental elements that should shape the Evaluation Team’s (ET) research. It is anticipated that the ET will address these, but it is not limited to working solely within this guidance. Q1: To what extent have the projects met their defined goals, purposes and outcomes? The ET will evaluate the contribution of GHG and RWANU to USAID’s efforts to reduce food insecurity among chronically food insecure households. 11 The ET will support its determination using both quantitative and qualitative methodologies when discussing the following: (1) project final on indicators against targets set by both the partners and the key FFP indicators* of Depth of Poverty, Stunting, and Undernutrition; (2) factors that promoted or inhibited the achievement of the project objectives, including, but not limited to the effectiveness of food-for-asset and/or cash-for￾asset interventions; (3) plausibility of pathways and the determinants of achieving the key outcomes; (4) targeting strategies and their contribution to achieving project goals (especially with regard to gender and reaching the most vulnerable); (5) the appropriateness and effectiveness of interventions on the poorest individuals. 1 2012 Food for Peace Country Specific Information: Uganda http://pdf.usaid.gov/pdf_docs/pdacu288.pdf *FFP’s established targets are: a minimum of 2 to 2.5 percentage point annual reduction of prevalence of stunting, a minimum of 3 to 4 percentage point annual reduction of prevalence of underweight, and a minimum of 4 percentage point annual reduction of depth-of-poverty. Q2: Based on the evidence, which project outcomes are likely to be sustained? The ET will evaluate the functionality and final of systems and processes established independently by the projects, as well as in collaboration with the private sector, Government of Uganda, community organizations, and research organizations to achieve project outcomes and sustainability. It will support its determination using both quantitative and qualitative methodologies that explore the following: (1) the quality of the processes, systems, and institutional arrangements developed and/or strengthened to sustain the necessary and critical services; (2) communities’ perceptions on the quality, frequency, effectiveness, and sustainability of the services provided by the project; (3) the likelihood that service providers will continue providing services after the project ends; (4) the motivation of the community and beneficiaries to demand and pay (or invest time) for the services; (5) whether the necessary resources and capacity strengthening will exist to sustain service providers; (6) the extent to which the projects leveraged other USG and non-USG investments to achieve sustained outcomes as identified in the theories of change; (7) evidence of enhanced linkages with other service providers. Q3: In each technical sector, what are the strengths of and challenges to the efficiency and effectiveness of the interventions’ implementation and their acceptance in the target communities? The ET will evaluate the effectiveness and relevance of the technical interventions, including food￾for asset and/or cash-for-asset interventions, to achieve project outcomes, and discuss those findings in relation to the projects’ theories of change. It will support its determination using both quantitative and qualitative methods when discussing the following: (1) factors in the implementation and context associated with greater or lesser efficiency and effectiveness in producing Outputs of higher or lower quality; (2) the interventions and implementation processes deemed more/less acceptable to members of the target communities. Q4: What key lessons learned and best practices should inform future projects in the country? During the course of its research, the ET should identify best practices, strengths, and challenges in the projects’ designs (including theories of change) and risks to their effective implementation, supporting project achievements, as well as approaches that should be considered in designing future food and nutrition security projects. The ET will support its determination using both quantitative and qualitative methodologies when discussing the following: (1) the unintended positive and/or negative consequences of the projects, and (2) ways to minimize potential unintended negative consequences and systematically capture positive consequences. AUDIENCE & INTENDED USES The primary audience of the evaluation reports will be MC and ACDI (and their sub partners), and USAID (FFP/Washington, USAID/Uganda). The reports will also be shared with the Government of Uganda. Findings from the final evaluation will be used to determine the final of the two DFAPs; influence the implementation of the current development food security activities (DFSA), as appropriate; as well as inform and shape future USAID projects. USAID will make extensive use of findings from the evaluations to make different presentations and bulletins as part of a wider dissemination of best practices and lessons learned. The evaluation recommendations may be used by FFP to refine proposal guidelines and project policy. FINAL EVALUATION METHODOLOGY The final evaluation will use a mixed-methods approach. The ET will begin with a desk review of project documents, validate its understanding of the projects via consultations with USAID and implementing partners, conduct a population-based household survey using all implementation villages as the sampling frame, and conduct qualitative research in villages selected via non-probability sampling method. It is preferred that, if possible, the firm conducts quantitative and qualitative components sequentially to allow the quantitative data to inform the qualitative research. a) Desk Review The evaluation team should review the following documents to contextualize and refine the evaluation questions, as well as to gain an in-depth understanding about the project design, implementation, and the food security situation in the area. Partner annual monitoring data should be reviewed when preparing for qualitative research, considered in relation to the evaluation findings, and incorporated into the report in support of/contrast to select evaluation findings. While FFP recommends the below documents for pre-evaluation learning, the literature review should not be limited to the following: Project proposals Pipeline Resource Estimate Proposals (PREPs) Annual results reports (ARR), including Indicator Final Tracking Tables (IPTT) for final against targets Midterm review reports and corresponding action plans developed by the two projects Baseline Study for the Title II Development Food Assistance Projects in Uganda, 2013 Uganda Demographic Health Survey 2016 Partner formative research and barrier analyses so as to better understand the context and if/how the studies influenced programming Monitoring data and reports b) Consultations As a supplement to the desk review, consultations with FFP and partner staff in Washington, DC and Uganda will allow the ET to corroborate its understanding of the design, approaches and interventions employed by each DFAP and acquired through the desk review. It is recommended that the ET engage with the staff at each organization prior to beginning fieldwork. Equally important to engaging pre-data collection is to reconnect post-data collection to “ground truth” findings with FFP/Uganda and the partner staff. In the case of major disagreements, the program staff should provide evidence in support of the argument, and pending time constraints, the ET may revisit the field. c) Quantitative Endline Survey The quantitative endline household survey for the 2012 DFAPs will be integrated with the baseline survey for the 2017 DFSAs. FFP made this decision to increase survey efficiency and reduce the data collection burden. The 2018 PBS (the term used to denote both the combined 2018 endline and baseline survey) will collect data on the same population-level impact and outcome indicators (presented below) that were collected during the 2013 baseline survey, as well as, resilience indicators2 and a few additional indicators identified by the new DFSA partners (following the 2018 baseline workshop). DFAP baseline data were collected between late February and the end of April, and data collection for the final evaluation must occur during the same time period. The 2018 PBS should use the same data collection instrument for the endline indicators, level of statistical precision (95 percent confidence intervals), and statistical power (80 percent) as the baseline study3. The 2018 PBS design does not need to be identical 2 Resilience indicators will be calculated, and analysis conducted, for the baseline study only. 3 Baseline Study for the Title II Development Food Assistance Projects in Uganda https://www.usaid.gov/sites/default/files/documents/1866/Uganda%20Baseline%20Study%20Report%2C% 20March%202014.pdf to the baseline; if the projects reduced their target areas, for example, the sampling frame of households used for the baseline may need to be adjusted. Prevalence of underweight children under five years of age Prevalence of Poverty: Percent of people living on less than $1.25/day Mean Depth of Poverty Per capita expenditures (as a proxy for income) of USG-targeted beneficiaries Prevalence of stunted children under five years of age Prevalence of underweight women (of reproductive age) Percentage of farmers who used at least [a project-defined minimum number of] sustainable agriculture (crop/livestock and/or NRM) practices and/or technologies in the past 12 months Percentage of farmers who used improved storage practices in the past 12 months Percentage of farmers who used financial services (savings, agricultural credit, and/or agricultural insurance) in the past 12 months Percentage of farmers who practiced the value chain activities promoted by the project in the past 12 months Average Household Dietary Diversity Score (HDDS) Prevalence of children 6-23 months receiving a minimum acceptable diet (MAD) Women's Dietary Diversity (Score): Mean number of food groups consumed by women of reproductive age (WDDS) Prevalence of exclusive breastfeeding of children under 6 months of age Percentage of children under age five who had diarrhea in the prior two weeks Percent of children under five years old with diarrhea treated with Oral Rehydration Therapy (ORT) Percent of households using an improved drinking water source Percent of households using an improved sanitation facility Percent of households with soap and water at a handwashing station commonly used by family members 6 Percentage of births receiving at least four antenatal care (ANC) visits during pregnancy Percentage of livestock farmers accessing government or private sector veterinary services (diseases diagnosis and drugs) and livestock vaccination Percentage of respondents who know at least 3 of the 5 critical moments for hand washing 3 submodules of the Women's Empowerment in Agriculture Index (WEAI)* * The 2013 baseline study collected and reported on the WEAI, but the questions asked in the WEAI are largely not relevant to FFP target populations. WEAI is designed for communities/populations that are at slightly higher wealth quintiles, where families can report on having some assets. The majority of Karamajong are extremely poor and do not have any of these assets, and therefore, most of the questions asked in WEAI and the score itself is not a valid measure of gender empowerment for FFP target households. Note: Based on USAID’s strategic interests, a few additional questions may be incorporated into the household questionnaire (and any corresponding indicators added). All quantitative data is owned by USAID and made available to the public barring rare exceptions. d) Qualitative Research Qualitative methods will be used to collect information to answer evaluation questions and to support the interpretation of the quantitative data. The ET will design the overall qualitative study approach and should consider a variety of primary data collection methods, such as semi-structured in-depth interviews, group discussions, key informant interviews, direct observations, and case studies (the ET may choose to use the most significant change methodology to identify a selective set of case studies). These methods - to the maximum extent possible - will ensure that if a different, well-qualified evaluator were to undertake the same evaluation, he or she would arrive at the same or similar findings and conclusions. The ET should decide on specific methods before traveling to Uganda, and present those to FFP along with the number of interviews, FGDs, etc., per project, in the inception report. Following discussion and agreement, the ET will finalize the methods during the team meeting in￾country. The evaluation team leader and members will be responsible for interviewing the direct, indirect and non-beneficiaries in their households and communities, as well as look for evidence of ongoing learning and activities (such as home gardens, etc.). The ET will also be responsible for interviewing relevant stakeholders for the evaluation and analyzing the qualitative data. Should the ET decide to hire additional researchers to complement the data collection effort, they cannot replace the evaluation team members’ role of collecting primary data using qualitative methods. The ET will contribute to the interpretation of the quantitative results using qualitative findings. In addition to the factors specifically identified earlier as essential to responding to the evaluation questions, during its research during the qualitative study the ET should also consider the efficacy of the following cross-cutting interests: project management; final monitoring; strategies to improve gender equality at the participant and project management levels; environmental considerations; and conflict sensitivity. Lastly, it is expected that the evaluation will speak to lessons learned and best practices. The ET may find it useful to apply non-probability sampling methods to select a sub set of enumeration areas from the quantitative survey. In selecting interview sites, the evaluation team should strategically select large-enough-yet-manageable interview sites that generally represent the target area. As with the quantitative household survey, qualitative sampling should include both individuals who directly participated in the DFAP (beneficiaries) and those not specifically targeted with any intervention (indirect/non-beneficiaries). (The latter should be included to allow learning on spillover, triangulate the information provided by the direct beneficiaries, and to understand their perspectives on the achievements or limitations of the interventions offered by RWANU and GHG.) In addition, the qualitative team should interview USAID personnel, project staff, knowledgeable people from the community, local government staff, community leaders, host Government officials, and other agencies and individuals as appropriate. e) Data Analysis and Interpretation The evaluation team will statistically compare the endline data for each of the two strata with that of the baseline for that stratum, and also for the overall country level, in order to detect changes (if any) for all key indicators. The ET will conduct descriptive and inferential analyses to describe the results, as well as various econometric analyses to identify the determinants of key outcomes and the magnitude and direction of changes. In advance of fieldwork, the evaluation team should develop a data analysis plan and submit to the COR for approval. When analyzing the data, however, the evaluation team should not limit itself to the data analysis plan; rather, the evaluation team should keep an open and curious mind to look for correlations between variables. In presenting the analysis, the evaluation team should be cognizant about the readers’ familiarity with the statistical presentation. FFP suggests avoiding jargons, but rather describe the statistical terms in a common language. Interpreting the results is as critical as the analysis. Oftentimes, it can be difficult for a reader to fully understand the key points and utility of the findings conveyed in a report. The analysis and interpretation should be presented in a “story telling format” so that the readers can see a human face as they read the report. While it is important for the reader to understand whether level of stunting is reduced in the area, it is equally important to understand the pathway; for example, how learning derived from project participation influenced people’s practices, which in turn resulted in positive changes in food security outcomes at the household and/or community level. Similarly, it is equally important for the readers to know some of the challenges participants faced that might have prevented them from reaping the full benefits of the projects. REPORT The ET will produce two reports in English, not to exceed 50-pages, for each DFAP. The draft reports will be shared within FFP, USAID/Uganda, and the partners for review and comment over a two-week period. The final report should include a table of contents, table of figures (as appropriate), acronyms, executive summary, introduction, purpose of the evaluation, research design and methodology, limitations, findings, conclusions, lessons learned, and recommendations. All evaluation questions should be answered, and the evaluation methodology should be explained in detail. To ensure a high-quality deliverable, the reports should reflect a thoughtful, well-researched and well￾organized effort to objectively evaluate what worked in the project, what did not, and why. Where noteworthy, the discussion should highlight and discuss the outcomes and impacts on males versus females. The report must integrate the quantitative analysis from the PBS and include a statistical comparison between baseline and endline results. Learning from the qualitative research will help to contextualize and interpret the quantitative data. The ET should also draw from partners’ annual monitoring data, where appropriate, to substantiate findings. For example, if a qualitative finding is that beneficiaries report increased yields, the ET should compare/contrast that finding against data collected by the partners. Findings should be specific, concise, and supported by strong quantitative and/or qualitative evidence, and presented as analyzed facts/evidence/data, and not be based on anecdotes, hearsay or a compilation of people’s opinions. It should include analytical methods to include appropriate tests of differences; econometric analysis to evaluate the theories of change and to explore the causal relation between the outcome and activities/variables based on the theoretical models; it is expected that the contractor will interpret the analytical findings. The report should disclose limitations to the evaluation, with particular attention to the limitations associated with the evaluation methodology, e.g. selection bias, recall bias, unobservable differences between comparator groups, etc. Recommendations should be supported by a specific set of findings, and be action-oriented, practical, and specific. It is expected that the final reports will address and incorporate feedback, as appropriate, from the reviewers. Should the ET disagree with any of the comments, it should raise this with the COR immediately for discussion. EVALUATION TEAM The Evaluation Team Leaders will be responsible for designing and managing the evaluations and overseeing the work of the evaluation team members; coordinating with the implementing partners, USAID Mission and other stakeholders; coordinating with the endline PBS team; analyzing the findings and ensuring the quality of the report. As this is a mixed-method final evaluation, in addition to the evaluation team, the endline survey will require extensive participation of the following personnel: Senior Survey Method Specialist, Data Analyst, Survey Coordinator, Data Management Specialist, Anthropometry Specialist, Country Operations Manager, and Survey Monitors. The PBS data collection team should be hired locally, if possible. The evaluation team will require in-country participation from the Evaluation Team Leader and up to three subject matter specialists. As FFP projects are multi￾sectoral, the evaluation team must possess expertise and field experience with food security and rural integrated programming and demonstrate an in-depth knowledge of the following technical sectors and cross-cutting areas: agriculture and off farm livelihoods, nutrition; water, sanitation, and hygiene (WASH); gender, youth, resilience, and disaster risk management. The subject matter specialists must also possess experience and knowledge about the specific processes used by the projects (e.g., Care Groups, Farmer Field Schools, etc.) FIELD LOGISTICS The ET is responsible to arrange and pay for all logistics including anthropometric equipment, and transportation. It is strongly recommended that the ET consults the Mission on identifying a reputable car-rental firm. The Mission and partners also should be consulted on identifying interpretation services. The ET should request assistance from the partners on making introductions, as necessary, to local ministry representatives and community leaders. DELIVERABLES The ET shall produce the following deliverables during the evaluation and submit to the Contracting Officer’s Representative (COR) for review and approval. All draft documents should be submitted in Microsoft Word or Microsoft Excel, or in the rare occasion both PDF and Word/Excel. The COR must approve all deliverables. Deliverable Timeline Work Plan includes a brief synthesis and timeline for the Uganda final evaluation, with the timeline including major activities throughout the study, including dates by which field guides and training materials will be completed. Only one work plan detailing both baseline study and final evaluation activities is required TBD PBS Enumerator Guide, Supervisor Manual, and Anthropometry Guide* provide revised detailed instructions on supervisor, enumerator and anthropometry trainings. Note that the PBS should use the supervisor, enumerator and anthropometry training guides developed for the baseline. Minor adjustments will be needed to accommodate the new indicators. Only one set of guides that serves both the baseline and endline surveys is required TBD PBS Data Treatment and Analysis Plan • details how the data will be cleaned, weighted, and analyzed and must include: programming specifications and editing rules for cleaning data, data dictionary codebook, SPSS syntax and output for all analyses and variable transformations into indicators; and • includes a descriptive, inferential, and econometric analyses plan. TBD Only one DTAP that serves both the baseline study and final evaluation is required, but it must clearly differentiate between the different analytical approaches used for each. PE Inception Report (~20 pages) • briefly synthesizes the literature review; • describes the qualitative evaluation methodology (including evaluation questions contextualized based on the literature review, sample site selection strategy and number of sites to be selected, number of interviews/discussions per project, types of interviewees) • introduces the evaluation team members and their roles; and • details how the qualitative information will be analyzed and integrated with quantitative. TBD Deliverable Timeline Final Evaluation Protocol (a combined protocol that discusses both baseline/endline surveys and qualitative component) (~30 pages) • describes the evaluation questions to be answered; • present specific data collection methods by evaluation question; • identifies indicators to be collected; • introduces the local partner (data collection firm); • discusses the quantitative and qualitative analysis methods and plan; • presents PBS sample size, design and plan, survey design, questionnaire design, site selection plan for qualitative research; and • presents the fieldwork plan (including trainings and field support/supervision, data management, quality control, recording, analysis and reporting). TBD Pertinent Permissions and approvals demonstrate official approval from all relevant institutional review boards and from host country institutions to collect data, conduct the evaluation, and release data and reports, as required, as well as a statement affirming adherence to all requirements specified in USAID’s Scientific Research Policy. TBD PBS Quantitative Survey and Qualitative Instruments • include both English and Ngakarimojong versions of the household survey (note: the instrument must be back-translated to English via a second translator to ensure accurate translation. Following the pilot of the survey, any modifications based on field experience will again require translation and back translation to ensure accuracy). • describe site selection methodology and factors used to select sample communities; • list sample communities; • discusses groups to be interviewed; and • explain criteria used to select respondents. TBD [preliminary estimates submitted in advance to support qualitative research] Draft Evaluation Reports (one for RWANU and one for GHG) • will be no more than 50 pages per DFAP (excluding annexes); • include as an annex tables for FFP indicator estimates, including comparisons between baseline/endline values October 2018 In-country briefings to USAID/Uganda and FFP stakeholders in Uganda Two 60-minute presentations of the major findings of the baseline study to provide an opportunity for immediate stakeholder feedback that can be considered for the revision (as appropriate and without compromising the validity or independence of the evaluation): o One presentation to USAID/Uganda; o One presentation to FFP stakeholders in Uganda, including the DFSA partners, donors, and Government of Uganda (invited by USAID/Uganda and the partners), via a 60- minute PowerPoint presentation November 2018 Briefing to FFP/Washington comprises a 60-minute PowerPoint presentation on the evaluation, major findings, lessons learned, and recommendations November 2018 Deliverable Timeline Final Evaluation Reports • include items identified in the draft report as well as a three- to five-page executive summary of the purpose, background of the project, methods, findings, conclusions and recommendations, and the following annexes: the scope of work, tools used in conducting the evaluation (questionnaires, checklists, and discussion guides), and any substantially dissenting views by any Team member, USAID or the PVOs on any of the findings or recommendations; and • must be 508 compliant and uploaded to the Development Clearinghouse following COR approval. December 2018 Data (to be submitted at the time of the final report*) • include a separate electronic file of all quantitative data in an easily readable format that is organized and fully documented so as to facilitate use by those not fully familiar with the project or the evaluation; • provides cleaned data, sampling weights at each stage, final sampling weights, and all derived indicators; • includes a second final data set in CSV format that has been anonymized to protect individual confidentiality for use as a public data file in the USAID Open Data; and • include a separate electronic file of all transcribed (as applicable) qualitative data in machine-readable format; and • include a separate file detailing GPS coordinates of households that participated in the PBS. *FFP may request data sets earlier for internal use only December 2018 ANNEX 3: EVALUATION MATRIX No Evaluation Question Measure/Indicator4 Data Collection Methods Main Sources of Information Data Analysis Methods 1 To what extent did the project meet its defined goals, objectives and outcomes? The evaluation team will assess/determine the following: 1.1 The project’s final achievement indicators against targets set by both the partners and the key FFP indicators including of Depth of Poverty, Stunting, and Undernutrition Key impact, outcome, and output indicators from the PMP will be used, including Depth of Poverty; Prevalence of Stunting in children under 5 and Underweight for children under 5 (N) Review of Indicator Performance Tracking Table (IPTT) data Population-based survey (PBS) IPTT database PBS indicator estimates Descriptive analysis of quantitative data from IPTT and PBS to compare targets and actual achievements for the various indicators. IPTT data will be used to compare targets with actual achievements at the end of the project (Year 5). The analysis will also compare Year 1 and Year 5 figures. PBS data will be used to compare baseline and endline figures for RWANU project areas, disaggregated by beneficiary status. Endline figures will also be compared with targets set by the project. 1.2 Factors that promoted or inhibited the achievement of the project objectives, including, but not limited to the effectiveness of food and nutrition security interventions including conditional transfers. Number and range of factors that promoted achievement of project objectives (L) Number and range of factors that inhibited achievement of project objectives (L) Review of documentation Key informant interviews (KIIs) and focus group discussions (FGDs) Annual Results Reports (ARRs,) quarterly reports All stakeholders and beneficiaries Narrative/thematic analysis of documentation Triangulation of information from different sources, including quantitative data from EQ1.1 to determine level of achievement for different objectives 1.3 Targeting strategies and their contribution to achieving project goals (especially with regard to gender and youth reaching the most vulnerable) Key impact and outcome indicators from Performance Monitoring Plan (PMP) (N) Alignment of RWANU targeting criteria with key PBS Review of documentation KIIs, FGDs, in-depth interviews (IDIs) PBS database ARRs, quarterly reports All stakeholders and beneficiaries PBS: Comparison of participant / non-participant HHs disaggregated by gender and vulnerability Narrative/thematic analysis of documentation 4 Indicators measured primarily through quantitative data are noted by “N,” indicators to be measured with qualitative data are noted by “L,” and those indicators including both are noted by “NL.” No Evaluation Question Measure/Indicator4 Data Collection Methods Main Sources of Information Data Analysis Methods gender and vulnerability indicators for Karamoja (L) Range of stakeholder and beneficiary perceptions regarding targeting and effectiveness (L) Community and household (HH) case studies Case study communities and households Triangulation of information from different sources, including comparisons across case study HHs 1.4 The effectiveness of interventions on the target households including the poorest and most vulnerable Comparison of PBS baseline-endline figures for key indicators, disaggregated by HH poverty (N) Range of stakeholder perceptions on effectiveness for poorest individuals (L) Comparison of case study HH’s perceptions of effectiveness (L) PBS Review of documentation KIIs, FGDs, IDIs Community and HH case studies PBS database All stakeholders and beneficiaries Case study communities and households PBS: Comparison of beneficiary / non-beneficiary HHs disaggregated by poverty Triangulation of information from different sources, including comparisons across case study communities and HHs 1.5 Based on evidence from sub-questions 1-4, what has been the contribution of RWANU to USAID’s efforts to reduce food insecurity among chronically food insecure households? Synthesis of indicators above Synthesis of above Synthesis of above Synthesis of above 2 For each technical sector, what are the strengths of and challenges in the project design (including theories of change [ToCs]) that influence the effectiveness of the interventions? The evaluation team will assess the following: 2.1 The relevance of the technical interventions, including 1000 days ration and other conditional transfers, to achieve project outcomes, and discuss those findings in relation to the projects’ ToCs Clear evidence for the basis of the technical interventions Coherence of technical interventions in relation to ToC Review of documentation KIIs, FGDs, IDIs Community and HH case studies Proposal, ARRs, Pipeline Resource Estimate Proposals (PREPs), technical documentation All stakeholders and beneficiaries Narrative/thematic analysis of documentation Triangulation of information from different sources, including different communities and HHs No Evaluation Question Measure/Indicator4 Data Collection Methods Main Sources of Information Data Analysis Methods 2.2 The plausibility of pathways and the determinants of achieving the key outcomes, highlighting any unforeseen pathways leading to unintended positive or negative consequences of the projects Coherence of the pathways/ToC Number and range of unintended consequences identified Review of documentation KIIs, FGDs, IDIs Community and HH case studies Proposal, ARRs, PREPs, etc. All stakeholders and beneficiaries Narrative/thematic analysis of documentation Triangulation of information from different sources, including different communities and HHs 3 In each technical sector, what are the strengths of and challenges to the effectiveness of the interventions’ implementation? The evaluation team will evaluate the following: 3.1 The effectiveness and relevance of the technical interventions/ approach/methods, including 1000 days ration and conditional transfers to achieve project outcomes, and discuss those findings in relation to the projects’ ToCs Coherence of approach/methods with local context Timeliness of technical intervention activities Quality of services provided Number and range of implementation challenges reported by stakeholders Review of documentation KIIs and FGDs Community case studies Proposal, ARRs, PREPs, technical documentation All stakeholders and beneficiaries Narrative/thematic analysis of documentation Triangulation of information from different sources, including different communities 3.2 Factors in the implementation approach/methods, and the context associated with greater or lesser effectiveness in producing outputs of higher or lower quality Number and range of implementation factors that affect quality of outputs (L) Number and range of contextual factors that affect quality of outputs (L) Review of documentation KIIs, FGDs, IDIs Community and HH case studies Proposal, ARRs, PREPs, technical documentation All stakeholders and beneficiaries Narrative/thematic analysis of documentation Triangulation of information from different sources, including different communities and HHs 3.3 The interventions and implementation processes deemed more or less acceptable to members of the target communities. Number and range of factors relating to acceptability to target communities Willingness of community members to participate / Review of documentation KIIs, FGDs, IDIs Community and HH case studies Proposal, ARRs, PREPs, technical documentation Community-level beneficiaries and other Narrative/thematic analysis of documentation Outcome mapping Triangulation of information from different sources, including different communities and HHs No Evaluation Question Measure/Indicator4 Data Collection Methods Main Sources of Information Data Analysis Methods contribute to/pay for project activities and services stakeholders who work at community level 4 Which project outcomes are likely to be sustained? The evaluation team will evaluate the following: 4.1 The apparent quality and functionality of local systems, processes, and institutional arrangements established or strengthened independently by the projects, as well as in collaboration with the private sector, Government of Uganda, community organizations, and research organizations in relation to project outcomes and sustainability Number, range, and functionality of systems, processes, and institutional arrangements established by project Review of documentation KIIs and FGDs Community case studies Proposal, ARRs, PREPs, technical documentation Stakeholders at district and sub￾county levels Public and private sector service providers Community members Narrative/thematic analysis of documentation Triangulation of information from different sources, including different communities 4.2 Whether the necessary resources and capacity strengthening will exist to sustain service providers where they were intended to continue actions, and the likelihood that this service provision will continue after the project ends Number, range, and capacity of project-related service providers that are currently providing services such as agriculture/livestock inputs, credit, market services, health services, etc. Review of documentation KIIs, FGDs, IDIs Community and HH case studies Proposal, ARRs, PREPs, technical documentation Stakeholders at district and sub￾county levels Public and private sector service providers Community members Narrative/thematic analysis of documentation Triangulation of information from different sources, including different communities and HHs 4.3 Communities’ perceptions on the quality, frequency, and effectiveness of the implementation processes and services provided in the project Number and range of community perceptions on quality, frequency, and effectiveness KIIs, FGDs, IDIs Community and HH case studies Community members Triangulation of information from different sources, including different communities, different community groups, and different HHs No Evaluation Question Measure/Indicator4 Data Collection Methods Main Sources of Information Data Analysis Methods 4.4 The motivation of the community and beneficiaries to demand and pay (or invest time) for the services, and communities’ perceptions on sustainability of the services provided by the project Number and range of instances in which community members contributed to/paid for project services Number and range of perceptions on sustainability of services KIIs, FGDs, IDIs Community and HH case studies Community members Public and private sector service providers Triangulation of information from different sources, including different communities, different community groups, and different HHs 4.5 The extent to which the projects leveraged other U.S. Government and non-U.S. Government investments to achieve sustained outcomes as identified in the ToCs Number and value of investments KIIs ACDI/VOCA staff Former Chief of Party FFP staff Triangulation of information from different sources 4.6 Evidence of enhanced linkages with other service providers Number, range, and functionality of linkages with other service providers KIIs and FGDs Public and private sector service providers Community members and groups Triangulation of information from different sources 5 What are the key lessons learned and best practices that should inform future projects in the country? 5.1 What are the lessons, best practices, and approaches that should be considered in designing future food and nutrition security projects? Innovative and successful lessons, best practices, and approaches relating to design Synthesis of above Synthesis of above Synthesis of above 5.2 What are the lessons, best practices, and approaches that should be considered in implementing/implementation approaches for future food and nutrition security projects? Innovative and successful lessons, best practices, and approaches relating to implementation Synthesis of above Synthesis of above Synthesis of above 5.3 What are the risks to the effective implementation of these approaches to support project achievements? Risks to implementing innovative and successful approaches Synthesis of above Synthesis of above Synthesis of above No Evaluation Question Measure/Indicator4 Data Collection Methods Main Sources of Information Data Analysis Methods 5.4 Based on the lessons learned, how can potential unintended negative consequences be minimized and positive consequences be captured systematically? Synthesis of above Synthesis of above Synthesis of above Synthesis of above ANNEX 4: CONSENT FORM Study Title: USAID Food for Peace Final Performance Evaluations of the RWANU and GHG Development Food Assistance Projects This study has been approved by an accredited Ugandan Research Ethics Committee (MUREC). This study is funded by the United States Agency for International Development (USAID) Office of Food for Peace. About 200 people will be interviewed for the RWANU Project. INFORMED CONSENT STATEMENT and CONSENT FORM for INDIVIDUALS SCHEDULED to PARTICIPATE in a KEY INFORMANT INTERVIEW or IN-DEPTH INTERVIEW The same statement should be used for all key informant and in-depth interviews at the district, sub-county and village levels. This statement must be read at the beginning of each interview by the person leading the interview. Introductions and duration of interview Thank you for very much for agreeing to meet us for an interview today. My name is X and I am from ICF, a company in the United States. (Each other evaluation team member present will introduce themselves). This interview will take no more than one hour of your time. Purpose of the study, and purpose of this interview Duration Before we begin I would like to tell you about the purpose for this study. We have been hired by USAID to conduct an evaluation/assessment/study of Mercy Corps’ activities here in this district/village over the past several years. We want to know if any of the activities benefitted you and other people in this district/village, and if they did, in what ways. The reason why we want to interview you is because of your knowledge/experience with (X insert the specific intervention/activity that is the focus of the interview). Potential risks, our procedures to reduce potential risk, and confidentiality measures Alternatives for notes taken during the interview and for recording the interview There should not be any risk to you for agreeing to be interviewed. Many other people in this district and in other villages will be interviewed, too, for the same purpose. The way we reduce any risk that might possibly occur because of your answers to our questions, is by our guarantee to you personally that everything you say during our meeting will be kept confidentially. We will not tell other people in your district/village or in any other places in this district, what we talked about and what you said. We plan to take notes during this interview so that we will not forget this conversation, but these notes will not be shared with any other persons. Your name will not be included in these notes. I will type up these notes into my computer, and the notes we take from other people we interview, to determine if there are ways in which MC’s future activities can be improved for the benefit of people in this district. When we finish studying the notes from all the interviews we hold with people in this district, I will erase the notes from my computer so that no one else will be able to read the notes. I will also destroy the notes we take down on paper today. Alternative for instances where we also plan to record the interviews. I would like to use this recorder to record our discussion to make sure I do not forget any important information. Do you have any objections? (plan to take notes if the person objects to being recorded) I will listen to the recording of the interview and type it up into my personal lap top/computer to make it easier for me to study. No one else is allowed to listen to the recording or to use my computer to read the notes from the recording of our interview today. I will study these notes, and the notes we take from other people we interview, to determine if there are ways in which MC’s future activities can be improved for the benefit of people in this area. When we finish studying the notes from all the interviews we hold with people in this district, I will erase the recording and delete the notes I typed up into my computer so that no one else will be able to listen to this interview or to read the notes. Benefits from consenting to interview There are no specific benefits to you for agreeing to be interviewed today. We can say that the information you give us today will make a contribution to these types of activities in the future to improve X (insert one of these phrases depending on the focus of this interview: the health of children, the local economy, the livelihoods of people, peace) in this area. We value your contribution. Contact Information for any questions If you have any questions about this research after we leave here today, please contact Kate Longley at katelongley.consultant@gmail.com . We will be glad to answer your questions. You can contact Dr. Daniel Kibuuka Musoke TEL 0772587094 for information regarding the progress and findings of the study. If you have any concerns about the study, you can contact the MUREC chairperson, Ms. Harriet Chemusfo; Tel: 0392174236. Participation is Voluntary. No penalty involved in leaving any time during interview. Before I begin asking questions, we want to assure you that your participation in this interview today is completely voluntary. If after hearing what I just explained you decide that you do not want to be interviewed, you are free to leave. I will not ask you for an explanation. There is no penalty involved. If there are any questions I ask that you do not want to answer, please let me know. We will skip to the next question. If at any time during the interview you wish to stop, please let me know. I will not ask you for an explanation. Individual’s questions Do you have any questions? Ask for consent Do you agree to be interviewed? (If yes) Will you sign this consent form? Participant Name, Signature and Date: Witness for Participant Name, Signature and Date: Interviewer Name, Signature and Date: ANNEX 5: QUALITATIVE DATA COLLECTION INSTRUMENTS This Appendix contains the following data collection guides: 1. Key Informant Interview guides for: a) former RWANU staff members b) government stakeholders c) agro-vet service providers d) health service providers e) Community Leaders f) Lead / Model Farmers g) Former group leaders / members h) Innovation Award Grantees i) Lead Mothers / Male Change Agents 2. In-depth interview guide for case study beneficiary households 3. Focus Group Discussion Guide for existing groups (e.g. savings group) 4. Focus Group Discussion Guide for non-participants 5. Technical questions (based on an analysis of quantitative results) - to be used in conjunction with the more general interview guides listed above. Technical questions for the following sectors are presented: a) Technical questions for agriculture (including crops and livestock) b) Technical questions for household incomes, savings & financial services c) Technical questions for health d) Technical questions for nutrition e) Technical questions for WASH The evaluation team will obtain voluntary written informed consent from literate respondents before carrying out any data collection. For illiterate respondents a verbal consent will be obtained. Consent forms will be translated into the appropriate languages and back-translated into English and tested to ensure clarity for use in the field. Subjects will read or have the form read to them in the relevant local language and respondents will be asked to sign the form, if literate. Careful attention will be paid to ensure that respondents understand that their responses will be used for research purposes and that the information they provide will be non-attributable in the report, and that their confidentiality and anonymity will not be compromised. After the interviewee has consented to take part in the interview, then the interviewer needs to provide a brief synopsis of the RWANU Project and ensure that the interviewee understands which project / which types of project activities will form the topic of the interview / discussion. The Field Guide will provide inputs as to how best to describe the project, and a written synopsis will be available to the ET members and Field Assistants. KII Guide for former RWANU staff members 1. What do you consider to be RWANU’s greatest achievements in terms of reaching the project’s goals / objectives / outcomes? a. What were the factors that promoted these achievements? [1.2] b. What challenges were overcome in reaching these achievements? [1.2; 5.2] 2. In which areas was RWANU less successful in reaching its goals / objectives / outcomes? a. What were the factors that inhibited achievement in these areas? [1.2; 5.3] b. What do you think could have been done differently to have achieved success in these areas? [5.2] 3. For the technical sector that you were working on, please describe how a specific intervention [e.g. the intervention that the interviewee is most familiar with] was expected to lead to project outcomes. [2.1; 2.2] a. How effective was the intervention in achieving project goals / objectives? [2.1; 3.1] b. Did the intervention lead to any unintended positive or negative consequences? [2.2] c. What were some of the strengths and challenges in terms of the design of the intervention? [2.1; 2.2; 3.1] d. What were some of the strengths and challenges in terms of the implementation of the intervention? (e.g., aspects of the approach/methods used, the context, the acceptance by beneficiary communities, etc.) [3.1; 3.2; 3.3] 4. Which aspects of the RWANU project have proven to be the most sustainable? a. Describe the systems, processes, capacities, and/or institutional arrangements that RWANU put in place that led to this sustainability. [4.1] KII guide for government stakeholders 1. Describe your role / involvement with the RWANU project, including level of engagement and duration (in years). 2. Which aspects of the RWANU project do you consider to have been the most successful and why? [1.1; 1.2] 3. Which aspects of the RWANU project do you consider to have been the least successful and why? [1.1; 1.2] 4. If you are familiar with RWANU’s targeting strategies, please comment on how effective the targeting approach was in reaching the most vulnerable and/or the poorest individuals. [1.3; 1.4] 5. Based on your knowledge of the RWANU experience, what are some of the lessons, best practices and approaches that should be considered: a. in the design of future food and nutrition security projects? [5.1] b. in the implementation of future food and nutrition security projects? [5.2] 6. Which aspects of the RWANU project have proved to be the most sustainable, and why? a. Describe the systems, processes, capacities, and/or institutional arrangements that RWANU put in place that led to this sustainability, especially in relation to government structures. [4.1; 4.2] KII guide for agro-vet service providers 1. For how long have you worked in providing agricultural / veterinary services (number of years)? 2. What types of the agricultural / veterinary inputs and services do you currently provide? 3. How many full-time / part-time / seasonal staff do you employ (if any)? 4. How/when (year) did you first get involved with the RWANU project? 5. Do you know why you were selected to receive support from RWANU? 6. In what ways / how did the RWANU project support you / your business? What types of support, capacity-strengthening, inputs and/or services did you/your business receive from the RWANU project? 7. What types of resources and capacity did you / your business receive from RWANU to enable you to continue to provide agro-vet inputs and services after the closure of the RWANU project? [4.2] 8. Describe the systems, processes, capacities, and/or institutional arrangements that RWANU put in place that have supported your business after the closure of the RWANU project. [4.1; 4.2] 9. Did RWANU facilitate any linkages between you / your business and other suppliers / service providers / government bodies / projects, etc., and how effective were / are these links? [4.6] 10. At the community level, how did RWANU build up the demand for your inputs/services, and are people willing and able to pay for your inputs/services? What were the successes and challenges in this regard? [4.4] KII guide for health service providers 1. Describe your role / involvement with the RWANU project, including level of engagement and duration (in years). 2. Which aspects of the RWANU project do you consider to have been the most successful and why? [1.1; 1.2] 3. Which aspects of the RWANU project do you consider to have been the least successful and why? [1.1; 1.2] 4. If you are familiar with RWANU’s targeting strategies, please comment on how effective the targeting approach was in reaching the most vulnerable and/or the poorest individuals. [1.3; 1.4] 5. Based on your knowledge of the RWANU experience, what are some of the lessons, best practices and approaches that should be considered: a. in the design of future food and nutrition security projects? [5.1] b. in the implementation of future food and nutrition security projects? [5.2] 6. Which aspects of the RWANU project have proved to be the most sustainable, and why? a. Describe the systems, processes, capacities, and/or institutional arrangements that RWANU put in place that led to this sustainability, especially in relation to government structures. [4.1; 4.2] KII guide for Community Leaders 1. Provide a brief description of your community, including: a. Approximate number of households b. How long village has existed on this site and how it was established (if recent) c. Key facilities, e.g. school; churches; market; shops/kiosks; etc., including approximate distance to nearest health centre d. Main ethnic/language groups e. Main livelihood strategies of the population (including any differences based on ethnicity, poverty, etc.) 2. Do you know why your village was selected to participate in the RWANU project? Describe if so. [1.3] 3. Describe the different activities that the RWANU project and project partners implemented in your village, including the approximate duration (in years) of each project activity 4. Rate the appropriateness of the project activities/implementation approaches according to their level of acceptability to the community (where 1 = least acceptable and 10 = most acceptable) and explain. [3.3] 5. Were there other RWANU activities / types of support (not necessarily implemented in the village itself) that have also helped members of this community? Describe if so. 6. Which aspects of the RWANU project do you consider to have been of greatest benefit to your community and why? [1.1; 1.2; 4.3] a. What are some of the positive changes that took place in your community because of the RWANU project? [2.2] 7. Which aspects of the RWANU project do you consider to have been of least benefit to your community and why? [1.1; 1.2; 4.3] a. Were there any negative changes that took place in your community because of the RWANU project? [2.2] b. What should have been done differently to have had greater benefit / avoided negative changes? [4.3; 5.1; 5.2; 5.3; 5.4] 8. How did the RWANU project identify / select the individuals and households to receive food rations and/or take part in project activities? [1.3] a. Who / what types of households (e.g. in terms of main livelihood strategy, ethnicity, wealth, gender, age, etc.) in your community benefitted most from the RWANU project and why? [1.3; 1.4] 9. How effective was the targeting / selection approach in reaching the most vulnerable and/or the poorest individuals in this community? [1.3; 1.4] a. Describe some of the ways in which the project benefitted the most vulnerable / poorest individuals in your community [1.3; 1.4] b. What should the project have done differently to help the most vulnerable / poorest individuals in your community more effectively? [1.3; 1.4; 5.1; 5.2] 10. Are any of the groups, activities or services that were established / supported by the project still continuing up to now? Describe if so. [4.1] KII guide for Lead/Model Farmers / former Group Leaders / former Group Members 1. Describe your role in the RWANU project, and the length of your involvement (number of years). 2. What were the different activities/practices that you/your group was promoting as part of the RWANU project, and how did you/your group promote these activities/practices? 3. Rate the appropriateness of the project activities/practices according to their level of acceptability to the community/group members (where 1 = least acceptable and 10 = most acceptable) and explain. [3.3] 4. What types of support, inputs and/or services did you/your group receive from the RWANU project? 5. Using a scale of 1 to 5 (where 1 = poor and 5 = excellent), rate the quality, timeliness, frequency and effectiveness of the different types of support, inputs and services provided by the RWANU project. [Record ratings for each of the 4 parameters for each specified support/input/service] [4.3] 6. Did community/group members pay for/invest in any of the support/input/services provided through the project? [4.4] 7. Are any of these support/input/services available to community members up to now? [4.1; 4.2; 4.4] a. If yes, who is providing them, and on what basis (i.e. commercial basis or through government structures or a different project)? b. Approximately what proportion of farmers/livestock keepers/community members currently use/purchase these support/input/services KII guide for Innovation Award Grantees 1. When were you granted an innovation award, and for what? 2. Describe the successes achieved and the challenges you’ve faced in developing your innovation, both during the RWANU project period and after the RWANU project ended. 3. What types of support, inputs and/or services did you/your group receive from the RWANU project? 4. Using a scale of 1 to 5 (where 1 = poor and 5 = excellent), rate the quality, timeliness, frequency and effectiveness of the different types of support, inputs and services provided by the RWANU project. [Record ratings for each of the 4 parameters for each specified support/input/service] [4.3] 5. Did you have to pay for/invest in any of the support/input/services provided through the project? [4.4] 6. Are any of these support/input/services available to you up to now? [4.1; 4.2; 4.4] a. If yes, who is providing them, and on what basis (i.e. commercial basis or through government structures or a different project)? b. Are you able to access these support/input/services, and how? KII guide for Lead Mothers / Male Change Agents 1. Describe your role in the RWANU project, and the length of your involvement (number of years). 2. What were the different activities/practices that you/your group was promoting as part of the RWANU project, and how did you/your group promote these activities/practices? 3. Rank the appropriateness of the project activities/practices according to their level of acceptability to the community/group members (where 1 = most acceptable) and explain. [3.3] 4. What types of support, inputs and/or services did you/your group receive from the RWANU project? 5. Using a scale of 1 to 5 (where 1 = poor and 5 = excellent), rate the quality, timeliness, frequency and effectiveness of the different types of support, inputs and services provided by the RWANU project. [Record ratings for each of the 4 parameters for each specified support/input/service] [4.3] 6. Did community/group members pay for/invest in any of the support/input/services provided through the project? [4.4] 7. Are any of these support/input/services available to community members up to now? [4.1; 4.2; 4.4] a. If yes, who is providing them, and on what basis (i.e. commercial basis or through government structures or a different project)? b. Approximately what proportion of mothers/community members currently use/purchase these support/input/services? In-depth interview guide for case study beneficiary households Clearly visible, wealth- and health-related information that can be compiled through observation alone will be noted by the ET member using the form in Appendix 13. This includes: house & roofing materials; any visible assets such as furniture, cooking facilities / utensils, mobile phone, chickens, kitchen garden, etc.; and general appearance of HH members in relation to health status. 1. Describe your household in terms of: a. Household members currently residing here (gender, age, relationship to HH head) b. Other household members not currently in residence (and where they are / what they’re doing) c. Main sources of income throughout the year 2. Describe your involvement with the RWANU project and over what time period (number of years), i.e. what types of support, inputs and/or services did you/your household receive from the RWANU project? Were you or members of your HH involved in any RWANU groups; what activities you took part in, etc. 3. Do you know why you / your household was selected to take part in the RWANU project? Describe if so. [1.3] 4. Using a scale of 1 to 5 (where 1 = poor and 5 = excellent), rate the quality, timeliness, frequency and effectiveness of the different types of support, inputs and services provided by the RWANU project. [Record ratings for each of the 4 parameters for each specified support/input/service] [4.3] 5. Did you/your HH pay for/invest in any of the support/input/services provided through the project? [4.4] 6. Are any of these support/input/services available to community members up to now and are you able to access them? [4.1; 4.2; 4.4] a. If yes, who is providing them, and on what basis (i.e. commercial basis or through government structures or a different project)? 7. Describe the ways in which you / your household benefitted from the RWANU project – need to be specific, with examples where possible. [1.3; 1.4; 2.2 8. Did your involvement with the RWANU project lead to any negative consequences? Describe if so. [2.2] Focus Group Discussion Guide for existing groups (e.g., savings group) 1. What is the name of your group? What is its purpose? 2. How many members (male / female) does the group currently have? 3. When was it formed (year)? 4. How many members were there when it was established? How many members joined later? When? What attracted the new members to join the group? 5. Describe how the RWANU project helped to establish the group and train the leaders/members. 6. What types of support, inputs and/or services did you/your group receive from the RWANU project? 7. Using a scale of 1 to 5 (where 1 = poor and 5 = excellent), rate the quality, timeliness, frequency and effectiveness of the different types of support, inputs and services provided by the RWANU project. [Record ratings for each of the 4 parameters for each specified support/input/service] [4.3] 8. Did group members pay for/invest in any of the support/input/services provided through the project? [4.4] 9. Are any of these support/input/services available to group members up to now? [4.1; 4.2; 4.4] a. If yes, who is providing them, and on what basis (i.e. commercial basis or through government structures or a different project)? b. Approximately what proportion of group members currently use/purchase these support/input/services? Focus Group Discussion Guide for non-participants 1. [Check the composition of the group…] Was anyone here a member of a group set up by the RWANU Project? Was anyone in your household a member of a group? Did anyone here receive any training or inputs from the RWANU Project? Did anyone in your household receive any training or inputs from RWANU? 2. What are some of the benefits that RWANU brought to this community? Describe. [2.1, 3.1] 3. What type of people benefitted most from the RWANU Project (gender, age, livelihood, wealth, etc.) [1.3, 1.4] 4. What type of people did not benefit? Was this because they did not participate, or because the project was not effective for these people? 1.3, 1.4] 5. Even though you yourselves did not participate in the RWANU project activities, are there any ways in which you feel that you have benefitted from RWANU? Describe if so, with specific examples of technologies/practices learned/adopted and from whom/how. [2.1, 3.1] 6. Did the project have any negative consequences in this community? Describe if so. [2.2] 7. Were there any negative consequences for you and/or your household? Describe if so. [2.2] 8. What should the project have done to avoid or reduce these negative consequences? [5.4] Technical questions for agriculture (including crops and livestock) Questions arising from quantitative data results Questions for technical specialists and stakeholders (e.g. govt staff, private sector providers, former RWANU staff, etc.) Questions for project participants (beneficiaries) at village level Why was there little or no improvement in HH dietary diversity - despite 25,000K hectares under improved technology/management practice, goats (protein), market linkage/value chain efforts such as bees (income), and food aid (resource transfer)? [Project area suffered under erratic rainfall resulting in limited success with home gardens/horticulture & lower than planned agriculture production - but is this enough of an explanation?]  Did RWANU aim to address nutrition through its agricultural interventions? How? [RWANU staff]  In what ways did RWANU promote horticulture activities? Where (which livelihood zone / agro-ecologies)? Who was targeted? How successful were the keyhole gardens? What were the successes and/or challenges? [RWANU staff]  How have other projects successfully promoted horticulture activities? [RWANU staff, govt staff] Were there cross project discussions/learning events during the project period that explored the most effective approaches, e.g. RWANU vs GHC approach (i.e. keyhole gardens vs permaculture; SO1-SO2 participation overlap)?  What was the RWANU experience in working to improve dietary diversity in the context of underlying differences in livelihood (LH) strategies among target population, e.g. - more pastoral LH vs more agriculture LH, other differences?  Did the RWANU project do anything to promote the cultivation and consumption of diverse foods such as vegetables in this village? Describe how this was done and with whom (women, men, youth, etc.). What were the successes and/or challenges?  Did any project activities (e.g., income earning, kitchen/keyhole gardens, savings) actually help to maintain or increase your household food consumption (amount and types)? How or why not? Questions arising from quantitative data results Questions for technical specialists and stakeholders (e.g. govt staff, private sector providers, former RWANU staff, etc.) Questions for project participants (beneficiaries) at village level Pest management interventions were 7,300 percent over plan - reportedly due to an army worm infestation in early 2017, which created great demand for integrated pest management – is this the case? How sustainable were these interventions?  What accounts for the high adoption rate of integrated pest management practices? What are the specific interventions that RWANU promoted in response to the fall army worm infestation? Do farmers still practice these? Are the inputs available up to now? Who / how are the inputs supplied? [RWANU staff, government staff, agro-input dealers]  Was your village affected by the fall army worm infestation in 2017? How did farmers deal with this? Did RWANU help farmers to combat fall army worm? How? Are these inputs still available if you need them? Where? From whom? The adoption of early maturing crop varieties appears to have been a project-based success, exceeding overall targets by 48 percent due to a coordinated engagement with seed companies, input dealers, and radio advertising. How was this achieved? What are the lessons? Is it sustainable? The average numbers of crop produced also went down slightly. Why were the projects not effective in achieving improvements in these areas? (USAID)  What are some of the early-maturing varieties that the project promoted? How were these identified and promoted? Which partners were involved, and what were their roles? Is the seed of these varieties still available? Who is supplying the seed, and how? Is there sufficient demand from farmers? What was done well / should have been done differently? [RWANU staff, govt staff, agro-input dealers]  What are some of the crops and varieties that the RWANU Project introduced to your group / village? Are any of these varieties still being grown up to now? Why do you like / did you not like these crops / varieties? How did you learn about them? How/where did you acquire the seed? If a farmer wanted to acquire seed of these crops / varieties, where/from whom/how could they get it? How many varieties do you typically grow, and for what features (early maturing, yield, pest tolerant, drought tolerant, taste, tradition…?) Are your preferences changing (past 5 years, going forward)? Questions arising from quantitative data results Questions for technical specialists and stakeholders (e.g. govt staff, private sector providers, former RWANU staff, etc.) Questions for project participants (beneficiaries) at village level Project sought a 5.4 percentage point increase from 52.2% to 57.6% but achieved just 47.1% of farmers using improved storage practices in the past 12 months, equivalent to a 5.1 percentage point decrease since the start of project, among the general population. 2017 annual data showed 55% usage amongst project participants against a 65% target for that year. It is difficult to identify whether seasonal production challenges have outweighed the purported increase in farmer storage practices amongst targeted groups.  What are some of the post-harvest storage and handling practices that the project promoted? How were these identified and promoted? Which partners were involved, and what were their roles? What was done well / should have been done differently? [RWANU staff, govt staff]  What are some of the post￾harvest storage and handling practices that the RWANU Project introduced to your group / village? Are any of these practices still being used up to now? Why do you like / did you not like these practices? How did you learn about the practices? The endline assessment showed that approximately half (50%) of the targeted 100% of farmers, carried out 3 or more sustainable agriculture activities promoted by the project, as compared to a baseline of 19%. This is a substantial increase of 30 percentage points - close to 1/3 of the overall population showed improved adoption one year after project activities were completed. A target of 100% across the general population may not have been realistic over the project period. While this increase in three sustainable agricultural practices in the past 12 months, looks impressive, not all areas envisioned were taken up. Acknowledging impact of erratic weather, what practices were most frequently cited, which ones do farmers appreciate the most, where did they learn them, and are planning and able to continue to use those practices in the future.  What agriculture practices did the RWANU project promote? Were they successful? What about introduction of improved crop varieties, soil management, post-harvest, climate mitigation, community vet services? Which of these were more widespread? Useful? What factors led to success, what challenges were faced? What do you think should they have done (instead)? (Govt, RWANU staff, other NGOs/stakeholders)  What agricultural practices were promoted by the project? Did you participate in training? Received other support for them (tech asst, materials). Were they new for you? What changed? Were they successful for your household? Why or why not? Are you still doing them? Who in your community was also included? Do you plan to use these practices in future? Why or why not? Are community members who were not part of the group using these practices? Why or why not? The RWANU project intended to increase wealth and improve diets in ways consistent with government strategy to encourage farmers to engage in markets. Erratic weather was cited as affecting production, and therefore the ability to engage in market preparation/value chain activities as planned. Is this the sole reason? Were the delayed rains/harvest in 2018 the main reason why value chain activities promoted by the project appeared to actually decrease by endline (43% in 2018 before harvest?) What activities were included in  What value chain activities (refer to list in survey) were promoted during project period – over how many harvesting seasons? Could farmers in Karamoja easily take up these activities? What was challenging for the project to promote value chain activities? Who was targeted to engage? Do you have perspectives on the most effective sequencing and layering of agricultural  What value chain practices were promoted by the project? Did you participate in training? Received other support for them (tech asst., materials)? Were they new for you? Were they successful for your household? Why or why not? Are you still doing them? Who in your community was also included? Do you plan to use Questions arising from quantitative data results Questions for technical specialists and stakeholders (e.g. govt staff, private sector providers, former RWANU staff, etc.) Questions for project participants (beneficiaries) at village level the 77% level reported at baseline) vs the activities reported in endline and/or in 2017 (also poor rainfall year)? Were scope of value chain activities realistic given the state of agriculture development in Karamoja during the project period? (USAID) [Note - The low achievement in percentage of target increase was surmised to be influenced by the delay of crops harvest (at the time of survey), and inclusion of non-project participants in endline. A more nuanced picture is provided by the annual figures, which showed rates of adoption between 58% and 216%, which were influenced heavily by the general success of the agricultural season.] production and value chain activities for the different product groups (list)? Should the project have promoted commercial linkages and income generation in other ways? Why? (govt, RWANU staff, stakeholders) these practices in future? Why or why not? Are community members who were not part of the group using these practices? Why or why not? Should the project have promoted commercial linkages and income generation in other ways? Why? What is contributing to significant declines in livestock owners’ access to vet care? Were the baseline numbers realistic (they are very high)? [Note - Just 23% of livestock owners in the endline survey reported accessing vet services as compared to the 2017 monitoring data showing 78% of livestock owners accessing vet services. This contrasts with the 69% usage at baseline. The steep drop in service usage of over 54 percentage points from baseline could have to do with changing herd sizes or with differences in average wealth/livestock assets amongst the general population and project participants. Another key difference could be expected in the access to CAHW in project villages, and a scaling up of this approach elsewhere. [Is this backed up by participant / non-participant disaggregation in the survey as well as cross evaluation with wealth categories, assets??]  What are the sources of veterinary care / medicines currently available to livestock owners in this area? Has there been any change in the past 5 or 6 years? Describe if so. Are you familiar with RWANU’s community vet worker program? What impact has it had?  What are the sources of veterinary care / medicines currently available to livestock owners in this area? Did the RWANU project help livestock owners to access veterinary care / medicines in any way? How? On the percentage of farmers adopting farmer managed natural regeneration practices in the past 12 months, it would be interesting to know what those practices are, which ones they appreciate the most, where they learned  What natural regeneration practices did the RWANU project promote? Were they successful? Which of these were more widespread? Useful? What factors led to success, what challenges were  What natural regeneration practices were promoted by the project? Did you participate in training? Received other support for them (tech assistance, Questions arising from quantitative data results Questions for technical specialists and stakeholders (e.g. govt staff, private sector providers, former RWANU staff, etc.) Questions for project participants (beneficiaries) at village level them, and if they are planning and able to continue to use those practices in the future. faced? What do you think should they have done (instead or in addition)? (Govt, RWANU staff, other NGOs/stakeholders) materials)? Were they new for you? Were they successful for your household? Why or why not? Are you still doing them? Who in your community was also included? Do you plan to use these practices in future? Why or why not? Are community members who were not part of the group using these practices? Technical questions for household incomes, savings & financial services Questions arising from quantitative data results Questions for technical specialists and stakeholders (e.g. govt staff, private sector providers, former RWANU staff, etc.) Questions for project participants (beneficiaries) at village level Was the increase in household income sufficient to make a difference to the lives of the project participants; was it enough to encourage them to maintain the improved practices / new income￾generating activities promoted by the project? What do the increased levels of male and female control over and use of income mean for household dynamics, specifically around food security? Also, what contributed to such large percentage increases in both male and female control over and use of income?  [For former RWANU staff:] What kind of training did the RWANU project provide in terms of household financial decision-making, especially in relation to the roles of women and men in household decision-making? What were some of the changes brought about by this training?  Did your household income change as a result of the activities promoted by the RWANU project? How did it change? What specifically accounted for this change (e.g. farming/marketing/savings practice / IGA)? Is the extra effort required for this new practice/IGA worth continuing in the long term? Why? How has this change impacted your household (e.g. what can/do you buy now that you couldn't before)?  Within your household, who decides how money earned should be spent / invested? Did the RWANU project provide any training in the roles of men and women in financial decision-making? Describe if so.  Has this training had any impacts on your household? Describe if so. What caused the percentage of male and female farmers using financial services to go down? Do the causes differ for each gender? Note: The population survey indicated just 9% coverage as compared with 27% at baseline, indicating an 18 percentage point drop as opposed to the hoped for 25 percentage point increase. Given 52% 2017 achievement of farmers using financial services, against an 85% target, from annual monitoring information, it may reflect a stronger result among participants, and/or be unclear whether facilitated usage dropped off following the end of the project, and/or whether poor cropping conditions had an impact. Project participants vs. non-participants may have utilized financial services at different rates. [More analysis of quantitative data needed here.] It is positive that the  Are savings and credit groups common in the region? What were success factors for the RWANU savings and credit groups? What were challenges for the promotion and sustainability of the savings and credit groups? Did these factors differ with respect to male/female project participants?  What did savings group members typically take loans for? Agricultural activities related to weather? Income/consumption  Did you/another member of your household participate in a savings/credit group? How many others in your community participated? How were they selected or why did they decide to join/form a group? Was the group composed of men, women or both? How did participation in the savings/credit group benefit your household?  Within your household, who decided how the money received through the savings / credit group should be spent / invested? Did the RWANU project provide any training in the roles of men and women in financial decision-making? Describe if so. Questions arising from quantitative data results Questions for technical specialists and stakeholders (e.g. govt staff, private sector providers, former RWANU staff, etc.) Questions for project participants (beneficiaries) at village level 2017 annual monitoring data of project participants showed a close parity between male and female access to financial services. This can be compared to a 9 percentage point difference between male and female access in the endline survey results (22% and 14% respectively). Project assumptions regarding spill over to overall population seem unrealistic in the given timeframe. smoothing? Non-agricultural revenue generating activities? Other HH expenses (wedding/funeral/health/school fees?) Technical questions for health Questions arising from quantitative data results Questions for technical specialists and stakeholders (e.g. govt staff, private sector providers, former RWANU staff, etc.) Questions for project participants (beneficiaries) at village level Immunization rates show a successful campaign, increasing from 64 percent to 83 percent (below the overall target of 90 percent but still a substantial increase (with Karamoja showing the highest immunization rates in the country—an important accomplishment shared with other stakeholders). (IR)  Please describe the RWANU interventions related to child immunization. (Probe regarding health staff training; community level SBC around immunization; whether immunizations were provided only at health facilities or also through mobile brigades and if through mobile brigades, the frequency of community visits; supply chain for vaccines).  What factors contributed to the levels of child immunization in the project area when the project was implemented?  What health-related activities did the RWANU project implement? (Probe for activities related to immunizations; training received on immunizations for community leaders or MC leaders, SBC messages on immunizations, quality of care provided by health service providers regarding immunizations; for IDIs ask to see immunization card and probe on why immunizations received or not).  For caregivers who have older children: Did you do anything different regarding immunizations with your child who participated in RWANU compared to your older children? Explain why/why not. Diarrhea rates among children under five fluctuated without clear improvement, with 23 percent at baseline, 34 percent in 2014, 31 percent in 2015, 28 percent in 2016, and 35.8 percent in 2017. (IR) The use of ORT for children with diarrhea did not reach targets – what contributed to the prevalence of diarrhea and the low use of ORT?  Tell me about the RWANU activities related to management of child diarrhea (Probe regarding health staff training; community level SBC around management of diarrhea; supply chain for ORT). What do you think contributed to the diarrhea prevalence in the project area when the project was implemented? What factors affected the use of ORT for children with diarrhea?  Probe for project activities related to management of diarrhea in children (training, messages about diarrhea management, seeking care from a health center, use of ORT and availability at health center). Describe how you feed your child when your child has diarrhea (probe for feeding more, the same, or less than when child is not ill). Describe how you feed your child when your child has recovered from the diarrhea (probe for provision of additional food).  For caregivers who have older children: Did you do anything different regarding managing your child with diarrhea, during or after the diarrhea, for your child who participated in RWANU, compared to your older children? Explain why/why not. Technical questions for nutrition Questions arising from quantitative data results Questions for technical specialists and stakeholders (e.g. govt staff, private sector providers, former RWANU staff, etc.) Questions for project participants (beneficiaries) at village level Is the lack of change in stunting rates after five years of programming alarming? What would explain this? Underweight rates of children under two appear to have increased consistently throughout the project period, from 22.7 percent in 2014 to 32.1 percent in 2017 (according to IPTT data)—this is surprising and will need to be explored further. The differential between boys and girls (with boys at higher underweight) also increased significantly over the period. The 2016 Uganda Demographic and Health Survey (DHS) recorded rates of children under five underweight (weight for height) in Karamoja at 13 percent but did not provide rates for children under two. Exclusive breastfeeding improved substantially, from 58.5 percent to 75.8 percent, which is below the target of 86.7 percent but a strong improvement. EBF in boys skyrocketed per baseline/endline results, and barely moved for girls. May be worth exploring? What contributed to the large increase in the percentage of underweight women? * Women’s dietary diversity did not improve in the project area – what contributed to the lack of impact on women’s dietary diversity? 1. How long have you been in your current job? Was this the same job that you had during the RWANU project? Please describe your job at the time of the RWANU project. 2. Please describe your experience with the RWANU project. What did the project do/ what were the project activities? What, if any, was your involvement regarding these activities? 3. Did you receive any training by the project? If yes, what training did you receive? Can you tell me what you learned? (Probe regarding training received in management of acute malnutrition, prevention of chronic malnutrition, breastfeeding, micronutrient malnutrition, IYCF, nutrition for pregnant or lactating women). Was the training useful? Why or why not? 4. Did the RWANU project provide supportive supervision, mentoring, or coaching? If yes, please tell me more about this. (Probe for content, frequency). Was the supportive supervision helpful? If yes, how? If not, why not? 1. What were the primary health problems that you saw among young children during the time the RWANU project was being implemented? (Probe for prevalence of acute malnutrition, diarrhea, malaria). Was the prevalence of these health problems the same over 1. What activities related to food and nutrition did the RWANU project implement? 2. Did you participate in any of these RWANU project activities? If yes, can you tell me what you did/how you participated? 3. Did you receive any training as part of your involvement in the project? What did you learn in the training? What did you like, if anything, about the training? Do you think the training could be improved? If yes, how? 4. Did you participate in the MCGs/ MCA groups? What did you do in the groups? What did you learn, if anything, in the groups? Are the groups still meeting? Why do you think this is the case? 5. Were you pregnant at any time during the RWANU project? If yes, was this your first pregnancy? If not, is there anything you did differently in this pregnancy compared to prior pregnancies? If yes, what did you do differently? (Probe regarding foods eaten, prenatal visits, etc.). Were there things you learned about in the MCGs that you wanted to do during the pregnancy but you could not? If yes, could you provide a few examples? Why was it hard to practice the new behavior? Were there any foods you could not eat during the pregnancy? Why? Were there any special foods that you ate during your pregnancy? If yes, what were they? Did you eat more, the same, or less than when you were not pregnant? 6. Did you breastfeed your child after the pregnancy, when the child was born? When did you first breastfeed the infant? Why? Did Questions arising from quantitative data results Questions for technical specialists and stakeholders (e.g. govt staff, private sector providers, former RWANU staff, etc.) Questions for project participants (beneficiaries) at village level What would explain this? *Note: Prevalence of underweight women (of reproductive age) rose significantly during project, to 45.5% for general population, and exceeded the baseline by 18.7 % points. This corroborates FEWSNET and project reports indicating a very unstable food security situation over the project period. More information is also needed about conditions and changes amongst surveyed households since the project closed. While targeted reductions may have been unrealistic, the reversal of progress is unusual and indicates greater vulnerability, rather than increased resiliency. Food assistance for mothers and children may have helped these vulnerable groups prevent a worsening of their situation. time, or did the prevalence decrease or increase? Why do you think that was the case? What, if anything, do you think could have been done to improve the situation? (Probe regarding detection of children with acute malnutrition, treatment, availability of RUTF or RUSF, supply chain for therapeutic food, loss to follow-up; also probe on EBF, IYCF practices (frequency, variety, quantity, responsive feeding, feeding when child is ill e.g. when is semi-solid food introduced to infants (at what age)? Describe the first foods an infant is given? How often does a child eat at one year of age? Same for boys and girls? How often does a child breastfeed at one year of age? Same for boys and girls? Do mothers empty the breast when breastfeeding their infant? At what age does a woman stop breastfeeding completely? Is this the same for boys and girls? When a child presents as malnourished, how do you care for him/her? What do you counsel the parents? 5. What are the primary health problems you see among pregnant women? Lactating women? How has this changed, if at all, over time? Why do you think this is the case? (Probe regarding diet, customs regarding women’s diets when they are pregnant; any special foods to eat or avoid? Do they eat more, same, or less than when they are not pregnant?) you give the infant anything other than breastmilk after the child was born? If yes, what did you give him/her? When? Why? Did you have any problems breastfeeding your child? If yes, what was difficult? Were there things you learned about in the MCGs that you wanted to do when breastfeeding your child but you could not do? Could you provide a few examples? Why was it hard to practice the new behavior? 7. If this is not your first child, is there anything you did differently regarding feeding or caring for this child compared to your older children? If yes, what did you do differently? (Probe regarding IYCF). Were there things you learned about in the MCGs that you wanted to do for your child but you could not? Could you provide a few examples? Why was it hard to practice the new behavior? Were there any foods you could not provide to your child? Why? Were there any special foods that you gave your child when he/she was less than 2 years of age? If yes, what did you give him/her to eat that you felt was special? How often did you feed your child each day? Why this number of times? How much of each food did you feed your child? Why this amount? When your child is fussy and does not want to eat, what do you do? 8. Do women with infants breastfeed their boy and girl infants the same, or differently? Tell me more about this. 9. When is semi-solid food introduced to infants (at what age)? Describe the first foods an infant is given? How often does a child eat at one year of age? Is that practice the same for boys and girls? Questions arising from quantitative data results Questions for technical specialists and stakeholders (e.g. govt staff, private sector providers, former RWANU staff, etc.) Questions for project participants (beneficiaries) at village level 6. Are you familiar with the MCGs and MCAs in the RWANU project? If yes, could you tell me more about them? Did you interact with them? If yes, how? Were the groups effective? Why or why not? 7. How do you feel about the overall impact of the RWANU Project? What were its strengths, if any? What were its areas for improvement? (Probe regarding impact on breastfeeding, especially among boys, and potential reasons behind this outcome; lack of impact on nutritional status and perceived reasons for this outcome). 10. How often does a child breastfeed at one year of age? Is that practice the same for boys and girls? 11. At what age does a woman stop breastfeeding completely? Is that practice the same for boys and girls? 12. Do children generally get enough to eat here? Both boys and girls? Are children often hungry? 13. How do you feel about the RWANU project? What do you think went well? What do you think could be improved? Why do you feel this way? Technical questions for WASH Questions for former RWANU staff) Questions for technical specialists and stakeholders (e.g. govt staff, service providers, etc.) Questions for project participants (beneficiaries) at village level 1. The project successfully improved knowledge on WASH, as it came close to achieving the monitoring target at 87% of participants reported knowing at least 4 of 6 IYCF and 3 of 8 MCH practices. Knowledge gains have been an important milestone for the project. Just over 100% of targeted number of people were trained on environmentally appropriate hygiene and sanitation behaviors, 94% of whom were women. Despite the impressive increase in knowledge, the application of good WASH practices remained very low or even declined (as described in the points below). Did the relatively small number of men trained inhibit uptake of behaviors? 1.1 Who was targeted to receive WASH training / messaging? What are the different ways in which WASH training was provided to these groups? Were men included in any of these? 1.2 Did the RWANU project address gender-related issues in promoting hygiene and sanitation behaviors? If so, how? What were some of the successes and challenges in addressing gender issues related to WASH? 1.1 The RWANU project was very successful in increasing people’s knowledge about environmentally appropriate hygiene and sanitation behaviors, but less successful in changing behaviors. Is this the same for other projects? Based on your experience, what are some of the reasons for the limited changes in WASH behaviors? 1.2 Do you know of any projects / interventions that have achieved success in promoting good WASH behaviors in the Karamoja region? What are the lessons that can be learned from this project? 1.1 What are some of the hygiene and sanitation practices that you learned through the RWANU project that you have been able to apply in practice? What are the benefits of these practices for your household? 1.2 Can you remember learning about any hygiene and sanitation practices that you have not been able to apply in practice? Why have you not been able to apply these? 2. What the barriers are to using soap and water for hand washing? At baseline, just 4% of households had soap and water at a hand washing station commonly used by family member. The general population target was just 6%, while the actual achievement was even less than baseline at 1.2%. RWANU updated its targets for participants during the project period, aiming to achieve 3% of households with tippy taps and soap. 2017 monitoring data exceeded this target achieving 3.5% of population surveyed - over double the general population incidence, but still very limited. Why is hand hygiene compliance still low after all the efforts in recent years? Is soap cost, lack of water, or behavioral priorities the main challenge? 2.1 How did the RWANU project package and implement the hand washing messages to the communities? Who was targeted? What were some of the successes and challenges in promoting good handwashing? 2.2 What are the constraints of hand washing with soap in the RWANU project intervention area? Pls rank these constraints (1=greatest constraint). 1.1 What are the constraints of hand washing with soap in the RWANU project intervention area? Pls rank these constraints (1=greatest constraint). 1.2 What are some of the local communities’ knowledge, attitudes and perceptions about hand washing practices? 1.3 For households unable to buy soap, what other detergents are generally used? Please rank them. 1.1 Did you receive any training / information from RWANU about hand washing? Describe if so, including the messages received; how often messages / trainings were provided; where; by whom; who was included in these trainings / messages (male / female / youth, etc.) 1.2 What are the critical times for hand washing? How / with what do you and others in your household normally wash your hands? 1.3 Are / were there any tippy taps in this village? (If no, why not?) Do you use the tippy tap? (If no, why not?) Questions for former RWANU staff) Questions for technical specialists and stakeholders (e.g. govt staff, service providers, etc.) Questions for project participants (beneficiaries) at village level 2.3 What are some of the local communities’ knowledge, attitudes and perceptions about hand washing practices? 2.4 For households unable to buy soap, what other detergents are generally used? Please rank them. 1.4 [If yes, ask to see the tippy taps to observe if water and soap are available and ask why not (if appropriate) and why people are not using it (if appropriate)] 3. Why did the overall percentage of children under age 5 with diarrhea in the last two weeks increase to almost one third of all children? Why was there a decrease in the number of people using improved drinking water sources? At baseline, 41.9% of households used an improved drinking water source. The general population target was 86%, while the actual achievement was even less than baseline at 36.9%, a 5% point decline. [Unclear whether monitoring data is ONLY for communities where water activities were being targeted as opposed to the full RWANU participant population - if so, the monitoring data are representative only of discrete (and sometimes quite limited) activity focus areas. CONFIRM? Did RWANU target more disadvantaged areas than the general population survey?] 3.1 Were there any improved water sources provided by the project? If yes, how were the targeted communities selected? What were some of the successes and challenges in relation to the provision, use and maintenance of improved water sources? 3.2 Describe the water user committees established / trained by RWANU project in relation to the operations and maintenance of water services. What were some of the successes and challenges encountered? 3.3 Did the RWANU project train communities on safe water chain handling methods? How was the health promotion packaged and delivered; who was targeted? 3.4 In your view, what could have caused the increased percentage of children under 5 with diarrhea illness in the RWANU project? Is this situation different from other 3.1 In your view, what could have caused the increased percentage of children under 5 with diarrhea illness in the RWANU project? Is this situation different from other parts of Karamoja Region? If yes, why? 3.1 [For village leaders / water user committee leader:] Were any improved water sources provided by the project for this village? If yes, why was your village selected? How is the improved water source being maintained currently? What are the constraints and challenges in maintaining the water source? 3.2 [For water user committee leader / members:] When was the water user committee formed? Did the RWANU project provide any training/support to the committee? Describe if so. What are the successes and challenges encountered by the committee in the operations and maintenance of water services in this village? 3.3 [For community members:] What is the source of water that you normally use for drinking? What is the distance / time that you take to collect water for drinking? Has there been any change in the distance / time taken to collect water due to the RWANU project? Describe if so. 3.4 Do you/ your household treat drinking water in any way? If yes what are methods do you use for water treatment and why? How often do you Questions for former RWANU staff) Questions for technical specialists and stakeholders (e.g. govt staff, service providers, etc.) Questions for project participants (beneficiaries) at village level parts of Karamoja Region? If yes, why? treat drinking water? How do you store your drinking water? 3.5 Did RWANU provide any training / messages on safe drinking water? Describe if so, including the messages received; how often messages / trainings were provided; where; by whom; who was included in these trainings / messages (male / female / youth, etc.) 3.6 What do you think are the causes of water borne diseases like diarrhea in the community, more especially with children under the age of 5? 4. At baseline, just 4% of households were using an improved sanitation facility. The general population target was 25%, while the actual achievement was even less than baseline at 1.2%. The project achieved a 2.8% point decline. However, the population with access to an unimproved sanitation facility did improve from 11.3% at baseline to 18.7% at end line, indicating lower level progress. Population level impacts in this area are challenging, without strong engagement and community mobilization. Among participants targeted by this project component, even a more modest target of 15% was not achieved by 2017, falling short by 6.2% points. It is unclear whether the latrine coverage report was not representative of communities targeted, or other challenges represented themselves. The relationship between open defecation at over 80% and diarrhea incidence, influencing nutritional status in children, is established and a likely factor in limiting progress. 4.1 How did the RWANU project address the need for improved sanitation facilities? What approaches were implemented, e.g. CLTS- Community led total sanitation, others? 4.2 In your view, what are the challenges limiting communities to move up the sanitation ladder? Please rank (1=greatest limitation). 4.3 Did the RWANU project attempt to address any of the factors mentioned, if yes how? 4.1 Are there existing functional community-based management structures in Southern Karamoja promoting health education activities? If so how effective is their role in sanitation promotional activities? 4.2 In your view, what are the challenges limiting communities to move up the sanitation ladder? Please rank (1=greatest limitation). 4.1 [For village leaders:] Are there any functional community-based structures promoting health education in your village? If so, how effective is their role in sanitation promotional activities? Were they trained by the RWANU project? 4.2 [For community leaders in villages with improved sanitation facility:] Who owns the facility (if owned by an individual household, what are the characteristics of that household)? How many people / households use the facility? Who is responsible for its maintenance? What are some of the factors that helped to allow for the construction of such a facility? What were some of the challenges that were overcome? Did RWANU provide any inputs to implement/ construct/maintain any improved sanitation facilities in this village? 4.3 [For village leaders:] Did RWANU provide any training on improved sanitation? / [For Questions for former RWANU staff) Questions for technical specialists and stakeholders (e.g. govt staff, service providers, etc.) Questions for project participants (beneficiaries) at village level community members:] Did you receive any training on improved sanitation? Describe if so. 4.4 [For community members:] Does your household have its own latrine? If yes, what type of latrine; is it shared with others; how many? How easy / difficult was it to construct? If no, where do you go to defecate, and how many other households use this facility? What are the challenges limiting households in this village to improve their sanitation facilities? Could you rank the most pressing factor? Did the RWANU project address any of the factors mentioned, if yes how? ANNEX 6: SELECTION OF VILLAGE SITES Table 6.1: Villages, Sub-county Towns, and District Towns Visited by the Evaluation Team No table of figures entries found. Subcounty Village Livelihood Zone Language/Ethnic Group Food Distribution*/ SO1/SO2 Nakapiripirit Moruita Kopedur Agricultural Mixed Karamojong and Pokot All three Nakapiripirit Lolachat Natirae Agro-pastoral Karamojong Food distribution and SO2 Napak Iriiri Morusapir Agricultural Karamojong SO1 and SO2 Napak Lopeei Naoi Agro-pastoral Karamojong All three Napak Lopeei Loitakwa Agro-pastoral Karamojong SO1 and SO2 *Food distribution is noted as a component in some but not all of the visited communities because the RWANU food rations were provided to participants in villages located greater than five kilometers from a health facility, while participants located in communities within five kilometers of a health facility received a World Food Program (WFP) maternal child health and nutrition ration for pregnant and lactating women and children ages 6–23 months. Table 6.2: PBS-sampled villages (sample frame) and sampling process District Sub-county Village Livelihoo d zone Interval [2] selection of eligible sites by District and Livelihood zone Food Dist / SO1 / SO2 Notes AMUDAT KARITA KANGONDI agro￾pastoral none District not selected AMUDAT LOROO NAKIPON pastoral none District not selected AMUDAT LOROO LOBOROKOCHA pastoral none District not selected AMUDAT LOROO NAMOSING pastoral none District not selected MOROTO NADUNGET LOKORIROT agro￾pastoral SO2 only District not selected MOROTO NADUNGET NABOKAT agro￾pastoral SO2 only District not selected MOROTO TAPAC KATIKEKILE agro￾pastoral none District not selected MOROTO TAPAC NARACHUCH agro￾pastoral none District not selected District Sub-county Village Livelihoo d zone Interval [2] selection of eligible sites by District and Livelihood zone Food Dist / SO1 / SO2 Notes MOROTO TAPAC NAUT agro￾pastoral none District not selected MOROTO TAPAC LONYILIK agro￾pastoral none District not selected NAKAPIRIPIRIT KAKOMONGOLE ACELEL agricultural 1 All 3 Eligible NAKAPIRIPIRIT KAKOMONGOLE LODOKET ANGITOME agricultural SO1 & SO2 Two villages listed with similar name - might have just been SO2 only – ineligible NAKAPIRIPIRIT LOREGAE AJOKOKIPI agricultural NUYOK village NAKAPIRIPIRIT MORUITA AYAS agricultural none? Not listed in RWANU database NAKAPIRIPIRIT MORUITA KOPEDUR agricultural 2 All 3 Selected NAKAPIRIPIRIT NAMALU LOKOMAIT agricultural SO1 & SO2 Eligible NAKAPIRIPIRIT NAMALU NAKILORO agricultural NUYOK village NAKAPIRIPIRIT NAMALU LOKITELA ALOKWA agricultural NUYOK village NAKAPIRIPIRIT NAMALU MORUAJORE OKUDUD agricultural All 3 Three villages listed with similar name - might have just been 2 activities, SO1 & SO2 - ineligible NAKAPIRIPIRIT LOLACHAT NATHINYONOIT agro￾pastoral 1 SO1 & SO2 Eligible NAKAPIRIPIRIT LOLACHAT KANANGAKINOI agro￾pastoral 2 All 3 To be replaced with another village with FD+SO2 NAKAPIRIPIRIT LOLACHAT MOAUANGAMIO N agro￾pastoral All 3 Eligible NAKAPIRIPIRIT LOLACHAT NATIRAE agro￾pastoral FD & SO2 Replacement for above NAKAPIRIPIRIT LORENGEDWAT LOKWAKWA agro￾pastoral All 3 Eligible NAKAPIRIPIRIT LORENGEDWAT LONANGAT agro￾pastoral SO1 & SO2 Eligible District Sub-county Village Livelihoo d zone Interval [2] selection of eligible sites by District and Livelihood zone Food Dist / SO1 / SO2 Notes NAKAPIRIPIRIT NABILATUK LONGAROI agro￾pastoral SO1 & SO2 Eligible NAKAPIRIPIRIT NABILATUK NAKOBEKOBE agro￾pastoral All 3 Eligible NAKAPIRIPIRIT NABILATUK NAPONGAE SOUTH agro￾pastoral FD & SO2 Eligible NAPAK IRIIRI ALEKILEK agricultural 1 All 3 Eligible NAPAK IRIIRI KALOPIDINGA agricultural FD only NAPAK IRIIRI MORUSAPIR agricultural 2 SO1 & SO2 Selected NAPAK IRIIRI NAMINIT ALICIA agricultural NUYOK village NAPAK IRIIRI LOJOM agricultural All 3 Eligible NAPAK IRIIRI NAMINIT agricultural FD & SO2 Eligible NAPAK LOKOPO KATUBAKUYON agricultural FD & SO2 Eligible NAPAK LOKOPO NAMORU￾AKWANGAN agricultural All 3 Eligible NAPAK LOKOPO NAPUSILIGOI agro￾pastoral NUYOK village NAPAK LOKOPO LOPANA agro￾pastoral none? Not listed in RWANU database NAPAK LOPEEI NAOI agro￾pastoral only eligible agro￾pastoralist site in Napak All 3 Selected NAPAK LORENGECORA NAWATOM agricultural NUYOK village NAPAK LORENGECORA NAKWAKWA agricultural NUYOK village NAPAK LOTOME ADWARAMUKUN Y agro￾pastoral NUYOK village NAPAK LOTOME NAKAALE agro￾pastoral NUYOK village District Sub-county Village Livelihoo d zone Interval [2] selection of eligible sites by District and Livelihood zone Food Dist / SO1 / SO2 Notes 5th village site needed in Napak District, Lopeei Sub-county must be in agro-pastoral zone and must have SO1 + SO2 NAPAK LOPEEI LOITAKWA agro￾pastoral SO1 & SO2 Selected from RWANU database as the first site that met the selection criteria above ANNEX 7: DESCRIPTIONS OF TECHNICAL INTERVENTIONS AND THEIR SUSTAINABILITY ANNEX 7A: CROP-BASED AGRICULTURAL AND MARKETING INTERVENTIONS Farmers were trained through farmer training groups (FTGs), and some FTGs were subsequently selected to become block farming groups or marketing groups to encourage commercial farming. The RWANU project formed FTGs of 25 male and female members, including one or more lead farmers. The Annual Results Report for 2014 reveals that the lead farmer model was, in fact, introduced in the second year of the project, after the uptake of improved technology and farming practices was seen to be slow. The report states that, “Going forward, RWANU will intensify the extension support to the farmers groups through more structured and systematic training processes and by deploying a network of lead farmers to encourage peer learning and expand reaches.” The lead farmers attended various training courses on improved agronomic and post-harvest practices, as well as courses on group dynamics and gender issues. The improved practices were demonstrated to the group members using demonstration plots, established under the supervision of the farm extension worker, a RWANU employee. Farm extension workers were primarily educated youth from the local communities. The FTG received regular training from the extension worker, who also provided regular supervision to the lead farmers and demonstration plots. RWANU supported selected FTGs to establish “block farms” of 10 acres to promote commercial production and improved practices.5 Some groups (60 groups from 2015 and 42 groups from 2014) were given a pair of oxen and a plow, and others used hired tractors to open up new farm land. Due to the lack of availability of tractors in Karamoja Region, RWANU had to hire tractors from Soroti and Mbale. Block farmers contributed 30 percent of the cost of inputs, such as tractor hire and seed. 5 An additional purpose mentioned in one report was to test whether farmers could work together. ANNEX 7B: LIVESTOCK INTERVENTIONS In this report, activities included under the livestock sector refer mainly to goat distribution and support to 211 women’s livestock groups (WLGs) and CAHW training. In addition to these two main activities, a pair of work oxen for plowing was given to each of 102 block farming groups, and training was provided for beekeeping and honey production. The project provided goats to 2,242 women in 211 groups (RWANU Annual Review Report, 2017). Five young female goats6 were given to each member; individual ownership increased the sense of responsibility (Lepillez, 2016). A study had been undertaken to determine how many goats would be economically viable for milk production; previous experience had found that two goats was not enough. The goats provided were quite young and had to be looked after for about a year before they started kidding and producing milk. A goat kids once a year and produces milk for about one month or slightly more after each kid. In addition, each group received one buck for breeding. Local does, Galla does, Galla bucks, and Toggenburg cross-bred bucks were distributed. Galla goats were introduced due to low adaptability of Toggenburg crosses bucks in the drier areas; 40 percent of kids born to Toggenburg crosses bucks died, compared to 4 percent of kids born to Galla bucks. Galla are a high milk yielding breed common in northern Kenya and southern Ethiopia (ARR 2014). Women were trained in herd management, buying and marketing of goats, planning and budgeting, and group savings and credit. Each WLG was linked to a local CAHW who had been trained by the project for health monitoring and veterinary services. CAHW training included animal husbandry as well as business and livelihood skills. The CAHWs also received bicycles; in addition, some received a “starter kit” containing livestock medicines. In general, the educational background and experience of CAHWs are very mixed due to the different approaches of earlier projects. Some projects trained those with formal education, others trained those without formal education but with experience in treating animals. RWANU selected CAHWs who had previously received some training and were located outside the main trading centers. Under the beekeeping component, 24 groups were trained, and individual group members were supplied with improved beekeeping equipment. Where possible, the residential training sessions included one day of gender training and one day of conflict training. The residential training sessions were supplemented by hands-on, practical onsite training conducted by one of the two apiary officers employed by the project. Toward the end of the project, efforts were made to establish a beekeepers’ association to support ongoing training and marketing needs, 6 A study had been undertaken to determine how many goats would be economically viable for milk production; previous experience had found that two goats was not enough. The goats provided were quite young and had to be looked after for about a year before they started kidding and producing milk. A goat kids once a year and produces milk for about one month or slightly more after each kid. The evaluation team found that group members tended to pair up and share milk between them when one woman’s goats had milk but the other’s didn’t. By sharing the milk from 10 goats between them, two women could each have milk for about 6–7 months of the year. ANNEX 7C: MARKET DEVELOPMENT AND VSLAS In this evaluation report, the market development sector includes the following interventions: marketing of honey and vegetables; bulking and marketing of grain crops; support to private sector agro-vet input dealers; and the grants made through the women’s livelihood innovation award (WLIA) and youth livelihood innovation award (YLIA). Each intervention was implemented on a relatively small scale, encompassing 24 beekeeping groups, 12 horticulture groups, 102 block farming groups (approximately 2,000 individual farmers); 36 innovation awards for women and youth (totaling 465 individuals); and 10 individual agro-vet input dealers. Each is described in the paragraphs below. RWANU supported a value chain approach to honey, providing bee-keeping training and equipment to the individual members of 24 groups. A honey marketing company (Golden Bees) was identified to establish a buying center in Moroto. The marketing company supplied bee-keeping equipment to the project, helped with training in beekeeping, and also trained 15 local artisans from all four districts in the construction of improved hives, contributing towards the local availability of hives. Toward the end of the project, efforts were made to establish a beekeepers’ association to support ongoing training and marketing needs, Support to vegetable production and marketing appears not to have taken a value chain approach, despite its inclusion in the project proposal document. Different interventions included the provision of vegetable seeds, establishment of horticulture production groups, provision of drip irrigation kits, and promotion of keyhole gardens under SO2. Farmers met by the evaluation team reported that RWANU provided minimal support in the marketing of vegetables; farmers were simply told to sell their vegetables outside local health centers. Interestingly, the 2016 market assessment did not include vegetables among the five market interventions assessed (Kayobyo et al, 2016). The Youth Livelihood Innovation Awards (YLIA) and the Women Livelihood Innovation Awards (WLIA) were introduced in the fourth year and intended to achieve the following:    To empower youth and women in enterprise building and business development skills. To provide productive and decent means of livelihoods for youth and women so that they do not become a security risk as a result of being idle and without a source of income. To expand opportunities for the RWANU supported farmers in the districts of Napak, Moroto, Nakapiripirit and Amudat by expanding markets, employment, income and expertise. In the last two years of the project, 28 YLIAs and 8 WLIAs were supported, totaling 465 individual members (166 men, 275 women) (Ishara, 2017). The applicants were encouraged to generate practical and innovative business ideas basing on the community needs which at the same time can lead to successful businesses and serve as a source of living to the youth and women. Some applicants were already RWANU project participants, but applicants did not necessarily have to be participating in other RWANU activities in order to apply for the awards, thus reducing the likelihood of “layering”; grants were awarded on the strength of the group’s business application. RWANU’s intervention to establish a private sector agro-vet input market system included training, mentorship and in-kind grants for eight agro-vet dealers. Training focused on business management (record keeping and financial management). In addition, RWANU linked the agro-input dealers to national seeds companies, including FICA, Victoria Seeds, Equator Seeds and Pearl Seeds. The project organized business talks and discussions between the dealers and seed companies on modalities of partnership and operations. Types and amount of seeds required and modalities of payment were also agreed upon. Through these linkages, agro-input dealers signed MoUs to purchase seeds on credit with flexible terms of payment. As a result, seed companies identified several of the agro￾input dealers as agent that represents them and promote their seed brands in South Karamoja. RWANU facilitated the formation of a marketing association, the Karamoja Agri-business Association (KABA), which intended to pull all the final pieces of the market linkages developed by RWANU together for long-term sustainability. By creating a common platform for agricultural inputs and commodities to move in and out of Southern Karamoja (i.e., the information transfer under the brand), the brand introduced a new language through which farmers change their production practices. The 25 KABA members included the eight agro-input dealers mentioned above, two veterinary drug shop operators, four farmers, three bulking center operators, and eight cereal traders located across the four districts of the project. The government registered the association as a community-based organization. All groups established under SO1 were trained as VSLAs and provided with the necessary stationery for record-keeping and specially designed savings boxes (made with three padlocks so that all three key holders had to be present for the box to be opened). Group members contribute money each week – depending on the group, either all members contribute the same amount, or each member contributes what they can afford up to an agreed maximum. The money from the weekly contributions is saved in the box in two different accounts: a social welfare fund and a savings account. Group members can access money from the social welfare fund in emergency cases, e.g. to pay for medical costs. Money from the savings account can be used as short-term credit by individual members who must apply for a loan and repay the money with interest. After nine months or one year, a ‘share-out’ takes place, when the savings that have accumulated in the box is shared out among the members according to each individual’s contributions. ANNEX 7D: HOUSEHOLD-LEVEL NUTRITION AND HEALTH INTERVENTIONS The technical interventions in the household-level nutrition and health component included:       mother care group (MCG) approach integrated management of acute malnutrition (IMAM) SURGE approach, where lead mothers screen children for severe acute malnutrition (SAM) and moderate acute malnutrition (MAM) using mid-upper-arm circumference (MUAC) tapes male change agent (MCA) approach drama groups goat distribution keyhole kitchen gardens Mother Care Groups (MCGs). An MCG is a group of 10 to 15 community-based volunteer lead mothers who meet regularly with project staff for training and supervision, and then hold meetings and visit regularly with 10 to 15 neighbors, sharing what they have learned and facilitating behavior change at the household level. In the RWANU project, lead mothers were trained on seven modules and 28 different health and nutrition topics through a cascade training approach, first through four-week cycles, and then in 2016 through six-week cycles to provide more time for follow-up on knowledge and practice in the home. The cycle included RWANU project nutritionists training field coordinators, who trained health promoters, who trained lead mothers, who trained mother caregivers, with the nutritionists supervising the trainings at different levels of the cascade to ensure quality, as did health promoters with lead mothers. The trainings were followed by home visits to encourage and document knowledge and adoption of behaviors. The training topics were (1) IYCF, (2) maternal health and nutrition, (3) agriculture and nutrition linkages, (4) WASH, (5) family planning, (6) child health, and (7) health user rights. The MCGs included the construction and planting of keyhole gardens, and a limited number of mothers (approximately 420) received goats to diversify women’s and children’s diets. The total number of MCGs by the end of the project was 345, just short of the life of activity (LOA) target of 350, and were comprised of 3,499 lead mothers (ACDI/VOCA 2017b).7 Table 7D.1: Best Practice in MCG Compared to MCG Implementation Under the RWANU Project Best practice in MCGsa RWANU implementation of MCGs The workload of Care Group Volunteers is limited: no more than 15 households per Care Group Volunteer Mother care group volunteer workload was generally between 10 to 14 households Care Group Volunteer contact with her assigned participant mothers— and Care Group meeting frequency—is monitored and should be at a minimum once a month, preferably twice monthly. Mother Care Group Volunteer contact with her assigned participant mothers— and Care Group meeting frequency—was initially once per month, but in 2016 was changed to every 6 weeks to allow more time for follow-up and refresher training. 7 In March 2015, Samaritans Purse completed a three-year DFID-funded Maternal and Child Health project in Napak. The project included training women using the MCG approach, with 219 MCGs comprised of 2,130 leader mothers/grandmothers and 35,597 mothers. In April 2015, RWANU started the incorporation and restructuring process of the Napak MCGs to align them with the RWANU MCG inclusion criteria, which included pregnant and lactating mothers and women with children under the age of 5 (ACDI/VOCA. 2015). RWANU staff shared that Samaritans’ Purse had rolled out their own maternal and child nutrition curriculum, which covered all the optimal practices, and the transition from the MCGs being under Samaritan’s Purse to RWANU went smoothly; after about two months of training the Napak groups were performing well. Best practice in MCGsa RWANU implementation of MCGs Each Care Group Volunteer is to reach 100% of households in the targeted group on at least a monthly basis, and the project should attain at least 80% monthly coverage of households within the target group. Coverage is monitored RWANU staff shared that coverage of eligible households with the MCG model was 100%. During monthly meetings with the mother caregivers, the average attendance among mother caregivers was 60–70%, and mother leaders conducted monthly home visits with approximately 60– 70% of mother caregivers. Lead mothers prioritized home visits for mother caregivers who missed the group sessions, so mother caregivers should have received information through the mother caregiver meetings or home visits or both. Data from 2014 showed that lead mothers were not making home visits with the expected frequency, so staff re-oriented lead mothers about the importance of the home visits. The Care Group Volunteers use some sort of visual teaching tool (e.g., flipcharts) to do health promotion at the household level. RWANU developed flipcharts to support health promotion and tested the flipcharts, which resulted in changes to improve the comprehension and acceptability of the materials, such as photos instead of drawings and ensuring the materials were appropriate for low literacy populations. Participatory methods of behavior change communication are used in the Care Groups with the Care Group Volunteers and by the volunteers when doing health promotion at the household or small-group level/effective interpersonal communication. RWANU used the ASPIRE participatory method to train, using games, songs, storytelling, use of pictures, activities, taking commitment of MCG members, discussing challenges, following up on practice in the home, and telling someone else about positive practice. RWANU staff ranked lead mothers 8 out of 10, or very good, regarding their capacity in interpersonal communication because of the ASPIRE methods of adult learning used and the training that focused on appropriate delivery style and content. The ASPIRE steps include: 1. Game to break the ice 2. Attendance, troubleshooting and recap of previous topic 3. Story (predetermined, and relevant for lesson at hand) 4. Ask questions to generate discussion around current practices brought to memory by the story 5. Show and explain picture/counselling card with key lesson messages 6. Probe on perceived barriers/problems with the new practice 7. Inform or hold a discussion on possible practical actions to reduce the barriers 8. Request commitment to try the new/recommended lesson practice 9. Examine previous commitments and how the new one adds to it Supervision of Promoters and at least one of the Care Group Volunteers occurs at least monthly RWANU had 7 field coordinators who each supervised 7 health promoters on a monthly basis. Health promoters supervised about 70 mother care leaders. If a health promoter supervises one mother care leader a month, this would be inadequate to be able to supervise each of the mother care leaders during the life of the project. Health promoters did meet with the mother care group twice in each 6-week period, first to train them and then to follow up on the training. Data from 2017 showed that after the project training on the last MCG module on Health User Rights, health promoters supervised 83% of the lead mothers as the lead mothers conducted the meetings with mother caregivers (ACDI/VOCA 2017b). Formative research to develop training modules RWANU used the TIPS methodology to investigate nutrition and hygiene practices to inform the development of the training materials and flipchart for interpersonal communication. Tailoring of training topics to the life cycle The training topics were: (1) IYCF, (2) maternal health and nutrition, (3) agriculture and nutrition linkage, (4) WASH, (5) family planning, (6) child health, and (7) health user rights The messages were short, simple and practical; each behavior was promoted for a period of four to six weeks, and the messages periodically repeated if the annual survey demonstrated Best practice in MCGsa RWANU implementation of MCGs that knowledge or practice among participants was poor. However, topics were not necessarily tailored to the life cycle stage of each participant or groups of participants during the group meetings with mothers. The topics were rolled out following the RWANU predetermined module lesson sequence, which was operationally more feasible. During lead mother home visits, lead mothers were supposed to check previous behavior change topics as well as the current topic, but it was only possible for RWANU staff to monitor this during supervision visits. Training for up to 25 promoters per training that covers each lesson and coaching and practicing for promoters. Groups of up to 25 promoters were trained over 2 to 3 days for each module, with follow-up supervision and coaching in the field. A cascade training approach was used, where RWANU project nutritionists trained field coordinators, who trained health promoters, who trained lead mothers, who trained mother caregivers. The quality of the promoter cascade training was monitored monthly during supervision visits by field coordinators. Given the number of lead mothers that each health promoter supervised, the supervision visits may have been less frequent than if there were a lower ratio of health promoters to mother caregivers. a TOPS. 2016 Table 7D.2: Positive Nutrition and Health Outcomes of RWANU Project Mentioned During FGDs and KIIs Positive nutrition and health outcomes of RWANU project mentioned during FGDs and KIIs Number of FGDs or KIIs that mentioned the outcome Increased knowledge by mothers of nutrition and health 4 Improved dietary diversity from the greens from keyhole gardens or sale of vegetables from keyhole gardens to purchase other food 3 Improved child health and less illness among children 3 Increased number of mothers introducing complementary foods, such as enriched porridge with greens, beans, or small fish at age 6 months 2 Increased adoption of family planning methods 2 Increased numbers of mothers exclusively breastfeeding their child during the first 6 months of life 1 Increased use of health facilities when children were sick 1 Fewer deaths during childbirth because more pregnant women were delivering at the health center 1 Women’s workload and child nutritional status. Regarding women’s workload, if women’s time is occupied by activities related to food production or the income generating activities, there may be less time for caregiving, which contravenes the RWANU project gender strategy to not increase women’s workload. The RWANU 2016 third quarter report indicated that RWANU staff participated in a two￾day meeting in which Action Against Hunger (ACF) disseminated the preliminary findings of a nutrition causal analysis study conducted in Moroto, which found that the five major causes of undernutrition included high workload for mothers, as well as poor complementary feeding, poor sanitation and hygiene, high instability of food access, and low purchasing power (ACDI/VOCA 2016). In the proposal issues letter, RWANU indicated it would monitor women’s workload to ensure that the project was not adding additional burden to women and that any additional time required for program activities would be offset through labor-saving means, but no monitoring results of women’s workload were found in the quarterly reports or annual reports. The last RWANU report for the calendar year 2016 does state that there were 1,123 energy saving stoves in use, which reduce firewood consumption and ultimately reduce mothers’ workload, leaving more time for care of their children, but there were no other reports of technologies to relieve women’s burden of work (ACDI/VOCA 2017c). Monitoring of women’s workload and more interventions to reduce women’s workload is important to increase women’s time for providing quality care for their children, or other options for quality childcare need to be considered. Prevalence of underweight among boys and girls. Endline survey results showed that the prevalence of underweight among boys less than age 5 increased significantly from 23 percent to 34 percent, while that of girls also increased significantly from 18 percent to 25 percent (Table 7D.3). RWANU annual monitoring data showed that for 2014 and 2016 there was a relatively small difference between the proportion of boys ages 6–23 months who were underweight compared to the proportion of girls. In 2015 the proportion of boys who were underweight was 9 percentage points higher than the proportion of girls, and in 2017 the proportion of boys who were underweight was 17 percentage points higher than that of girls. Studies have shown that there is a higher prevalence of stunting and underweight among boys than girls in Sub-Saharan Africa (Wamani et al 2007; Svedberg 1988), although the reasons for this finding are unclear. Energy requirements for boys under the age of 5 are higher than that of girls of the same age by approximately 8 to 10 percent (FAO 2001). Evidence from another country and region, Guatemala, showed that indigenous mothers reported that, compared with female infants, male infants were hungrier, were not as satisfied with breastfeeding alone, and required earlier complementary feeding, which can expose male infants to pathogens and illness at a younger age than girl infants (Tumilowicz et al 2015). Similarly, two FGDs with mothers in the RWANU project area shared their perception that boys have a greater need for food and eat more than girls; however, the proportion of boys who were exclusively breastfed in the RWANU project area significantly increased from baseline to endline, so early introduction of complementary foods does not appear to be the problem. 8 Mothers in RWANU FGDs also shared that boys are more active than the girls starting at about the age of 18 months, and that at the age of 4 to 6 years, boys begin to help with the cows and get less to eat in the bush than girls do around the home, although they also shared that with fewer cows, often only one boy in a family may go to herd the cattle. FGDs with mothers indicated that they did not treat male and female infants differently. Table 7D.3: Underweight Among Children Ages 6–23 Months, Total and by Sex, by Year Age group and sex 2014 2015 2016 2017 6–23 Months, Total 22.7% 25.6% 27.3% 32.1% 6–23 Months, Boys 22.3% 29.9% 28.5% 40.7% 6–23 Months, Girls 23.0% 20.5% 26.0% 23.7% Source: ACDI/VOCA 2017a. Exclusive breastfeeding. From March to October of 2014, RWANU staff had trained lead mothers, and lead mothers in turn trained mother caregivers, on the MCG infant and young child feeding module. In November–December 2014, RWANU staff conducted a module post-test survey and found that among mother caregivers who participated in the mother caregiver groups, 94 percent of the respondents with children less than age 6 months reported to have initiated breastfeeding within an hour of giving birth, which was higher than findings from the Karamoja FSNA in May 2014 (77 percent, Concern Worldwide, 2015). The proportion of mothers reporting to have exclusively breastfed their child in the 24-hour period before the survey was 69 percent, which was similar to the RWANU annual 2014 survey (68 percent), and lower than the May 2014 food security and nutrition assessment findings (82 percent). 8 In addition, an analysis of the PBS data showed that for infants less than age six months, there was no significant difference in child nutritional status between baseline and endline, including for between boys at baseline and endline, and for girls at baseline and endline. However, it is important to note that the sample size of infants less than age 6 months was very small, and it would only be possible to detect a very large change between baseline and endline for this age group. In the third quarter of 2016, given the poor exclusive breastfeeding result, RWANU staff conducted a barrier analysis on exclusive breastfeeding to better understand why exclusive breastfeeding among children under the age of 6 months did not increase as a result of the training, and to implement actions to achieve the project target around exclusive breastfeeding. The barrier analysis revealed that those who did not exclusively breastfeed their child for the first six months of life were less confident that they could do so, did not consider that breast milk was the most adequate food for a baby, felt they did not have enough breast milk, did not have sufficient food to produce enough breast milk because of lack of purchasing power or limited household-level food availability, felt they were “on their own” with little approval from their social group for exclusive breastfeeding, lacked the time to exclusively breastfeed, and did not feel that their child was at risk of diarrhea or malnutrition because of not exclusively breastfeeding (Stallkamp 2016). During subsequent trainings with lead mothers and male change agents, the project increased its emphasis on exclusive breastfeeding, focusing on the latter topics. By the end of 2016, monitoring data showed that exclusive breastfeeding among infants less than the age of 6 months had increased to 86 percent. Complementary feeding—RWANU barrier analysis on animal source foods. At the end of 2016, RWANU conducted a barrier analysis study on the use of animal source foods by pregnant and lactating women and children under the age of 5 among households participating in MCG training. The findings showed that primary barriers to the consumption of animal source foods were: lack of financial resource to purchase animal source foods; absence of animal source foods in the communities; lack of women’s decision-making about family livestock; and insufficient knowledge on the role of animal source foods for a healthy diet. The barrier analysis report included recommended actions to improve behavior change messages on the importance and use of animal source foods and improve access to animal source foods. Lead mothers were encouraged to discuss the messages with their household care groups. However, regarding the recommendations to improve access to animal source foods, given the study was conducted late in the project, it was difficult for the project to implement them. This was also difficult given the lack of a strong linkage between SO1 and SO2. There appear to be two consistent themes regarding the capacity of the RWANU project to influence adoption of promoted IYCF practices: (1) lack of participant funds to purchase recommended foods to improve dietary diversity and meal frequency; and (2) inadequate linkages between SO1 and SO2 to positively influence adoption of improved dietary intake. Mothers’ workload also appears to be a factor contributing to poor child nutrient intake, in terms of lack of time for appropriate childcare. Sustainability of MCGs. Two of five FGDs of lead mothers indicated that they were still meeting and that what was bringing them together after the closure of the RWANU project was being part of a VSLA. In one community, the lead mothers said that they started the VSLA on their own, with no support from any project, and registered it with the county, while the mother caregivers in the same community said that the VSLA was started with support from Mercy Corps, and that Mercy Corps was providing them with seeds. In the other community, the lead mothers said the VSLA had existed before the RWANU project but was strengthened by RWANU. Three of the five FGDs with lead mothers that are no longer meeting indicated that the lead mothers stopped meeting in 2017 after the RWANU trainer stopped coming. Three of five FGDs with lead mothers said that there are projects that are now registering them or that they are working with, including the USAID-funded CRS Nuyok project, Mercy Corps (project/donor not specified), and a UNICEF-funded project similar to RWANU with Doctors with Africa. It would be useful for new development food security activities to collect comprehensive data on the MCGs that operated in their program areas when the new projects were in start-up phase to determine how many MCGs were still functioning and the reasons why they were, or were not, still in operation. The findings could be used to strengthen the sustainable implementation of the MCGs after project closure. Sub-county level registration of MCGs for sustainability. One of five FGDs with lead mothers shared that some of the mothers’ groups had registered at the sub-county and as such, were able to access the government’s women’s empowerment fund program. One FGD of lead mothers indicated that they had not been registered at the sub-county. A KII with one district health staff indicated the MCGs had not been registered with the sub-county, and interviewed staff at one of the health facilities indicated that the MCGs in the area were registered with the sub-county but had not yet received support or partner engagement because the Ministry of Health does not have funds to support the MCGs. Sub-county officials emphasized that the formal registration of groups at the sub-county level allows them to benefit from government programs. Four of nine interviewed former RWANU staff said that, as part of the exit strategy, the MCGs were registered with the district community development office in the four districts to link them to the local government structure, and some MCGs were assisted with VSLAs, which created a sense of group belonging and togetherness. The final project annual report indicates that 292 MCGs out of 345 (85 percent) and all 16 MCA groups were successfully registered at the sub-county and district levels, and that the registration formalized the groups within the local government system for sustainability, and positioned groups to take advantage of livelihood grants operated by local authorities. The study on MCG sustainability suggested above should also include a component analyzing the registration of the MCGs at sub-county level to determine how registration and follow-up with the MCGs and officials could be improved. New development food security activities should consider registering groups much sooner in the implementation process, to support MCGs and MCAs, to fully understand and take advantage of the benefits of registration, and to link the groups to sources for sustainable support. Sustainable practice of promoted nutrition and health behaviors. All five FGDs with lead mothers indicated that mothers were either no longer motivated to practice the behaviors learned during the RWANU project because they were not receiving food rations, or there were very limited practices that some mother caregivers were willing to continue, such as family planning and handwashing. However, as noted in Section 4.2.5, five of seven FGDs with lead mothers and mother caregivers appeared to still be using their keyhole gardens. One FGD with mother caregivers reported that they destroyed their keyhole gardens because someone told them that if they had the gardens they would not be selected for another project, and another group said they were “taking a break” from the keyhole gardens and that they did not have funds for seeds, although access to water was not a problem. One FGD with mother caregivers indicated that they stopped preparing enriched porridge for children because they receive CSB+ from Andre Foods International for all children ages 6–23 months, and they no longer consider the enriched porridge necessary because CSB+ has all the nutrients the child needs. Studies are needed to better understand the reasons why some nutrition and health practices appear to have been sustainably adopted and others were not adopted or were no longer practiced after RWANU project closure. Findings from such studies would inform improved strategies for social and behavior change to improve long-term adoption of promoted behaviors. Integrated Management of Acute Malnutrition Surge approach. About 80 percent of lead mothers were trained in 2013–2014 to screen and identify acute malnutrition in children ages 6–59 months using the MUAC tape, to assess for nutritional edema, and to refer suspected acute malnutrition cases to the VHTs as part of the Integrated Management of Acute Malnutrition Surge approach. The linkage with the village health workers structure of the Ministry of Health reinforced the work being done by the VHTs in nutrition surveillance and referrals. The IMAM Surge approach was implemented by Concern Worldwide in response to a May 2013 food security and nutrition assessment in Karamoja that showed a GAM rate of 20.2 percent and a SAM rate of 6.1 percent. Through the IMAM Surge response, Concern Worldwide supported government health facilities to manage increased caseloads of acute malnutrition. Sustainability of IMAM Surge approach at the community level. Only two of five FGDs with lead mothers indicated that they were still measuring children’s MUAC in the community every two weeks and had just done so in the 2 weeks prior to the FGD. In two of the three communities where the FGDs with lead mothers were no longer conducting MUAC screening, the VHT members did say that they were screening children for acute malnutrition using MUAC, and one indicated that the number of children suffering from acute malnutrition was high—20 of 85 children screened in September had SAM or MAM, and 12 of 60 children screened in early October had SAM or MAM. Two of five interviewed VHT members in two communities said that they were trying to support MCGs, but a third VHT in another community indicated that there was no formal role for VHTs with the MCGs after the RWANU project ended. As indicated above, recent research has shown that mothers with low literacy and numeracy in developing countries can successfully identify children with SAM and MAM using MUAC tapes (Blackwell et al 2015; Grant et al 2018). Given the potential for the IMAM SURGE approach to be effective, it should continue to be supported in Karamoja, in conjunction with quality improvement approaches, and its use documented and shared with the wider development community. In addition, strong links should be developed with professionals and researchers using the approach in other settings to establish a network for learning and adaptation. There is also a need to formalize support for MCGs from VHTs, to monitor the successes, challenges, and sustainability of such support, and to adjust implementation to overcome challenges that arise. New development food security activity experiences with the latter should be documented and shared. Sustainability of outcomes from goat distribution. The mother caregivers that each received 5 goats now each have from 7 to 12 goats. The mother caregivers said that they continue to benefit from the goats in terms of improved dietary diversity, frequency of eating for their children and themselves, and having options to pay for emergency needs. It was only possible to meet with one women’s group that had received goats. It would be useful for a study to be conducted among the women’s groups that received goats to determine their sustainability, and the long-term impacts on dietary diversity and feeding frequency for young children of these mother caregivers. Male Change Agents (MCAs). The technical interventions also included a pilot program on MCAs introduced in Year 2 of the RWANU project. MCAs aimed to involve men in promoting joint decision￾making and positive household-level health and nutrition behavior change. The MCA curriculum was developed for health educators (HEs) to support their training of MCAs. The contents of the curriculum were adapted from the RWANU MCG modules developed by Concern Worldwide, a training manual for male group leaders on infant and young child feeding and gender developed by PATH in collaboration with CARE, and Promundo’s Programme P Manual on “Engaging Men in Fatherhood, Caregiving and Maternal and Child Health”, developed by The Manoff Group and University Research Co., LLC. The curriculum focused on gender topics, men’s role in maternal and child health and nutrition, hygiene, sanitation, family planning, needs and rights of children, public speaking, and tips on responding to strong negative opinions from peers and friends. MCA commitments and action plans were developed to monitor progress and provide support. Demonstrative home practices included constructing bathing shelters, compound cleaning, and accompanying their spouses to antenatal check￾ups. RWANU MCA monitoring findings and implications. Key findings from a Concern Worldwide survey on the MCA approach in August 2016 found that 70 percent of MCAs participated in caring for their child under the age of 5, compared to only 17 percent of “other men” whom MCAs had been working to influence; 88 percent of MCAs accompanied their pregnant wife to antenatal care visit at the health facility in the prior month, compared to 58 percent of “other men” influenced by the MCAs; 68 percent of MCAs and 85 percent of their wives indicated that women in the household were consulted in household decision making, compared to about 56 percent of “other men” who had been influenced by MCAs and the wives of the “other men”; 44 percent of MCAs lived in a household with a functioning tippy tap, compared to 19 percent of “other men” influenced by MCAs; and 88 percent of MCAs demonstrated good hand washing behavior with soap, compared to 50 percent of “other men” (Concern Worldwide. 2016). The findings showed that MCAs were serving as positive role models, but there was still much more effort needed to positively influence the adoption of behaviors among men the MCAs were working to influence. The report recommended supporting MCAs as they engage with other men to achieve the same levels of behavior change that they themselves had achieved. Sustainability of MCAs. Two of two interviewed MCAs said that they continue their work; for example, they are invited by the village elders to give advice in family disputes, they convinced two of eight men to start digging latrines in a recent small meeting, they helped a friend dig a latrine, and they maintain contact with other MCAs and discuss how work is going. One MCA shared that under the RWANU project in his community, they got 50 percent of school-age children to attend primary school, but this has decreased to 30 percent because families lack funds because of the need to purchase more food now compared to when the project was implemented. As indicated in section 4.2.5, given the importance of this approach, MCAs should be expanded and improved, and experiences and lessons learned should be documented and shared with the wider development community. Drama Groups. In addition to the MCGs and the MCAs, the RWANU project trained drama groups to present on topics, such as nutrition, health, family planning, WASH, and HIV/AIDs. There were 16 drama groups composed of approximately 314 members. Drama groups performed at health facilities, outreach sessions, food distribution sites, and for national campaigns like World Water Day, World Food Day, World AIDS Day. The groups were trained by the RWANU staff health educators. ANNEX 7E: FOOD DISTRIBUTION Food ration package. The RWANU project design and results framework for food distribution included provision of food rations to pregnant and lactating women and children ages 6–23 months regardless of nutritional status, and an additional household ration to partially fill the household food gap during the lean season and prevent sharing of the individual rations. The RWANU food rations were given to participants located greater than five kilometers from a health facility.9 The project proposal estimates that approximately 75 percent of project participants lived farther than five kilometers from a health facility (ACDI/VOCA 2012). According to the project proposal, the ration for children ages 6–23 months was composed of 75 grams of corn soy blend (CSB) and 15 grams of vegetable oil per day; the ration for pregnant and lactating women was 50 grams of split green peas, 133 grams of CSB, and 15 grams of vegetable oil per day; and the household lean season ration was composed of 133 grams of split green peas, 31 grams of vegetable oil, and 400 grams of cornmeal per day. Table 7E.1 provides the RWANU ration type and composition by distribution year (ACDI/VOCA 2013b; ACDI/VOCA 2015). RWANU adjusted the length of time for the lean season household ration distribution each year and in specific project areas based on an analysis of the food security situation. In 2014, for example, RWANU proposed increasing the lean season ration from four months to six months, and changing the time period from January to June, rather than May to August (ACDI/VOCA 2013b), due to anticipated poor harvests in 2013. In 2016 the project increased the ration of vegetable oil for pregnant and lactating women to 45 grams per day, for children ages 6–23 months to 23 grams per day, and for the household to 158 grams per day. The rationale for the increase in vegetable oil was not explained in project documents. In 2016 the project also requested CSB+ in the call forward rather than CSB, given the nutrient content of CSB+ is better than that of CSB. RWANU staff shared that in August of 2015 they discussed with the USAID Office of Food for Peace the composition of the ration but did not provide more details regarding the discussion. Table 7E.1: RWANU Ration Type and Composition Target Group Ration Type and Composition, 2014 and 2015 Ration Type and Composition, 2016 Commodity Monthly Ration size (kg) Commodity Monthly Ration Size (kg) Pregnant and lactating women CSB 4.00 CSB+ 4.00 Veg oil 0.46 Veg oil 1.38 Lentils 1.50 Lentils 1.50 Child age 6–23 months CSB 2.25 CSB+ 2.25 Veg oil 0.46 Veg oil 0.70 Household (Lean season) Lentils 4.00 Lentils 4.00 Cornmeal 12.00 Cornmeal 12.00 Veg. oil 0.92 Veg oil 4.85 9 Women could participate in the food distribution with each subsequent pregnancy; there was no limit to participation. Table 7E.2: RWANU Food Rations Compared to USAID Preventing Malnutrition in Children Under Age 2 Approach (PM2A) Guidance Target Group RWANU 2016 Ration PM2A Pregnant and lactating women Energy/day: 1,066 Protein/day: 29 grams Energy/day: ~1291–1,491kcal (maximum gap)a Protein/day: ~40–47 grams (maximum gap)b Children ages 6–23 months Energy/day: 479 kcal Protein/day: 9.7 grams Percent of energy from fat: 26–34 Energy/day: 300–550 kcal Protein/day: 11–13 grams Percent of energy from fat: ~17–42 Household (lean season)c Energy/day: 3,304 kcal Energy/day: 3,940 kcal gap a The lower range includes ~475 kcal for the third trimester of pregnancy and the upper range includes ~675 kcal for lactation, plus the estimated kilocalorie gap for adult women in Karamoja of ~645 plus an additional 10 percent of minimum daily energy requirement of 1,710 for women given their level of activity (FANTA-2 2010; ACDI/VOCA 2012). b Calculated as 12.5 percent of the maximum kilocalorie gap (kcal*0.125)/4. c PM2A household-level energy levels are based on mean dietary energy consumption gap of approximately 32.6 percent of total kilocalorie needs for a family of average size of 6.9 (ACDI/VOCA 2012). Table 7E.3: RWANU Food Distribution: Length of Activity Target, Achievement, and Percentage of Target Achieved Indicator Length of Activity Target Length of Activity Achievement Percentage of Target Achieved Number of children ages 6–23 months of age receiving PM2A rations Male: 23,417 Female: 23,548 Male: 15,230 Female: 15,732 Male: 65% Female: 67% Number of pregnant and lactating women receiving rations 31,193 29,978 96.1% Number of individuals receiving a protection ration 204,826 88,041 43% Impact of RWANU food rations on child nutritional status. RWANU staff felt that the rations might have kept the nutrition situation from getting much worse. Table 7E.4 shows data from the food security and nutrition assessments conducted by UNICEF and WFP in June 2016, two years before the RWANU endline survey and 6 months before RWANU food distribution ended; in June 2017, one year before the RWANU endline survey and 6 months after the last RWANU food distribution; in January 2018, 6 months before the RWANU endline survey, and in July 2018, at the same time as the RWANU endline survey. Note that the food security and nutrition assessments included anthropometric data for children ages 6–59 months, so the results are not comparable to the RWANU endline survey, which included anthropometric data for children under the age of 5, but the information provides a rough comparison showing that the prevalence of stunting does appear to have been slightly higher in the project districts of Nakapiripirit, Napak, and Moroto, compared to the endline value for the project area, and that food distribution may have kept the prevalence of stunting from getting much worse. However, this is impossible to say definitively without data from appropriate comparison or control groups during the corresponding time periods. The prevalence of stunting among children under the age of 5 in Karamoja was 35 percent in the 2016 demographic and health survey (UBOS et al 2018). The prevalence of stunting in the RWANU endline survey and in the UNICEF and WFP food security and nutrition assessments remains classified as very high. Table 7E.4 also shows the prevalence of wasting among children ages 6-59 months of age during the same time periods noted above. Wasting among children under the age of 5 was 12 percent in the RWANU endline survey, and 11 percent in the baseline survey. The prevalence of wasting among children under the age of 5 years in Karamoja was 10 percent in the 2016 demographic and health survey (UBOS et al 2018). In Napak and Moroto in 2016, for example, food distribution might have assisted in preventing the prevalence of wasting among children under the age of 5 from getting worse, but as with stunting, a comparable control group in the various time periods would be necessary for an appropriate comparison. Table 7E.4: Stunting and Wasting among Children Ages 6–59 Months in Food Security and Nutrition Assessments in Karamoja, June 2016, July 2017, January 2018, and July 2018 Location Prevalence of Stunting among Children ages 6–59 months (%)a Prevalence of Wasting among Children ages 6–59 months (%)a FSNA June 2016b FSNA July 2017c FSNA January 2018d FSNA July 2018e FSNA June 2016b FSNA July 2017c FSNA January 2018d FSNA July 2018e Karamoja 28 33 34 36 11 14 10 11 Napak 40 32 36 40 14 13 9 9 Moroto 34 41 35 38 14 19 15 12 Nakapiripirit 26 30 32 40 8 12 11 11 Amudat 18 26 24 26 11 12 15 10 a Note the surveys included anthropometric data for children ages 6–59 months, so the results are not comparable to the RWANU endline survey, which included anthropometric data for children under the age of 5 (RWANU endline stunting prevalence was 36 percent). b UNICEF et al 2016. c UNICEF et al 2017. d UNICEF et al 2018a. e UNICEF et al 2018b. Brief description of the integration of health service provision and health messaging at food distribution points. In 2014, the RWANU project staff and the Ministry of Health initiated integration of health service provision and health messaging at food distribution points (ACDI/VOCA 2014 [Q2 report]). Messaging included information about family planning, screening and referrals of malnourished children, antenatal care, nutrition education, vitamin A supplementation, and vaccinations. The 2014 quarterly reports indicated that food demonstrations at food distribution points were to begin, including demonstrations on the use of food rations and local foods to improve the nutritional value of staple foods and dietary diversity (ACDI/VOCA 2014 [Q3 report]). The purpose of the integration of health services and food distribution was to improve access to services, save time for both participants and health staff, and reduce workload at the village level. Sustainable outcomes of food distribution: One of nine interviewed former RWANU staff said that the project’s food distribution component did not have an exit strategy, which created a great deal of dependence on food rations. As seen in section 4.2.5 above, findings from the RWANU project endline survey showed that maternal and child nutritional status did not improve, nor did women’s dietary diversity or children’s dietary diversity and meal frequency, despite the provision of food rations during the project. Some, but not all mother caregivers reported that they were receiving food rations from other sources at the time of the qualitative data collection. Two FGDs with mother caregivers found that they were receiving rations from the health center, through WFP and AFI, but a third FGD with non-participants indicated that they did not receive food rations from the health center, which is located six kilometers from the community. WFP does provide food rations for pregnant and lactating women when they come to antenatal care visits and for children ages 6–23 months when they come for their immunizations at health facilities, but health staff shared that the ration availability was inconsistent and only full rations are provided, not partial rations. An FGD with lead mothers indicated that because there is no food ration, mother caregivers no longer listen to them or respect them, and the mother caregivers do not wish to continue with the practices that they had learned. During one case study, a vulnerable mother indicated that her children were not as healthy as they had been when they received the food rations. A former RWANU staff member shared that despite having been told regularly that the food distribution would come to an end, the end was not well received by participants. New USAID/FFP development food security activities in Karamoja plan to use cash transfer and vouchers, instead of food rations and direct distribution, to improve access to nutrient-rich foods for vulnerable groups. Whether food rations or cash vouchers are provided, new development food security activities will need to implement strong interventions to sustainably increase incomes and access to affordable, nutrient-rich foods for women and young children, in addition to improving knowledge and intent to practice the promoted nutrition and health behaviors. ANNEX 7F: HEALTH SERVICE DELIVERY A detailed list of RWANU technical interventions, as cited by health staff during interviews, included:              Outreach to communities greater than five kilometers from a health facility, involving: immunizations GMP nutrition assessment deworming supplementation with vitamin A antenatal care HIV testing educational sessions drugs for patients Community scorecard to increase dialogue between community members and health facility staff about quality of health services Suggestion boxes at health facilities Monthly training for VHTs, including screening and referral of children for acute malnutrition using MUAC (the Concern Worldwide IMAM SURGE approach) The RWANU project also included provision of anthropometric equipment, lab supplies, equipment and supplies for maternity wards, and solar lighting systems; transport of health facility supplies when needed; support to drama groups that presented educational messages at health facilities, outreaches, and food distribution points; training on the community scorecard, GMP, and the integrated management of acute malnutrition; supervision with district staff; and monthly coordination meetings with sub-county stakeholders. Findings of the RWANU KAP survey on Health User Rights training module and community scorecard impact assessment survey. RWANU’s pre- and post- knowledge, attitude, and practice (KAP) survey on the last MCG module on Health User Rights showed a reduction in the number of health workers treating or talking to clients (mothers) disrespectfully at health centers and hospitals. In addition, the post-test showed that there was an increase in the number of caregivers with money saved for health expenses, more caregivers able to talk to other caregivers without feeling fearful, and an improvement in joint decision-making by caregivers and their spouses (ACDI/VOCA 2017b). RWANU conducted a community scorecard impact assessment survey in 12 health facilities in November 2017 and found significant improvements as a result of the CSC approach, including in staffing levels, waiting times, staff attitude, drug availability, availability of medical equipment, and connection to the electrical grid, solar power, and water supply. Health workers and health service users noted improvements in infrastructure at all health facilities including the construction of staff houses, pit-latrines, placenta pits, and maternity and general wards. Overall, the CSC was considered successful and is a best practice that should be replicated in the new development food security activities. Sustainability of health outreach activities. All five district-level health staff and seven of eight health facility staff who were interviewed said that they have not been able to continue with most outreach activities that had been supported by the RWANU project, that the community scorecard activity introduced by RWANU was no longer used, and that after the RWANU project stopped providing support, health service coverage and use has not been sustained at the same level as when the RWANU project was being implemented. One district health officer attributed the lower health service coverage to the departure of Baylor College of Medicine Children’s Foundation-Uganda and Concern Worldwide, decreased support overall from projects, and to the RWANU closure.10 In all four health facilities, staff indicated that after RWANU closed, there was an organization that provided support for outreach activities, but in all cases the support covered only one or two communities, leaving a gap of six to 10 communities that were not covered through outreach because the health facility lacked transport.11 Sustainable use of equipment and knowledge and skills from training. About half (6 of 13) of the health staff indicated that the equipment that had been received from the RWANU project was still present and functional in the health facilities, but staff at one facility indicated that there was no provision for maintenance of donated equipment, such as weighing scales. About half (6 of 13) of the health staff shared that staff who had been trained and coached still possessed the knowledge and skills that they had developed with support from RWANU. For example, two staff shared that GMP and antenatal care clinics continue to be implemented; VHTs trained to measure MUAC in children ages 6– 59 months still conduct screening at the community level with MUAC tapes provided by the RWANU project; length boards and weighing scales provided by RWANU were still being used for growth monitoring and promotion activities; and the refresher training staff received on integrated management of acute malnutrition still motivated staff to implement the protocols. 10 Baylor College of Medicine Children’s Foundation-Uganda (Baylor-Uganda) is a child health not-for-profit nongovernmental organization that supports pediatric and family HIV/AIDS prevention, care and treatment services, health professional training, and clinical research in Uganda. It is affiliated to the Baylor College of Medicine International Pediatric AIDS Initiative based at Texas Children’s Hospital in Houston, Texas, USA. Baylor-Uganda’s work in Karamoja was funded by UNICEF through September 30, 2016. 11 Examples of organizations mentioned that provided support include Doctors with Africa, which works in prevention of HIV/AIDS, treatment of malnutrition, and diagnosis of tuberculosis, funded by various donors; Voluntary Service Overseas, which receives funding in Uganda from UK Department for International Development, IrishAid, European Union, UNICEF, USAID, and several private donors; Caritas Uganda; and UNICEF. ANNEX 7G: WASH WASH activities were initially implemented as a training module within the MCG training curriculum. WASH was embedded in the training curriculum as module four (water hygiene and sanitation) and included four lessons: L1–handwashing; L2–safe disposal of feces; L3–safe transportation and storage of water; and L4–food hygiene for mothers. The WASH component was expanded in the third year of the project to promote the construction and use of latrines and to rehabilitate 24 boreholes for 64 communities in three out of the four project districts. The RWANU project mainly embraced the Community-Led Total Sanitation (CLTS) approach, but also used home improvement campaigns and the Participatory Hygiene and Sanitation Transformation approach (PHAST). CLTS is an innovative methodology for mobilizing communities to completely eliminate open defecation. Communities are facilitated to conduct their own appraisal and analysis of open defecation and take their own action to become open defecation free. The crux of CLTS triggering is based on stimulating a collective sense of disgust and shame among community members as they confront the crude facts about open defecation and its negative impacts on the entire community. Former RWANU staff explained that the principle of “name and shame” could not be applied in the context of Karamoja because communities were very defensive about their own feces. PHAST is an approach that uses a sequence of participatory steps, methods, and tools to enable communities to assess their own risks and plan, organize for, and monitor their own improvements. The design of WASH activities was influenced by various studies undertaken by Concern Worldwide staff and specialist consultants, including doer/non-doer analysis of latrine usage,12 barrier analysis of selected behaviors including handwashing, 13 trials of improved practices (TIPs),14 and a knowledge, attitude, practice, and coverage survey.15 The barrier analysis study confirmed hand washing as a critical behavior to be addressed; TIPs research demonstrated that small doable actions were appropriate in promoting hygiene behavior change, and that “dig and bury” was the most feasible and acceptable method for designated defecation. WASH stakeholder mapping found that few actors were trained in the necessary methodologies; training activities were therefore initiated with 24 health staff and community development officers as well as Water User Committees (WUC). In addition to the CLTS/PHAST components, general WASH training and awareness-raising was also undertaken by the lead mothers through the MCGs, with additional sensitization carried out by the VHTs. CLTS/PHAST facilitation at the village level was done by sub-county level health assistants, supported community development officers, both of whom had been trained by RWANU project staff in CLTS and PHAST approaches. The facilitation was reported to have been done well, but the main challenge was the limited subsequent CLTS/PHAST follow-up. RWANU staff made at least quarterly visits to each of the 64 villages targeted for the CLTS approach, mainly for monitoring purposes. RWANU staff made at least quarterly visits to each of the 64 villages targeted for the CLTS approach, mainly for monitoring purposes. At the same time, the health assistants and community development officers were also expected to undertake follow-up CLTS activities. Interviews at sub-county level 12 This was undertaken by Concern staff to understand the motivators and barriers in practicing latrine usage. The subsequent TIPs study was then conducted to get a deeper understanding of the behavior. 13 Barrier analysis is a formative research method that identifies challenges or barriers to adopting identified positive behaviors. The analysis was undertaken by a barrier analysis consultant. 14 Trials of improved practices (TIPs) is a method for testing the feasibility and acceptability of recommended practices. The TIPs process was led by a specialist consultant. 15 The knowledge, attitude, practice, and coverage survey was implemented as the baseline (pre-training) for a pre- and post￾training survey conducted by the Concern Moroto M&E team to assess the success of the WASH training curriculum which was implemented from September 2015 to January 2016 through the Mother Care Groups. suggested that some sub-county staff occasionally accompanied RWANU staff on these visits for monitoring purposes, but the RWANU project did not provide the expected out-of-office allowances to sub-county staff for these visits and, as a result, the necessary CLTS/PHAST follow-up activities did not always take place. Senior RWANU staff, such as the CoP and the M&E Director occasionally visited the more successful CLTS villages to offer motivation to community members and staff alike, and this was reported to have successfully led to the first attainment of ODF status. Successful community exchange visits were also organized in which 15 community members were taken to visit a high-performing CLTS village and encouraged to construct latrines and apply WASH practices in their own villages. While considerable efforts were made to bring in those from outside the communities, In partnership with UNICEF and the Ministry of Water and Environment, the RWANU project rehabilitated a total of 24 boreholes and trained the associated water user committees in borehole operation and maintenance. The water user committees (WUCs) had been established at the time of borehole construction. Although many had become inactive, RWANU re-established and trained the WUCs in clean water supply, good hygiene and sanitation practices, basic borehole maintenance, water user fee collection, bookkeeping, and mainstreaming HIV/AIDS and gender. Each committee developed operations and management plans and instituted the collection of water user fees. WUCs included at least three women, including one in a key position. The fieldwork found that the WUC gender composition was 60 percent male and 40 percent female, and that each household pays UGX 1,000 per month for operation and maintenance. RWANU trained six sub-county hand pump mechanics. These mechanics were later registered at the sub-county level and were supposed to be linked into the water user committees if water sources needed rehabilitation. Sustainable promotion and adoption of improved WASH practices. Continued promotion of appropriate WASH behaviors is needed for the sustainable adoption of good practices. Although RWANU established MCGs and MCAs during the time of project implementation to, in part, share key WASH messages and promote appropriate WASH behaviors, it is unclear the extent to which MCGs and MCAs are still active, given during field visits less than half of interviewed FGDs with lead mothers indicated that they were still meeting, and even when meeting, they were not active in WASH-related activities. Some lead mothers indicated that handwashing continued to be practiced even after project closure, but endline survey results suggest that at the population level, fewer households practiced appropriate handwashing compared to the baseline. The tippy taps promoted by the project for handwashing proved to be inappropriate and, therefore, unsustainable because of the cost for the jerrycan, vandalization and misuse by children, theft, and destruction by the sun, livestock, and termites. Very few participants in FGDs had latrines, and few mentioned appropriate treatment and storage of water at the household level. Both MCAs who were interviewed were still active in promoting WASH messages. A study regarding sustainability of MCGs and MCAs and factors that promote or inhibit continued group function is needed, and findings used to improve sustainability of these community groups in new development food security activities, as well as their promotion of improved WASH practices. Overall, sustained promotion and adoption of improved WASH practices was not achieved by the RWANU project. Community structures, such as VHTs, MCGs, MCAs, and drama groups need continued support to promote the uptake of WASH practices, which also requires support from community leaders. Sustainability of latrines. The RWANU project did not achieve sustained adoption of improved sanitation facilities, namely, improved latrines. Endline data demonstrate that use of improved sanitation decreased in the project area, from 4 percent at baseline to 1.2 percent at endline. The lack of sustainability of the latrines is attributed to lack of investment in quality latrines, poor or collapsing soils, and frequent flooding that washed away latrines. Project participants become demotivated when it is necessary to re-construct latrines on an ongoing basis due to latrine collapse or flooding, given the labor-intensive work and competing priorities, especially activities that directly improve access to food. Only seven of 67 project communities were declared ODF, and some of these may have reverted to open defecation. A more intensive approach is needed to improve sanitation, including frequent follow￾up visits and support from local traditional leaders. Given the harsh environment in Karamoja, innovative, locally appropriate technologies are needed for the construction of sustainable latrines. Additionally, there is a shortage of trained masons for the construction of pit latrines. The 2017 RWANU Annual Report noted that 42 youth were trained in sanitation marketing through molding sunbaked blocks to support latrine construction in project areas, though no block-built latrines were observed during field visits, and there was no knowledge about the masons among those met by the evaluation team. The conclusion from this is that an insufficient number of masons had been trained within the project area. Provided that there is demand for their services, there is a need to increase the number of technical personnel at community levels for latrine construction. Sustainability of boreholes. An increase in sustainable access to improved water sources was not achieved in the long term, primarily because the RWANU project did not create effective linkages with hand pump mechanics to attend to timely repairs at the grassroots level. Field visit interviews revealed that most of the water points were reported to have an existing functional water source committee; four KIIs and three FGDs shared that community members contributed towards the cost of maintaining existing water sources. However, the qualitative field work also revealed that it takes between 6 and 12 months for broken borehole pumps to be repaired due to the bureaucracy involved and the lack of linkage between the water user committees and the pump mechanics. The water use committee members interviewed did not appear to have information about the pump mechanic or how they should contact the mechanic, and instead reported breakdowns to the sub-county office, which then informed the district water officer. Once reported to the district level, if the district water officer had the necessary budget available for the repairs, there were bureaucratic processes involved in obtaining approval to spend the budget and procure the necessary spare parts, which were generally not readily available within Karamoja but had to be purchased from Kampala or Mbale. Despite the efforts of RWANU staff to train hand pump mechanics, there was still a shortage of trained hand pump technicians to offer immediate services in case of breakdowns. The conclusion is that not enough technicians had been trained within the project area, nor were effective linkages developed between the WUCs and the technicians. Provided that there is demand for their services, there is a need to increase the number of technical personnel at community levels for borehole maintenance, and access to appropriate and affordable quality spare parts. ANNEX 8: SOURCES OF INFORMATION ANNEX 8A: PROJECT DOCUMENTS REVIEWED Project Proposal, Annual Results Reports, and PREPS Annual Results Reports (ARRs) (2013-2017) Pipeline Resource Estimate Proposals (PREPS) for 2014, 2015, 2016 and 2017 Quarterly Reports Analyses, Studies, and Assessments A Cost of the Diet analysis in Karamoja, Uganda: Agropastoral livelihood zone (Livestock Sorghum Bulrush Millet zone). July 2013. Christine Fernandes, Trials of Improved Practices in Karamoja: Investigating Behaviours of Nutrition and Hygiene. Final Report, RWANU. August 2013. Paul Wagubi, Agriculture Market Assessment Study Findings for RWANU Project In South Karamoja Districts of Amudat, Nakapiripirit, Napak and Moroto. April 2013. Bonnie L. Kittle, Report of Barrier Analysis Study: Social and Behavior Change Strategy (SBC) and Strategic Action Plan (SAP). RWANU. January 2013 Report on Community-based Animal Health Workers’ status in Napak District. April 2013. Doer-Non Doer Analysis Report on Latrine Usage in South Karamoja (2013) Hilda Kawuki with RWANU team, Karamoja, Health Facility Functionality Assessment Report for project selected health facilities in Napak, Moroto, Nakapiripirit and Amudat districts. 2013. Concern Worldwide (2014). Application of a Community Score Card in Moroto and Napak districts: A pilot study. 2014. Ecau Peter, Honey Market Assessment Report for RWANU Project in Southern Karamoja. 2014 Bernd Mueller and Samuel Bbosa, Wage Labor for Food Security in Southern Karamoja: A labor market assessment for the RWANU Program. December 2015. Gudrun Stallkamp, 2016. Report on Barrier Analysis survey on Women’s Consumption of Animal Source Foods. Location: Selected sites in four districts in South Karamoja (RWANU project implementation area). Concern Worldwide Uganda, November2016 Gudrun Stallkamp, 2016. Report on Barrier Analysis survey on Exclusive Breastfeeding. 2016 Danny Harvey, Susan Anibaya, Jennifer Lorika, 2013. Gender Assessment Report. March 2013. Strategy Documents Social and Behavior Change Strategic Action Plan (SBC/SAP) for RWANU Strategic Objective 2 (2013) Danny Harvey, 2013. RWANU Gender Strategy and Social and Behaviour Change Interventions (RFP￾12-0001) Hayden Aaronson, May 2015. RWANU Agricultural Input Sector Strategy RWANU-Resilient Market Systems (RWANU-RMS) RWANU Exit Strategy: Strengthening market systems for sustainability. PowerPoint presentation, ?2016 IPTTs, Monitoring and Impact Assessment Reports, Baseline and Annual Surveys ICF International, Inc. Baseline Study for the Title II Development Food Assistance Programs in Uganda. Contract #: AID-OAA-M-12-00009. March 5, 2014. Annual Survey for the Resilience through Wealth, Agriculture and Nutrition Project: Final Survey Report Submitted to ACDI VOCA by Service for Generations (SFG) International. June 2014. Annual Beneficiaries Survey 2014: Outcomes Evaluation Report. October 2014. Annual Beneficiaries Survey Report (Outcome Report) for 2015 RWANU Annual Survey 2016 RWANU Annual Survey 2017 Annual Indicator Performance Tracking Tables (2013-2017) Lydia Mbevi and Jennifer Lorika, Women’s Empowerment in Agriculture Index (WEAI) Impact Assessment Report. January 2016 USAID & ACDI-VOCA, 2016. Uganda RWANU Gender and Youth Audit: An Assessment of RWANU’s Institutional Capacity to Mainstream Gender and Engage Youth. November 2016. Concern Worldwide, 2016. Mother Care Groups and Behavior Change: Lessons from South Karamoja. Concern Worldwide Learning Brief. Lepillez, Karine, 2016. Livestock Activity Gender Impact Assessment. RWANU. Post WASH Module Assessment Report: Napak, Moroto, Nakapiripirit and Amudat District, South Karamoja Region, September 2015 – February 2016 Post Male Change Agent Training Survey Report, Concern S02-SBCC Team, 16 November 2016. Market Resilience Assessment Report (conducted in the final quarter of FY 2017) Susan Chomba, 2017. Integration of climate smart agriculture within RWANU (PowerPoint presentation, 2017) Agnes Ishara, 2017. Grants Impact Assessment of WLIA & YLIA Grants Groups Success Stories Introducing improved maize varieties through demonstration plots (2014) Southern Karamoja women seize the opportunity to control their own income (2014) Malnutrition is not new to Karamoja, but RWANU offers hope for the hungry (2014) Weeding out malnutrition, one child at a time (2014) Keyhole gardens gone viral in Iriiri sub-county (2015) Lotheutheu Women Livestock Group (WLG) (2015) Maize Farming Helps Family Build a Home (2016) Training Introduces Youth to New Opportunities (2016) RWANU Promotes Sunbaked Blocks for Environmental and Social Protection (2017) Youth Groups Launch New Businesses with Grant Funding (2017) Beekeeping Boosts Incomes in Uganda (2017) ANNEX 8B: FIELDWORK ITINERARY Date Activity Location Friday Oct 5 In-briefing meeting at USAID Mission Kampala Saturday Oct 6 Travel from Kampala to Matany Sunday Oct 7 Training for Field Assistants Matany Town, Napak District Monday Oct 8 Data collection in village, health center and sub￾county offices Loitakwa Village, Lopeei Sub￾county, Napak District Tuesday Oct 9 (Independence Day) Data collection in Matany and Moroto Matany Town & Moroto Town Wednesday Oct 10 Data collection in village and district offices Naoi Village, Lopeei Sub-county, Napak District Thursday Oct 11 Data collection in village, health center and sub￾county offices Morusapir Village, Iriiri sub￾county, Napak District Friday Oct 12 Team analysis workshop and interviews in Moroto Town Matany Town & Moroto Town Saturday Oct 13 Continuation of team analysis workshop and interviews in Moroto Town Moroto Town Sunday Oct 14 Interviews in Nakapiripirit Town Nakapiripirit Town Monday Oct 15 Data collection in village, health center, sub-county and district offices Kopedur Village, Moruita sub￾county Nakapiripirit District Tuesday Oct 16 (World Food Day) Interviews at district offices and health center Nakapiripirit Town, Tokora Health Center Wednesday Oct 17 Data collection in village, health center, and sub￾county offices Natirae Village, Lolachat sub￾county, Nakapiripirit District16 Thursday Oct 18 Team analysis workshop; travel to Jinja Nakapiripirit Town Friday Oct 19 Travel to Kampala; de-briefing presentation at USAID Mission; interview with Mission staff Kampala Saturday Oct 20 Evaluation team meeting Kampala 16 Although this was part of Nakapiripirit District during the RWANU project, it has since been made into a new district. ANNEX 8C: SUMMARY OF KEY INFORMANTS Position Location Date Former RWANU staff (Concern Worldwide) remote 27-Sep Former RWANU staff (Concern Worldwide) remote 27-Sep Former RWANU staff (Concern Worldwide) Kampala 5-Oct Government technical specialist Lopeei Sub County 8-Oct Government administrative officer Lopeei Sub County 8-Oct Government administrative officer Lopeei Sub County 8-Oct Government technical specialist Lopeei Sub County 8-Oct Government technical specialist Lopeei Sub County 8-Oct VHT Loitakwa village 8-Oct Government health specialist Lopei sub-county 8-Oct Government health specialist Lopei sub-county 8-Oct Village Chief (LC1) Loitakwa Village 8-Oct Former RWANU staff (Concern Worldwide) Moroto Town 9-Oct Former RWANU staff (Concern Worldwide) Moroto Town 9-Oct Former RWANU staff (Concern Worldwide) Moroto Town 9-Oct Former RWANU staff (Concern Worldwide) Moroto Town 9-Oct Former RWANU staff (Concern Worldwide) Matany Town 9-Oct Former RWANU staff (ACDI/VOCA) Matany Town 9-Oct Government administrative officer Napak District Office 10-Oct Village chief Naoi village 10-Oct Government technical specialist Napak District Office 10-Oct Government technical specialist Napak District Office 10-Oct VHT Naoi 10-Oct Government health specialist Napak District Office 10-Oct Government health specialist Napak District Office 10-Oct Government technical specialist Iriiri Sub-County Office 11-Oct Government technical specialist Iriiri Sub-County Office 11-Oct CAHW (Private sector) Iriiri Town 11-Oct Government administrative officer Iriiri Sub-County 11-Oct Agro dealer (Private sector) Iriiri Town 11-Oct VHT Morusapir village 11-Oct Government health specialist Iriiri Sub-County 11-Oct Mother Leader Morusapir village 11-Oct Credit Officer (Private Sector) Moroto Town 12-Oct Former RWANU staff (WHH) Moroto Town 12-Oct Former RWANU staff (Concern Worldwide) Moroto Town 12-Oct Former Government health specialist Moroto Town 12-Oct Former RWANU staff (ACDI/VOCA) Moroto Town 13-Oct Honey Buyer (Private Sector) Moroto Town 13-Oct Position Location Date Former RWANU staff (ACDI/VOCA) Nakapiripirit Town 14-Oct Former RWANU staff (ACDI/VOCA) Nakapiripirit Town 14-Oct Former RWANU staff (Concern Worldwide) Nakapiripirit Town 14-Oct Lead Farmer / FTG Chairman Kopedur Village 15-Oct Government technical specialist Nakapiripirit District 15-Oct Agro-vet input dealer (Private Sector) Moruita Trading Centre 15-Oct Agro-input Dealer Nakapiripirit Town 15-Oct Government administrative officer Nakapiripirit District 15-Oct Male Change Agent Kopedur village 15-Oct VHT Kopedur village 15-Oct VHT Kopedur village 15-Oct Government administrative officer Moruita sub-county 15-Oct Government technical specialist Moruita sub-county 15-Oct Government health specialist Nakapiripirit district 15-Oct Former RWANU staff (ACDI/VOCA) Kopedur village 16-Oct Former RWANU staff (ACDI/VOCA) Nakapiripirit Town 16-Oct Government health specialist Tokora health center 16-Oct Former RWANU staff (ACDI/VOCA) Lolachat sub-county 17-Oct Village chief (LC-1) Natirae Village 17-Oct VHT Natirae Village 17-Oct MCA Natirae Village 17-Oct Government health specialist Tokora Health Center 17-Oct Government health specialist Natirae Health Center 17-Oct Government health specialist Natirae Health Center 17-Oct Government health specialist Natirae Health Center 17-Oct Former RWANU staff (ACDI/VOCA) Kampala 19-Oct Former RWANU staff (Concern Worldwide) Kampala 19-Oct Former RWANU staff (WHH) Kampala 19-Oct Former RWANU staff (WHH) Kampala 19-Oct Former RWANU staff (ACDI/VOCA) Kampala 19-Oct Former RWANU staff (ACDI/VOCA) Kampala 19-Oct USAID Mission staff Kampala 19-Oct USAID Mission staff Kampala 19-Oct USAID Mission staff Kampala 19-Oct ANNEX 8D: SUMMARY OF FGD AND IDI PARTICIPANTS IN VILLAGE SITES Focus Group Discussions Group type FGD participants Location Date M F Farmer Group 5 24 Loitakwa village 8-Oct Lead Mothers 2 Loitakwa village 8-Oct Mother Care Givers 17 Loitakwa village 8-Oct Non-participants 8 15 Loitakwa village 8-Oct Non-participants 2 16 Naoi village 10-Oct Mother care givers 17 Naoi village 10-Oct Lead Mothers 3 Naoi village 10-Oct Farmer Training Group 8 12 Naoi village 10-Oct Farmer Training Group 7 12 Morusapir village 11-Oct Non-participants 3 Morusapir village 11-Oct Block farmer group 7 8 Morusapir village 11-Oct Mother care givers 8 Morusapir village 11-Oct Farmer Training Group 3 5 Kopedur village 15-Oct Lead mothers 10 Kopedur village 15-Oct Non-participants 1 9 Kopedur village 15-Oct Mother care givers 7 Kopedur village 15-Oct VSLA & Innovation Awardees 6 Lolachat Town 17-Oct Lead Mothers 10 Natirae village 17-Oct Women’s Livestock Group 8 Natirae village 17-Oct Non-participants 2 11 Natirae village 17-Oct Farmer group 13 12 Natirae village 17-Oct Mother care givers 10 Natirae village 17-Oct TOTAL 59 222 In-Depth Interviewees Household type Location Date Poor, vulnerable (non-participant) Loitakwa village 8-Oct Average (participant) Loitakwa village 8-Oct Poor, vulnerable (non-participant) Naoi Village 10-Oct Average (participant) Naoi Village 10-Oct Average (participant) Morusapir village 11-Oct Poor, vulnerable (non-participant) Morusapir village 11-Oct Average (participant) Kopedur village 15-Oct Poor, vulnerable (participant) Kopedur village 15-Oct Average (participant) Natirae Village 17-Oct Poor, vulnerable (participant) Natirae village 17-Oct ANNEX 8E: SAMPLE HOUSEHOLD PROFILES Names have been changed to maintain anonymity. Abigail is a 36-year old widow with six children aged 2-18. She lives at the edge of Iriiri Town (Napak District) which is located at the foot of a small mountain, and is advantaged by better rains than other areas in the district. Her husband died two years ago, and her relations now control her husband’s extensive landholdings which he established 25 years ago when he settled in the area. Ana was a member of a RWANU farmer training group, and now farms about 3-6 acres which she plowed with the former block farm group’s remaining ox, which members still share. Her farm contains unusually diverse plantings: tall coffee bushes in one corner, small papaya and mango trees, and a 50x50 kitchen garden directly in the compound, with evidence of the recent harvest of sorghum and maize. She also grows green gram, and sunflower. She also owns 10 chickens and 3 ducks. She was not an MCG member but had small children and copied their keyhole garden – which was a small but useful source of additional vegetables, although like her others, she destroyed it after the project was over to reduce chances of being passed over for follow-on project participation. Abigail was a member of the first farmer group training in their village, which was then ‘graduated’ to a block farmer group – she participated in 3 years of joint work on 10 acres with group of 15-25 farmers. Ana reported that the group members learned new techniques of planting and spacing crops, and received free seeds for additional crops (sim, green gram, others) as well as tractor assistance in opening 10 acres – which were temporarily ‘donated’ by a better-off land-holder. In year 3 they received 2 oxen for ploughing and purchased their own seed. When they purchased their own seed, they had greater success in timely planting and obtained some crop. The prior two years had been unsuccessful, as free seed distribution had been late and there was considerable crop loss due to erratic rains in earlier years. Living just 10 minutes’ walk from the town of Irirri, Ana is poor, but advantaged by good access to the local market, health care, and other services. Ana also labors for other farmers nearby to earn income, and also hires laborers to help her during cultivation season. She balances her cash needs by working 2 days a week for others, either field labor or breaking rocks for a daily wage of 2000 shillings and also collects firewood for sale. She also makes brew to sell to neighbors, and uses residue for chicken feed, although children also snack on the soft, alcohol-soaked grains. She did increase her income in some years by growing more diverse crops and was better able to feed her household, and increase income to allow to pay school fees, health costs, and buy food during lean season. She is still a savings group member with the former block farm group, and she has expanded her personal production area. During the collective work period she farmed smaller amount outside of her fields – but now prefers to work independently. The block farm group still feels connected – they save together and assist each other and share the surviving ox. They also rent oxen to other community members for traction – and use money jointly to pay a CAHW and for its medications. She is grateful – “Even though my husband has died, I can manage to support my children and pay for their education.” Her eldest son (18) is in boarding secondary school in Moroto and her eldest daughter (16) was recently recruited for a tailoring training program in Kampala. One child is learning disabled and 2 others attend primary school. Anna lives in Kopedur Village (Nakapiripirit) and is 36 years old, the second of four wives; her husband is 40 years old. Each wife has her own compound which is adjacent, but separate. She was reported as a ‘typical’ household but seemed to be at the upper end of ‘middle’ in the community. Anna was hard￾working and participated both in a farmer group, VSL and was also a lead mother. In a fresh pleated skirt and clean top, new plastic sandals she appeared to be healthy, well fed, and strong. She proudly noted that none of her 7 children were ever malnourished. Her two youngest children played nearby, a thin 6- year-old girl, and a 1.5-year-old boy with cold, but otherwise healthy looking. Her other children were away at boarding school: 19 yrs – g – P9; 16 yrs – g – P6; 14 yrs – g – P5; 12 yrs – b – P5; 10 yrs – g – P4. Anna’s compound was larger than most – about ¾ acre, with 3 sleeping huts (1 for her and young children, 1 for boys, 1 for girls); mud/wattle with thatch roof; no electricity. (The adjoining compound of the first wife, had a solar panel and satellite dish – but fewer buildings). Anna’s compound included a cooking hut, a large elevated traditional granary for maize, a keyhole garden with bean seedlings (covered with an old mosquito net against insects), a latrine and washing station, and was surrounded by a traditional stick fence. It also included a long L-shape building with 6 rooms, mud covered with concrete, tin roof. Anna rents 6 rooms to others, along with 2 additional huts – from which she collects rent, and with discussion/agreement with husband pays boarding school fees for all children in school. Her farming equipment included a hoe, slasher, and machete. Her household used 6 saucepans, 10 jerry cans, 3 basins, 3 buckets, cell phone, mattress/blanket for each child, 3 mosquito nets, Anna had 15 goats and 10 ducks, but the goats were stolen in June and the ducks stolen in Sept – so she has no remaining animals, and her husband has no cattle. Thieves from outlying villages are also known to steal food. Anna noted, “They come in the night. They think those of us near the town are rich.” Anna farms 6 acres with maize and beans, while the other wives farm 1-2 acres. In the kitchen garden she grows cowpeas, local greens, pumpkins, maize. The small keyhole garden included small seedlings for greens. Her income from rooms rental gives her: 3 at 20k per month; 3 at 30k per month, 2 huts at 10k per month = 150k per month towards boarding school fees for 5 children. (Ugandan shillings) She also hires labor for farm work at 3k per day – over full season costs 1.5 million for clearing and weeding; 500k for harvesting – total 2 million for maize harvest. This year grain store 80% full – last year about 75% full. Last year they had problems with army worm although the weather was good in their area – they tried ash, detergent, nothing worked. This year there was no problem with pests, but growing conditions were less favorable. Anna also labors herself for others 5 x per month in wet season and 3 x per month in dry season. Her savings group has given her capital for her brewing – she buys 2 bags sorghum for 60k each, prepares brew and is able to sell for 200k, giving 80k profit – after returning the loan. She felt the most successful RWANU activities were the following:     Hygiene and sanitation – most households have pit latrines Immunization emphasis has been successful Women encouraged to go for antenatal visits – successful She has learned about importance of diet for child nutrition – for example, she used to cook liver for her husband but did not give it to children – now if she cooks for husband also gives some to young children – e.g. 2x per month. (chicken/goat/beef). ANNEX 8F: PBS INDICATOR RESULTS TABLES FOR RWANU Table 8F.1: FFP Uganda Endline Indicators - RWANU Project Area Indicators, 95% Confidence Intervals and Base Population [Uganda, 2018] Indicator Value Confidence Interval Number of Records Weighted Population Standard Deviation Standard Lower Upper Error DEFT FOOD SECURITY INDICATORS Average Household Dietary Diversity Score (HDDS) 2.9 2.6 3.2 991 46,568 2.0 0.15 2.4 Prevalence of moderate and severe food insecurity in the population, based on the Food Insecurity Experience Scale (FIES) [30 day recall] 91.0 88.3 93.6 1,209 57,014 24.2 1.34 2.0 Male and female adults 90.9 88.1 93.7 978 45,234 24.4 1.41 1.8 Adult female, no adult male 91.6 86.8 96.3 196 9,752 22.5 2.43 1.5 Adult male, no adult female 88.5 79.1 97.9 34 1,951 28.6 4.79 1.0 Child, no adults NA NA NA 1 76 NA NA NA POVERTY INDICATORS Per capita expenditures (as a proxy for income) of USG-assisted areas $.69 $.55 $.82 6,486 305,097 $1.09 0.07 2.1 Male and female adults $.69 $.56 $.82 5,590 258,537 $1.04 0.06 1.9 Adult female, no adult male $.69 $.49 $.90 821 41,495 $1.30 0.10 1.1 Adult male, no adult female $.67 $.20 $1.13 74 4,987 $1.29 0.23 1.1 Child, no adults NA NA NA 1 77 NA NA NA Prevalence of poverty: Percent of people living on less than $1.25/day 92.9 89.8 95.9 6,486 305,097 25.7 1.51 2.1 Male and female adults 92.9 89.9 95.9 5,590 258,537 25.1 1.47 1.8 Adult female, no adult male 93.0 87.5 98.5 821 41,495 27.9 2.72 1.4 Adult male, no adult female 90.3 79.0 101.6 74 4,987 39.1 5.57 0.8 Child, no adults NA NA NA 1 77 NA NA NA Mean depth of poverty 60.9 57.3 64.5 6,486 305,097 25.9 1.79 2.4 Male and female adults 60.9 57.4 64.4 5,590 258,537 25.3 1.73 2.2 Adult female, no adult male 60.2 54.6 65.7 821 41,495 27.9 2.76 1.4 Adult male, no adult female 65.4 45.2 85.6 74 4,987 37.1 9.96 1.6 Child, no adults NA NA NA 1 77 NA NA NA WASH INDICATORS Percentage of households using an improved source of drinking water 36.9 28.8 45.0 1,228 57,951 48.3 4.00 2.9 Percentage of households using improved sanitation facilities 1.2 0.6 1.9 1,228 57,951 11.1 0.31 1.0 Percentage of households with soap and water at a handwashing station commonly used by family members 1.2 0.2 2.1 1,228 57,951 10.7 0.47 1.5 AGRICULTURAL INDICATORS Percentage of farmers who used financial services in the past 12 months 9.0 5.8 12.1 1,667 82,640 28.6 1.53 2.2 Male 11.0 7.4 14.5 773 37,824 31.5 1.75 1.6 Female 7.2 4.0 10.5 894 44,816 25.8 1.60 1.9 Indicator Value Confidence Interval Number of Records Weighted Population Standard Deviation Standard Lower Upper Error DEFT Percentage of farmers who practiced value chain activities promoted by the project in the past 12 months 33.6 28.3 38.8 1,667 82,640 47.2 2.58 2.2 Male 38.8 32.0 45.6 773 37,824 49.0 3.37 1.9 Female 29.2 24.8 33.6 894 44,816 45.2 2.17 1.4 Percentage of farmers who used at least three sustainable agricultural practices in the past 12 months 35.0 29.1 40.9 1,667 82,640 47.7 2.91 2.5 Male 41.2 34.2 48.2 773 37,824 49.5 3.45 1.9 Female 29.8 23.3 36.2 894 44,816 45.5 3.16 2.1 Percentage of farmers who used at least two sustainable agricultural (crop) practices (past 12 months) 28.8 22.9 34.6 1,667 82,640 44.8 2.88 2.6 Percentage of farmers who used at least two sustainable agricultural (livestock) practices (past 12 months) 14.4 10.7 18.0 1,667 82,640 40.7 1.80 2.1 Percentage of farmers who used at least one sustainable agricultural (NRM) practice (past 12 months) 10.0 5.1 14.9 1,667 82,640 30.0 2.42 3.3 Percentage of farmers who used improved storage practices in the past 12 months 47.1 40.5 53.6 1,667 82,640 49.9 3.24 2.7 Male 44.5 37.7 51.4 773 37,824 50.0 3.38 1.9 Female 49.2 41.9 56.5 894 44,816 49.8 3.60 2.2 WOMEN'S HEALTH AND NUTRITION INDICATORS Prevalence of underweight women 45.5 41.0 49.9 775 38,613 49.8 2.20 1.2 Women’s Dietary Diversity Score (WDDS) 2.7 2.6 2.9 1,005 56,454 1.3 0.07 1.6 CHILDREN'S HEALTH AND NUTRITION INDICATORS Prevalence of underweight children under 5 years of age (Total) 29.3 24.2 34.3 1,185 57,544 45.5 2.50 1.9 Male 34.2 27.6 40.9 573 27,139 48.3 3.30 1.6 Female 24.8 19.1 30.5 612 30,405 42.8 2.82 1.6 Prevalence of stunted children under 5 years of age (Total) 36.1 32.5 39.6 1,177 57,202 48.0 1.76 1.3 Male 39.4 34.4 44.4 565 26,769 49.8 2.46 1.2 Female 33.1 29.1 37.1 612 30,432 46.6 1.96 1.0 Prevalence of wasted children under 5 years of age (Total) 12.1 9.6 14.5 1,178 57,177 32.6 1.21 1.3 Male 12.8 9.6 16.0 567 26,821 34.0 1.59 1.1 Female 11.5 8.4 14.5 611 30,356 31.5 1.52 1.2 Percentage of children under age 5 with diarrhea in the last two weeks (Total) 27.2 23.0 31.4 1,262 61,991 44.5 2.06 1.6 Male 25.8 20.4 31.2 613 29,164 44.5 2.66 1.5 Female 28.5 24.0 33.0 649 32,828 44.5 2.23 1.3 Percentage of children under age 5 with diarrhea treated with ORT (Total) 84.0 79.7 88.3 328 16,875 36.7 2.12 1.0 Male 82.6 76.9 88.3 151 7,521 38.1 2.81 0.9 Female 85.1 79.0 91.3 177 9,354 34.2 3.04 1.2 Prevalence of exclusive breast-feeding of children under six months of age 75.6 63.5 87.6 142 6,868 43.1 5.93 1.6 Male 86.2 75.4 97.1 60 2,799 35.8 5.35 1.2 Female 68.2 51.6 84.8 82 4,070 45.7 8.19 1.6 Indicator Value Confidence Interval Number of Records Weighted Population Standard Deviation Standard Lower Upper Error DEFT Prevalence of children 6-23 months of age receiving a minimum acceptable diet (MAD) 7.7 2.8 12.5 367 17,362 26.7 2.40 1.7 Male 7.1 1.7 12.5 188 8,991 26.3 2.68 1.4 Female 8.3 2.0 14.6 179 8,371 27.8 3.11 1.5 CUSTOM INDICATORS Percentage of respondents reporting increased movement in areas that were previously not accessible due to insecurity 82.0 76.1 87.9 1,225 57,783 38.4 2.90 2.6 Percentage of households with access to a sanitation facility – not necessarily improved 18.7 11.8 25.6 1,226 57,817 39.0 3.38 3.0 Average number of crops produced per farmer in the past 12 months 2.0 1.9 2.1 1,662 82,390 1.1 0.05 2.0 Percentage of farmers adopting farmer managed natural regeneration practices in the past 12 months 30.6 28.3 33.0 1,667 82,640 46.1 1.17 1.0 Percentage of livestock owners accessing government or private sector vet care in the past 12 months 22.6 17.7 27.5 526 23,081 41.8 2.42 1.3 Percentage of mothers of children (0-23 months) attending 4 or more ANC visits with youngest child 83.9 79.5 88.3 466 25,580 36.8 2.17 1.3 WEAI COMPONENTS Female- Access to and decision on credit 17.2 13.1 21.3 1,113 51,975 37.8 2.04 1.8 Male- Access to and decision on credit 17.5 13.7 21.3 764 35,232 38.0 1.87 1.4 Female- Control over and use of income 79.2 75.1 83.3 1,113 51,975 40.6 2.01 1.7 Male- Control over and use of income 83.0 79.0 86.9 764 35,232 37.6 1.94 1.4 Female- Group membership 25.9 20.9 31.0 932 44,245 43.9 2.50 1.7 Male- Group members 30.4 24.7 36.1 637 29,328 46.0 2.81 1.5 NA : Not available Table 8F.2: Comparison of Baseline and Endline Indicators - ACDI/VOCA RWANU Program [Uganda 2013, 2018] 2013 Baseline 2018 Endline Raw Difference (Endline - Baseline) Significance Level1 Number of Observations Baseline Endline FOOD SECURITY INDICATORS Average Household Dietary Diversity Score (HDDS) 2.7 2.9 0.2 ns 2,055 991 POVERTY INDICATORS Per capita expenditures (as a proxy for income) of USG-assisted areas $0.52 $0.69 $0.24 * 14,532 6,486 Male and female adults $0.51 $0.69 $0.24 * 13,359 5,590 Adult female, no adult male $0.61 $0.69 $0.24 ns 1,001 821 Adult male, no adult female $0.76 $0.67 $0.24 ns 172 74 Child, no adults NA NA NA NA 0 1 Prevalence of poverty: Percent of people living on less than $1.25/day2 95.6 92.9 -2.7 ns 14,532 6,486 Male and female adults 95.8 92.9 -2.9 † 13,359 5,590 Adult female, no adult male 96.2 93.0 -3.2 ns 1,001 821 Adult male, no adult female 78.9 90.3 11.4 ns 172 74 Child, no adults NA NA NA NA 0 1 Mean depth of poverty (expressed as percent of poverty line) 66.7 60.9 -5.8 ** 14,532 6,486 Male and female adults 67.4 60.9 -6.5 ** 13,359 5,590 Adult female, no adult male 60.5 60.2 -0.3 ns 1,001 821 Adult male, no adult female 51.5 65.4 13.9 ns 172 74 Child, no adults NA NA NA NA 0 1 WASH INDICATORS Percentage of households using an improved source of drinking water 41.9 36.9 -5.0 ns 2,367 1,228 Percentage of households using improved sanitation facilities 4.0 1.2 -2.8 *** 2,367 1,228 Percentage of households with soap and water at a handwashing station commonly used by family members 4.0 1.2 -2.8 ** 2,155 1,228 AGRICULTURAL INDICATORS Percentage of farmers who used financial services in the past 12 months 27.1 9.0 -18.1 *** 3,063 1,667 Male 26.3 11.0 -15.3 *** 1,414 773 Female 27.7 7.2 -20.5 *** 1,649 894 Percentage of farmers who used at least three sustainable agricultural practices in the past 12 months 19.0 35.0 16.0 *** 3,080 1,667 Male 22.9 41.2 18.3 *** 1,418 773 Female 15.5 29.8 14.3 *** 1,662 894 Percentage of farmers who used at least two sustainable agricultural (crop) practices (past 12 months) 13.2 28.8 15.6 *** 3,080 1,667 Percentage of farmers who used at least two sustainable agricultural (livestock) practices (past 12 months) 14.9 14.4 -0.5 ns 3,080 1,667 2013 Baseline 2018 Endline Raw Difference (Endline - Baseline) Significance Level1 Number of Observations Baseline Endline Percentage of farmers who used at least one sustainable agricultural (NRM) practice (past 12 months) 15.8 10.0 -5.8 † 3,080 1,667 Percentage of farmers who used improved storage practices in the past 12 months 52.2 47.1 -5.1 ns 2,915 1,667 Male 50.2 44.5 -5.7 ns 1,343 773 Female 54.1 49.2 -4.9 ns 1,572 894 WOMEN'S HEALTH AND NUTRITION INDICATORS Prevalence of underweight women 26.8 45.5 18.7 *** 1,778 775 Women’s Dietary Diversity Score (WDDS) 2.6 2.7 0.1 ns 2,184 1,005 CHILDREN'S HEALTH AND NUTRITION INDICATORS Prevalence of underweight children under 5 years of age (Total) 20.5 29.3 8.8 ** 2,588 1,185 Male 23.1 34.2 11.1 ** 1,260 573 Female 17.9 24.8 6.9 * 1,328 612 Prevalence of stunted children under 5 years of age (Total) 38.0 36.1 -1.9 ns 2,588 1,177 Male 42.6 39.4 -3.2 ns 1,260 565 Female 33.7 33.1 -0.6 ns 1,328 612 Prevalence of wasted children under 5 years of age (Total) 11.2 12.1 0.9 ns 2,588 1,178 Male 12.9 12.8 -0.1 ns 1,260 567 Female 9.6 11.5 1.9 ns 1,328 611 Percentage of children under age 5 with diarrhea in the last two weeks (Total) 22.9 27.2 4.3 ns 2,756 1,262 Male 22.9 25.8 2.9 ns 1,324 613 Female 23.0 28.5 5.5 * 1,432 649 Percentage of children under age 5 with diarrhea treated with ORT (Total) 82.6 84.0 1.4 ns 587 328 Male 84.3 82.6 -1.7 ns 303 151 Female 80.9 85.1 4.2 ns 284 177 Prevalence of exclusive breast-feeding of children under six months of age 58.5 75.6 17.1 * 276 142 Male 56.0 86.2 30.2 *** 140 60 Female 61.4 68.2 6.8 ns 136 82 Prevalence of children 6-23 months of age receiving a minimum acceptable diet (MAD) 7.0 7.7 0.7 ns 818 367 Male 6.9 7.1 0.2 ns 398 188 Female 7.1 8.3 1.2 ns 420 179 CUSTOM INDICATORS Percentage of respondents reporting increased movement in areas that were previously not accessible due to insecurity 71.4 82.0 10.6 ** 2,367 1,225 Percentage of households with access to a sanitation facility – not necessarily improved 11.3 18.7 7.4 † 2,367 1,226 Average number of crops produced per farmer in the past 12 months 2.5 2.0 -0.5 *** 3,075 1,662 2013 Baseline 2018 Endline Raw Difference (Endline - Baseline) Significance Level1 Number of Observations Baseline Endline Percentage of farmers adopting farmer managed natural regeneration practices in the past 12 months 15.9 30.6 14.7 *** 3,061 1,667 Percentage of mothers of children (0-23 months) attending 4 or more ANC visits with youngest child 75.2 83.9 8.7 ** 862 466 WEAI COMPONENTS Female- Access to and decision on credit 17.0 17.2 0.2 ns 2,221 1,113 Male- Access to and decision on credit 18.2 17.5 -0.7 ns 1,466 764 Female- Control over and use of income 58.5 79.2 20.7 *** 2,258 1,113 Male- Control over and use of income 54.2 83.0 28.8 *** 1,878 764 Female- Group membership 63.8 25.9 -37.9 *** 1,545 932 Male- Group members 64.9 30.4 -34.5 *** 1,083 637 1 ns = not significant, † p<0.1,* p<0.05, ** p<0.01, *** p<0.001 2 Expressed in constant 2010 USD 3 The baseline indicator result was recalculated to exclude the use of “increased fluids” which is considered a treatment for diarrhea but not an ORT treatment. NA : Not available Note: Comparisons between baseline and endline for value chain activties and farmers accessing vet care services are not provided due to methodological differences in the data collected at baseline and endline. Table 8F.3: FFP Uganda Endline Indicators - Comparison Across Project Areas [Uganda, 2018] ENDLINE INDICATOR VALUES ALL RWANU GHG FOOD SECURITY INDICATORS Average Household Dietary Diversity Score (HDDS) 3.3 2.9 3.6 Prevalence of moderate and severe food insecurity in the population, based on the Food Insecurity Experience Scale (FIES) [30 day recall] 90.1 91.0 89.1 Male and female adults 90.1 90.9 89.1 Adult female, no adult male 91.4 91.6 91.3 Adult male, no adult female 83.9 88.5 80.2 Child, no adults NA NA NA POVERTY INDICATORS Per capita expenditures (as a proxy for income) of USG-assisted areas $0.80 $0.69 $0.92 Male and female adults $0.80 $0.69 $0.91 Adult female, no adult male $0.77 $0.69 $0.86 Adult male, no adult female $1.30 $0.67 $1.80 Child, no adults NA NA NA Prevalence of poverty: Percent of people living on less than $1.25/day 90.0 92.9 87.0 Male and female adults 90.2 92.9 87.3 Adult female, no adult male 90.9 93.0 88.5 Adult male, no adult female 76.4 90.3 65.2 Child, no adults NA NA NA Mean depth of poverty 57.2 60.9 53.3 Male and female adults 57.1 60.9 53.2 Adult female, no adult male 58.0 60.2 55.5 Adult male, no adult female 51.9 65.4 41.1 Child, no adults NA NA NA WASH INDICATORS Percentage of households using an improved source of drinking water 37.4 36.9 37.9 Percentage of households using improved sanitation facilities 7.1 1.2 13.7 Percentage of households with soap and water at a handwashing station commonly used by family members 1.9 1.2 2.6 AGRICULTURAL INDICATORS Percentage of farmers who used financial services in the past 12 months 19.0 9.0 31.1 Male 19.1 11.0 29.4 Female 18.9 7.2 32.4 Percentage of farmers who practiced value chain activities promoted by the project in the past 12 months 27.9 33.6 21.0 Male 32.6 38.8 24.8 Female 24.1 29.2 18.1 Percentage of farmers who used three sustainable agricultural practices in the past 12 months 36.0 35.0 37.2 Male 42.7 41.2 44.6 Female 30.5 29.8 31.4 Percentage of farmers who used at least two sustainable agricultural (crop) practices (past 12 months) 31.5 28.8 34.8 Percentage of farmers who used at least two sustainable agricultural (livestock) practices (past 12 months) 15.9 14.4 17.8 Percentage of farmers who used at least one sustainable agricultural (NRM) practice (past 12 months) 10.2 10.0 10.4 ENDLINE INDICATOR VALUES ALL RWANU GHG Percentage of farmers who used improved storage practices in the past 12 months 48.9 47.1 51.0 Male 47.3 44.5 50.9 Female 50.1 49.2 51.1 WOMEN'S HEALTH AND NUTRITION INDICATORS Prevalence of underweight women 35.0 45.5 24.8 Women’s Dietary Diversity Score (WDDS) 2.9 2.7 3.1 CHILDREN'S HEALTH AND NUTRITION INDICATORS Prevalence of underweight children under 5 years of age (Total) 28.6 29.3 27.7 Male 32.8 34.2 31.2 Female 24.5 24.8 24.2 Prevalence of stunted children under 5 years of age (Total) 36.7 36.1 37.6 Male 39.9 39.4 40.5 Female 33.8 33.1 34.6 Prevalence of wasted children under 5 years of age (Total) 11.0 12.1 9.7 Male 12.5 12.8 12.2 Female 9.6 11.5 7.3 Percentage of children under age 5 with diarrhea in the last two weeks (Total) 30.9 27.2 35.4 Male 29.3 25.8 33.4 Female 32.4 28.5 37.4 Percentage of children under age 5 with diarrhea treated with ORT (Total) 82.9 84.0 81.8 Male 81.0 82.6 79.6 Female 84.4 85.1 83.7 Prevalence of exclusive breast-feeding of children under six months of age 73.9 75.6 71.7 Male 78.7 86.2 71.0 Female 69.8 68.2 72.5 Prevalence of children 6-23 months of age receiving a minimum acceptable diet (MAD) 8.4 7.7 9.4 Male 8.0 7.1 9.2 Female 8.9 8.3 9.7 CUSTOM INDICATORS Percentage of respondents reporting increased movement in areas that were previously not accessible due to insecurity 84.7 82.0 87.8 Percentage of households with access to a sanitation facility – not necessarily improved 32.9 18.7 49.0 Average number of crops produced per farmer in the past 12 months 2.2 2.0 2.5 Percentage of farmers adopting farmer managed natural regeneration practices in the past 12 months 28.9 30.6 26.8 Percentage of livestock owners accessing government or private sector vet care in the past 12 months 19.0 22.6 15.7 Percentage of mothers of children (0-23 months) attending 4 or more ANC visits with youngest child 80.9 83.9 77.1 WEAI COMPONENTS Female- Access to and decision on credit 23.1 17.2 30.0 Male- Access to and decision on credit 21.7 17.5 27.2 Female- Control over and use of income 77.6 79.2 75.6 Male- Control over and use of income 79.7 83.0 75.5 Female- Group membership 35.5 25.9 46.0 Male- Group members 38.1 30.4 47.2 NA : Not available Table 8F.4: Population and Household Characteristics in the RWANU Project Area [Uganda, 2013, 2018] 2013 Baseline 2018 Endline Total population 424,056 305,097 Male 204,455 140,220 Female 219,601 164,877 Total households 67,762 57,951 Male and Female Adults 60,990 45,890 Adult Female no Adult Male 5,681 10,009 Adult Male no Adult Female 1,091 1,975 Child No Adults 0 77 Women of reproductive age (15-49 years) 84,632 56,454 Children ages 0-59 months 79,687 61,991 Males ages 0-59 months 38,153 29,164 Females ages 0-59 months 41,534 32,828 Children ages 6-23 months 25,436 17,362 Males ages 6-23 months 12,227 8,991 Females ages 6-23 months 13,209 8,371 Children ages 0-5 months 8,908 6,868 Males ages 0-5 months 4,712 2,799 Females ages 0-5 months 4,196 4,070 Source: FFP Baseline (2013) and Endline (2018) PBS in Uganda ANNEX 8G: USE OF SUSTAINABLE AGRICULTURAL PRACTICES This indicator on farmers’ use of sustainable agricultural practices captures farmers who use at least three of the following practices (regardless of whether they are crop, livestock or NRM practices): Crop practices (for sorghum, red sorghum, white sorghum, maize, beans, cow peas, pigeon peas, green grams and ground nuts):                Soil preparation with ox-plow Planting seeds in rows Crop rotation Apply fertilizer Intercropping Livestock practices (for goats and cattle): Animal shelters Vaccination Deworming Homemade animal feeds made of locally available products Use the services of community animal health workers Purchased drugs/medicines to give to animals NRM practices: Agro-forestry or cultivation of fruit trees Management of natural regeneration Soil conservation on hillsides Construction of water catchments Table 8G.1: Percentage of Farmers by Type of Agricultural Practice Agricultural Practice Baseline Endline Crops Soil preparation by hand 84.7 75.9 Soil preparation with ox plow 23.1 41.1 Soil preparation with tractor 5.8 3.2 Broadcasting seed 64.0 69.2 Planting seeds in rows 15.2 20.5 Crop rotation 4.0 6.4 Fertilizer application 0.7 0.3 Intercropping 17.5 30.4 Pest and disease control N/A 2.1 Weed control N/A 67.7 Mulching N/A 8.1 Thinning N/A 13.2 Contouring land with berms and swales N/A 1.0 Other 0.9 3.7 Agricultural Practice Baseline Endline Livestock Animal shelters 14.4 14.2 Kraals 13.6 16.2 Vaccinations 12.9 13.0 Deworming 12.4 11.6 Homemade animal feeds made of locally available products 2.4 1.7 Used the services of community animal health workers 3.2 2.0 Purchased drugs/medicines to give to animals 4.6 7.0 Rotational grazing N/A 1.9 Dehorning N/A 1.2 Castration N/A 3.7 Natural resource management Management of watersheds or reforestation 8.9 1.3 Agro-forestry or cultivation of fruit trees 6.9 1.6 Management of forest plantation 6.7 1.2 Management of natural regeneration 7.7 6.2 Collecting products from forest plants 3.8 0.1 Soil conservation on hillsides 3.2 1.6 Construction of water catchments 2.4 3.2 Number of farmers that raised crops or livestock 3,080 1,667 N/A = Not available Source: BL and EL PBS ANNEX 8H: MULTIPLE REGRESSION RESULTS, STUNTING, WASTING, AND WOMEN’S UNDERWEIGHT Table 8H.1: Results of Logistic Regression of the Prevalence of Moderate-to-Severe Stunting, ACDI RWANU [Uganda 2013, 2018] Variables Odds Ratio SE Year (2018) (2013= reference year)) 0.99 (0.25) Project participation variables HHs regularly participated in RWANU activities 1.12 (0.38) HHs received food ration 0.53 (0.20) HHs regularly received nutrition training 0.98 (0.31) HHs regularly participated in agriculture related training 1.39 (0.65) HHs participated in any other activities 0.57* (0.14) HHs received food ration, agriculture and nutrition training 1.72 (0.81) Child age (<6 month= reference age) 6 to 11 months 1.50 (0.42) 12 to 23 months 3.12*** (0.93) 24 to 35 months 3.79*** (0.94) 36 to 47 months 3.47*** (0.92) 48 to 59 months 2.36** (0.68) Child's gender (ref: male) 0.70** (0.07) Consumption expenditure quintiles (Poorest quintile= reference group) Quintile 2 1.16 (0.24) Quintile 3 1.11 (0.28) Quintile 4 1.28 (0.36) Quintile 5 (Richest) 1.68 (0.45) Household size 1.03 (0.03) Gendered household type (both male and female adults= reference group) Adult female only households 1.00 (0.22) Adult male only households 0.36 (0.34) Household education (Ref: HHs with no educated HH members) At least one member with primary level education 0.99 (0.21) At least one member with >=secondary level educ. 0.50* (0.14) WASH Households with soap and water at handwashing station 1.43 (0.52) Households using improved water sources 1.12 (0.17) Agriculture variables Households using financial services (savings, agricultural credit or agricultural insurance) 1.10 (0.23) Households using value chain practices promoted by the project 0.76 (0.11) Households using at least three sustainable agriculture practices 0.71* (0.10) Households using improved storage practices 0.99 (0.14) Household consuming >=4 food groups 0.86 (0.13) Geographic districts (Napak= reference district) Moroto 2.50*** (0.59) Nakapiripirit 1.04 (0.18) Amudat 0.38*** (0.08) Percent of children 0-59 months (1,825 days) of age who had diarrhea in the prior two weeks 1.59** (0.22) Variables Odds Ratio SE CHN Variables for children 6-23 months Dietary diversity of 7 food groups 1.11 (0.10) Frequency of feeding of solid, semi-solid and soft foods yesterday 0.93 (0.07) Constant 0.26** (0.13) Observations (<5years old children in BL and EL dataset) 1,740 F-statistic 4.63 Prob>F 0.00 SE = Standard error, HH = household *** p<0.001, ** p<0.01, * p<0.05 Table 8H.2: Results of Logistic Regression of the Prevalence of Moderate-to-Severe Wasting, ACDI RWANU [Uganda 2013, 2018] Variables Odds Ratio SE Year (2018) (2013= reference year)) 1.40 (0.35) Project participation variables HHs regularly participated in RWANU activities 0.56 (0.25) HHs received food ration 1.59 (0.82) HHs regularly received nutrition training 1.22 (0.62) HHs regularly participated in agriculture related training 1.49 (0.78) HHs participated in any other activities 1.20 (0.55) HHs received food ration, agriculture and nutrition training 0.28* (0.16) Child age (<6 month= reference age) 6 to 11 months 1.77 (0.56) 12 to 23 months 1.77 (0.61) 24 to 35 months 1.12 (0.41) 36 to 47 months 0.66 (0.24) 48 to 59 months 1.29 (0.48) Sex (ref: male) 0.93 (0.16) Consumption expenditure quintiles (Poorest quintile= reference group) Quintile 2 1.05 (0.24) Quintile 3 1.08 (0.27) Quintile 4 1.03 (0.24) Quintile 5 (Richest) 0.86 (0.27) Household size 1.08 (0.04) Gendered household type (both male and female adults= reference group) Adult female only households 1.10 (0.41) Adult male only households Household education (Ref: HHs with no educated HH members) At least one member with primary level education 1.30 (0.33) At least one member with >=secondary level educ. 0.28 (0.19) WASH Households with soap and water at handwashing station 0.54 (0.33) Households using improved water sources 0.98 (0.15) Agriculture variables Households using financial services (savings, agricultural credit or agricultural insurance) 0.87 (0.25) Households using value chain practices promoted by the project 0.99 (0.19) Variables Odds Ratio SE Households using at least three sustainable agriculture practices 0.81 (0.16) Households using improved storage practices 1.04 (0.21) Household consuming >=4 food groups 0.98 (0.24) Geographic districts (Napak= reference district) Moroto 1.28 (0.53) Nakapiripirit 0.87 (0.19) Amudat 0.64 (0.23) Percent of children 0-59 months (1,825 days) of age who had diarrhea in the prior two weeks 1.06 (0.25) CHN Variables for children 6-23 months Dietary diversity of 7 food groups 1.16 (0.11) Frequency of feeding of solid, semi-solid and soft foods yesterday 0.93 (0.07) Constant 0.07*** (0.04) Observations (<5years old children in BL and EL datasets) 1,732 F-statistic 2.50 Prob>F 0.00 SE = Standard error, HH = household *** p<0.001, ** p<0.01, * p<0.05 Table 8H.3: Results of Logistic Regression of the Prevalence of Women's Underweight (15-49 years), ACDI RWANU [Uganda 2013, 2018] Variables Model 6 Odds Ratio SE Year (2018) (2013= reference year) 2.10*** (0.35) Project participation variables HHs regularly participated in RWANU activities 0.83 (0.34) HHs received food ration 2.17 (0.86) HHs regularly received nutrition training 0.68 (0.41) HHs regularly participated in agriculture related training 1.56 (0.93) HHs participated in any other activities 0.45* (0.17) HHs received food ration, agriculture and nutrition training 0.73 (0.43) Woman age (>=15 & <=20 year= reference age) 21 to 25 years 0.40** (0.11) 26 to 30 years 0.74 (0.17) 31 to 35 years 0.79 (0.22) 36 to 50 years 0.90 (0.22) Women's minimum dietary diversity 0.97 (0.05) Consumption expenditure quintiles (Poorest quintile= reference group) Quintile 2 1.24 (0.29) Quintile 3 0.90 (0.24) Quintile 4 1.16 (0.23) Quintile 5 (Richest) 1.03 (0.24) Household size 0.95 (0.03) Gendered household type (both male and female adults= reference group) Adult female only households 0.71 (0.14) Adult male only households 20.67 (23.85) Household education (Ref: HHs with no educated HH members) At least one member with primary level education 1.24 (0.26) Variables Model 6 Odds Ratio SE At least one member with >=secondary level educ. 1.43 (0.55) WASH Households with soap and water at handwashing station 0.76 (0.39) Households using improved water sources 1.18 (0.16) Agriculture variables Households using financial services (savings, agricultural credit or agricultural insurance) 1.04 (0.20) Households using value chain practices promoted by the project 0.86 (0.12) Households using at least three sustainable agriculture practices 1.43** (0.18) Households using improved storage practices 1.10 (0.15) Household consuming >=4 food groups 0.86 (0.17) Geographic districts (Napak= reference district) Moroto 1.36 (0.39) Nakapiripirit 1.02 (0.14) Amudat 0.71 (0.16) Constant Observations (women of 15-49 years in pooled BL and EL datasets) 2,487 F-statistic 4.87 Prob>F 0.00 SE = Standard error, HH = household *** p<0.001, ** p<0.01, * p<0.05 Table 8H.4: Results of Logistic Regression of the Nutritional Status Among Children and Women by Household's Goat Ownership Status (Pooled BL and EL datasets), ACDI RWANU [Uganda 2013, 2018] Nutritional Status Goats Ownership No Yes sig Children between 6-23 months Stunting 40.2 35.7 Wasting 22.6 14.7 Underweight 38.7 27.5 Dietary Diversity- 7 food groups 2.3 2.5 Meal Frequency (average number in the last 24 hours) 2.3 2.8 * Women Underweight 39.7 37.2 WDDS 2.6 3.1 *** ANNEX 8I: IPTT TABLES Indicators ID No. Source Indicator Dir. (+/-) Frequency Project Goal: Reduced Food Insecurity among vulnerable people in South Karamoja 1 IM1 Ft/F -2 Prevalence of poverty: Percent of people living on less than $ 1.25 / day (-) Baseline & Endline 2 IM2 FTF -3 Mean depth of poverty (-) Baseline & Endline 3 OC1 FTF/F -5 Daily per capita expenditure (as a proxy for income) of USG targeted beneficiaries (+) Baseline & Endline 4 OP1 FTF/F-34 Number of vulnerable households benefiting directly from USG assistance (+) Annual Strategic Objective 1: Improved access to food for men and women Outcome and Impact Indicators 5 IM3 FFP-29 Average Household Dietary Diversity Score (HDDS) (+) Baseline & Endline 6 IM4 FTF/F-28 Prevalence of households with moderate or severe hunger (Household Hunger Scale - HHS) (-) Baseline & Endline 7 OC2 FTF/F-9 Number of farmers and others who have applied improved technologies or management practices with USG assistance (+) Annual 8 OC3 FTF F-15 Number of hectares of land under improved technologies or management practices with USG assistance (+) Annual Output Indicators 9 OP2 FTF F-11 Number of individuals who have received USG supported short￾term agricultural sector productivity or food security training (+) Annual IR. 1.1: Improved smallholder farm management practices adopted Outcome Indicators 10 OC4 Mission FFP 22 Percentage of farmers who practiced the value chain activities promoted by the project in the last 12 months (+) Baseline & Endline 11 OC4 RWANU Percentage of farmers who practiced the value chain activities promoted by the project in the last 12 months (%) (+) Annual 12 OP3 FFP-27 Number of farmers who practiced the value chain activities promoted by the project in the last 12 months (#) (+) Annual 13 OC5 FFP-14 Percentage of farmers who used at least THREE sustainable agriculture practices and/or technologies in the past 12 months (+) Baseline & Endline 14 OC6 FFP-17 Percentage of farmers who used improved storage practices in the past 12 months (+) Baseline & Endline 15 OC6 RWANU Percentage of farmers who used improved storage practices in the past 12 months (+) Annual 16 OC7 FTF/F-8 Farmer’s gross margin per hectare, per animal, per cage obtained with USG assistance (+) Annual Output Indicators 17 OP4 FTF-84 Number of technologies or management practices in one of the following phases of development: in Phase II: under field (+) Annual Annual ID No. Source Indicator Dir. (+/-) Frequency testing as a result of USG assistance in Phase III: made available for transfer as a result of USG assistance Annual 18 OP5 FTF/F-10 Number of for-profit private enterprises, producers’ organizations, water users’ associations, women’s groups, trade and business associations and community-based organizations (CBOs) that applied improved organization-level technologies or management practices with USG assistance (+) Annual IR. 1.2: Improved smallholder livestock management practices adopted Outcome Indicators 19 OC8 RWANU Percentage of livestock farmers accessing government or private sector veterinary services (diseases diagnosis and drugs) and livestock vaccination (+) Baseline & Endline 20 OC8 RWANU Percentage of livestock farmers accessing government or private sector veterinary services (diseases diagnosis and drugs) and livestock vaccination (+) Annual Output Indicators 21 OP5 RWANU Number of persons receiving livestock inputs (+) Annual IR. 1.3: Increased linkages to markets Outcome Indicators 22 OC9 RWANU Number of farmers groups linked to at least one buyer (+) Annual Output Indicators 23 OP7 FTF/F-12 Number of for-profit private enterprises, producers’ organizations, water users’ associations, women's groups, trade and business associations, and community-based organizations (CBOs) receiving USG food security related organizational support (+) Annual IR 1.4 Access to credit increased Outcome Indicators 24 OC10 FFP-21 Percentage of farmers who used financial services (savings, agricultural credit, and/or agricultural insurance) in the past 12 months (+) Baseline & Endline 25 OC10 RWANU Percentage of farmers who used financial services (savings, agricultural credit, and/or agricultural insurance) in the past 12 months (+) Annual Output Indicators 26 OP8 RWANU Number of saving and credit groups formed (+) Annual OP21 FTF/F-25 Number of MSMES, including farmers, receiving business development services from USG assisted sources (+) Annual Strategic Objective 2: Reduced malnutrition in Pregnant and Lactating Mothers and Children under 5 27 IM5 FTF/F-6 Prevalence of stunted children under five years of age (-) Baseline & Endline 28 IM6 FTF/F-1 Prevalence of underweight children under five years of age (-) Baseline & Endline 29 IM6 RWANU Prevalence of underweight children under two years of age (-) Annual 30 OC11 FTF/F-36 Women's Dietary Diversity: Mean number of food groups consumed by women of reproductive age (WDDS) (+) Baseline & Endline ID No. Source Indicator Dir. (+/-) Frequency 31 OC11 RWANU Women's Dietary Diversity: Mean number of food groups consumed by women of reproductive age (WDDS) (+) Annual IR 2.1 Improved health and nutrition practices at household level Outcome Indicators 32 OC12 FTF/F-37 Prevalence of exclusive breastfeeding of children under six months of age (+) Baseline & Endline 33 OC12 RWANU Prevalence of exclusive breastfeeding of children under six months of age (+) Annual 34 OC13 FTF/F-35 Percentage of children 6-23 months receiving a minimum acceptable diet (MAD) (+) Baseline & Endline 35 OC13 RWANU Percentage of children 6-23 months receiving a minimum acceptable diet (MAD) (+) Annual 36 OC14 RWANU Percentage of caregivers who know at least 4 of 6 IYCF and 3 of 8 MCH practices (+) Annual 37 OC15 USAID / HIP Percentage of respondents who know at least 3 of the 5 critical moments for hand washing (+) Baseline cf OC15 RWANU Percentage of respondents who know at least 3 of the 5 critical moments for hand washing (+) Annual 39 OC16 Mission Percentage of households with soap and water at a hand washing station commonly used by family member (+) Baseline & Endline 40 OC16 RWANU Percentage of households with soap and water at a hand washing station commonly used by family member (+) Annual 41 OC16 F-40 Percent of households using an improved drinking water source (+) Baseline & Endline 41 OC17 F-40 Percent of households using an improved drinking water source (+) Annual 42 OC17 F-41 Percent of households using an improved sanitation facility (+) Baseline & Endline 42 OC18 F-41 Percent of households using an improved sanitation facility (+) Annual 43 OP7 F-50 Number of communities verified as “open defecation free” (ODF) as a result of USG assistance (+) Annual Output Indicators 44 OP8 RWANU Number of active mother care groups (+) Annual 45 OP9 F-32 Number of people benefiting from USG supported social assistance (+) Annual 46 OP10 RWANU Number of people trained on environmentally appropriate hygiene and sanitation behaviors (+) Annual IR 2.2: Improved service delivery for prevention and treatment of maternal and child illnesses Outcome Indicators 47 OC19 RWANU Percentage of mothers with children aged 0-12 months who had four or more antenatal visits when they were pregnant with their youngest child (+) Annual 48 OC19 FFP-52 Percentage of births receiving at least four antenatal care (ANC) visits during pregnancy (+) Baseline & Endline 49 OC20 F-38 Percentage of children under age of five who had diarrhea in the prior two weeks (-) Baseline & Endline 50 OC20 RWANU Percentage of children under age of five who had diarrhea in the prior two weeks (-) Annual ID No. Source Indicator Dir. (+/-) Frequency 51 OC21 F-39 Percent of children under 5 years old with diarrhea treated with Oral Rehydration Therapy (ORT) (+) Baseline & Endline 52 OC21 RWANU Percent of children under two years old with diarrhea treated with Oral Rehydration Therapy (ORT) (+) Annual 53 OC22 RWANU Percent of children aged 12-23 months who are fully immunized (BCG, DPT3B, polio3/OPV3, measles/MMR) (+) Annual Output Indicators 54 OP13 FTF/F-56 Number of people trained in child health and nutrition through USG-supported programs (+) Annual 55 OP14 RWANU Number of people trained on using the health center community score card (+) Annual 56 OP15 FTF/F-57 Number of children under five (0-59 months) reached by nutrition￾specific interventions through USG-supported nutrition programs (+) Annual 57 OP16 RWANU Number of health centers measuring performance using the community score card (+) Annual Cross-Cutting Gender IR: Increased number of households where women and men jointly make informed decisions regarding productive assets, food security, and nutrition Outcome Indicator 58 OC23 RWANU Percentage of women ages 15-49 and currently living with a man reporting that she makes decisions either by herself or jointly with the man regarding seeking health services for her own health; seeking health services for her children's health; how to spend money she herself has earned; and how to use productive assets (+) Baseline Output Indicator 59 OP17 F-60 Percentage of participants in USG-Assisted programs designed to increase access to productive economic resources (assets, credit, income or employment) that are female (+) Annual Cross-cutting: Conflict Mitigation and Environment Outcome Indicators 60 OC24 RWANU Percentage of farmers adopting Farmer Managed Natural Regeneration practices (+) Annual 61 OC25 RWANU % of CAG members reporting that its meetings are useful and achieving their objective (+) Annual Output Indicators 62 OP18 RWANU Number of Community Action Groups that meet a minimum of once every three months (+) Annual 63 OP19 RWANU Number of individuals receiving USG supported training in natural resource management and/or bio-diversity conservation (+) Annual Cross-cutting: Disaster Risk Management and Reduction Output Indicators 64 OP20 RWANU Number of farmer groups assisted to develop response plans to early warning information (+) Annual Contextual Indicator 65 CI1 FTF/F-7 Prevalence of underweight women (of reproductive age) (-) Baseline & Endline ID No. Source Indicator Dir. (+/ - ) Frequency 66 CI2 Mission Women's Empowerment in Agriculture Index (WEAI) (+) Baseline & Endline (some WEAI indictors only at EL) ( - ) FTF -71 ( - ) ( - ) ( - ) Poverty ID No. Source Indicator Dir. Disaggre￾gation Frequency Baseline (or FY13) LOA Target LOA or 2017 Achieved % LOA Target Achieved (nominal) Percentage point changes Endline Result Statistically (+/-) Significant? (see note 5) Project Goal: Reduced Food Insecurity among vulnerable people in South Karamoja 1 IM1 FtF/F -2 Prevalence of poverty: Percent of people living on less than $ 1.25 / day (-) Total Baseline & Endline 95.6% 88.7% 92.9% 104.7% 2.7 % point decrease Decrease not Statistically Significant and Well Below Target 2 IM2 FTF -3 Mean depth of poverty (-) Total Baseline & Endline 66.7% 61.9% 60.9% 98.4% 5.8 % point decrease Statistically Significant Improvement Slightly More than Target Decrease HHFNM 60.5% 55.7% 60.2% HHMNF 51.5% 46.7% 65.4% HHM&F 67.4% 62.8% 60.9% HHCNA 0.0% 0.0% NA 3 OC1 FTF/F -5 Daily per capita expenditure (as a proxy for income) of USG targeted beneficiaries (+) Total Baseline & Endline $0.52 $0.94 $0.69 73.7% NA Statistically Significant Improvement but Well Below Target Increase HHFNM $0.61 $1.10 HHMNF $0.76 $1.37 HHM&F $0.51 $0.92 HHCNA - $0.00 4 OP1 FTF/F-34 Number of vulnerable households benefiting directly from USG assistance (+) Total Annual 27,794 42,859 154% NA NA HHFNM 0 5397 8,517 158% HHMNF 0 150 150 100% HHM&F 0 22213 34,147 154% HHCNA 0 34 45 132% New 0 27,794 53,211 191% Continuing 0 0 42,859 Note 1: OP = Output; OC = Outcome, IM = Impact and CI = Contextual, NA = Not Available, Note 2: Membership composition in groups are: Farmer Training Groups were formed with 25 members, Honey Groups with 20 members, Goats groups and MCG with 10 - 15. Note 3: Where possible census data were used to estimate targets of project beneficiaries 0 Note 4: Initial targets provided by ACDI-VOCA based on national targets and/or feasible results based on prior experience. Actual target determined and agreed upon with USAID. Note 5: Indicators source renamed to RWANU for some FFP or FTF indicators since the reporting frequency changes to annual as required by RWANU and not FTF or FFP Note 6: Percentage point changes - important esp. where % of achieved is calculated w/ 0 as start point and/or is described for % targets, and where results worse than baseline Color coding: reversed <60 % achieved 60-79% achieved 80-99% achieved 100-120% achieved > 120% achieved Food Access ID No. Source Indicator Dir. (+/-) Disaggre￾gation Frequency Baseline (or FY13) LOA Target LOA or 2017 Achieved % LOA Target Achieved (nominal) Percentage point changes Endline Result Statistically Significant? (see note 5) Strategic Objective 1: Improved access to food for men and women Outcome and Impact Indicators 5 IM3 FFP-29 Average Household Dietary Diversity Score (HDDS) (+) Total Baseline & Endline 2.7 3.5 2.9 83% NA Improvement Not Statistically Significant & Well Below Target Increase 6 IM4 FTF/F-28 Prevalence of households with moderate or severe hunger (Household Hunger Scale - HHS) (-) Total Baseline & Endline 69% 59.0% dropped as endline indicator dropped as endline indicator NA HHM&F 69% 59.0% HHMNF 76% 66.0% HHFNM 69% 59.0% HHCNA - NA 7 OC2 FTF/F-9 Number of farmers and others who have applied improved technologies or management practices with USG assistance (+) Producers Annual 0% 18,292 18,814 103% NA NA Total -- w/one or more Annual 0 18,292 18,814 103% NA Female 0 11,630 11,807 102% NA Male 0 6,662 7,007 105% NA Crop genetics 0 5,937 8,800 148% NA NA Cultural practices 0 15,326 15,790 103% NA NA Livestock management 0 2,267 2,258 100% NA NA Pest management 0 211 15,423 7309% NA NA Disease management 0 211 0 0% NA NA Soil-related 0 12,463 12,847 103% NA NA ID No. Source Indicator Dir. (+/ - ) Disaggre - gation Frequency Baseline (or FY13) LOA Target LOA or 2017 Achieved % LOA Target Achieved (nominal) Percentage point changes Endline Result Statistically Significant? (see note 5) Irrigation, 0 49 4 8% NA NA PHH & storage 0 10,695 12,724 119% NA NA Climate mitigation 0 12,211 14,319 117% NA NA Crop Farmers 0 15,326 15,882 104% NA NA Horticulture farmers 0 126 4 3% NA NA Apiary 0 670 670 100% NA NA 8 OC3 FTF F -15 Number of hectares of land under improved technologies or management practices with USG assistance (+) Total (with one or more technology) Annual 0 76,722 24,448 32% NA NA Crop genetics 0 19,371 14,017 72% NA NA Cultural practices 0 40,109 24,347 61% NA NA Pest management 0 787 23,896 3036% NA NA Disease management 0 974 0 0% NA NA Soil related 0 25,312 20,743 82% NA NA Others (Land preparation) 0 864 972 112% NA NA Climate mitigation or adaptation’ 0 17,280 22,537 130% NA NA Of overall TOTAL for this indicator Female 0 46,194 13,312 29% NA NA Male 0 30,048 10,953 36% NA NA Joint 0 N/A N/A N/A NA NA Association - Applied 0 479 183 38% NA NA Output Indicators 9 OP2 FTF F -11 Number of individuals who have received USG supported short - term agricultural sector productivity or food security training (+) Total Annual 0 40,812 16,319 40% NA NA ID No. Source Indicator Dir. (+/-) Disaggre￾gation Frequency Baseline (or FY13) LOA Target LOA or 2017 Achieved % LOA Target Achieved (nominal) Percentage point changes Endline Result Statistically Significant? (see note 5) Farmers, male (Producers) 0 10,814 4,822 45% NA NA Farmers, female (Producers) 0 25,529 8,619 34% NA NA Honey producers, male (Producers) 0 1,175 398 34% NA NA Honey producers, female (Producers) 0 987 205 21% NA NA Horticulture farmers, male (Producers) 0 724 142 20% NA NA Horticulture farmers, female (Producers) 0 2,063 151 7% NA NA Goat rearing group members, male (Producers) 0 0 0 0% NA Goat rearing group members, female (Producers) 0 6,815 1,824 27% NA NA CAHWs, male (Producers) 0 422 133 32% NA NA CAHWs, female (Producers) 0 74 25 34% NA NA Subtotal Male 0 11,593 5,495 47% NA NA Subtotal Female 0 29,219 10,824 37% NA NA Note 1: OP = Output; OC = Outcome, IM = Impact and CI = Contextual, NA = Not Available, Note 2: Membership composition in groups are; Farmer Training Groups were formed with 25 members, Honey Groups with 20 members, Goats groups and MCG with 10 - 15. Note 3: Where possible census data were used to estimate targets of project beneficiaries Note 4: Initial targets provided by ACDI-VOCA based on national targets and/or feasible results based on prior experience. Actual target determined and agreed upon with USAID Note 5: Indicators source renamed to RWANU for some FFP or FTF indicators since the reporting frequency changes to annual as required by RWANU and not FTF or FFP Note 6: Percentage point changes - important esp. where % of achieved is calculated w/ 0 as start point and/or is described for % targets, and where results worse than baseline Color coding: reversed <60 % achieved 60-79% achieved 80-99% achieved 100-120% achieved > 120% achieved Farming-Livestock ID No. Source Indicator Dir. (+/-) Disaggre￾gation Frequency Baseline (or FY13) LOA Target LOA or 2017 Achieved % LOA Target Achieved (nominal) Percentage point changes Endline Result Statistically Significant? (see note 5) IR. 1.1: Improved smallholder farm management practices adopted Outcome Indicators 10 OC4 Mission Percentage of farmers who practiced the value chain activities promoted by the project in the last 12 months (+) Total Baseline & Endline 77.4% 80.0% 34% 43% Short of target by 37 % points (and worse than baseline by 43.4% points) Methodologic al differences in the way this indicator was measured mean that it cannot be compared at BL/EL males 76.3% 79.0% 39% 39% FFP-22 females 78.4% 81.0% 29% 29% 11 OC4 RWANU Percentage of farmers who practiced the value chain activities promoted by the project in the last 12 months (%) (+) Total Annual N/A 99% 92.28% 93% NA NA males N/A 99% 90.61% 92% females N/A 99% 93.44% 94% 12 OP3 FFP-27 Number of farmers who practiced the value chain activities promoted by the project in the last 12 months (#) (+) Total Annual N/A 17,926 16,097 90% NA NA males N/A 6,529 6,619 101% females N/A 11,397 9,478 83% 13 OC5 FFP-14 Percentage of farmers who used at least THREE sustainable agriculture practices and/or technologies in the past 12 months (+) Total Baseline & Endline 19.0% 100.0% 35.0% 50% Short of target by 49.6 % points Statistically Significant Improvement but Well Below Target Increase males 22.9% 100.0% 56.7% females 15.5% 100.0% 45.1% ID No. Source Indicator Dir. (+/ - ) Disaggre - gation Frequency Baseline (or FY13) LOA Target LOA or 2017 Achieved % LOA Target Achieved (nominal) Percentage point changes Endline Result Statistically Significant? (see note 5) Crop farming 13.2% 100.0% missing? Livestock farming 14.9% 100.0% missing? NRM farming 15.9% 100.0% missing? 14 OC6 FFP -17 Percentage of farmers who used improved storage practices in the past 12 months (+) Total Baseline & Endline 52.2% 57.6% 47.1% 82% Short of target by 10.5% points (and worse than baseline by 5.1% points) Decline Not Statistically Significant but Well Below Target Increase males 50.2% 56.0% 44.5% 79% females 54.1% 60.0% 49.2% 82% 15 OC6 RWANU Percentage of farmers who used improved storage practices in the past 12 months (+) Total Annual N/A 65% 55.01% 85% Short of target by 10 % points NA males N/A 65% 54.71% 84% females N/A 65% 55.23% 85% 16 OC7 FTF/F - 8 Farmer’s gross margin per hectare, per animal, per cage obtained with USG assistance (+) males Annual N/A 544.5 0 0% - NA females Annual N/A 437.3 18 4% - NA Output Indicators 17 OP4 FTF -84 Number of technologies or management practices in one of the following phases of development: in Phase II: under field testing as a result of USG assistance in Phase III: made available for transfer as a result of USG assistance (+) Total Annual 0 34 25 74% NA NA Under field testing as a result of USG assistance Annual 0 22 21 95% NA NA Made available for transfer as result of USG assistance Annual 0 12 4 33% NA NA 18 OP5 FTF/F -10 (+) Total Annual 0 1,290 1,511 117% NA NA ID No. Source Indicator Dir. (+/-) Disaggre￾gation Frequency Baseline (or FY13) LOA Target LOA or 2017 Achieved % LOA Target Achieved (nominal) Percentage point changes Endline Result Statistically Significant? (see note 5) Number of for-profit private enterprises, producers’ organizations, water users’ associations, women’s groups, trade and business associations and community-based organizations (CBOs) that applied improved organization-level technologies or management practices with USG assistance Honey groups new (Producers) Annual 0 41 36 89% NA NA Farmer group New (Producers) Annual 0 662 650 98% NA NA Horticulture groups new (Producers) Annual 0 39 46 118% NA NA Producers sub-Total 0 NA 732 Livestock groups new (Women) Annual 0 218 188 86% NA NA Livestock sub Total 0 NA 188 Savings groups (New) Annual 0 330 591 179% NA NA Savings Sub￾Total 0 NA 591 Total New 0 1,290 1,511 117% NA NA IR. 1.2: Improved smallholder livestock management practices adopted Outcome Indicators 19 OC8 RWANU Percentage of livestock farmers accessing government or private sector veterinary services (diseases diagnosis and drugs) and livestock vaccination (+) Total livestock Farmers Baseline & Endline 69% 77.0% 22.6% 29% Fell short of reduction target by 54.4 % points (and worse than baseline by 46.4 % points) Methodologic al differences in the way the data were collected at BL and EL mean that comparisons cannot be drawn. 20 OC8 RWANU Percentage of livestock farmers accessing government or private sector veterinary services (diseases diagnosis and drugs) and livestock vaccination (+) Total livestock Farmers Annual N/A 100% 78.6% 79% NA NA ID No. Source Indicator Dir. (+/-) Disaggre￾gation Frequency Baseline (or FY13) LOA Target LOA or 2017 Achieved % LOA Target Achieved (nominal) Percentage point changes Endline Result Statistically Significant? (see note 5) Output Indicators 21 OP5 RWANU Number of persons receiving livestock inputs (+) Total (Females) Annual 2,100 2,267 108% NA NA Amudat 160 72 45% Moroto 160 170 106% Nakapiripirit 830 1,027 124% Napak 950 998 105% Note 1: OP = Output; OC = Outcome, IM = Impact and CI = Contextual, NA = Not Available, Note 2: Membership composition in groups are; Farmer Training Groups were formed with 25 members, Honey Groups with 20 members, Goats groups and MCG with 10 - 15. Note 3: Where possible census data were used to estimate targets of project beneficiaries Note 4: Initial targets provided by ACDI-VOCA based on national targets and/or feasible results based on prior experience. Actual target determined and agreed upon with USAID Note 5: Indicators source renamed to RWANU for some FFP or FTF indicators since the reporting frequency changes to annual as required by RWANU and not FTF or FFP Note 6: Percentage point changes - important esp. where % of achieved is calculated w/ 0 as start point and/or is described for % targets, and where results worse than baseline Color coding: reversed <60 % achieved 60-79% achieved 80-99% achieved 100-120% achieved > 120% achieved Markets ID No. Source Indicator Dir. (+/-) Disaggre￾gation Frequency Baseline (or FY13) LOA Target LOA Achieved % LOA Target Achieved (nominal) Percentage point changes (see note 5) Endlne Result Statistically Significant? IR. 1.3: Increased linkages to markets Outcome Indicators 22 OC9 RWANU Number of farmers groups linked to at least one buyer (+) Total Annually 0 264 74 28% NA NA Grain buyer Annual 0 196 28 14% NA NA Livestock buyer Annual 0 22 0 0% NA NA Honey buyer Annual 0 35 35 100% NA NA Horticulture buyer Annual 0 11 11 100% NA NA Output Indicators 23 OP7 FTF/F-12 Number of for-profit private enterprises, producers’ organizations, water users’ associations, women's groups, trade and business associations, and community-based organizations (CBOs) receiving USG food security related organizational support (+) Total Annual 0 1,639 1,504 92% NA NA Honey groups new (Producers) Annual 0 41 40 99% NA NA Horticulture groups new (Producers) Annual 0 53 46 87% NA NA Farmer groups new (Producers) Annual 0 662 650 98% NA NA Subtotal Producer Groups NA 736 Livestock groups new (Women) Annual 0 218 188 86% NA NA Subtotal Women's Groups NA 188 Savings groups new (Trade) Annual 0 666 580 87% NA NA Subtotal Trade Groups NA 580 New Annual 0 1,639 1,504 92% NA NA IR 1.4 Access to credit increased Outcome Indicators 24 OC10 FFP-21 Percentage of farmers who used financial services (savings, agricultural credit, and/or agricultural insurance) in the past 12 months (+) Total Baseline & Endline 27.1% 52.0% 9.0% 17% Fell short of reduction target by 9.3 % points (and worse than baseline by 8.8 % points) Statistically Significant Decline and Well Below Targeted Increase males 26.3% 51.0% 11.0% 22% females 27.7% 53.0% 7.0% 13% ID No. Source Indicator Dir. (+/-) Disaggre￾gation Frequency Baseline (or FY13) LOA Target LOA Achieved % LOA Target Achieved (nominal) Percentage point changes (see note 5) Endlne Result Statistically Significant? 25 OC10 RWANU Percentage of farmers who used financial services (savings, agricultural credit, and/or agricultural insurance) in the past 12 months (+) Total Annually N/A 85.0% 52.51% 62% Short of target by 32.5 % points NA males N/A 85.0% 51.64% 61% females N/A 85.0% 53.11% 62% Output Indicators 26 OP8 RWANU Number of saving and credit groups formed (+) Total Annually N/A 723 580 80% NA NA Amudat N/A 41 32 78% NA NA Moroto N/A 31 27 86% NA NA Nakapiripirit N/A 332 302 91% NA NA Napak N/A 319 219 69% NA NA OP21 FTF/F-25 Number of MSMES, including farmers, receiving business development services from USG assisted sources (+) Total 287 110 38% NA NA micro 287 110 38% small NA NA NA medium NA NA NA male 144 74 51% Female 143 36 25% joint NA NA NA n/a NA NA NA Producer 287 110 38% Input supplier NA NA NA Trader NA NA NA Processors NA NA NA Non-agricultural NA NA NA Other NA NA NA Note 1: OP = Output; OC = Outcome, IM = Impact and CI = Contextual, NA = Not Available, Note 2: Membership composition in groups are; Farmer Training Groups were formed with 25 members, Honey Groups with 20 members, Goats groups and MCG with 10 - 15. Note 3: Where possible census data were used to estimate targets of project beneficiaries 0 Note 4: Targets are provided based on national targets and/or feasible results based on past organization performance. Actual targets will be determined and agreed upon with USAID after review of the baseline levels Note 5: Indicators source renamed to RWANU for some FFP or FTF indicators since the reporting frequency changes to annual as required by RWANU and not FTF or FFP Note 6: percentage point changes - important esp. where % of achieved is calculated in such a way to imply 0 start point and/or is described for % targets, and where indicators are worse than baseline Color coding: reversed <60 % achieved 60-79% achieved 80-99% achieved 100-120% achieved > 120% achieved Nutrition-Health ID No. Source Indicator Dir. (+/-) Disaggre￾gation Frequency Baseline (or FY13) LOA Target LOA or 2017 Achieved % LOA Target Level (nominal) Percentage point changes (see note 5) Endlne Result Statistically Significant? Strategic Objective 2: Reduced malnutrition in Pregnant and Lactating Mothers and Children under 5 27 IM5 FTF/F-6 Prevalence of stunted children under five years of age (-) Total Baseline & Endline 38% 28.0% 36.1% 129% Fell short of reduction target by 8.1 % points (and improved on baseline by 2.1 % points) Increase Not Statistically Significant but Well Above Targeted Reduction males 43% 33.0% 39.4% 119% females 34% 24.0% 33.1% 138% 28 IM6 FTF/F-1 Prevalence of underweight children under five years of age (-) Total Baseline & Endline 20.5% 16.0% 29.3% 183% Fell short of reduction target by 9.3 % points (and exceeded baseline by 8.8 % points) Increase Statistically Significant and Well Above Targeted Reduction males 23.1% 17.1% 34.2% 200% females 17.9% 15.0% 24.8% 165% 29 IM6 RWANU Prevalence of underweight children under two years of age (-) Total Annual N/A 23.3% 32.1% 138% Fell short of reduction target by 8.8 % points NA males N/A 15.0% 40.7% 271% females N/A 13.0% 23.7% 182% 30 OC11 FTF/F-36 Women's Dietary Diversity: Mean number of food groups consumed by women of reproductive age (WDDS) (+) n/a Baseline & Endline 2.6 3.0 2.7 90% NA Increase Not Statistically Significant and Well Below Targeted Increase 31 OC11 RWANU Women's Dietary Diversity: Mean number of food groups consumed by women of reproductive age (WDDS) (+) n/a Annual N/A 4.0 3.1 78% NA NA ID No. Source Indicator Dir. (+/-) Disaggre￾gation Frequency Baseline (or FY13) LOA Target LOA or 2017 Achieved % LOA Target Level (nominal) Percentage point changes (see note 5) Endlne Result Statistically Significant? IR 2.1 Improved health and nutrition practices at household level Outcome Indicators 32 OC12 FTF/F-37 Prevalence of exclusive breastfeeding of children under six months of age (+) Total Baseline & Endline 58.5% 68.5% 75.6% 110% exceeded target by 10 % points Statistically Significant Increase Overall and for Boys (Increase not SS for Girls) males 56.0% 66% 86.2% females 61.4% 71.4% 68.2% 33 OC12 RWANU Prevalence of exclusive breastfeeding of children under six months of age (+) Total Annual N/A 86.0% 63.0% 73% short of target by 23 % points NA males N/A 88.0% 67.8% 77% females N/A 84.0% 55.38% 66% 34 OC13 FTF/F-35 Percentage of children 6-23 months receiving a minimum acceptable diet (MAD) (+) Total Baseline & Endline 7% 10.0% 7.7% 77% .7 %-point increase Increase not Statistically Significant and Below Target males 7% 10.0% 7.1% females 7% 10.0% 8.3% 35 OC13 RWANU Percentage of children 6-23 months receiving a minimum acceptable diet (MAD) (+) Total Annual N/A 25.0% 16.9% 68% short of target by 8.1 % points NA males Annual N/A 22.1% 14.3% 65% females Annual N/A 28.0% 13.5% 48% 36 OC14 RWANU Percentage of caregivers who know at least 4 of 6 IYCF and 3 of 8 MCH practices (+) Total Annual N/A 90.0% 87.0% 97% short of target by 3 % points NA 37 OC15 USAID / HIP Percentage of respondents who know at least 3 of the 5 critical moments for hand washing (+) Total Baseline & Endline 71% 75.0% dropped NA NA NA ID No. Source Indicator Dir. (+/ - ) Disaggre - gation Frequency Baseline (or FY13) LOA Target LOA or 2017 Achieved % LOA Target Level (nominal) Percentage point changes (see note 5) Endlne Result Statistically Significant? 38 OC15 RWANU Percentage of respondents who know at least 3 of the 5 critical moments for hand washing (+) Total Annually N/A 80.0% 75.1% 94% short of target by 4.9 % points NA 39 OC16 Mission Percentage of households with soap and water at a hand washing station commonly used by family member (+) Total Baseline & Endline 4% 6.0% 1.2% 20% 2.8 % -point decline Statistically Significant Decline and Well Below Targeted Increase 40 OC16 RWANU Percentage of households with soap and water at a hand washing station commonly used by family member (+) Total Annual N/A 3.0% 3.5% 116% NA NA 41 OC16 F -40 Percent of households using an improved drinking water source (+) none Baseline & Endline 41.9% 86.0% 36.9% 43% 5 % -point decline Decline not Statistically Significant but Well Below Targeted Increase 41 OC17 F -40 Percent of households using an improved drinking water source (+) none Annually 41.9% 95.0% 93.6% 99% short of target by 1.4 % points NA 42 OC17 F -41 Percent of households using an improved sanitation facility (+) none Baseline & Endline 4.0% 25.0% 1.2% 5% 2.8 % -point decline and short of target by 23.8 % points Statistically Significant Decline and Well Below Targeted Increase 42 OC18 F -41 Percent of households using an improved sanitation facility (+) none Annual 4.0% 15.0% 8.80% 59% short of target by 6.2 % points NA 43 OP7 F -50 Number of communities verified as “open defecation free” (ODF) as a result of USG assistance (+) none Annual N/A 71 7.00 10% NA NA Output Indicators ID No. Source Indicator Dir. (+/-) Disaggre￾gation Frequency Baseline (or FY13) LOA Target LOA or 2017 Achieved % LOA Target Level (nominal) Percentage point changes (see note 5) Endlne Result Statistically Significant? 44 OP8 RWANU Number of active mother care groups (+) Total Annually N/A 350 345 99% NA NA 45 OP9 F-32 Number of people benefiting from USG supported social assistance (+) Total Annually N/A 282,984 148,981 53% NA NA Pregnant and Lactating Women N/A 31,193 29,978 96% NA NA Male Children N/A 23,417 15,230 65% NA NA Female Children N/A 23,548 15,732 67% NA NA Other HH members N/A 204,826 88,041 43% NA NA Male 125,830 64,681 51% Female 157,154 84,300 54% 46 OP10 RWANU Number of people trained on environmentally appropriate hygiene and sanitation behaviors (+) Total Annually N/A 140,756 141,071 100% NA NA Male 7,747 7,793 101% Female 133,009 133,278 100% IR 2.2: Improved service delivery for prevention and treatment of maternal and child illnesses Outcome Indicators 47 OC19 RWANU Percentage of mothers with children aged 0-12 months who had four or more antenatal visits when they were pregnant with their youngest child (+) N/A Annually N/A 85.0% 81.4% 96% short of target by 3.6% points NA 48 OC19 FFP-52 Percentage of births receiving at least four antenatal care (ANC) visits during pregnancy (+) N/A Baseline & Endline 75% 80.0% 83.9% 105% exceeded target by 3.9 % points Statistically Significant Results Exceeded Target 49 OC20 F-38 Percentage of children under age of five who had (-) Total Baseline & Endline 23% 20% 27.2% 136% short of target decrease by Results not Statistically Significant ID No. Source Indicator Dir. (+/-) Disaggre￾gation Frequency Baseline (or FY13) LOA Target LOA or 2017 Achieved % LOA Target Level (nominal) Percentage point changes (see note 5) Endlne Result Statistically Significant? diarrhea in the prior two weeks 7.2 % points & 4.2 % points worse than baseline males 23% 20% 25.8% 129% females 23% 20% 28.5% 143% 50 OC20 RWANU Percentage of children under age of five who had diarrhea in the prior two weeks (-) Total Annual N/A 25.0% -10.84% ? NA males N/A 25.0% -10.55% -237% females N/A 25.0% -11.13% -225% 51 OC21 F-39 Percent of children under 5 years old with diarrhea treated with Oral Rehydration Therapy (ORT) (+) Total Baseline & Endline 83% 85.0% 84.0% 98% short of target by 1 % point Not Statistically Significant males 84% 85.0% 81.0% 95% females 82% 85.0% 84.4% 99% 52 OC21 RWANU Percent of children under two years old with diarrhea treated with Oral Rehydration Therapy (ORT) (+) Total Annual N/A 95.0% 84.40% 89% short of target by 10.6 % points NA males N/A 95.0% 89.33% 94% females N/A 95.0% 80.00% 84% 53 OC22 RWANU Percent of children aged 12-23 months who are fully immunized (BCG, DPT3B, polio3/OPV3, measles/MMR) (+) none Annually N/A 90.0% 83.23% 92% short of target by 6.8% points NA Output Indicators 54 OP13 FTF/F-56 Number of people trained in child health and nutrition through USG￾supported programs (+) Total Annually N/A 319,915 346,958 108% NA NA Male N/A 8,000 15,119 189% NA Female N/A 311,915 331,839 106% ID No. Source Indicator Dir. (+/-) Disaggre￾gation Frequency Baseline (or FY13) LOA Target LOA or 2017 Achieved % LOA Target Level (nominal) Percentage point changes (see note 5) Endlne Result Statistically Significant? 55 OP14 RWANU Number of people trained on using the health center community score card (+) Total Annually N/A 3,053 2,780 91% NA NA males N/A 1,527 1,378 90% females N/A 1,526 1,402 92% 56 OP15 FTF/F-57 Number of children under five (0-59 months) reached by nutrition-specific interventions through USG￾supported nutrition programs (+) Total Annually N/A 78,319 92,865 119% NA NA males N/A 38,771 45,221 117% females N/A 39,548 47,644 120% 57 OP16 RWANU Number of health centers measuring performance using the community score card (+) none Annually N/A 34 31 91% NA NA Note 1: OP = Output; OC = Outcome, IM = Impact and CI = Contextual, NA = Not Available, Note 2: Membership composition in groups are; Farmer Training Groups were formed with 25 members, Honey Groups with 20 members, Goats groups and MCG with 10 - 15. Note 3: Where possible census data were used to estimate targets of project beneficiaries Note 4: Initial targets provided by ACDI-VOCA based on national targets and/or feasible results based on prior experience. Actual target determined and agreed upon with USAID Note 5: Indicators source renamed to RWANU for some FFP or FTF indicators since the reporting frequency changes to annual as required by RWANU and not FTF or FFP Note 6: Percentage point changes - important esp. where % of achieved is calculated w/ 0 as start point and/or is described for % targets, and where results worse than baseline Color coding: reversed <60 % achieved 60-79% achieved 80-99% achieved 100-120% achieved > 120% achieved Cross-cutting ID No. Source Indicator Dir. (+/-) Disaggre￾gation Frequency Baseline (or FY13) LOA Target LOA or 2017 Achieved % LOA Target Achieved (nominal) Percentage point changes (see note 5) Endlne Result Statistically Significant? Cross-Cutting Gender IR: Increased number of households where women and men jointly make informed decisions regarding productive assets, food security, and nutrition Outcome Indicator 58 OC23 RWANU Percentage of women ages 15- 49 and currently living with a man reporting that she makes decisions either by herself or jointly with the man regarding seeking health services for her own health; seeking health services for her children's health; how to spend money she herself has earned; and how to use productive assets (+) Total Baseline & Endline 80.5% dropped Seeking health services for her own health 74.0% 95.0% dropped Seeking health services for her children's health 74.1% 95.0% dropped How to spend money she herself has earned 69%* 77.0% dropped How to use productive assets 74%* 55.0% dropped ID No. Source Indicator Dir. (+/-) Disaggre￾gation Frequency Baseline (or FY13) LOA Target LOA or 2017 Achieved % LOA Target Achieved (nominal) Percentage point changes (see note 5) Endlne Result Statistically Significant? Output Indicator 59 OP17 F-60 Percentage of participants in USG-Assisted programs designed to increase access to productive economic resources (assets, credit, income or employment) that are female (+) Total Annually N/A 75.0% 76.0% 101% exceeds target by 1 % point NA 10-29 years N/A 77.0% 79.0% 103% 30 and above N/A 74.0% 75.0% 101% No age available N/A 0.0% 0.0% Cross-cutting: Conflict Mitigation and Environment Outcome Indicators 60 OC24 RWANU Percentage of farmers adopting Farmer Managed Natural Regeneration practices (+) Total Annually N/A 97.0% 90.54% 93% short of target by 6.5 % points NA males N/A 95.0% 88.73% 93% females N/A 98.0% 91.80% 94% 61 OC25 RWANU % of CAG members reporting that its meetings are useful and achieving their objective (+) Total Annually N/A 80.0% 100% 125% exceeded target by 20 % points NA Amudat 80.0% 100% 125% Moroto 80.0% 100% 125% Nakapiripirit 80.0% 100% 125% Napak 80.0% 100% 125% Output Indicators 62 OP18 RWANU Number of Community Action Groups that meet a minimum of once every three months (+) Total Annual N/A 208 115.00 55% NA NA Amudat 11 7.00 64% ID No. Source Indicator Dir. (+/-) Disaggre￾gation Frequency Baseline (or FY13) LOA Target LOA or 2017 Achieved % LOA Target Achieved (nominal) Percentage point changes (see note 5) Endlne Result Statistically Significant? Moroto 11 6.00 55% Nakapiripirit 93 40.00 43% Napak 93 62.00 67% 63 OP19 RWANU Number of individuals receiving USG supported training in natural resource management and/or bio￾diversity conservation (+) Total Annual N/A 3,432 1320 38% NA NA Amudat, Male 81 18 22% Amudat, Females 87 11 13% Moroto, Males 66 53 80% Moroto, Females 78 52 67% Nakapiripirit , Males 754 321 43% Nakapiripirit , Females 817 410 50% Napak, Males 710 159 22% Napak, Females 838 296 35% Cross-cutting: Disaster Risk Management and Reduction Output Indicators 64 OP20 RWANU Number of farmer groups assisted to develop response plans to early warning information (+) Total Annual N/A 284 121 43% NA NA Amudat 14 15 107% Moroto 14 3 21% Nakapiripirit 133 58 44% Napak 123 45 37% ID No. Source Indicator Dir. (+/ - ) Disaggre - gation Frequency Baseline (or FY13) LOA Target LOA or 2017 Achieved % LOA Target Achieved (nominal) Percentage point changes (see note 5) Endlne Result Statistically Significant? Contextual Indicator 65 CI1 FTF/F - 7 Prevalence of underweight women (of reproductive age) ( - ) none Baseline & Endline 26.8% 14.0% 45.5% 325% Fell short of reduction target by 31.5 % points (and exceeded baseline by 18.7 % points) Increase Statistically Significant and More than Triple Targeted Reduction 66 CI2 Mission Women's Empowerment in Agriculture Index (WEAI) (+) Index Baseline & Endline 0.806 0.845 See Below NA ( - ) % of disempower ed women 44.7% 34.7% Dropped NA FTF -71 ( - ) Average inadequacy score of disempower ed women 37.8% 27.2% Dropped NA ( - ) % of dual adult HH with no parity 40.2% 30.0% Dropped NA ( - ) Average % gap in adequacy score in dual adult HH with no parity 20.5% 10.5% Dropped NA REVISED for ENDLINE Female - Access to and decision on credit 17.0% ? 17.2% NA 0.2 % point increase Change Not Statistically Significant REVISED for ENDLINE Male - Access to and decision on credit 18.2% ? 17.5% NA 0.7 % point reduction REVISED for ENDLINE Female - Control over and use of income 58.5% ? 79.2% NA 20% points = 35% increase Increase / Improvement Statistically Significant ID No. Source Indicator Dir. (+/-) Disaggre￾gation Frequency Baseline (or FY13) LOA Target LOA or 2017 Achieved % LOA Target Achieved (nominal) Percentage point changes (see note 5) Endlne Result Statistically Significant? REVISED for ENDLINE Male - Control over and use of income 54.2% ? 83.0% NA 28.8% points = 53% increase REVISED for ENDLINE Female - Group membership 63.8% ? 25.9% NA 37.9 % point reduction Decrease / Decline Statistically Significant Male - Group membership 65.9% ? 30.4% NA 35.5 % point reduction Note 1: OP = Output; OC = Outcome, IM = Impact and CI = Contextual, NA = Not Available, Note 2: Membership composition in groups are; Farmer Training Groups were formed with 25 members, Honey Groups with 20 members, Goats groups and MCG with 10 - 15. Note 3: Where possible census data were used to estimate targets of project beneficiaries Note 4: Initial targets provided by ACDI-VOCA based on national targets and/or feasible results based on prior experience. Actual target determined and agreed upon with USAID Note 5: Indicators source renamed to RWANU for some FFP or FTF indicators since the reporting frequency changes to annual as required by RWANU and not FTF or FFP Note 6: Percentage point changes - important esp. where % of achieved is calculated w/ 0 as start point and/or is described for % targets, and where results worse than baseline Color coding: reversed <60 % achieved 60-79% achieved 80-99% achieved 100-120% achieved > 120% achieved U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523