vvvv Final Evaluation: Burkina Families Achieving Sustainable Outcomes (FASO) No. AID-FFP-10-00013 Prepared for United States Agency for International Development and Catholic Relief Services August 2017 Evaluation Team: Brian Sage, Kathy Tilford, Batamaka Somé, CEMES-Afrique Remote Support and Technical Assistance: Bryan Crawford-Garrett and Oxu Support Staff FASO Final Evaluation i Table of Contents Table of Contents........................................................................................................................i Acknowledgments.................................................................................................................... iii List of Tables ............................................................................................................................iv List of Figures...........................................................................................................................iv List of Annexes..........................................................................................................................v List of Acronyms ......................................................................................................................vi Executive Summary................................................................................................................... I 1. Introduction ........................................................................................................................1 2. Overview of the Programmatic Context and Implementation............................................2 2.1. Context at FASO’s Inception ......................................................................................2 2.2. Contextual Challenges During Implementation..........................................................3 2.3. FASO Program Summary ...........................................................................................4 3. Summary of Evaluation’s Scope of Work..........................................................................5 4. Evaluation Methodology ....................................................................................................6 4.1. Methods.......................................................................................................................7 4.2. Constraints and Limitations ......................................................................................10 5. Baseline/Endline Data ......................................................................................................13 6. Findings, Conclusions and Recommendations by Sector.................................................20 6.1. SO1: Agriculture .......................................................................................................21 6.2. SO2: Health/Nutrition and WASH............................................................................34 6.3. SO3: Governance ......................................................................................................44 6.4 Cross-Cutting Recommendations..............................................................................60 7. Overall Findings, Conclusions and Recommendations....................................................60 7.1. Assessment of FASO Evaluation Main Objectives....................................................61 7.2. Review of Evaluation Questions...............................................................................63 7.3. Review of Specific Objectives..................................................................................71 7.4. Overall Recommendations........................................................................................72 FASO Final Evaluation ii Cover photos Left: Women in a Care Group who participated in a final evaluation FGD. (Photo by Djamila Sanfo, Bartiebougou, 10 February 2017) Right: Young girl all dressed up as her mother participates in a FGD with a GASPA group. (Photo by Djamila Sanfo, Koulbila, 6 February 2017) FASO Final Evaluation iii Acknowledgments Oxu Solutions would like to acknowledge the many people who made this work possible: - Beneficiaries, government administrative and technical representatives, past and present consortium members, and community leaders who agreed to be interviewed or participate in focus groups; - Community members who took part in the quantitative survey; - Translators: Etienne Lankoandé, David Noali and Djamila Sanfo; - Drivers: Saïdou Ouedraogo and Francisse Nikiema; - Chief of Party Adjavon Vewonyi, who ensured that the qualitative evaluation team had the best possible working conditions and support, along with his colleagues Fabris Compaore, MEAL Coordinator, and Edouard Nonguierma, Deputy Chief of Party; - FASO Regional Coordinators Hervé Koné and Amidou Idani, who organized the qualitative field work schedules in the districts; - The FASO teams in the three health districts who set up the focus group discussions and key informant interviews and coordinated daily schedules; and - Aisseta Boundaogo, who arranged logistics for the qualitative evaluation team and facilitated our arrival. Finally, Oxu takes full responsibility for the content of this work. Any errors or misrepresentations in the document are our own. FASO Final Evaluation iv List of Tables Table 1: Local structures Table 2: Consortium members’ geographic and thematic coverage (first phase) Table 3: FASO program goal & SOs Table 4: M&E and research activities Table 5: Summary of agriculture and health & nutrition surveys planned and completed Table 6: Sampling criteria for qualitative fieldwork Table 7: SO1 baseline-endline survey data Table 8: IR1.1 baseline-endline survey data for IR and select output indicators Table 9: IR1.2 baseline-endline survey data for IR and select output indicators Table 10: SO2 baseline-endline survey data Table 11: IR2.1 baseline-endline survey data for IR and select output indicators Table 12: IR2.2 baseline-endline survey data for IR and select output indicators Table 13: SO3 baseline-endline survey data Table 14: Summary of FASO SO1 design Table 15: SO1 indicators Table 16: Beneficiaries reached through agricultural activities (disaggregated by gender) Table 17: IR1.1 indicators Table 18: IR1.2 indicators Table 19: Baseline-Endline Data for Indicator 1.2.1 disaggregated by gender Table 21: Summary of FASO SO2 design Table 22: SO2 indicators Table 23: Stunted and underweight children under 2 years of age Table 24: IR2.1 indicators Table 25: IR2.2 indicators Table 25: Summary of FASO SO3 design Table 26: SO3 indicators Table 27: IR3.2 indicators Table 28: IR3.1 indicators Table 29: IR3.4 indicators Table 30: IR3.3 indicators Table 31: IR3.5 indicators List of Figures Figure 1: Production trend of Cowpea, sesame, and rice from baseline through year 5 Figure 2: Percentage of Targeted Beneficiaries who plant rice Figure 3: The centrality of SILC, income-generating activities and productive activities, and agricultural production Figure 4: Percentage of targeted beneficiaries shell rice using plastic instruments FASO Final Evaluation v List of Annexes Annex A: Scope of Work for Final Evaluation Annex B.1: FASO Final Evaluation Plan Annex B.2: Quantitative Data Analysis and Treatment Plan for Final Evaluation Annex B.3: Updated Qualitative Methodology for FASO Final Evaluation Annex C: FASO Final Evaluation Team Annex D.1: FASO Results Framework January 2016 Annex D.2: FASO Results Framework History Annex D.3: FASO Results Framework FY11-15-16 Comparison Annex D.4: Shifts in FASO SO3 Results Framework Annex E: List of Documents and Information Consulted by Oxu for FASO Final Evaluation Annex F: FASO Qualitative Data Collection Tools Annex G: Final Evaluation Calendar Annex H: Final Evaluation Qualitative Fieldwork Achievements Annex I.1: Output-level findings for SO1 Annex I.2: Output-level findings for SO2 Annex I.3: Output-level findings for SO3 Annex J: FASO Program Timeline Annex K: Local Structures Annex L: IPTT Updated with Endline Survey Data Annex M: Endline Survey Report and its Annexes FASO Final Evaluation vi List of Acronyms Acronym Description ACDI/VOCA Agricultural Cooperative Development International / Volunteers in Overseas Cooperative Assistance ANJE Alimentation du Nourrisson et du Jeune Enfant (French for: IYCF - Infant and young child feeding) AQIM Al-Qaeda in the Islamic Maghreb ATT Association Tin Tua ARR Annual Results Reports CBO Community-Based Organization CEMES Centre d’Etudes et de Management Economique et Social - Afrique CHW Community Health Worker CLTS Community-Led Total Sanitation CMO Chief Medical Officer (English translation for: MCD - Médecin Chef de District) COGES1 Comité de Gestion (Farmers’ Group Management Committee) CRS Catholic Relief Services EA Enumeration Area EGSSA Environmental Guidelines for Small-Scale Activities in Africa ENIAM Etude Nationale sur l’Insécurité Alimentaire et la Malnutrition (National Study on Food Security and Malnutrition) FANTA Food and Nutrition Technical Assistance FAO Food and Agriculture Organization of the United Nations FASO Families Achieving Sustainable Outcomes FFP Food for Peace FFW Food for Work FGD Focus Group Discussion FY Fiscal Year GASPA Groupe d’Apprentissage et de Suivi des Pratiques Optimales d’ANJE (Learning and Monitoring Group for Optimal Infant and Young Child Feeding Practices) GoBF Government of Burkina Faso GRET Groupe de Recherche et d’Echanges Technologiques (Research and Technical Exchange Group) DREA Direction régionale de l’eau et de l’assainssement Regional Directorate of Water and Sanitation HA Hectare HH HHs HKI Helen Keller International IEE Initial Environmental Examination IGA Income-Generating Activity IPTT Indicator Performance Tracking Table IR Intermediate Result IYCF Infant and Young Child Feeding (English translation for: ANJE - Alimentation du Nourrisson et du Jeune Enfant ) 1 This body should not be confused with the official school management committees at village level, also called COGES (Comité de Gestion). In this report, the acronym COGES will only be used to designate farmers’ group management committees. FASO Final Evaluation vii KII Key Informant Interview LOA Life of Activity M&E Monitoring and Evaluation MA Mothers Association MEAL Monitoring, Evaluation, Accountability and Learning MFI Micro Finance Institution MIYCN Maternal, Infant and Young Child Nutrition ML Mother Leader MoA Ministry of Agriculture MoH Ministry of Health MT Metric tons MTE Mid-Term Evaluation MWS Ministry of Water and Sanitation MYAP Multi-Year Assistance Program NGO Non-governmental organization OCADES Organisation Catholique pour le Développement et la Solidarité (Catholic Organization for Development and Solidarity) ODK Open Data Kit PLW Pregnant and Lactating Women PM2A Preventing Malnutrition in Children Under 2 Approach PREP Pipeline and Resource Estimate Proposal PSP Private Sector Provider PTA Parent Teachers Association (English translation for APE – Association des Parents d’Élèves) REGIS-AG Resilience and Economic Growth in the Sahel – Accelerated Growth REGIS-ER Resilience and Economic Growth in the Sahel – Enhanced Resilience RISE Resilience in the Sahel Enhanced RUTF Ready-to-Use Therapeutic Foods SAREL Sahel Resiliency Learning SBCC Social and Behavior Change Communication SILC Savings and Internal Lending Communities SO Strategic Objective SPRING Strengthening Partnerships, Results, and Innovation in Nutrition Globally SPSS Statistical Package for the Social Sciences TMC Tool Management Committee UN United Nations UNFPA UN Population Fund USAID United States Agency for International Development VDC Village Development Committee or Council ViM Victory against Malnutrition VPEA Village Poultry Extension Agent WASH Water, Sanitation and Hygiene WRA Women of Reproductive Age WUA Water User Association (English translation for AUE - Association d’Usagers d’Eau) FASO Final Evaluation I Executive Summary Introduction. This report presents the findings, conclusions, lessons learned, and recommendations of the final evaluation of the Burkina Families Achieving Sustainable Outcomes (FASO) Multi-Year Assistance Program, a United States Agency for International Development (USAID) /Food for Peace (FFP) Title II food security program. FASO is being implemented from June 2010 through September 2018 by a consortium led by Catholic Relief Services (CRS). Current consortium members include the Tin Tua Association (ATT) and the Organisation Catholique pour le Développement et la Solidarité (OCADES). In addition, Helen Keller International (HKI) and Groupe de Recherche et d’ Échanges Technologiques (GRET) were part of the CRS-led consortium during FASO’s first five years (first phase). The goal of the FASO program is to sustainably reduce vulnerability to food insecurity in the Tougouri, Manni, and Gayéri health districts in Burkina Faso; this goal is supported by three Strategic Objectives (SOs): - SO1: 56,126 households have improved access to food of sufficient quantity and quality throughout the year; - SO2: 51,426 mother/child units have improved health and nutrition; - SO3: Community-based organizations (CBOs) in at least 100 villages are using principles of good governance to make strategic decisions regarding sustainability of community investments and food insecurity reduction. The overall purpose of the final evaluation was to determine the outcomes and effectiveness of the FASO program using a mixed-methods approach that included a quantitative endline survey and qualitative fieldwork. Accordingly, this final evaluation analyzes: the SOs and Intermediate Results (IRs) based on the pre- post comparison of outcome and impact level indicators at population levels; the effectiveness of the use of Title II food commodities in achieving the intended results; and the stakeholders’ satisfaction with the program’s quality and the implementation processes and how these processes supported or inhibited program performance. Overall Conclusions. The evaluation team identified findings, conclusions and lessons learned, and recommendations for programmatic sectors organized by SO, as well as for cross-cutting themes and for the program as a whole. The full narrative report and annexes provide the sector-level details, including specific findings and conclusions for each SO and cross-cutting theme. The key overall conclusions can be summarized as follows: 1. First and foremost, FASO is a successful program that has made important contributions to the program goal of sustainably reducing vulnerability to food insecurity in the program’s three targeted health districts. These contributions can be seen at the individual, household, and community levels for targeted beneficiaries and were consistently validated during the qualitative fieldwork. Moreover, FASO has consistently met annual indicator targets for program beneficiaries (as measured through annual surveys). For instance, for the final year of data that is available for SO and IR-level indicators, FASO reached at least 90 percent of the target for 31 of 35 indicators. In a number of cases, the program greatly exceeded the established targets (some of which had even been revised upward). For population-level measures, while significant difference testing was not able to be done conclusively, there were sizable differences FASO Final Evaluation II noted between baseline and endline values for several indicators, especially in SO2. These findings, combined with the qualitative data, suggest that the program has contributed to certain important improvements in program beneficiaries and at population-level alike.2 2. Title II food commodities were used effectively to advance the program’s intended results. In SO1 Food for Work enabled beneficiaries to reclaim degraded lands, an activity that continued in many places even when the program stopped providing food. In SO2, qualitative data indicated that food was helpful in promoting participation in Care Groups, use of health services, and a more diverse diet where commodities were distributed. In SO3, food also seemed to encourage girls’ participation in school but it is not known if these gains will be sustained with the transition from Title II food to locally-sourced food. 3. There was strong satisfaction among beneficiaries and institutional stakeholders with program performance and implementation processes were solid and well-executed. Program beneficiaries and key stakeholders interviewed by the evaluation team highlighted many FASO achievements. These include, but are not limited to: adoption of promoted agricultural and land improvement techniques; the establishment of the Savings and Internal Lending Communities (SILC) groups; changes in nutrition-related behaviors and practices; clean water provision and latrine construction; and strengthening of several community-based structures such as Water User Associations (WUAs) and school-based Mothers Associations (MAs). In addition, although FASO lacked a comprehensive Social and Behavior Change Communication (SBCC) strategy, the program effectively promoted behavior change to consolidate achievements across all three SOs. 4. FASO was participatory, adaptable and responsive to local community needs. The program responded to many of the most important food security-related needs of the beneficiary populations, which included, in particular, supporting enhanced agricultural production and marketing; access to some level of savings and credit via SILC; water, sanitation and hygiene (WASH); and Infant and Young Child Feeding (IYCF) practices, among others. In addition, FASO employed a more sustainable approach by working with government services and several local structures, either pre-existing or established for the purpose of the program – and as such enjoyed strong support at the community level, including from community leaders, and at the administrative level from mayors, government representatives and other key stakeholders. FASO was also a program that, on the whole, adapted as the situation evolved, as seen for instance through the introduction of several new SO1 activities during program implementation (market gardening, warrantage, and poultry-raising) and an increased emphasis on WASH as the needs become more apparent. The flexibility of the donor helped facilitate these program adaptations. 5. The program design for SO2 was logically sound, appropriate and relevant, SO1 was imperfect but solid, whereas SO3 had major design flaws which did not match the quality of program implementation for SO3 activities. The evaluation team found that SO2 was well-designed, contextually appropriate, and not overly ambitious given the program timeframe and local needs. SO1 was fairly strong but there were questions related to crop selection and the fact that greater emphasis was placed on production (and corresponding lesser emphasis on marketing). With SO3, however, there were numerous design flaws and shifts at the 2 As noted in the main narrative in Section 5: Baseline/endline comparison (which presents the baseline-final survey quantitative findings), any causality or attribution from program implementation and final outcomes at the population level are limited to plausible or possible contributions given that there were no comparison or control groups. FASO Final Evaluation III SO and IR levels. Even with modifications made during the program, SO3 did not have a tight design, which made it extremely difficult to assess the degree to which FASO fulfilled what was planned. The interventions implemented under SO3 did, however, appear quite effective in contributing to the sustainability of a number of SO1 and SO2 results. 6. FASO was very strong in promoting and ensuring female participation across all program activities. However, there was not an intentional strategy aimed at going beyond participation to address gender inequities more broadly. FASO was successful in including women as priority beneficiaries in numerous interventions, yet there was very little done to transform gender relations and promote women’s leadership through more in-depth and deliberate work. 7. The program deliberately and successfully designed integration across the main program components, targeting the same households for activities among the different SOs. This multiplied opportunities and benefits for these households and strengthened the overall positive impact of the program. There was evidence of this throughout the qualitative field work and in the FASO program documents. 8. Challenges related to improving dietary diversity persisted throughout the program period. Consumption of micronutrient-rich food remains a challenge for improving dietary diversity and nutrition status, especially for women and children. Although this was less of a problem during the first phase in Gayéri and Tougouri when women and young children were consuming Title II food commodities, FASO beneficiaries spoke frequently during the fieldwork of the unavailability of nutrient-rich foods during certain periods of the year. The very slight but statistically significant baseline-endline decline in levels of household dietary diversity observed at the population level further supports this point. While there were reported increases in revenue from agricultural activities when comparing baseline and endline values and FASO did promote market gardening in SO1 and some SO1-SO2 linkages related to dietary diversity, the program still struggled overall to succeed in facilitating greater access to a diverse diet in the target areas. 9. Across all three SOs, there seem to be sustainable program achievements. The tools developed for the qualitative fieldwork included a number of questions related to sustainability and what will continue after the program ends. Across the board, respondents were adamant that the positive impact of the program would endure, although positive statements from beneficiaries and stakeholders are not necessarily guarantees of sustainability. Obstacles to sustainability do remain, as FASO lacked a concrete sustainability plan to guide program implementation efforts towards promoting sustainability and post-program ownership. However, the adaptations made during the extension period to focus on building the capacity of community structures to sustain program achievements show promise. Four Principal Recommendations. The main report presents recommendations for each SO, the cross￾cutting themes and program management. The four overarching recommendations are summarized below: 1. Introduce more intentional and results-based planning in the areas of: program learning; gender; SBCC; and sustainability The evaluation team understands that the FASO program was designed before FFP required a learning agenda, a gender analysis and strategy, an SBCC strategy based on formative research, and a sustainability and exit strategy. Future food security programs need to have these strategic documents in place – whether FASO Final Evaluation IV they are required by a donor or not – as well as a timetable for their periodic review and revision as necessary. 2. Ensure all elements of the program are well-designed and fit within a clearly-articulated theory of change that explains the desired outcomes and how the various program components contribute to those outcomes. The program staff indicated to the evaluation team that they know that a theory of change will be required as part of new FFP proposal requirements. For future programs, we suggest developing a clear theory of change for each SO and linking these to a consolidated overall theory of change for the program. Each IR or core intervention area should have a documented strategy and activities should have an implementation guide to ensure harmonization of approach across geographic zones. During a program’s implementation period, the theory of change should be reviewed as part of an annual review process (prior to the annual planning process) to assess progress not only against indicators but against the full logic of the program design. This will help avoid some of the inconsistencies of programming design and implementation that were seen in FASO. For future programs these documents can also help identify knowledge gaps to plan operational research and serve as the basis of a program’s learning plan. In the meantime, we also recommend that the current FASO program (not as a formal submission to USAID, but as an internal exercise) take advantage of the extension period to consolidate the learning to date by developing an updated theory of change for the current program or at least document the logic of how activities with direct project beneficiaries will lead to changes in their attitudes and/or behaviors and how those will create changes in the broader community. This will help the team revisit the current logic of interventions, which has not been well-documented, and discuss and recognize gaps that can be addressed in future programming. 3. Aim higher with respect to transforming gender perceptions and relations. FASO has made some significant achievements in engaging women in program activities across the SOs. Future programming should articulate through a gender strategy the ultimate desired change for female beneficiaries (distinguishing among different sub-groups of women [adolescent, elderly, etc.] as needed) and map out how that change should occur within the broader program design. The desired outcomes need to be ambitious yet realistic given the extent of gender inequity in the given context. While the evaluation identified a number of successes with regard to women’s engagement in the program, this is not a substitute for a plan that includes elements such as: a deliberate learning process; gender training for staff (and key government counterparts if feasible); dedicated staff to ensure that gender considerations and learning are fully integrated into the overall program; and concrete strategies for involving men and women in activities with an aim towards improving gender relations and promoting greater gender equity. 4. Incorporate best practices for promoting agriculture-nutrition linkages, especially as a means to improve dietary diversity of women and children beneficiaries. Future food security programs need to ensure that beneficiaries have access to the key ingredients for a diverse diet that contains the essential nutrients, especially for Pregnant and Lactating Women (PLW) and young children. Where year-round food availability is the issue, solutions could include promoting the following: production of nutritious crops for household consumption; household gardening on a wider scale; and/or improved food conservation and transformation measures. Where access (income) is an issue, Income-Generating Activities (IGAs) for women, promoting crops for marketing, and encouraging household discussions around purchasing more nutritious foods are possible solutions. These issues also confirm the need to ensure consideration of nutrition and dietary diversity when selecting food products to promote for agricultural or livestock production activities. The FASO program did not seem to clearly FASO Final Evaluation V articulate how each targeted crop would contribute to improved household nutrition, in terms of crop promotion for specific outcomes related to the more direct nutrition benefits that can be obtained from consumption of household production and the indirect benefits that can accrue through purchasing more nutritious food as a result of increased sales of targeted crops. While the FASO proposal stated that the program would focus on seven “dual purpose” (for household consumption and for sales) value chains, the program design did not specify how promotion of each of these seven crops would explicitly contribute to improved nutritional outcomes for targeted beneficiaries or clearly articulate how promoting these crops would address specific nutrient deficiencies in the local population. For instance, program activities for crops designed to improve nutritional status primarily through sales (and consequently income) would require training in marketing, and potentially work to develop those markets, whereas targeting crops designed to address nutritional gaps through consumption would have different interventions depending on whether the project wanted to increase broad availability (and possibly reduce cost) through the market or via own-consumption of the farmers’ own production. For example, increasing millet consumption without increasing sales would be unlikely to have nutritional gains if net caloric intake is not a key challenge identified in targeted geographic areas. As part of the development of theories of change noted above, it is therefore important for future programs to plan out these linkages, determine appropriate measurement, and monitor the progress toward the intended results so that any necessary adjustments can be made accordingly. Final Conclusion. FASO is a well-managed food security program that has responded to local needs, has generally been adaptable to the context, and has been most successful in changing health and nutrition￾related attitudes and behavior. Moreover, it is clear that FASO has made important and a number of measurable differences in the lives of benefiting individuals, households, and communities. There are also several key lessons learned based on the FASO experience that can be incorporated into future food security programs in Burkina Faso and other similar contexts. It is the evaluation team’s hope that this exercise both provides an objective assessment of the FASO program as well as documents concrete and meaningful ways for improving these types of programs in the future. FASO Final Evaluation 1 1. Introduction This report details the findings, conclusions, lessons learned, and recommendations of the final evaluation of the Burkina Families Achieving Sustainable Outcomes (FASO) Multi Year Assistance Program (MYAP), a United States Agency for International Development (USAID)/Food for Peace (FFP) Title II food security program. FASO is being implemented from June 2010 through September 2018. The report has been written for USAID and the program consortium led by Catholic Relief Services (CRS) to evaluate the performance of the FASO program. This final evaluation analyzes performance against the Strategic Objectives (SOs) and Intermediate Results (IRs); the effectiveness of the use of food commodities in achieving the intended results; and the stakeholders’ satisfaction with the program’s quality and the implementation processes (including human resource management) and how these processes supported or inhibited program performance. The report consists of seven sections. After this Introduction, Section 2 describes the Burkina Faso context and summarizes the FASO program and its implementation. Section 3 includes a synopsis of the evaluation purpose. Section 4 describes the mixed-methods methodology that was used to conduct the evaluation and the key constraints and limitations of the evaluation. Section 5 presents a comparison of FASO’s indicators that were measured during the baseline and the endline survey. Section 6 provides findings, conclusions and lessons learned, and recommendations for each of the program’s three SOs and cross-cutting themes. Section 7 then outlines the evaluation team’s overarching findings, conclusions and recommendations. The recommendations in Sections 6 and 7 are relevant to the program and context and provide concrete and realistic steps for programming in similar contexts for future implementing partners of CRS and USAID. FASO worked with and through several local structures, some of which are part of the formal government administration, while others were established by FASO. The table below provides a brief orientation to the different groups with which the program worked (in alphabetical order). More detailed descriptions of each group can be found in Annex K: Local Structures. Table 1: Local structures Group Description Care Groups Groups composed of pregnant women and mothers of young children. Established by FASO. COGES – Farmers’ Group Management Committees (Comité de Gestion) Farmers groups that manage community agriculture infrastructure. Established by FASO. GASPA - Learning and Monitoring Groups for Optimal IYCF [Infant and Young Child Feeding] practices (Groupes d’Apprentissage et de Suivi des Pratiques Optimales d’ANJE [Alimentation du Nourrisson et du Jeune Enfant]) Group model introduced by the Ministry of Health and piloted by FASO in one commune to promote positive behavior changes in maternal, infant, and young child nutrition. Established by FASO. MAs - Mothers Associations Committees that oversee school feeding and hygiene at school. They exist within the MOE guidelines. PTAs - Parent Teacher Associations These are just Parents’ Associations in this context. They oversee activities related to curriculum, attendance, school fees, and the FASO Final Evaluation 2 management of the school canteen, including the mobilization of parents’ contributions of food to the canteen. SILC - Savings and Internal Lending Communities User-owned, self-managed savings and credit groups. Established by FASO (CRS model). TMCs - Tool Management Committees Village committees that manage agricultural tools and equipment provided by FASO. Established by FASO. WUAs - Water User Associations Government-mandated structures that manage local drinking water sources. They report to the commune. VDCs - Village Development Committees or Councils Smallest formal administrative level (the village). They report to the commune. 2. Overview of the Programmatic Context and Implementation 2.1. Context at FASO’s Inception The goal of the FASO program is to sustainably reduce vulnerability to food insecurity in some of the most chronically food insecure regions. Although 80 percent of the nation’s population relies on subsistence agricultural production, at the program inception 35 percent of households were not self￾sufficient in cereal production,3 with children under five being the most vulnerable to food insecurity: 39 percent of children under 5 were stunted, and 12 percent wasted, surpassing emergency thresholds of 10 percent. 4 Only 6 percent of infants nationally were exclusively breastfed, only 43 percent of children 6 to 23 months had minimum dietary diversity,5 and only 43.9 percent were fully vaccinated.6 A total of 64 percent of women reported inadequate access to health care7 and, while 60.2 percent of the population had access to potable water, less than 10 percent had access to sanitation.8 CRS used secondary data and field assessment findings to target the health districts of Boulsa/North (now Tougouri), Manni and Gayéri as the Burkina FASO program implementation zone for agriculture and nutrition interventions.9 These areas had a stunting rate for children under five of over 40 percent (weight for height <-2 Z score), 10 a food coverage rate of less than 75 percent for at least two of the last five years, 11 a poverty rate of over 50 percent, 12 an incidence of diarrhea rate in children under five of over 20 percent, 13 and more than three occurrences of drought or flooding in the last five years.14 3 Evolution in the Agricultural Sector and Quality of Life of HHs, Ministry of Agriculture, 2009. 4 Enquête Nationale sur l’Insécurité Alimentaire et la Malnutrition (ENIAM), Ministry of Agriculture, Water Resources and Fisheries, 2009. 5 ENIAM, 2009. 6 Demographic and Health Survey, 2003. 7 Annual Health Yearbook, 2007. 8 National Project for Water Supply and Sanitation, 2005. 9 Multi-Year Assistance Program - MYAP Proposal, Families Achieving Sustainable Outcomes - The Burkina FASO Program, January 15, 2010, pp. 3-4. 10 ENIAM, 2009. 11 DGPER/CILSS, using data from 2003-2007. 12 Defined as percentage of people living on less than 85,000 CFA per year, or approximately 170 dollars (National Institute of Demographic Statistics, 2004). 13 DHS 2003. 14 DGPER/CONASUR/CRS. FASO Final Evaluation 3 2.2.Contextual Challenges During Implementation The FASO program reported several key contextual challenges during implementation: 1. Political instab ility. In October 2014, Burkina Faso’s stable regime of 27 years collapsed when Blaise Compaoré fell from power. FASO was impacted by the political insecurity that followed, as 800 metric tons of food commodities designated for program activities were looted after the October 2014 insurrection.15 Furthermore, the general and presidential elections in 2016-17 disrupted the regular functioning of a number of government institutions. According to program documentation this led to a significant reduction in the presence of government-subsidized inputs and agro-dealer distribution points and rescheduling of some governance activities as municipalities struggled to function during the political transition and before the election of new municipal leaders.16 17 2. Sahel food crisis. Nearly one-fourth of Burkina Faso’s population was estimated to be undernourished because of the regional food crisis that engulfed much of the Sahel and started in in 2012.18 3. National capabilities negatively impacted FASO. Health infrastructure throughout the program areas was weaker than expected and focused on a recuperative rather than preventive approach toward malnutrition, the latter being the one promoted by FASO. The capacity of the seed certification system of GoBF, which aimed to improve the quality of seed, was also limited due to the small number of inspectors. Additionally, during the program period GoBF introduced a policy establishing certain requirements to be a certified seed producer, in part to make it easier for seed inspectors to fulfill their obligations. According to the FASO mid-term evaluation (MTE) report, this shift created a challenge for FASO and program farmers in being able to meet the new obligations.19 4. Attr ition of beneficiaries and staff. Events that led to loss of beneficiaries participating in the program were due to floods or drought in lowlands areas in Fiscal Year (FY) 2016, coupled with alternative economic opportunities in the recent gold boom that attracted beneficiaries away from agricultural activities. Loss of staff was due to the opening of new programs that hired them away. 5. Differences am ong the three health distr icts. Gayéri faced more security threats, as seen by the loss of one consortium staff member from Helen Keller International (HKI) in 2012 in an attack.20 Gayéri is also a less developed area in terms of basic services infrastructure, which included fewer schools and no community warehouses for food storage. FASO adapted by building new community warehouses to store commodities. Despite these constraints, Gayéri has a smaller population and is less densely populated than the other districts, therefore, land is less scarce. In addition, there also appear to be socio-cultural differences in all three districts, e.g., agricultural practices and women’s customary access to land. 21 15 FASO 2015 Annual Results Report 16 FASO 2016 Annual Results Report 17 Additionally, the late approval and disbursement of funds for the extension period caused delays and high staff turnover for implementing partners as some staff took jobs elsewhere due to the uncertainty of the program’s possible extension. 18 http://www.fao.org/docrep/field/009/i3760e/i3760e.pdf 19 FASO Mid-Term Evaluation Report, 2013 20 From the FY2012 AAR: “The district of Gayeri experienced an increased level of banditry on the roads. One of HKI animators was killed by armed gangs causing a temporary suspension of program activities in the area”, p. 17. 21 Women’s Access to Land, Markets and Local Institutions in East and Central-North Burkina Faso, June 2011. FASO Final Evaluation 4 2.3. FASO Program Summary FASO is a $53.3 million USAID/FFP-financed MYAP led by CRS that has aimed to improve food security for approximately 300,000 people in three targeted Health Districts – Manni, Gayéri, and northern Boulsa (now Tougouri)22 – in the Eastern and North-Central Regions of Burkina Faso. Initially awarded in 2010 as a five-year program, FASO received a two-year extension from June 2015 to September 201723 and expects to receive another extension until September 2018. The program has included sub-awards to Helen Keller International (HKI), Groupe de Recherche et d’Échanges Technologiques (GRET), Organisation Catholique pour le Développement et la Solidarité (OCADES-Kaya) and Association Tin Tua (ATT). During the first five years, each organization was responsible for implementing various components of the program as per the table below: Table 2: Consortium members’ geographic and thematic coverage (first phase) District† Health, Nutrition, and Water, Sanitation and Hygiene (WASH) Agriculture Manni GRET ATT Gayéri HKI ATT Tougouri OCADES-Kaya OCADES-Kaya †As of the extension period, the consortium no longer includes HKI and GRET, and therefore ATT also has led implementation for health, nutrition and WASH in Manni and Gayéri. The table below summarizes the program’s goal and three objectives: Table 3: FASO program goal & SOs Goal: Vulnerability to food insecurity is sustainably reduced in the Health Districts of Boulsa-North, Manni and Gayéri (North Central and Eastern Burkina Faso) SO1: 56,126 HHs have improved access to food of sufficient quantity and quality throughout the year SO2: 51,426 mother/child units have improved health and nutrition SO3:24 Community-based organizations in at least 100 villages are using principles of good governance to make strategic decisions regarding sustainability of community investments and food insecurity reduction. To achieve FASO program objectives, the program has worked in approximately 250 villages across 10 communes in the three targeted health districts. SO1 activities have focused on sustainable improvement of household agricultural production and increasing beneficiary farmer revenues; SO2 activities have promoted health, nutrition, and hygiene practices by mothers of children aged 0-23 months and pregnant women, coupled with improving access to quality nutrition and health services for women and children aged 0-59 months; SO3 activities have focused on ensuring local community organizations are better able to manage community resources and community development activities, that women occupy leadership roles in their communities, community access to service providers is improved, and promoted girls education and local provision of school meals by community groups. 22 In the early stages of FASO, the third health district was known as "northern Boulsa," referring to the northern part of Boulsa Health District. Later, this became a separate health district known as “Tougouri.” This report will therefore use Tougouri throughout the document and this corresponds to what was previously known as “northern Boulsa.” 23 Note that this includes $12,976,393 of additional resources approved to cover the two-year extension period from June 2015 to September 2017. 24 During the initial years of implementation, SO3 was formulated as “800 community structures have improved their local governance practice.” Following the program’s MTE, SO3 was changed in FY15 to be: “Community-based organizations in 100 villages are using principles of good governance in implementing community development plans for reducing food insecurity,” and this was modified slightly for FY16 through the end of FASO. FASO Final Evaluation 5 Per the evaluation scope of work, the extension period was intended to help align the FASO program to USAID’s Resilience in the Sahel Enhanced (RISE) initiative and create conditions for greater linkage with the Resilience and Economic Growth in the Sahel – Enhanced Resilience (REGIS-ER) and REGIS￾Accelerated Growth (REGIS-AG) projects. The consortium undertook a number of Monitoring and Evaluation (M&E) and research initiatives: Tab le 4: M&E an d resear ch activities Period Activity Purpose 2010 Population-based baseline survey Provide reference data for programt impact, outcome, and key output indicators for the three FASO program technical sectors (agriculture, health and nutrition, and governance) 2011 Gender study on women’s access to land and markets Strengthen FASO’s gender approach. 2013 MTE Qualitative evaluation to review implementation progress, focusing on recommendations to improve program implementation in the remaining years of FASO 2013 Nutrition barrier analysis Identify barriers to key nutrition-related behaviors promoted by FASO 2015 Formative evaluation of FASO and Victory Against Malnutrition (ViM)25 Inform the extension of the FASO program through September 2017 2016 Agriculture barrier analysis Qualitative research to look at barriers to exploiting lowlands and renting agricultural equipment from program-established TMCs. CRS led FASO’s Monitoring, Evaluation, Accountability and Learning (MEAL) activities. CRS monitored program progress towards achieving the annual indicator targets through consortium members’ quarterly reports, annual surveys and data collection results from the health districts. Key indicators were tracked in the Indicator Performance Tracking Table (IPTT). Annex L: IPTT Updated with Endline Survey Data presents the most recent IPTT and integrates data from the baseline and endline surveys. 3. Summary of Evaluation’s Scope of Work As articulated in the evaluation’s Scope of Work (see Annex A: Scope of Work for Final Evaluation) and Evaluation Plan (Annex B.1: FASO Final Evaluation Plan), the overall goal of the final evaluation is to determine the outcomes and effectiveness of the FASO program using a mixed-methods approach. As such, the evaluation will allow for: an independent assessment of program achievements against expected results outlined in the Results Framework; determining the relevance, effectiveness, efficiency, impact and sustainability of activities; measurement of outcomes, including FASO contributions to those outcomes; identification of lessons learned; and formulation of conclusions and recommendations for future program design and management. The final evaluation of the FASO program focused on three main objectives: 25 ViM project is led by Agricultural Cooperative Development International / Volunteers in Overseas Cooperative Assistance (ACDI/VOCA) and aims to reduce food insecurity affecting vulnerable populations in Sanmatenga Province by improving farmers’ incomes and household health and nutrition, especially among pregnant and lactating women and children under five years old. It is also funded by USAID/FFP. FASO Final Evaluation 6 1. To evaluate the performance of the FASO program’s SOs and IRs based on the pre- post comparison of outcome and impact level indicators and explore the lessons learned. 2. To evaluate the effectiveness of the use of food commodities in achievement of the intended results. 3. To evaluate stakeholders’ satisfaction with the program’s quality and effectiveness of implementation processes (including human resource management); to determine how these processes supported or inhibited program performance. In addition, seven specific objectives and 10 key questions were defined by CRS and approved by USAID. These focused on: - Baseline-endline comparisons for key outcome and impact indicators; - Effectiveness of program activities in contributing to achievement of planned outcomes; - Major factors influencing achievement or non-achievement of planned outcomes; - Lessons learned from program interventions and management approaches that can inform future program design, strategy, and implementation; - Extent to which gender was addressed and integrated into the program, along with women’s level of participation; - Degree to which local community-based organizations have been supported (through technical assistance, capacity building, etc.) and are able to sustain and manage program investments; - Changes in individual and group capacities; - Level of integration of the core program components; - Appropriateness and relevance of program design based on the context; - Effectiveness of the food commodities and other subsidies the program provided; - Efficient implementation of program activities (time, resources, etc.); and - Any unintended (positive or negative) consequences of FASO. 4. Evaluation Methodology The evaluation methodology was presented in an Evaluation Plan package submitted in October 2016. In the Evaluation Plan, the evaluation team also provided a framework for guiding the evaluation as it relates to the program design and the Results Framework (see Annex B.1: FASO Final Evaluation Plan). The evaluation framework helped ensure that the final evaluation would be closely tied to the logic of the program design showing how quantitative and qualitative methods are used to address the high-level components of the Results Framework and corresponding program indicators. The evaluation was carried out from October 2016 to March 2017 (see Annex G: Final Evaluation Fieldwork Calendar ). The evaluation included two main phases of work: i) A quantitative population-based household endline survey, conducted in November-December 2016. The endline survey was conducted to provide values for the key indicators at the end of the program and to be able to compare results with baseline information; and ii) Qualitative data collection took place in January-February 2017, and helped to identify factors that contributed to or constrained the adoption of best practices promoted by the FASO program. Tools were informed by a desk review of program documentation and relevant secondary sources. FASO Final Evaluation 7 4.1. Methods 4.1.1. Desk Review Prior to conducting the fieldwork, Oxu carried out a desk review of internal program documents and information as well as select relevant external documents. The desk review continued during the other components of the evaluation as new documentation or information was provided (See Annex E: List of Documents and Information Consulted by Oxu for FASO Final Evaluation). 4.1.2. Quantitative Survey The final evaluation included a quantitative population-based household survey in the three program health districts. The survey was conducted to be able to compare results to indicators measured during the program’s baseline survey. CEMES (Centre d’Etudes et de Management Economique et Social) Afrique, a Burkinabé consulting firm, administered the quantitative survey. Quantitative data was collected electronically on Android devices using the Open Data Kit (ODK) platform. Data was then converted to .csv data files and then cleaned. Stata was the primary statistical package used for data treatment and analysis, although Statistical Package for the Social Sciences (SPSS) was used at certain points and all files have been converted to (SPSS). Data files are included with the supporting documents as part of Annex M: FASO Endline Survey Report and its associated sub-annexes.26 The endline survey covered the program’s three targeted health districts and the sampling was designed to be representative at the level of the entire program area. The baseline survey sampling was intended to be representative of the population in each district, as well as for the program overall. The sample size for the endline survey was calculated based on the following elements: i) the proportion of stunted children of 0-23 months of age that, prior to program commencement, fell below -2 standard deviations on the Height-for-Age z-score distribution; ii) the magnitude of the estimated expected effect at the end of the program; iii) the probability of detecting a true change; iv) the power to detect if a change has occurred; v) a design effect appropriate for the use of cluster sampling; vi) the number of HHs needed to survey to be able to reach enough children 0-23 months of age (which is also sufficient for reaching 0-59 months of age given the proportion of households with 0-59 month old children is greater than those with only 0-23 month olds); and vii) a non-response factor (of 10 percent). The full calculation is detailed in the survey report; the total sample size equaled 2,550 HHs to be surveyed in 85 EAs. Although the baseline report was not clear and explicit on this issue, through discussions with CRS and review of email correspondence it became clear that: the sample size of the baseline survey was calculated to ensure the sample’s significance for the program’s impact indicators, which focus on the segment of the population “children 0-59 months old”. However, while the sample size was calculated to this aim, during the baseline survey anthropometric measurements were collected only for 0-23 month old children and infants, and not for 24-59 month olds. Based on this information, the Scope of Work required a sample calculation for the endline survey with the following criteria: i) significant for 0-23 month olds to enable comparison to baseline data (although accurate comparison is still a challenge because of the different ways in which the baseline and final sample sizes were calculated and the lack of detail on how baseline values were calculated); and (ii) significant for the overall program level and not for each of the three districts in order to economize on data collection costs. This issue is discussed further in Section 4.2 Constraints and Limitations below. 26 This has not yet been completed for analysis syntax but will be included in the final package. FASO Final Evaluation 8 A two-stage cluster sampling design was employed, taking as a starting point the FASO program target health districts. The first stage involved random selection of 85 Enumeration Areas (EAs)27 from a total of 389 EAs in the three health districts. The second stage entailed a household listing exercise in each of the selected EAs. Thirty households were then selected through systematic random sampling from the lists obtained in the household listing exercise. As was done with the baseline, the survey team surveyed the head of each household selected as well as any other member relevant to the indicators for which data were collected. The household survey had the following main components: 1. Agriculture survey assessing: general demographics, agricultural production, food security status and coping strategies, income generation, expenses, marketing, etc.; 2. Mother and/or caregiver survey to determine: young child feeding practices and children’s health, access to health care, and delivery practices; 3. Anthropometric survey for all children aged 0 to 59 months (note that height was measured only for children 6-59 months); and 4. Progress out of poverty index (PPI) survey that also included women’s empowerment in agriculture elements related to women’s participation in production and revenue-related decision-making. The table below provides a summary of the quantitative survey data collected for the agriculture and health/nutrition components. Table 5: Summary of agriculture and health & nutrition surveys planned and completed District # HHs planned # Agriculture surveys % Completion # Health & Nutrition surveys % Completion Manni 930 931 100.1% 909 97.7% Tougouri 1055 1047 99.22% 1028 97.4% Gayéri 565 570 100.9% 529 93.6% Total 2550 2548 99.9% 2466 96.7% In addition, governance-related information was collected from 113 local community-based organizations. Sampled organizations are based in the selected EAs and were supported by FASO. 4.1.3. Qualitative Data Collection28 Upon conclusion of the quantitative survey, the evaluation team carried out the qualitative component of the evaluation, which was planned sequentially so that the qualitative phase of the evaluation could be informed by preliminary findings from the quantitative survey. This was, however, not feasible due to the delays in obtaining preliminary quantitative findings (see Section 4.2: Constraints and Limitations). For the qualitative data collection, the evaluation team developed data collection tools and note-taking templates prior to commencing the fieldwork. The tools were organized for each sector: agriculture, health/nutrition and WASH, and governance. They also included cross-cutting topics based on a set of generic questions, covering beneficiaries’ expectations, relevance of activities, integration of sectors, 27 EAs are an official geographic administrative area used by the GoBF for the census and other large surveys and are typically smaller units than villages. The final evaluation used EAs to align with the GoBF survey practices so the endline data might be more useful in the future, although recognizing that it was difference from the household sampling done in the baseline. 28 A detailed updated qualitative methodology that was revised just prior to the qualitative field work is provided in Annex B.3: Updated Qualitative Methodology for FASO Final Evaluation. A summary is offered here. FASO Final Evaluation 9 impact, gender, sustainability, governance and Social and Behavior Change Communication (SBCC). The tools can be found in Annex F: FASO Qualitati ve Data C ollection Tools. Data collection for the qualitative component was carried out by Oxu’s qualitative leads with support from local language translators. The qualitative component of the final evaluation used a variety of methods for collecting information. These methods ensured, to the maximum extent possible, that if a different, well￾qualified evaluator were to undertake the same evaluation, he or she would arrive at the same or similar findings and conclusions. As such, the qualitative methods employed were the following: - 121 Key Informant Interviews (KIIs) with community and institutional stakeholders, including but not limited to: men and women program beneficiaries; FASO program staff (at Ouagadougou and field levels); government structures such as Ministry of Agriculture (MoA) and Ministry of Health (MoH) staff, the national Nutrition Directorate and other actors in agriculture, health/nutrition and WASH; local leaders including village chiefs and mayors; non-governmental actors working with FASO; WUAs and farmer group leaders; women members of VDCs; school directors or head teachers; current and past consortium members; and agro-dealers; - 44 Focus Group Discussions (FGDs) were conducted for program beneficiaries in each of the sectoral areas including: women participating in GASPA (Groupe d’Apprentissage et de Suivi des Pratiques Optimales d’ANJE - Learning and Monitoring Group for Optimal Infant and Young Child Feeding Practices) and Care Groups; Mothers Leaders (MLs); VDCs, Community-Based Organisations (CBOs), and PTA members; members of school management committees; primary school teachers; girl primary students; and farmers’ group leaders; and men and women beneficiaries of farming and SILC activities. FGDs were composed of mixed gender groups as well as female-only and male-only groups for a variety of environments to gain multiple perspectives; - 17 Site observations: Evaluation team members also observed a limited number of activities and achievements in each of the program sectoral areas, including latrines, boreholes, SILC record books, CBO record books or documents. Data collection activities are listed in Annex H: Final Evaluation Qualitati ve Fieldwork Achievements. The evaluation team shared sampling criteria with the FASO team in January. Tab le 6: Sam pling criteria for qualitat ive fieldwork Each district team was asked to propose six villages, in at least two different communes, with these criteria: - Not more than 90 minutes from the evaluation team’s lodging - Two villages have a borehole - Two villages participate in the Community-Led Total Sanitation (CLTS) activity - Two villages have a lowlands agricultural activity or other agricultural infrastructure - At least one village with a vegetable gardening activity - At least three villages with a VDC that FASO has worked with - At least one village with a PTA that FASO has worked with - Villages of varying population sizes - All three SOs implemented in each village (not necessarily all the activities of each SO) - No market nearby on the days of data collection Once the evaluation team received the lists, they also used additional criteria: - Villages that had not been included in the formative evaluation or the midterm evaluation; - Mix of villages close to the district headquarters and far from the headquarters; - Including some sites that had not benefitted from boreholes and/or CLTS to permit a comparison with sites that had boreholes and/or CLTS activities. - At least one villages in Thion commune (the only commune in which FASO implements the new MoH FASO Final Evaluation 10 - Ensuring that the major ethnic groups were included in the final list of sites; initiative called GASPA, which is comparable to the Care Group model in other FASO areas). The team used a multi-step process to analyze and validate the qualitative data both during and after the actual fieldwork. During the fieldwork the evaluation team: - Conducted daily internal team debriefs to discuss and reflect upon findings from the day and identify particular themes that were emerging from the fieldwork; - Conducted internal team summary debriefs at the end of data collection in each health district. - Presented preliminary findings to the three FASO district teams for validation and additional information; and - Consistently updated the data collection questions based on emerging themes and patterns. After the fieldwork, each Oxu evaluator conducted a full review of the summary briefings and of the data he or she had collected to confirm and sharpen findings. The team then worked together to examine across the data salient general and cross-cutting themes, as well as findings related to individual sectors or topic areas. The team presented initial qualitative findings to the FASO consortium and USAID during a validation workshop in Ouagadougou on February 16th . 4.2. Constraints and Limitations During an evaluation, there are numerous challenges that often arise. The evaluation team does not believe the challenges encountered substantially affected the quality of the data; nonetheless these constraints and limitations are presented below with analysis on the potential impacts on the findings and subsequent analysis for readers’ consideration. 4.2.1. Constraints Two main factors may have had an impact on the evaluation: First, and most significantly, the qualitative data collection was not informed by quantitative data, as initially proposed. It was planned that having the preliminary analysis of survey findings prior to conducting the qualitative fieldwork would allow the team to focus more narrowly the lines of inquiry for the qualitative work. The survey was completed on schedule; however, data storage and consolidation problems resulted in significant delays in producing data and preliminary findings. Based on the planned availability of the qualitative team, the hope that the preliminary analysis was still going to be available in time, and an earlier specification that the evaluation report needed to be completed by March 2017, the team ended up conducting the qualitative work without the preliminary endline survey analysis. The result was a qualitative approach more typical of mixed methods evaluations where the qualitative and quantitative work is conducted simultaneously. Oxu maintains that the qualitative analysis following the quantitative work is the better approach, and regrets this missed opportunity. However, the evaluation team feels that this constraint does not raise questions about the accuracy of findings but rather resulted in an opportunity cost in terms of not being able to say more and in greater detail. Second, there were several significant challenges with the baseline survey: 1. The baseline survey report presented values for all intervals but did not report confidence intervals for the vast majority of baseline measures (all indicators except those using anthropometric measures). FASO Final Evaluation 11 Furthermore, documentation for determining confidence intervals and values, including syntax for construction of the indicators, for the baseline was not available. This prevents comparison of the indicator value’s proportion or mean, as it is impossible to determine if confidence intervals overlap. Furthermore, statistical tests of difference of means (t-tests) cannot be calculated as we do not have standard deviations for these values. 2. Survey weights are missing from the data, and documentation in the baseline report and annexes for how weighting was done did not provide a coherent explanation of how the weights were created or calculated. Not having a clear understanding of the baseline weighting procedure prevents us from being able to replicate the weighted baseline values and calculate the confidence intervals from the baseline. 3. Program-level indicators at the baseline were calculated as an average of the three districts, but program-level indicators at the endline are weighted in proportion to the districts. While an average of the districts may be approximate to the overall program zone, it is not directly comparable with the endline indicators which have been sampled and weighted on the program area. 4. The baseline data provided was not clearly cleaned or coded, and we identified several weaknesses when we attempted to replicate the baseline values using the same indicator construction applied at endline. 5. As explained above in Section 4.1.2: Quantitative survey, for the baseline survey the sample size was calculated to ensure significance for the program’s two “impact indicators”, which focus on the segment of the population of children 0-59 months old.29 However, during the baseline data collection anthropometric data such as height and weight were collected only for 0-23 month-old infants/children, and no anthropometric data was collected for 24-59 month-olds.30 This means that the data presented in the baseline survey for these indicators only applies to 0-23 month-olds. As a result, comparisons for the two indicators covering children under five years of age are not possible. 6. In order to make comparisons with the available baseline data, the final evaluation team decided to ensure sufficient data to be significant for the 0-23 month-old age group. To this purpose, the endline survey sample size was calculated based on the number of 0-23 month-olds in targeted areas (which, as noted above, is also sufficient to be statistically significant for the 0-59 month age range). 7. Another challenge linked to the baseline survey is that it employed 90 percent confidence intervals, whereas the endline survey used 95 percent (per USAID Food and Nutrition Technical Assistance - FANTA guidelines). 8. This change of approach for calculating the endline survey’s sample resulted in an increased sample size, as there are fewer 0-23 month-olds than 0-59 month-olds in HHs, a higher number of HHs need to be surveyed, which requires more resources. Therefore for budget reasons CRS requested that the endline survey sample be calculated to be significant for the entire program area rather than for each district (as it was in the baseline survey). As a result, the endline survey permits comparison on the two impact indicators (albeit at 6-23 month-olds and 0-23 month-olds only), but no analysis of statistically significant differences by district. This being said, to compare stunting for 6-23 month￾olds and underweight measure for 0-23 month-olds is still a challenge because of the different ways in which the baseline and endline sampling procedures and analysis were conducted. 29 Indicator 2.1. Percentage of stunted children 6-59 months of age (Height-for-Age Z-score <-2) and 2.2. Percentage of underweight children 0-59 months of age (Weight-for-Age Z-score <-2). 30 The first indicator in the baseline report shows stunting for 6-59 month-old children. There are no footnotes to qualify or explain this, but it is well-understood by FASO and USAID/FFP that anthropomorphic data was only collected for 6-23 month￾olds and not for 24-59 month-olds. The evaluation team does not raise this point to criticize the baseline report, but rather to highlight it as an issue when considering baseline-final comparisons. FASO Final Evaluation 12 Due to these constraints, significance testing is possible only for certain indicators, keeping in mind the limitations of and concerns with the baseline data, which are detailed further in Annex M: FASO Endline Survey Narrative Report. Therefore, we present baseline-endline comparisons below and in Annex M with reservation. 4.2.2. Limitations The survey data collection generally proceeded well, although the following limitations were reported: - The most significant challenge related to data collection and storage was in relation to the Open Data Kit server. At one point the user base was over-subscribed so the survey firm cut off access to the server, meaning that supervisors had to save data to laptops and external drives (which was the back￾up option) and there was not a copy on the ODK server. This meant that the consolidation of data needed to be done manually and omitted the automatic linkage between the primary and secondary databases (which links the agriculture, nutrition, anthropometric, and poverty modules). This required that the survey team generate proxy linkages based on the survey numbers after consolidating the data from the different laptops and back-up drives to link the different question modules to a unique household. This ultimately led to significant delays in finalizing the data and the data analysis. Moreover, some data that could not be uploaded stayed on enumerator tablets and was never found. Overall, the number of surveys that were not found was relatively small, as it ranged from 0-7 percent per district and per type of survey (agriculture, nutrition…), which is still less than the 10 percent non￾response rate that was incorporated into the sample size calculation, and was distributed across districts. - The survey was very lengthy and took a long time to complete, but to ensure consistency with the baseline, there was limited room to shorten the endline survey. To the extent that long surveys can be tiring for both respondent and surveyor, it can affect the quality of the data, although it would have likely been the same for the baseline as well as the endline. - Collection of data on governance: The governance data collection for the endline survey was set up in a way that governance surveys would only be done in villages where EAs were sampled. Furthermore, it only considered community organizations that have been supported by FASO. This resulted in a relatively low number of organizations that were surveyed during the endline. However, the governance data was not meant to be representative of the population or analyzed looking at statistical significance in terms of baseline-endline change. To address having a smaller sample size, the final evaluation team used the qualitative study and annual program data to triangulate the information obtained through the endline governance survey. - There are endline values for three output-level indicators in SO2 that we were not able to calculate.31 This was mainly due to an issue with ODK not properly accepting multiple responses for the question, as numeric variables were not programmed accurately in ODK to accept multiple answers for the question being asked, which means that we have no way to know if more than one answer was given. There were text values in response to the choice “other” but it would not be possible to know all of the coded answers provided, which corresponded to the key knowledge changes sought for each particular indicator. 31 This includes the following indicators: 2.1.1.1. Percentage of household heads and mothers of children 0-23 months who knew all critical handwashing moments; 2.1.1.2. Percentage of mothers of children 0-23 months who can correctly cite 2 advantages of exclusive breastfeeding; and 2.1.4.1. Percentage of men who can cite 3 advantages of delivering in a health center. FASO Final Evaluation 13 - The lack of data connection and electricity in several areas meant that some enumerators had to do additional travel to find ways to charge devices or upload data; this could have contributed to delays in data uploads and fatigue of some enumerators. In the qualitative data collection process, translators were used for translating from the local language into French. The translators selected were fluent in Gourmanché and in Moré, the two major languages of the program health districts and were familiar with FASO activities and technical terms as they had worked as enumerators on the endline survey. However, the use of translators can always introduce some level of error. Additionally, several program areas were composed of mixed ethnic/linguistic groups, so participants belonging to the minority may have been more comfortable in another language than the one used for the focus group. This applies to both Gourmanché or Moré-speaking groups (both being the majority in some of the sites and the minority in others), but more significantly also to Fulani-speaking Peul groups, which also existed as a minority in parts of the FASO area, and whose language was not employed during data collection.32 The qualitative methodology included data collection for 12 days in program areas. The team collected data in six villages, opting to have approximately two days per village rather than spending only one day per village to reach 12 villages. In addition, while the qualitative work was focused in six villages, some FGDs and KIIs included program staff and beneficiaries from a number of surrounding villages, which helped provide a broader perspective. The sampling methodology above also notes that the evaluation team selected the two villages from a list of six proposed by the FASO program staff. It is possible that the program staff only suggested the best villages, but given the other criteria that were included we do believe that any potential bias was mitigated. 5. Baseline/Endline Data This section presents baseline and endline survey data for all indicators at the SO and IR levels along with some of the output-level indicators that were measured during the baseline or endline surveys with population-based samples. The tables in this section provide baseline and endline survey values with associated confidence intervals (which, for the baseline, are only available for anthropometric indicators). For baseline data, because the indicator construction is uncertain, we calculated unweighted baseline values as a way to try to check whether our formula calculations for the endline data are comparable to those used in the baseline (to a degree more specific than is available in the generic formulas in the baselie report). Where we find minimal differences between the weighted and unweighted baseline values we can be more confident that the indicator construction is identical at baseline and endline. However, for where there are substantial differences between the baseline report values and the unweighted baseline values that we have recreated it is very difficult to be confident in these values or the means of calculation. We proceeded to make a comparison where baseline report values and unweighted re-calculated baseline values were within 10 percent. To address baseline weighting issues, Oxu also recalculated weighted and unweighted baseline values to be proportionate to district size. In the tables below we note where the data show statistically significant differences (at 95 percent confidence level) from conducting two z-tests: one comparing the recalculated weighted baseline value and the weighted endline value, and a second that compares the recalculated unweighted baseline value and the unweighted endline value. In this way, by 32 The evaluation team did learn of villages or divisions/ neighborhoods within villages that were majority Peul, as it seems they sometimes tend to live together. The evaluation team did conduct WASH observations in one of these areas. FASO Final Evaluation 14 only showing where the weighted and unweighted comparisons are both statistically significant, then we are more confident in a change having occurred at the population level. As such, these comparisons are useful to the extent that they suggest change, particularly when triangulated against annual survey data and qualitative field work. For SO-level indicators, we also provide final targets in the tables in this section, 33 and for SO1 and SO2 we include a summary of findings from baseline-endline comparisons. Finally, note that the wording of result-levels and indicators is taken from the IPTT.34 Given that the evaluation design was not set up as an impact evaluation, it is important to remember that it is not possible to determine causality or direct attribution from FASO program implementation to any outcomes at the population level. Any statistically significant changes between the baseline and endline indicators would suggest changes that occurred at the population level, and it is presumed that FASO has possibly contributed to these results. As discussed in Section 4.2, there are several constraints that disallow standard testing for statistically significant differences between the baseline and endline indicators, the primary constraints being: the lack of confidence intervals for the majority of baseline indicators, the baseline values from the baseline report not being proportional to district population size, and it not being clear from the baseline report how weighting has been taken into account in the calculation of specific indicators. However, the data can still be considered to approximate the population, and improvements in the indicator values provide descriptive evidence to triangulate with findings from the rest of the evaluation. Because of this, and as requested in the SOW, we have decided to present a summary of the baseline-endline data separately in this section prior to providing the more detailed findings for each of the SOs (which will be presented in Section 6: Findings, conclusions and recommendations by sector and make reference to the baseline-endline data where relevant, in the context of the qualitative study). A complementary analysis of baseline-endline comparisons is available in Annex M: FASO Endline Survey Narrative Report. To most fully utilize the data, while still maintaining good practice in statistical analysis, the endline survey report presents comparisons that have been made between: 1) original weighted baseline indicators, re-averaged to be proportional to district size, compared with weighted endline indicators, and 2) indicators of unweighted data at baseline and endline, proportional to district size. In this way, the comparison of unweighted values serves as an additional check on the data so that like-for-like comparisons are being made. Please see Annex M for further information and complete data comparison tables showing weighted and unweighted baseline indicators, proportional baseline indicators, and endline indicators. 33 We did not include final targets for IR and output-level indicators as many of the “LOA (Life of Activity) targets” in the FY16 IPTT seem to refer to final program-level beneficiary targets rather than at the population level, though this is not entirely clear when reviewing the IPTT. 34 Note that some of the wording in result levels and indicators is not completely grammatically correct (such as several indicators that mention “% of targeted beneficiaries farmers” rather than “beneficiary farmers”), some indicators refer to beneficiaries but actually were measured at population levels during the baseline and endline surveys (such as % of targeted beneficiaries), and there is some mixing of French and English acronyms, but we kept the language the same as is used in program documents for consistency sake. FASO Final Evaluation 15 SO1: 56,126 HHs have improved access to food of sufficient quantity and quality throughout the year Table 7: SO1 baseline-endline survey data Indicator Baseline report value Baseline report N Baseline Proportional to District Size Endline value [95% CI] Endline N Target final 1.1 Average number of Months of Adequate Household Food Provisioning (MAHFP) – no longer applicable for the program 8.71 1619 8.52 10.16 2548 N/A [10.09, 10.23] 1.2 Average Household Dietary Diversity Score (indicator HDDS) 3.3 1713 3.36 3.28 2486 6 [3.22-3.34] 1.3 % of household with moderate or severe hunger scale (FFP) N/A† 6.2335 2548 29 [5.29-7.17] † Indicator 1.3 was not measured at baseline but was measured at the beneficiary level during 2011 Annual Survey (with a value of 34.3, which was presumably used to set the final population target). IR1.1: 33,375 HHs sustainably improve their agricultural production Table 8: IR1.1 baseline-endline survey data for IR and select output indicators Indicator36 Baseline report value Baseline report N Baseline Proportional to District Size Endline value [95% CI] Endline N 1.1.1a Average household production of cowpea (kg) 130 1780 132.96 143.34 [88.94, 197.74] 2548 1.1.1b Average household production of sesame(kg) 58.75 1780 60.64 45.99 [38.48, 53.50] 2548 1.1.1c Average household production of rice (kg) 126.31 1780 130.53 155.1 [93.77, 216.44] 2548 Average household production of sweet potato (kg) 14.71 1780 16.02 26.75 [8.50, 45.01] 2548 1.1.2.1f. % men & women using thinning for sesame 61.89 958 66.08 72.69* [70.33, 75.04] 1381 % of targeted beneficiaries who plant sesame 52.2 3433 56.14 54.96 [53.01, 56.91] 2512 *statistically significant at 5 percent alpha-level 35 Note that the endline value for this indicator may have been affected by the timing of data collection, as endline data collection occurred the last week of November and during the first three weeks of December, which is typically a more abundant period after the harvest. While this indicator was not measured at baseline, during the 2011 Annual Survey the data was collected in October when food scarcity can still be a challenge, and when FASO found that 34.3 percent of beneficiary households reported moderate or severe hunger. 36 For household production of select crops the baseline and endline surveys calculated average production across all sampled households, not just those who produced the crop. This gives a broader sense of availability but should not be misconstrued as the average household quantity produced by only those households that produced the crop. FASO Final Evaluation 16 IR1.2: At least 50,500 beneficiary farmers sustainably increase revenue Table 9: IR1.2 baseline-endline survey data for IR and select output indicators Indicator Baseline report value Baseline report N Baseline Proportional to District Size Endline value [95% CI] Endline N 1.2.1. % of targeted beneficiaries farmers (by gender) reporting increased revenue from agricultural activities 31.19 3433 24.29 52.68* [50.72, 54.63] 2512 1.2.1.1d. % of beneficiaries farmers who use plastic material to shell their rice 17.47 391 22.27 45.67* [41.11, 50.23] 462 1.2.1.1a. Percentage of sesame producers who hulled their sesame on a plastic sheet 69.60 1699 68.87 79.7* [77.58, 81.83] 1381 1.2.1.1b. Percentage of sesame producers who cleaned their sesame before bagging 89.62 1668 89.02 98.83* [98.27, 99.40] 1381 % of households (HH) who declared that access to fertilizer is relatively easy 3.00 1780 2.35 11.93* [10.66-13.2] 2512 % of households (HH) who declared that access to improved seeds is relatively easy 11.8 1780 12.41 28.75* [26.97-30.52] 2509 *statistically significant at 5 percent alpha-level Summary of SO1 Comparison Findings: There are few definitive conclusions that can be made from comparisons of the SO1 baseline and endline data. For the lone objective-level indicator, HDDS, we conducted a t-test but did not find any statistically significant difference (comparing a recalculated weighted baseline value of 3.36 to an endline of 3.28). The program also did not achieve the endline HDDS target of 6. For average household production data, the program’s indicators at the IR1.1 level, there are several issues with the baseline data that make it impossible to make meaningful comparisons, even though the household production indicators are very important for this objective. As a complement to the production data collected at baseline and endline, we also reviewed secondary source data from the government’s Directorate of Agricultural Statistics, which included annual average household production statistics by region and nationwide from 2009-2016. However, there were wide discrepancies between the FASO data (gathered at district level) and the secondary statistics. This can be expected given the secondary data was for regional and national levels; moreover, there are varying agro-ecological conditions even within regions. As a result, there is very limited use in trying to make meaningful comparisons with this data or even in attempting to validate or triangulate baseline or endline findings using the regional and national statistics. For the program’s one indicator at the IR1.2 level, % of individuals in the agriculture sector who declared increasing their agriculture revenue from the previous year, we observed a statistically significant increase of nearly 30 percentage points when comparing the endline weighted value with the baseline report value recalculated proportional to district size. The program, however, did not reach the endline target of 84 percent, and the endline value for men was actually much higher than the value for women (at 53.39 percent for men and 34.86 percent for women, though men comprised 2403 out of 2512 endline respondents). FASO Final Evaluation 17 Many of the program’s output-level indicators in SO1 (for outputs under IR1.1 and IR1.2) sought to measure improvements in agricultural producer practices or behaviors. For these output-level indicators we expected to see increases in indicator values, which would suggest improvements over the course of the program period at the population level. There are a handful of these practice-related indicators where the data suggest statistically significant changes. These include the following (with the percentage point increase comparing weighted endline and weighted baseline proportionate to district size): For the other SO1 indicators, there are very few conclusions that can be made from the survey data. SO2: 51,426 mother/child units have improved their health and nutrition Table 10: SO2 baseline-endline survey data Indicator Baseline report value [90% CI] Baseline report N Baseline Proportional to District Size Endline value [95% CI] Endline N Target 2.1 Prevalence of stunted children under 5 years of age (Rise) N/A 32.57 [30.76, 34.44] 2759 17.5 2.1a Prevalence of stunted children under 2 years of age (Height-for￾Age Z-score <-2) 26.5 [23.7-29.6] 1288 25.82 27.58 [24.66, 30.71] 909 N/A 2.2 Prevalence of underweight children under 5 years of age (Rise) N/A 20.46 [19.02-21.90] 3042 23.7 2.2a Prevalence of underweight children under 2 years of age (Weight-for-Age Z-score <-2) 38.50 [35,3-41,8] 1832 38 22.89* [20.47, 25.51] 1192 N/A *statistically significant at 5 percent alpha-level IR2.1: 18,500 mothers of children 0-23 months and pregnant women adopt recommended health, nutrition and hygiene practices: Table 11: IR2.1 baseline-endline survey data for IR and select output indicators Indicator Baseline report value Baseline report N Baseline Proportional to District Size Endline value [95% CI] Endline N 2.1.1 % of infants 0–5 months of age who are fed exclusively with breast milk (FFP)/Rise 40.00 N/A 37.39 56.21 [49.93, 62.3] 267 2.1.2 % of HHs using the available (improved) sanitation facility (Initial Environmental Examination - IEE) 22.60 1800 24.51 42.28* [40.19, 44.4] 2453 Indicator Percentage point increase % of producers that practice thinning for sesame 6.6 % of sesame producers who hulled their sesame on a plastic sheet 10.8 % of sesame producers who cleaned their sesame before bagging 9.8 % of producers who thresh their rice on plastic surface 23.4 % of households (HH) who declared that access to fertilizer is relatively easy 9.6 % of households (HH) who declared that access to improved seeds is relatively easy 16.3 FASO Final Evaluation 18 2.1.3. Percentage of children 6–23 months of age who receive a minimum acceptable diet (apart from breast milk) (FFP)/Rise† N/A N/A N/A 7.90 [6.18, 10.07] 826 2.1.4 % of children aged 0-23 months with diarrhea in the last two weeks whose mothers sought outside advice or treatment for the illness 54.80 1832 54.42 63.99* [58.01, 69.56] 297 2.1.5 % of mothers of children age 0-23 months who had four antenatal visits when they were pregnant with the youngest child 35.57 1274 35.23 50.22* [47.17, 53.27] 1160 2.1.6 % of mothers of children 0-23 months who had a postnatal visit within 45 days of delivery 22.00 1816 21.15 29.94* [27.26, 32.76] 1160 2.1.1.3 % of mothers of children aged 0-23 months who know at least two danger signs of childhood illness that indicate the need for treatment 74.20 1816 73.60 90.04* [88.01, 91.76] 1160 2.1.3.1 % of households with access (possessing) to an improved sanitation facility (latrines) (IEE) 17.40 1800 18.98 40.08* [37.99, 42.19] 2453 2.1.5.1 % of children 6-23 months of age who receive foods from 4 or more food groups (7 food groups) during the previous day 13 N/A 13.88 10.12 [8.18, 12.47] 826 2.1.5.2.a % of pregnant women and mothers of children aged 0-23 months who consumed at least 5 of 10 food groups in the last 24 hours†† N/A N/A N/A 34.51 [31.9, 37.21] 1395 2.1.5.3. % of targeted children 6-23 months consuming Vitamin A rich foods in the last 24 hours 21 1288 20.75 22.85 [19.93, 26.06] 826 2.1.5.4.% of mothers of children 0- 23 months consuming Vitamin A rich foods in the last 24 hours (FFP) 31 1816 32.47 86.63* [84.35, 88.62] 1160 2.1.5.5. Percentage of children aged 6-23 months who consumed locally fortified flour as complimentary food 21 1288 22 31.48* [28.27, 34.88] 826 *statistically significant at 5 percent alpha-level †For this indicator, the baseline measured the percentage of children 6-23 months benefitting from at least 3 appropriate infant feeding practices, whereas the endline measured the minimum acceptable diet indicator, which is Indicator 2.1.3 in the FY16 IPTT. For this reason, we only present the endline data here. It should be noted that for those children not breast-fed, the data permitted only identifying if dairy was consumed 1 of the 4 times in the day, as opposed to ensuring that it is 2 of the 4 times according to the indicator guidance. The baseline values presented are unweighted values calculated for the baseline data, given the lack of a value for this indicator in the baseline report. FASO Final Evaluation 19 †† Indicator 2.1.5.2 was originally the “% of pregnant women and mothers of children aged 0-23 months who consumed at least 4 of 7 food groups in the last 24 hours,” and this was later updated to include 5 of 10 food groups. The endline survey measured both, but we present the 5 of 10 groups indicator here. IR2.2: 79,590 women and children 0-59 months access quality nutrition and health services Table 12: IR2.2 baseline-endline survey data for IR and select output indicators Indicator Baseline report value Baseline report N Baseline Proportional to District Size Endline value [95% CI] Endline N 2.2.2 % of mothers of targeted children 0- 23 months who gave birth in the health center 62.90 1816 62.87 84.06* [81.81, 86.07] 1160 2.2.2.1 % mothers of children 0-23 months who declared being supported by the community in accessing healthcare (care group, SILC, CVD, COGES-basfond) (structures communautaires) 43.60 1816 46.66 56.39* [53.35, 59.39] 1160 *statistically significant at 5 percent alpha-level Summary of SO2 Comparison Findings: For SO2, the baseline-endline comparisons provide evidence of several population-level improvements for nutrition and health-related indicators. Many show not only statistically significant changes but also substantively significant improvements. For example, we found a statistically significant reduction of over 15 percentage points in underweight children 0-23 months of age when comparing the weighted baseline value (recalculated to be proportional to district size) and the endline value. For most of the SO2 indicators at the IR and output levels where we could conduct z-tests, we found statistically significant and some sizeable improvements when comparing the weighted baseline and endline values. These are discussed in more detail in Section 6 below and include improvements in access to and usage of improved sanitation facilities, use of health services, and dietary consumption for pregnant women and mothers of children 0-23 months, among others. There were numerous other indicators where there was an increase, and often a sizeable one, between the baseline and endline values (both weighted and unweighted); however, because the difference for these indicators in the weighted and unweighted baseline values was also greater than 10%, it is difficult to draw definitive and statistically rigorous conclusions. We can, however, note in a descriptive or qualitative way substantial positive differences in the baseline and endline data. SO3: Community-based organizations in 100 villages are using principles of good governance in implementing community development plans for reducing food insecurity Table 13: SO3 baseline-endline survey data Indicator Baseline report value Baseline N Endline value Endline N 3.2. [alt.] Percentage of targeted community committees (including CVDs) applying at least 2 of 4 good governance principles † 55.50 209 95.74 94 3.1.1. Percentage of CVDs that abide by 3 of 4 criteria of functioning CVDs 40.57 69 60.71 28 3.2.1. [alt.] Percentage of community structures with at least one woman in a decision-making position [calculated for 76.74 172 94.68 94 FASO Final Evaluation 20 women in any committee position; this is comparable to baseline calculation] 3.2.2.1. % of PTAs and village development counsels who undertake at least 2 actions to support girl's education (enrollment, attendance) 80.6 215 85.42 48 3.2.2.2. Percentage of targeted PTAs who provide sufficient food for 4 months of school lunches 4.6 65 95 20 † The baseline indicator for 3.2 measures the application of at least two of four good governance principles, not three of four as the indicator is documented in the final IPTT. 6. Findings, Conclusions and Recommendations by Sector This section presents the evaluation findings, conclusions and lessons learned, and recommendations for the program’s three SOs, including: SO1: Agriculture; SO2: Health/Nutrition and WASH; and SO3: Governance. For each SO, we provide a summary of the objective’s design and key activities; present findings at the SO and IR levels (output-level findings are presented in Annex I.1-3: Output-level findings for each SO); discuss conclusions and lessons learned based on the findings; and finally, offer recommendations. At the end of the section we also include cross-cutting recommendations. The findings for each objective are based on three principal sources of information: i) results of the endline population-based survey; ii) the qualitative field work; and iii) FASO program documents and program data (including monitoring data, annual survey data, the baseline report, MTE report, etc.). Within the findings sections, quantitative achievements for select program indicators are provided in tables for each SO and IR to show some of the important numerical outcomes at these levels and to complement the qualitative findings. For these tables, we show baseline and endline survey values for indicators measured at the population level (Pop.), as well as the values for the indicators concerning the program’s direct beneficiaries (Dir. Ben.) measured at the beneficiary level through annual program surveys. Note that the baseline values provided in this section are the original values taken from the baseline report, though as explained at length in previous sections we were not able to be certain that the endline values are indeed comparable, given that we did not have access to key information from the baseline such as weighting, confidence intervals (for nearly all indicators), syntax, and indicator construction. We include the values from the baseline report to ensure consistency in our approach, even though there are questions regarding the baseline data, and for a small number of indicators we noted discrepancies between the baseline report value and the value presented in the IPTT. For the beneficiary-level indicators, we include information sourced from the FY2016 IPTT and for population-level indicators we use the baseline report and endline survey data. The tables include: “initial data,” which provides the value from the initial year for which program-level beneficiary data is available (and notes the program fiscal year) and the baseline report value for population-level data; “final target,” which is the target included for the final year in which program-level beneficiary data is available and the life of activity target for population-level; “final achieved:” the value for the final year in which data is available for beneficiary-level data (with the FY noted) and the endline survey value for population data; and “% final achieved,” which shows the percentage achieved of the target for the final year in which data is available for program-level data and for the life of activity for population data. Compiling this FASO Final Evaluation 21 information was a rather complicated process given that initial and final program data was available for different indicators in different years. It is also noteworthy that many of the findings included in this section were presented to FASO program staff and USAID during the validation workshop in Ouagadougou. Several findings were questioned by workshop participants, which is a natural part of such a process, and such feedback was helpful to the evaluation team in continuing to refine the findings, conclusions and recommendations. Finally, the recommendations are relevant to the FASO program as well as other future similar multi-year food security interventions in similar contexts. 6.1. SO1: Agriculture 6.1.1. Summary of FASO SO1 Design37 Table 14: Summary of FASO SO1 design SO1 56,126 HHs have improved access to food of sufficient quality and quantity throughout the year IR1.1 33,375 HHs sustainably improve their agricultural production Output 1.1.1 At least 56,126 beneficiaries access diversified agricultural inputs Output 1.1.2 At least 33,375 beneficiary farmers apply improved agricultural production techniques Output 1.1.3 14,300 hectares of productive land sustainably improved and protected IR1.2 At least 50,500 beneficiary farmers sustainably increase revenue Output 1.2.1 At least 22,450 beneficiary farmers thresh sesame and rice production appropriately Output 1.2.2 At least 50,500 beneficiary farmers hermetically store cowpea Output 1.2.3 800 women improve parboiled rice value chain Output 1.2.4 10 producer unions and 50 village level groupements have improved their access to markets (22,500 HHs, 7,000 women) Output 1.2.5 17,100 vulnerable beneficiaries sustainably finance productive activities 6.1.2. Introduction to SO1 Activities SO1 activities were implemented by two local partners: ATT in Manni and Gayéri and OCADES-Kaya in Tougouri. SO1 sought through a wide variety of activities to diversify agricultural production and livelihoods opportunities, leverage opportunities for Income-Generating Activities (IGAs), and improve natural resource management. FASO focused on production for both nutrition and income. This informed its orientation towards dual-purpose commodity value chains, that is, crops that can be marketed for cash and/or used for household consumption. The program thus emphasized rice, sesame, cowpea, and orange￾fleshed sweet potato. Yet, since the program is being implemented in chronic food deficit areas, FASO included the three traditional staple cereal value chains in the portfolio: sorghum, maize and millet, with the underlying assumption that production of these food staples would primarily serve for the purpose of consumption. Key activities from the start of the program included: - Provision of agricultural inputs, including improved seeds (cowpea, sorghum, orange-fleshed sweet potato, rice, and maize) through subsidized seed coupons and tools (carts, wheelbarrows, shovels, etc.) through a tool rental system managed by local Tool Management Committees (TMCs); 37 Source: FASO Results Framework, as of January 2016. FASO Final Evaluation 22 - Reclaiming degraded agricultural lands through Food for Work (FFW), to improve soil and water conservation using the zaï technique as well as half-moons, stone bands, and other agro-ecological land improvements;38 - Improvement and reclamation of lowlands areas for rice production; - Promoting improved agricultural techniques for target crops39 through: demonstration plots, subsidized provision of triple-lined storage bags (for cowpeas) and urea fertilizer (for lowland rice production), thinning, line sowing, spacing, as well as other technical assistance; - Support producer groups such as women’s rice parboiling associations and lowland rice producers by facilitating access to markets and credit providers, including linking them to Micro Finance Institutions (MFIs) and implementing the warrantage activity (discussed below); - Promoting linkages among service providers and producers, including: agro-dealers, seed multipliers, the MoA, government research institutions, and small-scale farmers; - Establishing self-selected Savings and Internal Lending Communities (SILC) groups, primarily comprised of women, to provide opportunities to beneficiaries to mobilize financial resources within small community groups. To implement SILC, FASO worked through a network of local Private Sector Providers (PSPs); the PSPs are paid by SILC groups through a fee-based service and provide support to graduated groups as well as help establish new groups. Other activities were introduced during the program (at least three years in) to capitalize on additional opportunities: - Market gardening: introduced in FY13 targeting primarily women with the aim of improving availability of food during the dry season, as well as responding to household nutrition needs; - Warrantage: 40 starting in FY14, FASO used the program’s warehouses that were no longer needed for Title II commodities to provide access for farmers to credit (via MFIs) and storage through an inventory credit system (also commonly known as a warehouse receipts system); this was done by linking farmer groups and micro-finance institutions; and - Poultry: also introduced in FY14, female-headed households received poultry inputs and training, and were linked to the Village Poultry Extension Agents (VPAE) system. 6.1.3. Findings for SO1 The findings begin with the SO population-level indicators, which are then followed by findings for IR1.1 and IR1.2. Where relevant, we present and discuss the endline survey results and data from the IPTT, to show the population-level data as well as data reported from direct program beneficiaries. The evaluation team’s qualitative findings are also woven into this section. Detailed output-level findings are provided in Annex I.1: Output-level Findings for SO1. SO1: 56,126 households have improved access to food of sufficient quality and quantity throughout the year 38 Zaï is a simplified pit planting technique in which small pits are filled with covered compost; half-moons are semi-circle holes dug in a degraded land to capture water in hostile climate areas where rainfall is scarce and erratic; and stone bands are stone-hedges laid in a line on a degraded or vulnerable soil with a slope. The hedge thus impedes flowing water to carry away substantial soil nutrients. 39 The techniques promoted focused on: correct application of urea fertilizer for rice production; rice and sesame threshing using tarpaulins; intercropping for cowpea; and cowpea storage. 40 Warrantage, a French word derived from the English borrowing “warrant”, is commonly used in West Africa to describe the inventory credit system. In practice, warrantage is about granting credit with grain as collateral in secure warehouses, where a third independent party holds the collateral on behalf of both the creditor and the debtor. FASO Final Evaluation 23 Table 15: SO1 indicators Indicator Baseline† Final Target Endline Survey Result 1.2. Average household dietary diversity score 3.3 6.0 3.28 1.3. % of household with moderate or severe hunger scale N/A†† 29.0 6.23 † As mentioned in the Introduction to Section 6, in this section we provide baseline report values as they were presented in the baseline report. †† A true baseline-endline comparison is not possible, as this indicator was not measured during the baseline (though it was measured in the 2011 annual survey at the beneficiary level and was 34.3 percent, which presumably allowed for FASO to set the final program target). Through FY16, FASO reached the following number of beneficiaries through agricultural activities: Table 16: Beneficiaries reached through agricultural activities (disaggregated by gender)41 Male Female Total # % # % # % 58,603 44% 75,151 56% 133,754 100% IR1.1: 33,375 HHs sustainably improve their agricultural production Table 17: IR1.1 indicators Indicator Source Initial Data Final Target Final Achieved % Final Achieved 1.1.1a Average household production of cowpea (kg) (in population) Dir Ben. 101.19 (FY11) 300 322 (FY15) 107 1.1.1b Average household production of sesame (kg) (in population) Dir Ben. 50.1 (FY11) 150 142 (FY15) 95 1.1.1c Average household production of rice (kg) (in population) Dir Ben. 177 (FY11) 400 754 (FY15) 189 1.1.1e Average household production of sorghum (kg) (in population) Dir Ben. 1122 (FY13) 1150 1229 (FY15) 107 1.1.1f Average household production of millet (kg) (in population) Dir Ben. 442 (FY13) 520 535 (FY15) 103 1.1.1g Average beneficiary production of onion (kg) Dir Ben. None 250 387 (FY16) 155 1.1.1h Average number of live chickens per beneficiary per year No data for this indicator; the FY16 IPTT states that the program is awaiting a specific survey on poultry breeding. IR1.1 findings: Farmers encountered in FGDs consistently reported improved agricultural production. Production-level data for FASO beneficiaries generally confirms this finding, as program-level data shows increases in average household production for FASO beneficiaries for all crops (shown in the table above and for cowpea, sesame, and rice in Figure 1 below). Moreover, average household production levels for cowpea and rice for program beneficiaries in FY15 (the last year for which we have annual program data) are higher than the population-based numbers from the endline survey. This is not, however, the case with rice, which had an endline value of 877.6 kg (though with a wide confidence interval of 531-1224 kg given the limited number of endline survey respondents that produce rice), and the production level in the FY15 annual survey was 775 kg. Another important IR-level finding is that FGD participants noted that they have reduced the number of hungry months during the year, though they still face hungry months. Finally, in all 19 FGDs held farmers highlighted using land improvement techniques to upscale their 41 Many of the SO1 beneficiaries also participated in SO2 and SO3 activities as part of the program’s deliberate integration strategy. FASO Final Evaluation 24 production. These techniques were also mentioned often in SO2 FGDs. In terms of production, gaps have persisted between male and female farmers for certain crops (and such gaps remain at the population-level as well).42 Respondents during the qualitative field work also noted that in some seasons production levels have been lower due to environmental factors (such as unexpected drought pockets, flooding, late rains, gullies, infrastructural default, labor, shortage of and constraints to accessing farm inputs such as chemical fertilizers and compost). FASO attempted to address access to quality seed through offering the partially subsidized seed coupons that included early-maturing and drought-resistant seed varieties. The use of improved seeds, except for market gardening, was not highlighted frequently during the fieldwork. Based on beneficiary farmers’ declarations during FGDs and KIIs across the districts, there is an indication that many farmers do not renew the seeds after the second or third generation of the one received, which raises a problem of sustainability in promoting improved variety seeds. At the same time, seed provision is complicated in the FASO program areas (and in Burkina Faso more broadly), as seed is often distributed by the GoBF or other NGOs free or at highly subsidized rates. As a result, FASO did not reach planned targets for seed coupons as some beneficiaries even preferred to get seeds late or of lower quality than to use the less subsidized FASO seed coupon to purchase seeds. This was consistent with a general finding within SO1 that many producers have an expectation for NGOs to provide or distribute materials/ inputs, contributing to a growing sense of dependency. To strengthen the seed system the program has also been working with 42 See the FY2016 annual results report (ARR): “While the average production of 387kg of onions per farmer exceeded the FY2016 target of 300 kg for market gardeners, men tended to harvest more than women (443 kg. for men vs. 371 kg for women). This is mainly due to women having less access to manure and technical information. The gross margin per HA also shows a big difference favoring men. This situation is the result of the combination of lower levels of production and higher expenses by women than men for labor costs. Competing domestic chores often compel women to hire paid labor to help them with land preparation or seedling transplantation.” On the other hand, KIIs with program staff suggest that an operational research activity actually showed that rice production levels were fairly comparable for men and women, but that it was in the revenue obtained for rice that there seemed to be the greatest difference between women and men. This report has not yet been produced, nor has the program apparently made any modifications in activities to address this finding. Assuming the validity of this research, the tentative explanation for the difference could be found, at least, at two levels, including the characteristics of the women (head of household, widow managing her own farm, dependent of a husband…) and the intended use of the production, as most women’s top objective is household food security. Besides, as the market for rice is not so vibrant in most of the program areas, transportation and mobility issues could also be a hurdle to women’s maximizing in returns to rice. Figure 1: Production trend of Cowpea, sesame, and rice from baseline through year 5 (Data source: FY16 IPTT) FASO Final Evaluation 25 some local agro-dealers and recently with seed multipliers in FY16; but FASO encountered challenges in identifying regional seed multipliers and conditions required by the government seed certification schemes meant slower progress in increasing available certified seed. Upon recommendation of the MTE, FASO also introduced Tool Management Committees (TMCs), which were set up to be sub-committees of the VDCs (which are allowed to create sub-committees under the GoBF guidelines for VDCs). Though the vision behind the TMCs was to support them to operate as a business instead of giving tools to individuals in the village, they also came in handy as a community response to agricultural equipment that quickly deteriorated or was lost. The TMCs were trained to establish user-fee systems, as well as a system for storing, maintaining, and monitoring the tools. During FGDs farmer groups and other individuals noted the usefulness of these committees, as they established clear conditions for accessing the equipment (see also section 6.3.3. SO3 Findings, IR3.4, below). The final main finding related to access to inputs is that promotion of organic and inorganic fertilizers was appreciated across districts. By and large the qualitative field work also found during all 19 FGDs and four KIIs that program beneficiaries have knowledge of the agricultural practices promoted by FASO within and across districts, and declare applying those practices. However, annual survey data shows more mixed results in terms of the IR1.1 practices being adopted, as discussed in the detailed output-level findings for Output 1.1.2 (Annex I.1: Output-level Findings for SO1). Moreover, the evaluators found that the reclamation of lowlands for rice production also had mixed results. Some households improved rice production and are increasingly adopting rice consumption. At the same time, during the field work in Gayéri and Tougouri especially, several challenges were raised related to the program’s promotion of lowland rice production. These include: lowland production being labor-intensive and not being a customary practice in the social ecosystem, unavailability and expense of fertilizer, and conflicts related to time and financial resources for farmers to perform their traditional agricultural activities. In addition, there is a limited market for rice, especially in Gayéri where the lowlands also seem less relevant, given that more fertile land is available for commonly cultivated crops and an emerging perception that youth are being pushed out of agriculture as they are pulled by mining sites, which are reportedly more lucrative. As a result, a high number of plots in certain lowlands were not exploited.43 The MTE also 43 A barrier analysis was conducted in 2016 to explore this issue, and an SBCC plan was elaborated as part of the analysis. Among others, it emerged from the analysis that women needed a more varied and women-friendly set of tools as part of the stock of the TMC. Besides, the lack of a guidance handbook (in the manner of the Mother Leaders ideogram handbook) that synthesizes the technical itinerary for rice appeared to be an obstacle to many producers. 0 10 20 30 40 50 2014 2015 2016 % % of tar geted beneficiar ies who plant rice Manni Tougouri Gayeri Total Figure 2: Percentage of Targeted Beneficiaries who plant rice (Data sources: annual program data by district) FASO Final Evaluation 26 raised concerns about this activity, noting that it did not appear to be farmer-driven and that the production of rice in lowlands was risky as it depends on timely rainfall patterns. In some areas in FY14 and FY15 in particular, rains arrived late. As Figure 2 shows, the percentage of targeted program beneficiaries planting rice also declined significantly from FY14-FY16, which can be explained by late rains, constraints with water and flood management, and beneficiaries pursuing other activities. This reduction is more striking considering that the endline survey found that 19 percent of the population plant rice, which means that the proportion of the overall population of producers planting rice is nearly double the approximately 10 percent of program beneficiaries that planted in 2016. FASO used FFW through FY14 to reclaim degraded areas to improve soil and water conservation. In FY15, once FFW had concluded, according to the FY15 annual results report (ARR) more than 3,500 hectares (HA) of degraded land were improved by 7,650 farmers (of whom 52 percent were female), in comparison to 829.25 HA improved in FY14 with the use of FFW. Some government agricultural staff, however, regretted the fact that beneficiaries declined to participate in similar government-initiated lowland development activities that were occurring simultaneously but that did not offer any compensation (money, food, etc.). This suggests that the impressive increase could have been more motivated by the compensation than the beneficiaries’ intrinsic adoption of the lowland development activities. Farming households have also integrated agro-ecological intensification and land improvement techniques in their daily agricultural practices. It is important to note that the zaï technique has primarily benefited women.44 During the qualitative field work, particularly in Tougouri and Manni, respondents in farmer-oriented FGDs and KIIs as well as some in SO2 and SO3 groups frequently cited the benefits of composting, zaï, half-moons, and stone bands. This was less frequently reported in Gayéri (this is likely due to the fact that there is readily available productive land, hence less need to promote such techniques). In the villages visited in Gayéri, the evaluation team observed that the lowlands were not being used in an optimal manner compared to the other districts. FASO also promoted other conservation agriculture techniques like mulching with crop residues, composting and use of compost as organic fertilizer, and "conservation agriculture with trees," which integrates tree species (such as moringa, acacia species and baobab) with the other techniques. IR1.2: At least 50,500 beneficiary farmers sustainably increase revenue Table 18: IR1.2 indicators Indicator Source Initial Data† Final Target Final Achieved % Final Achieved 1.2.1. % of targeted beneficiaries farmers (by gender) reporting increased revenue from agricultural activities Dir Ben. 21.8 (FY11) 31 65 (FY15) 210 Pop. 31.19 84 52.68 63 44 At the beginning of the program this activity was exclusively for women since the zaï method was used on unproductive lands and the FASO gender study on women and land noted that such lands tend to be allocated to vulnerable women. But as men became more interested, they were also included in this activity. Heaps of composted manure laid on an arid and impoverished soil protected with stone bands, near Barga, in preparation of the coming farm season, (Bouroum Commune, 2 February 2017) FASO Final Evaluation 27 1.2.1a. % of targeted beneficiaries men farmers reporting increased revenue from agricultural activities Dir Ben. 22.7 (FY11) 34 69 (FY15) 203 Pop. 33.60 85 53.55 63 1.2.1b. % of targeted beneficiaries women farmers reporting increased revenue from agricultural activities Dir Ben. 20.5 (FY11) 28 61 (FY15) 218 Pop. 28.10 81 34.8 43 † As mentioned in the Introduction to Section 6, in this section we provide baseline report values as they were presented in the baseline report. IR1.2 findings: The table above shows that at FASO-beneficiary and population levels there are positive signs of increased revenue from agricultural activities. These findings, especially at the beneficiary level, were triangulated through the qualitative fieldwork and desk review, according to which the most promising activities seemed to include: i) SILC groups. These groups offer a strong springboard for access to financial resources for reinvestment, as illustrated in Figure 3 (which the evaluation team developed). However, the initiative has been facing a series of challenges over the recent years. During FY2016 some SILC groups in Tougouri, Manni, and Gayéri ceased to function, mainly because their members migrated to gold-mining sites; this contributed to a slump in the number of SILC beneficiaries. In addition, the FY16 ARR data on the creation of new groups by Private Sector Providers (PSPs) suggest that more than half of the groups are underperforming. Finally, the relatively high monthly compensation that REGIS-ER offers to FASO-coached PSPs is affecting the program’s SILC objectives in negative ways; REGIS-ER is paying SILC field agents approximately 1000 percent as much money as FASO, which affects the PSP’s motivation to doing its FASO-focused work. ii) The warrantage system. This has allowed beneficiaries quality storage and access to credit for crops stored in warehouses, as well as an opportunity to sell crops when prices go up. It was however noted that some beneficiaries ended up storing rice in the warehouses for later household consumption, in the absence of market opportunities. Note that warrantage was implemented in only in two health districts (Gayéri and Manni). In Gayéri, the activity took advantage of PM2A warehouses constructed by the program. iii) Rice parboiling groups. These groups reached just over 750 women and provided a value￾added product for marketing. Groups in FY15 reported annual per capita profits of $148 in Tougouri, $264 in Gayéri, and $344 in Manni and 18 groups accessed a total of over $60,000 in loans. However, FGDs with rice producer groups, especially in Gayéri, noted a failure to sell rice at the local market. Though parboiled rice can fetch a higher price, the market for parboiled rice was generally timid, despite some signs of promise, as women succeeded in selling a few platefuls of parboiled rice in village and area markets. Figure 3: The centrality of SILC, income￾generating activities and productive activities, and agricultural production FASO Final Evaluation 28 In terms of other activities, market gardening was highlighted especially in Manni for providing a new market opportunity with vegetable sales, though there were some reports of market gardens being abandoned, due to infrastructure default or water shortage. In addition, the Formative Evaluation report for FASO and ViM noted potential land security issues for this activity, which allows groups primarily comprised of women to produce vegetables on previously unused land.45 An important land security issue was also raised during field work in Tougouri.46 In addition, there are large and statistically significant differences for farmers reporting a year-on-year gain in revenue from agricultural activities, especially in Manni and in Tougouri. KIIs and FGDs with staff and community members in Gayéri reported that prior programs provided significant monetary incentives for people to participate in program activities, which may have biased baseline data collection there. At the program level, based on annual data and as shown in the table above, male and female beneficiaries across the program zones enjoyed a three-fold increase for this indicator (from 21.8 percent for all beneficiaries in FY11 to 65 percent in FY15). Table 19: Baseline-Endline Data for Indicator 1.2.1 disaggregated by gender Manni Tougouri Gayéri FASO-wide M F All M F All M F All M F All 1.2.1 % of targeted beneficiaries farmers (by gender) reporting increased revenue from agricultural activities Baseline 10.2 9.5 9.9 12.7 9.3 11.3 76.8 66.1 72.1 33.6 28.1 31.2 Endline 58.6 33.6 57.9 46.9 33.8 45.8 55.6 65.3 55.7 53.6 34.8 52.7 45 From p. 34: “Land owning families often welcome the development of market gardens on land of little current value, but once the gardens generate significant revenues the heads of the land owning families (usually men) are tempted to reassert their traditional ownership rights and claims on income from those lands. This is an issue that is easily neglected and needs to be adequately addressed up front before investments are made. Because these market gardens invariably are made on lands that fall under the jurisdiction of local leadership, they must also be involved in and committed to any decisions.” 46 Of the 10 market gardening sites in Tougouri, five use the semi-Californian irrigation system, and one of these does not work because of land security issues. The land plot for the site was granted to FASO by the village customary Chief for off￾season gardening. However, animals caused significant damage to the garden plants. Therefore, the beneficiaries worked with FASO to protect the site with a hedge of trees, to which the Chief initially agreed. However, in certain areas in rural Burkina Faso, the planting of a tree is perceived as a “materialization of ownership” by the person who planted the tree. In this case, at the onset of the materialization of the hedge, the Chief and members of his family then opposed the project, which subsequently brought exploitation of the site to a stop. This kind of incident is common and clearly voices the urgency of mapping pathways of land security in future programs. FASO Final Evaluation 29 Similar to the IR1.1 findings, the practices promoted under IR1.2 were also very valued, and this was generally corroborated by annual data (for rice shelling, for example, as shown in the graph). However, certain IR1.2 practices, such as hermetically storing cowpeas or using urea fertilizer for rice production, depend on being able to access materials. When FASO no longer provided subsidies to purchase triple-lined storage bags for cowpeas, the rate of usage went down (from a high of 42.1 percent in FY13 to 15.4 percent in FY16).47 However, some program beneficiaries reported using simpler and less costly alternative containers for storing cowpea, as mentioned in the detailed output-level findings for Output 1.2.2 (Annex SO1 – IR1.2 Output-level findings). During the qualitative fieldwork it was evident that program beneficiaries understand the benefits of the improved practices. But they either are not willing or are unable to purchase associated inputs at full price (based on a combination of low availability in certain locations and producers lacking purchasing power to buy the bags and fertilizer). As for the poultry activity, the evaluation team found that the VPEA services are being used, and the local poultry market is active, vibrant, and accessible, and therefore represents a market opportunity for the beneficiaries (all of whom are female) of this activity. In terms of other value chain strengthening interventions, the program has also provided access to credit and some technical assistance (particularly in terms of helping to set up basic organizational structure and operating guidelines in collaboration with SO3 activities) to lowland rice producer groups (in addition to the rice parboiling groups mentioned above), though producer group capacity is quite limited overall. Finally, the evaluation team did not find evidence of FASO implementing key activities with producer unions or groupements, although warrantage logically could be considered such activities, since farmers from different groupements get together to constitute a warrantage group. 6.1.4. SO1 Conclusions & Lessons Learned IR. 1.1. 33,375 households sustainably improve their agricultural production Conclusions: There have been important accomplishments within IR1.1, including most critically increases in total production for targeted crops at the program beneficiary level. The generally comparatively lower production levels for women are likely the result of women still having consistently lower purchasing power and access to improved inputs, less access to technical information, lower levels of literacy, higher expenses for labor costs – in short, there remains unequal access to productive means. 47 As measured by the percent of targeted beneficiaries who hermetically stored at least 100 kg of cowpeas for at least 6 months. 0 20 40 60 80 100 2012 2013 2014 2015 2016 % % of tar geted beneficiar ies shelling rice using plastic instruments Manni Tougouri Gayeri Total Figure 4: Percentage of targeted beneficiaries shelling rice using plastic instruments (Data sources: annual program data by district) FASO Final Evaluation 30 The program faced challenges related to input provision on two main fronts. First, despite some collaboration between FASO and GoBF district level MoA staff in program areas (who actually assisted the program in checking seed quality at seed fairs), there were different types of seed provision in the program: FASO offered its seed coupon scheme, the GoBF was providing more heavily subsidized seed, and other NGOs had their own different modalities. While the evaluation team supports FASO’s general approach to avoid handouts and to ensure beneficiaries provide at least something to receive inputs, it is not clear why FASO did not find ways to harmonize seed provision approaches more closely with other actors. Secondly, while acknowledging the constraints inherent in the local seed system and its management, it seems the program was overly focused on providing short-term seed access to farmers and could have done more to support the longer-term efforts of strengthening the overall seed system, including working more closely with key stakeholders and seed multipliers, as it recently started to do more of in FY16, with the aim of establishing reliable and consistent access to quality seed after FASO’s end. In addition, although one perceives the longer-term vision of the program to encourage new opportunities for food security and revenue and we recognize that lowland development was part of the 2003-2015 GoBF strategy for the development of irrigated agriculture, it can still be difficult to fully understand the program emphasis on lowland rice production, especially in some of the geographic areas and in considering the opportunity cost in terms of the efforts made to promote lowland paddy rice as opposed to promoting production of other crops in lowland or other areas (the issue of crop selection is discussed further under the IR1.2 Conclusions below). One can therefore conclude that one of the shortfalls of the IR1.1 strategy is that is was implemented in the same way across geographies without flexibility according to environmental and contextual differences. The program could have adapted activities to each local context and analyzed with key stakeholders the most contextually-appropriate interventions to introduce in each area. Outputs to IR logic: The logic of the outputs to the IR are sound, with the exception of “sustainable” improvements in production. This is especially related to the issue of input subsidies, as FASO did not seem to help producers understand that the costs of improved inputs could be compensated by increased production (either by providing more food for household consumption or greater income from sales). Farmers participated in the activities when inputs were subsidized, but adoption dropped off significantly when subsidies ended. This was the case especially in Gayéri, where FASO participants reported significant drop-offs in use of these inputs after subsidies ended but where inputs were generally available. (This is consistent with the report that the Gayéri population in general expects handouts or monetary compensation for participation in development activities, inherited from previous projects.) On the other hand in Tougouri, where shortages of subsidized inputs were common, some producers expressed their intention to buy their own inputs because FASO and government supplied inputs often arrived too late. IR. 1.2. At least 50,500 beneficiary farmers sustainably increase revenue Conclusions: In the program areas, there were notable increases in reporting increased agriculture revenue from the prior year, as discussed above. Although this type of annual comparison based on self-reporting is useful, it would have also been helpful for the program to have had other program indicators in the IPTT from the outset for this IR to help measure “sustainably increased revenues,” such as volume and dollar value of crop sales of targeted crops (total quantities and/or annual percentage change) or average prices received for crop sales of targeted crops. Based on the evaluation and information available, it is FASO Final Evaluation 31 impossible to determine with any sort of certainty the extent to which the program contributed to increased revenues in the target areas. While several activities are contributing to increased revenue for beneficiary households, the program did little to strengthen the capacities of local producer groups other than the groups producing paddy rice and parboiled rice. Moreover, the program did not have a very strong value chain or market systems strengthening approach, as output 1.2.4 (“10 producer unions and 50 village level groupements have improved their access to markets”) was the least prioritized of all the IR1.2 outputs. Outputs to IR logic: There are some gaps in the proposed logic, as partly revealed through the qualitative fieldwork. The logic assumes that if there are improvements in rice and sesame processing and if groups in these areas access greater markets, their incomes will increase. There are two primary logic gaps: 1. The fourth output regarding access to markets is more than an output, unless the program knows of one or two very basic obstacles to gaining access to the markets, such as transport. In all three districts, the factors seem inherently more complicated than that, suggesting the need for a more complex logical design looking at knowledge and skills of those groups in marketing and the market demand overall; 2. The choice of crops of sesame, rice, and cowpea, with heavy emphasis in program activities on rice, does not appear to have been based on a detailed market analysis, which is critical if the activities with these crops are expected to yield increases in agriculture revenues (e.g., in Gayéri there does not appear to be a significant market demand for rice locally). SO1 Overall Conclusions & Lessons Lear ned FASO has made important progress towards improving access to food of sufficient quantity and quality throughout the year; however, clear conclusions as to the fulfillment of the objective are difficult to make. 1. Based on the qualitative and quantitative findings it appears that households in the program ar ea have great er quantities of food available for longer periods during the year . This is also further supported by endline data showing very low population-based percentages of reported severe and moderate hunger (totaling 6.03 percent). 2. Changes in household dietar y diversity score ar e inconclusive. In terms of the SO1-level indicator, HDDS, there is no statistically significant change in the baseline and endline values (comparing a recalculated baseline value proportionate to district size of 3.36 with the endline value of 3.28), and the endline value did not reach the target of 6 (out of a scale of 12). While there were reported increases in revenue from agricultural activities when comparing baseline and endline values and some SO1 activities (e.g., market gardening, poultry, sweet potato production) have promoted dietary diversity and were seen as successful and relevant interventions, the number of beneficiaries was relatively low in comparison with many of the other SO1 activities and the program still struggled overall to succeed in facilitating greater access to a diverse diet in the target areas. 3. FASO’s intentional promotion of women’s par ticipation in SO1 activities is a significant success (but work still remains). The efforts FASO made to ensure SO1 activities benefited women were immense and are laudable. Many of the SO’s key interventions – such as SILC, market gardening, parboiled rice and producer groups, promoting cowpea production, poultry raising, the zaï technique and lowland improvements – were introduced primarily with female beneficiaries in mind. The use of the zaï technique also opened venues for women to access more farm lands, as men found it easier to pass those less fertile plots to women. As a result, women increased production and revenues. However, a look beyond the surface of women’s impressive numerical participation reveals insightful FASO Final Evaluation 32 subtleties on unveiled inequities and household dynamics worthy of interest. Women’s overcrowded schedules and competing priorities on limited financial budgets do not always allow them to optimize resources and labor in some of their personal farm endeavors as required, which affects their yields and subsequent revenues. Many women resort to interesting, though informal, negotiation strategies within their households, in their quest to maintain the benefits of harmonious household relations that do not frustrate the male ego of their head of household within the context of a “hegemonic masculinity.” 4. From the onset, the program placed greater emphasis on production than on commercialization and mar keting. While some efforts were made with paddy rice and parboiled rice producer groups to strengthen their capacities and link them to loans with MFIs, overall the program spent more effort and resources on production-related activities. This was even evident in terms of measurement; for IR1.2 all that was measured was beneficiaries self-reporting on whether or not they increased their agricultural revenue from the previous year. As a result, the program did not include a holistic approach to value chain/market systems strengthening or capacity building of local producer groups for crops beyond rice/parboiled rice. However, during the extension period the program has made more efforts to promote the marketing component, including hiring a marketing specialist on the core FASO team. 5. Use of Title II food comm odities for reclaiming degraded lands was appropriate. Providing food via a FFW scheme to rehabilitate degraded lands and for conservation agriculture with trees was a good use of food commodities, and it was impressive that some communities continued the same types of work even without food. In certain places it appeared in retrospect that it may have been less the lowlands per se that interested beneficiaries than the access to food via FFW and subsidized inputs, as evidenced by the decline in lowland HAs exploited for rice production. It is not, however, possible to know this for certain. 6. The approach to input provision was not very successful, in spite of program efforts. FASO struggled in some ways with how best to provide inputs such as seeds, tools, and other inputs (storage sacks, commercial fertilizers, etc.) to beneficiaries. The activities provided beneficiaries with short￾term access to inputs, and the program did make a number of efforts to address longer-term challenges (e.g., through: the introduction of the TMCs, initiating linkages between wholesalers and small traders; pursuing support to emerging groups of small scale local seed producers; etc.). Such efforts on the whole did not, however, yield the expected overall results in terms of sustainably improving access to inputs for smallholders in the target areas. From the outset of the program it did not appear that there was a clear and cohesive plan for how best to strengthen the seed system and the local input supply systems, given the challenges such systems faced, (though some producer groups were able to build on their rewarding collaboration with the program to develop pathways for sustainable access to inputs). Key Lessons Learned: - It is critical to work with government and other key stakeholders to try to analyze and address holistically seed system and other input system challenges and issues, so as to use a comprehensive, strategic, and long-term approach. - Agriculture programming can successfully ensure women’s participation and benefits for women through careful selection of activities that draw from the cultural and sub-cultural contexts to create safer spaces for women and their households. Future interventions would gain by having a strategy that sets gender-transformative targets which enhance the economic and social empowerment of women and relevant vulnerable social identities in due consideration of the local and sociocultural FASO Final Evaluation 33 contexts. In the short and medium term, such targets could consider addressing inequities in resources access and labor allocation, levels of literacy and women’s leadership and standing in groups and society. - It is important to have a multi-sectoral strategy with associated means of measurement for agriculture and nutrition to address dietary diversity. - Flexibility in terms of initiating new activities based on an evolving understanding of the context and needs can be extremely useful, as FASO did with the introduction of the warrantage, poultry, and market gardening activities. - To maximize promoting market access a program needs a clear and strategic approach to strengthening value chains and market systems for food insecure households. 6.1.5. SO1 Recommendations 1. Seed and Input Supply Systems. Based on a thorough context analysis, work with the host-country government and other key stakeholders to ensure a comprehensive strategy for strengthening seed and input supply systems that includes a clear plan for what levels of support and subsidy to provide and how those will change over time, as well as how sustainability will be built into the approach. Such a strategy must also ensure that the fertilizer and triple-line bag companies are equipped and able to supply and respond to farmers’ demands. 2. Gender. Address underlying inequities for women through integrating activities such as: functional literacy and numeracy classes focused on agriculture and marketing; developing leadership skills and women’s leader networks; creating staff positions focused on ‘gender and agriculture’ to pursue a more transformative gender approach; incorporating household gender dialogues into ongoing program-supported groups (SILC, producer groups, etc.); and advocating for women and vulnerable men’s access to land plots. 3. Marketing Production. Use a value chain and market systems approach to analyze opportunities for local producers and to introduce a more intentional approach to strengthening producer group capacities, especially in financial and organizational management, business planning, and technical skills. This should also include ways to measure concrete change in terms of market access and producer group capacities. 4. Dietary Diversity. Scale up efforts at dietary diversity and ensure there are annual indicators to measure associated change in program beneficiaries. This can be done through initiatives such as: marketing campaigns for orange-fleshed sweet potatoes and market garden vegetables to strengthen demand within the population; promoting widespread household vegetable production for home consumption; and understanding micronutrient deficiencies and needs in program areas and introducing a more detailed locally contextual nutrition/micronutrient lens into selection of a program’s targeted crops. 5. Crop Selection. Select priority crops based on a combination of: market potential, nutrition, and gender-friendliness. Include current lucrative opportunities that seem to be common and rewarding practices in the targeted areas, such as peanut production and small ruminant raising. 6. Risk and Contextual Analysis in Design. Carry out context analyses that present the sociocultural subtleties, strengths, weaknesses, opportunities, and potential threats, prior to intervention. Ensure FASO Final Evaluation 34 risks associated with introduction of new types of activities are appropriately considered across all program areas. For instance, such a contextual analysis would inform that while cowpea is clearly a women’s crop in one health district, peanuts is a more women-friendly crop in another. It might also reveal that rice production in lowlands areas is appropriate in some areas, whereas a focus on other crops in lowlands or other lands would be more appropriate in other areas. While the program did adapt during implementation to ongoing learning (e.g., through introducing market gardening), one of the shortfalls of the original design for the production activities is that it was viewed in the same way across all of the program areas without flexibility according to geographic and contextual difference. In the future, programs should take the time required to conduct risk and contextual design prior to activity implementation and, as needed, adapt activity strategies to the location accordingly. 6.2. SO2: Health/Nutrition and WASH 6.2.1. Summary of FASO SO2 Design48 Table 20: Summary of FASO SO2 design SO2 51,426 mother/child units have improved their health and nutr ition IR2.1 18,500 mothers of children 0-23 months and pregnant women adopt recommended health, nutrition and hygiene practices Output 2.1.1 18,500 caregivers of children 0-23 months have the recommended health, nutrition and hygiene knowledge Output 2.1.2 26,230 caregivers of children 0-23 months develop positive attitudes for appropriate health, nutrition and hygiene practices Output 2.1.3 Hygiene and sanitation conditions are improved in 148 communities Output 2.1.4 Influential people at the community level support pregnant women and mothers of children 0-23 months in adopting recommended health, hygiene and nutrition practices Output 2.1.5 25,700 mothers and children consume food rich in micronutrients IR2.2 79,590 women and children 0-59 months access quality nutrition and health services Output 2.2.1 38 health centers provide quality services for preventing and treating malnutrition of women and children Output 2.2.2 248 communities facilitate women’s access to health care 6.2.2. Introduction to SO2 Activities During the first phase (through May 2015), the program implemented the Preventing Malnutrition of Children under the age of Two Approach (PM2A) in Tougouri (led by OCADES) and Gayéri (led by HKI) as a strategy to: i) improve the health and nutrition status of Pregnant and Lactating Women (PLW) and children under the age of two and ii) to strengthen the quality and delivery of health care services. Key activities in these two districts included Care Groups to promote and provide support for household and community level behaviors that prevent maternal and childhood illnesses. Care Groups are composed of pregnant women and mothers of young children; MLs, supported by promoters, are responsible for groups of 10-15 women with whom they meet regularly in group and home settings. Additional activities during the first phase included cooking demonstrations, monthly distributions of Title II commodities to improve 48 Source: FASO Results Framework, as of January 2016. FASO Final Evaluation 35 the caloric and nutrient intake of mothers and children under two, and training and supervision activities for MOH staff at health centers. In Manni, where GRET led the SO2 component during the first phase, no Title II foods were distributed in this health district as GRET was promoting a fortified flour product using locally available ingredients as a supplementary food for young children. GRET implemented the SBCC activities through village￾level field agents and committees (instead of MLs) and participated in the capacity-strengthening activities for MoH health agents. Starting in the second year, the FASO program began to support improved water, sanitation and hygiene (WASH), a welcome programmatic addition that reinforced the health and nutrition activities and responded to a need felt on the part of the population. WASH activities initially included rehabilitating non-functioning boreholes and drilling new boreholes. In the third year the Community-Led Total Sanitation (CLTS) approach was progressively introduced and now includes latrine construction and sanitation promotion in 85 villages in the three health districts. In the extension period three programmatic changes occurred for SO2. First, the leadership for implementing SO2 in Manni and Gayéri changed with ATT assuming oversight of health, nutrition and WASH in these two health districts; OCADES continued as the implementing partner for SO2 in Tougouri. Second, there is no longer any distribution of Title II commodities for supplementary feeding. Third, in Manni the FASO program is continuing the WASH component in all three communes, but the other SO2 activities are only being implemented in one commune, Thion. In Thion FASO is piloting GASPA, a new strategy initiated by the Government of Burkina Faso (GoBF) to promote positive behavior changes in Infant and Young Child Feeding (IYCF). Under this initiative, Community Health Workers (CHWs) are trained in IYCF and they in turn meet monthly with pregnant women and mothers of children up to two years of age. The message content is basically the same as that used in the Care Groups but CHWs fill the role of MLs and promoters. 6.2.3. SO2 Findings SO2: 51,426 mother/child units have improved their health and nutr ition stat us Table 21: SO2 indicators Indicator Baseline† Final Target Endline Survey Result 2.1. Prevalence of stunted children under 5 years of age N/A†† 17.5 32.57 2.2. Prevalence of underweight children under 5 years of age N/A†† 23.7 20.46 † As mentioned in the Introduction to Section 6, in this section we provide baseline report values as they were presented in the baseline report. †† A baseline value is not available given that anthropometric measures were not taken for children 24-59 months during the baseline, as discussed in detail previously in Section 5: Baseline/endline comparison in this report. However, for underweight 0-59 month olds, a proxy point of comparison is from the SMART survey from 2009-10, which found 32.7 percent of children 0-59 months were underweight nationwide. FASO Final Evaluation 36 Table 22: Stunted and underweight children under 2 years of age Indicator Baseline Report Final Target Endline Survey Result Prevalence of stunted children under 2 years of age 26.5 N/A 27.58 Prevalence of underweight children under 2 years of age 38.5 N/A 22.89 Tables 21 and 22 above show the results for the SO2 impact-level indicators (stunting and underweight); comparisons of baseline and endline results were only possible for 6-23 month olds for stunting and 0-23 month-olds for underweight. Although there is not a statistically significant difference for stunting in the 6-23 month olds, there is a statistically significant (at 95 percent confidence level) reduction of nearly 16 percentage points in underweight children 0-23 months of age. In addition, the prevalence of underweight children 0-59 months at the endline survey was 20.46 percent, which exceeded the endline target of 23.7 percent for this indicator. For stunting, the endline target for children 6-59 months was 17.5 percent, but this was not met as the endline value for this indicator is 32.57 percent. Without additional data, it is not possible to determine why the stunting rate did not decline for the 6-23 month-olds and why the stunting target for 6-59 month-olds was not met. Among the possible explanations are poor maternal nutritional status, especially during pregnancy, and non-food causes of stunting such as enteric enteropathy. And in retrospect, an endline target of 17.5 percent for stunting seems unrealistic given that the FASO program document states that the stunting rate for the three health districts prior to the program start was over 40 percent, a figure based on the 2009 Etude Nationale sur l’Insécurité Alimentaire et la Malnutrition (ENIAM). IR2.1: 18,500 mothers of children 0-23 months and pregnant women adopt recommended health, nutrition and hygiene practices Tab le 23: IR2.1 indicator s Indicator Source Initial Data† Final Final Achieved % Final Target Achieved 2.1.1 % of infants 0–5 months of age who are fed exclusively with breast milk (FFP)/Rise Dir Ben. 62.4 65 83 128 (FY12) (FY16) Pop. 40 60 56.21 94 2.1.2 % of households using the available (improved) sanitation facility (IEE) Dir Ben. 34.5 60 48 80 (FY12) (FY16) Pop. 22.6 32.6 42.28 130 2.1.3 % of children 6-23 months of age who receive a minimum acceptable diet (apart from breast milk). (FFP)/Rise Dir. Ben. 26.5 (FY12) 20 41 (FY16) 203 Pop. N/A 30 7.90 26 2.1.4 % of children aged 0-23 months with diarrhea in the last two weeks whose mothers sought outside advice or treatment for the illness Dir Ben. 77.8 85 85 100 (FY12) (FY16) Pop. 54.80 70 63.99 91 2.1.5 % of mothers of children age 0-23 months who had four antenatal visits when they were pregnant with the youngest child Dir Ben. 25.5 70 61 87 (FY12) (FY16) Pop. 35.57 70 50.22 72 2.1.6 % of mothers of children 0-23 months who had a postnatal visit within 45 days of delivery Dir Ben. 35.9 90 85 94 (FY12) (FY16) Pop. 22 90 29.94 33 FASO Final Evaluation 37 † As mentioned in the Introduction to Section 6, in this section we provide baseline report values as they were presented in the baseline report. IR2.1 findings: As shown in Section 6.2.1: Summary of FASO SO2 Design above, IR2.1 is broken down into five outputs; these five outputs, representing pre-conditions to achieving the adoption of improved practices, are: knowledge levels, positive attitudes for changing practices, improved hygiene and sanitation conditions, community support, and the consumption of foods rich in micronutrients. Four of the five outputs have been achieved and progress has also been made on the fifth output, consumption of foods rich in micronutrients. Targeting for IR2.1 focuses primarily on pregnant women and mothers of children under two and to a more limited extent, on husbands and fathers. One group, however, that was not included as a key target group was male and female adolescents, a consideration for future programs. Detailed output-level findings are presented in Annex I.2: Output-level Findings for SO2. Overall, we found impressive changes in knowledge levels and declared adoption of better health and nutrition practices during the qualitative field work. These findings were corroborated by the results of the quantitative survey when the baseline values (recalculated to be proportionate to district size) and endline values were compared, as we found statistically significant and sizeable improvements for a number of IR2.1 indicators, including: These findings provide compelling evidence for FASO’s focus on WASH, and promotion of other health and nutrition practices. Based on the results of the FGDs and KIIs conducted during the qualitative fieldwork, it is clear that caregivers, including some men, have a high level of knowledge concerning the recommended health, nutrition and hygiene practices. And given their active participation in the FGDs, their conviction that change was here to stay, their enthusiasm for the Care Groups and GASPA, and the deep appreciation they expressed for the MLs, the evaluation team concluded that the caregivers interviewed have developed the positive attitudes needed to institutionalize change in their own households and to promote change among their peers. It is no surprise that MLs have been so successful in promoting behavior change as the Care Group model is known worldwide for its effectiveness. MLs are an invaluable resource for the community and for the health system. In the latter half of the program, the FASO team has made a concerted effort to show how much MLs are valued and to encourage them in their work. This has included prioritizing them for SO1 activities (including income-generating opportunities) and organizing celebrations for all MLs. And Indicator Percentage point increase % of households using improved sanitation facilities 17.8 % of mothers of children 0-23 months who know at least 2 danger signs of childhood illnesses that require treatment 16.4 % of households with access to an improved sanitation facility (latrines) 21.1 % of mothers of children aged 0-23 months who consumed food rich in Vitamin A in the last 24 hours 54.2 % of children aged 6-23 months who consumed locally fortified flour as complimentary food 9.45 FASO Final Evaluation 38 although it has only been in place for a little over a year, the GASPA initiative also seems to be a very effective method for promoting change. The FY16 annual survey results from Manni, where CHWs are implementing GASPA, are better than the results from Gayéri and Tougouri across almost all indicators. Whether implemented through Care Groups or through GASPA, the SBCC approach is a m ajor factor in the positive IR2.1 results. The approach has included a number of innovations. For example, during the first phase in Manni FASO trained religious leaders (pastors, priests and imams) in IYCF and then carried out nutrition promotion activities before, during or after the weekly religious services. In Manni FASO also trained literacy instructors (established through SO3 activities) in health and nutrition and assisted them with incorporating these themes into their literacy classes. Also in Manni FASO set up stands in markets and other public venues and used games and contests to attract listeners. Although there was relatively little formative research to inform the SBCC activities for SO2, FASO did conduct a barrier analysis in 2014 in all three districts, looking at IYCF and maternal nutrition (exclusive breastfeeding, diverse diet for children aged six months and older, enriched porridges for children, and dietary diversity for pregnant women); the results were used to sharpen messages on these topics. The SO2 team has also been able to conduct SBCC activities with SO1 beneficiaries in SILC and producer groups to further disseminate health and nutrition messages. As for SBCC topics, women respondents in three of the four Care Group FGDs asked for more information on fam ily planning. The nurse interviewed at the Tiboudi health center also mentioned that more needs to be done to inform couples about the advantages of birth spacing.49 As expressed by beneficiaries, community leaders and MoH counterparts during the FGDs and KIIs, WASH is among the most successful and appreciated components of the FASO program. In addition to increasing access to potable water and improved sanitation facilities, the WASH team is currently piloting community-managed sanitation shops (boutiques d’assainissement) to make latrine construction supplies readily available to those interested in having their own household latrine. FASO provides an initial stock of materials such as cement and rebar at competitive prices; when the stock is depleted the management committee buys new materials using the proceeds to pay for the materials and the transport. IR2.2: 79,590 women and children 0-59 months access quality nutr ition and health services Table 24: IR2.2 indicators Indicator Source Initial Data† Final Target Final Achieved % Final Achieved 2.2.2 % of mothers of targeted children 0- 23 months who gave birth in the health center Dir Ben. 90.3 (FY12) 90 92 (FY16) 102.2 Pop. 62.9 80 84.06 105.1 † As mentioned in the Introduction to Section 6, in this section we provide baseline report values as they were presented in the baseline report. IR2.2 findings: In the program framework, IR2.2 is broken into two outputs. The first aims to increase the quality of the health centers’ services, especially those related to nutrition. During the first phase the FASO team in collaboration with the MoH carried out a number of activities to improve the quality of health center services for preventing and treating malnutrition. These activities included trainings for health center staff and CHWs, especially in IYCF; monitoring visits conducted by FASO staff and MoH 49 For additional information on SBCC, see the final section of the report that assesses the evaluation objectives, questions, and specific objectives. FASO Final Evaluation 39 personnel, usually the CMO and/or members of his team; and quarterly malnutrition screening of children, carried out jointly by FASO staff and health center personnel. In Gayéri during the first phase FASO went a step further with the Health and Nutrition Supervisors regularly carrying out formative supervisions at the health centers. These supervisions were day-long visits and consisted of observations, providing on￾the-job refresher training and assisting the often-understaffed health center with consultations, thereby modelling good techniques and current MoH protocols. Since the extension phase started, there are fewer activities directly related to improving the quality of services at the health centers. FASO program managers cited budget constraints and competing programmatic priorities as reasons. In addition, a number of FASO staff noted in KIIs that the frequent transfer of MoH staff outside of the three FASO districts means that investments in capacity-building do not always result in better-trained agents at the health centers since replacement staff would also need to be trained. Although the reasons evoked for this shift in focus are understandable, it has not gone unnoticed among MoH colleagues. During KIIs with Head Nurses and with the Gayéri CMO, it was evident that they valued the close working relationship they had previously enjoyed with the SO2 staff. The second output focuses on increasing women’s access to health care. As shown by the survey data presented in Table 23 above (specifically indicator 2.1.5 and 2.1.6) and Table 24, the FASO program has had a very positive impact on the utilization of health center services for program beneficiar ies: i) services for four antenatal consultations increased for beneficiaries from 25.5 percent in FY12 to 61 percent in FY16; ii) post-natal care increased for beneficiaries from 25.5 percent in FY12 to 61 percent in FY16; and deliveries at the health center were consistently high for beneficiaries as indicator 2.2.2 shows above. Moreover, at the population level, we found statistically significant baseline-endline increases (when comparing the baseline values recalculated proportional to district size) for the following: 22.7 percentage points for the percentage of mothers of children 0-23 months who delivered in the health center; 9.15 percentage points for mothers attending a postnatal visit within 45 days of delivery; and 8.3 percentage points for mothers of children aged 0-23 months who declared being supported by their community in access to health care. Initially, in Gayéri and Tougouri this increased utilization of services was directly related to the distribution of Title II foods as pregnant women needed to attend antenatal consultations in order to be on the distribution list. MoH personnel and FASO staff interviewed during the qualitative fieldwork stated that attendance at maternal and child health consultations soared once FASO began, and attributed much of the increase initially to the incentive of food distribution and later to the fact that women discovered the services available to them and the benefits of using them. Two of the three Chief Medical Officers (CMOs) showed the team data from annual reports confirming a steady increase in utilization of services. Husbands supported women going to the health centers because the food was an incentive for them as well. Although there was a slight downturn in attendance when the food distribution ended, the habits and practices adopted during the first phase were largely continued. SBCC activities also likely resulted in the increased utilization of services since the increase in utilization of services also holds true for Manni where there was no food distribution. FASO Final Evaluation 40 During the extension phase there have been a number of positive developments as the program has evolved; the following examples from the qualitative fieldwork illustrate the changes put in place to reinfor ce linkages with the MoH system and to ensure susta inability: - In Tougouri and Gayéri, the MLs have assumed responsibility for conducting routine screening for malnutrition in children, freeing up the health center staff; - MLs now use a referral system (with slips of colored paper corresponding to the screening armband colors) to refer suspected cases of malnutrition to the health centers; - The FASO program is working more closely with CHWs. This includes: i) the CHWs in Thion Commune (Manni) who conduct the GASPA groups and ii) plans to transition oversight of MLs in Tougouri and Gayéri from the promoters to CHWs; - FASO has signed contracts with each health district for a nine-month period (January – September 2017), complete with a budget. Under this contract the district health team will organize IYCF training for CHWs; supervise Care Groups and GASPA; advocate for the inclusion of Care Groups into the MoH’s overall strategy for improving IYCF; and oversee the training and maintenance of community scoreboards,50 which are a recent addition to the SBCC activities for SO2. - FASO is also working to ensure that health districts incorporate FASO program activities into their annual action plans on an ongoing basis. 6.2.4. SO2 Conclusions & Lessons Learned IR. 2.1 18,500 mothers of children 0-23 months and pregnant women adopt recommended health, nutrition and hygiene practices Conclusions: The qualitative evaluation team found highly motivated women who demonstrated a commitment to continuing the improved health, nutrition and hygiene practices they have adopted; a cadre of approximately 4,700 community-based volunteers including MLs, promoters and CHWs who are dedicated to continuing their work as change agents at the community level; much-improved access to clean water and sanitation facilities; and communities that seem committed to supporting women in their efforts to improve family health, especially for pregnant women and young children. Overall, the qualitative and quantitative results show that IR2.1 has been achieved with convincing results that are likely to be sustained. Care Groups and GASPA are both effective in the FASO program and both promote solidarity and build social capital among women who participate. Each approach has its strong and weak points. For example, GASPA works through the CHWs who are attached to the local health center. As such, CHWs have official status and are in an excellent position to reinforce linkages between the health center and the community. However, they have multiple duties and GASPA may not always be their first priority. The Care Group approach, on the other hand, relies on female volunteers who are mothers themselves and role models. The Care Group approach definitely helps women develop leadership skills 50 Community scoreboards are large permanent structures made of cement that depict MCH messages and provide information on the community’s performance in EBF, antenatal visits and child malnutrition. Using wooden markers, the community and health center work together to periodically update the data, providing an opportunity to reinforce their collaboration. Example of a highly visible community score￾board for the village of Taffogo. Community scoreboards typically convey information about child malnutrition and births in the health center. (Taffogo, 2 February 2017) FASO Final Evaluation 41 and encourages women to have a voice in community affairs, an advantage over GASPA. However, sustaining the volunteers’ motivation over the long term can be a challenge. Outputs to IR logic: The outputs to IR design for IR 2.1 is logical in that the necessary and sufficient elements to achieve changes in practices are included in the five outputs: knowledge, positive attitudes, improved hygiene and sanitation conditions, community support, and the consumption of foods rich in micronutrients. However, achieving output 2.1.5 (consuming foods rich in micronutrients) will remain a challenge during certain times of the year when these foods are not available unless activities such as gardening or better methods for food transformation and storage can be expanded. IR. 2.2 79,590 women and children 0-59 months access quality nutr ition and health services Conclusions: The IR 2.2 activities have without a doubt increased the utilization of maternal and child health services at the health centers and to a more limited extent, improved the quality of those services. Increased utilization of services was initially influenced by food distribution in two districts and by a successful SBCC approach across all three districts. It is noteworthy that women continued to utilize the health services at high rates even after food distribution ceased. However, FASO missed an opportunity to document in a systematic and rigorous way the effectiveness of the use of food by not setting up a learning agenda to compare results between districts where Title II commodities were distributed and the district where there was no distribution. Although IR2.2 has been partially achieved, in retrospect the FASO team could have reviewed and revised this IR and its outputs when it became evident that they could not control all the factors necessary to sustainably improve the quality of services. Notwithstanding this observation, there have been a number of important achievements that are likely to continue. For example, families now have more community￾level resources and support for improving maternal and child health and for preventing malnutrition: MLs; promoters; peer support for women through Care Groups and the GASPA groups; CHWs who have been trained in IYCF; supportive spouses and community leaders; monthly screening of children for malnutrition at the community level; and a fledgling referral system for malnourished children. And as described in the preceding section, new developments during the extension phase will enhance the potential for sustainability of these gains and reinforce ties with the formal health system. Outputs to IR logic: There is an inherent flaw in the program design for IR2.2 as the two outputs are necessary but not sufficient to ensure quality services. Many factors are involved in improving the quality of services, including services for preventing and treating malnutrition. Based on the evaluation team’s experience with similar programs for improving the quality of care, one needs at a minimum: Mother Leaders with shirts saying "I am a Mother Leader. I support women and children's health in my community." (Bartiebougou, 10 February 2017) FASO Final Evaluation 42 - Availability of a full complement of well-trained, motivated, and incentivized health agents at each health center; - A welcoming environment; - Adequate equipment, materials and supplies including Ready-to-Use Therapeutic Foods (RUTF) and drugs for treating other conditions exacerbating malnutrition; - Adequate budget for the district team to monitor the health centers; - A functioning referral system from the community to the health center level and on to a higher level of care; and - Year-round physical accessibility of the health structure. Obviously, some of these factors are beyond the control of a program such as FASO. However, the formative supervision introduced in Gayéri’s health centers was much appreciated and likely improved the quality of services. If an external program wants to provide this type of support to the MoH, there are other conditions to be met such as qualified program staff who have more experience than the health center staff they are mentoring; the full support of the CMOs, the Head Nurses and MoH managers at higher levels; training modules and a system for measuring improvements (e.g. pre- and post-tests); and an adequate budget. SO2 Overall Conclusions & Lessons Lear ned Overall, the FASO program is well on its way to achieving impressive and sustainable results for SO2, especially where IR 2.1 (adoption of improved practices) is concerned. Much has also been accomplished for IR 2.2 (accessing quality health care services). Given the high rates of improvements in knowledge levels, the reported behavior changes in adopting recommended health and nutrition practices, the increased use of health facilities, the supportive environment for women seeking to improve their family’s health, and the community-level resources now available for health, hygiene and nutrition, it is logical to conclude that SO2 has been lar gely achieved, resulting in improved health and nutrition for mother/child units participating in the FASO program. Key factors that contributed to this achievement include: 1. This SO and its activities were appropriat e to needs prioritized by beneficiar ies. Improved health and nutrition and better access to potable water and sanitation facilities are high on the list of priorities, especially for women who have primary responsibility for the family’s health. 2. In the first phase the distr ibution of Title II food comm odities was a definite factor in encouraging participation in SO2 activities in Tougouri and Gayéri. 3. Both the Car e Group approach and the GASPA initiat ive as well as SBCC activities have been very successful in increasing knowledge levels and promoting sustained changes in practices. 4. The addition of the WASH component, especially the rehabilitation of old boreholes and the drilling of new ones, has made the FASO program even more attractive to the target communities and likely improved health outcomes. And since women need to spend less time finding water, they have more time for childcare and more flexibility to participate in program activities, such as SILC and producer groups, and to be more active in their communities. 5. Including older women as MLs and directly involving community leaders are two actions that resulted in strong support for the SO2 activities. Older women have a strong influence on younger women and community leaders are in a good position to persuade husbands to support their wives who participate in SO2. The engagement of older women and community leaders helped to create a certain level of peer pressure so that practices such as exclusive breastfeeding, delivering at the health center and attending four antenatal consultations became the norm. However, there was a real missed opportunity to engage adolescent boys and girls. FASO Final Evaluation 43 Key Lessons Learned: - Periodically review the Results Framework to ensure that activities and associated program outputs are contributing to outcomes and objectives and revise the framework when it becomes evident that a major objective or goal cannot be met even if the proposed activities and outputs are achieved. - Ensure that diverse, nutrient-rich foods are available and accessible if the program is promoting an adequate and diverse diet for PLW and young children. - With over 4,000 MLs as well as numerous women represented on community committees and groups in other sector activities, there is a natural opportunity to include concrete initiatives to strengthen women’s leadership skills. Future programs can invest in reinforcing women leaders through activities such as: mentoring, classes on leadership and public speaking, creating networks of women leaders from different SOs, etc. Challenges for future food security programs with a health/nutrition component remain: How can a program help to improve the quality of services at health facilities when resources, especially services for preventing and treating malnutrition, are limited? How can a program ensure that the necessary ingredients for an adequate, diverse diet are available year-round? How can a program engage the MoH at the health center and higher levels in a cost-effective and sustainable way? In spite of these challenges, there is high potential for sustainability of both the positive impact and the activities of SO2. Knowledge gained through SBCC activities has become ingrained and new practices for improving health have become a reflex. The use of health center services is likely to continue and communities involved in CLTS seem to be making steady progress toward better sanitation. Through FASO’s work with WUAs, there are strong indications that most communities will be able to maintain their boreholes and ensure that potable water remains potable from the borehole to the home. And last but not least, MLs, CHWs and community leaders seem committed to continuing in their role as change agents. 6.2.5. SO2 Recommendations 1. Lear ning agendas. If a food security program is distributing food in some zones and not others, establish a learning agenda to determine whether the food has any effect on nutritional status and other program performance indicators. This should include quantitative indicators to track as well as a limited number of questions for qualitative purposes. 2. Gender. Develop strategies to more fully engage men, including adolescent males, in health and nutrition activities. Initiatives that have proven successful in similar programs include the “Schools for Husbands” activity sponsored by the United Nations Population Fund (UNFPA) in Niger and other countries,51 and Father Leaders, modelled after the ML approach. 3. Adolescents. Future programs with gender and health components should more directly include adolescents since they are future parents and leaders and also have special health concerns. As the USAID/FFP Burkina Faso Food Security Country Framework FY2010-2014 document recommended, targeting adolescent girls in particular is important given that: maternal mortality rates were reported to be at least double for adolescents 15-19 years of age; adolescent girls have the least 51 www.unfpa.org/news/schools-husbands-gaining-ground-rural-niger FASO Final Evaluation 44 decision-making influence of all women; and early marriage of girls is common, especially in rural areas. 4. WASH. a. Continue to emphasize the WASH component. Both potable water and adequate sanitation facilities are essential for improving health in a food security program. In addition, having an accessible supply of clean water allows women and girls to spend less time finding water and more time on education, childcare, IGAs and participation in community affairs. b. Carefully document the experience of piloting the sanitation shops. If the sanitation shops prove to be successful, incorporate them into future food security programs that have a WASH component. 5. Fam ily planning. Since health centers do have a range of family planning methods available, future food security programs should include family planning in at least the SBCC activities. If the implementing partners do not have the capacity to include additional activities, partner with an organization in Burkina Faso that has this capacity such as Marie Stopes International. 6. Improving the quality of health and nutr ition services. If a program increases the demand for services at health facilities, there should be a quid pro quo for the health facilities to ensure that the quality of services does not suffer from the increased workload. This could range from providing equipment and materials to formal training to supportive supervision, depending on needs and the program’s resources. If supportive supervision is included in a program, this would require qualified program staff, the full cooperation of the MoH at all levels, and an adequate budget. 7. Retaining community volunteers. a. The FASO program has managed to retain the majority of MLs and promoters, most of whom are now entering their seventh year as volunteers. In future programs, it will be important to develop retention strategies before the program begins. Successful strategies used in the FASO program include opportunities for training, ML appreciation days, providing materials for their activities, and perhaps most important for long-term retention, prioritizing MLs for other program activities such as SILC and agriculture. b. Organize cross-visits between ML groups as a way to promote learning and exchanges. c. Develop similar retention strategies for other community volunteers such as promoters and hygienists. At a minimum, provide each volunteer with an official certificate to include details on dates of service, location and his/her major responsibilities. 6.3. SO3: Governance 6.3.1. Summary of FASO SO3 Design [Note that the design is not shown in numerical order to better reflect chronological and logical order] Table 25: Summary of FASO SO3 design52 SO3 Community-based organizations in at least 100 villages are using principles of good governance to make strategic decision regarding sustainability of community investments and food insecurity reduction IR3.2 1600 women occupy leadership roles in their communities [only through FY2015] 52 Source: FASO Results Framework, as of January 2016 (unless stated otherwise). FASO Final Evaluation 45 Output 3.2.1 30,561 girls are enrolled in preschool and primary school [only through FY2015] Output 3.2.1 45,000 primary school students and 3,000 pre-school children per year have received mid￾day meals through school canteens in 340 schools and 41 preschools. Output 3.2.2 300 PTA/AME/CVD support girl’s education [only through FY2015] IR3.1 100 VDCs are coordinating the development and implementation of community development action plans Output 3.1.1 100 VDCs have expanded capacities for coordinating planning and implementation of community development activities using principles of good governance Output 3.1.2 2000 leaders from VDCs and other CBOs participating in FASO Program activities have expanded functional literacy capacities IR3.4. Internal governance among community structures supported by FASO reinforced Output 3.4.1 Sustainably self-managed investment IR3.3. Increased inclusion of communes in the process of sustaining WUA and VDC activities IR3.5 Community access to service providers improved 6.3.2. Introduction to SO3 Activities The results framework, activities and approaches of SO3 have shifted several times during the program, as is illustrated in Annex D.2: FASO Results Framework History, Annex D.3: FASO Results Framework (FY11, FY15, FY16 Comparison, and Annex D.4: Shifts in FASO SO3 Results Framework). In summary, the activities associated with SO3 are largely focused around: i) school feeding to increase enrollment and attendance of primary school students, especially girls (via school canteens and take-home rations for girls); ii) training and technical support to community committees, with emphasis on WUAs and VDCs, but also other groups; and iii) more recently, supporting linkages between communities and private sector service providers as well as supporting other types of groups (e.g. PTAs, MAs, TMCs, management committees of rice producer groups and market garden group management committees, etc.) to improve their capacities to sustain the investments made under FASO. This section focuses on evaluating the program against expected results; however: i) the IRs are addressed in a different sequence to better reflect logical and chronological order and ii) design issues are discussed separately, as all activities do not fit neatly within their IRs. 6.3.3. SO3 Findings In general, the findings for SO3 reveal that community members and beneficiaries in SO1 and SO2 activities noted positive improvements in the functioning of community committees and genuinely seemed to value these improvements. However, there have been so many changes in the SO3 design that it is difficult to assess fulfillment of what was planned. Moreover, as discussed below under the IR findings, at no time during the program does the SO3 design appear to have been logically clear and tight. The approach of the extension phase to focus on areas which can improve the sustainability of program investments related to food security is promising, but the individual components do not have documented strategies (this is somewhat understandable given that they were expected to be one-year interventions as part of an exit strategy but they will now become two-year interventions with the additional extension year). The lack of clear documented strategies makes it difficult to assess the SO’s logic and expected results. And while there appears to be strong and capable staff members who are adapting well to the evolution of SO3 activities, this is different from executing a clear and logical strategy. SO3: CBOs in at least 100 villages are using principles of good governance to make strategic decisions regarding sustainability of community investments and food insecurity reduction FASO Final Evaluation 46 Table 26: SO3 indicators Indicator Source Initial Data FY16* Final Target Endline Result 3.2. Percentage of targeted community committees (i.e., CBOs, including CVDs) applying at least 3 of 4 good governance principles Annual 98 76.6 81.91 3.2 [alt.] Percentage of targeted community committees (including CVDs) applying at least TWO of four good governance principles [calculated for comparison with baseline, but not consistent with FY2016 IPTT] Baseline/ endline 55.5 95.74 3.3. Percentage of community structures sustainably managing investments Annual 36 50 *Annual data not available other than for year 6. The SO3 findings are presented for each IR, which include summary tables of the outcomes at the beneficiary and population levels for the IR-level indicators along with overall IR findings. Detailed output-level findings are presented in Annex I.3: SO3 Output Level Findings. The conclusions sections for each IR and for the overall SO3 seek to examine, to the extent possible, what was achieved compared to what was planned. IR3.2: 1600 women occupy leadership roles in their communities [first phase only] Table 27: IR3.2 indicators Indicator Source Initial Data‡‡ Final Target Final Achieved % Final Achieved 3.2.1. [alt.] Percentage of community structures with at least one woman in decision-making positions [calculated for women in any committee position; this is comparable to baseline calculation] Baseline/ endline 76.74 ‡ 94.68 ‡ 3.2.1. Percentage of community structures with at least one women in a decision-making position [calculated for woman as president, VP, secretary, treasurer only; to be consistent with IPTT] Annual 25.8 (FY10) 57 57 (FY16) 100 Baseline/ endline ‡ ‡ 65.96 ‡ 3.2.1a. % of targeted VDCs with at least one woman in decision-making positions Annual 44.2 (FY10) 58 58 (FY16) 100 3.2.1b. % of targeted PTAs‡‡‡ with at least one woman in decision-making positions Annual 32.6 (FY10) 45 31 (FY16) 68 3.2.1c. % of targeted "COGES"/AUE with at least one woman in decision-making positions Annual 12.1 (FY10) 67 81 (FY16) 127 3.2.2.1. % of PTAs and village development counsels who undertake at least 2 actions to support girl's education (enrollment, attendance) Baseline/ endline 80.6 88.6 85.42 97.5 ‡ The target appears set relative to annual survey data, and indicator calculation for annual surveys seems to focus on restricted positions (e.g. President, Vice President, etc.), whereas the baseline survey appears to calculate for all elected representatives of the structure. ‡‡ Out of these five indicators, only 3.2.1 and 3.2.2.1 were measured during the baseline. ‡‡‡ Does not include data for Mothers’ Associations to be consistent with baseline, and due to the fact that members are nearly all women. IR3.2 findings: In terms of achievements, the evaluation team found that the program does appear to have supported enhanced women’s participation in community committee leadership roles, which is arguably FASO Final Evaluation 47 more important than simply increasing the number of women occupying those roles. Through the qualitative field work there does also appear to have been increases in the numbers of women occupying leadership roles, the numbers still remain rather low. For example, at the end of the program, WUAs seemed to have the highest representation of women in decision-making positions (i.e. President, Vice President, Treasurer, or Secretary), and largely due to the high number of women in the Treasurer position. It should be noted that the WUAs also reportedly have the greatest amount of money under management (due to the larger number of user fees) and showed the greatest increase in women occupying decision￾making positions from baseline. The evaluation team also found that it was rare for women’s participation to decline in committees, based on KIIs and FGDs, and the majority of committees we met with, especially those in Tougouri, noted more active participation from women. Indeed, the program has seen an increase in committees with at least one woman in leadership positions, but very little increase in those with two women. The only activities FASO specifically targeted to increase women’s membership in leadership committees were sensitization of the public at the point of committee elections and literacy classes targeted at committee members (potentially increasing the chance that a woman might be re-elected to a more senior position); these activities on their own, however, are not likely to bring about profound changes in women in leadership. For example, sensitization at the time of elections seems unlikely to lead to sustainable changes in this regard. In fact, the evaluation team suspects, but is unable to provide concrete evidence to support this theory, that the work with Care Groups and Mother Leaders, and possibly also the support to MAs, has led to more active participation by some women. This may actually have more of an impact on women’s leadership in targeted communities, but the team is still not convinced of a significant change in this regard. Although the intent of this IR was apparently to increase the number of women in leadership positions, the vast majority of program activities and resources for this IR focused on working with MAs and PTAs to mobilize community contributions to the school canteen to cover gaps in the school year because the contributions of the GoBF and the FASO program did not cover all nine months of the school year. It was an important intervention early in the program as it helped make the 2015 transition from Title II support to school canteens to locally-sourced supplies somewhat less dramatic,53 and schools were able to continue the school feeding programs, albeit with a few gaps at different points in the year. In FY15, PTAs and MAs were able to source 130.7 MT of local food to feed nearly 17,000 children in the locally-run canteens. These community-provided quantities, however, remained significantly less than what had been provided through Title II commodities. For instance, FASO provided take-home rations to more than 4,000 girls in year one growing to more than 10,000 girls in year 4, before ending this component with the termination of FASO commodity assistance. The take-home rations went to girls who met attendance targets in the previous month and was highly appreciated by PTAs and MAs, who insisted that while there was some drop in girls enrollment after the take-home rations ended, most girls continued to go to school. The MAs were supported by FASO to follow up on girls’ attendance and to work with the PTA to ensure that parents continued to send their girls to school. Neither of the outputs of these activities had any direct apparent linkages to achievements toward the IR, although indirect linkages are discussed further in the analysis and conclusions section. 53 The evaluation team was informed that due to looting in the last planned year of distribution, this transition was more abrupt than initially planned. FASO Final Evaluation 48 During the extension period, FASO did not provide any school canteen support or take-home rations for girls, and this actually made it easier for PTAs and MAs to scale-up parent contributions. Finally, FASO support to select PTAs and MAs through IGA matching funds was highly appreciated and required minimal FASO financial or human resources. Mothers in the MAs are accessing small loans for IGAs and repay those loans with interest. Some groups use the interest entirely for school activities and some use the interest to grow the capital fund as well as to fund school activities. IR3.1: 100 VDCs are coordinating the development and implementation of community development action plans [first phase only] Table 28: IR3.1 indicators Indicator FY15* Final Target Final Achieved FY16 % Final Achieved 3.1.1. % of VDCs which are coordinating the development and implementation of community development action plans. 94 95 90 95 3.1.1.1. Percent targeted VDCs who have developed community development plans by the end of the program. 82 90 90 100 3.1.1.2. Percent targeted VDCs who have implemented community development plans by the end of the program. No data 90 83 92 3.1.1.3. Percent of adult education participants who are able to pass a basic literacy exam. 95 95 87 91 *FASO activities related to these indicators were introduced as of FY14 or year FY15. IR3.1 findings: The mayors and the communal councils organized elections in 2016 for all VDCs. VDCs in all FASO villages visited during the qualitative data collection have been active, though many committees visited were newly elected. Per program documentation, all 100 VDCs supported have created community development plans. The fieldwork revealed, however, that despite the existence of these plans, some VDCs are not able to articulate anything specific about the plans, and the two plans actually seen do not include concrete action items to be completed by certain actors within a specified timeframe. FASO staff members suggest that this is a function of the limited number of sites visited, but the two plans from two completely different communes suggest that the plans originated from a template used by the program; both examples are included in Annex I.3: SO3 Output Level Findings. These documents are quite useful diagnostic documents, but the plan in and of itself consists of a one-page budget table. In two of the five program-supported VDCs with which the evaluation team met, the large majority of members were replaced during the 2016 elections; and in at least two other VDCs, there was not a formal election but the committee (or some members) continued “by consensus.”54 The committees in which nearly all members were replaced offered very few details of activities and had limited understanding of the role of the VDC (apart from the simple and vague formulation of “overseeing development activities in the village”). In one VDC (out of six visited, five of which were supported by the program), the committee noted that it had advocated successfully for a new secondary school. At least three of the remaining four FASO-supported VDCs mentioned their actions to support the activities of other committees, mostly helping WUAs or PTAs mobilize funds. FASO staff indicated plans to do follow-up training for these new committees, but there is not a clear connection to the continuing responsibility of commune staff (i.e. in terms of sustainability of the activities), who are invited to the training as participants. 54 This was confirmed to be consistent with the bylaws governing VDCs whereby elections (done by “lining up” behind a candidate) are held only if there is not consensus to continue the same committee. FASO Final Evaluation 49 Finally, although not obviously linked to the logic of the IR, the literacy activities were greatly appreciated but were not implemented on a wide scale. The program contracted local partners ATT and OCADES Kaya to run pop-up literacy centers in different FASO villages that offered three- to six-month courses for members of VDCs or other community-based organizations. Based on the IPTT, in FY16 the program trained 1,286 persons in literacy activities, and this is the only year for which data is reported on. While this is useful for existing VDC members, or those who access the limited spaces devoted to other community members, communities still struggle overall with high illiteracy rates; this also means that many future committee representatives will likely be illiterate, particularly if there are changes in these representatives when elections occur. IR3.4: Internal governance among community structures supported by FASO reinforced Table 29: IR3.4 indicators Indicator Final Target Final Achieved FY16 % Final Achieved 3.4.1. # of community events held to discuss local governance issues 75 82 109 3.4.2. # of community leaders who have completed literacy training 2000 1286 64 3.4.3. % of VDC that financed the implementation of at least one project within the action plan 35 45 129 *FASO activities related to these indicators were introduced as of FY16. IR3.4 findings: In addition to the SO3 KIIs and FGDs, several FGDs or KIIs for other sectors noted that committees, particularly WUAs, functioned better after FASO began working in their communities. WUAs have been collecting money for repairs more consistently and at greater levels than ever before (confirmed in FGDs with WUAs, KIIs with commune representatives and corroborated in annual survey data). All five PTAs and MAs participating in FGDs demonstrated understanding of committee functions and spoke of practices promoted by the program several years after FASO stopped working with them, particularly about managing resources and ensuring joint planning between the school director, the PTA and the MA. All three TMCs interviewed are managing the tools provided by FASO and collecting fees, though they did not have plans for acquiring new equipment to replace lost and worn-out items. Moreover, some COGES, especially for areas where there was a dam constructed, seemed very dynamic in mobilizing users and ensuring maintenance and use of common assets (e.g. pump repair) as well as identifying inputs, etc. At least three COGES visited with rice producer groups or market gardens cited examples of actions and initiatives taken by the COGES on behalf of the group, and all seemed to demonstrate the practices necessary to manage the financial resources of the group. Several program staff and a few VDCs spoke of committee self-assessments conducted jointly by FASO staff and committee members. Although self-assessments are a common and appropriate component of capacity strengthening programs, it does not appear that FASO then undertook specific activities on the basis of those assessments. Most committees reportedly have undergone a formal three-day training which has some general skills aspects and some very specific aspects related to the role of the specific committee. The FASO community facilitators and their supervisors acknowledged that community facilitators, based in the communes they covered, did a lot of mentoring with these committees. However, this is somewhat different than offering tailored trainings to committees with the same self-identified weaknesses, or targeted visits to address a particular theme. Consequently, it is not obvious that the self-assessments served a concrete function and the self-assessments do not seem to have been repeated at a later date with the committees even as a progress monitoring mechanism. The program insists that they have been used FASO Final Evaluation 50 and cite the examples of improving gender balance in committee structures or introducing the literacy programs as results of these assessments; however, the FASO program had no indicators, outputs, or even intermediate results for these, and there was no evidence in annual reports, or otherwise during the field work to suggest that there was some sort of tailored, or even aggregate analysis of self-assessments and decisions taken by the committees. These are by all means positive actions, but they do not appear to show evidence of systematic use of self-assessments. IGAs were highly appreciated and a source of pride in FGDs with VDCs, PTAs, and MAs. The IGAs were used by MAs and PTAs (although only MAs were visited and the activity was often referred to as IGAs with the MAs) to disburse loans and collect repayments and interest; the interest was used to grow the capital fund and/or pay for school activities. The IGAs were frequently cited in FGDs and KIIs as among FASO’s most beneficial interventions (usually along with nutrition sensitization activities with Care Groups and MLs in those districts). Even with probing, there do not seem to have been significant problems in the management of the IGA funds. FASO continue to attend repayment/disbursement days to monitor what happens with funds, but the activity seems to have closed in FY2015. There was little documentation in ARRs or Annual Surveys (through 2015) of what schools have done with the funds or the repayment rates and performance of the funds (the FY2015 ARR suggested that $17,480 was distributed in FY2014 and that $86,640 was disbursed by the 38 PTAs/MAs in FY2015). These are impressive numbers, but there are little additional details as this activity was not part of the core program design, although it was a widely appreciated addition. IR3.3: Increased inclusion of communes in the process of sustaining WUA and VDC activities Table 30: IR3.3 indicators Indicator Final Target* Final Achieved % Final Achieved 3.3.1. % of VDC monitored by the communes (twice a year) 90 (FY17) N/A** N/A 3.3.2. # of formalized MMCs, Village Hygiene Committee 120 114 (FY16) 95 3.3.3. % of WUA with an updated borehole management transfer agreement with the municipality [commune] (cumulative) 30 39 (FY16) 131 3.3.4. % of borehole with at least two preventive maintenance per year conducted by municipalities [communes] 19 36 (FY16) 189 3.3.5. % of breakdowns of borehole that have been repaired within the following 21 days by the WUA 50 (FY17) N/A** N/A *FASO activities related to these indicators were introduced as of FY16 or FY17. **In the latest FASO information, there are no data points for this indicator. IR3.3 findings: From staff interviews, corroborated by FGDs with WUAs and KIIs with commune water focal points, the vast majority of WUAs in FASO-supported villages have an updated agreement with the commune whereby the WUA oversees the care of the borehole on behalf of the commune, and the commune guarantees to conduct at least two preventive maintenance visits per year if the WUA has deposited an annual fee of 10,000 CFA per borehole. The qualitative visits and meetings with commune water focal points and WUAs suggest that most WUAs only began to deposit the required maintenance fees at the commune in late FY16 and early FY17, although they have been collecting the fees to manage repairs at the community level for some time. And WUAs and participants in other FGDs certainly noted the improvements in the repairs of their boreholes. Program data suggests that 36 percent of boreholes FASO Final Evaluation 51 have been preventively maintained by the communes in FY2016.55 Since the maintenance visits are supposed to be spread out throughout the year, it is difficult to understand how two maintenance visits could have happened already in FY2016 for 36 percent of the boreholes. At least three of four communes visited during the field work indicated that they had not yet benefitted from preventive maintenance, which is possibly just the coincidence of the sites visited. In any event, this level of achievement would be surprising given that the agreements with the commune were not signed until after the new municipal councils (and mayors) were elected in May/June 2016, WUA remittances to the commune happened later in FY2016 and early FY2017, although perhaps there is some ambiguity in the formulation of the indicator. For the 2016 VDC elections, the program provided financial and technical support for the elections but distanced itself from direct involvement in the process.56 FGDs with VDCs, and KIIs with staff, mayors and other commune representatives all confirmed that the program facilitated some VDC representatives to participate in commune council meetings to understand what happens in council meetings, and to better understand the role and responsibilities of the council.57 In addition, this participation enables them to begin to understand their role as representatives of the community at the village level and the importance of how the communal council activities and decisions also link to development activities at the village level. Indicator 3.3.1, the only one that involved the monitoring of VDCs, does not seem to have been collected as part of program monitoring. During the qualitative fieldwork, no such structured monitoring visits by the commune were noted, although FASO notes that this was postponed until FY2017 given that municipalities were not elected until later in FY2016 and VDCs not elected until early in FY2017. Finally, there are no outputs associated with IR3.3. IR3.5: Community access to service providers improved Table 31: IR3.5 indicators Indicator Final Target* Final Achieved % Final Achieved 3.5. % of community expressed needs satisfied by service providers within 10 days of request 75 (FY17) N/A** N/A *FASO activities related to this indicator were introduced as of FY2016. Note that there are no outputs associated with this IR. **In the latest FASO information, there are no data points for this indicator, and the FY16 IPTT notes that “The MEAL system in the field was not able to accurately report on this indicator.” IR3.5 findings: This IR was added in FY2016 and, based on interviews and activities listed in Pipeline and Resource Estimate Proposals (PREPs), it focuses on developing lists of service providers for: i) borehole repair persons; ii) irrigation pump repair persons; and iii) VPEA (Villages Poultry Extension Agents). For the latter, the lists aimed to facilitate access to ministries so that VPEAs can obtain official 55 See Annual Report FY2016, p. 18. 56 KIIs, particularly in Gayeri, noted that there were accusations of commune representatives disagreeing over whether there was “consensus” to renew the existing committee. Reports suggest that political parties in the commune at times seek to ensure that their party retains the presidency of the VDC. FASO wisely distanced itself from the elections, and the fact that political agents, and not civil servants, oversee these elections, combined with the discretionary assessment of whether there is “consensus” prior to holding an actual vote, appears to be a risk contained within the guidelines governing the VDC elections, which is outside FASO’s control, but is a reality. 57 The commune council has two representatives from each village, and they are elected to make decisions for the overall commune. The VDC members are also elected per village but their roles and responsibilities are focused on development activities at the village level. FASO Final Evaluation 52 recognition that they meet specific quality standards and facilitate market linkages with suppliers to ensure quality inputs (e.g. parts or vaccines) at better prices. The evaluation team saw lists of qualified repair persons for both boreholes and irrigation pumps for the supported communes in Tougouri and Gayéri districts, and interviews with staff in Manni district confirmed their existence there as well. The list of qualified borehole repair persons only has about six people per district (reflecting two repair-persons for each of three communes supported by the program in each district). Some of these service providers have been approved and are awaiting formal certificates from the Regional Directorate of Water and Sanitation (DREA - Direction régionale de l’eau et de l’assainissement). As for irrigation pump repair, in general it appeared that there were challenges in identifying an adequate number of service providers for irrigation pumps, though at least three KIIs with staff confirmed that this identification process is still underway. All WUAs met were aware of the qualified service providers for boreholes and indicated that they were required to seek repair services from the repair person (or to contact the commune who would send the repair person) who held the contract for regular maintenance with the commune. However, four of six WUAs indicated that it is possible to get repairs done by other repair persons, and in fact in the past they have gotten repairs from people other than the commune-designated maintenance person. The commune focal person in five of six communes visited was a volunteer. In the remaining commune, the focal person was a full-time staff member funded by another program, but only through September 2017. The lists of VPEAs for each district contains 15-20 qualified individuals. FGDs and KIIs with livestock groups and other agriculture beneficiaries cited having received services from the VPEAs, although beneficiaries did not specify when (nor were they specifically asked). As noted in the findings for SO1, program beneficiaries were accessing these vaccination services and appreciated them. Program staff indicated that efforts are underway to increase transparency of prices for spare parts, including encouraging communes and the repair-persons link to suppliers in larger cities with good quality spare parts at better prices. 6.3.4. SO3 Conclusions & Lessons Learned IR. 3.2: 1600 women occupy leadership roles in their communities Conclusions: The IR’s focus on women in leadership positions seems to have been accomplished in part, but not due to the achievement of the outputs given the lack of clear logic between the two levels as explained below. The evaluation team believes this partial achievement was a result of: - The SO2 Care Groups and ML activities demonstrating that women can be very active and achieve concrete changes in their communities; - The program’s encouragement of women candidates when facilitating committee elections; - The improved functioning and activities of the MAs who were supported to manage the school canteen and encourage girls’ education: this gave them leadership experience and served as an example of women’s potential; - The IGA matching funds provided to MAs used to grow the capital fund and/or fund school activities (although this positive effect was not included in program design documents and the activity was described as a strategy for easing the program’s exit from school feeding activities); - Women gained greater access to inputs and skills improvements for livelihoods activities in participating in lowlands improvement, market gardening, and rice parboiling groups. FASO Final Evaluation 53 There have also been significant achievements in terms of increasing school enrollment, particularly for girls. Moreover, the evaluation team asserts that there appears to have been strong capacity improvements for the PTAs and MAs given that they managed the school canteen and worked together successfully. Outputs to IR logic/achievements: One of the most critical challenges with IR3.2 is in its design: the IR formulation focuses on increasing women in leadership roles within the program lifetime; however, none of the stated outputs or activities would seem to have a direct effect within the program timeframe (except perhaps the literacy activities). The lack of a clear, logical connection between increasing the number of women in leadership roles and the program activities of providing school meals, focusing on child enrollment in school (presumably pre-school and primary school, although not explicit in the IRs), and distributing take-home rations to encourage girls’ enrollment means these interventions will likely have little influence on IR3.2, particularly in the short- to medium-run. The intervention was aimed at targeting girls who are quite young, and therefore likely 15 to 25 years away from the age usually seen in leadership positions. Consequently, the intervention is appropriate for an intermediate result focused on girls education, but not for the stated intermediate result of having women occupy leadership roles. The program staff recognize this shortcoming and acknowledged it as a major reason for the design shift in the extension period. These design challenges translate into illogical indicators as well. The evaluation team certainly acknowledges apparent achievements with regard to this IR, but we feel those are likely to have been driven by the activities with Care Groups and Mother leaders, the IGAs with some MAs, and the leadership roles women assumed in certain SO1 activities such as market gardening and poultry raising, all of which gave women concrete experience with women-led community activities which were seen by women and men in the communities, and possibly through the sensitization at the time of elections. However, it is important to acknowledge the design issues if any future programs seek to replicate or expand any successes. IR. 3.1: 100 Village Development Councils are coordinating the development and implementation of community development action plans Conclusions: As the findings note, there is little evidence from the qualitative field work or program documentation that VDCs are able to sustainably lead and coordinate the development activities in their areas. While it is true that in several villages VDCs were very engaged in support for development activities in their areas, in general VDCs were not the strongest of the community structures. The best performing structures were those with a narrower mandate (such as WUAs, PTAs, TMCs, etc.). The program has made significant contributions to those, which is a worthwhile achievement given that the aim of SO3 is a combination of strengthening governance and developing the general capacity of local structures. IR3.4 provides more general information on the governance and capacity development of local structures. We acknowledge that the FASO staff disagree with this finding and stated that they have significant experience seeing VDCs implementing their action plans. The evaluation team, though, questions the characterization of the VDC plans seen (and included in Annex I.3) as true action plans. We do, however, recognize the value of the plans in initiating VDCs into their roles to undertake a thorough diagnosis of their village in order to be able to interact with government and NGO stakeholders who may come to do assessments or develop plans for future projects. Outputs to IR logic: There are also significant design flaws in IR3.1, specifically in the logical connection between the IR and its supporting indicators. There is a weak logical link between the desired outcome of VDCs’ active leadership and involvement in “coordinating the planning of development” on the one hand and activities that increase the literacy of VDC members (voluntarily, for those with the time and desire FASO Final Evaluation 54 for literacy training) and support VDCs to create village development plans on the other hand. Moreover, any link to VDC’s engagement with food security and those aspects of development would be even less convincing. IR. 3.4: Internal governance among community structures supported by FASO reinforced Conclusions: IR3.4 (with the effects of work completed as part of activities under IR3.1 and 3.2) is perhaps the most significant achievement of SO3. The key contributions of IR3.4 stemmed from: i) the election of community committees according to accepted term limits; ii) the basic training in roles and responsibilities; and iii) the training in financial record-keeping. This last financial component contributed to transparent and accountable resource management in various committees and has strong potential for helping to ensure the future sustainability of FASO investments (which is how many program staff articulated the intent of SO3). In low-resource contexts, money and resources capture attention. Therefore, it is how those resources are managed that provide useful teaching moments that can have a significant effect on other activities in the long term. The TMCs’ tool rental system, the IGAs of the MAs, the fees collected by the WUAs, and the deposits gathered by the commune for borehole maintenance are all concrete examples of proper resource management. SILC groups are well known in other contexts, and equally so in the FASO program, for their very transparent and successful management of the members’ resources. However, it is also worth noting that emphasizing VDC activities like holding regular annual meetings and having meeting notes are useful additions and will become more useful over time, but these are not as important as the key contributions noted above, and the outputs might meet not quality standards (see photo of “meeting notes” to the left, which is more of a log of meetings that took place, which is actually quite useful but not likely what one would expect by reading program documents which state “meeting notes”). Outputs to IR logic/achievements: The ambiguity of IR3.4 makes it difficult to discern how findings from outputs 3.4.1 to 3.4.3 contribute to this IR. FASO has focused its efforts in this regard during the FY2016 extension year and has made some significant achievements in a single year, despite the lack of a documented strategy. In addition, one output – “3.4.3: percentage of VDC that financed the implementation of at least one project within the action plan” – could be understood as both an IR and an outcome in and of itself and might have been a more specific result of FASO than the stated IR. Followers of community development programs will likely recognize how difficult it can be to achieve financing of community projects; this usually occurs several steps (and several years) into a community development program. FASO has focused its efforts in this regard during the FY2016 extension year, despite the lack of a documented strategy. Record book from VDC. (Barga II, 1 February 2017) FASO Final Evaluation 55 IR. 3.3: Increased inclusion of communes in the process of sustaining WUA and VDC activities Conclusions: This component was introduced in the one-year extension for FY2016 and has proved to be a promising and vital contribution, particularly with regard to WUAs. IR3.3 capitalizes on an important opportunity to satisfy mutual interests: the commune and the Ministry of Water and Sanitation (MWS) at the regional level sought to increase their ability to have technical oversight over borehole repairs, and the villages wanted more support from the government, including preventive maintenance. FASO has made significant strides in creating linkages between government service providers and local community representative structures, primarily in the management of boreholes. Further, it has strengthened the interaction between WUAs and the commune to ensure more sustainable preventive maintenance of boreholes. That said, the responsibility for actual repair of boreholes and the distinction between repairs and preventive maintenance were not as clear even though FASO organized joint trainings of WUA members (1-2 per WUA), commune representatives and pump repairmen. The terms “preventive maintenance” and “repair” were often used interchangeably by FASO staff members, commune representatives, and WUA members. This is particularly true regarding who ensures that work is done with quality and at fair cost, and who troubleshoots irregularities. For example, what aspects of needed repairs discovered during a preventive maintenance will be covered by the commune versus by the WUA could be the subject of eventual tension and disagreement. It is true that the GoBF has rolled out a series of policies and official texts but what repairs the commune will actually pay for remains a question mark. WUAs interviewed during the qualitative work indicated that they expected that the community would need to pay for any repairs even though WUAs are required to make an annual contribution per borehole to the commune for routine maintenance visits twice a year (taken from the user fees they collect from communities to cover repairs and preventive maintenance). To build confidence in the system being established and to ensure continued contributions from the WUA to the commune, it will be essential that WUAs and communes have the same understanding of what the commune will pay for in terms of maintenance and repairs and what remains the WUAs’ responsibility. Many commune focal points for WASH were most frequently volunteers and not technical persons themselves and they now theoretically oversee the regular maintenance contracts for the commune (for the two preventive maintenance visits per year – already a sizable contract). Managing and overseeing sizable contracts poses a significant risk of corruption, especially when unpaid volunteers are making the decisions. The program staff seem to disagree strongly with the evaluation team on this point, but FASO’s limited experience to date with monitoring the interaction between WUAs and the communes on preventive maintenance and repairs is not enough to reassure the evaluation team that the system will work well everywhere when it is just getting started, and feels it is important to acknowledge and prepare for these risks. Further, repair persons who conduct the two annual routine monitoring visits are selected by the commune focal points and WUAs are not allowed to choose which repair person from the commune will do the work, which suggests a potential gap in governance and leaves open huge potential for complicity at best and corruption at worst. Indeed, repair persons may not feel accountable to the WUA and focal points might simply expect compensation from repair persons for access to the “market”; there may be few mechanisms for cross-checking the technical work given that a single technician will be repairing and conducting regular maintenance on the pumps. Even if, as program staff suggest, the government FASO Final Evaluation 56 guidelines for WUAs58 stipulate that the commune handles repairs and maintenance of boreholes (although the MWS guidelines for WUA seem only to require that the commune has the authority to ensure the quality of repairs and work done on boreholes in the commune), the risk of poor service provision (and/or corruption) remains. In addition, there does not seem to be a clear understanding of how complaints should be handled and what support FASO will provide WUAs if they are confronted with such a problem. During discussions with program staff, some staff disagreed with these observations and did not share the same concerns regarding the gaps in accountability and the potential for corruption. These activities are just beginning, but the evaluation team cautions that should such problems arise, they will likely affect behaviors regarding community and WUA contributions, potentially undoing the strong achievements to date. Finally, the absence of a plan for training newly-elected WUAs or VDCs is a risk to the sustainability of activities. We recognize that this is not directly within the control of the program, though we would like to emphasize the need to consider possibilities to encourage capacity building or to mitigate the inevitable loss of knowledge that happens when new committees are elected. Outputs to IR logic/achievements: It is difficult to speak to logic of outputs to IR3.3 given that there is no documented design or strategy document for the component. First, it is not evident that either the communes or the VDCs recognize the specific roles or points of interaction between the two structures apart from the commune’s involvement in organizing the elections (or the “consensus” of the community to renew the existing VDC without elections, per ministry rules governing VDCs). There do not seem to be clear and concrete VDC activities that the VDC might conduct or implement that will contribute to food security, and how the commune would sustain them going forward; further, the sole indicator (3.3.1. % of VDC monitored by the communes (twice a year) regarding monitoring of VDCs was not collected. On the other hand, there is a strong link for WUAs and the building and sustaining of boreholes in the villages and it is worth emphasizing that the program has made great strides in this area in a short time. There is a large overlap with IR3.5 both in terms of the network of repair persons and access to service providers, especially for borehole management and the WUAs. A significant factor in this potential difference between VDCs and WUAs and the relation to the commune is that the WUA has a much more concrete and specific role that is more easily understood, and reflects a mutual concrete need, in emerging local government structures. IR. 3.5: Community access to service providers improved Conclusions: IR3.5 activities – the VPEAs in particular – have made a noteworthy contribution to program achievements. Community members report using them, and there is greater choice when selecting a vaccinator whose qualifications can be confirmed by the MoA. As for repair service providers mentioned under IR3.3, while there is quality certification59 for the borehole repair persons, WUAs have little choice in selection of providers. Though there is certainly the possibility of providing users with higher quality service providers, the fact that they cannot patronize pump repair persons they have used previously, and in most cases do not have a choice of provider, makes it unclear how this arrangement allows them greater access than before. 58 Water Policy Management Act (2001) 59 Understandably, certification for borehole repair has a longer timeline given it happens at the regional level and the skills are different than those for vaccinators. FASO Final Evaluation 57 The program activities for VPEAs seems to be consistent with increasing access to service providers, as the logic seemed to have been to seek MoA certification for all those in the area who had the qualifications. For WUAs, they seem to be increasing commune services to village infrastructure, by initiating the process of communes providing a service of preventive maintenance that it had not provided previously. Ultimately, however, evaluating the issue of increased access is difficult without a clear and documented strategy and vision for the intervention package. Outputs to IR logic: With only one indicator and no outputs and no strategy document for implementation, it is difficult to comment on the logic of the design. SO3 Overall Conclusions and Lessons Learned The FASO program has indeed made significant achievements in SO3. The six villages seen during qualitative data collection show improved governance practices regarding community investments, although largely around financial record-keeping and openness of meetings. The annual program results and the endline survey results confirm application of many of the key practices promoted by FASO. The most dramatic observable changes in the visited areas directly linked to food security is around the management of boreholes, which strengthens access to clean water and therefore offers a positive contribution towards the reduction of food insecurity, and the strengthening of COGES for RPGs and market gardening groups.60 However, there are many good things that have been done under SO3, from increasing girls enrollment, to strengthening local school feeding programs, to increasing women’s engagement in community structures. The measurement of the two components related to transparency is very significant in terms of program sustainability and promoting practical and understandable governance. And overall, the program has made contributions to improving key governance principles like transparency, accountability, participation and inclusion61, just perhaps not exactly as the program intended or as conceived in a typical governance strategy. In many documents, the results, outputs, and indicators read as a very broad governance program, vaguely worded, rather than a discrete component of a food security program that seeks to change the behaviors of some structures or groups of individuals that will have a direct impact on food security. The extension focus on TMCs, WUAs and repair systems, the COGES for rice producer groups and market gardens to manage a communal asset, IGAs with MAs to support the functioning of school canteens, are all very specific, important, and achievable contributions with direct linkages to food security. However, these seem to get lost at times amid all the other SO3 activities. It is probably more appropriate to say the program has contributed more to the sustainability of certain activities - which was paraphrased as the “purpose” of this program component by several FASO staff. The observable positive practices generated by this component, particularly for a food security program, are impressive. What the program lacked, though, was a clear, concise, and logically ordered design to make the activities more efficient and effective.62 As this section makes clear, there were numerous achievements from SO3. Some of the strongest benefits, however, are masked by broad, vague language around governance, which is exacerbated by a lack of a clear strategy showing activities to outputs to results to a clearly defined 60 This is positive but not sufficient, because reduction in diarrheal disease and improved absorption would also be necessary elements in a complete food security strategy. 61 Although there is no universal definition of “governance principles”, there seems to be a consensus among development actors around the principles of transparency, accountability, participation and inclusion. See USAID’s Democracy, Human Rights and Governance Strategy http://pdf.usaid.gov/pdf_docs/pdacx557.pdf and an analysis on this theme by Carnegie Endowment http://carnegieendowment.org/files/development_consensus_brief.pdf 62 Annexes D2, D3, and D4 as cited earlier summarize the many changes in the Results framework during the FASO Program life. FASO Final Evaluation 58 objective that contributes to the program’s overall food security goals. The weaknesses of design and documentation mean that the MTE, formative evaluation, and final evaluation exercises had to spend an inordinate amount of time sorting through design issues related to what is claimed to be happening or have happened, rather than what is important in terms of the objective’s key contributions to food security improvements. FASO has been reflective and generally managed resources well. For instance, the Title II food incentives in the education component (which does not really fit neatly or logically within an objective focused on governance) seem to have had an effect on school enrollment, and helped create an interest in school feeding that community structures have then taken over (with government support, albeit with inconsistent timing and quantities); even after the removal of the Title II food there seems to be still a significant net gain in enrollment and attendance, especially for girls. In addition, staff members for SO3 appear committed and knowledgeable. These factors have led to the program finding its way despite a series of changes in direction. With a more focused and specific design focusing on the sustainability of a few key investments with targeted involvement from community structures, the program could have achieved more. Key Lessons Learned: - Design activities and outputs to contribute to outcomes and objectives. - Document the logic of interventions, intended effects and learning objectives, remembering to think about program sustainability. - Analyze activities for risks and elaborate mitigation strategies tailored to the context. 6.2.1. SO3 Recommendations 1. Planning, design, and measurement. a. Develop a precise and logical design for all components of the objective, including how the objective fits within the broader goal of the program as well as a theory of change for how activities are supposed to lead to achievement of outputs and how outputs should combine to achieve IRs. The design should also articulate the linkages with other program objectives and sectors. b. For any mid-course modifications, be sure that they are also thought through in detail and reviewed for coherence against the objective and program goal. While it is understandable that a program’s design may change at certain points during the life of the program, modifications should also have a sound design and logic in their contributions to the program. c. Ensure the M&E system has the most appropriate indicators for measuring planned program outcomes. As noted previously in this section, many of the FASO indicators did not really provide strong measurements of the impact of the program’s interventions (e.g., all of the SO3-level indicators focused solely on local structures implementing three of four governance practices, when the SO is targeted at use of good governance principles; all IR3.2 indicators measure percentage of community structures with at least one woman in a decision-making role when the IR focuses on 1600 women occupying leadership roles; etc.). As a result, for SO3 it is extremely difficult to assess achievement of the objective and its IRs through reviewing the extent to which indicator targets were met. d. Make sure that any measures that are designed to contribute towards sustainability are part of a comprehensive sustainability plan and strategy. While there were many activities within SO3 that were aimed at sustainability, FASO lacked a clear strategy for what was expected for locally￾supported structures to be able to continue on their own after FASO’s end and without further external FASO Final Evaluation 59 support. A sustainability plan and strategy would lay out clearly for each relevant activity the expected outcomes for local structures along with the steps and tasks required to reach those results. e. Fairly simple interventions like the MMCs and the IGAs are potentially excellent entry points for promoting good governance. They were both introduced relatively late in the program and would have been excellent interventions earlier in the program upon which to build more complex activities. 2. Local CBO support. a. VDCs and others: Clearly define the program’s planned involvement with VDCs and other CBOs and develop program interventions and activities accordingly. For support to VDCs and other locally￾supported development structures, a program should determine the following: What are the expected organization’s contributions to the program’s overall goals and objectives (which will presumably be related to improving food security)? In what ways can the program achieve these expectations (i.e. what types of activities are necessary)? What human and financial resources are required? What does coordination with the host-country government look like? A FFP program could ensure there is general skills training for VDCs and then accompany them through a process of planning and implementing specific activities related to analysis of the food security context in the village (and be explicit that this is a sub-area of their role as the VDC looking at overall development). b. All program-supported CBOs: Either introduce a system for using the self-assessments, or stop using them. While some elements of the self-assessment seem that they could be helpful for CBOs in their development, if the results of the assessments are not used systematically then there is no point in doing them. The activity should therefore be analyzed to determine its usefulness, and if it is implemented in the future then a system for using the assessments will be needed. c. WUAs: Ensure greater clarity in terms of who will pay for which maintenance and repair services, who provides quality assurance of services provided by the commune repair person, and what a WUA will do if they are unsatisfied with the services provided. While there was clear progress in borehole maintenance during FASO, a gap remains in terms of longer-term precision around payment for maintenance and repairs as well as provision of quality assurance. How these points will be addressed should be clearly documented and understood among key stakeholders 3. Gender. a. Introduce women’s leadership trainings and/or women’s networking meetings. Such trainings could be offered to women participating in the locally-supported structures (VDCs, MAs, PTAs, etc.) and would be used to make sure that activities go beyond bolstering women’s participation in local structures but also help ensure women’s contributions to the groups. It might be a useful complement to also organize meetings for women representatives on different committees across SOs to come together to discuss common challenges and strategies for success. At the very least, it can encourage increases in social capital of women who access positions of power. 4. Access to service providers. a. Continue to explore linkages with other sectors and how best to design activities and measurement for increased access to and/or linkages between private service providers. Even though IR3.5 was introduced in the very latter stages of the program, it has shown signs of promise especially in terms of linking governance with the agriculture and WASH sectors. It will be important to continue to explore these linkages and to think through the design and measurement elements of this type of result so that it can be seen as an important part of an integrated program rather than a less well thoroughly considered add-on. FASO Final Evaluation 60 b. Find a way to monitor user satisfaction with service providers. It important to have strong monitoring in place particularly when the government and private sector are linking together, as abuse of the system risks damaging trust in government actors/services which could work to the detriment of overall intended program outcomes. From a governance perspective, there needs to be formal feedback loops and accountability mechanisms for when there are misuses of the system (including cases of corruption). 6.4 Cross-Cutting Recommendations Though we discuss cross-cutting conclusions in greater detail in Section 7: Overall findings, conclusions and recommendations, we close this section by offering several important cross-cutting recommendations based on the evaluation team’s findings across the program’s key sectors. SBCC - Include in the SBCC strategy a list of cross-cutting messages that can be disseminated across all components. Depending on the program content, generic themes might include topics related to good governance, gender, girls’ education, nutrition for PLW and young children, the advantages of joining SILC, etc. - If radio is used as an SBCC strategy, develop an annual calendar of programs and conduct periodic checks to see what messages are being heard. Engagement of local authorities and technical services - Institute regular meetings between the program staff, technical services and local authorities. The FASO program has in each district a local monitoring committee (comité local de suivi) that brings together mayors, program staff and representatives from the technical services on a quarterly basis. Although meetings are not always held every quarter, they provide an opportunity for the FASO team to present achievements and challenges and to plan jointly with administrative and technical partners. Coordination: - Improve coordination and coherence among USAID-funded initiatives working in the same area to maximize efficiency and avoid undue competition. This should ideally be done during the design phase and should be seen as a shared responsibility between USAID and implementing partners. Targeting: - Ensure inclusion of key vulnerable target groups – i.e. adolescents in this case. 7. Overall Findings, Conclusions and Recommendations This section presents an overall review of the evaluation’s key elements, and these are organized according to assessment of: i) the three main objectives of the final evaluation; ii) the evaluation’s 10 key questions to address; and iii) the evaluation’s seven specific objectives. FASO Final Evaluation 61 7.1. Assessment of FASO Evaluation Main Objectives The evaluation team has drawn conclusions related to the three main final evaluation objectives: Objective 1: Evaluate the performance of the FASO program’s SOs and IRs based on the pre- post comparison of outcome and impact level indicators and explore the lessons learned. First and foremost, FASO is a successful program that has made important contributions to the program goal of sustainably reducing vulnerability to food insecurity in the program’s three targeted health districts. These contributions can be seen at the individual, household, and community levels for targeted beneficiaries and were consistently validated during the qualitative fieldwork. Moreover, FASO has frequently met annual indicator targets for program beneficiaries (as measured through annual surveys). For instance, for the final year of data that is available for SO and IR-level indicators, FASO reached at least 90 percent of the target for 31 of 35 of these indicators that were measured.63 In a number of cases, the program greatly exceeded the established targets (some of which had even been revised upward).64 For population-level measures, while significant difference testing was not able to be done conclusively, there were sizable differences noted between baseline and endline values for several indicators, especially in SO2. These findings, combined with the qualitative data, suggest that the program has contributed to certain important improvements in program beneficiaries and at population-level alike. 65 Lessons learned related to performance of the SOs and IRs have been provided at the end of each SO Conclusions section.66 Objective 2: Evaluate the effectiveness of the use of food commodities in achievement of the intended results. Overall, food was used effectively to advance the program’s intended results. In SO1, FFW enabled reclaiming degraded lands, an activity that continued in many places even once the program stopped providing food. A main reason why this continued is that the food attracted farmers towards a method that convinced them of its practical utility and benefit even in the absence of receiving food. In SO2, qualitative information indicated that food was helpful in promoting participation in Care Groups, use of health services, and a more diverse diet where commodities were distributed. The positive benefits of food distribution for SO2 activities continued once distribution ended. For example, participation in Care Groups continued at a high rate and as both the qualitative and quantitative data showed, women continued to utilize health center services at much higher rates than during the pre-program period. For SO2, it is also very likely that many positive effects of FASO activities will continue post-project. Pillars for sustainability include availability of resources, capacity, motivation, and linkages to other programs and interventions. For the Care Groups, the main resources are the Mother Leaders (MLs) who have convincingly demonstrated their capacity to promote key messages and their ability to influence others. 63 There are three indicators in IR3.3 and IR3.4 that were introduced during the FASO extension but not measured, and in IR1.1 the indicator for average number of live chickens per beneficiary per year has never been measured; the FY16 IPTT notes that for this indicator, FASO is awaiting the report of a specific survey on poultry breeding. 64 These figures are based on a review of the FASO FY16 IPTT. 65 As noted above in Section 5: Baseline/endline comparison presenting the baseline-final survey quantitative findings, any causality or attribution from program implementation and final outcomes at the population level are limited to plausible or possible contributions given that there were no comparison or control groups. 66 See Section 6: Findings, conclusions and recommendations by sector above. FASO Final Evaluation 62 The MLs’ motivation is likely to continue as they enjoy a certain level of prestige in their community, take great pride in being a ML and benefit from the support of community leaders and the health centers. These motivational factors will remain once the project ends as will the MLs’ knowledge and skills. In terms of linkages, FASO is making a concerted effort to work more closely with the three district health teams to link the MLs more closely to CHWs and to health center outreach. It will be important for the MLs to have the continued support of the MoH staff post-project. In addition, Mother Leaders and Care Group members normally participate in SILC groups for accessing savings and loan opportunities. These are locally-perceived practical and collateral gains derived from Care Group participation, which also speak or point to the potential for the sustainability of the Care Groups-induced behavior after FASO has left. Finally, because most Mother Leaders are true leaders within the community, they certainly have a catalyst role in promoting the sustainability after FASO leaves. These leaders will use other local and endogenous networks to further the adopted practice towards sustainability and perpetuation. In SO3, food also seemed to encourage girls’ participation in school but it is not known if these gains will be sustained with the transition from Title II food to locally-sourced food. While FASO did use food effectively to achieve intended results, the program also missed two key opportunities for learning and measurement related to the effectiveness of using food commodities. First, during the first phase FASO used different models in distinct geographic areas for SO2: in two health districts Title II food commodities were distributed as part of a more traditional PM2A approach that also used Care Groups; in the third district no commodities were distributed and village field agents instead of Care Groups were responsible for the SBCC activities. It would have been extremely interesting and beneficial to FASO as well as wider audiences if the program had established a focused learning component with key learning questions and a means to address those questions to explore the differences in these models. This type of study could have looked at the outcomes from each model as well as how Title II commodities were phased-out in areas where they were distributed. We did observe that many practices acquired during the program period occurred continued once distributions ceased, and while this type of retrospective look at program data is useful, it is no replacement for a more intentional study planned and established from the outset of the program. Second, FASO used Title II food for three years (and reportedly from 2004-2009 under a previous Title II program) for school feeding and then shifted completely within a period of less than one year to supporting school canteens to do their own local food sourcing. If this exit/transition was planned from the FASO program onset, it would have been useful for CRS to have planned to study more closely this transition, including strengths and challenges of each approach as well as what could have been done differently and any lessons for future Title II programming that plan for some sort of shift to local sourcing. If this were not planned, USAID/FFP and FASO should have arranged to study this, as it is a commonly recurring challenge under commodity programming, where one year (or even less as was the case here) is not an adequate period of time to manage such a shift in food insecure environments without assurances of government (or other institutional support) to fill in the gaps, or to ensure adequate production changes or school-managed production sources. Objective 3: Evaluate stakeholders’ satisfaction with the program’s quality and effectiveness of implementation processes (including human resource management); determine how these processes supported or inhibited program performance. There was overall strong satisfaction among beneficiaries and institutional stakeholders with program performance and implementation processes were solid and well-executed. For SO2, for instance, all three CMOs interviewed by the evaluation team lauded FASO’s accomplishments in changing behaviors, providing clean water, promoting latrine construction and increasing the use of FASO Final Evaluation 63 services at health centers, especially for antenatal consultations and deliveries. The National Nutrition Director also expressed her satisfaction with FASO’s support to her department, including training for health agents and piloting GASPA in Manni. For SO3, commune authorities, village chiefs, and community members interviewed expressed appreciation for program achievements in improving the functioning of community committees, particularly WUAs, TMCs, and VDCs. FASO was also participatory, adaptable and responsive to local community needs. The program responded to many of the most important food security-related needs of the beneficiary populations, which were articulated in the USAID Food Security Country Framework for Burkina Faso and included in particular agricultural production and marketing, water, sanitation, and IYCF practices, among others. In addition, FASO employed a participatory approach by working with several local structures, either pre-existing or established for the purpose of the program67 - and as such enjoyed strong support at the community level, including from community leaders, and at the administrative level from mayors, GoBF representatives and other key stakeholders. FASO was also a program that, on the whole, adapted as the situation evolved, as seen for instance through the introduction of warrantage and market gardening activities and an increased emphasis on WASH as the needs become more apparent. The flexibility of the donor helped facilitate these program adaptations. 7.2. Review of Evaluation Questions The table below shows the evaluation’s 10 key questions and a summary of conclusions for each question. 1. To what extent were the FASO interventions, as designed, consistent with beneficiaries' (men's and women's) expectations and the food security needs of the target areas? Was the program design valid, appropriate, and relevant given the needs and the context? FASO interventions met beneficiary expectations and food security needs of target areas, as discussed in response to the evaluation’s third main objective just above. The program design for SO2 was logically sound, appropriate and relevant, SO1 was imperfect but solid, and SO3 had major design flaws which did not match the quality of program implementation. As discussed in detail in the SO findings sections, the evaluation team found that SO2 was well-designed, contextually appropriate, and not overly ambitious given the program timeframe and local needs. SO1 was fairly strong but there were questions related to crop selection and the greater emphasis placed on production (and corresponding lesser emphasis on marketing). With SO3, however, there were numerous design flaws and shifts at the SO and IR levels. Even with the modifications, SO3 did not have a logical or tight design, which made it extremely difficult to assess the degree to which FASO fulfilled what was planned. The interventions implemented under SO3 did, however, appear quite effective in contributing to the sustainability of a number of SO1 and SO2 results. 2. How effective was the FASO program in achieving the program's specific SOs and IRs? What are the beneficiary and other stakeholders’ perceptions of benefits and positive and negative consequences? To what degree does a food or subsidy incentive appear to make a difference? This question is addressed in the conclusions of each of the three main evaluation objectives just above as well as in the IR and SO-level conclusions in Section 6: Findings, conclusions and recommendations 67 See Annex K: Local Structures for a complete list and description. FASO Final Evaluation 64 by sector and in Annexes I.1-3: Output-level Findings for SO1-SO2-SO3, respectively, which provide detailed findings and conclusions at the output level for each SO. 3. How effectively and efficiently were FASO program resources (funds, expertise, time, etc.) used to produce quality results? Were activities implemented on time and according to implementation plans? Were target numbers of beneficiaries reached and planned outputs achieved? What elements of the program promoted or inhibited adherence to timely achievement of program targets? Given that this was a final program evaluation focusing on program outcomes, the evaluation team spent comparatively less time looking at this question with its emphases on processes and outputs. Instead, the team prioritized questions that were geared towards program quality, broader-level results, and lessons learned. This being the case, we did still find that program resources were used effectively and efficiently and activities were generally well-implemented (on time, meeting targets, with quality, etc.). 4. To what extent did the program address gender? How effective was FASO in reaching women? What could have been done to improve women's participation? Further, how did male and female beneficiaries perceive changes in: decision-making roles and opportunities; participation in community and social institutions (particularly in leadership positions); access to and control over household and community resources based on participation in FASO activities? What are the lessons learned? Female beneficiaries were prioritized in activities across all three SOs. FASO was successful in including women as priority beneficiaries in program activities. Within SO1, women were prioritized for lowland and parboiled rice, market gardening, poultry, and SILC. In SO2, women, especially Women of Reproductive Age (WRA), are the main target group. This is logical since they are the primary caregivers for young children and also have their own set of health challenges. In SO3, girls in school received take-home rations as a means of supporting girls’ education, and women were members of FASO-supported local structures and committees. Moreover, the program encouraged female candidates for committee elections and helped improve the functioning and activities of the MAs. Promoting women in leadership had mixed results. Each of the SOs included women in positions of leadership in various structures. Many women who were selected as MLs for Care Groups seemed to develop leadership skills, increased self-confidence, and enhanced status in their communities. In addition, during the fieldwork it was not unusual to find MLs represented on other committees such as WUAs, PTAs and even the VDCs, and women’s participation in local structures seems to have increased. The women in MAs were supported to manage the school canteen and girls’ education, giving them leadership experience and serving as an example of the potential contribution of women. The MAs met during the field work were generally very dynamic, in particular those who were running an IGA revolving fund. Men in the PTAs as well as VDCs and other groups pointed to the success of the IGA matching fund, which in turn appears to strengthen the credibility of the MAs and their leaders. However, we also found that there was a missed opportunity in terms of the program strengthening the leadership skills of these women and creating networks of female leaders, as discussed in the SO2 and SO3 findings sections. During the fieldwork it was rare to find a woman in a president or vice-president post and there was no particular leadership training or capacity building support for these women, other than access to literacy classes (which was also available to men). Limited engagement of male and female adolescents in general and of men in SO2. Male and female adolescents were conspicuously absent as a key target group in FASO activities. This is in contrast to FASO Final Evaluation 65 the suggestions in the USAID/FFP Burkina Faso Food Security Country Framework FY2010-2014 document that recommended including adolescent girls in particular as a target group, especially given that: maternal mortality rates were reported to be at least double for adolescents 15-19 years of age; adolescent girls have the least decision making influence of all women; and early marriage of girls is common, especially in rural areas. Although adolescent girls who were pregnant or mothers did participate in SO2, no specific activities were developed for them as a group with special or particular needs or vulnerabilities. Other than discussion groups, there was also limited engagement of men in SO2. Some SBCC activities encouraged men to be more supportive of their spouses in general and of pregnant spouses in particular. However, men were not very involved in family health care issues and FASO did not include many activities that attempted to bring men and women into a broader discussion of gender inequities and the potential for transformative gender changes. To conclude, FASO was very strong in promoting and ensuring female participation in program activities. However, there was not an intentional strategy aimed at going beyond participation to address gender inequities more broadly. Accordingly, there was very little done to transform gender relations through more in-depth and deliberate work (for example, with men, with leaders, with couples, etc.). Although the emphasis on women’s participation strengthened the gender element of the program, there were thus missed opportunities to: i) improve/ transform gender relations; ii) strengthen female leadership more intentionally and consistently; and iii) include men more in discussions around gender￾related issues (e.g., family health, child rearing, household responsibilities, livelihoods, etc.). 5. What are the unintended positive and/or negative consequences of the program? What lessons can be learned to minimize unintended negative consequences in the design of future programs? There were a few unintended consequences as a result of the program, which are summarized here: - The Care Group approach with the MLs apparently achieved its intended effect to promote behavior change to improve maternal and child health. In addition, the evaluation team believes based on reports from program participants that the Care Groups also had the unintended benefit of contributing to women’s leadership in community groups ("unintended" since this was not part of the program design related to promoting women’s leadership). This was accomplished as MLs gained confidence from doing important work, building their skills, and enjoying an elevated status in their communities. This unintended effect is potentially useful for learning and an important contribution to program impact, and it is supported by broader experience from other programs where Care Groups result in other positive changes beyond health and nutrition behavior change. However, the evaluation team noted that the GoBF GASPA model does not have a similar built-in mechanism for promoting women’s leadership in the way that Care Groups do, which is one drawback we see in GASPA. - In Gayéri, the farmers joined the warrantage primarily so they can add value to their farm products by benefiting from a quality storage facility to store the crop to sell at a more favorable market price. However, due to lack of market for rice, the farmers use part of the stored food for family consumption, which was not the intention but still benefited these households. - Some women in a few FGDs expressed concerns that their husbands are not always happy to see them succeed in more lucrative agricultural activities. Although no specific mention of domestic violence or significant harm was specifically cited, the program does not seem to have intentional activities to monitor or mitigate such risks. Yet other FGD participants noted that their husbands were very happy about their success and that their success in the program improved their marital FASO Final Evaluation 66 relationship. A culturally-sensitive household negotiation coaching of participant households could therefore be of use. 6. What was the level of integration of the core program components? Were there further opportunities for integrating program components that might have resulted in greater food security impacts and sustainability? The program deliberately and successfully designed integration across the main program components and did a good job targeting beneficiaries for multiple activities among the different SOs. This was evident throughout the qualitative fieldwork and in the FASO program documents including the ARRs. Illustrative examples observed during the qualitative fieldwork include: - SO1 and SO2: The SO2 field agents provide health and nutrition training for SILC groups and for agricultural production groups such as gardeners. - SO2 and SO3: SO2 has a strong WASH component that includes drilling and rehabilitating boreholes; SO3 has trained and strengthened the WUAs responsible for maintaining these boreholes. - SO1 and SO2: Although not started until later in the program, the MLs have become a priority group for SO1 activities such as SILC and poultry-raising. For instance, the majority of those interviewed for SO2 (women in Care Groups, the MLs and the promoters) participated in SO1 activities, especially SILC, poultry and learning how to use improved agricultural techniques such as zai. - SO3 and SO1: The SO3 team contributed to the organizational strengthening of certain groups such as COGES and TMCs. - SO1, SO2, and SO3: All SOs linked communities to service providers such as VPEAs and WUAs. We can therefore conclude that the three major FASO components are well-integrated, bringing multiple opportunities and benefits to households engaged in the program and strengthening overall positive impact. That the three SOs were well-integrated was also confirmed during the FGDs and KIIs with FASO beneficiaries, officials such as mayors, and community leaders as it was clear that they were familiar with all the principal activities. However, challenges related to improving dietary diversity – an integration-related issue especially between agriculture and nutrition – persisted throughout the program period. Consumption of micronutrient-rich food remains a challenge for improving dietary diversity and nutrition status, especially for women and children. Although this was less of a problem during the first phase in Gayéri and Tougouri when women and young children were consuming Title II food commodities, FASO beneficiaries spoke frequently during the fieldwork of the unavailability of nutrient-rich foods during certain periods of the year. The minimal differences in levels of dietary diversity when looking at baseline and endline survey data further supports this point. While FASO did promote market gardening in SO1 and some SO1-SO2 linkages related to dietary diversity, the program struggled to succeed in facilitating greater access to a diverse diet in the target areas (either through increased production of nutritious crops for household consumption or increased revenues to enable purchase of more nutritious foods). For consideration of further opportunities for integrating program components, please see the Overall Recommendations, which will come in Section 7.4: Overall recommendations below. 7. To what extent has the program been effective in developing the technical and organizational capacities of the beneficiaries and community structures? To what extent did local committees (e.g., FASO Final Evaluation 67 Care Groups, SILC, WASH and water management committees, etc.) promoted by FASO take ownership of their advancement? What are expected obstacles for their functionality in the future? How can these findings be addressed in future programming? 8. How effective have capacity building efforts for community-based organizations been, especially in helping assure sustainability of program interventions/investments (Lowland, market gardening sites, water points)? What are the lessons learned? 9. What were the major factors that influenced the achievement or non-achievement of the expected outcomes? What is the level of ownership that has been acquired by the program’s stakeholders? What is the likelihood for sustained benefits after the end of the intervention? 10. What can be learned about the effectiveness and quality of different behavioral change communication and extension strategies, technical interventions, and management approaches used by the FASO programs that may inform future strategy and program design? These final four questions (7-10 above) include significant levels of overlap regarding four key themes: i) sustainability, including capacity building and ownership; ii) social and behavior change communication; iii) technical approaches and interventions; and iv) program management. As such, we present conclusions for these four questions organized according to these key themes. i. Sustainability, including capacity building and ownership Although certain activities were aimed at promoting sustainability, FASO did not have a concrete, comprehensive sustainability plan or strategy. While FASO introduced through the FY15 PREP an overarching vision and strategy to guide its efforts towards promoting sustainability and post-program ownership, the program did not have a concrete plan to achieve the strategy, and though sustainability was promoted through numerous activities, this was done without clear linkages to the aims or approaches in the program graduation/ exit strategy. The high-level strategy in the FY15 PREP was later used to introduce key components of the extension proposal. Across all three SOs, there seem to be sustainable program achievements. The qualitative fieldwork guides included a number of questions related to sustainability and what will continue after the program ends. Across the board, respondents were adamant that the positive impact of the program would endure. Overall, a number of examples of sustainable achievements were found, including: - SO1: SILC groups that are functioning without further support; poultry beneficiaries accessing markets as well as technical assistance and services from VPEAs; self-financing of certain productive activities and other IGAs; and promoted agricultural techniques, especially those that are low-cost, being practiced; - SO2: Care Groups that are functioning after six years with little attrition from MLs; promoted health/nutrition and WASH behaviors widely known and practiced; boreholes that are functioning and being maintained by WUAs (linked with SO3); increased utilization of maternal and child health services at the health center; - SO3: A number of structures have much greater capacity for functioning on their own in the future, including the WUAs, MAs and VPEAs in particular. Obstacles, however, do remain. In addition to the positive examples of the program facilitating sustainability, there have been challenges as well. Those most evident include: - SO1: Many producers continue to have a mentality of dependence on assistance and subsidies. For example, even when producers are aware of the benefits of fertilizers and other inputs, many would not agree to buy them at prices that are not subsidized. - SO2: Keeping community volunteers like CHWs/MLs motivated post-program may be a challenge. FASO Final Evaluation 68 - SO3: The periodic election of new members for WUAs and VDCs without adequate training by non-FASO structures is a big obstacle to sustainability of those structures. ii. SBCC The program lacked a comprehensive, program-wide SBCC strategy document; nevertheless, the program was quite successful in promoting improved behaviors and practices across all three SOs. Although SBCC was included for each SO, not having a strategy based on formative research was a shortcoming for FASO. It should be noted that having a program-wide SBCC strategy would have allowed for an even greater dissemination of cross-cutting messages. Another weakness was that during the first phase, there was not a full-time SBCC person; one staff person was responsible both for SO3 and for SBCC. The SBCC team is currently composed of staff with a solid background in communication for development and in media. However, their focus seems to be more on documenting successes, media (e.g., radio spots and videos) and organizing events such as ‘ML appreciation days’ and less on finding better ways to promote and sustain behavior change per se. As stated in the MYAP proposal (p. 22), the initial SO1 approach aimed to “use aggressive BCC campaign to create interest, generate demand, and establish the necessary “enabling environment” that will facilitate household adoption of promoted practices”. Multiple radio shows, information briefs, and demonstration fields were used to promote dual-purpose crops and create and encourage adoption and use of improved techniques, diversified inputs, conservation agriculture, improved production practices, seed management, postharvest handling, and processing. The radio broadcasts and shows, and some on￾site promotional events in village markets combined with on-site demonstrations on model farmers’ plots had the merit of creating awareness of a set of opportunities and alternatives ways of farming. In some cases, this inspired adoption as a result, but did so in varying degrees. There were greater challenges in adopting behaviors without subsidies as incentives, for example, the use of chemical fertilizers when not subsidized. Adopting agricultural techniques such as spacing and thinning seedlings, sowing in line, using zaï, half-moons, and stone bands appear less costly and easier to adopt, especially when farmers found them relevant to their socio-ecological contexts. Despite the merit of the SBCC efforts for SO1, there is an impression at times that it is more about communication to stakeholders other than community members (e.g. donors, etc.) than about communication around specific behaviors to change. For instance, focus group discussions during the program were also counted as SBCC activities, which rather seems like a first step about sensitization and raising awareness, and would not necessarily result in changing individual social perceptions and thereby influencing behavior. The SBCC approach for SO2 uses a number of standard methods – interpersonal communications led by the MLs, promoters, hygienists and CHWs (for GASPA groups); community-wide meetings; and short radio programs, usually spots. MLs and the GASPA CHWs use the MoH-approved flipchart for their sessions and the hygienists had loose-leaf visual aid sheets with WASH themes. Spot checks conducted during the qualitative field work showed that these community volunteers know how to use the visual aids appropriately. The MLs have been quite successful in promoting behavior change. In addition, although it has only been in place for a little over a year, the GASPA initiative also seems to be a very effective strategy for increasing knowledge and promoting behavior change. FASO Final Evaluation 69 More recent additions to the SBCC methodology for SO2 are community scoreboards and community video. Although there are fewer than 20 scoreboards, they are large enough to be seen from the road and the qualitative evaluation team observed several of them. According to FASO staff and two Head Nurses, they are real motivating factors for communities who want to see visible declines in child malnutrition and increases in deliveries at health centers. The community video is an initiative of Strengthening Partnerships, Results, and Innovation in Nutrition Globally (SPRING) whose team trained promoters from 15 villages on how to make a video promoting health and nutrition by using community members as the “actors”. This is a promising innovation but it is too early to tell what impact it will have on behavior change. As for radio, the FGD respondents from SO2 were able to cite messages they had heard on the radio (health, nutrition, sanitation, agricultural techniques, and governance) but the FASO program did not seem to have a long-term plan for use of the radio particularly for health/nutrition and WASH, such as a list of themes, a calendar, who to present the spot or program, etc. The spots seemed to be developed on an ad hoc basis, rather than targeted at specific behaviors of specific groups. Overall, the impressive changes in knowledge levels and the declared adoption of better health and nutrition practices by respondents in SO2 FGDs indicate that the SBCC approach has been quite effective. This finding is corroborated by the positive changes measured during the endline survey. The SO2 team has also been able to conduct SBCC activities with SO1 beneficiaries to further disseminate health and nutrition messages. iii. Technical Approaches and Interventions These are described in detail in the findings, conclusions and lessons learned, and recommendations presented by SO in Section 7.3: Review of specific objectives and 7.4: Overall recommendations. iv. Program Management In terms of program management, the evaluation team categorized these according to: i) overall program management; ii) MEAL; and iii) coordination with other key stakeholders. Overall program management: The FASO program was well-managed overall, though there were some challenges with consortium management. FASO benefited from strong leadership and an experienced program management team. One noteworthy fact is that a number of senior staff have been on the team since FASO started (e.g., Chief of Party, Deputy Chief of Party, Tougouri Regional Coordinator and the Health/Nutrition Program Manager), providing solid leadership and continuity as well as program memory. This has helped to promote learning about what works, what does not work, and what needs to be changed. Another advantage of the current program organization is that most field agents reside in FASO program communities, bringing them closer to the populations and leaders for a better understanding of problems, felt needs and possible solutions. In the first phase there were a total of five organizations in the consortium, including HKI and GRET. According to senior managers these two organizations expected to be part of the extension phase but were informed with only one month’s notice that they would not be participating. Whatever the reasons for this decision, the short notice created problems for both organizations (e.g. staff had to be released on short notice, negative impact on financial pipelines, etc.) FASO Final Evaluation 70 Another challenge with any consortium is that each organization has its own compensation package and this is normal. However, when staff fulfilling similar functions in the same geographic zones (e.g., field agents) have wide disparities in their salaries and benefits, this can create tension and have a negative impact on team spirit. There is not necessarily a solution to this issue since each organization is responsible for its compensation system but in such circumstances, greater attention should be given to team building and finding other ways to foster a more cooperative spirit. MEAL: The MEAL system was strong, but the program did not include a robust accountability mechanism or any sort of learning plan or agenda other than documenting success stories. The MEAL system was strong in terms of collecting annual indicators on progress. At the same time, while the program reportedly initiated some efforts to elicit feedback, FASO did not appear to have a solid beneficiary accountability system; the evaluation team did not see any examples during the qualitative fieldwork of how the program consistently solicited feedback from beneficiaries or other stakeholders. In addition, other than documenting success stories, the program did not have a concrete learning plan or agenda, which meant missed opportunities for broader learning about program effectiveness (as discussed under evaluation objective two above). In addition, while annual surveys were conducted the data produced was not consolidated or summarized at district-level. For the evaluation team, this made it a challenge to review district-level data for the indicators that were measured through the annual survey (so ultimately the evaluators ended up going through the process of pulling all of this data together themselves). For the program team, this type of summary analysis would have been useful for analyzing differences between districts each year and for looking at changes over time, and might be considered for future reporting cycles where the IPTT aggregates information for the diverse socio￾cultural areas of the program with some divergent results for some indicators. In 2015 FASO began a process of converting to a fully electronic-based monitoring system using tablets. It is an impressive system and understandably it took more than a year to put it in place and for it to really start to provide meaningful and accurate data. FASO seems to have already arrived at the point where the data is very useful and while there are still complaints as to the effort it requires, sector coordinators and other staff also recognize the benefits in terms of more accurate data, particularly the ability to have access to data within days after an activity and the time savings that comes with addressing data quality issues immediately after collection rather than waiting until a report is due. The new system includes many dimensions including scan cards for beneficiaries as well as for field agents to confirm attendance at trainings (for beneficiaries and for the field agents assigned to run the training). Given the enormous effort and cost to put this system in place, it is understandable that there seems to be a sense that the system will solve all of the monitoring problems; this could lead to a potential overemphasis on what was done and not how it was done, which is a crucial dimension of quality. There does not seem to be structured quality monitoring (of activities and of outputs) by senior program staff such as: observing (spontaneously) activities to assess the quality of an activity session (extension, Care Group, etc.); post-activity monitoring to assess the quality of the activity or retention of key themes; or talking with a WUA or VDC to assess what skills they are applying and what their ongoing challenges are to help focus or refine future activities. The advancements provided by the new data system offer benefits in terms of efficiently measuring outputs and using quantitative data to help monitor certain aspects of quality, but it should not be considered sufficient in and of itself to ensure quality monitoring. FASO Final Evaluation 71 Coordination: FASO has generally supported the GoBF in its initiatives and followed official policies and protocols. In all three objectives FASO followed government policies and worked effectively with relevant ministries such as agriculture, health (including the National Nutrition Directorate), water and sanitation, as well as with local governance committees (VDCs, WUAs, etc.). For SO1 the program coordinated with relevant stakeholders; however, there were challenges implementing the seed coupon scheme in areas in which there were GoBF distributions that provided seed for free. Within SO2, FASO uses the MoH SBCC materials, trainers and training modules, and as of 2015 FASO is piloting the new MoH GASPA model in one commune in Manni health district, while continuing to implement the Care Group model in Tougouri and Gayéri. Each model has pros and cons, as discussed in the SO2 section above, and although both have been successful in effecting behavior change, future programs may benefit from selecting only one model in close coordination and discussion with the MoH regarding the model to use. FASO had mixed experiences coordinating with other RISE actors. The program has made a concerted effort to coordinate with several RISE actors, namely REGIS-ER/AG, SPRING and SAREL (Sahel Resiliency Learning). This includes numerous meetings, detailed plans incorporated into PREPs, and coordination at the field level. Although the donor encourages layering, complementarity and coordination among the actors under the RISE umbrella, this has been a challenge for programs that started their activities in different years, may have different objectives and approaches, and develop their annual work plans on different cycles. The evaluation team feels that the major responsibility for effective coordination and complementarity lies with the donor and measures must be put in place before programs are designed: expecting implementing partners to achieve a high level of complementarity after programs have started can result in an undue burden and even set up unrealistic expectations. Also, where two or more actors are working in similar activities within the same geographic zones (e.g. FASO and REGIS-ER), there may be more of a tendency toward competition rather than collaboration, especially if activities are being funded by the same donor. Within SO1, we found that REGIS-ER is implementing interventions that are similar to the FASO SILC activities and are building off the service provider networks that FASO established. It was not clear, however, the extent to which FASO and REGIS-ER coordinated on this, especially using the same service providers and having similar savings and loan models in the same geographic areas. For health/nutrition and WASH, REGIS-ER is using the promoters trained by FASO and as with the SILC service providers for SO1, is paying them significantly more than FASO for their services. This can create problems in terms of service providers being available for both programs. On a more positive note, SPRING recently trained SO2 promoters from 15 FASO villages on the community video initiative and the promoters have already begun to work with communities on making videos promoting health and nutrition by using community members as the “actors”. 7.3. Review of Specific Objectives The table below provides the evaluation’s seven specific objectives and how they were addressed: Specific Objective Comment or reference (narrative section or annex) Compare the results of the final evaluation with the baseline values to detect changes to the key outcome and impact indicators identified in the IPTT. Completed to the extent possible in Section 5: Baseline/endline comparison and in Annex M (endline survey report) but with limitations as discussed in FASO Final Evaluation 72 Section 4.2: Constraints and Limitations and Section 5. Evaluate the extent to which the FASO program contributed to RISE objectives in terms of partnership with RISE actors. Completed – see information on Coordination in response to evaluation questions 7-10 just above. Evaluate the effectiveness of the use or non-use of commodities in achieving the intended results in the following components: health and nutrition, agriculture, education, and capacity building of community-based organizations. Completed – see summary conclusions under main objective 2 above as well as Section 6: Findings, conclusions and recommendations by sector. Document lessons learned about the effectiveness and quality of behavioral change communication and extension strategies, technical interventions, and management approaches used by the FASO program that may inform future strategy and program design. Completed – lessons learned have been provided at the end of each SO in sections 6.1/2/3.4: SO1/2/3 Conclusions and lessons learned and are also reflected in the 6.1/2/3.5 Recommendations sections Evaluate the extent to which gender was integrated in each activity and how the program might have contributed to women’s economic and social empowerment and to household/ community food security. Completed – see response to evaluation question 4 above. Evaluate the effectiveness of community-based organizations supported by the program to sustain long-term impact and management of the community investments (water points, lowlands, market gardening sites, warehouses, etc.). Completed – see information on sustainability in response to evaluation questions 7-10 above. Evaluate a) the effectiveness of specific activities (Care Groups, SILC, SWC, new agricultural techniques, post-harvest techniques, lowlands, market gardening, CLTS) in achieving the intended results and b) the sustainability trend of CBO self-management of infrastructural investments (lowland, market gardening, warehouses, water points) and equipment. Completed – for part a) see sections 6.1/2/3.3 Findings for SO1/2/3 and 6.1/2/3.4 Conclusions and lessons learned ; and for part b) see information on sustainability in response to evaluation questions 7-10 above. 7.4. Overall Recommendations This final section presents the evaluation team’s overall recommendations for future programs. 1. Introduce more intentional and results-based planning from the outset of a program to ensure maximum impact, especially in the areas of: gender; SBCC; program learning; and sustainability. The evaluation team understands that the FASO program was designed before FFP required a gender analysis and strategy, an SBCC strategy based on formative research, a learning agenda, and a sustainability and exit strategy. Future food security programs need to have these strategic documents in place – whether they are required by a donor or not – as well as a timetable for their periodic review and revision as necessary. When a program is being implemented by different teams, it is also important to have sector strategies and detailed activity descriptions to ensure harmonization across geographic zones. Including field agents in annual and quarterly planning workshops and organizing cross visits of program staff are two suggestions for promoting learning, staff development and harmonization of approaches. 2. Ensure all elements of the program are well-designed and fit within a clearly-articulated theory of change that explains the desired outcomes and how the various program components contribute to those outcomes. FASO Final Evaluation 73 The program staff indicated to the evaluation team that they know that a theory of change will be required as part of new FFP proposal requirements. For future programs, we suggest developing a clear theory of change for each SO and linking these to a consolidated overall theory of change for the program. Each IR or core intervention area should have a documented strategy and activities should have an implementation guide to ensure harmonization of approach across geographic zones. During a program’s implementation period, the theory of change should be reviewed as part of an annual review process (prior to the annual planning process) to assess progress not only against indicators but against the full logic of the program design. This will help avoid some of the inconsistencies of programming design and implementation that were seen in FASO. For future programs these documents can also help identify knowledge gaps to plan operational research and serve as the basis of a program’s learning plan. In the meantime, we also recommend that the current FASO program take advantage of the extension period to consolidate the learning to date by developing an updated theory of change for the current program, or at least document the logic of how activities with direct program beneficiaries will lead to changes in their attitudes and/or behaviors and how those will create changes in the broader community. This will help the team revisit the current logic of interventions, which has not been well-documented, and discuss and recognize gaps that can be addressed in future programming. 3. Aim higher with respect to transforming gender perceptions and relations. As noted above, FASO has made some significant achievements in engaging women in program activities across the SOs. Future programming should base women’s engagement on a detailed gender strategy, which will result from a program gender analysis. The gender strategy should articulate the ultimate desired change for female beneficiaries (distinguishing among different sub-groups of women such as adolescent, elderly, etc., as needed) and map out how that change should occur within the broader program design. The desired outcomes need to be ambitious yet realistic given the extent of gender inequity in the given context. While the evaluation has identified a number of successes with regard to women’s engagement in the program, this is not a substitute for a deliberate plan that includes elements such as: a learning process; gender training for staff (and key government counterparts if feasible); dedicated staff to ensure that gender considerations and learning are fully integrated into the overall program; and concrete strategies for involving men and women in activities with an aim towards improving gender relations and promoting greater gender equity. Future programs can capitalize on the following key elements: i) mobilizing and building confidence in women in contexts where women were not already active in community life; ii) strengthening specific skills among women such as leadership and public speaking through trainings as well as promoting networks of women leaders; iii) promoting enabling attitudes and behavior change among men and among leaders; iv) addressing gender-related inequities in agricultural production and marketing; and v) scaling up activities that stand out as successful (such as basic literacy classes, SILC, etc.).68 Of notable concern 68 There are numerous resources for promoting greater gender awareness in standard food security activities, including: the SILC + Gender Transformative Approach (http://pubs.iclarm.net/resource_centre/SILC-GTA-Facilitation-Manual.pdf); the Economic and Social Empowerment (EA$E) Facilitator Guide for VSLA (http://www.fsnnetwork.org/sites/default/files/EA$E%20Facilitators%20Guide-VSLA.pdf); PROSHAR, a care group curricula to leverage the role fathers and in-laws play in decision-making related to health and nutrition behaviors in Bangladesh (http://caregroups.info/care-group-curricula/proshar-pci-care-group-curricula/); farmer field and business schools (http://www.care.org/work/world-hunger/farmers-field-and-business-school-toolkit); and promoting gender equity in value chain development (https://www.cordaid.org/en/wp-content/uploads/sites/3/2013/02/Challenging_chains_to_change.pdf and http://ilo.ch/wcmsp5/groups/public/---ed_emp/---emp_ent/documents/instructionalmaterial/wcms_106538.pdf); among others. FASO Final Evaluation 74 for future programming is the potential shift from Care Groups to GASPA if the MoH does not include the Care Group model in its overall IYCF strategy. The evaluation team believes that the Care Groups were a critical building block for the successful engagement of women in the SO1 and SO3 activities, particularly in geographic areas where women were reported to be less engaged in community life. 4. Incorporate best practices for promoting agriculture-nutrition linkages, especially as a means to improve dietary diversity of women and children beneficiaries. Future food security programs need to ensure that beneficiaries have access to the key ingredients for a diverse diet that contains the essential nutrients, especially for PLW and young children. Where year￾round food availability is the issue, solutions could include promoting the following: production of nutritious crops for household consumption; household gardening at a wide scale; and/or improved food conservation and transformation measures.69 Where access (income) is an issue, income-generating activities for women, promoting crops for marketing, and encouraging household discussions around purchasing more nutritious foods are possible solutions. These issues also confirm the need to ensure consideration of nutrition and dietary diversity when selecting food products to promote for agricultural or livestock production activities. The FASO program did not seem to clearly articulate how each targeted crop would contribute to improved household nutrition, in terms of crop promotion for specific outcomes related to the more direct nutrition benefits that can be obtained from consumption of household production and the indirect benefits that can accrue through purchasing more nutritious food as a result of increased sales of targeted crops. While the FASO proposal stated that the program would focus on seven “dual purpose” (for household consumption and for sales) value chains, the program design did not specify how promotion of each of these seven crops would explicitly contribute to improved nutritional outcomes for targeted beneficiaries or clearly articulate how promoting these crops would address specific nutrient deficiencies in the local population. As part of the development of theories of change noted above, it is therefore important for future programs to plan out these linkages, determine appropriate measurement, and monitor the progress toward the intended results so that any necessary adjustments can be made accordingly.70 69 In Manni, for instance, during the qualitative field work it was reported that one producer engages in the production of orange-fleshed sweet potato, using the tuber for food and revenue, and he uses the forage to feed his small livestock, thus adding a third purpose for this crop. In a future initiative, program activities could also explore, where appropriate, the encouragement of the use of the leaves for household consumption, as they are normally richer in Vitamin A than the tuber. 70 The USAID-funded SPRING project would be a good reference point for future program design in terms of the agriculture￾to-nutrition pathways framework that SPRING has developed (see here: https://www.spring-nutrition.org/technical-areas/ag￾nut/pathways-and-principles-linking-agriculture-and-nutrition) along with the series of briefs that discuss principles for how agriculture-nutrition linkages can be strengthened (https://www.spring-nutrition.org/publications/series/improving-nutrition￾through-agriculture-technical-brief-series).