September 2015 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Rees Warne, Beth Miller, Fulgence Mishili, and John Msuya for Social Impact, Inc. Insert high-quality photograph representing the project being evaluated to replace this instruction and fill this text box. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program Cover Photo: Peer support group meeting (Photo Credit: Fulgence Mishili) MID-TERM PERFORMANCE EVALUATION OF THE USAID/TANZANIA FEED THE FUTURE PROGRAM: MWANZO BORA, NAFAKA, AND TUBEROSHE CHAKULA September 2015 USAID Contract AID-621-C-15-00002 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program ii CONTENTS Acronyms ................................................................................................................................................................................................. iv Executive Summary................................................................................................................................................................................ vi 1. Introduction ......................................................................................................................................................................................... 1 2. Feed the Future Program Description .......................................................................................................................................... 4 3. Purpose of the Evaluation................................................................................................................................................................. 6 4. Evaluation Design and Methodology ............................................................................................................................................. 7 Evaluation Questions ........................................................................................................................................................................ 7 Evaluation Approach ......................................................................................................................................................................... 8 Data Collection Methods ................................................................................................................................................................. 8 Limitations .........................................................................................................................................................................................11 5. Findings and Conclusions ...............................................................................................................................................................12 Evaluation Question 1 .....................................................................................................................................................................12 Evaluation Question 2 .....................................................................................................................................................................17 Evaluation Question 3 .....................................................................................................................................................................23 Evaluation Question 4 .....................................................................................................................................................................27 Evaluation Question 5 .....................................................................................................................................................................32 Evaluation Question 6 .....................................................................................................................................................................37 Evaluation Question 7 .....................................................................................................................................................................41 Evaluation Question 8 .....................................................................................................................................................................45 Evaluation Question 9 .....................................................................................................................................................................48 6. Lessons Learned ...............................................................................................................................................................................51 7. Recommendations ............................................................................................................................................................................54 Evaluation Question 1 .....................................................................................................................................................................54 Evaluation Question 2 .....................................................................................................................................................................55 Evaluation Question 3 .....................................................................................................................................................................56 Evaluation Question 4 .....................................................................................................................................................................58 Evaluation Question 5 .....................................................................................................................................................................59 Evaluation Question 6 .....................................................................................................................................................................60 Evaluation Question 7 .....................................................................................................................................................................61 Evaluation Question 8 .....................................................................................................................................................................62 Evaluation Question 9 .....................................................................................................................................................................64 Annexes ...................................................................................................................................................................................................66 Annex I: Evaluation Statement of Work ....................................................................................................................................67 Annex II: Evaluation Methods and Evaluation Design Matrix ...............................................................................................74 Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program iii Annex III: Data Collection Instruments .....................................................................................................................................87 Annex IV: Documents Reviewed .............................................................................................................................................. 104 Annex V: Persons Interviewed .................................................................................................................................................. 108 Annex VI: Performance Against PMP Indicators ................................................................................................................... 109 Annex VII: Disclosure of any Conflicts of Interest ............................................................................................................... 117 Tables Table 1. Key Informant Statistics by Data Collection Method ..................................................................................................10 Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program iv ACRONYMS CDCS Country Development Cooperation Strategies CFSVA Comprehensive Food Security and Vulnerability Assessment CHW Community Health Workers CONSENUTH Centre for Counseling, Nutrition, and Health Care CRS Catholic Relief Services CSO Civil Society Organization DCT Diocese of Central Tanganyika DQA Data Quality Assessment EQ Evaluation Question FA Field Agent FDG Focus Group Discussion FeFA Folic Acid/Iron Supplements FIPS Farm Input Promotions Africa FTF Feed the Future FY Fiscal Year GAP Good Agricultural Practices GDP Gross Domestic Product GOT Government of Tanzania HFW Health Facility Worker IFDC International Fertilizer Development Center IP Implementing Partner IR Intermediate Result KII Key Informant Interview LF Lead Farmer MBNP Mwanzo Bora Nutrition Project MNP Micronutrient Powder MVIWATA Muungano wa Vikundi vya Wakulima Tanzania MYS Multi-Year Strategy NAFAKA Tanzania Staples Value Chain Project NGO Non-Governmental Organization PIRS Performance Indicator Reference Sheet PM Program Manager PMP Performance Management Plan PSG Peer Support Groups PSP Private Sector Service Provider QA Quality Assurance RCH Reproductive and Child Health SI Social Impact SILC Savings and Internal Lending Communities SOW Statement of Work STA Senior Technical Advisor TAWG Tanga AIDS Working Group TC Tuboreshe Chakula TFNC Tanzania Food and Nutrition Center TL Team Leader TOT Training of Trainers USAID United States Agency for International Development Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program v USD United States Dollar USG United States Government VBAA Village Based Agriculture Advisor ZOI Zone of Influence Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program vi EXECUTIVE SUMMARY INTRODUCTION The confluence of environmental, political, and macroeconomic factors complicates Tanzania’s development narrative and widespread poverty has remained a chronic problem, with 68 percent (2007) of the population living below the extreme poverty line of 1.25 United States dollars per day.1 Although the overall gross domestic product (GDP) has grown at seven percent per year since 2000, the majority of this has been concentrated in the urban areas and capital-intensive sectors. Much of the economic growth is not reaching the rural areas or the agriculture sector, which employs 77 percent of all Tanzanians.2 According to the Feed the Future (FTF) Country Profile for Tanzania, “chronic undernutrition is the greatest contributor to under-5 mortality and is estimated to cost the country 2.65 percent of its GDP due to lost revenues from poor cognitive and physical development in early life.”3 These statistics clearly highlight that its high rate of chronic under-nutrition remains a key challenge to accelerating Tanzania’s economic growth and reducing poverty nationwide. The United States Agency for International Development (USAID)/Tanzania prioritizes FTF activities as a vital means of supporting its work to reduce poverty and hunger in Tanzania. FTF Tanzania sought to integrate agriculture and nutrition interventions with the theory that together both will have a greater impact on improving the nutritional status than if they were implemented separately. In order to address this problem, USAID/Tanzania developed three activities: the Tanzania Staples Value Chain Project (NAFAKA); Tuboreshe Chakula (TC); and the Mwanzo Bora Nutrition Program (MBNP). These three projects focus on value chain development, post-harvest processing and food fortification, and household nutrition, respectively. THE FEED THE FUTURE PROGRAM DESCRIPTION This evaluation covered three FTF activities: MBNP, NAFAKA, and TC. In the spring of 2015, MBNP and NAFAKA received extensions under which they are expanding their work into Mbeya and Iringa. The TC activity closed in June 2015. Mwanzo Bora Nutrition Program: The goal of MBNP is to bring about significant and measurable changes in the nutritional status of Tanzanian people through the implementation of the National Nutrition Strategy, the Tanzania Agriculture and Food Security Investment Plan, and other relevant policies and strategies. MBNP’s objectives are 1) strengthen the capacity of government and local non￾governmental organizations to deliver quality education and communication on nutrition, and 2) strengthen the delivery of integrated community-based nutrition services and social behavior-changing education, resulting in a model that can quickly be scaled-up. 1 U.S. Government. USAID Country Development Cooperation Strategy, October 3, 2014 – October 3, 2019. U.S. Government, 2014. Web: https://www.usaid.gov/sites/default/files/documents/1860/CDCS%20Tanzania%20FINAL.pdf. 2 Ibid. 3 U.S. Government. Tanzania. Feed the Future. Washington, DC: U.S. Government. Web: . Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program vii Tanzania Staples Value Chain Project: The goal of the NAFAKA Staples Value Chain Activity is to increase incomes and food security in Tanzania by working with smallholder farmers and other value chain actors to strengthen the value chains of two extremely important crops in Tanzania—rice and maize. NAFAKA’s objectives are 1) improving competitiveness and productivity of the rice and maize value chains; 2) facilitating greater domestic and regional trade; 3) increasing benefits from the growth of the maize and rice subsectors, particularly those to women and youth; and 4) unleashing innovation and private sector development Tuboreshe Chakula: The goal of the TC Food Processing and Consumption Project was to transform a critical segment of Tanzania’s agro-processing industry—maize, rice, and sunflower oil—and to increase the supply of and demand for nutritious and fortified foods, especially among vulnerable populations. The objectives of TC are 1) to increase competitiveness of agro-processors, and 2) to increase consumption of nutritious processed products. PURPOSE OF THE EVALUATION Social Impact, Inc. (SI) was contracted to conduct a mid-term performance evaluation to provide an independent examination of the overall progress of three FTF activities in Tanzania and to identify achievements, performance issues, and constraints related to activity implementation and effectiveness. This mid-term evaluation examined each of these key investments to provide USAID/Tanzania with principle project results, lessons learned, and targeted recommendations for future programming. The primary audience for the evaluation is USAID/Tanzania. Findings and recommendations are also expected to be shared with USAID’s Bureau for Food Security in Washington, DC and the implementing partners as well as with relevant Government of Tanzania (GOT) partners. EVALUATION DESIGN AND METHODOLOGY The evaluation team used an evaluation approach that prioritized utilization of evaluation results through the use of a mixed-methods design. There are four approach foci that were woven into the evaluation activities: 1) utilization-focused evaluation; 2) value chain analysis; 3) nutrition; and 4) gender. The qualitatively-dominant but still mixed-methods formative evaluation approach involved: 1) a desk review of available primary and secondary documents; 2) USAID consultations; 3) semi-structured key informant interviews; 4) focus groups discussions (FGDs); and 5) observation visits to activity implementation sites (e.g., nutrition clinics) to observe their operations. EVALUATION QUESTIONS Questions for this evaluation were specified in the evaluation statement of work provided to SI by USAID/Tanzania. There are nine evaluation questions (EQs). The first four have been designated as priority questions, while questions five through seven have been designated as ancillary questions. Among these nine EQs there are 11 sub-questions. The sub-questions are not listed here. 1. To what extent have women and youth been integrated into the design and implementation of the activities? 2. In what specific ways have the activities integrated nutrition and agriculture interventions in their work? Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program viii 3. To what extent have the activities reached scale in terms of outreach to direct and indirect beneficiaries, adoption of new technologies and practices, and adoption of innovative business models? 4. To what extent are activities promoting and measuring diffusion of improved technologies or sharing knowledge of improved practices beyond direct beneficiaries? 5. What progress has each activity made in achieving its respective objectives as outlined in its performance management plan? 7. To what extent have the activities worked with local partners? 8. How are the activities perceived and valued by stakeholders, including direct beneficiaries, indirect beneficiaries, and GOT counterparts? 9. What unexpected results (negative consequences or positive results) of the activities have been realized? FINDINGS AND RECOMMENDATIONS Because of the number of EQs and activities evaluated, the Executive Summary includes principle findings and recommendations arranged by activity rather than by EQ. Principle Findings Overall A. Breadth vs. Depth: Activity monitoring and evaluation (M&E) systems have a strong focus on numbers reached and outputs accomplished (breadth of impact) rather than on progress towards objectives (depth of impact). Activity implementation therefore skews toward meeting numerical targets, rather than accomplishing objectives and goals. This creates risks of superficial outcomes. B. Sustainability: Rapid scaling up and an extremely large beneficiary base with weak connections to program activities create risks for lost opportunities and wasted effort if activities exit before behavior changes and institutional capacity are fully sustainable. This is of particular importance as activities are asked to shift their focus to new implementation areas in Mbeya and Iringa—especially when new villages and large numbers of new beneficiaries have been brought into the activities within the last 18 months. C. Integration of Nutrition and Agriculture: Nutrition and agriculture are not well integrated in the overall United States government FTF results framework. Activity managers indicated that if they were required to integrate nutrition and agriculture, or inter-activity work, they would—but that their indicators did not require them to do so. Mwanzo Bora Nutrition Program: Women are well-integrated into MBNP implementation. “Kits” that include recordings for training modules appear to be a sound way to keep key MBNP messages clear and effective, and health facility workers requested access to kits. Some community health workers (CHWs) appeared to have over-reported outcomes. Gardens were not as prevalent as expected, and some demonstration gardens were in poor condition. Outreach to men could be improved. A revolving fund mechanism was introduced in some peer support groups (PSGs) but its function was not well Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program ix understood by some CHWs, and some members reported that the requirement to contribute to the fund reduced PSG participation. Based on its Performance Management Plan, MBNP is able to claim a large proportion of women of reproductive age and children under age five as beneficiaries. Many of these appear to have been only peripherally “reached,” and attribution to MBNP for some indicators is not clear. The custom proxy indicator for nutrition (5.1.1.1) was recently redefined and no longer provides information on MBNP outcomes. NAFAKA Staples Value Chain Activity: Support to rice production has been especially successful, and uptake of new practices and technologies is relatively high. However, many farmers are applying these to limited portions of their crops. Cost and access to inputs was consistently cited as a critical limiting factor. Few farmers had access to credit. Full rice and maize productivity achievements are not yet sustainable, especially in areas where farmers were recently brought into the program, and there is a risk of loss of incipient production gains. Results for vulnerable households served under Component 4 are particularly at risk. NAFAKA works through many sub-partners that use a variety of approaches, and their implementation areas sometime overlap. It is not clear which of the approaches are more effective than others. Those responsible for field-level indicator data collection did not use consistent definitions of beneficiaries, and one method of extrapolating indicator data from survey results appeared flawed. While staff told evaluators the activity did not include nutrition, nutrition is in activity documents. The proxy indicator for nutrition appears to have been recently redefined and now may give less useful information. Tuboreshe Chakula: The reorientation of TC’s work helped the activity to focus on achieving fortification, particularly of maize flour. The millers and processers interviewed all reported that their business had grown due to participation in TC. Many millers reported selling large quantities of fortified maize meal. However, most of the fortified meal appeared to be being shipped outside the FTF zone of influence (ZOI), including some being sold to outlets in Dar es Salaam and some outside of Tanzania. While consumers somewhere are benefitting from the fortification, little of the benefit was available to those living in the ZOI other than those who living close to mills running fortification dosifiers. Although TC’s internal research showed a high rate of consumption in the direct vicinity of fortifying mills, the team found no familiarity with the concept of fortified maize meal in its FGDs. Some millers with dosifiers were not using them, citing low demand. While mechanized dosifers are effective for bulk production of fortified maize flour, a large proportion of household, particularly in rural areas, take their maize to small nearby mills and have small batches milled. Mechanized dosifiers are too expensive for most of these small neighborhood mills. Millers and government representatives noted that support for regulatory structures would be important for facilitating access to fortified foods in more rural areas. Principle Recommendations Overall Breadth vs. Depth: M&E systems should include more monitoring of outcomes and reduce both the number of and the focus on targets that can only be accomplished with superficial “reach” (sometimes with questionable attribution) to large numbers of beneficiaries (who may not receive substantive or sustainable benefits). Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program x Principle Recommendations Sustainability: Activities shifting to new implementation areas in Mbeya and Iringa should allocate staff and budget toward consolidation of gains in “older” implementation areas, particularly where beneficiaries recently joined and where community-level leaders’ skill many not yet be strong. Integration of Nutrition and Agriculture: Improved integration of agriculture and nutrition and cross￾activity collaboration is more likely to occur if the requirement for it is required by the Mission, through objectives, indicators that must be reported to USAID, and detailed work plans. Mwanzo Bora Nutrition Program M&E: Work with USAID to assess suitability of using broad definitions of beneficiaries that have inherent attribution difficulties. Add indicators to measure progress towards nutrition objectives for people directly participating in MBNP activities. Conduct a data quality assessment (DQA). CHWs: Continue to provide support to CHWs where needed in “old” implementation areas to assure that they understand and are implementing activities as expected and are accurately reporting indicator data. Exit Plan: Develop an exit plan for continued work by district and regional nutrition committees and CHWs. Kits: Roll out agriculture kits in all implementation areas, support broader use of nutrition kits (e.g., at health facilities), and provide backstopping support and training to CHWs. PSGs and Dues: Strongly consider dropping the revolving funds recently introduced into the PSGs. NAFAKA Fill Knowledge Gaps: Given the shift of resources to Mbeya and Iringa and the large increase in beneficiaries in the last 18 months, NAFAKA should 1) conduct an assessment of key gaps in Village Based Agriculture Advisor (VBAAs), lead farmers (LFs), and local sub-partners abilities to sustainably and effectively support farmers; 2) allocate sufficient staff and budget to on-going support of work already begun in current implementation areas where gains are at risk; and 3) provide targeted support and additional training where needed. Data Quality: Conduct a DQA; adjust indicator definitions and calculations where needed to assure clarity and consistency; and assure that those who report indicator data understand and consistently use definitions. Prioritize Most Successful Approaches: Assess strengths and weaknesses of the variety of approaches being used by the many different sub-partners and sub-components (e.g., VBAAs [village-based￾agriculture advisors] vs. LFs vs. agro-dealers and sub-partner efficacy), and prioritize those that are most successful, particularly for new implementation areas. Input Affordability: Increase attention to affordability of inputs, especially access to credit and support for farmers’ cost/benefit analysis of different input types. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program xi Principle Recommendations Nutrition: Fulfill the language on nutrition in activity documents by: 1) providing nutrition training to staff; VBAAs, LFs, and Savings and Internal Lending Community service providers; and to beneficiaries (particularly those in vulnerable groups) and 2) refining the custom indicator on gardens (e.g., to focus on consumption of garden produce). *TC Consumer and Retailer Education and Demand: Increase outreach and education and focus on benefits of fortified maize for shopkeepers and consumers, particularly those in the FTF ZOI. Large Millers, Traders, and Retailers: Work with large millers who have substantial throughput for bulk sales and work with traders and retailer with a focus on sales within the FTF ZOI. Rural Millers and the Scoop Method: In rural areas and with small millers, promote the “scoop” method of fortification (this uses a small scoop to measure and add prepared bulk micronutrients to buckets of maize brought to the mill by consumers for milling into flour). This is especially useful in rural areas where small-scale millers are unlikely to invest in a mechanized dosifier. Government Agencies and Regulations: Collaborate with and support government agencies working on fortification and nutrition; support regulatory structures and regulations. * The TC activity has closed. Future activities that focus on food fortification should provide attention to assuring benefits reach intended targets in the FTF ZOI. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 1 1. INTRODUCTION The United States Agency for International Development (USAID)/Tanzania contracted Social Impact, Inc. (SI) to conduct a mid-term performance evaluation of the progress of three Feed the Future (FTF) activities in Tanzania and to identify achievements, performance issues, and constraints related to activity implementation and effectiveness. The three activities were the Tanzania Staples Value Chain Project (NAFAKA); Tuboreshe Chakula (TC); and the Mwanzo Bora Nutrition Project (MBNP). This mid-term evaluation examined each of these key investments to provide USAID/Tanzania with principle project results, lessons learned, and targeted recommendations for future programming. The confluence of environmental, political, and macroeconomic factors complicates Tanzania’s development narrative and widespread poverty has remained a chronic problem, with 68 percent (2007) of the population living below the extreme poverty line of 1.25 United States dollars (USD) per day.4 Although the overall gross domestic product (GDP) has grown at seven percent per year since 2000, the majority of this has been concentrated in the urban areas and capital-intensive sectors. Much of the economic growth is not reaching the rural areas or the agriculture sector, which employs 77 percent of all Tanzanians.5 According to the FTD Country Profile for Tanzania, “chronic undernutrition is the greatest contributor to under-5 mortality and is estimated to cost the country 2.65 percent of its GDP due to lost revenues from poor cognitive and physical development in early life.”6 These statistics clearly highlight that its high rate of chronic under-nutrition remains a key challenge to accelerating Tanzania’s economic growth and reducing poverty nationwide. With the strong connection between maternal health and children’s under-nutrition, FTF’s investments in Tanzania focus on identifying and addressing the root causes of under-nutrition. Evidence suggests that Tanzanian women contribute significantly to all food crop production, processing, and marketing activities. Tanzanian women are key participants in all staple value chains, constituting of 75 percent of the agricultural labor force. Not to mention, they are involved in all aspects of the rice and maize value chain—the target of USAID's FTF interventions. In spite of this, there are significant gender related inequalities (e.g., education, social norms, and earning potential) that disfavor women in the agricultural sector. Young male and female farmers in Africa and other parts of the developing world face challenges similar to those faced by women in Tanzania. Youth employed in the agricultural sector often struggle with seasonal underemployment and this tends to lead to rural exodus. The limitations and inequalities suffered by women and youth are limiting to overall growth and to equitable development, which is an important reason for USAID/Tanzania FTF’s approach of combining agriculture development and nutrition interventions with the empowerment of women and youth. 4 U.S. Government. USAID Country Development Cooperation Strategy, October 3, 2014 – October 3, 2019. U.S. Government, 2014. Web: https://www.usaid.gov/sites/default/files/documents/1860/CDCS%20Tanzania%20FINAL.pdf. 5 Ibid. 6 U.S. Government. Tanzania. Feed the Future. Washington, DC: U.S. Government. Web: . Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 2 USAID/TANZANIA FEED THE FUTURE CONTEXT The Development Hypothesis USAID/Tanzania prioritizes FTF activities as a vital means of supporting its work to reduce poverty and hunger in Tanzania. According to the statement of work (SOW) for the evaluation, the development hypothesis of FTF in Tanzania is: By dramatically increasing agricultural productivity, improving access to markets, and providing support for the development of agribusiness, rural incomes in the FTF zone of influence (ZOI) will increase. This increase in productivity and income, combined with empowerment of women and youth, and a massive, integrated social behavior change and communication intervention will influence how the increased income of farm families can be used to improve the nutrition status of women and children.7 FTF Tanzania sought to integrate agriculture and nutrition interventions with the theory that they will have a greater impact on improving the nutritional status than if they were implemented separately. In order to address this problem, USAID/Tanzania developed three activities: NAFAKA, TC, and MBNP. These three projects focus on value chain development, post-harvest processing and food fortification, and household nutrition, respectively. USAID’s Response The FTF Multiyear Strategy 2011-2015 outlines the portfolio of FTF Tanzania activities. The three commodity value chains of maize, rice, and horticulture are the focus for increasing productivity given their importance to the agricultural economy in Tanzania. Maize is the most important staple grain and is produced throughout the country. Rice is produced predominantly by small farmers with demand for locally produced rice projected to triple over the next decade. Rice is also being exported to neighboring countries, and as Tanzanian rice becomes more competitive it is expected that rice exports will increase. The horticulture sector has grown significantly in recent years with fast growing demand for horticultural products in domestic, regional, and international markets. All three commodities form a large part of the Tanzanian diet and are key to food security in the country. The growing market opportunities offered by these value chains present small farmers, many of whom are women, with increasing income earning opportunities.8 As the second largest food consumed in the Tanzanian diet, rice was selected as the primary value chain for investment. Because of this, Tanzanian paddy farmers have the potential to supply growing regional markets.9 As it is also a major staple food in the Tanzanian diet, Maize was selected as a secondary value 7 U.S. Government. Statement of Work, USAID/Tanzania – Mid-Term Evaluation of Feed the Future (FTF) Tanzania. Washington, DC: U.S. Government, 2015. Print. 8 Ibid 9 Ibid Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 3 chain. The maize milling process offers opportunities to enhance capacity of the medium private-sector milling enterprises to fortify food to improve nutrition. Horticulture is a third secondary value chain. It offers the opportunity for increased incomes and increased variety of food available on the domestic market. Because we know that rising incomes alone are not sufficient to significantly improve nutrition of women and children, targeted nutrition activities are underway in all areas where FTF works. This integration of the nutrition, direct agricultural value chain work, and demand-stimulated activities of marketing and processing is one of the key hallmarks of FTF work in Tanzania.10 10 Ibid. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 4 2. FEED THE FUTURE PROGRAM DESCRIPTION FEED THE FUTURE ACTIVITIES INCLUDED IN THIS EVALUATION This evaluation covered three FTF activities: MBNP, NAFAKA, and TC. In the spring of 2015, MBNP and NAFAKA received extensions under which they are expanding their work into Mbeya and Iringa. The TC activity closed in June 2015. Mwanzo Bora Nutrition Project The goal of MBNP is to bring about significant and measurable changes in the nutritional status of Tanzanian people through the implementation of the National Nutrition Strategy, the Tanzania Agriculture and Food Security Investment Plan, and other relevant policies and strategies. MBNP seeks to increase the nutritional status of children as well as pregnant and lactating women, with specific focus on reducing maternal anemia and child stunting by at least 20 percent in Manyara, Morogoro, and Dodoma regions, while also supporting community and health facility-level interventions in Zanzibar. Under its recent extension, MBNP initiated work in Mbeya and Iringa. MBNP’s objectives are: 1) strengthen the capacity of government and local non-governmental organizations (NGOs) to deliver quality nutrition education and communication, and 2) strengthen the delivery of integrated community￾based nutrition services and social behavior-changing education, resulting in a model that can quickly be scaled-up. NAFAKA Staples Value Chain Activity The goal of the NAFAKA Staples Value Chain Activity is to increase incomes and food security in Tanzania by working with smallholder farmers and other value chain actors to strengthen the value chains of two extremely important crops in Tanzania, rice and maize. To achieve this goal, the project helps farmers improve productivity, facilitates private sector investment, increases market access, assists vulnerable households, and builds institutional capacity. In addition to farmers, NAFAKA partners with other value chain actors, such as farmer organizations and agro-dealers. NAFAKA interventions are focused in portions of the Southern Agricultural Growth Corridor of Tanzania and are also active in Zanzibar. NAFAKA is initiating work in Mbeya and Iringa. NAFAKA’s 2013 Performance Management Plan (PMP) states that its objectives are 1) improving competitiveness and productivity of the rice and maize value chains; 2) facilitating greater domestic and regional trade; 3) increasing benefits from the growth of the maize and rice subsectors, particularly those to women and youth; and 4) unleashing innovation and private sector development. In its fiscal year (FY) 14 Annual Report, NAFAKA says that the four means it uses “to sustainably reduce poverty and hunger” are “improving competitiveness and productivity of the rice and maize value chains; improving domestic and regional trade in rice and maize; expanding the depth and breadth of benefits from the growth of the rice and maize subsectors; and enhancing rural household nutrition by promoting women-focused value chain development and improved consumption of a quality diet.” Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 5 Tuboreshe Chakula The goal of the TC Food Processing and Consumption Project was to transform a critical segment of Tanzania’s agro-processing industry—maize, rice, and sunflower oil—and to increase the supply of and demand for nutritious and fortified foods, especially among vulnerable populations. TC did this by building the capacity of small-scale agro-processors to improve the quality, safety, presentation, sale, and distribution of new and existing food products and to develop the capacity to fortify these products in the maize and oil sectors. To maximize impact and reach vulnerable groups, particularly pregnant or lactating women and children between six and 59 months of age, TC worked through a range of public and private sector actors. TC also distributed micro-nutrient powders through a social marketing approach to increase uptake of fortified foods. The stated objectives of TC are 1) to increase competitiveness of agro-processors and 2) to increase consumption of nutritious processed products. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 6 3. PURPOSE OF THE EVALUATION This formative mid-term evaluation examines the overall progress and achievements of three of USAID/Tanzania prioritizes FTF activities as a vital means of supporting its work to reduce poverty and hunger in Tanzania. The primary audience for the evaluation is USAID/Tanzania. Findings and recommendations are also expected to be shared with USAID’s Bureau for Food Security in Washington, DC and the implementing partners (IPs) as well as with relevant Government of Tanzania (GOT) partners. Consistent with the Office for Learning, Evaluation, and Research’s 2011 Evaluation Policy, this performance evaluation is intended to provide empirical evidence to respond to evaluation questions designed to support learning and continuous improvement for USAID/Tanzania’s FTF work. SI’s approach draws upon utilization-focused methodologies to ensure that the information and guidance generated by the evaluation are useful to USAID/Tanzania and its stakeholders. The USAID/Tanzania FTF team plans to use this evaluation in an overall sense to inform decisions about scaling-up and modifying or re-designing FTF investments. More specifically, it plans to use the evaluation results to help identify and act on options to improve implementation performance, to enhance collaboration among FTF activities, and to expand the productive involvement of female and young farmers in the maize, rice, and horticulture value chains. Most immediately, findings and recommendations from this evaluation are intended to be used to inform work planning for on-going activities. Elements of the evaluation report may also be used to inform the management of other current FTF activities and the design of new activities with similar objectives and components. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 7 4. EVALUATION DESIGN AND METHODOLOGY EVALUATION QUESTIONS Questions for this evaluation were specified in the evaluation SOW provided to SI by USAID/Tanzania. Given the large number of evaluation questions (EQs) in the evaluation SOW (nine EQs and eleven sub￾questions were listed, for a total of 20 questions), it was not possible to undertake in-depth work on them all. USAID/Tanzania identified four of the EQs as the priority questions for the evaluation.11 These four questions are related to women and youth, agriculture and nutrition integration, achieving scale, and indirect beneficiaries. These four questions have seven additional sub-questions. Because of the breadth of each of these questions, the evaluation team, hereinafter referred to as “the team,” limited its focus to the sub-questions within the primary EQs which best address the priority questions. The EQs included in the evaluation SOW are listed below. 1. To what extent have women and youth been integrated into the design and implementation of the activities?  Which approaches have proven most effective in reaching women and youth?  How have women and youth benefitted from the activities? 2. In what specific ways have the activities integrated nutrition and agriculture interventions in their work?  Have these integration efforts delivered intended results?  What are these results and how have they been achieved? 3. To what extent have the activities reached scale in terms of  Outreach to direct and indirect beneficiaries;  Adoption of new technologies and practices; and  Adoption of innovative business models? 4. To what extent are activities promoting and measuring diffusion of improved technologies or sharing knowledge of improved practices beyond direct beneficiaries?  If indirect beneficiaries are being reached, what are the primary vehicles through which this is taking place, how are indirect beneficiaries being measured, and how are benefits calculated? 11 Note that, in the SOW for this evaluation, these were questions 3, 4, 5, and 6. For the purposes of this evaluation itself, in agreement with USAID/Tanzania, these were renumbered as questions 1, 2, 3, and 4. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 8 5. What progress has each activity made in achieving its respective objectives as outlined in its performance management plan?  Are activity objectives and targets realistic and achievable before the respective activity end dates?  What results/outcomes have been achieved to date? 6. How effective are the implementation approaches used by each implementing partner in achieving intended objectives and results and reaching scale? 7. To what extent have the activities worked with local partners?  What are the results in terms of strengthening local institutional capacity, ownership, and long￾term sustainability? 8. How are the activities perceived and valued by stakeholders, including direct beneficiaries, indirect beneficiaries, and GOT counterparts? 9. What unexpected results (negative consequences or positive results) of the activities have been realized?  Which of these should be considered in implementation or follow-on activities, or documented for further dissemination? EVALUATION APPROACH The evaluation approach prioritized utilization of evaluation results through the use of a mixed-methods design. Four approach foci were woven into the evaluation activities: 1) utilization-focused evaluation; 2) value chain analysis; 3) nutrition; and 4) gender. DATA COLLECTION METHODS Methodology The evaluation methodology focused on generating findings that allowed the team to draw evidence￾based conclusions and design well-substantiated recommendations for USAID/Tanzania’s use. The team lead a qualitatively-dominant but still mixed-methods formative evaluation approach involving: 1) a desk review of available primary and secondary documents; 2) USAID consultations; 3) semi-structured key informant interviews (KIIs); 4) focus groups discussions (FGDs); and 5) observation visits to activity implementation sites (e.g., nutrition clinics) to observe their operations. Following data collection, the team analyzed and triangulated the data to determine evidence-based findings from which to draw conclusions and recommendations. The data collection activities and analytical approaches were designed to be value chain-based, gender and youth-sensitive, utilization-focused, and rigorous. See the Evaluation Matrix in Annex II for a summary of the information sources and methods used to collect data for each EQ. Site Selection: As agreed with USAID/Tanzania, the evaluation focused the field work in regions where it could find the greatest overlap in implementation work among the three activities: Dodoma, Manyara, and Morogoro. The team conducted interviews and FGDs in five of the nine districts in the FTF ZOI within the three regions: Morogoro Urban, Mvomero, Kongwa, Kiteto, and Dodoma Urban. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 9 As bases for operation in five districts, the team identified nodes where a number of beneficiaries that have been affected by the three FTF activities reside. Sampling and Respondent Selection: For TC, the team selected respondents using purposive sampling based on a list of millers and processers who had participated in the TC fortification work during TC’s final phase. The team chose non-participating millers and retailers for opportunistic interviews through a combination of convenience sampling and snowball sampling, and retailers for opportunistic interviews through a combination of maximum variation sampling and snowball sampling. There were two stages of field work in which different types of sampling were used for KIIs and FGDs for NAFAKA and MBNP. During the first eight days of field work in Morogoro and Dodoma regions, the respondents were chosen by the IPs (see the section on limitations below). For the remainder of the field work, the team worked with the IPs and their sub-partners to use maximum variation sampling for beneficiary KIIs and FGDs, with criteria that included high, medium, and low levels of performance, distance from population centers, and agro-ecological zones. The team selected non-beneficiaries (NAFAKA) and people who were not active participants (MBNP) for interviews through opportunistic and snowball sampling. Data Collection Desk Review: The team reviewed activity-related documents provided by USAID, IPs (ACDI/VOCA, Abt Associates, and Africare), and other relevant stakeholders as well as background documents relevant to the activities being evaluated. These included design documents, quarterly and annual reports, monitoring data, data quality assessments (DQAs), materials prepared by the implementers, special purpose publications, contextual data, and other information from government sources and program implementers. USAID/Tanzania FTF Staff Consultations: The team held consultations with a sub-set of USAID staff members prior to departure for Tanzania. Before fieldwork, the team held discussions with the FTF Tanzania team to learn more about the context, priorities, and potential challenges. Key Informant Interviews: The team conducted semi-structured KIIs with a broad spectrum of stakeholders. A total of 212 people were interviewed (see Table 1), including 1) USAID/Tanzania staff; 2) GOT representatives at the national, regional, district and local levels; 3) IPs and sub-partners; 4) NGOs associated with the activities; 5) staff at health facilities; 6) community-level leaders (village based agriculture advisors [VBAA], lead farmers [LFs], field agents [FAs] and professional sector service providers who supported savings groups; and community health workers [CHWs]); 7) agro-dealers; 8) maize millers and sunflower oil processers; and 9) selected beneficiaries. The team conducted the majority of the KIIs with individuals. Some were conducted as joint interviews with two people who worked together or with small groups, such as district or regional officials. The semi-structured KII guides are provided in Annex III. Focus Group Discussions: The team facilitated 29 FGDs with a total of 332 activity beneficiaries (see Table 1), including, for NAFAKA, farmer groups, farmer associations and farmers supported by NAFAKA who did not belong to a farmer group; and, for MBNP, members of peer support groups (PSGs). When appropriate to facilitate the flow of discussion, FGDs were held with either all male or all female participants and, in a few cases, FGDs were split by ethnic group. The FGD guide instrument is included in Annex III. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 10 Opportunistic Interviews: To collect information from people not participating in the FTF activities, the team conducted 41 short opportunistic interviews with individuals selected using convenience sampling (see Table 1). These included women who were pregnant or with small children, farmers, maize mill operators, shop keepers, and government extension agents. Observations: To observe program activities, the team visited selected sites, including maize and rice demonstration plots and farmers’ fields (NAFAKA); health facilities and garden demonstration plots (MBNP); and maize mills and sunflower oil processing facilities (TC). Table 1. Key Informant Statistics by Data Collection Method Region Gender KIIs FGDs Opportunistic Interviews Subtotal Total Dar es Salaam Males 15 0 0 15 24 Females 9 0 0 9 Morogoro Males 58 44 6 108 284 Females 31 140 5 176 Dodoma Males 45 17 8 70 165 Females 35 49 11 95 Manyara Males 10 19 6 35 112 Females 9 63 5 77 Total 212 332 41 Male: 228 585 Female: 357 Data Analysis: Data analysis centered on identification, examination, and interpretation of patterns and themes and how these patterns and themes help answer the EQs. The data analysis approach included five steps: 1) process and record data immediately; 2) begin analyzing as data is being collected; 3) data reduction; 4) identify meaningful patterns and themes; and 5) draw and verify conclusions. Evaluation Team: The team included evaluators from both Tanzania and the United States. There were eight members on the team, four evaluators (two from Tanzania and two from the United States), two support staff, a logistician, and one senior advisor. Team Leader (TL) and Senior Evaluation Specialist Rees Warne is an international development professional with more than 25 years of experience and a focus on designing and leading rigorous mixed methods evaluations. She currently serves as Program Director for Performance Evaluation at SI. Value Chain, Market Systems Development, and Gender Specialist Dr. Beth Miller brings 23 years of experience linking gender analysis with agricultural value chains for sustainable livelihoods. After 10 years as Director of the Gender Equity Program for Heifer International, she started a consulting company that has provided services to an array of clients. Agricultural Economics Specialist Dr. Fulgence Mishili serves as Senior Lecturer in the Department of Agricultural Economics and Agribusiness at the Sokoine University of Agriculture in Morogoro, Tanzania. His research interests and consulting work span agribusiness management, agricultural production analysis, and marketing and trade studies. Nutrition Specialist Dr. John Mbonea Msuya is the chair of the Nutrition Department at Sokoine University of Agriculture in Morogoro, Tanzania. Dr. Msuya is an agriculture and nutrition specialist with over 20 years’ experience carrying out surveys, evaluations, and assessments in these sectors. Vicent Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 11 Temba served as logistician for the team in the field. The SI support staff consisted of Program Manager (PM) Georgie Almon and Program Assistant Kristen Grimsland. Social Impact’s Vice President Dennis Wood served as Senior Technical Advisor. LIMITATIONS There were several limitations that may have affected this evaluation. First, all three activities operate in large geographical areas. The evaluation did not cover them all, so some results based on beneficiary interviews and FGDs, particularly for NAFAKA, cannot be fully generalized across the full spectrum of activity implementation sites. The evaluation SOW designated that the evaluation field work be done in three regions: Dodoma, Morogoro, and Manyara (so, for instance, activity work in Zanzibar was not included). It was agreed with USAID/Tanzania that the field work would be conducted in districts where there was the greatest overlap among TC, MBNP, and NAFAKA implementation. Although this resulted in maize growing areas being slightly more heavily represented than rice growing areas, this is not expected to substantively affect results because of the evaluation methods used. Second, as noted above, during the first week of KIIs and FGDs with NAFAKA and MBNP beneficiaries, the respondents were largely chosen by the IPs. IPs were supportive in working with the team to apply the team’s sampling strategy thereafter. While every effort was made to maximize diversity in the sampling, some selection bias may have been introduced as a result of the respondents made available for interviews. Third, particularly during the early days of field work, NAFAKA, MBNP, and/or their sub-partner staff closely accompanied the evaluation. IP and/or sub-partner staff were often present when the team arrived (usually accompanied by an activity vehicle) at a KII or FDG site, and staff introduced the respondents to the team. At one FGD site, there were seven IP and sub-partner staff. In one case, a sub￾partner staff was leading cheers with FDG participants when evaluators arrived. In another, when the team arrived for a KII, a high-level IP staff person was meeting with the person who was to be interviewed. Over time, the team was able to diminish the level of activity staff’s presence and involvement, but it was rarely completely absent. After the first few days, staff agreed not to be present during FGDs and most KIIs, though they were present during joint interviews with regional and district officials. Overall, this likely introduced some response bias, particularly in early stages of the field work. Selection and response bias were not expected to be significant issues with the TC interviews. The original team leader had to leave the evaluation after six days of field interviews. The Social Impact Senior Technical Advisor (STA) for the evaluation was present in the field for all but two of those days of interviews. SI does not expect that her taking on the team leader role had a negative effect on the data collected or evaluation results. Finally, two of the four team members do not speak Kiswahili. This was compensated for by the language skills of the Tanzanian team members and by interpreters used during the field work. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 12 5. FINDINGS AND CONCLUSIONS EVALUATION QUESTION 1: To what extent have women and youth been integrated into the design and implementation of the activities? Which approaches have proven most effective in reaching women and youth? How have women and youth benefitted from the activities? EQ1: Mwanzo Bora Findings Women: MBNP targets pregnant women and mothers of young children by strengthening existing reproductive and child health (RCH) services at health facilities and the CHWs already present in villages. Some health facility workers (HFWs) told the team that most pregnant women and mothers use the RCH services and indicated that they thought activities increased women’s visits to the clinic and compliance with advice. Through MBNP, the CHWs formed and supported PSGs: MBNP reports 2,136 PSGs with 24,131 members. The PSGs are based in the villages, so they should be reasonably accessible to people who are interested in participating. However, only a small percentage of pregnant women or mothers of young children in each village have joined a PSG. MBNP staff, HFWs, and CHWs all noted that men are often gatekeepers to women’s compliance with nutritional and health advice. MBNP targets men as well as women, and reports that men constitute 39 percent of PSG members.12 However, although the team asked about men’s participation at each FGD and CHW interview, PSG members cited few males as members, and none of the CHWs mentioned all￾male PSGs. While MBNP encourages men to accompany their wives on visits to health facilities, the team observed few men at the facilities, and HWFs reported that their attendance with wives was rare. A staff person said that men think it will waste their time, or worse, expose them to ridicule if they participate. MBNP and HFWs cited several different means of encouraging men to be involved in RCH, including:  Special health care and training for men who accompanied wives to clinic.  Roadshows and cultural events about nutrition and child health, taken to where men gather, such as bars or community centers.  Training for community leaders, such as local government and religious leaders, to encourage healthy behaviors by women and men, with follow up visits to address obstacles. According to MBNP and sub-partner staff, the most effective way to reach men are the roadshows and series of meetings with influential local leaders. Youth: MBNP does not monitor age, but women of reproductive age are defined as 18-45. Some older women and grandmothers brought children to the clinic or participated in PSGs. Some mothers are quite young, especially in Kiteto, where HFWs indicated that the average age of marriage is 14 years for girls. 12 MBNP Year 3 Annual Report 2014 Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 13 Staff, HFWs, and CHWs indicated that mothers and mothers-in-law have a strong influence over the ability of a young mother to implement what she is taught at the clinic or a PSG. For example, older women may impose traditional beliefs such as limiting eggs or other protein-rich foods during pregnancy. First-time mothers and fathers reported that they had not learned anything about prenatal and early childhood nutrition while they were in school. EQ1: Mwanzo Bora Conclusions Women: While CHWs indicated that MBNP influenced women to visit health facilities earlier in their pregnancies and that women are now more compliant with suggested behaviors, this is difficult to verify or demonstrate through MBNP’s monitoring system, as it has no indicators that collect information on this type of result. (See also EQs 3 and 6.) Underlying causes of poor maternal and child health include women’s low status in society, their lack of control over family resources, and limited influence in family decision making. Therefore, outreach to men is important because their support is needed for raising women’s status. It is not clear whether the reported data on men’s membership in PSGs is currently accurate; it is not consistent with the information given to the team in interviews and focus groups. Youth: There is little documented information about the participation of very young mothers, and there appears to be no specific outreach to them. It would be valuable to deliberately target grandmothers, particularly mothers-in-law, with nutrition messages because they may have more influence in household decision making than young mothers. Schools provide a valuable opportunity to educate both men and women about the nutritional needs of mothers and children as well as gender equality within the family. EQ1: NAFAKA Findings Women Beneficiaries: NAFAKA reported that 48 percent of FY14 beneficiaries were women, and of these 41 percent are heads of household. While this is commendable, only 28 percent of NAFAKA’s beneficiaries are married women, which may indicate that married women are having relatively less access to NAFAKA services than married men. Savings and Internal Lending Community (SILC) groups are reported to be 65 percent female. NAFAKA reports that most farmer groups are gender balanced, which is significant in the Tanzanian context. Further, NAFAKA reports that the sex of the respondent was not related to adoption of good agricultural practices (GAPs) in Morogoro. NAFAKA monitors women’s decision-making within the household, but survey results do not seem to be consistent with the qualitative data collected by NAFAKA and by the team. For example, the Annual Outcome Survey 2013 Results13 reports: “…with regards to crop production related decisions….about 65 percent of the decisions are made jointly [by husbands and wives] while there is no significance difference on who make these other decisions among male and females. This is an initial indication that project 13 NAFAKA, Annual Outcome Survey 2013 Results, (p. 24) Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 14 gender strategy is already positively impacting on the way men and women interact in project activities and decision making on key agricultural activities.” Later, the same report presents results from a women’s FGD results that paint a different picture of women’s involvement in post-harvest decision making—one which was consistent with results from the team’s FGDs: “ …most of the farm operations are mainly carried out by women, but after harvest women are not involved in decision making on what to sell and when and the entire ownership of the product shifts to men. Women are not involved in decisions about the use of money from the sales.” 14 Gender inequality, poor communication within families, and harmful gender assumptions continue to be widespread, according to the men and women farmers who participated in FGDs. This is of particular importance regarding issues such as distribution of protein rich foods within households and sharing information about crop yields, income, and use of income. Gender-related information collected during this evaluation is consistent with the qualitative data reported in the Gender Baseline (2012)15, 16 and Active Learning Approach (2014).17 This data identified women’s constraints to benefiting from activities as continuing to include lack of time due to drudgery from collecting water and fuel, and differences in access and control of resources compared to men in the same household. FGD participants reported that increased income affected women and men differently, and was sometimes disruptive. For example, warehousing rice and maize harvests gave married women access to information on total harvests and income earned from sales of family crops, which a few respondents indicated reduced men’s level of control over information on income. A staff informant reported that as women began to earn more income themselves, some men withdrew financial responsibility from the family, while others appreciated women more. It was reported in FGDs that some men used their income from increased production to marry additional wives, while some women used their new income to leave problematic husbands. Training: NAFAKA’s technical training materials depict both men and women performing agricultural activities. Staff and farmers expressed appreciation for the introductory Human Rights training, which includes training on women’s rights. NAFAKA’s Gender Team began to roll out a longer and more intense Human Rights curriculum for community-level leaders (VBAAs, LFs, the FAs, and private sector service providers [PSPs] who work with the SILCs), but this has not yet reached all groups. A few farmers leaders trained in the Human Rights curriculum reported that they do not yet have the skills or confidence to initiate controversial gender discussions on their own. Some women farmers in FGDs reported that they were asked to suggest times, places, and length of training (important because women have more time and travel constraints than men because of other family and household responsibilities), and they indicated that these factors were key to their ability to 14 NAFAKA, Annual Outcome Survey 2013 Results, (p. 31) 15 Gender Analysis for the Maize Value Chain, NAFAKA August 2012 16 Brief on Gender-Based Roles, Constraints, and Opportunities in Rice Value Chain- Kilombero snd Mvomero Districts 17 Active Learning Approach for the Cross Cutting Component: Increase Women and Youth Empowerment in Agriculture, NAFAKA, 2014 Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 15 participate in community-based training. Some women stated that they preferred all women training so that men would not “take over” and get all of the practice. NAFAKA and sub-partner agricultural trainers were overwhelmingly male. Women in Leadership Roles: NAFAKA promotes women in leadership positions in farmer groups, as VBAA, LFs, SILC FAs, and PSPs. 48 percent of LFs are women (in part because of NAFAKA policies), and women hold many leadership positions in farmer groups, particularly as treasurers. However, among the VBAAs selected by communities, 26 percent are women. The difference may be due to how people are selected (by unspecified stakeholders in the community versus by NAFAKA), differences in the training model, or differences in the approaches used by the sub-partners. Women VBAAs and lead farmers reported that the training they received and the leadership role they play gave them increased confidence and increased status in the community. One lead farmer noted that she thought she could be more effective than a male lead farmer because it was easy for her to communicate with women and easy for them to trust her. However, women in leadership positions reported that it was difficult for them to participate in multi-day trainings far from their homes. Primary hurdles included the need for permission from husband or father for travel, difficulty being away from home responsibilities for so long, and potential damage to their reputations. One said that this discouraged her peers from taking on such roles. Others said that they were only able to participate themselves because they had no husband or their husband was not present. Leadership training for women leaders was especially appreciated, although many requested more training for themselves and other women who still lacked confidence to speak in public. Men also appreciated the leadership training. In the FGD for one SILC with women officers, male community members spoke for the women saying that they were “helping” the shy ones, but they seemed to have their own agenda, and women would not speak in their presence. (The team did not always have the time or resources to interview men and women separately). Many NAFAKA and sub-partner staff as well as some VBAAs and two agro-dealers noted that it was good to have women in leadership roles in farmer and SILC groups, particularly where management of member cash was involved, because women were seen to be trust-worthy with handling money. Staff and Sub-partners’ Knowledge and Attitudes about Gender Equality: NAFAKA has a Gender Team and a staff gender specialist, who is also the training specialist, giving her influence over gender aspects of training and gender-related activities. There was a general appreciation and acceptance of the importance of gender balance in membership and leadership of groups among staff, sub-partners, and farmers themselves. Among NAFAKA and sub-partner staff, there was a range of opinions on whether NAFAKA should or was required to make efforts toward gender equality beyond gender balance among beneficiaries and in leadership roles. A few staff expressed concern that promoting gender equality was a distraction from NAFAKA’s core business of increasing household productivity. Some business or production trainers were uncomfortable discussing gender, while others indicated that they thought that NAFAKA was not doing enough to address the attitudes, beliefs, and behaviors that undermined women’s confidence and control of resources. Many staff were appreciative of the Gender Advisor and requested additional training, noting that they had little experience thinking about gender and culture, or knowing how to address deeply rooted inequality. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 16 In the goals and objectives section of its FY14 Annual Report, NAFAKA appears to list as an objective “enhancing rural household nutrition by promoting women-focused value chain development and improved consumption of a quality diet.” NAFAKA sub-partners staff who were interviewed were unsure if this applied only to vulnerable households and could not define “women-focused value chains,” or how SILCs with vulnerable people were part of woman-focused value chains. Youth: Information obtained during farmer FGDs was consistent with NAFAKA’s report of 27 percent of farmer group members being under age 35. Many older farmers opined that when access to credit improved, and the financial benefits of the productivity packages are apparent, then more young people would join. Members of several farmer groups reported that many youth had joined but had subsequently dropped out. Reasons given for relatively low youth participation included interest in faster return on investment, and interest in access to more frequent income than is available in farming; reluctance to wait to sell harvest in a block with other group members; and general lack of interest in farming. Members of the one youth farmers group that was interviewed said that a primary advantage of forming their own group was that they could set lower dues. EQ1: NAFAKA Conclusions Women: NAFAKA’s achievement of gender balance in farmer groups and activity leadership roles is significant, especially given the history of low levels of women’s participation in training and leadership roles in agricultural development activities. Participation in farmer groups is important as development benefits are no longer assumed to trickle down to all family members when male household heads received resources or training. However, gender balance in meetings and leadership is only a rough, and sometimes misleading, proxy for gender equality. Qualitative methods are needed to measure changes in beliefs, attitudes, and behaviors such as shared decision making. A rapid and dramatic increase in household income controlled by either a man or a woman can disrupt household dynamics. Farmer groups with both men and women working on rice or maize productivity and marketing present a powerful opportunity for meaningful engagement about gender equality and the implications of sharing workload, decision making, and benefits. In addition to disaggregating data by sex overall, it is important to differentiate married and unmarried women (or between women in female-headed households vs male-headed households), especially in farmer groups and training opportunities. This information will reveal if married women are benefitting as much as unmarried women or if female spouses of male group members are also accessing training and inputs. If not, appropriate interventions can be planned. The value of counting the members of a male beneficiary’s household as indirect beneficiaries may need to be verified, because benefits do not always trickle down. Women in Tanzania (and elsewhere) are more likely to spend their income on family rather than personal purchases compared to men. NAFAKA’s development hypotheses, “if gender is integrated in project activities women will be empowered”18 coupled with the common conflation by staff and sub-partners of empowerment and having roughly equal men and women participants, implies that gender balance in leadership roles and 18 NAFAKA, Staples Value Chain NAFAKA Activity: Performance Management Plan (FY 2011 – FY2016), January 2013 (p. 2). Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 17 farmer groups is the same as empowerment. Participation in groups and leadership is part of empowerment, but is only one of the five domains identified in the Women’s Empowerment in Agriculture Index, which FTF is using to “provide conclusive data on the effectiveness of some of the approaches undertaken by FTF to empower women smallholder farmers.” 19 (The other domains are production, resources, income, and time). Youth: Interviews with staff confirmed the finding from the 2013 Annual Outcome Survey: "More needs to be done in Youth Development… there is only a nascent youth development strategy in our program.” Many rural youth are seeking alternatives to the subsistence agriculture of their parents and are interested in mechanization and technology, but are at a disadvantage in accessing credit. Agriculture programing for youth needs to take their perspectives and time horizons into account. The farmer youth groups that NAFAKA initiated appear to be effective means of supporting young farmers. EQ1: Tuboreshe Chakula Findings Women: Most millers, blenders, and processers supported by TC were men. TC used gender as a criterion when selecting millers or blenders for participation in activities, seeking to move towards gender balance. A gender action plan from 2012 (with an update in 2014) was referenced, but the team could not access a copy. Youth: TC noted that some processers are young, but did not systematically target or monitor their participation. The team met with young processors who appeared to be very successful. EQ1: Tuboreshe Chakula Conclusions Women: Female processors benefited from expanded business opportunities. While it is typically women who take maize to be milled and who make food purchasing decisions, the specific concerns about additives that millers mentioned consumers having (e.g., that they could make a man sterile) mean that both men and women are important audiences for education about fortification. Youth: Youth are not necessarily disadvantaged in business, because family resources and education can matter more than age. However, future fortification projects could give particular attention to younger processors with the expectation that they would be leaders in the future. EVALUATION QUESTION 2: In what specific ways have the activities integrated nutrition and agriculture interventions in their work? Have these integration efforts delivered intended results? What are these results and how have they been achieved? EQ2: Overall Findings Overall FTF Integration of Nutrition and Agriculture: Nutrition and agriculture are not well integrated in USAID/Tanzania’s FTF Multi-Year Strategy, which sets as its goal “to reduce the poverty rate and increase the agricultural sector annual growth rate from 3.2 to 6.3 percent … by 2015 in the 19 The Gender Assessment for the USAID/Tanzania Country Development Cooperation Strategy, 2012. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 18 target areas” and, in the next paragraph, goes on to say “improving nutrition is also a high level FTF goal.”20 This is not a critique of the Mission’s strategy; the separation is also present in the overall United States government (USG) FTF Results Framework, which has two First Level Objectives: 1) “inclusive agricultural sector growth,” and 2) “improved nutritional status especially of women and children.” Only one second level objective, “increased resilience of vulnerable communities and households,” feeds into both first level objectives.21 That said, overall FTF Intermediate Result 3 (IR) is “increased investment in agriculture and nutrition-related activities.” In USAID/Tanzania’s Multi-Year Strategy, this IR includes a point on an education and communication program.22 Still, USAID/Tanzania intended for its FTF nutrition and agriculture programming to work together. Its FTF Multi-Year Strategy (MYS) states “households reached through the nutrition flagship program will intentionally overlap with those targeted through the agriculture programs in order to maximize programmatic synergies and impact.”23 In practice, each activity perceived the need to integrate nutrition and agriculture differently. MBNP delivers nutrition training and promotes home gardens through the nutrition kits and CHWs, and most PSG members subjectively attributed improved family health to increased vegetable consumption. TC linked a small percentage of maize producers supported by NAFAKA to processors who fortify maize flour. While NAFAKA staff indicated that the activity does not prioritize nutrition, the concept of nutrition is present in its project documents, and home gardens were promoted to some vulnerable households participating in SILC groups. Inter-activity Integration: TC actively collaborated with MBNP and NAFAKA. There is little active collaboration between NAFAKA and MBNP in the field. Although NAFAKA and MBNP work in many of the same villages (MBNP reports 98 percent coverage of villages in the three districts), the team found little overlap between NAFAKA-assisted farmer groups and MBNP’s PSGs. While this may reflect different target group demographics, most women farmers are of reproductive age. Understanding of Nutrition: The concept of “good nutrition” is not well understood by many agricultural professionals or government officials, or even by some health professionals. For example, many agronomists and government officials interviewed indicated that high availability of staple foods, such as cereals, is sufficient for good nutrition. Some officials and NAFAKA staff expressed that good nutrition was only a concern for vulnerable persons such as children under five, pregnant women, and people with chronic diseases, such as Human Immunodeficiency Virus/Acquired Immunodeficiency Syndrome. 20 Ibid (p. 13). 21 Feed the Future “Feed the Future Results Framework,” http://feedthefuture.gov/sites/default/files/resource/files/ftf_results_framework_2013.pdf (accessed August 23, 2015) and included as Figure 1 in “Tanzania: FY2011-201 Multi-Year Strategy.” 22 USAID, “Tanzania: FY2011-2015 Multi-Year Strategy,” February, 2011 (p. 17). 23 USAID, “Tanzania: FY2011-2015 Multi-Year Strategy,” February, 2011 (p. 10). Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 19 EQ2: Overall Conclusions Overall FTF Integration: Effective integration of agriculture and nutrition objectives and outcomes is a challenge at a higher level within USAID than the three activities evaluated here. Inter-activity Integration: MBNP’s nutrition and NAFAKA’s agriculture activities appear to be proceeding in parallel, and the advantages of integration between the activities are not yet realized. Despite occasional one off events, there was little evidence of joint planning or even understanding of each other’s work. However, there is ample scope for increasing collaboration between the activities. Understanding of Nutrition: To achieve FTF goals, a clear understanding of good nutrition is required of all stakeholders, including implementers, sub-partners, and community-based leaders. EQ2: Mwanzo Bora Findings Vegetable Gardens: CHWs were trained in planting vegetable gardens (including sack gardens) and promoted gardening to community members through PSGs and demonstration gardens. Uptake of home vegetable gardens varied widely, with water being the most critical variable, along with access to and affordability of inputs. Most PSG members claimed to be eating more vegetables at home, and some claimed that they and their families are healthier because of this. In areas where vegetables are commonly consumed and access to water was good, the uptake of gardens was encouraging (e.g., Mwembesongo in Morogoro Municipality, Hembeti in Mvomero District, and Mwanya in Kiteto District). PSG-motivated garden uptake was particularly very low in many areas of Kongwa, Kiteto, and Dodoma Urban Districts where water access is a challenge. In some dry areas, like Mahoma-Makulu in Dodoma, vegetables are traditionally grown by men as cash crops, because men are usually in charge of irrigation water. These vegetables are transported to distant markets such as Dodoma Town and Dar es Salaam, and home consumption does not appear to be a priority. Women have other income generating activities (such as pottery or salt collection) that they use to earn income to purchase food for the home. The promotion of vegetable growing to women in these circumstances does not appear to be very effective. Most vegetables promoted by MBNP are exotic varieties (e.g., Chinese cabbage), which require purchase of seeds and pesticides. Women who received trial packets used them, but many women expressed concern about the cost of inputs in the future. Some PSG members (particularly those in more remote villages) expressed reluctance to invest time in preparing gardens if they could not be sure they could access seeds and inputs. Staff indicated that the introduction of highly nutritious orange￾fleshed sweet potatoes was delayed due to difficulties procuring planting materials. In areas with a strong tradition of gathering wild vegetables for home consumption, PSG members indicated that home gardens were not necessary. In these areas, some PSG members asked for training on improved preservation of edible native plants. Small Livestock: Keeping of local free range chickens was already common among women in PSGs, because they require minimal investment for feed and shelter. MBNP encourages women to increase their consumption of animal protein, and in most areas, women do not need a husband’s permission to Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 20 slaughter a chicken. Some women sell their surplus poultry for cash that they can control without the husband’s influence. MBNP encourages rabbit keeping because it is another animal source food that women can control independently from men, but uptake is very low and challenging. Raising rabbits requires more investment and skill than raising free-range chickens. Rabbit keeping is not traditional in most of the targeted areas, and many CHWs have little experience or training. The one set of demonstration rabbits observed (at the home of one CHW) appeared unhealthy and poorly managed. Training: The much anticipated Agriculture Kits were not yet available, although the team did see a prototype that looked promising. MBNP staff said the Kit includes material on staple crops such as rice and maize as well as vegetable, chicken, rabbit, and dairy goat production. EQ2: Mwanzo Bora Conclusions Vegetable Gardens: Although many PSG members said they and their families were healthier because of increased vegetable consumption, MBNP’s monitoring system’s lack of means to differentiate between PSG members and non-members makes assessment and attribution difficult (see also EQs 4 and 5). While home gardens combined with nutritional education can increase vegetable consumption under some conditions, the appropriateness of gardens cannot be assumed for all districts and villages. Sustainability of vegetable gardening for some beneficiaries may be limited by cost and availability of inputs. Small Livestock: It is expected that the new agriculture kits will be mobilized soon and will be useful. Chickens are already widely kept. Uptake of rabbit keeping has lagged due to poor skills. Milk goats can also be useful for improving nutritional outcomes, but only when adequate technical training and health care are included. EQ2: NAFAKA Findings Design and Implementation: Several high-level NAFAKA staff indicated that NAFAKA does not have any nutrition objectives or indicators. Two high-level staff expressed the opinion that nutrition interventions would “dilute” the “core objective” of increased productivity of staple crops. However, there is language on nutrition within NAFAKA’s description of its work.24 In the Goals and Objectives section of its FY14 annual report,25 while there are no explicit objectives listed, NAFAKA follows its goal statements with four actions it is taking; the fourth goal is “enhancing rural household nutrition by 24 USAID/Tanzania also appears to have expected NAFAKA to address nutrition. Its current MYS states that NAFAKA will “enhance rural household nutrition by promoting consumption of a nutritious basket of fortified foods that include but are not limited to staple crops.” (USAID, “Tanzania: FY2011-2015 Multi-Year Strategy,” (p. 27).) 25 The team does not have access to the NAFAKA contract, so it is using activity reports as proxy information on activity design. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 21 promoting women-focused value chain development and improved consumption of a quality diet.”26 In the same report, NAFAKA starts the introduction to its IR.4/Component 4 “increased resilience for vulnerable smallholders” with “NAFAKA is increasing incomes for vulnerable smallholders and enhancing rural household nutrition.”27 Under this component, NAFAKA promotes vegetable gardening to vulnerable households through SILC groups, with the expectation that the vegetables would be consumed to improve family nutrition. The only indicator related to nutrition is the custom indicator “number of beneficiaries with home gardens or alternate crops as proxy for access to nutritious foods and income.” As written, it appears that NAFAKA is responsible for some work on nutrition. Gardens and Alternate Crops: According to NAFAKA’s FY14 annual report, “three hundred seventy-five SILC group members received training on sustainable organic farming and vegetable crop production to improve household-level nutritional intake. Four hundred SILC members adopted home gardens.”28 Only a few SILC groups interviewed had received garden training; in those groups few members had established gardens. Among those, all treated the vegetables as a cash crop to increase income. None of the SILC groups interviewed had received any nutritional training, and most members who were asked defined “nutrition” as having adequate staples through the year. NAFAKA also introduced high protein maize and soy beans in some areas. In a village in Mvomero, none had been harvested yet, and farmers indicated that they did not know if they would like the taste. It remains to be seen if they will consume any of the crops or sell them. Implementation Expertise: Many field and sub-partner staff demonstrated low levels of nutritional knowledge, although they appeared interested to learn more about models for nutrition-sensitive agriculture. Most expressed the opinion that increased agricultural productivity would lead to improved nutrition. Many appeared to conflate “nutrition” and “adequate staple crops.” Gardening training of SILC groups was done by the community-based FAs rather than by agriculture or nutrition experts. Inter-activity Integration: NAFAKA helped to link some farmer groups to TC-supported millers. EQ2: NAFAKA Conclusions NAFAKA’s development hypothesis includes the assertion that “if [agricultural] production and productivity increases, and farmers have access to markets, then household income will increase. If household income increases then households will be food secure.” This is an invalid or incomplete 26 NAFAKA, “NAFAKA Staples Value Chain Activity: Annual Performance Report (October1, 2013 – September 30, 2014),” (p. 6) 27 Ibid. (p. 27). 28 Ibid. (p. 6). Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 22 assumption.29 30 31 32 Traditionally, agricultural interventions have focused on increasing food production and raising incomes to reduce malnutrition, hunger and poverty. Although this remains part of a valid approach, it is now recognized that higher levels of production and income alone have limited impact on improving nutrition. Design and Implementation: Nutrition is already present in NAFAKA’s objectives, an IR and Component 4 as well as in one indicator. There are relatively simple and low-effort ways that NAFAKA can enhance its contribution to its own and the Mission’s FTF nutrition objectives. Gardens and Alternate Crops: The work on vegetable gardens with vulnerable populations under Component 4 appears to be a good opportunity to enhance focus on nutrition. NAFAKA’s contact with many farmer groups is another valuable opportunity for teaching good nutrition to both men and women at the community level and for reinforcing MBNP’s messages, especially when sub-activities are in the same village. Implementation Expertise: Agricultural professionals do not have to become expert nutritionists to link nutritional and agricultural messages appropriate and useful for the farmers with whom they work. EQ2: Tuboreshe Chakula Findings Internal Integration: TC integrated agriculture and nutrition internally through fortification of maize, blended flours and cooking oils. Inter-activity Integration  TC worked with NAFAKA to link some millers with NAFAKA-supported farmer groups.  TC worked directly with MBNP (through development and provision of micronutrient powder (MNP) supplement sachets). However, some messaging from TC agents indicated that there was not sufficient integration with messaging within MBNP. In one village in Mvomero District, PGS members reported receiving contradictory information from CHWs promoting vegetables and TC’s agents saying that MNP sachets were far superior to vegetables. Several HFWs noted that the children most likely to be malnourished had the poorest parents who would not purchase the sachets in sufficient quantity to make the needed difference for undernourished children. Therefore, they encouraged increased vegetable consumption as a lower cost alternative. However, it is difficult to 29 IFAD, 2014. Improving nutrition through agriculture. http://www.ifad.org/pub/thematic/nutrition/nutrition_e_web.pdf 30 BMGF, 2015. Agricultural Development: Optimizing Nutrition Outcomes from Investments in Agriculture. http://www.gatesfoundation.org/What-We-Do/Global-Development/Agricultural-Development/Optimizing￾Nutrition-Outcomes-from-Investments-in-Agriculture SPRING. 2014. Understanding the Food Production Pathway. Brief 2. Improving Nutrition through Agriculture Technical Brief Series. Arlington, VA: USAID/Strengthening Partnerships, Results, and Innovations in Nutrition Globally (SPRING) Project. https://www.spring-nutrition.org/publications/briefs/understanding-food-production￾pathway 32 International Food Policy Research Institute (IFPRI), 2015. Identifying Opportunities for Nutrition-Sensitive Value-Chain Interventions. http://www.ifpri.org/publication/identifying-opportunities-nutrition-sensitive-value-chain￾interventions Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 23 reach adequate levels of micronutrients for an undernourished child through vegetable consumption alone. EQ2: Tuboreshe Chakula Conclusions Inter-activity Integration: TC and MBNP coordinated production and promotion, but did not anticipate that price and availability would be such obstacles. Better communication could have avoided the confusing and contradictory messages. EVALUATION QUESTION 3: To what extent have the activities reached scale in terms of: outreach to direct and indirect beneficiaries; adoption of new technologies and practices; and adoption of innovative business models? Note: This EQ is closely related to EQ5. Please see EQ5 for a more specific discussion on indicators and outcomes. EQ3: Overall Findings  In all three activities, implementation faces an internal tension between breadth of reach and depth of change. High numbers of beneficiaries (breadth) often seemed to be given more weight in implementation than depth of training or other information provided or adoption of promoted practices and/or technologies. This is in part a function of working towards indicators, most of which are output indicators.  People interviewed indicated appreciation of the promoted behaviors and technologies, and many have adopted at least some of them. However, there remains much scope for improved depth of understanding and application of practices and technologies. EQ3: Overall Conclusions  Indicators are increasingly seen as drivers of program implementation, so particular care must be taken to design indicators that will lead implementation in the desired directions.  A large-scale shift of resources (staff and budget) to new implementation areas places at risk some newly reported gains in older implementation areas. In current implementation regions, on-going support toward consolidation and sustainability would be useful, particularly for community-level leaders (VBAAs, LFs, CHWs, etc.) and SILCs. Gains may be lost if support towards sustainability is not provided. EQ3: Mwanzo Bora Findings HFWs: All HFWs interviewed reported that they have incorporated many of MBNP’s messages into their work with the women and young children who come to their health facility. However, they did not all demonstrate full grasp of key MBNP messages. The majority of HFWs interviewed requested additional training and some requested additional informational materials. HFWs are crucial for sustainability of outreach. CHWs and PSGs: Almost all CHWs interviewed requested additional training and support. Many of the PSGs appeared to be functioning well, with members reporting being engaged with their group and benefitting from the information received. In some groups, there were clear needs for the CHW to have additional training and support from MBNP to fully master information and gain skills (such as some Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 24 nutrition messages, gardening, and rabbit-keeping). Some CHWs ceased or greatly reduced working on MBNP activities after funding and/or support from their MBNP sub-contractor was perceived as insufficient or stopped (particularly in Dodoma and Kiteto). In these cases, PSG members did not appear to continue meeting as a PSG. CHWs and PSGs that had been working with MBNP longer appeared to have better mastery of promoted concepts and better adoption of promoted practices (with the exception of vegetables gardens) than did those that had been brought into MBNP more recently. Reach vs. Adoption of Practices: MBNP-supported outreach has been conducted (through HFWs, a media campaign, CHWs, demonstrations of cooking and of vegetable gardens, etc.) to women in a wide catchment area, including, at least at some level, women in nearly all villages in the three regions that were included in the evaluation. While many HFWs interviewed reported that women they served were, for instance, using micronutrient powder sachets frequently and/or following other HFW instructions closely, most of the women interviewed were not applying practices as frequently or as effectively as the HFWs indicated. According to a qualitative impact assessment following the radio campaign broadcasting nutrition information, most respondents could not recall the campaign.33 The team interviewed a small number of women at health facilities who were pregnant and/or carrying babies and who were not in PSGs. None of these women had heard of MBNP or demonstrated awareness of central MBNP messages such as “1000 days” or exclusive breast feeding. Most had heard from HFWs of MNP sachets and the importance of taking folic acid/iron supplements (FeFA). EQ3: Mwanzo Bora Conclusions Reach vs. Adoption: While it appears that MBNP can legitimately claim having reached large numbers of women in some way, adoption rates appear to be relatively low and appear to be concentrated among those in functioning PSGs. The MBNP indicator “number of women of reproductive age reached by USG-supported nutrition programs” does not distinguish between women who have heard of a technology or practice, women who have had substantive training that would allow them to independently apply it, and those actually using it. Levels of exposure to practices promoted by MBNP vary widely. There is a clear distinction in the level of understanding and adoption of practices promoted by MBNP between PSG members and women who just heard information through media or HFWs. Though PSGs appear to be effective, they are resource-intensive (see EQ 7.) HFWs: HFWs appeared to be an effective means of reaching out to women at a broad scale. However, their advice did not appear to be followed as closely as many claimed. CHWs and PSGs: While working through existing local government institutions is important for both results and as a step towards sustainability of the supported work and its outcomes, it is not sufficient. Efficacy of CHWs varied widely. Most CHWs who were well supported by MBNP and/or an effective sub-partner appeared to be effective at getting MBNP’s central messages across to PSG members, though the team observed instances where the messages were imperfectly delivered and/or understood by PSG members. Information did not appear to be flowing well to people outside PSG members’ immediate circles. MBNP may risk undermining incipient gains if CHWs (especially newer ones) do not receive additional support both during the remaining period of performance, if not beyond. 33 MBNP Year 3 Annual Report, October 2014. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 25 EQ3: Nafaka Findings Scaling-up: NAFAKA’s FY14 annual report refers to scaling up as ”scaling out interventions into new villages, deepening interventions along the rice and maize value chains on the mainland, and also expanding rice productivity activities in Zanzibar.”34 Some staff indicated that reaching scale meant hitting targets, and others thought of their work as pilots, with only the best practices transferring to new areas. Nutrition activities do not appear to have been scaled up yet. In FGDs, no SILC members reported that they had had training on nutrition. As noted under EQ2, NAFAKA promotes vegetables gardening to SILC members “to help improve nutritional intake and increase incomes for vulnerable households.”35 Rather than being delivered by NAFAKA agricultural experts, the vegetable garden training in FY14 was provided by the SILC FAs who received training of trainers (TOT). It is worthy of note that FAs are normally expected to be competent in managing finances, not in agricultural work. The FAs provided training on vegetable gardening to 375 SILC members (just five percent of those participating in SILCs). NAFAKA reported that 10 SILC members received training from TC on adding soy to make a flour blend to improve nutrition; this is less than 0.1 percent of SILC members. There is potential for expanding the relatively limited work done on agricultural mechanization, water management, and soil fertility management. Reach vs. Adoption: According to its FY14 Annual Report, in FY14 NAFAKA brought in 55,066 new beneficiaries (which amounts to 54 percent of total beneficiaries supported since activity start-up). In FY14, NAFAKA started work in 117 new villages and started 277 SILCs (65 percent of all SILC members started in FY14). A large number of new VBAAs and LFs were brought in in FY14 as well. This is a commendable expansion of services. With the shift of the bulk of NAFAKA’s resources to the new implementation areas, several NAFAKA staff noted that there would be little opportunity to continue to provide services in the areas where they worked in FY14. Some sub-partner staff indicated that, in previous years, they had to reduce support to some incipient beneficiaries to move on to new areas. In its FY13 Annual Report, NAFAKA reported that, according to its Annual Outcome Survey, 80 percent of farmers “who have worked with NAFAKA since inception applied at least one new technology.” It is not clear whether this definition includes farmers who have been working with NAFAKA since FY12, though the survey is carried out in places where NAFAKA has been working for several years. M&E staff noted that the figure for “number of farmers and others who have applied improved technologies or management practices as a result of USG assistance” (FTF indicator 4.5.2-5) in the FY13 Annual Report was incorrectly calculated and was too high. While it is clear that many farmers have applied technologies and practices that they learned about through NAFAKA (especially using introductory free trial packets of seeds or inputs), many also indicated that they have thus far only applied the new practices or technologies to a small part of their maize or rice crop. Many farmers with whom the team spoke indicated that, while they appreciated the support and advice they had gotten, they were not planning to spend the funds needed to fully apply them on their fields and/or were not yet 34 NAFAKA, “NAFAKA Staples Value Chain Activity: Annual Performance Report (October1, 2013–September 30, 2014),” (p. 6). 35 Ibid. (p. 6). Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 26 confident enough in the results to do so. As well, SILCs need time, generally one to two full cycles, before they are sufficiently consolidated to operate independently. VBAAs and LFs: VBAAs appear to be a useful business model, and both VBAAs and LFs were appreciated by the farmers they supported. Both VBAA and agro-dealer business models are being promoted, sometime in the same areas. Some VBAAs reported that they were now in competition with agro-dealers, though a few VBAAs have become agro-dealers themselves. EQ3: NAFAKA Conclusions There remains much room for improving support to vulnerable groups. Additional training and support for sub-partners, VBAAs, and LFs is still needed. Whether the many new community-level leaders and new beneficiaries (as well as those who have worked with NAFAKA for longer) will independently and appropriately apply new information and technologies they received without additional support from NAFAKA is an important question that should be addressed. Gains made could be at significant risk of not being sustainable. EQ3: Tuboreshe Chakula Findings Millers and Processers: Before its reorientation at the end of FY14, TC had reached larger than expected scale in terms of outreach to a wide range of millers and processers, having worked with 566 micro, small and medium-sized businesses—343 percent of its target.36 Discussions with USAID resulted in redirecting TC’s FY15 work to focus on direct beneficiaries implementing food fortification. In FY15, TC’s target was 56 processers fortifying with dosifiers or micro-feeders by the end of the activity in June 2015.37 In May, at least 44 millers were doing so.38 Those millers, along with three sunflower oil processers (half the anticipated total), adopted new technological approaches and innovative business practices. While TC supported the growth of a broad set of millers, blenders, and oil processers, actual food fortification was not accomplished as anticipated. Many millers who have scaled up are now expanding their markets well beyond the FTF ZOIs, and less fortified meal is staying in the ZOIs than anticipated. Consumers: It is not clear how many indirect beneficiary consumers were reached through TC awareness raising or through consumption of maize meal or sunflower oil fortified due to TC. In May 2015, TC conducted its final panel survey of consumers who lived within a one kilometer radius of millers (most of which received dosifiers through TC). TC’s survey indicated that 50 percent of those households had consumed fortified maize meal within the past week. For many of these, the fortification was added when they took their own maize to a miller who has a dosifier. This contrasts with the qualitative data collected by the team in a broader cross section of the FTF ZOIs, where very few respondents had heard of fortified maize meal. Most millers using the dosifiers (some of those interviewed were not using them) indicated that they were selling fortified maize to traders who largely sold the product outside of the local area (see also the TC sections under EQ4 and EQ9). 36 Tuboreshe Chakula, “Annual Report: Tuboreshe Chakula: October 2013-September 2014,” October, 2014. 37 Tuboreshe Chakula, “USAID Tanzania Feed the Future Tuboreshe Chakula Project Final Survey Report,” May, 2015 (p. 6). 38 See the discussion on this indicator under Evaluation Question 5. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 27 EQ3: Tuboreshe Chakula Conclusions Millers and Processers: TC’s mid-course change did shift its work from breadth to depth. This change meant that fewer millers and processers were supported during the final phase of the activity, but more fortified food was produced—and, therefore, more households’ nutrition will be affected. In this case, reducing the breadth of implementation focus resulted in greater breadth of ultimate impact on nutrition. Consumers: At the end of the TC activity, less fortified maize entered the diet of households in FTF ZOIs (particularly in rural areas) than anticipated. However, maize fortified due to the TC activity is entering the broader market and is being consumed in a much larger area, including in Dar es Salaam and outside of Tanzania. As yet, the impact of fortification on nutrition in FTF ZOIs is limited (see MBNP regarding MNP sachets). EVALUATION QUESTION 4: To what extent are activities promoting and measuring diffusion of improved technologies or sharing knowledge of improved practices beyond direct beneficiaries? If indirect beneficiaries are being reached, what are the primary vehicles which this is taking place, how are indirect beneficiaries being measured, and how are benefits calculated? EQ4: Mwanzo Bora Findings Diffusion of Knowledge: Some of the practices promoted by MBNP are familiar to and sometimes being followed by women who are not PSG members. Many of those who are directly participating in MBNP PSGs said they are helping to diffuse information beyond their group, primarily by telling their friends and neighbors about what they were learning. That said, many women, including some who CHWs said were PSG members,39 were not familiar with the practices that MBNP promotes. The health facility workers that were interviewed all said they were giving advice based on information that they had received from colleagues who had been directly trained by MBNP to the pregnant and lactating women who visited the clinic. However, opportunistic interviews conducted with pregnant non-PSG members revealed that many of the primary MBNP messages were not familiar to them. For example, most pregnant women said they had taken FeFA supplements, and most were familiar with micronutrient sachets (having heard of them or having been given them by HFWs). However, almost no women who were interviewed or who participated in a FDG used MNP sachets as advised, indicating that this was because they could not afford to buy them and/or did not know where to purchase them. Many of the demonstration gardens observed by the team did not appear to be operated in a way that would promote diffusion of good practices. While a few of those were observed to be heathy, having been cultivated based on good practices, others appeared to be in poor condition or planted without following the good practices meant to be promoted by MBNP (e.g., seeds much too close together). Others appeared to have been planted or heavily worked within a few days prior to the evaluators’ visit. 39 This appeared to be primarily (though not solely) an issue in areas supported by MBNP sub-contractor, the Diocese for Central Tanganika, which was replaced in early 2015 by Sharing Worlds. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 28 At one health center, respondents reported that the demonstration sack gardens could not be viewed because they had recently been stolen. Other means of diffusion of information were cited as useful. These included brochures; theater, songs, and roadshows; training of village, district, and religious leaders; and cooking demonstrations. Data on custom indicator MBNP 5.1.1.1 appears to be problematic. This indicator is worded in the PMP as “number of households who have established gardens for their domestic consumption.” In MBNP Year 3 Annual Report it is worded as “number of beneficiaries with home gardens or small livestock as a proxy for access to nutritious foods and income.” There is a significant difference in meaning between these. Many PSG members reported that they had gardens and/or small livestock before they became MBNP beneficiaries. In addition, it appears that at least some CHWs have inaccurately reported on the number of gardens that PSG members have established with MBNP support. For example, one CHW told the team that “almost all” PSG members had established a garden because of MBNP but was unable to show the team any MBNP-supported gardens within walking distance. Measuring and Calculating Indirect Beneficiaries Indirect beneficiaries: MBNP does not specifically collect, calculate, or report data on non-beneficiaries. This is in part because, with the way MBNP has defined its indicators, there are few, if any, possible indirect beneficiaries. In addition, interviews with MBNP staff indicated that they were not consistent in their definitions of direct beneficiaries. As an example, some counted all women of reproductive age and all children under five years of age in the villages where MBNP has operations, while others counted all those in the entire district where those villages are located. Some said women’s family members were direct beneficiaries. A few counted all people who heard a promotion message about pregnant and lactating mothers’ and babies’ nutrition or health from HFWs. This latter definition is consistent with Indicator 9.1.1.1 “number of women of reproductive age reached with support from MBNP-health facilities, community program.” According to its performance indicator reference sheet (PIRS), this indicator includes people who heard information on the radio (though an assessment of the campaign found very little message retention), heard MBNP messages from friends or neighbors, and who were exposed to MBNP messages through other means. Most said that, given their definition, there are no indirect beneficiaries.40 Based on the PIRS for 9.1.1.1, this appears to be the case. Attribution: MBNP collects data from health facilities that cover the entire population served in certain ways. MBNP reports data on seven indicators in its periodic reporting to USAID. Of these, two are on children under five: 1) 3.1. 9-15 “children under five who received Vitamin A from USG-supported programs” and 2) 3.1.9.2-3 “number of children under five reached by USG-supported nutrition programs.” The data for these are collected from the Health Facilities in the zones of interest where MBNP operates. HFWs told evaluators that the health facilities provided to MBNP information on all of the children who received Vitamin A at that facility and on all of the children who came to the facility for the first time. This raises the question of attribution. It is not clear that all children (as stated in the PIRS for this indicator) “coming for nutrition services to the health facilities for the first time and those who came for the first visit in the first year” are coming because of MBNP. Likewise, it is not clear that 40 One MBNP staff person said the CHWs and HFWs are indirect beneficiaries because they are getting the information even if they are not themselves pregnant or do not themselves have small children. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 29 MBNP is the reason that children receive Vitamin A at health facilities, though some HFWs asserted that some mothers are demanding Vitamin A for their children and attributed this to MBNP. According to the PIRS for Indicator 9.1.1.2 “number of women of reproductive age reached with support from MBNP-health facilities, community program” (reported in the 2013 annual report as “number of women of reproductive age reached by USG-supported nutrition programs”), data on women reached is to be collected from both CHWs and health facilities. One sub-partner staff person noted that, in some areas, the number of reported beneficiaries is higher than what the population of women and children was thought to be. It is not clear how double-counting is prevented, whether all women recorded in the health facilities’ data are specifically attributable to MBNP or how CHW, health facility records, and other estimates (such as number reached by radio) are compared and integrated. EQ4: Mwanzo Bora Conclusions Measuring and Calculating Indirect Beneficiaries Indirect beneficiaries: MBNP defines beneficiaries extremely broadly, such that all women of reproductive age and all children under five who visit a health center or who may have heard about MBNP through mass media are considered beneficiaries. This is too broad a definition to be used to assess indirect program benefits. While this definition generates high numbers for reporting to USAID, it obscures actual activity results (see also EQ5). Attribution: It is unclear whether the performance data routinely collected by health facilities, and reported to USAID, can be attributed to MBNP. In any case, in other contexts, these people might be considered as indirect beneficiaries. EQ4: NAFAKA Findings Diffusion of Knowledge/Promotion of Improved Technologies to Indirect Beneficiaries: The main vehicles for promoting improved technologies and sharing knowledge beyond direct beneficiaries used under NAFAKA are demonstration plots, informal sharing of information, and observation by non￾beneficiaries. A few FGD participants said that they actively shared information with family, friends, and/or people in neighboring fields. Some VBAAs and Lead Farmers indicated that they offered ad hoc support to farmers beyond those in their farmer group or with whom they were otherwise not working intensively. VBAAs and LFs were not consistent in how they counted those other farmers as direct or indirect beneficiaries when they report to NAFAKA. In most sites visited, uptake of new technologies beyond direct beneficiaries did not appear to be widespread. Participants of most FGDs provided anecdotes about other people observing the participants’ fields and copying some technologies, such as plant spacing. The most frequently cited limitations to wider adoption were cost and availability of the inputs promoted by NAFAKA, along with the labor intensity of some of the technologies. Measuring and Calculating Indirect Beneficiaries: NAFAKA reports on the custom indicator “number of beneficiaries reached.” The definition of this indicator has changed over time. In the 2013 PMP, it is listed as “number of targeted beneficiaries reached.” (The definition of direct beneficiaries in this indicator is different from the PMP’s definition under FTF indicator 4.5.2-13.) In periodic reports to USAID, the indicator is listed as “number of beneficiaries reached.” This indicator includes both direct Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 30 beneficiaries and their household members, who are counted as indirect beneficiaries who benefit from the direct participant’s ability to increase the household’s income or access to staple crops. These household members do not necessarily learn about the technologies. Indirect beneficiaries are not being counted under any other indicator. However, the way that they are included here is confusing and makes it appear as though they are direct beneficiaries. For Standard FTF indicator 4.5.2-13 “number of rural households benefiting directly from USG interventions,” the NAFAKA Annual PMP Data Matrix for FY14 states “a household is a beneficiary if it contains at least one individual who is a direct beneficiary. An individual is a direct beneficiary if s/he is engaged with a project activity or s/he comes into direct contact with the set of interventions (goods or services) provided by the project. Individuals merely contacted or involved in an activity through brief attendance (non-recurring participation) does not count as a direct beneficiary.” One senior NAFAKA staff person charged with M&E told the team that a person who attended a demonstration plot field day would be counted as an indirect beneficiary because that would fall under brief attendance and not￾recurring participation. Another senior staff person used a different definition, saying that anyone who attended a field day would be counted as direct. Different VBAAs and LFs used each definition. As one put it, he reported as a beneficiary “anyone who comes to my field day and hears the music and my announcements … If they do not sign in, I find out later who they were.” The .967 multiplier used to adjust for the potential of having more than one direct beneficiary in the same household (that is. 3.3 percent of direct beneficiaries live in a household with one other direct beneficiary) may be too high, leading to over-estimating the total number of beneficiary households (FTF 4.5.2-13) as well as of direct plus indirect beneficiaries. This multiplier is particularly problematic if those who attend demonstration plot field days are counted as direct beneficiaries; several field day organizers said families often attend. EQ4: NAFAKA Conclusions Diffusion of Knowledge: NAFAKA is measuring diffusion of benefits (to household members) rather than diffusion of knowledge (to others who can use it). It is difficult to assess the extent of promotion and diffusion of knowledge to indirect beneficiaries as different VBAAs and lead farmers—and different NAFAKA staff—define indirect beneficiaries differently. Measuring and Calculating Indirect Beneficiaries  NAFAKA indicator data is insufficient to provide evidence of program outcomes for indirect beneficiaries. Data issues include inconsistent indicator definitions and inconsistent interpretation of definitions by various project staff and by those who collect data on the ground. Reporting to USAID would be more accurate and management of sub-activities would be more robust if all were using the same definitions.  The utility of the custom indicator that confusingly combines direct and indirect beneficiaries is not clear, particularly since direct beneficiaries and number of beneficiary households are already being reported. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 31 EQ4: Tuboreshe Chakula Findings Diffusion to Indirect Beneficiaries: Many millers reported that fortification had helped them to greatly expand their sales. This included selling to new large-scale traders and wholesalers who did not serve local markets. As one put it “I am now about to be international…. selling to big wholesalers in Dar and farther away.” While the team was able to find fortified maize flour in local shops, it did not find fortified maize flour produced by TC-supported millers. Some millers said they used the dosifier for people who brought their own maize in to be milled. Some of the millers with micronutrient dosifiers were not using them at the time the team visited. A few said that demand for fortified flour was low, and one said he did not use the dosifier for maize milled for local people “because some people think it will make them sterile.” Another miller said that he did not tell local buyers that he was fortifying the maize flour because he was concerned they might not want to buy it, but said he was fortifying anyway because “it would make people healthier.” Measuring and Calculating Indirect Beneficiaries: TC was designed to focus work on millers and processers in order to create a supply of fortified food for households in FTF ZOIs. TC staff sometimes categorized the people who consume fortified products as direct beneficiaries and sometimes as indirect beneficiaries. For instance, TC (in contrast to MBNP) categorized people who heard radio advertisements, switched to a fortified product, or used micronutrient sachets as indirect beneficiaries. In this report, consumers are categorized as indirect beneficiaries. TC formally measured consumption by indirect beneficiary households in selected areas. To collect data for three of its four custom indicators related to FTF IR 6 “improved nutrition behaviors,” TC conducted a series of panel studies41 of households within a kilometer radius of selected millers. In its final survey42 (conducted in March and April 2015) TC found that, within the previous seven days, approximately 50 percent of the households surveyed had consumed fortified meal and 35 percent of households with children under five had used micronutrient sachets. However, the qualitative interviews in the urban and village areas where the team conducted its work do not provide evidence that supports these high numbers. The team did not hear from any PSG members who had used micronutrient sachets in the previous week (and PSG members are people who could be expected to be particularly sensitized to their utility). Almost no PSG members had heard of fortified maize meal. Some did blend flours on their own. In addition, the team members had difficulty locating shops that sold MNP sachets in many villages (including near health facilities) as well as in Dodomo and Morogoro towns.43 EQ4: Tuboreshe Chakula Conclusions Diffusion  TC’s benefits appear to go well beyond what are typically considered indirect beneficiaries and reach many unintended beneficiaries well beyond FTF ZOIs. This may not be the impact the Mission was seeking. 41 In panel studies, the same respondents are interviewed at several different times. 42 Tuboreshe Chakula, USAID Tanzania Feed the Future Tuboreshe Chakula Project Final Survey Report, May 2015. 43 As an example, team members spent more than 40 minutes seeking sachets in Dodoma town before locating any. While some former TC staff noted that there had been some MNP sachet supply problems, others said that the sachets should be readily available. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 32  While expanding their operations and fortifying maize flour is undoubtedly good for supported millers’ business growth, it appears that much of the fortified flour is leaving the local market, and is not being sold the intended indirect beneficiary households in FTF ZOIs.  Benefits to indirect beneficiary consumers are limited by low levels of awareness on the part of shopkeepers and consumers of the value of fortified maize flour and by some consumers’ suspicions about changes to their basic food staple. Measuring and Calculating Indirect Beneficiaries: The sample for the survey on consumption of promoted products did not adequately represent the expected range of consumers in the FTF ZOIs. While a panel survey can be useful in many circumstances, a survey that repeatedly (in this case, up to eight times in three years) asks a household about consumption of targeted “healthy” foods runs a high risk of introducing bias in the responses. EVALUATION QUESTION 5: What progress has each activity made in achieving its respective objectives as outlined in its performance management plan? Are activity objectives and targets realistic and achievable before the respective activity end dates? What results/outcomes have been achieved to date? EQ5: Overall Findings Breadth vs. Depth: Both NAFAKA and MBNP reached hundreds of thousands of beneficiaries, as those beneficiaries are defined under their indicators. However, it can be difficult to assess actual impact from these numbers. Interviews and site visits suggested that ideas and technologies promoted by the activities are far from internalized or institutionalized; significant reinforcement is still needed. The overall picture is of very high numbers and broad coverage that is sometimes superficial in terms of what people actually gain or in terms of FTF results. Some sub-partners and community-level leaders reported feeling pressure to provide reports with high numbers. A few community-level leaders said that the sub-partner supporting them primarily called or came to pick up reports and not to offer guidance, assistance, or even interest in progress. This led to some resentment. Reporting Accuracy: Some reported outputs were not observed as expected when the teams visited. The frequency of this suggests some exaggeration, at least some of which appears to be over-reporting by CHWs, VBAAs, LFs, and agro-dealers who self-report outputs. Targets and Delays: While outreach to beneficiaries exceeded targets in many cases, delays of inputs, training, or follow up limited impact. In NAFAKA, delays in seeds and input arrivals postponed implementation of new technology by a season in some places. For MBNP, the delays in organizing the Agriculture Kit and associated training limited potential outcomes. TC’s delays in installing fortification capacity limited the availability of fortified foods, and delays in market research and advertising fortified maize flour limited demand for maize fortifiers’ products. EQ5: Overall Conclusions Progress toward indicators is not the same as progress toward objectives. All activities devoted considerable resources to monitoring data collection and often meet or exceed targets, but that has not necessarily translated into meeting activity objectives or making substantial progress toward FTF IRs. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 33 When resources are limited, it is necessary to balance breadth (high numbers of beneficiaries affected) with the depth and quality of interventions to achieve outcomes and objectives. Some current custom indicators are not meaningful, such as using the number of beneficiaries with home gardens or alternate crops as proxy for access to nutritious foods and income. Similarly, collecting data on home gardens or raising small animals like poultry may be misleading if they are traditional practices and not attributable to the interventions. Delays in inputs, training packages, or follow up may limit progress, even when outreach exceeds targets. Some achievements are still incipient or fragile and may unravel without continued training, backstopping, and support. Lack of credit (particularly for women) limits farmers’ ability to take advantage of training and recommendations. SILC groups for the vulnerable households do not generate adequate amounts of credit for agricultural investment, many are not yet capable of independent operation, and there is no meaningful monitoring of changes in nutritional outcomes. EQ5: Mwanzo Bora Findings MBNP’s objectives are 1) strengthen the capacity of government and indigenous NGOs to deliver quality nutrition education and communication; and 2) strengthen the delivery of integrated community￾based nutrition services and social behavior-changing education, resulting in a model that can quickly be scaled-up to reduce child stunting and maternal anemia. Local Capacity: Progress toward MBNP’s first objective appears to be uneven. Several local NGO sub-partners have received support and capacity-building from MBNP. Most of these have successfully leveraged these in support of MBNP’s goals, but at least one did not. Many CHWs have been trained, and they interact with pregnant and nursing mothers in their communities; in the PSGs visited, the CHWs were greatly appreciated. More FeFA tablets were reported to be available at RCH facilities from improved use of existing government supply systems. MBNP plans to distribute tools to measure stunting and anemia in the RCH clinics. It is not clear what level of support will be allocated to regions outside new extension areas. MBNP’s progress in building the capacity of Tanzania Food and Nutrition Centre (TFNC) and Centre for Counseling, Nutrition, and Health Care (CONSENUTH) has been slow, and expected handover of responsonsibilities has not occurred as planned. The team observed frustration on the part of these organizations with the delays and poor communication, the financial implications of the responsibilities they would be expected to take on, and the slow pace of the transfer of responsibilites from MBNP to them. Breadth, Depth, and Attribution: Progress towards MBNP’s second objective appears high in terms of numbers reported, but appears to be lower in terms of substantive outcomes achieved. MBNP reported that, as of the end of 2014, it had covered 826 (98.2 percent) of the 841 health facilities providing RCH services and reached 1,335,618 women of reproductive age. However, in opportunistic interviews with pregnant or nursing women in those villages (some in NAFAKA farmer groups), most indicated they had not heard of MBNP or PSGs or the “1,000 days” program for child growth. Those who visited the RCH clinic may have still benefited from greater availability of FeFA or better advice about how to take it, but attribution to MBNP of all women taking FeFA is difficult. Data on many of MBNP’s outcome indicators will not be collected until near the end of the activity. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 34 Reporting on home gardens provides an example of the difference in approach between measuring outcomes and reporting large numbers. The MBNP PMP from December 2013 includes the custom indicator “number of households who have established home gardens for their domestic consumption.” The PIRS stipulates that it measures households establishing a home garden for domestic consumption after specific MBNP trainings. The PMP also includes “Number of people trained on home vegetable gardening.” Together, these indicators would provide MBNP with information on outcomes and efficacy of training. However, in the FY14 Annual Report, neither of these indicators is reported. Instead, “number of beneficiaries with home gardens or small livestock as a proxy for access to nutritious foods and income” is reported (with 82,000 beneficiaries). The FY15 first quarter report uses the same text but splits it in two, reporting individually on number of beneficiaries “with home gardens” (41,337) and “with small livestock” (82,760). It is noteworthy that these indicators say that the beneficiaries have a garden or small livestock but do not include any attribution to an MBNP role in their existence or productivity. This is further complicated by the origin of the data. For example, some CHWs that had reported gardens (including one who reported that “almost all” PSG members had gardens) were not able to show the team a PSG member garden within a reasonable walking distance. According to one staff person, the number of beneficiaries reported in some areas appeared to be larger than the total population of women of reproductive age and children under five. The latter could be due to double-counting, such as of people who attended both demonstration days and health facilities and of people who are thought to have been reached by the media campaign. In its FY14 Annual Report, MBNP reported that its Siku 1000 radio campaign had reached approximately 2,663,153 people. However, the British Broadcasting Corporation Media Action, in a consultancy for MBNP, researched the impacts of MBNP’s “Siku 100” radio campaign and found that most respondents did not recall the campaign. This was attributed to the time lapse between the campaign and the research. 44 No information was available to the team on whether the research also looked at whether people remembered or applied any messages from the campaign. Still, the more than two million people reached appear to be included in MBNP set of beneficiaries. MBNP’s 2014 Annual and Quarterly reports report only outputs, such as number of beneficiaries reached or kits distributed, but no outcomes, such as changes in behavior or impacts like reduced anemia or stunting. Health facility staff, mothers, and CHWs reported a subjective impression of healthier children and of fewer mothers needing referrals to larger health centers, but there was no hard data on this. A quick analysis of monthly reports at several health facilities showed a slight decrease in children in the “red” (under-nutrition) category compared to a year ago. Given all of the factors that can affect under-nutrition (household income, rainfall and crop harvests, other potable water and sanitation projects being implemented in the same villages, etc.), it is not clear how much of this can be attributed to MBNP specifically. Primary reports older than one year were not available to the team (see also the section on attribution under EQ4). EQ5: Mwanzo Bora Conclusions The long term impact of MBNP’s interventions depends on strong Tanzanian instititions to manage activities in the future, so this merits a higher priority and adequate transition time. Some targets appear 44 MBNP Year 3 Annual Report, October 2014. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 35 to be higher than realistic. To reach them, MBNP has to count people only marginally touched by the activity as direct beneficiaries. It is difficult to assess the number of people actually reached. Having stakeholders, such as CHWs, self-report of the number of beneficiaries served can introduce incentives for over-reporting, and it is not clear whether double-counting exists. It is difficult to see how a model can be developed (Objective 2) without data to compare the effectiveness of different interventions. The results of work with PSGs are likely quite good, but they are not being measured. The training on strategic health communication was commendable, but inadequate to develop an entire system to select the best practices for the model. Conclusions from the evaluation team’s field interviews are that the lack of a robust monitoring and evaluation system that can measure effects of key MBNP activities periodically during the course of its work substantively impedes MBNP ability to learn from its work, scale up its most successful sub-activities, showcase successes, and course￾correct when outcomes are not what was expected.45 The team also concurrs with the observation in the DQA report that the activity’s M&E seems oriented toward producing large numbers for reporting on standard FTF indicators to USAID.46 Focus on achieving the high numbers is not balanced by indicators that measure outcomes that are clearly attributable to the activity’s likely very good work. Some earlier versions of indicators that would have allowed MBNP to measure progress and impact were shifted to indicators that can generate large numbers—but some of those numbers may have little relation to MBNP itself. EQ5: NAFAKA Findings NAFAKA’s objectives are 1) improving competitiveness and productivity of the rice and maize value chains; 2) facilitating greater domestic and regional trade; 3) increasing benefits from the growth of the maize and rice subsectors, particularly those to women and youth; and 4) unleashing innovation and private sector development.47 For the most part, indicators for the first three objectives appear to cover the expected outcomes, and progress appeared to keep pace with or surpass targets. (See the indicator table in Annex VI.) NAFAKA achieved its targets for rice and maize productivity and for trainings and linkages to improve trade. In its FY14 Annual Report, NAFAKA reports that a cumulative total of more than 100,300 rural households had benefited (138 percent of its life-of-project target). The high numbers of beneficiaries reported do not directly translate into high levels of impact. Even farmers that are applying the new technologies may not be applying them very intensively. Many farmers (particularly maize farmers—and in some cases all members of a given FGD) noted that, while they achieved higher yields with the new technologies, they did not plan to apply the promoted maize (or rice) technologies to all of the land they planned to plant 45 This is consistent with the observation from the 2013 data quality assessment (DQA) that “…we are unable to discern a methodology or system for the on-going measurement of the effectiveness of MBNP-supported training, capacity-building, and social and behavioral change.” [The Mitchell Group, 2013. Data Quality Assessment (DQA) on the Mwanzo Bora Nutrition Project (MBNP) indicators during March-May 2013.] 46 As the 2013 DQA put it, the “M&E focus appears to be largely on “numbers,” causing us to question if the larger, more comprehensive, outcome-oriented vision of project achievement is being overshadowed and forsaken for the need to produce output-oriented results data.” 47 ACDI/VOCA. Staples Value Chain Activity – NAFAKA – Performance Management Plan (FY 2011 – 2016). Washington, DC: ACDI/VOCA, 2013. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 36 in maize (or rice). Reasons they cited for not using the new practices/technologies more intensively or on more land included, labor, high cost of inputs, lack of finance to purchase inputs, concern about earning enough from the harvest to make the investment worthwhile, concern about lack of market to sell their produce, becoming dependent on purchased inputs, etc.. This was the case for many who had been NAFAKA beneficiaries for multiple years as well as for new beneficiaries. Despite impressive improvement in yields and gross margins, nearly all of farmers, community, and NGO implementers interviewed asked for more training. Many of the rice and maize farmers interviewed were happy with increased productivity, but indicated that they lacked credit to continue to use inputs. Accessing markets continues to be a challenge for farmers. Some maize farmers noted that even though the improved seed performed better than traditional seed during drought, they did not see much economic gain. Trainings to increase trade and markets are in process, but for many farmers interviewed, the marketing was still a puzzle. Some expressed reluctance to become dependent on purchased inputs, leaving them vulnerable to delays or price increases. The indicators for Objective 4 are less effective in measuring accomplishment of the objective. Objective 4 does not define a “women focused value chain.” The primary intervention under Objective 4 is SILC groups, where 66 percent of members are women. The number of SILC members reported under FTF 4.5.2-14 “Number of vulnerable household benefiting directly from USG assistance” (NAFAKA somewhat tautologically defines those who join SILCs as “vulnerable”) and the value of saving accumulated both surpassed the targets. However, it should be noted that 67 percent of the vulnerable households participating in a SILC belong to a SILC that was formed during FY14. SILCs typically take several years to become independent. If support to these SILCs is substantively reduced or eliminated with the move to the new implementation areas, these incipient gains are at risk. While there are no indicators that explicitly trace the pathway to food quality, availability, consumption and utilization, NAFAKA does have an Objective 4 indicator on gardening, for which it accomplished just three percent of its FY14 target. In FY14, 375 SILC members (approximately 10 percent of all SILC members) received training on gardening. For vegetable gardens, NAFAKA recently scaled down its targets while changing the way it reports beneficiaries in a way that substantially increases reported output. NAFAKA’s FY14 Data Matrix shows a total of 400 for “Number of beneficiaries with home gardens or alternate crops as a proxy for access to nutritious foods and income.” In its FY15 second quarter report, NAKAKA uses a different definition for the indicator, this time counting beneficiaries as all household members who have access to a home garden, effectively nearly quadrupling its output. Data on Scale: There has likely been some over-reporting of direct beneficiaries. As described earlier, some VBAAs and LFs who were interviewed indicated that they used a broader definition for direct beneficiary than did their peers and some NAFAKA M&E staff. One agro-dealer who was interviewed said that his contract with NAFAKA stated that he was to work with 1200 farmers, but that he had not reached that many yet. Two other agro-dealers each told the team that they worked with 1200 farmers (the similarity of the numbers may or may not be a coincidence). One of them cited a much higher number for famers supported by an LF that he worked with than the LF herself gave. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 37 EQ5: NAFAKA Conclusions NAFAKA has made strong gains in many areas. It appears that a few indicators are designed to show high numbers in a way that can obscure actual impact. Reallocation of substantial staff and budgetary resources to the new implementation areas may put incipient results at risk. The good agricultural practices and rice intensification technical packages are good models for improved productivity, so integration with markets and credit must be high priorities next. NAFAKA still needs clarity on expectations regarding interventions and outcomes regarding gender equality and improved nutritional status. EQ5: Tuboreshe Chakula Findings TC has two objectives: 1) to increase competitiveness of agro-processors, and 2) to increase consumption of nutritious processed products. TC successfully met or exceeded many of its agro-processer business development targets. For example, in FY14, TC reported that it significantly exceeded its FTF FY14 targets for profitability (127 percent), and improved technology and management (141 percent)48 for processors. TC appears to have improved the profitability and efficiency of selected processors and increased their capacity to fortify flour. Agro-processors interviewed confirmed that they are now better run and more profitable. However, TC was less successful with its second objective. While processers legitimately claim increased sales to traders, local sales of fortified flour are minimal, and processors stated that there is little local demand. In 2014, TC reported that 29 percent of households with children under five near supported millers use MNP. The team could not confirm the increased consumption of fortified flour and MNP reported by TC. The team’s interviews and focus groups identified very few mothers who used MNP or who had heard of maize meal fortification, even in the PSGs who received the most intense nutritional education. EQ5: Tuboroshe Chukula Conclusions  There is now a cadre of processors producing fortified flour or capable of it, once demand increases. Many of the processors participating in TC activities have better financial and management skills as well as new business contacts. Selection of processors should have been more focused in the beginning. This would have helped to track impact and to consolidate gains.  Data on household consumption that does not provide a representative sample of overall consumption in FTF ZOIs is of limited utility for impact reporting and planning. EVALUATION QUESTION 6: How effective are the implementation approaches used by each implementing partner in achieving intended objectives and results and reaching scale? EQ6: Mwanzo Bora Findings MBNP partnered with government institutions and helped build their capacity to deliver improved RCH services in nearly every village in the implementation districts. Government institutions are challenging 48 Tuboreshe Chakula Annual Report 2013-2014 Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 38 partners because of frequent staff turnover (especially in rural areas), often poor accountability, chronic underfunding, and resistance to change. Many officials, especially within agriculture, have a poor understanding of nutrition, and therefore may not promote or fund nutritional interventions. PSGs appear to be effective for behavior change interventions. New mothers in PSGs said that they used exclusive breast feeding for six months. Most PSG members stated that they now consume more vegetables and animal protein as a result of MBNP’s education. They said the cooking lessons helped them make vegetables taste better and made it easier to increase their families’ consumption. Members without home gardens claimed to purchase vegetables for household consumption. Through CHWs and PSGs, MBNP educated women in the community on feeding practices, food selection and preparation, and complementary health practices (deworming, malaria treatment, and supplementary intake of FeFA tablets) to improve absorption and utilization of food. Most CHWs and PSG members (and some HFWs) equate good nutrition with eating more vegetables, rather than with consistent consumption of balanced meals. Most people who have not had any nutritional training equate good nutrition with adequate staples during the year, so this is an improvement, but still not an adequate understanding of human nutritional needs. Recorded messages minimize dilution of message, although it still occurs. For example, during a PSG interview, some mothers attributed inaccurate information to CHWs, including “red meat is bad for women.” This was probably a confused version of the message that women should eat more white meat (poultry, fish, and rabbits) because it is cheaper and less likely to be taken by men than red meat. Although every village has local extension officers assigned to them, very few were engaged with the PSGs. Staff indicated that that they would be working with extensionists more when the new agriculture kits are introduced. Extension Agents interviewed were aware of the MBNP activities and expressed interest in supporting them, but did not know of explicit expectations to work together. Some CHWs ceased working with PSGs and on MCH issues when expected sub-partner remuneration and support was not available. This raises questions about sustainability after MBNP ends. While use of a variety of sub-partners appeared to enhance local capacity in some sub-partner organizations, it also fragmented aspects of implementation and led to some inconsistencies in methods and messages. Use of multiple sub-partners diluted MBNP’s ability to control messages, monitor progress, and provide support where needed, which increased the amount of staff time and resources MBNP needed to train, support, and provide oversight. At least one sub-partner performed poorly for quite some time before corrections were made. Sub-partner staff and CHWs requested additional training and support. Many CHWs appeared to require more training, support, and oversight than they received. Although MBNP does not measure and compare impact of all its different modes of outreach, staff and sub-partners consistently stated that street theater and workshops with community leaders had the biggest impact, especially for engaging men and building widespread support for novel behaviors. EQ6: Mwanzo Bora Conclusions MBNP’s implementation strategy of strengthening existing government and community institutions is a sustainable approach as improved capacity in the health facilities and local government offices will remain Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 39 behind after MBNP funding ends. Given the starting point of some institutions, the time frame may be too short. The TOT approach to training CHWs reached a huge number of people very quickly, but the quality of the training varied and the resulting skills are uneven. PSGs are resource intensive, and it appears that, with strong CHWs, they can be effective at supporting behavior change. However, this has not been monitored or verified. EQ6: NAFAKA Findings NAFAKA uses a value chain approach and works with the private sector to support building up a strong value chain for rice and maize with increased production and profitability for farmers. The value chain approach does not necessarily fit well with farmer decision making and resources. Staff and sub-partners view small farms as enterprises, but many farmers are still operating with a subsistence approach. The behavior change materials, such as calendars and posters, are well designed and appreciated, but are limited to technical messages. The team saw little evidence of them outside of activity offices. Working through a variety of sub-partners helps to build local capacity and leverage the important organizational capacities and contacts that sub-partners can bring. The VBAA and LF extension models operate very differently, and VBAA and LF work areas sometimes overlap; they also overlap with areas where NAFAKA supports agro-dealers to do outreach to farmers. One LF listed over a dozen recent visitors and said most came to check data on progress rather than to offer substantive support. Individual VBAAs and LFs varied widely in their agricultural knowledge, understanding of agricultural inputs, and ability to help farmers choose the most appropriate product. VBAAs facilitated availability of more inputs to rural areas. Introducing a profit motive can be a good incentive for educating farmers and can be an important underpinning to sustainability (for example, a few VBAAs have already become agro-dealers). The downside is that the profit motive may color the types of recommendations they provide to farmers, such as recommending agro-chemicals over integrated pest management. In some areas, support for agro-dealers to market more aggressively to small farmers may be affecting the sustainability of VBAAs. NAFAKA has chosen not to work with government extension agents. NAFAKA’s recent work on collective storage for rice and maize was given mixed reviews by farmers interviewed. While many indicated that they liked the opportunity to delay sales to receive higher prices, others indicated that they disliked the delay because it prevented them from selling grain when they needed the cash. A few farmers noted that the transparency of record keeping meant that wives now knew the full extent of the harvest and its income. Work with vulnerable households was assigned to a separate sub-partner that brought in high-level SILC skills. However, there was almost no integration of the work with vulnerable households under Component 4 and the bulk of the NAFAKA activity, which is geared towards rice and maize production. The available loan sizes in the NAFAKA SILCs are largely insufficient for investing in the technologies and practices promoted by NAFAKA. Most SILC members invested their loans in trade or small livestock (e.g., goats or chickens), rather than crops or gardens. Many SILC members said that the opportunity to have cash available for emergencies was more important than loans. This is consistent with the design of the SILC mechanism and the way that SILCs usually operate. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 40 Work in many locations, with many community-level leaders, and with many beneficiaries began in 2014 and early 2015. Many are not yet able to independently apply the new ideas and technologies. EQ6: NAFAKA Conclusions The value chain approach and focus on just maize or rice does not address farmers’ needs to integrate all of their crops, livestock, and water resources into a meaningful year round cycle. Most small farmers in the ZOI practice mixed farming, and the transition to commercial farming can be disruptive. The behavior change posters and calendars are well done, but address only the technical aspects of the change to commercial production. Commercialization of smallholder agriculture goes beyond the marketing of surplus products, to include changes in household decision-making, objectives of production, household participation in input and output markets, degree of specialization in production and dependence on markets for income and consumption.49 The multiplicity of sub-partners results in a multiplicity of approaches, some of which are complementary and some of which are not. This has resulted in an array of people visiting community￾level leaders—and in an array of disparate messages being given to community leaders and farmers. This creates unnecessary competition and confusion. While this could offer an important means for testing the strengths, weaknesses and utility of the variety of approaches being used by different sub-partners and different sub-components of NAFAKA, it doesn’t appear that this is being done. At the same time, Component 4 vulnerable populations’ relative isolation from valuable support offered through other components limits their ability to improve rice and maize production. Cascading training resulted in some dilution of messages, so reinforcement of training is still needed. Incipient results and sustainability of early results are at risk, especially for vulnerable populations and those who recently began participating in NAFAKA sub-activities, if NAFAKA is not able to continue support in at least some locations after the expansion into the new implementation areas. SILCs are not an ideal mechanism for boosting agricultural productivity. While SILCs are documented to be useful for vulnerable households, it is not clear why SILCs were chosen as part of a rice and maize production activity or why support to vulnerable households is largely limited to SILCs. The SILC methodology is designed to be a savings rather than a loan mechanism. SILCS are not well suited to be providers of loans for agriculture. Using SILCs for agricultural loans risks jeopardizing a SILC’s sustainability. For example, they were not designed to be able to provide multiple large loans at once (e.g., during planting season). Because they are internally funded, members bear all risks, and drought or other agricultural shocks could deplete the core funds and wipe out all members’ savings. 49 Moti Jaleta, Berhanu Gebremedhin and Hoekstra D. 2009. Smallholder commercialization: Processes, determinants and impact. Discussion Paper No. 18. Improving Productivity and Market Success (IPMS) of Ethiopian Farmers Project, ILRI (International Livestock Research Institute), Nairobi, Kenya. (https://cgspace.cgiar.org/bitstream/handle/10568/27/SmallholderCommercializationProcesses_DiscPaper18.pdf?seq uence=1) Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 41 EQ6: Tuboreshe Chakula Findings TC’s implementation approach focused primarily on the private sector and included grants and training for private businesses to improve capacity for fortification and profitability. Grants appeared to increase profitability but were not tied to nutritional outcomes in FTF ZOIs. Fortification of flour was promoted only by grants to purchase the SANKU Fortification Company (SANKU) dosifier machines, with embedded services with the manufacturer. Consumers received little effective information promoting fortification, and consumer demand remained low. For example, a participating grain processor in Morogoro observed that while TC made a lot of effort to train the millers on the importance of micronutrient fortification, the same was not done to the consumers, and therefore effective consumer demand was not realized. The use of MNP to treat under-nourished children has not been taken up well. Although nearly all PSG members know about MNP, very few had used the sachets due to expense and poor availability. Some CHWs reported that mothers who purchased the MNP sachets in the past rarely use them in the adequate quantity or frequency, and often use one dose for an entire family’s meal, and only once a week; information from PSG members confirms this. Availability is low. The team spent half a day trying to find a local shop in Kibaya town (Kiteto District) to buy the sachets, and only one of the visited health facilities (in Mvomero District) had them for sale. EQ6: Tuboreshe Chakula Conclusions Dosifers are expensive, and may be too costly for small scale processors to purchase on their own. There appeared little impact on nutrition from fortified maize in the ZOIs because availability and use of remained so low. EVALUATION QUESTION 7: To what extent have the activities worked with local partners? What are the results in terms of strengthening local institutional capacity, ownership, and long-term sustainability? EQ7: Overall Findings All activities have worked with local partners of different types and capacities. Variation is seen in 1) levels of capacity of different institutions; 2) levels of knowledge and motivation of different individuals in the same roles; 3) approaches to implementation; and 4) levels of achievements and potential for sustainability. EQ7: Overall Conclusion There is an ample array of potential local partners to work with in the central government, local government, non-governmental sector, and private sector. Few of the sub-implementers observed are capable of continuing the activities begun without further financial support, and in some cases, organizational strengthening. EQ7: Mwanzo Bora Findings Civil Society Organizations (CSOs): MBNP is working with a local CSO in each district of operation. Examples include Faraja Trust Fund in Morogoro Municipality, Tanga AIDS Working Group Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 42 (TAWG) in Mvomero District, UMWEMA Group in Kongwa District, KINNAPA in Kiteto District, and Sharing World in Dodoma Municipality. These CSOs are supported by MBNP through training on necessary nutrition actions, including skills on behavior change communication, financial management, and some essential materials, such as office furniture, computer and printing facilities, and personnel expenses. Selection of CSOs was done competitively. Some CSOs perform better than others, and one recently had to be replaced due to poor performance.50 Government National: MBNP collaborated with the Ministry of Health and Social Welfare’s nutrition arm (the TFNC) to prepare standardized training manuals for various target audiences (workers in agricultural extension, reproductive health, social work, and community development along with community leaders, CHWs, and district multi-sectorial nutrition committees). Regional and District Nutrition Committees: It was clearly pointed out in all three Regions (Morogoro, Dodoma, and Manyara) that although District and Regional Multi-Sectorial Nutrition Committees were formed some years before the advent of MBNP, they were not functional (e.g., they had hardly met). Many of the stipulated members reported that it seemed that the meetings were not important. MBNP strengthened the committees by supporting their regular sittings (logistics support, etc.). Health Facilities: MBNP is working with the existing Health Facilities, particularly the RCH department. HFWs reported that trainings took place over a year prior to the interview and that some trained colleagues had been transferred out of the health facilities. The interviewed HFWs noted that much of the knowledge that was brought by the MBNP experts through training was not really new, except for techniques for social behavioral change communication. However, a few indicated that it was useful to have MBNP generating enthusiasm and support for reproductive and child health. Community-Level: MBNP revitalized the functioning of the CHWs in the communities. There are usually two per village (one man and one woman) appointed by village government. The positions were established by the National Health Policy, but were not very active before the MBNP intervention. CSO representatives (and the CHWs themselves) indicated that CHWs needed to get more frequent training on their day to day activities. CSO representatives indicated that they were not provided with sufficient budget for that. EQ7: Mwanzo Bora Conclusions CSOs: The CSOs are the ones responsible for implementing the day to day activities of MBNP. This is good practice to support sustainability. Poorly performing CSOs severely handicap MBNP’s ability to achieve desired results. However, these CSOs do not have the budget (and some do not have the capacity) to continue the project activities on their own after MBNP leaves. Government National: With more support, TFNC can play a greater role in working towards MBNP objectives. 50 Diocese of Central Tanganyika, which was working with MBNP in Dodoma Municipality and Bahi District, was replaced by another CSO known as Sharing Worlds. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 43 Regional and District Nutrition Committees: MBNP strengthened these committees and has given their work new meaning. The challenge now is how to sustain them when MBNP ends its support. Health Facilities: MBNP built the capacity of RCH staff to better understand and adopt the necessary actions for improving health of women and babies in the first 1,000 days of life Health Facilities: HFWs are key to on-going outreach to women of childbearing age through educational messages and physical inputs like FeFA and health monitoring. Community-Level: MBNP is supporting the CHWs as vehicles to impart necessary knowledge and skills to community members. They have become important links between community members and health facilities. CHWs need additional backstopping and refresher trainings in order to improve their functioning and to improve prospects for sustainability of impact. The PSGs appear to be a useful platform for learning from person to person. EQ7: NAFAKA Findings NAFAKA worked with several categories of local partners: 1. Sub-Partners: Local service providers were engaged as sub-partners: Rural and Urban Development Initiative (RUDI), Muungano wa Vikundi vya Wakulima Tanzania (MVIWATA), Catholic Relief Services (CRS), Farm Input Promotions Africa (FIPS), International Fertilizer Development Center (IFDC), and KINNAPA. These sub-partners helped to provide services, including formation of farmers’ groups (MVIWATA) and establishment of SILCs by CRS. NAFAKA provided capacity-building for local sub￾partners (particularly in financial management) and paid for project personnel. 2. Community Extension Service Providers: To increase farmer access to improved services and inputs— such as seed, fertilizer, and agro- chemicals—NAFAKA established and/or provided support to VBAAs and LFs. NAFAKA builds their capacity through Training of Trainers on good agricultural practices and business management. This helped to create linkages amongst various service providers and farmer customers. Use of demonstration plots supervised by these service providers is an important aspect of their extension work. Many of the VBAAs and Lead Farmers interviewed indicated that their ability to sustainably continue their work would be greatly improved by additional training and access to credit. NAFAKA is encouraging formation of VBAA Associations, which can serve as official distribution points for seed and fertilizer inputs at the village level. The association structure is designed to allow VBAAs to purchase inputs in bulk at a lower cost. So far there are four such associations in Kiteto, Kongwa, Mvomero, and Kilombero. 3. District Extension Officers: To a limited extent, NAFAKA worked with district or ward extension officers in supervising farmers who are involved in producing Quality Declared Seeds. 4. Partnerships with Private Entities: To increase farmer access to improved services and inputs, such as seeds, fertilizers, and agro-chemicals, NAFAKA worked with agro-dealers who were selected using a competitive process. Those selected were given training on business management and supported to link with agro-input traders. Those who were interviewed indicated that their businesses have grown since joining the NAFAKA collaboration. A few agro-dealers were also serving as lead farmers among the farmer groups. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 44 5. Agricultural Research and Training Institutions: Ilonga Research Institute, Kilombero Agricultural Research and Training Institute (KATRIN), Cholima Research Institute, and Mkindo Farmer’s Training Centre have conducted trials to determine suitable fertilizers and good agricultural practices in various ecological conditions and provided training the farmers. For example, Ilonga Research Institute was involved in identifying appropriate alternative crops for intercropping with maize in Kongwa and Kiteto. EQ7: NAFAKA Conclusions NAFAKA worked with many of the local and international organizations already present in the FTF ZOI. Work with the Tanzanian organizations contributes more to local organizational sustainability than work with international organizations if appropriate and sufficient organizational strengthening and capacity building is provided. NAFAKA has attempted to create structures at lower levels (VBAAs, LFs, and associations of VBAAs). It has also strengthened local entities (e.g., agro-dealers) by building capacity to market to and address smallholder farmer needs. After a critical mass of demand for inputs is established in remote areas, this farmer-supportive system is likely to be able to continue (i.e., once promoted good agricultural practices and input purchases have been adopted by a sufficient number of farmers, this will create a demand for necessary inputs, such as improved seeds, which can be provided by the agro-dealers and/or the VBAAs). While this critical mass appears to have been built in some locations where NAFAKA operates, creating this critical mass in other locations will take more time and more support. EQ7: Tuberoshe Chakula Findings Most importantly, TC engaged small and medium milling and oil processing enterprises in fortifying cereals and sunflower oil to improve the population’s nutrition. TC worked with a number of other local partners, including:  Government based: Ministry of Health and Social Welfare, TFNC, Tanzania Food and Drug Authority (TFDA), National Food Fortification Alliance (NFFA), and Tanzania Bureau of Standards (TBS).  Community-based: Community leaders and CHWs. (However, where CHWs were not functional, TC appointed community facilitators referred to as Community-owned Resource Persons.) TC initiated a process of setting a regulatory policy to enforce cereal grain and oil fortification in Tanzania by working with various government institutions. TC did some work with community leaders toward creating a general acceptance of MNP sachets in the communities. EQ7: Tuboreshe Chakula Conclusions TC’s recent work with local millers substantially increased the amount of fortified maize flour potentially available to Tanzanians in FTF ZOIs. Efforts to implement regulations on food fortification and to create substantive awareness of and demand for fortification in rural areas have not yet succeeded. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 45 EVALUATION QUESTION 8: How are the activities perceived and valued by stakeholders, including direct beneficiaries, indirect beneficiaries, and GOT counterparts? EQ8: Overall Findings While most of those interviewed reported that they appreciated what they learned from the three activities, many intended beneficiaries said they lacked the resources to fully adopt or implement the new ideas and/or purchase the promoted inputs. EQ8: Overall Conclusions In general, all three activities were valued by beneficiaries. Key factors in the lack of application of new knowledge (agricultural or nutritional) include price of inputs, accessibility of inputs, lack of knowledge about inputs, and climate/weather. EQ8: Mwanzo Bora Findings PSGs: PSG members voiced a clear appreciation for the concept of six-month exclusive breast feeding. Many members liked the image/metaphor of a child as a flower. Members expressed appreciation for learning about the value of vegetables in the diet and new cooking techniques. In most PSGs, nearly all reported increased consumption of green vegetables by the family. Gardens: Many members expressed appreciation of the idea of vegetable gardens. Adoption of home vegetable gardens was not widespread, especially in dry areas and among poorer families. Some expressed concern about the price and availability of seeds and pesticides. Small Animals: Where poultry production was promoted by CHWs, PSG members reported receiving little new information or support. Few members mentioned raising rabbits to improve family nutrition (and the few interviewed CHWs appeared not to be skilled in raising them). Micronutrient Powder Sachets: Most PSG members had heard of MNP sachets. Some PSG members had purchased MNP sachets and others had received them for free from a health facility. Many noted that they were expensive. (The price of the retail sale sachets was in general found to be 100 Tanzanian shillings—a price based, in part, on a pricing study that included information from target purchasers). Many PSG members (and CHWs and HFWs) noted that it was difficult to find MNP sachets to purchase.51 Almost no PSG members reported purchasing and using MNP sachets in the quantity recommended by MBNP. Revolving Loans: In some PSGs where a revolving fund mechanism had recently been introduced, women noted that that the requirement to pay into the fund had led some women to reduce or cease their participation in the PSG. While the mechanism was described by some as a SILC, the best practices for operating a SILC did not appear to be well-understood. Objectives and protocols for operating the mechanism protocols were not consistently understood by CHWs. 51 The team also had difficulty in finding MNP sachets, once spending 40 minutes in Dodoma town following leads until a shop that stocked them was found. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 46 CHWs: CHWs expressed strong appreciation for the kits. Most CHWs requested more frequent training and more in-depth training. Some CHWs expressed frustration with the level of financial and other support provided by MBNP, especially those serving large villages without transportation. HFWs: All HFWs interviewed expressed appreciation for MBNP. They attributed the following to MBNP: 1) improved uptake of FeFA tablets, and 2) women coming to health facilities earlier in their pregnancy. HFWs cited MBNP’s emphasis on having men accompany women to the health facilities as positive, but noted that, while more men were coming to the health facilities, there were still very few. HFWs expressed appreciation for the training provided by MBNP. They asked for more HWFs to be trained, for more frequent training, and for more in-depth training. Some noted that MBNP’s attention to nutrition motivated them to emphasize it more, though it had always been part of their job. Government: Members of Multi-sectoral District Nutrition Committees (MSDNCs) reported benefiting from MBNP training, financial support, and advocacy at all levels of government EQ8: Mwanzo Bora Conclusions PSGs Gardens and Small Animals: While useful, gardens may not be the most appropriate vector for improved nutrition for women participating in PSGs, particularly in dry areas. CHWs have provided limited added value on raising poultry or rabbits. The newly developed agriculture training kit appears useful for improving training on small livestock, gardens, and staple crops. Micronutrient Powder Sachets: While most PSG members expressed appreciation of the concept of MNP sachets, uptake is limited by cost and availability. Revolving Loans: As implemented, the PSG revolving fund mechanism does not appear to be seen as providing sufficient value to override its negative effects. CHWs: Kits are an important and effective source of information for PSG members and CHWs. HFWs: While much of the content of MBNP training was not new to most HFWs, MBNP increased HFWs’ focus on nutritional aspects of maternal and child health promoted by the activity. Government: MBNP’s support to regional and district government representatives appears to have increased local government interest in nutrition for the MBNP target population. EQ8: NAFAKA Findings Local Leaders: VBAAs, SILC PSGs, and Lead Farmers (particularly women) noted that experience in NAFAKA had increased their skills, their confidence, and their status in the community. Some group leaders expressed frustration that the level of financial support and/or other material support provided by NAFAKA was low in comparison what was provided by other activities. Farmers Individual Farmers: Most participating farmers said they appreciated the new knowledge and technologies. More farmers expressed more enthusiasm for future use of specialized seeds than for future use of agro-chemicals. Most reported increased yields (particularly of rice). Many reported that, even though they experienced yield increases, they did not plan to apply the new technologies at a large scale. Primary reported barriers to application of NAFAKA technologies were lack of cash or credit for inputs, Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 47 risk of dependence on purchased seed, lack of water/rainfall, timeliness of availability of promoted inputs, labor, and low expected sales prices. Farmer Groups: Most participating farmers valued membership in a farmers’ groups. Youth were reported to be less interested in participating in farmers’ groups than older farmers. A few farmers reported that poor health could make their membership in a group too burdensome. Some farmers received access to credit through participation in farmers’ groups and some did not. A few farmers reported that group membership led to some loss of independence in ability to sell harvest when best for their family, which they saw as imposing a financial burden. Agro-dealers: All agro-dealers and most VBAAs indicated that their participation was already or was expected to be profitable. SILC Groups Loans: SILC membership is steady and loan repayment rate is within industry standards. Many SILC members were especially appreciative of the “social fund” used to help each other in emergencies. Some SILC members indicated that the loans available through their groups were too small to allow them to invest in the agricultural technologies being promoted by NAFAKA. Payment for Services: Very few SILC groups had made the transition from free (project supported) FA to paying a PSP for managing their savings and lending. Many SILC members were reluctant to pay for business training offered through NAFAKA. Some sub-partner staff indicated that the time allocated for supporting SILCs to start in new areas and to become independent was insufficient. Government: Many government agricultural extension agents reported knowing little about NAFAKA. Some district-level government officials reported knowing little about NAFAKA. EQ8: NAFAKA Conclusions Local Leaders: Women VBAAs and lead farmers particularly valued participation in NAFAKA. Farmers Individual Farmers: Farmers’ application and scaling-up of use of inputs promoted by NAFAKA is limited by risk perception and cost of inputs. Farmer Groups: Different types of farmers perceived different types of costs and benefits from participation in farmers’ groups. SILC Groups: Loans: While SILCs are valued by members, available credit from NAFAKA SILC groups (which are made up of people from vulnerable households) is insufficient to cover the costs of the technologies promoted by NAFAKA because of limitations in on-lending and/or because of the SILC group rules that constrain borrowers’ maximum. Payment for Services: The short time frame (one or two cycles) during which SILCs are able to receive NAFAKA support to establish the groups appears not to be enough to convince some members (especially in vulnerable groups) to pay for the PSP, which limits sustainability. Government: NAFAKA is missing an opportunity for potential government uptake of and/or support for the interventions supported by NAFAKA. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 48 EQ8: Tuberoshe Chakula Findings Millers and Processers  Value of TC Support: Maize millers and sunflower oil processers interviewed all expressed appreciation for training, grants, equipment, and expanded networks gained through TC.  Value of Fortification: Many millers noted that the value of fortification was the ability to sell to large￾scale traders whose markets were outside FTF ZOIs, including outside of Tanzania. Many millers reported that they were not selling (any or much) fortified maize flour to local customers and/or that local customers did not ascribe much value to fortified maize flour (or actively avoided it). Some millers interviewed were not using the dosifiers provided through TC. Consumers: A very small minority of people interviewed in towns and villages in FTF ZOIs had heard of fortified maize flour. Some retailers visited by the team did not stock it. None of the visited retailers stocked fortified maize flour from millers who participated in TC. Some retailers were not aware of what fortified maize flour is. Government: Government counterparts appreciated work with TC, but wanted more support. EQ8: Tuberoshe Chakula Conclusions Millers and Processers: Millers and processers valued TC for having helped them to improve and expand their business, including beyond their local markets. Many millers saw less value in fortification than in the other benefits provided by TC. Consumers: General consumers (particularly in rural areas) and many retailers do not appear to value maize flour fortification. Government: Increased work with the government could have added value. EVALUATION QUESTION 9: What unexpected results (negative consequences or positive results) of the activities have been realized? Which of these should be considered in implementation or follow-on activities, or documented for further dissemination? EQ9: Overall Findings Representatives from NAFAKA and MBNP indicated that they would have done more work together and more internal integration of nutrition and agriculture if that had been required of them by their indicators and by USAID. EQ9: Overall Conclusions See EQ2 overall conclusions on overall FTF integration and interactivity integration. Sufficient incentives for integrating agriculture and nutrition are not yet fully realized in the design and implementation of NAFAKA and MBNP. EQ9: Mwanzo Bora Findings Communication  According to both partners and ex-post research, the radio programming implemented through the Social Behavioral Change Communication campaign did not have the anticipated effects. “Despite Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 49 wide accessibility, communication through radio may not be the best medium to channel one time nutrition messages; and for sustainability and good behavior change outcomes, one to one communication could be the best option.”52  HFWs expressed particular appreciation for brochures produced by MBNP and for the recordings in the kits. They requested more brochures and educational materials as well as kits for the health facilities.  The PSGs appear to be a good platform for dissemination of information from person to person within the community. Vegetables  Support for vegetable gardening appeared to be particularly appreciated where water was plentiful and not as widely adopted in areas without reliable rainfall or easy access to piped water or irrigation.  While MBNP promoted vegetables as a crop for consumption, many PSG members reported that they were selling much of what they grew in order to obtain cash.  Respondents in several PSGs noted that their increased vegetable production (for household consumption and/or sale) reduced community dependence on vegetables from elsewhere—and reduced the income for vegetable sellers in their community.  Women in several PSG’s reported that, because they are preparing more vegetables and preparing them in tastier ways (due to MBNP cooking demonstrations), their husbands are eating at home more and the marriage is happier. Sustainability: In areas in Dodoma where the Diocese of Central Tanganyika (DCT) had been the primary sub-partner, some CHWs reported that they had stopped working because they were no longer getting direct support from MBNP. Some PSGs in this area reported that they had stopped meeting well before the sub-partner was replaced. EQ9: Mwanzo Bora Conclusions Vegetables: Because some of the harvest from improved vegetable gardens was sold and/or replaced vegetables that the family had previously purchased, the presence of vegetable gardens is not a valid proxy for improved nutrition. Sustainability: It appeared that the problems with the sub-partner in Dodoma could have been identified and rectified sooner. The reactions of CHWs and PSG members interviewed raise the question of how sustainable the work of the CHWs and PSGs will be after direct MBNP ceases. EQ9: NAFAKA Findings  Rice and Vegetables: In areas where NAFAKA promoted fertilizers for rice crops, farmers reported increased production from vegetables they grew on the rice plots after rice harvest. 52 MBNP Year 3 Annual Report (2014) Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 50  Barriers to Adoption: Cost of seeds and other inputs was consistently cited by farmers as significant barriers to adoption (including adoption on more than a portion of a farmers’ land) of technologies promoted by NAFAKA. Timeliness of inputs provided or facilitated by NAFAKA was also frequently cited as a barrier. Some farmers asserted that increased harvests (of both rice and maize) had reduced the market price of those commodities. They expressed reluctance to invest in inputs because they were concerned they would not recuperate the outlay when selling their harvest.  SILC: Most SILC group members reported that they had not received information on the agricultural technologies promoted by NAFAKA and that the technologies were too expensive for them to engage in with the size of the loans they were able to leverage in their groups. EQ9: NAFAKA Conclusions  Rice and Vegetables: Those who farmed both rice and vegetables with NAFAKA support saw substantive benefits in both vegetable yield and income.  Barriers to Adoption: Access to credit and markets continues to be a challenge for many farmers; the latter is particularly important when they have increased yields.  SILCs: The SILCS did not appear to be well integrated into the overall implementation of NAFAKA’s productivity and marketing work. EQ9: Tuberoshe Chakula Findings See TC findings under EQ4. Consumer Preferences: There was less consumer awareness, understanding, and interest in fortified maize meal than anticipated. Some consumers were concerned about negative effects of fortification. Some millers interviewed were not using dosifiers. Others indicated that they were not labeling maize flour as fortified or otherwise not telling consumers the flour was fortified. Sale of fortified maize outside of the FTS ZOIs: According to millers, a large percentage of the fortified maize they were producing was being sent to markets far away from the milling location and outside of the FTF ZOIs. Therefore, much of the maize that was fortified was not available to consumers in the FTF ZOIs whose nutrition FTF programming was designed to enhance (the production of fortified sunflower oil was too recent for outcomes to be identified). EQ9: Tuberoshe Chakula Conclusions See TC “Diffusion” conclusions under EQ4. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 51 LESSONS LEARNED A. The current M&E system focuses on numbers reached or outputs accomplished, rather than progress towards objectives (depth of impact). Activity implementation therefore skews to meeting numerical targets, rather than accomplishing objectives and goals. This tension between producing large numbers for reporting to Congress (breadth of reach) rather than progress towards objectives (depth and sustainability of impact) creates risks of superficial outcomes. M&E systems should include more monitoring of outcomes and reduce both the overall number of and focus on targets that can only be accomplished with superficial “reach” (sometimes with questionable attribution) to large numbers of beneficiaries. B. FTF interventions are designed for long-term impact; yet activity cycles are short for accomplishing behavior change and changes in government, farmer, and household capacity and practices. Rapid scaling up and an extremely large (more than a million people) beneficiary base with weak connections to program activities risks lost opportunities and wasted effort if activities exit before behavior changes and institutional capacity are fully sustainable. This is of particular importance as activities are asked to shift their focus to new implementation areas in Mbeyi and Iringa and where new villages and large numbers of new beneficiaries have been brought into activities within the last 18 months. 1. Sufficient staff and budget resources are needed to consolidate gains in “older” implementation areas—particularly in areas with beneficiaries that have been brought in recently. 2. Reaching fewer beneficiaries with longer contact and more intensive content can boost retention of information and use of new practices. 3. An M&E system should provide timely information to support assessment and management of activity components. 4. Consider a longer project cycle for such ambitious goals. C. The activities (NAFAKA and, to a lesser but still important extent, MBNP) are so large, cover such extensive geographical areas, and include so many sub-partners, sub-contractors, and collaborating entities that it is difficult for managers to have adequate knowledge about field situations. This makes it difficult for them to make agile and informed decisions and to correct problems in a timely fashion. Future activities would do well to have fewer moving parts and fewer subcontractors with different outreach styles and to have strong systems in place to monitor sub-partner/sub-contractor work and provide support as needed. D. The use of multiple sub-partners that are expected to do the same work (with different institutional cultures, approaches, and levels of capacity) along with working through a large number of community-level leaders with a wide range of skills and capacities creates risks of dilution and distortion of key activity messages. The use of “kits” with recorded training/modules has been very successful at helping accurate training reach local populations. The concept is worth continuing to build on. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 52 E. High-level activity staff indicated that improved integration of agriculture and nutrition and cross￾activity collaboration are more likely to occur if the requirement for it is spelled out with objectives, indicators, and outcomes, as well as detailed work plans but that there is little incentive to take on what is perceived as additional work that is outside of what is required to be monitored and reported to USAID. F. All stakeholders do not have a good working knowledge of nutrition or gender. 1. Most stakeholders, including many staff and government officials, define good nutrition as adequate staples for the year, and do not recognize the importance of the diverse basket of nutritious food on a regular basis, for all people, not just the vulnerable populations. All stakeholders, including IPs, sub-partners, community-level leaders and government, should understand good nutrition and be able to define the pathways through which agriculture supports good nutrition within their area of expertise. 2. Most stakeholders, including most staff and government officials, define gender equality as women’s empowerment (which results in promoting gender balance in groups and in leadership) or as women’s ability to earn income. USAID identifies gender equality as an essential element of all successful development and defines it as changes in attitudes, behaviors, roles and responsibilities at home, in the workplace, and in the community. Women’s empowerment goals, outcomes, and indicators function best when they include changes of beliefs, attitudes, and behaviors of both men and women. 3. Orientation of all stakeholders on the goals of FTF and how it is different from earlier (and less successful) interventions because it seeks to overcome the traditional isolation and separation of agriculture and nutrition would support integration of agriculture and nutrition. G. All activities are aware that attempts to change women’s behavior, status, and health require active engagement with men, but progress is still limited. Workshops with staff and focus groups as well as brief surveys can elicit enough qualitative and quantitative information to establish the best interventions to reach men in the new target areas. H. Working through community-level leaders, such as VBAAs, LFs, and CHWs, is an important means of reaching scale and of building local capacity and sustainability. Use of local leaders who area already part of local government structures (such as CHWs) is an effective practice. Supporting people to become extension entrepreneurs (such as VBAAs) also appears to be effective. Without a comparison of the outcomes achieved by VBAAs and LFs—as well as clarity on the strengths and weaknesses of the VBAA and LF models—it is difficult to know which of these is the more appropriate model for expanding work into new areas or for designing new activities. I. Affordability of inputs was consistently named as the primary barrier to adoption of new technologies and practices (including agricultural practices for NAFAKA beneficiaries and MNP sachets and gardens for MBNP beneficiaries). 1. Small farmers can have difficulty implementing many of the suggestions for improving productivity at scale in their fields without access to adequate credit. Credit (especially credit that does not require land as collateral and beyond what is available and appropriate through Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 53 SILCs) and market linkages for small farmers and smaller processers are vital links in the rice and maize value chains. 2. Pricing and availability of important additions to diet, such as MNP sachets can stymie uptake, even when the concept is appreciated by mothers of vulnerable small children. J. SILCs are well-suited as a savings mechanism for vulnerable households, but the SILC concept was not designed as a mechanism for providing loans for income-generating activities. Leveraging SILC funds for agricultural loans that tie up the funds for long period of time and carry risks that can affect many borrowers at once (such as drought) can risk the loss of an entire group of vulnerable people’s savings. K. Assuring that the benefits from fortified maize flour reach vulnerable populations and others in FTF ZOIs will require a combination of: 1. Outreach and education of consumers and retailers to create demand (with the education design based on a sound understanding of family food purchase decision-making and drawbacks to fortification perceived by consumers). 2. Facilitation of distribution of fortified foods to rural retailers (including outreach to and education of retailers). 3. Creating means of maize flour fortification that are suitable for small millers in urban and rural communities, where households take their own maize to be milled in small batches. While large￾scale dosifiers like the SANKUs are appropriate for producing large amounts of fortified maize meal for distribution and retail sale, many vulnerable households produce, receive from family, or purchase maize and have it ground at nearby mills as needed. For these people, and for small millers, batch fortification, such as the scoop method, can be a better vehicle for getting the benefits of micronutrient-fortified maize flour. . Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 54 RECOMMENDATIONS EVALUATION QUESTION 1: To what extent have women and youth been integrated into the design and implementation of the activities? Which approaches have proven most effective in reaching women and youth? How have women and youth benefitted from the activities? EQ1: Mwanzo Bora Nutrition Program Recommendations 1. Women A. Gender Analysis: MBNP should conduct gender analysis before initiating work in new areas to help identify beliefs, attitudes, and behaviors that impede adoption of health and nutrition advice, and to inform design of strategies to address them. Indicators on social change at the local level can help evaluate the most effective strategies and to provide feedback on progress. B. Outreach: MBNP should strengthen outreach to men, community leaders, and older women (particularly mothers-in-law) to help change attitudes towards healthy nutritional behaviors (e.g., pregnant and lactating women eating more high-protein foods) and increasing young mothers’ role in household decision-making. Because meetings with community leaders and roadshows were said to be effective, plan and budget for more of these, along with follow up. C. Data Quality: MBNP should review the data on men’s participation in PSGs and make adjustments (to the data collection or to outreach on men’s support for RCH and nutrition) if needed. 2. Youth A. Young Mothers: MBNP should determine whether very young mothers need a more specific outreach strategy. B. Older Women: MBNP should design and conduct specific outreach and messaging for older women who have influence over young mothers and family decision-making. C. Schools: For future similar programming, consider expanding outreach into schools to help prepare young men and women for parenthood. EQ1: NAFAKA Recommendations 1. Women A. Training: NAFAKA should enhance gender training for staff and sub-partners to assure that all are aware of how implementation, facilitation decisions, and practices can positively or negatively affect women’s participation and outcomes for women. NAFAKA’s new Human Rights package should include strategies for discussing gender issues in different cultural settings. In the expansion areas, NAFAKA should provide the full Human Rights training early in the implementation cycle, along with follow up workshops. 2. Youth A. Youth Strategy: NAFAKA should finalize its Youth Outreach Strategy. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 55 B. Youth Farmer Groups: NAFAKA should facilitate creation of new youth farmer groups. C. Youth Farmer Activities: NAFAKA should assess the effectiveness of trial projects such as agro￾chemical handling as a business for youth groups, and its feasibility for scaling up. NAFAKA should consider facilitating farmer youth groups’ applications for the Mabilioni ya Kikwete (Presidential Fund for Youth = Billions from President Kikwete). EQ1: Tuboreshe Chakula Recommendation Consumers: Future fortification projects should start with research on the concerns that female and male consumers have about fortification and should address the benefits of fortification to the family. EVALUATION QUESTION 2: In what specific ways have the activities integrated nutrition and agriculture interventions in their work? Have these integration efforts delivered intended results? What are these results and how have they been achieved? EQ2: Overall Recommendations 1. Overall FTF Integration: If USAID/Tanzania wants activities to more actively and effectively integrate agriculture and nutrition as part of their implementation, the Mission should: A. Make the expectation of integration clear in the activities’ objectives at the design phase. B. Make sure that the activities’ PMPs explicitly include both indicators that measure nutrition outcomes and those that measure agricultural outcomes. C. Consider including one or more custom indicators that specifically address integration. D. Require that activities’ work plans clearly lay out how the integration will take place. E. Require and support FTF activities (including MBNP and NAFAKA) to create opportunities to provide cross-fertilization and collaboration. Encourage collaboration among the activities by having regular meetings to reflect on their common mission to improve the nutritional status of the FTF target population. Plan concrete steps for collaboration, such as cross-trainings. 2. Inter-activity Integration: Work plans for NAFAKA and MBNP’s extension periods should include collaboration between NAFAKA and MBNP (with other FTF activities as appropriate) to take advantage of their unique expertise and of opportunities to reach different target groups. EQ2: Mwanzo Bora Nutrition Program Recommendations 1. Site Appropriate Interventions: MBNP should use basic information on existing conditions in each area to select the best nutrition interventions for local circumstances. Variables include water availability; traditions of vegetable production, preservation, and consumption; women’s autonomy; availability of edible wild plants; access to seeds and inputs; women’s available time; and existing food practices. Possible interventions in addition to gardens, poultry, and rabbits should be explored. Rabbit keeping should not be encouraged unless adequate support and training is made available. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 56 2. Agriculture Kits: MBNP should ensure that adequate training is provided to CHWs and local extension officers who will use the new Agriculture Kit, including refreshers and backstopping. The Kit should be provided to villages where Nutrition Kits are in use (many CHWs are expecting them). 3. Improve Understanding of Nutrition: Strengthen training and outreach to regional and district government officials and agricultural professionals to build understanding of key concepts and commitment and momentum toward improved nutritional outcomes. EQ2: NAFAKA Recommendations 1. Increase Action on Nutrition Language in Activity Documents: NAFAKA should fulfill language currently in activity documents by providing increased attention to nutrition. A. Provide Nutrition Training to Beneficiaries: If SILC groups are designed to be the primary vehicle for improving the nutrition of vulnerable populations, they should be provided with training on nutrition, expanded training on vegetable gardens, and assigned core staff with knowledge of vegetable gardens and nutrition to provide or supplement garden trainings. B. Measure a Nutrition Proxy Output: Refine the definition of the custom indicator on gardens to focus on home consumption of garden produce. C. Provide Nutrition Training to Staff: Provide training for core and sub-partner agronomists, managers, business experts, and others on the linkage between agriculture and good nutrition. Emphasis should be on demonstrating that well implemented agriculture can lead to improved nutrition through consumption of a diverse basket of food by all members of the household and that good nutrition can enhance agricultural productivity through improved health. EQ2: Tuboreshe Chakula Recommendations 1. Linkages and Coordination A. Coordinate with FTF Activities: Future FTF activities should create opportunities for coordination and for linking participants to potential benefits available through other FTF activities, including maize producers and consumers in vulnerable populations and FTF ZOIs. B. Coordinate Messages: Where activities do work together, assure close coordination so that messages provided by each activity complement (rather than contradict) each other. EVALUATION QUESTION 3: To what extent have the activities reached scale in terms of: outreach to direct and indirect beneficiaries; adoption of new technologies and practices; and adoption of innovative business models? EQ3: Overall Recommendations 1. Depth of Impact A. Depth vs. Breadth: For future programming, consider using more outcome indicators that can both demonstrate results and incentivize implementers to focus on achieving depth of impact. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 57 B. Sustainability: When allocating budget and staff to new implementation areas, assure that adequate budget and staff are allocated to consolidate gains in previous implementation areas. EQ3: Mwanzo Bora Nutrition Program Recommendations 1. Depth of Impact: MBNP should increase attention to adoption of promoted practices including conducting additional site visits and back-stopping for CHWs to improve utility of trainings, success of gardens, etc.; broadening use of the Nutrition Kit recordings; and using theater and other community-level messaging. 2. Kits and Training for HFWs: Provide additional training and support to HFWs, including providing support for the use of educational recordings from the Nutrition Kits with women waiting to be seen by HFWs. EQ3: NAFAKA Recommendations 1. Depth of Impact/Sustainability: Fill Knowledge Gaps: NAFAKA should conduct a diagnostic needs assessment to assess key gaps in the knowledge that VBAAs, LFs, and local sub-partners need to effectively support farmers and the potential sustainability of their work, while also providing targeted support and additional training where needed. This is particularly important for areas where NAFAKA initiated work in the last two years. 2. Data Quality: Assess the methods used for collecting and extrapolating data from the Annual Outcome Survey to ensure that they accurately represent the overall scope of NAFAKA’s work. EQ3: Tuboreshe Chakula Recommendations 1. Assure Benefits Reach Intended Targets in ZOIs: Future fortification activities should: A. Research and Design: Clearly and carefully define who the intended (direct and/or indirect) beneficiaries of the fortified products are and where those beneficiaries are located. Based on that, the activities should build in mechanisms to facilitate (and, if necessary incentivize) distribution in FTF ZOIs (other intended target areas) and sales to targeted consumers. B. Consumer and Retailer Education and Demand: Increase outreach and education, focusing on benefits of fortified maize for shopkeepers and consumers. There should be particular focus on those in FTF ZOIs, both before and after the products appear. C. Rural Millers and the Scoop Method: In rural areas and with small millers, promote the “scoop” method of fortification, in which millers use a small scoop to measure and add prepared bulk micronutrients to buckets of maize that are brought to the mill for milling into flour. This is especially useful in rural areas where small-scale millers are unlikely to invest in a SANKU dosifier. (This will only be viable if consumers understand the importance of micronutrients in the diet.) D. Bulk Sales to ZOI shops: Work with millers, processers, wholesalers, traders, and other bulk buyers of fortified products to facilitate linkages to sales outlets in ZOIs. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 58 EVALUATION QUESTION 4: To what extent are activities promoting and measuring diffusion of improved technologies or sharing knowledge of improved practices beyond direct beneficiaries? If indirect beneficiaries are being reached, what are the primary vehicles through which this is taking place, how are indirect beneficiaries being measured, and how are benefits calculated? EQ4: Mwanzo Bora Nutrition Program Recommendations 1. Measuring and Calculating Indirect and Direct Beneficiaries A. Beneficiary Definitions: USAID should work with MBNP to determine whether it is programmatically reasonable—and whether it serves FTF programmatic interests—to define as direct beneficiaries 1) all women of reproductive age and all children under five; 2) all of those who visit RCH clinics, or 3) all women of reproductive age who are estimated to have heard any message related to MBNP. B. Data Quality i. PIRS: MBNP should assure that the PIRS in its PMP clearly and accurately describes who can be counted as a beneficiary. Indicators should remain consistent throughout the project cycle (except where changes are agreed to with the Mission). MBNP should assure that all MBNP and sub-partner staff, all CHWs, and all health facilities that are reporting data on beneficiaries consistently use the same definitions. ii. Field Visits to Verify CHW Data and Provide Support: MBNP and sub-partner staff should work closely with CHWs to assure they understand how to count beneficiaries for each indicator. MBNP and sub-partner staff should conduct more frequent and more in-depth field visits to verify data reported by CHWs (as well as to provide on-going training and support and recognize signs of poor performance earlier). 2. Diffusion of Knowledge A. Kits and Training: See EQ3 Recommendation 2. B. Depth of Impact: Model Approach: Especially if MBNP intends to develop and scale up a model approach, MBNP should 1) increase the pace of developing and using a system to measure and assess which interventions produce the best, most cost effective, and most sustainable outcomes; 2) monitor increased benefits to those who participate in PSGs to assess the utility of resources spent; and 3) fine tune implementation for greater impact. EQ4: NAFAKA Recommendations 1. Data Quality: Measuring and Calculating Indirect Beneficiaries A. Have a DQA done for NAFAKA. B. NAFAKA should improve the reliability and validity of the data that it reports to USAID. Design clearer indicator definitions in its PIRS and PMP Data Matrix Activities, making sure that they are easy for all stakeholders (particularly those charged with collecting and reporting data to NAFAKA) to understand. They should periodically verify that all stakeholders involved in data collection are using the same definition and are collecting information as required. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 59 C. NAFAKA should consider changing the wording of the custom indicator from “Number of beneficiaries reached” to “Number of direct beneficiaries and their household members reached.” D. NAFAKA should review the data on which the .967 multiplier (which is based on an estimation that in 3.3 percent of households where one member is a direct beneficiary of NAFAKA there is also another household member who is a direct beneficiary) is based and make adjustments as needed. Assess the sampling frame to assure that it is adequately representative of the overall set of NAFAKA beneficiaries and regional variations. EQ4: Tuboreshe Chakula Recommendations 1. Diffusion: See EQ3 Recommendations A (Research and Design), B (Consumers and Retailer Education and Demand), C (Rural Millers and the Scoop Method), and D (Bulk Sales to ZOIs). 2. Measuring and Calculating Indirect Beneficiaries A. Measurement of consumption of fortified foods in future activities should be done by taking a sample that covers the range of indirect beneficiaries who are expected to be reached. B. Surveys that ask respondents relatively frequently about socially desirable behaviors should be designed to reduce potential response bias. In such cases, panel surveys should be avoided. EVALUATION QUESTION 5: What progress has each activity made in achieving its respective objectives as outlined in its performance management plan? Are activity objectives and targets realistic and achievable before the respective activity end dates? What results/outcomes have been achieved to date? EQ5: Overall Recommendations 1. Depth of Impact A. Depth vs Breadth: The Mission should consider reassessing the activities’ approach to monitoring, moving from a focus on numbers reached to more meaningful progress towards objectives.53 B. Sustainability: Implementers should carefully structure the allocation of staff and budget resources to assure sufficient support for consolidation of incipient or fragile gains in the earlier implementation areas while expanding into new areas. The Mission should balance support for the existing and new implementation areas and support appropriate transitioning. EQ5: Mwanzo Bora Nutrition Program Recommendations 1. Depth of Impact A. Indicators: MBNP and the Mission should revisit the M&E system. They should include indicators that allow for measurement of outcomes and that generate feedback on progress and impact directly related to MBNP implementation. MBNP and the Mission should reduce, redefine, or 53 This is in line with a recommendation from the 2013 Mwanzo Bora Data Quality Assessment: “…there is a need to develop better analysis methodologies across the entire FTF/T initiative to more readily extrapolate progress toward key project and program objectives from the plethora of output data, i.e. numbers, being generated.” Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 60 eliminate indicators that depend on data that is not directly attributable to MBNP implementation activities. B. Model Approach: See EQ4 Recommendation 2B. 2. Data Quality A. DQA: MBNP should have another DQA. B. Field Visits to Verify CHW Data and Provide Support to CHWs: See EQ4 Recommendation 1B ii. EQ5: NAFAKA Recommendations 1. Depth of Impact/Sustainability A. Continue to Support Non-extension Areas: NAFAKA should continue to support work already begun in Morogoro, Manyara, and Dodoma. Support to rice has been especially successful, but full rice and maize productivity achievements are not yet sustainable. Results for vulnerable household served under Component 4 are particularly at risk. B. Future Activities: Should consider a longer time frame or more intense training and refreshers. 2. Provide Nutrition Training to Beneficiaries: See EQ2 Recommendation A. EQ5: Tuboroshe Chakula Recommendations 1. Consumer and Retailer Education and Demand: See EQ3 Recommendation B. 2. Bulk Maize Flour and Small Batch Milling: Future fortification activities should work with small millers in rural areas who can provide fortification (e.g., by the scoop method) for small batch milling for household maize and with millers who have the potential for large throughput. EVALUATION QUESTION 6: How effective are the implementation approaches used by each implementing partner in achieving intended objectives and results and reaching scale? EQ6: Mwanzo Bora Nutrition Program Recommendations 1. Sustainability/Depth of Impact: A. Fill Knowledge Gaps: MBNP should allocate increased staff time to providing support and oversight to sub-partners as well as helping them to provide improved support and oversight to CHWs. This should include allocating budget to more frequent refresher trainings, observations for accuracy of messaging, and hands-on support for CHWs to improve the quality and sustainability of their work, especially in the extension area (Iringa and Mbeya Regions) as well as in areas where gains are not yet consolidated and sustainable. Local support for CHWs must be established before MBNP exits or activities are not likely to continue. 2. Government Collaboration and Coordination: A. CHW Sustainability: MBNP should plan an exit strategy so that support from local government will replace the honorarium, training, and supervision currently provided by MBNP. Options for Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 61 support may include payment of some allowance (particularly to cover travel costs), a bicycle, tax credit, assistance with farm work, and public recognition as an incentive to continue work past the end of MBNP. B. National Government Activities: MBNP and/or future FTF activities should intensify efforts to educate government officials and strengthen Tanzanian institutions tasked with creating and enforcing the regulatory framework impacting nutritional outcomes. This could include increased support provided to TFNC to ensure it functions as needed to provide technical support in implementing nutrition projects and programs as well as monitoring them, including fortification claims required by law and verification of labeling claims. C. Local Capacity Building: Future FTF programs should coordinate with and continue to build capacity of local institutions. EQ6: NAFAKA Recommendations NAFAKA should: 1. Assess Approaches Used: Assess strengths, weaknesses, and utility of the variety of approaches being used by different sub-partners and sub-components and prioritize use of the approaches that are most successful, particularly in the new implementation areas. 2. Depth of Impact/Sustainability: Fill Knowledge Gaps: See EQ3 Recommendation 1. 3. Vulnerable Populations: Incorporate Component 4 vulnerable populations more fully into other aspects of NAFAKA, including the rice and maize value chains. 4. Social Impacts: Address the social changes associated with commercialization, along with the technical and financial training. (See also EQ1). EQ6: Tuboreshe Chakula Recommendations For future fortification interventions: 1. Sequencing: Plan the sequencing of consumer outreach with capacity to deliver fortified products. 2. Linkages and Coordination: See EQ2 Recommendations A and B. 3. Rural Millers and the Scoop Method: See EQ3 Recommendation C. EVALUATION QUESTION 7: To what extent have the activities worked with local partners? What are the results in terms of strengthening local institutional capacity, ownership, and long-term sustainability? EQ7: Overall Recommendation Implementing partners should further emphasize capacity building of local partners in the planning of activities. This is particularly important where the activities have shorter implementation time frames, such as in the new expansion areas and in areas where implementation started in the last two years. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 62 EQ7: Mwanzo Bora Nutrition Program Recommendations 1. Government Collaboration and Coordination A. National Government Activities: See EQ6 recommendation 2B. B. Regional and District Nutrition Committees: Develop an exit plan so that the MSDNCs as well as the regional-level committees will continue after MBNP funding ends. Involve the MSDNCs and representatives from the Ministry of Agriculture and local extension workers in the deployment of the Agriculture Kits. C. Health Facilities: Increase training opportunities for HFWs, providing more frequent training to more individuals, and to cater for staff turnover. 2. Sustainability/Depth of Impact: A. Fill Knowledge Gaps: See EQ3 Recommendation 1. B. CHWs: See EQ6 Recommendation 2. C. Local Sub-Partners: MBNP should increase resources for staff working with local sub-partners to build skills where needed. This should include on-the-ground observations not arranged far in advance and not at locations chosen by sub-partners to identify potential issues with work quality before they become problems. EQ7: NAFAKA Recommendation Depth of Impact/Sustainability: Fill Knowledge Gaps: See EQ3 Recommendation 1. EQ7: Tuboreshe Chakula Recommendations Future activities that focus on food fortification should provide increased attention to 1. Government Agencies and Regulations: Collaboration with and support for government agencies working on fortification and nutrition and to supporting national-level regulatory structures and regulations. 2. Consumer and Retailer Education and Demand: See EQ3 Recommendation B. EVALUATION QUESTION 8: How are the activities perceived and valued by stakeholders, including direct beneficiaries, indirect beneficiaries, and GOT counterparts? EQ8: Overall Recommendations Assess primary barriers to application of new knowledge and technologies and develop and apply means of addressing them. EQ8: Mwanzo Bora Nutrition Program Recommendations 1. Nutrition, Gardens, and Small Livestock A. MBNP should consider alternatives to a primary focus on vegetable gardens when teaching and promoting access to improved nutrition. If needed, make adjustments to the training to further Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 63 encourage consumption of foods rich in nutrients typically missing from the local diet. MBNP should promote indigenous and traditional vegetable consumption, along with all the elements of a healthy diet. CHW’s skills in rabbit production must be improved before rabbits can be effectively promoted. B. MBNP should plan and budget for adequate TOTs for use of the new agricultural kit as well as for means for trainers to reach outlying communities. It would be valuable to apply the kit in existing all MBNP regions (not just the new extension regions). 2. Nutrition and Agriculture Kits: MBNP should continue to utilize kits as a central element of MBNP. See also EQ2 Recommendation 2. 3. Revolving Loans: MBNP should strongly reconsider introduction of a revolving fund into the PSG model, particularly the utility and appropriateness of imposing what is seen by many as a requirement to pay for PSG membership, and consider dropping the fund concept. If the revolving fund concept is retained, MBNP should redesign it and provide significantly higher levels of training and support to CHWs supporting the funds. They should create a clear means of participating in PSGs without contributing to the fund. 4. Sustainability/Depth of Impact: A. Fill Knowledge Gaps: CHWs: See EQ6 Recommendation 2. B. CHW Sustainability: See EQ6 Recommendation 2A. C. Government Collaboration and Coordination: Regional and District Nutrition Committees: See EQ7 Recommendation 1B. 5. Micronutrient Powder Sachets A. MBNP could support increasing the availability of MNP sachets, including availability at health facilities. B. MBNP, or a future FTF activity, could conduct a marketing study to determine customer perceptions of—and appropriate pricing for—MNP sachets at the village level and conduct outreach to shopkeepers about stocking them. EQ8: NAFAKA Recommendations 1. Depth of Impact/Sustainability A. Fill Knowledge Gaps: See EQ3 Recommendation 1. B. Continue to Support Non-extension Areas: NAFAKA should reassess the amount of time that SILCs receive support from NAFAKA and increase the time where needed. This is particularly important in areas where the SILCs have been active for less than three years and where they serve especially vulnerable people. 2. Input Affordability: NAFAKA should increase attention to affordability of inputs, especially access to credit and support for cost/benefit analysis of different input types. Help farmers work through Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 64 the cost/benefit analysis of input purchases, and include price risks in the analysis. Increase attention to safety nets, crop insurance (especially in drought-prone areas), etc. 3. Government Collaboration and Coordination: NAFAKA should consider doing more outreach to district-level government officials involved in agriculture and economic development. NAFAKA should keep government agricultural extension agents informed about—and actively involve them in—NAFAKA activities. 4. Vulnerable Populations: See EQ6 Recommendation 3. EQ8: Tuboreshe Chakula Recommendations 1. Millers and Processers: Future fortification activities should include a focus on sales of fortified products within the FTF ZOIs. 2. Consumers: Future work on maize flour fortification should include a strong focus on assessing and creating consumer demand for fortified maize flour—particularly within the FTF ZOIs. 3. Government Collaboration and Coordination: Future work should include more attention to collaboration with government agencies working on fortification and nutrition. EVALUATION QUESTION 9: What unexpected results (negative consequences or positive results) of the activities have been realized? Which of these should be considered in implementation or follow-on activities, or documented for further dissemination? EQ9: Overall Recommendations See EQ2 overall recommendations on overall FTF integration and interactivity integration. EQ9: Mwanzo Bora Nutrition Program Recommendations 1. Kits and Training for HFWs: See EQ2 Recommendation 2. 2. Nutrition and Vegetables A. Consumption vs. Sale: MBNP should assess whether a sufficient proportion of PSG members’ vegetable production is being consumed by the pregnant or lactating women and small children in the members’ household. B. Access to Water: MBNP could consider working with regional and district government to put in place boreholes that can be used for irrigating vegetable gardens through the Tanzania Social Action Fund livelihood enhancement program. C. Gender Analysis: See also EQ1 Recommendation 1A. EQ9: NAFAKA Recommendations 1. Barriers to Adoption: NAFAKA should increase attention to A. Input Affordability: See E8 Recommendation 2. B. Timeliness of Input Availability: Work with suppliers of key inputs being promoted to farmers in particular areas to facilitate availability when needed during production seasons. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 65 C. Post-Harvest Management: Increase attention to the post-harvest loss management, sales, and marketing links in the rice and maize value chains. 2. Vulnerable Populations: See EQ6 Recommendation 3. 3. Rice and Vegetables: Continue to promote vegetable production to rice farmers and include nutrition training for vegetable growers. EQ9: Tuboreshe Chakula Recommendations Diffusion: See EQ3 Recommendations A (Research and Design), B (Consumers and Retailer Education and Demand), C (Rural Millers and the Scoop Method), and D (Bulk Sales to ZOIs). Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 66 ANNEXES ANNEX I EVALUATION STATEMENT OF WORK ANNEX II EVALUATION METHODS AND EVALUATION DESIGN MATRIX ANNEX III DATA COLLECTION INSTRUMENTS ANNEX IV DOCUMENTS REVIEWED ANNEX V PERSONS INTERVIEWED ANNEX VI PERFORMANCE AGAINST PMP INDICATORS ANNEX VII DISCLOSURE OF ANY CONFLICT OF INTEREST Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 67 ANNEX I: EVALUATION STATEMENT OF WORK SECTION C: DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK USAID/Tanzania – Mid-Term Evaluation of Feed the Future (FtF) Tanzania C.1 BACKGROUND Feed the Future (FTF) is a Presidential Initiative that aims to address the root causes of global hunger by sustainably increasing agricultural productivity, supporting nutrition interventions, facilitating access to markets, and increasing incomes for the rural poor to meet their food and nutritional requirements. Tanzania has been designated as a priority country for FTF. The initiative supports growth of the agricultural sector and promotes good nutrition to attain its key goal, “to sustainably reduce global hunger and poverty by tackling their root causes and employing proven strategies for achieving large scale and lasting impact.” The prevalence of poverty and proportion of stunted children are the key indicators chosen to measure success towards reaching the FTF goal. FTF activities for Tanzania began in mid-2011 and are aligned with the Government of Tanzania’s strategies and development plans. FTF works with Tanzanian partners to bring about positive changes in the economic, food security and nutritional status of Tanzanian households. The FTF Multiyear Strategy 2011-2015 outlines the portfolio of FTF Tanzania activities. Rice has been selected as the primary value chain for investment. It is the second most important food in terms of consumption in the Tanzanian diet, and Tanzanian paddy farmers have the potential to supply growing regional markets. Maize was selected as a secondary value chain. It is an important staple food in most Tanzanians’ diets, and is grown by nearly two thirds of Tanzanian farmers. The maize milling process offers opportunities to enhance capacity of the medium private-sector milling enterprises to fortify food to improve nutrition. Horticulture is the other secondary value chain. It offers the opportunity for increased incomes and increased variety of food available on the domestic market. Because we know that rising incomes alone are not sufficient to significantly improve nutrition of women and children, targeted nutrition activities are underway in all areas where FTF works. This integration of the nutrition, direct agricultural value chain work and demand-stimulated activities of marketing and processing is one of the key hallmarks of the FTF work in Tanzania and drives the decision to have the evaluation team look at a trio of activities simultaneously. The development hypothesis (theory of change) of FTF in Tanzania is that by dramatically increasing agricultural productivity, improving access to markets, and providing support for the development of agribusiness, rural incomes in the FTF zone of influence (ZOI) will increase. This increase in productivity and income, combined with empowerment of women and youth, and a massive, integrated social behavior change and communication intervention will influence how the increased income of farm families can be used to improve the nutrition status of women and children. Therefore integrating agriculture and nutrition interventions will have a greater impact on improving nutritional status than if they were implemented separately. In spite of a relatively robust 6-7% growth of the Tanzanian economy in recent years, the agricultural growth rate in 2010 was only 4%, and the number of people living below the poverty line has actually increased due to population growth. Overall, human development indicators, though improving, remain low. While nationally 34% of the population is below the income poverty line, in some regions as much as half the population is unable to meet basic needs. The agricultural sector is the main source of livelihoods in rural areas as it employs approximately 77% of the labor force, with women providing 80% of agricultural labor. Given that agriculture employs the majority of Tanzanians, dramatic growth in the Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 68 sector is needed to achieve the growth targets set by the GOT and many international donors. The three commodity value chains of maize, rice, and horticulture are the focus for increasing productivity given their importance to the agricultural economy in Tanzania. Maize is the most important staple grain and is produced throughout the country involving 85% of all farmers. Rice is produced predominantly by small farmers with demand for locally produced rice projected to triple over the next decade. Rice is also being exported to neighboring countries and as Tanzanian rice becomes more competitive it is expected that rice exports will increase. The horticulture sector has grown significantly in recent years with fast growing demand for horticultural products in domestic, regional, and international markets. All three commodities form a large part of the Tanzanian diet and are key to food security in the country. The growing market opportunities offered by these value chains present small farmers, many of whom are women, with increasing income earning opportunities. While the further development of these value chains presents excellent opportunities for improving food security and income growth for the small farmer, many constraints are inhibiting value chain development. Important among these constraints is that the full potential of women and young farmers continues to be hindered due to issues related to land ownership and other assets, access to markets, access to training and technical information, and in the case of women, domestic responsibilities and managing the household. Key technical constraints are numerous, and include low productivity across all three value chains, poor infrastructure, including on farm storage, and a weak network of rural roads, resulting in high cost of agricultural inputs and inability of farmers and processors to access markets. FTF emphasizes the importance of focusing program efforts within specific geographic areas to permit improved impact monitoring and contribution of results to program funding. FTF/Tanzania’s ZOI includes five mainland regions (Dodoma, Manyara, Morogoro, Mbeya and Iringa) and all five regions of Zanzibar. Morogoro and Zanzibar have been the focus of irrigated rice production, with expansion into Mbeya region planned. Maize will be supported in selected districts of all five ZOI mainland regions. Horticulture value chains also have a broad geographical reach. Nutrition programs are (or will be) active in all areas of the ZOI. This evaluation will focus on those districts where the activities have been operating the most intensively. The nine in-depth districts are:  Dodoma Urban, Kondoa and Kongwa in Dodoma Region  Babati and Kiteto in Manyara Region  Kilombero, Morogoro Urban, Mvomero and Ulanga in Morogoro Region. Empowerment of women and youth is a core objective of the USAID program in Tanzania and gender is highlighted across the FTF portfolio. Gender-equitable interventions include strengthening women’s participation and leadership in farmer, processor, and irrigation organizations; and ensuring that market information and productivity enhancing technologies such as improved seed varieties, fertilizer, and production and processing equipment are accessible to both men and women. FTF also ensures that women are engaged in and benefit from initiatives to improve the quantity and quality of horticulture, maize, and rice products, and works with lending institutions to design women and pro￾poor loan instruments and advocate for the use of moveable assets as collateral. This mid-term performance evaluation is a formative evaluation intended to provide an independent examination of the overall progress and achievements of three of the FTF activities in Tanzania. The three activities are the Tanzania Staples Value Chain Project (NAFAKA); Tuboreshe Chakula (TC); and Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 69 Mwanzo Bora Nutrition Project (MBNP). The evaluation will identify achievements, performance issues, and constraints related to activity implementation and effectiveness, as well as identify how synergies between the three activities may contribute to FTF outcomes. The evaluation must also identify results and lessons learned from the three activities and will provide concise, actionable recommendations to determine which activities to scale up, modify, or re-design in order to improve overall activity performance. Evaluation findings and recommendations will be shared and discussed with USAID/Tanzania and the Bureau for Food Security in Washington, implementing partners, and relevant GOT partners. C.2 PURPOSE OF THE EVALUATION USAID/Tanzania has contracts and agreements with a number of implementing partners to execute activities supported under FTF key investment areas. This evaluation will focus on the following specific subset of activities: a) Tanzania Staples Value Chain Project (NAFAKA). This $30 million activity, led by ACDI/VOCA, integrates agricultural, gender, and nutritional development approaches to improve smallholder farmers’ productivity and profitability in maize and rice value chains. NAFAKA will improve the competitiveness and productivity of maize and rice; facilitate improved domestic and regional trade; expand the benefit reach of maize and rice production with particular attention to women and youth; and enhance rural household nutrition by promoting value chain development and improved food consumption practices. b) Tuboreshe Chakula (food fortification and post-harvest processing). This $22 million activity, led by Abt Associates, is strengthening the capabilities of agro-processors of rice, maize and selected horticultural products to build sustainable enterprises dealing with food processing and fortification and expand and diversify the production and marketing of nutritious processed foods. Key activities include business, enterprise, and market development; technology transfer; support to food processing industries; and demand creation for fortified products. c) Mwanzo Bora Nutrition Project. Mwanzo Bora, led by Africare, is a $30 million activity promoting household nutrition improvement, especially for women and children, with a focus on reducing stunting and anemia. The project will strengthen the capacity of government and indigenous NGOs to deliver quality nutrition education and communication, and strengthen the delivery of community based nutrition services and social behavior changing education resulting in a model that can be rapidly scaled-up to reduce child stunting and maternal anemia.  What progress has each activity made in achieving its respective objectives as outlined in its performance management plan? Are activity objectives and targets realistic and achievable before the respective activity end dates? What results/outcomes have been achieved to date?  How effective are the implementation approaches used by each implementing partner in achieving intended objectives and results and reaching scale?  To what extent have women and youth been integrated into the design and implementation of the activities? Which approaches have proven most effective in reaching women and youth? How have women and youth benefitted from the activities? Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 70  In what specific ways have the activities integrated nutrition and agriculture interventions in their work? Have these integration efforts delivered intended results? What are these results and how have they been achieved?  To what extent have the activities reached scale in terms of: outreach to direct and indirect beneficiaries; adoption of new technologies and practices; and adoption of innovative business models?  To what extent are activities promoting and measuring diffusion of improved technologies or sharing knowledge of improved practices beyond direct beneficiaries? If indirect beneficiaries are being reached, what are the primary vehicles through which this is taking place, how are indirect beneficiaries being measured, and how are benefits calculated?  To what extent have the activities worked with local partners? What are the results in terms of strengthening local institutional capacity, ownership, and long-term sustainability?  How are the activities perceived and valued by stakeholders, including direct beneficiaries, indirect beneficiaries, and GOT counterparts?  What unexpected results (negative consequences or positive results) of the activities have been realized? Which of these should be considered in implementation or follow-on activities, or documented for further dissemination? The analysis and findings regarding each of the evaluation questions must be followed by specific, tailored, realistic and actionable recommendations aimed at improving outcomes. C.2.2 Evaluation Design and Methodology The mid-term evaluation is expected to apply both quantitative and qualitative methods for data collection and analysis. Site visits over approximately four weeks in the field to at least five of the “in￾depth districts” identified above plus additional sites selected in consultation with the USAID/Tanzania FTF team will provide qualitative data for analysis through methods such as in-depth and key informant interviews, focus groups, and direct observation. The evaluation team is expected to meet with farmers, local government officials, local NGOs, implementing partners, subcontractors, and other beneficiaries, partners and stakeholders in order to acquire the data needed to respond to the evaluation questions. In-depth key informant interviews, conducted face to face and based on a semi-structured questionnaire, will provide much of the necessary qualitative data. For producer groups, enterprises, and community groups which have implemented or received nutrition behavior change activities, focus group discussions are recommended as an efficient way to collect information. Primary collection of quantitative data and large-scale structured surveys are beyond the scope of this evaluation. It is expected that the evaluation team will use data provided by the implementing partners in their quarterly and annual reports, performance reporting, and special purpose publications for most of the quantitative data required. All three partners have dedicated teams of M&E personnel and mature IT systems for ongoing data collection and analysis. Program M&E output and outcome data is supplemented by implementing-partner conducted annual surveys of beneficiaries which will be available to the evaluation team. Population-based survey data on progress towards achieving FTF’s high-level objectives of poverty alleviation and improved nutrition are being collected by national statistics agencies (especially via the Demographic and Health Survey and National Panel Survey, which are scheduled to commence sampling in late 2014), and will not be available at the time of this evaluation. Improving agricultural statistics is a Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 71 major initiative of FTF Tanzania, in partnership with the local agencies and the United States Department of Agriculture, but again, improved national data will not be available by the time of this evaluation. Data collection methods must be selected in order to provide the highest quality and rigor in answering the performance evaluation questions. Methodological decisions will be determined by the information that is needed and by the cost of collecting the data. C.2.3 Data Sources The USAID/Tanzania Economic Growth Office will provide documents for the desk review, as well as contact information for prospective interviewees. The evaluation team will be responsible for identifying and reviewing additional materials relevant to the evaluation, as well as additional contacts. Illustrative data sources include but are not limited to: a. FTF/Tanzania Multi-Year Strategy (MYS) b. Program descriptions and statements of work c. Activity performance management plans d. FTF performance management plan e. Activity quarterly and annual reports f. Activity data quality assessment reports g. FTF baseline data from secondary sources h. Activity baseline survey reports and beneficiary surveys i. Comprehensive Food Security and Vulnerability Assessment (CFSVA) j. Gender Assessment for the USAID/Tanzania Country Development Cooperation Strategies (CDCS) k. Tanzania Demographic and Health Survey 2010 l. Tanzania National Panel Survey 2010 m. Tanzania Household Budget Survey 2011/12 C.3 EVALUATION DELIVERABLES C.3.1 Evaluation deliverables include: a. Evaluation Team Planning Meeting (s) – essential in organizing the team’s efforts. During the meeting(s), the team must review and discuss the SOW in its entirety , clarify team members’ role and responsibilities, discuss the process for devising a work plan, develop data collection methods, review and clarify any logistical and administrative procedures for the assignment and instruments, and prepare for the in-brief with USAID/Tanzania b. Work Plan - Detailed draft work plan including task timeline, methodology outlining approach to be used in answering each evaluation question, team responsibilities, data analysis plan, and report writing tasks and timeline; c. In-Brief Meetings - In-brief with USAID/Tanzania and implementing partners upon international team members’ arrival in Dar es Salaam; Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 72 d. Data Collection Instruments–Development and submission of data collection instruments to USAID/Dar es Salaam during the design phase; e. Regular Updates - The Evaluation Team Leader (or his/her delegate) will brief the FTF team on progress with the evaluation on at least a weekly basis, in person or by electronic communication. Any delays or complications must be quickly communicated to USAID/Tanzania as early as possible to allow quick resolution and to minimize any disruptions to the evaluation process. f. Debriefing with USAID/Tanzania and Implementing Partners – The evaluation team will present the major findings from the evaluation to USAID/Tanzania and partners through a PowerPoint (or similar) presentation prior to the team’s departure from the country. The debriefing will cover initial findings, conclusions and preliminary recommendations to USAID/Tanzania; g. Original data and data sets -- Copies of secondary quantitative data sets, transcripts of interviews and focus groups, and notes from direct observations; h. Draft Evaluation Report - A draft report on the findings and recommendations must be submitted to USAID/Tanzania within two weeks after departure of international team members from Dar es Salaam. The written report must clearly describe findings, conclusions, and recommendations. USAID will provide comments on the draft report within ten working days of submission; i. Final Report - The Team will submit a final report that incorporates the Mission’s comments and suggestions no later than seven days after USAID/Tanzania provides written comments on the team’s draft report. The final evaluation report must meet the following criteria: a. The report must represent a thoughtful and well organized effort to objectively respond to the evaluation questions. b. The report must address all evaluation questions included in the SOW. c. Evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides must be included in an Annex in the final report. d. Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, etc.). e. Evaluation findings must be specific, concise and supported by strong quantitative or qualitative evidence. f. Recommendations must be action-oriented, practical, specific, and evidence-based. C.3.2 Reporting Requirements The contractor must submit the reports/deliverables to the Contracting Officer’s Representative (COR). From a technical management perspective and in-country scheduling, the contractor will be responsible for coordination with the COR. All reports and deliverables must be in English unless otherwise specified by the CO. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 73 The following content must be included in the final report: Executive Summary - concisely state the evaluation purpose, methodology, key evaluation questions, key findings and recommendations; 1. Introduction – Evaluation context, including a summary of any relevant history of FTF program, demography, socio-economic status etc.; 2. The FTF Program description - brief overview of the FTF program including the development hypothesis, key intervention areas and implementation arrangement/approach (may rely heavily on existing documents); 3. Purpose of the Evaluation - purpose, audience, and synopsis of task; 4. Evaluation design and Methodology - describe evaluation design and methods, including sampling procedure; 5. Findings/Conclusions - describe and analyze findings for each project and for FTF program using graphs, figures and tables, as applicable supported with concise narratives; 6. Lessons Learned - provide a brief of key technical and/or administrative lessons on what has worked, not worked, and why for immediate corrective measures and future project or relevant program designs; 7. Recommendations – prioritized for each key question; must be separate from conclusions and be supported by clearly defined set of findings and conclusions. Include recommendations for future project implementation or relevant program designs and synergies with other FTF Tanzania projects and other donor interventions as appropriate; Annexes – to include statement of work, list of documents reviewed, tools used, interview lists, meetings, and data tables. Annexes must be pertinent and readable. The format of the final evaluation report should strike a balance between depth and length. The report will be submitted electronically. The final report will be edited/formatted by the contractor and provided to USAID/Tanzania 7 working days after the Mission has reviewed the content and approved the final revised version of the report. The final evaluation report must be 508 compliant and comply with the USAID Evaluation Policy as it relates to performance evaluations, and must use the criteria for quality evaluation reports listed in Appendix I of the Evaluation Policy C.4 SCHEDULING AND LOGISTICS The contractor will be responsible for all international and in-country administrative and logistical support, including identifying and fielding appropriate consultants (International and local). The evaluation team must be able to make all logistical arrangements including vehicle rental for travel within and outside Dar es Salaam and should not expect any logistical support from the Mission. The team must also make their own arrangements for venues for team meetings, and equipment support for producing the report. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 74 ANNEX II: EVALUATION METHODS AND EVALUATION DESIGN MATRIX EVALUATION PURPOSE This formative midterm evaluation examined the overall progress and achievements of three of the FTF activities in Tanzania. The three activities are the NAFAKA; TC; and MBNP, which focus on value chain development, post-harvest processing and food fortification, and household nutrition, respectively. This midterm evaluation identified achievements, performance issues, and constraints related to activity implementation and effectiveness and as related to the priority EQs. The evaluation was designed to produce targeted, action-oriented recommendations for USAID/Tanzania to support decision making and next steps related to these and future activities. SI’s approach drew upon utilization-focused methodologies to ensure that the information and guidance generated by the evaluation were useful to USAID/Tanzania and its stakeholders. The evaluation produced results that were intended to help USAID/Tanzania and IPs identify and act on options to improve implementation performance, to enhance collaboration among FTF activities, and to expand the productive involvement of female and young farmers in the maize, rice, and horticulture value chains. EVALUATION QUESTIONS The questions that this evaluation addressed were identified in the evaluation statement of work (SOW). They are listed below. Given the large number of evaluation questions in the evaluation SOW (nine evaluation questions and 11 sub-questions were listed, for a total of 20 questions), it was not possible to undertake in-depth work on all of them. USAID/Tanzania identified evaluation four of the evaluation questions as the priority questions for the evaluation.54 These four questions are related to women and youth, agriculture and nutrition integration, achieving scale, and indirect beneficiaries. These four questions have seven sub-questions. Because of the breadth of each of the four primary evaluation questions (given their sub-questions), the evaluation team further focused on those sub-questions within the primary evaluation questions which best address the priority questions. The team will dedicate approximately 70 percent of its resources on the four priority questions and approximately 30 percent on the five ancillary questions in order to maximize high-priority information and analysis. The EQs included in the evaluation SOW are listed below. 1a. To what extent have women and youth been integrated into the design and implementation of the activities? 1b. Which approaches have proven most effective in reaching women and youth? 1c. How have women and youth benefitted from the activities? 2a. In what specific ways have the activities integrated nutrition and agriculture interventions in their work? 2b. Have these integration efforts delivered intended results? 2c. What are these results and how have they been achieved? 54 In the SOW for this evaluation, these were questions 3, 4, 5, and 6. For the purposes of this evaluation itself, the questions have been renumbered as questions 1, 2, 3, and 4. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 75 3a. To what extent have the activities reached scale in terms of: 3b. Outreach to direct and indirect beneficiaries; 3c. Adoption of new technologies and practices; and 3d. Adoption of innovative business models? 4a. To what extent are activities promoting and measuring diffusion of improved technologies or sharing knowledge of improved practices beyond direct beneficiaries? 4b. If indirect beneficiaries are being reached, what are the primary vehicles through which this is taking place, how are indirect beneficiaries being measured, and how are benefits calculated? 5a. What progress has each activity made in achieving its respective objectives as outlined in its performance management plan? 5b. Are activity objectives and targets realistic and achievable before the respective activity end dates? 5c. What results/outcomes have been achieved to date? 6. How effective are the implementation approaches used by each implementing partner in achieving intended objectives and results and reaching scale? 7a. To what extent have the activities worked with local partners? 7b. What are the results in terms of strengthening local institutional capacity, ownership, and long￾term sustainability? 8. How are the activities perceived and valued by stakeholders, including direct beneficiaries, indirect beneficiaries, and GOT counterparts? 9a. What unexpected results (negative consequences or positive results) of the activities have been realized? 9b. Which of these should be considered in implementation or follow-on activities, or documented for further dissemination? EVALUATION APPROACH There were several features of SI’s approach that were crucial to the success of the evaluation. The approach prioritized the utilization of evaluation results through the use of a mixed-methods design, which included value-chain analysis and attention to nutrition, and is grounded in contextualized, gender and youth-sensitive methodologies. These four approach foci (1. utilization-focused evaluation; 2. value chain analysis; 3. nutrition; and 4. gender and youth) are discussed below and were woven into the activities conducted throughout the evaluation. Data Collection Methods There were several features of SI’s approach that were crucial to the success of the evaluation. The approach prioritized utilization of evaluation results through the use of a qualitative-dominant mixed￾methods design, which included attention to value-chains and nutrition, and was grounded in Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 76 contextualized, gender and youth-sensitive methodologies. These four approach foci were woven into the evaluation activities. Utilization-Focused Evaluation A core tenant of SI’s mission is to focus on evaluation use in order to improve programming and, ultimately, development effectiveness. Aligned with this mission, SI actively engaged USAID, IPs, and other stakeholders in the evaluation process. It began by engaging USAID staff and other stakeholders to gain a full understanding of primary and secondary intended users and identify the range of potential evaluation uses. This allowed SI to refine and implement the evaluation so that it targeted these purposes and created evaluation products that were useful to key stakeholders. After the data was collected, SI held a debriefing for USAID, during which the team described the data collection process and preliminary findings. The team provided the Mission with an annotated outline that allowed the team to ensure that its framework was aligned with USAID’s expectations. Then the team returned to their respective homes where they completed the systematic data analyses. Once the analysis was completed and an initial report drafted, SI sent an electronic copy to USAID for review and comments. Value Chains Approach The team developed of a value chain map for maize and rice, which guided the selection of the stakeholders that were interviewed. Some of the individual interviews were with the directors and staff of NAFAKA, TC, MBNP, local government officials, and few key decision-makers in each of the value chains under FTF. The team also sampled from among value chain beneficiaries who participated in these activities and received training, grants, and other assistance. The team identified what the market opportunities for maize and rice were, and how these demand requirements were being transmitted up the value chain. Further, they assessed how the activity addressed critical points in the value chain through information collected from the interviews. The team paid special attention to improvements made in strengthening the market linkages and the flow of information between different stages: input suppliers to producers, and producers to middlemen (traders) or to food processors, with a particular focus on the maize value chain as it related to both NAFAKA and TC (as well as to MBNP beneficiaries). They also assessed the nature of relationships/integration and interaction between a particular level and other value chain stakeholders. The team examined the different interventions and how those interventions helped the local target sector to integrate successfully into the value chain and secure benefits. The analysis was done with the broader picture of the value chain stakeholders who participated in those value chains in mind. This lead to identification of value chain nodes, such as markets, processors, collection sites, etc. Nutrition The nutrition approach focused on specific ways in which nutrition and agriculture interventions have been integrated in NAFAKA, TC, and MBNP activities. The team reviewed activity plans to identify key issues, and then clarified them by interviewing activity managers who were responsible for implementing the activities and by conducting site observations during field visits to activity sites. This was Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 77 supplemented by interviews with representatives of IP and sub-partner staff, primary beneficiaries and other stakeholders. Through dialogues with representatives of IPs, local community leaders, beneficiary groups, and government officials, the team assessed how the nutrition dimensions of the activities are perceived and valued by stakeholders (including direct and indirect beneficiaries as well as GOT counterparts). The dialogues were in the form of KIIs and FGDs. These were supplemented with complementary site visits to health facilities that provide reproductive and child health services to assess the relationship of TC and MBNP and the use of MBNP and TC products along with the MBNP communication program. Women and Youth Documented evidence suggests that Tanzanian women contribute significantly to all food crop production, processing, and marketing activities. They are key participants the in all staple value chains, constituting 75 percent of the agricultural labor force and are involved in all aspects of the rice and maize value chains (as well as the horticultural value chains) that are the targets of USAID's FTF interventions. In spite of this, there are significant gender related inequalities (e.g., education, social norms, earning potential, etc.) that disfavor women in the agricultural sector. Young male and female farmers in Africa and other parts of the developing world face challenges similar to those faced by women. Youth employed in the agricultural sector often struggle with seasonal underemployment and this tends to lead to rural exodus. The limitations and inequalities suffered by women and youth are limiting both to overall growth and to equitable development, which is an important reason for USAID Tanzania FTF’s approach of combining agriculture development and nutrition interventions with the empowerment of women and youth. The team assessed the extent to which each of the activities addressed gender, age, and related dynamics and inequalities in program design, implementation, and results measurement to promote activity effectiveness and sustainability. They examined how the activities themselves have influenced gender, age, and other social statuses and relationships. They highlighted successes and gaps in the approaches used and identified potential opportunities to improve activity performance going forward. The team disaggregated all data collected by sex and age. Using sex-disaggregated FGDs, where appropriate, ensured that the representation of both women and men and allowed the team to address gender specific issues and gaps. For KIIs, the team considered the selection of interviewees so that women, men, and youth were effectively represented. FGDs were facilitated to create an environment that fostered participation by all. This methodological and analytical focus on gender and age sensitivity generated findings, conclusions, and recommendations that enabled the team to better understand the variable effects of FTF activities on women and young and older farmers. METHODOLOGY The evaluation methodology focused on generating findings that allowed the team to draw evidence￾based conclusions and design well-substantiated recommendations for USAID/Tanzania’s use. The team lead a qualitatively-dominant but still mixed-methods formative evaluation approach involving: 1) a desk review of available primary and secondary documents; 2) USAID consultations; 3) semi-structured KIIs; 4) FGDs; 5) opportunistic interviews; and 6) observation visits to activity implementation sites (e.g., health facilities and demonstration plots) to observe their operations. Following data collection, the Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 78 team analyzed and triangulated the data to determine evidence-based findings from which to draw conclusions and recommendations. The data collection activities and analytical approaches were designed to be value chain-based, gender and youth-sensitive, utilization-focused, and rigorous. Sampling The evaluation was grounded in purposive sampling of stakeholders, and some elements of maximum variation sampling and snowball sampling were used as well. The goal of the sampling was to allow the evaluation work to focus on particular characteristics of a population that are of interest and that can best address the evaluation questions. The SI team selected districts based on the level and type of activities carried out by the IPs. Specific stakeholders were then selected in each district at key points of interest where a number of interventions have been made. Given the team’s limited amount of time and resources for data collection, it was not possible to attain a representative random sample of the population. Therefore, in an effort to learn as much about the three FTF activities in the time available, the team used a maximum variation sampling approach. This is a purposive sampling technique used to capture a wide range of perspectives relating to the area of inquiry. Maximum variation sampling is a search for a variation in perspectives, ranging from those conditions that are viewed to be typical through to those that are more extreme in nature. As such, the basic principle behind maximum variation sampling is to gain greater insights into the question at hand by looking at it from as many angles as possible. This is designed to help researchers identify common themes that are evident across the sample from a range of examples. Interview respondents were selected through a multi-stage process, which started with recommendations from USAID, IPs, and local partners. The team purposively selected a sample who participated in FTF to speak with the team via focus groups and/or interviews. Once they were in the field, the team requested further contacts and informants from this group, using a snowball sampling approach to continue to reach out to relevant respondents. These discussions further explored reasons for high and low performance, and informed recommendations for relevant and effective programming. Site Selection As agreed with USAID/Tanzania, the fieldwork was conducted in three regions (Dodoma, Manyara, and Morogoro). There are nine districts in the ZOIs in the three regions: Dodoma Urban, Kondoa, and Kongwa in Dodoma Region; Babati and Kiteto in Manyara Region, and Kilombero, Morogoro Urban, Mvomero, and Ulanga in Morogoro Region. In consultation with USAID/Tanzania, five of the nine districts in the ZOI were selected for field work: Morogoro Urban, Mvomero, Kongwa, Kiteto, and Dodoma Urban. Within these five districts, the team visited the IPs’ primary partner organizations. To the extent these organizations had offices and personnel in these locations, the team visited partner offices as well. Respondent Selection The IPs produced a list of stakeholders participating in their activities at different locations. Drawing from these lists, the team interviewed participants where they currently work or reside. The team identified points of interest which are nodes where a number of beneficiaries that have been affected by the FTF activities reside. These nodes were key to examining the relationships between agriculture and nutrition interventions. Using these nodes as starting points, the team interviewed both direct and Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 79 indirect beneficiaries around each node. Names and contact information of direct beneficiaries came from USAID, IPs, and local organizations, while indirect beneficiaries were selected using opportunistic and snowball sampling. DATA COLLECTION Desk Review The team reviewed activity-related documents provided by USAID, IPs (ACDI/VOCA, Abt Associates, and Africare), and other relevant stakeholders as well as background documents relevant to the activities being evaluated. These included design documents, quarterly and annual reports, monitoring data, data quality assessments, materials prepared by the implementers, special purpose publications, contextual data and other information from government sources, and program implementers. SI also reviewed IP baseline and activity monitoring data, performance reporting, and special purpose publications. The team used these data sources both to triangulate with field data and as input for responding directly to EQs. The documents reviewed were revisited during the qualitative analysis and reporting so that the findings were presented in the larger context of the FTF program in Tanzania. Please see Annex IV Documents Reviewed for a list of the documents review by the team. These included all documents required under the evaluation SOW. USAID/Tanzania FTF Staff Consultations The team held consultations with a sub-set of USAID staff members prior to departure to Tanzania. They discussed the context, priorities, and potential challenges that the team might face during the evaluation. Key Informant Interviews The team facilitated 179 KIIs with a total of 211 stakeholders (some interviews, particularly those with regional and district government representatives and introductory interviews with sub-partners, included more than one person). Stakeholders included, for NAFAKA, ACDI/VOCA, and sub-partner staff, VBAAs, farmer group leaders, PSPs, and FAs supporting SILCs as well as agro-dealers, government representatives, etc. For MBNP, stakeholders included Africa and sub-partner staff, CHWs, health facility staff, government representatives, retailers, representatives of collaborating organizations, PSG members, other beneficiaries, etc. For TC, stakeholders included Abt Associates staff, maize millers (both participants and non-participants), sunflower oil processers, HFWs, etc. KIIs were implemented in two distinct phases. Phase I solicited information from respondents with high-level knowledge (i.e., program-wide or country-wide) in Dar es Salaam. These interviews covered the three IPs, relevant GOT Ministries and offices concerned with the FTF program, and other key stakeholders. Phase II was implemented in the five districts in the three regions. As noted in the sub-sections on sampling, key informants were chosen through purposive, maximum variation, and snowball methods. Interview candidates were identified through the document review and through suggestions from USAID, IPs, beneficiaries, and lists provided by IPs. Semi-structured KII guides were designed to be tailored for use with each primary stakeholder group, and all included a core set of common questions. Each KII was led by one of the four team members, depending on the language ability of the respondent. Two team members participated in each interview, which provided a cross disciplinary understanding of the information provided by the respondent and Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 80 provided guidance, note-taking, and support throughout the interview. The KII guides are provided in Annex III. Focus Group Discussions The team facilitated 30 FGDs with a total of 332 activity beneficiaries, including, for NAFAKA—farmer groups, farmer associations, farmers supported by NAFAKA who did not belong to a farmer group, and SILCs, and for MBNP—members of peer support groups. When appropriate to facilitate the flow of discussion, many FGDs were held with either all male or all female participants. In a few cases, FGDs were disaggregated by ethnic group. The FGD guide instrument is included in Annex III. Opportunistic Interviews To collect information from people not participating in the FTF activities, the team conducted short opportunistic interviews with individuals selected using opportunistic and convenience sampling. These included women who were pregnant or with small children, farmers, maize mill operators, and shop keepers. Observation To observe program activities, the team visited selected TC, NAFAKA, and MBNP sites. They carried out structured and unstructured site observations of program activities at these sites. DATA ANALYSIS Qualitative Data Qualitative data was the core source of learning in this evaluation design. When done carefully and systematically, it serves many purposes within the context of a mixed-methods evaluation design. Qualitative data can provide robust, objective evidence to answer the EQs. It is also a useful source of information about people’s experiences, perceptions, opinions, feelings, motivations, and knowledge; the context within which the activity is operating; and the factors (causal mechanisms) explaining observed results (or non-results), all of which add depth and nuance in answering the EQ. Qualitative data can also deepen understanding of quantitative monitoring data, particularly the how and why, and bring it to life with illustrative anecdotes. Finally, in-depth conversations through KIIs and FGDs are not constrained by pre-determined response categories and can thus explore unanticipated subjects that surface during the interview/discussion, adding further nuance and insight in answering the EQs. Qualitative data analysis involves the identification, examination, and interpretation of patterns and themes in textual data and determines how these patterns and themes help answer the EQs. We used a common approach to analyze qualitative data. This approach included the five steps described below.  Step 1: Process and Record Data Immediately: Immediately on completing an interview or group discussion, the team summarized interview/discussion notes, while also noting things that stuck out during the interview/discussion, identifying highlights, and recording other observations.  Step 2: Begin Analyzing as Data is Being Collected: As data was collected, the team began reviewing the data and mentally processing it to identify themes or patterns. To facilitate this process, the team held routine team ‘check-ins’ during the fieldwork to discuss findings and ask questions. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 81 During these check-ins, the gender specialist inquired about issues and ensured that they were being adequately addressed during the interviews/discussions.  Step 3: Data Reduction: Qualitative research generally produces a wealth of data, but not all of it is useful. After data collection was complete, the team undertook a data reduction process to identify and focus in on what was most meaningful. First, the team worked together periodically during field work to assess data in real time to assure that the needed information was being captured through the interviews. At the end of the field work, the team worked together to do an initial analysis of all collected data and prepare its preliminary findings for the out-briefing with the Mission. The data reduction work continued and was completed after the team members returned home. In this step, the team combed through the raw data, coded it by hand, and filtered it to determine what was significant. The team then transformed the data into a simplified format that could be understood in the context of the research questions. When trying to discern what the most meaningful data was, the team referred back to the research questions and used them as the framework in addition to relying on team members’ own experience and familiarity with the activities.  Step 4: Identifying Meaningful Patterns and Themes: For qualitative data to be analyzed, it must first be grouped into the meaningful patterns and/or themes. The team identified meaningful patterns early in the evaluation and then built simple coding to be used to decipher the patterns that appeared in the interviews. The team undertook both content analysis and thematic analysis. To carry out the qualitative analysis, the team used hand coding to sort the qualitative data by theme and attribute of the respondent and conduct queries of themes and attributes to disaggregate the data. The main themes that emerge from the qualitative data were then identified systematically and supported with key quotations and examples from the individual discussions.  Step 5: Conclusion Drawing and Verification: In this step, the team conducted a thorough review of its analysis and findings. This step involved all team members in an iterative process of review, reflection, and discussion to interpret what the findings mean; determine how the findings help answer the EQs; and draw conclusions solely from the findings. We then involved all team members in developing recommendations and lessons learned based solely on the conclusions. Quality Assurance SI employs a four-stage quality assurance (QA) process for all of its work products. Each stage of an assessment was reviewed with direct feedback provided to our TL. Stage I: Work Planning: The SI STA reviewed the feasibility and rigor of the proposed assessment evaluation design and work plan as well as the data collection tools and protocols. Consideration was given to gaps between males and females. Stage II: Field Work: The SI PM ensured the staff was properly on board with SI security protocols, mobilization procedures, and QA procedures. The PM worked with the TL to train staff on field work procedures specific for the activity and ensure QA measures were followed throughout the duration of work. Intermittent oversight was provided by the STA to ensure that the evaluation methodologies were followed. (In the case of this evaluation, the STA replaced the TL early in the field work, and SI’s Senior VP for Evaluations took on the former STA’s role.) Additionally, data QA processes were performed by the team in the field, in real-time, during data collection, during data entry, and in the Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 82 delivery of datasets. This included: 1) monitoring the sampling process and location of completed KIIs, FGDs, and site observations, and 2) reviewing all written notes from KIIs, FGDs, and site observation protocols at the end of each day to ensure completeness and accuracy Stage III: Drafting the Evaluation Report: The report was reviewed by the STA and PM to ensure proper structure and logical linkages among the findings, analysis, conclusions as well as the presentation of data and actionable recommendations. Stage IV: Final Report: The STA and PM followed a 45-point quality check of the executive summary, program, and methodology description; adequacy of findings, analysis, conclusions, and final recommendations; compliance with USAID evaluation policies; and overall report presentation (e.g., charts, graphs, and annexes). The STA for this evaluation maintained overall technical oversight and QA. Limitations There were several limitations that may have affected this evaluation. First, all three activities operate in large geographical areas. The evaluation did not cover them all, so some results based on beneficiary interviews and FGDs, particularly for NAFAKA, cannot be fully generalized across the full spectrum of activity implementation sites. The evaluation SOW designated that the evaluation field work be done in three regions: Dodoma, Morogoro, and Manyara (so, for instance, activity work in Zanzibar was not included). It was agreed with USAID/Tanzania that the field work would be conducted in districts where there was the greatest overlap among TC, MBNP, and NAFAKA implementation. While this resulted in maize growing areas being slightly more heavily represented than rice growing areas, this is not expected to substantive affect results because of the evaluation methods used. Second, as noted above, during the first week of KIIs and FGDs with NAFAKA and MBNP beneficiaries, respondents were largely chosen by IPs. IPs were supportive in applying the team’s sampling strategy thereafter. While every effort was made to maximize diversity in the sampling, some selection bias may have been introduced as a result of the respondents made available for interviews. Third, particularly during the early days of field work, NAFAKA, MBNP and/or their sub-partner staff, closely accompanied the evaluation. IP and/or sub-partner staff were often present when the team arrived (usually accompanied by an activity vehicle) at a KII or FDG site, and staff introduced the respondents to the team. At one FGD site, there were seven IP and sub-partner staff. In one case, sub￾partner staff were leading cheers with FDG participants when evaluators arrived. In another, when the team arrived for a KII, a high-level IP staff person was meeting with the person who was to be interviewed. Over time, the team was able to diminish the level of activity staff’s presence and involvement, but it was rarely completely absent. After the first few days, staff agreed not to be present during FGDs and most KIIs, though they were present during joint interviews with regional and district officials. Overall, this likely introduced some response bias, particularly in the early stages of the field work. Selection and response bias were not expected to be significant issues with the TC interviews. The original TL had to leave the evaluation after six days of field interviews. SI’s STA for the evaluation was present in the field for all but two of those days of interviews, so her taking on the team leader role is not expected to have had a negative effect on the data collected or evaluation results. Finally, two of the four team members do not speak Kiswahili. This was compensated for by the language skills of the Tanzanian team members and by interpreters used during the field work. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 83 Evaluation Design Matrix Evaluation Question Evaluation Sub-Question Data Source (Organization/Person) Data Collection Methods Priority Evaluation Questions Q1a. To what extent have women and youth been integrated into the design and implementation of the activities?  National Nutrition Strategy  Gender assessment and analysis reports for each program  Implementation plans and descriptions for integration  Activity M&E plans  M&E data from each contractor on data collection on certain indicators (M&E database information)  Male, female, and youth beneficiaries  Women’s associations  Farmer associations  Literature review  KIIs (leaders of associations)  FGDs (women associations, farmer associations, etc.) Q1b. Which approaches have proven most effective in reaching women and youth?  Activity reports  IPs  Male, female, and youth beneficiaries  Women’s associations  Farmer associations  Literature review  KIIs  FGDs (association leadership and beneficiaries (through recommendations from associations)  Field visits/observations Q1c. How have women and youth benefitted from the activities?  Activity M&E data  Gender focal points within IPs  USAID gender specialist (if possible)  Male, female, and youth beneficiaries  Women’s associations  Farmer associations  Literature review  FGDs  KIIs  Field visits/observations (kitchen gardens and demo fields) Q2a. In what specific ways have the activities integrated  National Nutrition Strategy  Activity reports  IP staff  Male, female, and youth beneficiaries  Literature review  FGDs Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 84 Evaluation Question Evaluation Sub-Question Data Source (Organization/Person) Data Collection Methods nutrition and agriculture interventions in their work? Q2b. Have these integration efforts delivered intended results?  IP Annual reports  IPs  Government partners  Literature review  KIIs Q2c. What are these results and how have they been achieved?  IP Annual reports  IP staff  Government partners  Literature review  KIIs Q3. To what extent have the activities reached scale in terms of: outreach to direct and indirect beneficiaries; adoption of new technologies and practices; and adoption of innovative business models?  IP staff  IP Annual reports  Other IP reports  Enterprise development co-investment grant holders  Local leaders who are close to these programs  Associations  KIIs  Literature review  FGDs  Site observations Q4a. To what extent are activities promoting and measuring diffusion of improved technologies or sharing knowledge of improved practices beyond direct beneficiaries?  Annual reports  M&E Data  Activity publications  IP staff  GOT staff  Association Leaders  Literature review  KIIs Q4b. If indirect beneficiaries are being reached, what are the primary vehicles through which this is taking place, how are indirect beneficiaries being measured, and how are benefits calculated?  Annual and quarterly reports  District government reports  LFs and others  Service providers (e.g., extension officers)  CHWs  VBAA advisors  Clinic directors  Literature Review  KIIs  Observations from field visits Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 85 Evaluation Question Evaluation Sub-Question Data Source (Organization/Person) Data Collection Methods  Ancillary Evaluation Questions Q5a. What progress has each activity made in achieving its respective objectives as outlined in its performance management plan (PMP)?  IP staff  Male, female, and youth beneficiaries  Activity M&E data  IP reports  Regional data (from local Government national bureau of statistics)  District-level data (from IPs)  Literature Review  KIIs  FGDs Q5b. Are activity objectives and targets realistic and achievable before the respective activity end dates?  Activity annual reports  IP staff  Government staff on the activities  Key partner NGOs  Leaders of associations  Literature Review  KIIs Q5c.What results/outcomes have been achieved to date?  Activity annual reports  M&E data  Leaders of associations  Literature Review  KIIs Q6. How effective are the implementation approaches used by each implementing partner in achieving intended objectives and results and reaching scale?  Activity annual reports  M&E data  IP staff  Government staff  Private sector partners  Local NGOs  Literature Review  KIIs  Site observations  FGDs Q7a. To what extent have the activities worked with local partners?  Directors of the local partners  Annual and quarterly reports on trainings and workshops  MOUs  Literature Review  KIIs Q7b. What are the results in terms of strengthening local institutional capacity,  Workshop/training reports, attendance  Annual reports of the local institution, board of directors  Literature Review  KIIs Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 86 Evaluation Question Evaluation Sub-Question Data Source (Organization/Person) Data Collection Methods ownership, and long-term sustainability?  M&E reports Q8. How are the activities perceived and valued by stakeholders, including direct beneficiaries, indirect beneficiaries, and GOT counterparts?  IPs  Local partners  Government staff  Beneficiaries  DOs  FGDs  KIIs Q9a. What unexpected results (negative consequences or positive results) of the activities have been realized?  IPs  Local partners  Government staff  Beneficiaries  DOs  KIIs Q9b. Which of these should be considered in implementation or follow-on activities, or documented for further dissemination?  IPs  Local partners  Government staff  Beneficiaries  DOs  KIIs Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 87 ANNEX III: DATA COLLECTION INSTRUMENTS Annex IIIA: Key Informant Interview: Implementing Partners and Sub-Partners Annex IIIB: Key Informant Interview: Participants and Beneficiaries Annex IIIC: Focus Group Discussion Guide: Beneficiaries Notes on the instruments  Given the wide variety of stakeholders, it determined that it was not practical to design a separate instrument for each one. Therefore, the team designed broader instruments that provided internal scope for questions needed for each stakeholder. Not all questions in the instruments were asked of each stakeholder.  The key informant interview and focus group discussion guides presented here have been modified to fit into the evaluation report format: spaces for interviewers to write responses have been removed.  The numbering in the instruments is based on the numbers of the evaluation questions assigned by USAID. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 88 Key Informant Interview: Implementing Partners & Sub-Partners Guide for FTF Activities in Dodoma, Manyara, and Morogoro Interviewers provide introduction that covers the following points:  Brief introduction of the evaluation team members  Brief background on Partnering to Innovation (P41) and its different components  Purpose of the evaluation and of the interview  Main topics of the of the interview  Confidentiality of responses  Request for and receipt of permission to interview Project Name: NAFAKA (NF), Tuboreshe Chakula (TC), Mwanzo Bora Nutrition Program (MBNP) Sub-partner: NAFAKA: CRS, IFDC, FIPS, MVIWATA, other MBNP: CONSENUTH, TAWG, Sharing Worlds, other Region: Dodoma; Manyara; Morogoro District: Ward: Village: Topic or Value Chain: Maize (MZ), Rice (RI), Nutrition (NU) Name(s) and Position(s): Organization: Male (#)__ or Female (#)___ Date of interview:_______________________ Time of Interview__________ Name of Interviewer:________________ Name of note taker:__________ Confidentiality explained: Yes___ or No _____ Respondent agreed to be interviewed: Yes___ or No _____ Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 89 Overarching A. Could you please describe the activity(ies) of NAFAKA/Mwanzo Bora/Tuboreshe Chakula that you have participated in/led? a. 3. What practice/activities that were introduced by the project did not exist before? b. 3. To what extent have they been adopted i. by the people you work with in this project? ii. By the wider community? B. How long have you been working with NAFAKA/Mwanzo Bora/Tuboreshe Chakula? C. For Sub-Partners a. How long have you worked for this organization? b. How is work on NAFAKA/Mwanzo Bora/Tuboreshe Chakula different from other your organization does? Q5c.What results/outcomes have been achieved to date? 5c I. What have you achieved so far? 5c II. Describe the outcomes on the beneficiaries you work with. Q2a: In what specific ways have the activities integrated nutrition and agriculture interventions in their work? 2a I. Have activities been implemented in the way they were outlined in the project plan? Yes/No a. If not, what has been changed? 5a Q5a. What progress has each activity made in achieving its respective objectives as outlined in its performance management plan (PMP)? 5a I. Which objectives have been or seem easiest to achieve? 5a II. Which objectives have been or may be the most challenging to achieve? 5a III. What were the main obstacles you faced in achieving the objectives? 7a Q7a. To what extent have the activities worked with sub-partners? 7a I. Implementing Partners (NAFAKA, MC and TC) a. Who were your sub-partners in each district? b. What are the central activities of each sub-partner? c. How are the activities among the sub-partners coordinated (particularly among those working in the same communities)? 7a II. Sub-partners a. What activities did you carry out on this project? b. Were you adequately informed or consulted on project activities? c. What do you think about the collaboration process with the prime? // with other subcontractors? 7a Q7b. What are the results in terms of strengthening local institutional capacity, ownership, and long-term sustainability? 7a I. Can you give evidence on how the capacity of how a LOCAL SUB-PARTNER has improved? 7a II. How would you describe the financial health of the LOCAL SUB-PARTNERs since the intervention started? 7a III. Were the expected outcomes of your collaboration achieved? Yes/No; If not, explain Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 90 7a LOCAL SUB-PARTNER: i. Are there any areas of management that you would like to improve (e.g., bookkeeping system to plan for and account for sources and uses of funds)? Q6. How effective are the implementation approaches used by each implementing partner in achieving intended objectives and results and reaching scale? 6 I. What implementation approaches worked best in meeting objectives? 6 II. If yes, what elements or incentives made the approach successful? Training, Grants, Advice? 6 III. What is the best way to incentivize target groups to engage? PRIORITY Q2a. In what specific ways have the activities integrated nutrition and agriculture interventions in their work? 2a I. Please describe what good nutrition means to you? Respondent’s level of understanding of nutrition is sophisticated///moderate///poor 2a II. For Mwanzo Bora a. What specific agricultural activities do you do? b. When you talk about farming or gardens, do you talk about how they are related to family nutrition? Yes/No c. If yes, how do you integrate the work on agriculture and nutrition? iii. Do you teach people about any of the following: a. Animal keeping (chicken, rabbits, etc.) Yes/No b. Growing Quality Protein Maize Yes/No c. Growing orange fleshed sweet potato d. Sensitization of husbands to support wives (accompanying to the clinic, etc.) Yes/No e. Empowering women to own the income resulting from production Yes/No 2a III. For NAFAKA a.What specific activities do you do that are related to nutrition? b.When you talk about agriculture, do you talk about how it is related to nutrition? Yes/No c. If yes, how do you integrate the work on agriculture and nutrition? 2b Q2b. Have these integration efforts delivered intended results? 2b I. What were the intended outcome results of integrating agricultural and nutrition related activities? 2b II. Have they been attained? a. If not, what is the variation? 2c Q2c. What are these results and how have they been achieved? 2c I. For Mwanzo Bora and NAFAKA: If nutrition and agriculture are integrated: a. How has bringing together agriculture and nutrition been beneficial? b. What challenges have you seen in trying to integrate agriculture and nutrition? c. What do you think are the main achievements of integrating nutrition and agriculture? d. What are the main challenges of trying to integrate nutrition and agriculture? e. What do you think contributed most to the success/achievements? f. What strategies do you think might help sub-partners to further integrate them? g. Do you think the outcomes could have been different if the agriculture and nutrition work was not integrated? Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 91 PRIORITY Gender 1a Q1a. To what extent have women and youth been integrated into the design and implementation of the activities? 1a I. Are women, men and youth encouraged to attend the activities and be included in decision-making processes? (For HFW or PSG, ask about MEN!) 1a II. Are women and youth involved in the project activities? – Yes/No a. If yes, indicate their levels of involvement (attendance, leadership, role models, promotion to neighbors, working on demo plots) b. Is there any difference between men/women regarding the activities or positions? 1a III. What strategies (if any) were adopted to cater for the needs of women and youth (access to land, water, information, credit, improved/laborsaving technologies). Were these strategies successful? Yes/No Why? 1a IV. How does the improvement of women’s status (earning income, speaking up in a group, receiving education) affect her husband and home life? 9 V. How do men respond to the empowerment of women? (Give positive and negative examples) 1a VI. How would you rate the involvement of men, women and youth in this program? Q1b. Which approaches have proven most effective in reaching women and youth? 1b I. How do you identify the needs of female farmers, processors, or entrepreneurs? 1b II. Do the approaches have an impact on the self-esteem of women and/or youth? 1b III. Which of these approaches were most effective? 9 IV. Have any of these approaches had negative results? 1b V. What other strategies could be employed to better integrate women and youth into these types of activities? Q1c. How have women and youth benefitted from the activities? 1c I. How has this project benefitted men, women, and youth? Have these groups benefitted in different ways? 1c II. Women already play a big role in small-scale processing. To what extent are women food processors benefiting from the FTF activities (particularly TC)? 1c III. Any thoughts regarding the similarities and/or differences in benefits between these three groups of beneficiaries? Or, how can we explain the similarities and/or differences in benefits? Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 92 PRIORITY 3 Q3. To what extent have the activities reached scale in terms of: outreach to direct and indirect beneficiaries; adoption of new technologies and practices; and adoption of innovative business models? 3 I. Do you have targets of numbers of people you are supposed to work with? Yes / No If yes, a. How many? b. So far how many have you worked with? 3 II. What are the challenges that make it hard to work with more people? 3 III. Do you have targets of numbers of demo plots/gardens/groups/other you are supposed to set up? Yes / No If yes, a. How many? b. So far how many have you worked with? 3 IV. What are the challenges that make it hard to set up more? 3 4b V. Do some of the benefits of your work also get to people who are not directly involved in the project? Yes / No If yes, a. How does that happen? VI. Among the beneficiaries, who were intended beneficiaries and who were the indirect beneficiaries? PRIORITY 4a Q4a. To what extent are activities promoting and measuring diffusion of improved technologies or sharing knowledge of improved practices beyond direct beneficiaries? 4a I. What steps are you undertaking to promote the technologies/capacity/information to people other than the people who directly participate in project activities? 4b Q4b. If indirect beneficiaries are being reached, what are the primary vehicles through which this is taking place, how are indirect beneficiaries being measured, and how are benefits calculated? 4b I. What are the main ways that people who are not the direct participants/beneficiaries receive benefits? (e.g., information from current beneficiaries, radio, drama, etc.) 4b II. What is the best way to encourage new beneficiaries to join? Word of mouth? Other? 4b III. What attracts new entrants to a project activity? Social bonding or other incentives? 4b IV. How do you know if you are reaching people who are directly participating in the project? V. How are you measuring/estimating the number of indirect beneficiaries/level of diffusion? VI. How do you think we could try to measure them better? 5b Q5b. Are activity objectives and targets realistic and achievable before the respective activity end dates? 5b I. Do you have a specific number of people you are supposed to reach or work with? Yes / No 5b II. Are you on track to be able to reach/work with/that many? Yes / No 5b III. Is there a risk that you might not make the target? 9 IV. How is the local government involved in the work you do for this project? Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 93 Value of the Project Q8. How are the activities perceived and valued by stakeholders, including direct beneficiaries, indirect beneficiaries, and GOT counterparts? 8 For participants I. What are the 2 most useful things you’ve learned through this project? 8 II. What have been the two most valuable results/benefits of your participation in this project? Unexpected Results Q9a. What unexpected results (negative consequences or positive results) of the activities have been realized? 9 I. Have there been positive outcomes that were not expected? Yes/No? What were they? 9 II. Have there been negative outcomes that were not expected? Yes/No? What were they? Sustainability 7b. What are the results in terms of strengthening local institutional capacity, ownership, and long-term sustainability? 7b I. Do you think the project has strengthened your own skills // the capacity of your institution? Yes/No? II. What do you think you are better at now (because of the project)? 7b III. Has your organization changed any of its practices because of anything you learned from working with this project? 7b IV. Do you have what you need (in terms of knowledge, skills and equipment) to continue the activities or work that were started through this project? Yes,/No If no, what would you need in order to continue? 7b V. What are the incentives for your participation in this project? l 7b VI. What would affect your ability to continue/sustainability? PRIORITY Recommendations If someone was going to do a project like this somewhere else, what recommendations do you have for the managers and implementers? R I. What should they change to make it better? Try to get at least 2 ideas R II. (optional) What should they make sure to keep the same? Observations of the context Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 94 Key Informant Interview: Participants/Beneficiaries Guide for FTF Activities in Dodoma, Manyara, and Morogoro Interviewers provide introduction that covers the following points:  Brief introduction of the evaluation team members  Brief background on Partnering to Innovation (P41) and its different components  Purpose of the evaluation and of the interview  Main topics of the of the interview  Confidentiality of responses  Request for and receipt of permission to interview Project Name: NAFAKA (NF), Tuboreshe Chakula (TC), Mwanzo Bora Nutrition Program (MBNP) Sub-partner: NAFAKA: CRS, IFDC, FIPS, MVIWATA, other MBNP: CONSENUTH, TAWG, Sharing Worlds, other Region: Dodoma; Manyara; Morogoro District: Ward: Village: Topic or Value Chain: Maize, Rice, Nutrition Institution type: Implementing Partner, Sub-Partner, CSO, SILC, VBAA, Producer Association Leader, Lead Farmer, Farmer Group Member, Health Facility/Clinic Worker, Community Health Worker, Agro-dealer, Miller/Processer, Other Private Business, Local Government Agency, Other (specify) ____________ Name(s) and Position(s): Institution:_____________________________________ Male (#)__ or Female(#)___ Date of interview:_______________________ Time of Interview__________ Name of Interviewer:________________ Name of note taker:__________ Confidentiality explained: Yes___ or No _____ Respondent agreed to be interviewed: Yes___ or No _____ Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 95 Project Name MBNP__ ,TC__, NF___ A. Could you please describe the activity(ies) of NAFAKA/Mwanzo Bora/Tuboreshe Chakula that you have participated in/led? a. What practice/activities that were introduced by the project did not exist before? b. To what extent have they been adopted 1) by the people you work with in this project? 2) By the wider community? B. How long have you been part of this activity(ies)? C. How did you hear about this project? (radio, friend, neighbor, extension worker, newspaper, government announcement?) Q5c.What results/outcomes have been achieved to date? 5c I. What have you achieved so far? 5c II. Describe the outcomes on the beneficiaries you work with. 5c For NAFAKA and Tuboreshe: Does the project increase buying of inputs and sales of production? PRIORITY Q2a. In what specific ways have the activities integrated nutrition and agriculture interventions in their work? 2a I. Please describe what good nutrition means to you? Respondent’s level of understanding of nutrition is sophisticated///moderate///poor 2a I. For Mwanzo Bora a. What specific agricultural activities do you do? b. When you talk about farming or gardens, do you talk about how they are related to family nutrition? Yes/No c. If yes, how do you integrate the work on agriculture and nutrition? II. Do you teach people about any of the following: a. Animal keeping (chicken, rabbits, etc.) Yes/No b. Growing Quality Protein Maize Yes/No c. Growing orange fleshed sweet potato d. Sensitization of husbands to support wives (accompanying to the clinic, etc.) Yes/No e. Empowering women to own the income resulting from production Yes/No 2a III. For NAFAKA a.What specific activities do you do that are related to nutrition? b.When you talk about agriculture, do you talk about how it is related to nutrition? Yes/No c. If yes, how do you integrate the work on agriculture and nutrition? 2c Q2c. What are these results [of integrating ag and nutrition] and how have they been achieved? 2c I. For Mwanzo Bora and NAFAKA: If nutrition and agriculture are integrated: a. How has bringing together agriculture and nutrition been beneficial? b. What challenges have you seen in trying to integrate agriculture and nutrition? c. What do you think are the main achievements of integrating nutrition and agriculture? d. What are the main challenges of trying to integrate nutrition and agriculture? e. What do you think contributed most to the success/achievements? f. What strategies do you think might help to further integrate them? g. Do you think the outcomes could have been different if the agriculture and nutrition work was not integrated? Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 96 2c II. Have you head of the Tuboreshe Chakula project? Yes/No 2c III. Do you promote the use of MNP sachets? Yes/No VI. Do you promote the use of fortified maize? Yes/No PRIORITY 1a Q1a. To what extent have women and youth been integrated into the design and implementation of the activities? 1a I. Are women, men and youth encouraged to attend the activities and be included in decision-making processes? (For HFW or PSG, ask about MEN!) 1a II. Are women and youth involved in the project activities? – Yes/No a. If yes, indicate their levels of involvement (attendance, leadership, role models, promotion to neighbors, working on demo plots) b. Is there any difference between men/women regarding the activities or positions? 1a III. What strategies (if any) were adopted to cater for the needs of women and youth (access to land, water, information, credit, improved/laborsaving technologies). Were these strategies successful? Yes/No Why? 1a IV. How does the improvement of women’s status (earning income, speaking up in a group, receiving education) affect her husband and home life? 9 V. How do men respond to the empowerment of women? (Give positive and negative examples) 1a VI. How would you rate the involvement of men, women and youth in this program? Q1b. Which approaches have proven most effective in reaching women and youth? 1b I. How do you identify the needs of female farmers, processors, or entrepreneurs? 1b II. Do the approaches have an impact on the self-esteem of women and/or youth? 1b III. Which of these approaches were most effective? 9 IV. Have any of these approaches had negative results? 1b V. What other strategies could be employed to better integrate women and youth into these types of activities? Q1c. How have women and youth benefitted from the activities? 1c I. How has this project benefitted men, women, and youth? Have these groups benefitted in different ways? 1c II. Women already play a big role in small-scale processing. To what extent are women food processors benefiting from the FTF activities (particularly TC)? 1c III. Any thoughts regarding the similarities and/or differences in benefits between these three groups of beneficiaries? Or, how can we explain the similarities and/or differences in benefits? PRIORITY 3 Q3. To what extent have the activities reached scale in terms of: outreach to direct and indirect beneficiaries; adoption of new technologies and practices; and adoption of innovative business models? Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 97 3 I. Do you have targets of numbers of people you are supposed to work with? Yes / No If yes, a. How many? b. So far how many have you worked with? 3 II. What are the challenges that make it hard to work with more people? 3 III. Do you have targets of numbers of demo plots/gardens/groups/other you are supposed to set up? Yes / No If yes, a. How many? b. So far how many have you worked with? 3 IV. What are the challenges that make it hard to set up more? 3 4b V. Do some of the benefits of your work also get to people who are not directly involved in the project? Yes / No If yes, a. How does that happen? PRIORITY 4a Q4a. To what extent are activities promoting and measuring diffusion of improved technologies or sharing knowledge of improved practices beyond direct beneficiaries? 4a I. What steps are you undertaking to promote the technologies/capacity/information to people other than the people who directly participate in project activities? 4b Q4b. If indirect beneficiaries are being reached, what are the primary vehicles through which this is taking place, how are indirect beneficiaries being measured, and how are benefits calculated? 4b I. What are the main ways that people who are not the direct participants/beneficiaries receive benefits? (e.g., information from current beneficiaries, radio, drama, etc.) 4b II. What is the best way to encourage new beneficiaries to join? Word of mouth? Other? 4b III. What attracts new entrants to a project activity? Social bonding or other incentives? 4b IV. How do you know if you are reaching people who are directly participating in the project? V. How are you measuring/estimating the number of indirect beneficiaries/level of diffusion? VI. How do you think we could try to measure them better? 5b Q5b. Are activity objectives and targets realistic and achievable before the respective activity end dates? 5b I. Do you have a specific number of people you are supposed to reach or work with? Yes / No II. Are you on track to be able to reach/work with/that many? Yes / No III. Is there a risk that you might not make the target? 5b IV. For agro-dealers: a. How many VBAAs are you working with? b. What is the total number of group members that those VBAAs work with? c. How many tons of fertilizer did you sell through VBAAs during this agricultural season? d. How many tons of fertilizer did you sell to farmers not working with VBAAs during this agricultural season? Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 98 e. Have adequate support or resources been made available to allow you to achieve the expected targets? f. If not, what else do you need? g. Knowing what you know now, do you think the activity objectives and targets were realistic? Yes / No h. If no, what would you change? 9 V. How is the local government involved in the work you do for this project? Value of the Project Q8. How are the activities perceived and valued by stakeholders, including direct beneficiaries, indirect beneficiaries, and GOT counterparts? 8 For participants I. What are the 2 most useful things you’ve learned through this project? 8 II. What have been the two most valuable results/benefits of your participation in this project? Unexpected Results Q9a. What unexpected results (negative consequences or positive results) of the activities have been realized? 9 I. Have there been positive outcomes that were not expected? Yes/No? What were they? 9 II. Have there been negative outcomes that were not expected? Yes/No? What were they? Sustainability 7b. What are the results in terms of strengthening local institutional capacity, ownership, and long-term sustainability? 7b I. Do you think the project has strengthened your own skills // the capacity of your institution? Yes/No? II. What do you think you are better at now (because of the project)? 7b III. Has your organization changed any of its practices because of anything you learned from working with this project? 7b IV. Do you have what you need (in terms of knowledge, skills and equipment) to continue the activities or work that were started through this project? Yes,/No a. If no, what would you need in order to continue? 7b V. What are the incentives for your participation in this project? l 7b VI. What would affect your ability to continue/sustainability? Recommendations If someone was going to do a project like this somewhere else, what recommendations do you have for the managers and implementors. R I. What should they change to make it better? Try to get at least 2 ideas R II. (optional) What should they make sure to keep the same? Observations of site (demo plot, agro-dealership, etc.) and/or context Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 99 Focus Group Discussion Guide Guide for FTF Activities in Dodoma, Manyara, and Morogoro Facilitators provide introduction that covers the following points:  Brief introduction of the evaluation team members  Brief background on Partnering to Innovation (P41) and its different components  Purpose of the evaluation and of the interview  Main topics of the of the interview  Confidentiality of responses  Request for and receipt of permission to interview Sub-partner: NAFAKA: CRS, IFDC, FIPS, MVIWATA, other MBNP: TAWG, Sharing Worlds, other Region: Dodoma; Manyara; Morogoro District: Ward: Village: Topic or Value Chain: Maize (MZ), Rice (RI), Vegetables (VG), Nutrition (NU), Institution type: Producer Association (PA), Farmer Group (FG), Farmer Association (FA), Women’s Peer Support Group (PSG), Civil Society Organization (CSO), Saving and Lending Cooperative (SILC), Other_________________ Male #__ // Female# ___ Date of interview:_______________________ Time of Interview__________ Name of Interviewer:________________ Name of note taker:__________ Confidentiality explained: Yes___ or No _____ Respondents agreed to be interviewed: Yes___ or No _____ Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 100 Overarching Project Name MBNP__ ,TC__, NF___ A. Could you please describe the activity(ies) of NAFAKA/Mwanzo Bora/Tuboreshe Chakula that you have participated in/led? B. What practice/activities that were introduced by the project did not exist before? C. To what extent have they been adopted a. by the people you work with in this project? b. By the wider community? D. How long have you been part of this activity(ies)? E. How did you hear about this project? (radio, friend, neighbor, extension worker, newspaper, government announcement?) Q5c.What results/outcomes have been achieved to date? 5c I. How has your participation in this group affected you and your family? Probe for income, nutrition, health, relationships within family, relationships with others, personal growth, etc. PRIORITY Q2a. In what specific ways have the activities integrated nutrition and agriculture interventions in their work? 2a I. Please describe what good nutrition means to you? Respondent’s level of understanding of nutrition is sophisticated///moderate///poor 2a I. For Mwanzo Bora a. Do you do any farming or vegetable garden training or work as part of this group? If yes – what? b. When you talk about nutrition, do you talk about how it is related to gardens? # Yes ___/ #No_____ i. If yes, please tell us about it c. Do you teach people about any of the following: i. Animal keeping (chicken, rabbits, etc.) Yes/No ii. Growing Quality Protein Maize Yes/No iii. Growing orange fleshed sweet potato sensitization of husbands to support wives (accompanying to the clinic, etc.) Yes/No iv. Empowering women to own the income resulting from production Yes/No 2a I. For NAFAKA a. What specific things do you learn or do with this group that are related to nutrition? b. When you talk about gardens, do you talk about how they are related to family nutrition? # Yes ___/ #No_____ i. If yes, please tell us about it c. Do you learn about any of the following: i. Animal keeping (chicken, rabbits, etc.) Yes/No ii. Growing Quality Protein Maize Yes/No Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 101 iii. Growing orange fleshed sweet potato sensitization of husbands to support wives (accompanying to the clinic, etc.) Yes/No iv. Empowering women to own the income resulting from production Yes/No 2a I. For SILCs a. What specific things do you learn or do with this group that are related to nutrition or to agriculture or vegetable gardens? b. When you talk about nutrition, farming or gardens, do you talk about how nutrition and gardens are related to each other? # Yes ___/ #No_____ c. If yes, please tell us about it Q2a. In what specific ways have the activities integrated nutrition and agriculture interventions in their work? 2a I. Please describe what good nutrition means to you? Respondent’s level of understanding of nutrition is sophisticated///moderate///poor 2a II. Have you ever heard of MNS packets? # Yes ___/ #No_____ a. If yes, Have you ever used MNS packets? # Yes ___/ #No_____ b. If yes, Probe for frequency and how used // If no, why not? c. Where do you get them? 2a III. Have you ever heard of fortified maize meal? # Yes ___/ #No_____ a. If yes, Have you ever used fortified maize meal? # Yes ___/ #No_____ b. If yes, Probe for frequency and how used /// If no, why not? c. Where do you get it? 2c Q2c. What are these results and how have they been achieved? 2c If nutrition and agriculture are integrated: I. What kind of differences has it made to you to link thinking about nutrition and agriculture/gardening? a. Do you do anything differently now? # Yes ___/ #No_____ b. If yes, please tell us about it PRIORITY 3 Q3. To what extent have the activities reached scale in terms of: outreach to direct and indirect beneficiaries; adoption of new technologies and practices; and adoption of innovative business models? 3 I. How many people are in your group? II. Has the size of your group changed over time? Yes / No If yes, a. What was the largest number? b. What was the smallest number? c. What has affected the number of people who actively participate? d. Was there something that happened that led to people joining or leaving? What? 3 III. What responsibilities do you have to commit to to be part of this group? (probe for fee, demo plot or other work responsibilities, etc.) 3 IV. What are the challenges that make it hard for more people to be part of your group? V. What kinds of things might make it more attractive for more people to participate? 3 VI. Do you encourage other people you know to join? VII. Do most of them join? Yes / No VIII. For those who do not join, why do you think they decided not to join? Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 102 3 IX. Do you teach to other people the things that you learn in this group? 3 X. Did you or other people in your community already do the things you are learning about in this project before the project started? Yes / No XI. Are other people in your community recently starting to do the same kinds of things that you have been learning about in this group? Yes / No If yes, How are they learning about these things? XII. About what percentage of people in your community are now doing the kinds of things you learn about in this project? PRIORITY 1a Q1a. To what extent have women and youth been integrated into the design and implementation of the activities? 1a I. Are women, men and youth encouraged to attend the activities and be included in decision-making processes? (For HFW or PSG, ask about MEN!) 1a II. Are women and youth involved in the project activities? – Yes/No a. If yes, indicate their levels of involvement (attendance, leadership, role models, promotion to neighbors, working on demo plots) b. Is there any difference between men/women regarding the activities or positions? 1a III. What strategies (if any) were adopted to cater for the needs of women and youth (access to land, water, information, credit, improved/laborsaving technologies). Were these strategies successful? Yes/No a. Why? 1a IV. How does the improvement of women’s status (earning income, speaking up in a group, receiving education) affect her husband and home life? 9 V. How do men respond to the empowerment of women? (Give positive and negative examples) 1a VI. How would you rate the involvement of men, women and youth in this program? Q1b. Which approaches have proven most effective in reaching women and youth? 1b I. How do you identify the needs of female farmers, processors, or entrepreneurs? 1b II. Do the approaches have an impact on the self-esteem of women and/or youth? 1b III. Which of these approaches were most effective? 9 IV. Have any of these approaches had negative results? 1b V. What other strategies could be employed to better integrate women and youth into these types of activities? Q1c. How have women and youth benefitted from the activities? 1c I. How has this project benefitted men, women, and youth? Have these groups benefitted in different ways? 1c II. Women already play a big role in small-scale processing. To what extent are women food processors benefiting from the FTF activities (particularly TC)? 1c III. Any thoughts regarding the similarities and/or differences in benefits between these three groups of beneficiaries? Or, how can we explain similarities and/or differences in benefits? Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 103 Value of the Project Q8. How are the activities perceived and valued by stakeholders, including direct beneficiaries, indirect beneficiaries, and GOT counterparts? 8 I. What have been the two most valuable things you’ve done or learned that you’ve gotten from this project/activity? 8 II. What have been the two most valuable benefits results of your participation in this project? Unexpected Results Q9a. What unexpected results (negative consequences or positive results) of the activities have been realized? 9 I. Have there been positive outcomes that were not expected? a. What were they? 9 II. Have there been negative outcomes that were not expected? a. What were they? Sustainability 7b. What are the results in terms of strengthening local institutional capacity, ownership, and long-term sustainability? 7b I. Do you think the project has strengthened your own skills // the capacity of your institution? # Yes ___/ #No_____ a. If yes, What do you think you are better at now (because of the project)? 7b II. Do you have what you need (in terms of knowledge, skills and equipment) to continue the activities or work that were started through this project? # Yes ___/ #No_____ a. If no, what would you need in order to continue? III. What are the incentives for your participation in this project? l 7b IV. Thinking about the future – after the project/organization that is supporting you ends, a. What will you continue to do on your own? b. What will be hard to do without support of the project? 7b V. What do you think might be the lasting results of this project five years from now? Recommendations If someone was going to do a project like this somewhere else, what recommendations do you have for the people managing that project. R I. What should they change to make it better? Try to get at least 2 ideas R II. (optional) What should they make sure to keep the same? Observations of site / Context Notes Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 104 ANNEX IV: DOCUMENTS REVIEWED Abt Associates. Annual Report 2012-13, Processing and Consumption Project - Tuboreshe Chakula. Washington, DC: Abt Associates, 2013. Print. Abt Associates. Tuboreshe Chakula Performance Management Plan. Washington, DC: Abt Associates, 2013. Print. Abt Associates. Tuboreshe Chakula – Performance Management Plan (FY 2015). Washington, DC: Abt Associates, 2014. Print. Abt Associates. USAID Tanzania Feed the Future, Tuboreshe Chakula – Fifth Survey Report, June 2014. Washington, DC: U.S. Government, 2014. Print. Abt Associates. USAID Tanzania Feed the Future, Tuboreshe Chakula – Rapid Survey Report, September 2014. Washington, DC: U.S. Government, 2014. Print. ACDI/VOCA. NAFAKA Staples Value Chain Activity, Annual Performance Report (October 1, 2012 – September 30, 2013). Washington, DC: ACDI/VOCA, 2013. Print. ACDI/VOCA. NAFAKA Staples Value Chain Activity, Annual Performance Report (October 1, 2013 – September 30, 2014). Washington, DC: ACDI/VOCA, 2014. Print. ACDI/VOCA. NAFAKA Staples Value Chain Activity, Annual Performance Report (October 1, 2014 – December 31, 2014). Washington, DC: ACDI/VOCA, 2015. Print. ACDI/VOCA. NAFAKA Staples Value Chain Gender Analysis, Maize Areas, Kongwa and Kiteto Final Draft. Washington, DC: ACDI/VOCA, 2012. Print. ACDI/VOCA. Staples Value Chain Activity – NAFAKA – Annual Outcome Survey 2013, Preliminary Results Report - Draft. Washington, DC: ACDI/VOCA, 2013. Print. ACDI/VOCA. Staples Value Chain Activity – NAFAKA – Baseline Report. Washington, DC: ACDI/VOCA, 2012. Print. ACDI/VOCA. Staples Value Chain Activity – NAFAKA – Performance Management Plan (FY 2011 – 2016). Washington, DC: ACDI/VOCA, 2013. Print. ACDI/VOCA. Tanzania Staple Value Chain – Introduction of NAFAKA Grantee Hub Agrodealer. Washington, DC: ACDI/VOCA. Print. Africare. Improving Nutrition in Tanzania Mwanzo Bora Nutrition Program. Washington, DC: Africare, 2013. Print. Africare. Mwanzo Bora Nutrition Program, Baseline Survey Report. Washington, DC: Africare, 2012. Print. Africare. Mwanzo Bora Nutrition Program, Quarterly Report (October – December 2014). Washington, DC: Africare, 2015. Print. Africare. Mwanzo Bora Nutrition Program, Report of Data Quality Assessment. Washington, DC: Africare, 2012. Print. Africare. Mwanzo Bora Nutrition Program, Year Two Annual Report. Washington, DC: Africare, 2013. Print. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 105 Africare. Mwanzo Bora Nutrition Program, Year Three Annual Report. Washington, DC: Africare, 2014. Print. Africare Mwanzo Bora Tanzania Feed the Future Baseline Indicators Deliverable #1: Data availability. By Alexis Murphy. Washington, DC: Africare. Print. Africare. Report of Data Quality Assessment Mwanzo Bora Nutrition Program (MBNP). Washington, DC: Africare, 2013. Print. Africare. USAID/Tanzania Feed the Future Mwanzo Bora Nutrition Program, Performance Monitoring Plan. Washington, DC: U.S. Government, 2013. Print. De Brauw, Allen, Aulo Gelli, and Summer Allen. Identifying Opportunities for Nutrition-Sensitive Value-Chain Interventions. Washington, DC: International Food Policy Research Institute, 2015. Web: http://www.ifpri.org/publication/identifying-opportunities-nutrition-sensitive-value-chain￾interventions Food and Agriculture Organization of the United Nations. The State of Food and Agriculture 2010-2011, Women in Agriculture Closing the Gender Gap for Development. Tech. Rome: Food and Agriculture Organization of the United Nations, 2011. Web. http://www.fao.org/docrep/013/i2050e/i2050e.pdf Hillenbrand, Emily. Baseline Gender Disparities Relevant to a Nutrition Education Strategy in Rural Bangladesh. Web. http://www.ipcbee.com/vol39/011-ICNFS2012-N019.pdf Hoddinott, John F. “Agriculture, Health, and Nutrition: Toward Conceptualizing the Linkages." Reshaping Agriculture for Nutrition and Health. Washington, DC: International Food Policy Research Institute, 2012. 13-20. Web. http://ebrary.ifpri.org/cdm/ref/collection/p15738coll2/id/127118 Hoddinott, John. “Agriculture, Health, and Nutrition: Toward Conceptualizing the Linkages. 2020 Conference: Leveraging Agriculture for Improving Nutrition and Health." International Food Policy Research Institute, Washington, DC (2012). International Food Policy Research Institute, 2015. Web. https://agrilinks.org/sites/default/files/resource/files/nutsens.pdf International Fund for Agricultural Development. Improving Nutrition through Agriculture. Tech. Rome: International Fund for Agricultural Development, 2014. Web. http://www.ifad.org/pub/thematic/nutrition/nutrition_e_web.pdf Jaleta, Moti, Berhanu Gebremedhin, and Dirk Hoekstra. "Smallholder commercialization: Processes, determinants and impact." (2009). Savoie, Rebecca. Annual Report, Tuboreshe Chakula, October 2013 – September 2014. By Rebecca Savoie. Washington, DC: Abt Associates, 2014. Print. U.S. Government. Brief on Gender-Based Roles, Constraints and Opportunities in Rice Value Chain- Kilombero and Mvomero Districts. Washington, DC: U.S. Government. Print. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 106 U.S. Government. Draft USAID Tanzania Feed the Future Performance Monitoring & Evaluation Plan (PMP). Washington, DC: U.S. Government, 2011. Print. U.S. Government. Gender Assessment for the Tanzania CDCS. Washington, DC: U.S. Government, 2012. Print. U.S. Government. Gender Equality and Female Empowerment Policy. Washington, DC: U.S. Government, 2012. Web. https://www.usaid.gov/sites/default/files/documents/1865/GenderEqualityPolicy_0.pdf U.S. Government. Household Hunger Scale: Indicator Definition and Measurement Guide. By Terri Ballard, Jennifer Coates, Anne Swindale, Megan Deitchler. Washington, DC: U.S. Government, 2011. Print. U.S. Government. Improving Nutrition through Agriculture Technical Brief: Understanding the Food Production Pathway. Washington, DC: U.S. Government, 2014. Web. U.S. Government. Tanzania. Feed the Future. Washington, DC: U.S. Government. Web. . U.S. Government. Tanzania FY 2011 – 2015 Multi-Year Strategy. Washington, DC: U.S. Government, 2011. Print. U.S. Government. USAID Country Development Cooperation Strategy, October 3, 2014 – October 3, 2019. U.S. Government, 2014. Web. https://www.usaid.gov/sites/default/files/documents/1860/CDCS%20Tanzania%20FINAL.pdf. U.S. Government. USAID/Tanzania Feed the Future Monitoring and Evaluation (TFTF-M&E) Project. Washington, DC: U.S. Government, 2013. Print. U.S. Government. USAID Tanzania Feed the Future Tuboreshe Chakula Project Final Survey Report, May 2015. Washington, DC: U.S. Government, 2015. Print. United Nations World Food Programme. Comprehensive Food Security & Vulnerability Analysis (CFSVA) Tanzania, 2012. Tech. Rome: United Nations World Food Programme, 2013. Print. United Republic of Tanzania. Basic Information Document, National Panel Survey (NPS 2010-2011). Dar es Salaam: United Republic of Tanzania, 2011. Print. United Republic of Tanzania. Tanzania Agriculture and Food Security Investment Plan (TAFSIP), 2011 – 2021. Dar es Salaam: United Republic of Tanzania, 2011. Print. United Republic of Tanzania. Tanzania Agriculture and Food Security Report 2011-12 to 2020-21. Dar es Salaam: United Republic of Tanzania, 2011. Print. United Republic of Tanzania. Tanzania Demographic and Health Survey 2010, Preliminary Report. Dar es Salaam: United Republic of Tanzania, 2010. Print. United Republic of Tanzania. Tanzania Mainland Household Budget Survey Main Report 2011/12. Dar es Salaam: United Republic of Tanzania, 2014. Print. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 107 United Republic of Tanzania. Tanzania National Panel Survey Report - Wave 2, 2010 – 2011. Dar es Salaam: United Republic of Tanzania, 2012. Print. World Bank. World Development Report 2012: Gender Equality and Development. Washington, DC: The World Bank, 2012. Web. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 108 ANNEX V: PERSONS INTERVIEWED Category Type Total Implementing Partner KII 29 Sub-Partner KII 30 CSO KII 3 VBAA KII 15 Lead Farmer KII 2 Agro-dealer KII 12 SILC FA/PSP KII 4 Miller/Processer KII 9 Health Center/Facility Worker KII 15 Community Health Worker KII 28 Local Extensionist KII 5 Government KII 54 USAID KII 6 Total 212 Farmers (FI/FG) FDG 131 SILC FDG 48 Peer Support Group FDG 153 Total 332 Retailers Opportunistic Interview 17 Women Opportunistic Interview 22 Other Opportunistic Interview 2 Total 41 Overall Total 585 Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 109 ANNEX VI: PERFORMANCE AGAINST PMP INDICATORS Mwanzo Bora Nutrition Program55 Indicators FY 2014 Target Achievement % Q1 Q2 Q3 Q4 Number of people trained in child health and nutrition through USG-supported programs 798 1,521 6,530 5,296 14,034 14,145 101% Number of children under five reached by USG-supported nutrition programs 801,272 134,117 202,778 135,038 927,808 1,273,205 137% Number of children under five who received Vitamin A from USG-supported programs 709,643 102,559 927,808 812,202 88% Number of districts with plans and budgets that include at least three essential nutrition actions 13 20 20 20 20 20 100% Number of beneficiaries with home gardens or small livestock as a proxy for access to nutritious foods and income 780 934 57,720 22,766 34,650 82,200 237% Number of women of reproductive age reached by USG￾supported nutrition programs 826,272 178,394 256,858 208,132 1,073,852 1,469,656 135% Number of people reached through community awareness supported by MBNP - 216 22,373 69,700 - 92,289 N/A 55 Africare. Mwanzo Bora Nutrition Program, Year Three Annual Report. Washington, DC: Africare, 2014. Print. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 110 Performance against PMP Indicators – NAFAKA Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 111 Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 112 Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 113 Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 114 60 ACDI/VOCA. NAFAKA Staples Value Chain Activity, Annual Performance Report (October 1, 2014 - December 31, 2014). Washington DC: ACDI/VOCA, 2015. Print Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 115 Performance against PMP Indicators – Tuboreshe Chakula FTF PROCESSING AND CONSUMPTION PROJECT Indicator Number Indicator Title FY14 Target FY 14 Actual Total / % Achieved Q1 Q2 Q3 Q4 4.5.2-5 (RiA) Number of farmers and others who have applied improved technologies or management practices as a result of USG assistance 550 cumu￾lative 567 621 699 776 776 / 141% Sex of Owner Female 16 5 8 10 39 Male 155 47 69 65 336 Joint 13 2 1 2 18 Status New 184 54 78 77 393 Continuing 383 567 621 699 766 Up-scaling technology Number of processors fortifying with dossifiers or micro-feeders 0 0 7 0 7 Number of processors refining oil 0 0 1 2 3 Number of processors using moisture meters 15 0 0 81 4.5.2-7 (RiA) Number of individuals who have received USG supported short term agricultural sector productivity / food security training 500 298 819 333 1527 2977 / 595% Sex Female 64 500 99 521 Male 234 319 234 1006 Sector Processors 295 444 278 156 People in Government 3 375 55 1371 4.5.2-29 (RiA) Value of Agricultural and Rural Loans USD 400,000 USD 335K USD 27.5K USD 653.7K USD 550.4K USD 1,566,600 392% Sex of Owner Female 0 0 USD 70K 0 USD 70K Male USD 335K USD 24.5K USD 546.7K USD 363.7K USD 1,269,900 Joint 0 USD 3K USD 37K USD 186.7K USD 226,700 Scale Small Scale USD 41K USD 27.5K USD 174.8K USD 272.3K USD 515,600 Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 116 FTF PROCESSING AND CONSUMPTION PROJECT Indicator Number Indicator Title FY14 Target FY 14 Actual Total / % Achieved Q1 Q2 Q3 Q4 Medium Scale USD 294K 0 USD 478.9K USD 278.1K USD 1,051,000 61 Savoie, Rebecca. Annual Report, Tuboreshe Chakula, October 2013 - September 2014. By Rebecca Savoie. Washington DC: .Abt Associates, 2014. Print. Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 117 ANNEX VII: DISCLOSURE OF ANY CONFLICTS OF INTEREST Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 118 Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 119 Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 120 Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 121 Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 122 Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 123 Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 124 Mid-term Performance Evaluation of the USAID/Tanzania Feed the Future Program 125 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523