USAID/INDIA SCALING NUTRITION-SENSITIVE FISHERIES TECHNOLOGIES AND INTEGRATED APPROACHES THROUGH PARTNERSHIP IN ODISHA Ex-Post Facto Final Evaluation Report IDIQ # 72038619D0000; Task Order # 72038620F00003 June 13, 2023 Disclaimer: The views of the expressed in this document do not necessarily reflect the views of the United States Agency for International Development or the United States Government PHOTO BY LUCY O’BRYAN FOR USAID Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 2 SCALING NUTRITION-SENSITIVE FISHERIES TECHNOLOGIES AND INTEGRATED APPROACHES THROUGH PARTNERSHIP IN ODISHA EX-POST FACTO FINAL EVALUATION REPORT Contract Title: Collaborating, Learning and Adapting in India Mechanism (CLAIM) Contract Number: 72038619D00001 Task Order Number: 72038620F00003 Contractor: Panagora Group COR: Chandan Samal Submitted: March 21, 2023 Cover photo: Woman at a solar dryer in Jagatsinghpur by Lucy O’Bryan for USAID Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 3 TABLE OF CONTENTS ACRONYMS 7 EXECUTIVE SUMMARY 9 1. PROJECT OVERVIEW 16 1.1 PROJECT BACKGROUND 16 1.2 PROJECT PARTNERS 16 1.3 PROJECT GOAL AND OBJECTIVES 17 1.4 PROJECT GEOGRAPHIC FOCUS 20 2. EVALUATION OVERVIEW 22 2.1 EVALUATION PURPOSE 22 2.2 EVALUATION TIMELINE 22 2.3 EVALUATION QUESTIONS 22 2.4 EVALUATION TEAM 23 2.5 DATA COLLECTION 23 2.6 ETHICAL CONSIDERATIONS 27 2.7 DATA ANALYSIS METHODS 27 2.8 LIMITATIONS AND POTENTIAL BIASES 28 3. FINDINGS AND CONCLUSIONS 30 EQ 1: PROJECT COMPONENTS - FINDINGS 30 EQ 1: PROJECT COMPONENTS - CONCLUSIONS 36 EQ 2: INTERVENTION SUCCESS FACTORS - FINDINGS 36 EQ 2: INTERVENTION SUCCESS FACTORS - CONCLUSIONS 41 EQ 3: CONTINUED USE AND HINDERING FACTORS - FINDINGS 42 EQ 3: CONTINUED USE AND HINDERING FACTORS – CONCLUSIONS 46 EQ 4: CONTINUED IMPLEMENTATION - FINDINGS 47 EQ 4: CONTINUED IMPLEMENTATION – CONCLUSIONS 48 EQ 5: SUSTAINABILITY FACTORS - FINDINGS 48 EQ 5: SUSTAINABILITY FACTORS - CONCLUSIONS 50 4. LESSONS LEARNED 52 4.1 TAILORING BENEFICIARY SELECTION 53 4.2 BUSINESS PLANS AND MARKET LINKAGES 54 4.3 DIRECT IMPLEMENTATION VERSUS LEVERAGING MARKET SYSTEMS 54 4.4 CHANGING CONSUMPTION AND TASTE 54 4.5 LEVERAGE CONVERGENCE OF AGENCY OBJECTIVES 55 5. RECOMMENDATIONS 56 5.1 TAILORING BENEFICIARY SELECTION 56 5.2 BUSINESS PLANS AND MARKET LINKAGES 56 5.3 DIRECT IMPLEMENTATION VERSUS LEVERAGING MARKET SYSTEMS 56 5.4 CHANGING CONSUMPTION AND TASTE 57 5.5 LEVERAGE CONVERGENCE OF AGENCY OBJECTIVES 57 ANNEX 1: SCOPE OF WORK 58 ANNEX 2: EVALUATION TIMEFRAME 61 ANNEX 3: EVALUATION TEAM SUMMARY 62 Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 4 ANNEX 4: REFERENCE CITATIONS 63 ANNEX 5: GAP ANALYSIS 66 ANNEX 6: DATA COLLECTION PROTOCOLS 71 ANNEX 7: KII CONTACT LIST 82 ANNEX 8: FGD SUMMARY 83 ANNEX 9: MINI SURVEY DATA SET 84 Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 5 LIST OF FIGURES Figure 1: Project Theory of Change.....................................................................................................................18 Figure 2: Project Zone Map ...................................................................................................................................21 Figure 3: Pre-Project Importance of Fish in Household Diets (left) and Incomes (right).......................37 Figure 4: Household Fishpond Owner Typologies...........................................................................................37 Figure 5: Survey Respondents’ Rankings of Taste of Products Made from Whole Dried Mola............40 Figure 6: Usefulness of Technical Training (left) and Post-Project Change in Incomes (right)..............42 Figure 7: Usefulness of Nutrition Training (left) and Post-Project Change in Diet (right).....................49 LIST OF TABLES Table 1: Project Components and Description.................................................................................................18 Table 2: Proposed Project Components and Activities..................................................................................19 Table 3: Project Zone - Districts and Blocks.....................................................................................................20 Table 4: Evaluation Questions by Topic..............................................................................................................22 Table 5: Project Stakeholder Groups..................................................................................................................24 Table 6: Site Visits by Type ....................................................................................................................................26 Table 7: Geographic Distribution of Data Collection Activities...................................................................27 Table 8: Reported Project Components and Key Activities with Targets and Results...........................30 Table 9: Evolution of Carp-Mola Polyculture Beneficiaries............................................................................32 Table 10: Evolution of WSHGs Drying Fish, Anganwadi Centers/Schools Using Fish in SNPs.............33 Table 11: Evolution of Improved Food and Nutrition Security and Scaling Activities.............................35 Table 12: Status of Community Fishponds in February 2023 ........................................................................44 Table 13 Status of Solar Dryers (February 2023_............................................................................................46 Table 14: Mola-Carp Polyculture on Communal Pond Illustrative Cost of Production..........................50 Table 15: Reported Project Components and Key Activities with Targets and Results (revisited) ....52 Table 16: Evaluation Timeline and Deliverable Dates......................................................................................61 Table 17: Summary of Evaluation Team LOE and Roles.................................................................................62 Table 18: Donor and Implementing Partner Staff KII Contact List ..............................................................82 Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 6 Table 19: Focus Group Discussion Summary ....................................................................................................83 Table 20: Mini Survey Data Set .............................................................................................................................84 Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 7 ACRONYMS APS Annual Program Statement CDCS Country Development Cooperation Strategy CGIAR Consultative Group on International Agricultural Research CLAIM Collaborating, Learning and Adapting in India Mechanism (Activity) EQ Evaluation Question F&ARD (Odisha State Government Department of) Fisheries and Animal Resource Development ERB Ethical Review Board FGD Focus Group Discussion GOI Government of India GB Gopabandhu Nagar (block) ICAR-CIFT Indian Council of Agricultural Research-Central Institute of Fisheries Technology ICDS Integrated Child Development Services INR Indian Rupees IPP India Partnership Program IR Intermediate Result KII Key Informant Interview KISS Kalinga Institute of Social Sciences MOU Memorandum of Understanding MSSRF MS Swaminathan Research Foundation MSME Micro, Small and Medium Enterprises (Department) OLM Odisha Livelihood Mission OSP Orange Sweet Potato PIO Public International Organization SBCC Social and Behavior Change Communication SHG Self-Help Group SNP Supplemental Nutrition Program Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 8 TA Technical Assistance USAID U.S. Agency for International Development WCD&MS (Odisha State Government Department of) Women and Child Development and Mission Shakti WSHG Women’s Self-Help Group Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 9 EXECUTIVE SUMMARY Between November 2022–March 2023, at the request of the USAID/India Mission, Panagora Group deployed a three-person team to implement an ex-post facto evaluation of the “Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha” project, which ended on March 21, 2021. PROJECT OVERVIEW In October 2017, USAID/India funded WorldFish to demonstrate approaches for nutrition-sensitive technologies involving fish and fish products. Since 2016, WorldFish had been embedded with the State Government of Odisha Department of Fisheries and Animal Resource Development (F&ARD) through a six-year memorandum of agreement to provide technical assistance (TA) for implementation of the Odisha Fisheries Policy. Originally a three-year project, in September 2020, USAID/India approved a six-month no cost extension to March 31, 2021. Additional project partners included the State Department of Women and Child Development and Mission Shakti (WCD&MS), which manages supplemental nutrition programs (SNPs) under the Integrated Child Development Services (ICDS) program through schools and a network of rural childcare centers called “Anganwadi centers.” Other partners included the State Micro, Small, & Medium Enterprises Department, the Indian Council of Agricultural Research-Central Institute of Fisheries Technology (ICAR-CIFT) based in Cochin, Kerala, and Odisha Livelihood Mission. The project’s stated goal was to “improve food and nutrition security in the Indian state of Odisha,” and its four integrated objectives were as follows: 1) increased availability of nutrient-rich fish; 2) increased accessibility of fresh fish and fish-based products; 3) increased consumption of nutritious foods; and 4) regional and national scaling of innovative, nutrition-sensitive fisheries technologies and integrated approaches. WorldFish and the Department of F&ARD implemented the project in seven blocks in the districts of Baleshwar Mayurbhanj and Jagatsinghpur. In addition, they identified ten coastal sites for construction of demonstration solar dryers for marine fish. EVALUATION OVERVIEW The purpose of this ex-post facto evaluation was to assess sustainability, and to learn how outcomes and impacts evolved after completion of the project. Evaluation findings and conclusions will also guide future programs in similar sectors. Panagora Group deployed a three-person evaluation team to implement this evaluation between November 21, 2022–March 31, 2023. The evaluation team collected data through 1) an initial literature review and gap analysis; 2) key informant interviews and focus group discussions with project beneficiaries; 3) a quantitative “mini survey” of beneficiaries; and 4) site visits to project￾related sites. The evaluation focused on five topics corresponding to the following five evaluation questions (EQs): EQ 1. Project Components: To what extent and how did the project components reach their objectives during implementation? EQ 2. Intervention Success Factors: What factors hindered interventions, and what factors contributed to their success? EQ 3. Continued Use and Hindering Factors: Are project beneficiaries and stakeholders continuing to use promoted behaviors related to use of innovative, nutrition-sensitive fisheries technologies and integrated approaches to improve food and nutrition security? If so, how? If not, what factors hinder continuation of these practices? Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 10 EQ 4. Continued Implementation: Have the implementing partners/stakeholders strengthened by USAID funding continued to implement project components? What is stakeholder perspective/feedback about project impact? EQ 5. Sustainability Factors: What are the factors that contributed to or impaired the long-term sustainability of project outcomes and outputs? The team used three data analysis methods: 1) content analysis; 2) trend analysis; and 3) triangulation. Potential biases and limitations inherent in the methodology included: 1) positive response (“halo”) bias; 2) selection bias; 3; sampling limitations; and 4) subjective measurements. CONCLUSIONS EQ1: To what extent and how did the project components reach their objectives during implementation? Increased Availability - WorldFish aimed to increase the availability of small fish (e.g., mola) among “backward” and vulnerable communities in target districts through training selected individual household fishpond famers and members of self-help groups (SHGs) managing community fishponds, and by providing 50 percent of operational set-up costs to individual farmers and 60 percent to women in SHGs managing community ponds. In addition, WorldFish undertook a one-time free distribution of mola and carp seed and initial fish feed. While the project surpassed initial proposed targets, it fell short of revised targets of 2,000 household fishponds, reaching only 789 ponds. It also fell slightly short of the revised target for community fishponds, reaching 22 ponds against the target of 24. Donors and implementing partner staff attributed these shortfalls to a beneficiary recruitment drop and limitations on activity implementation following a 2019 budget delay and subsequent COVID-19 restrictions. Increased Accessibility - WorldFish aimed to increase accessibility to fish and fish products through creating and leveraging private sector partnerships to develop new fish-based products as well as through institutional linkages with the ICDS program’s SNPs to distribute fish and fish products at Anganwadi centers and schools. However, challenges related to sanitation and quality of existing fish supplies forced the project to reduce its focus on private sector linkages. The project instead pivoted to focus on fostering the use of solar dryers by women’s self-help groups (WSHGs) to increase accessibility of sanitary and higher quality dried fish. WorldFish met its objective of establishing ten solar dryers in coastal areas managed by WSHGs through collaboration with ICAR-CIFT, which supplied the dryers and trained group members on their use. ICAR-CIFT also provided dried fish powder, manufactured in Kerala, to help WorldFish meet its objective of 50 Anganwadi centers using fish products in the mother and child health care ICDS. However, while WorldFish, through an agreement with Falcon Marine Ltd., was able to launch a pilot program for distribution of fish through WCD&MS-managed schools, the program was curtailed in 2019 due to USAID concerns. The project only reached 24 schools using fish in the mid￾day meal program, short of its target of 50. Increased Consumption - The project aimed to increase consumption of nutrient-rich fish and vegetables, especially among women and young children, through subcontracting with MAMTA￾Health Institute for Mother and Child to develop a set of social behavior change communication (SBCC) materials to promote fish and nutrition. This activity did not have a discrete quantitative target. However, the project documentation describes a number of institutional activities and linkages related to this component. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 11 Improved Food and Nutrition Security in Odisha and Scaling - The project aimed to improve food and nutrition security through a range of “collaboration arrangements” with government and non-government agencies to promote scaling throughout Odisha and in diverse regions beyond. WorldFish implemented these linkages through an array of mechanisms which included “collaboration arrangements,” a corporate social responsibility event, meetings with other regional, national, and international organizations for scaling up the program, and capacity building activities for government functionaries. Project target units were revised to reflect execution of memoranda of understanding (MOUs), establishment of partnerships, influence on policies, cross￾country visits, and training of officials. However, the project did not set quantitative targets for these project activities. EQ2: What factors hindered interventions, and what factors contributed to their success? Increased Availability - Activities that aimed to increase availability included promotion of mola￾carp polyculture through training, provision of start-up costs, and distribution of inputs to farmers and SHGs which were using household and communal ponds. The key factor determining the success of these activities was the market orientation of beneficiaries—which ranged from very low to very high—which influenced how much they were willing to invest in fish production. Additional factors included type of pond (seasonal or perennial); and in the case of communal ponds, the terms of leases, previously two years, now extended to five years. Increased Accessibility - Activities to increase accessibility focused on improving the quality and hygiene of fish products through promoting the use of solar dryers, as well as fostering linkages with institutional systems that could use fish and fish products, especially SNPs at Anganwadi centers and schools providing mid-day meals. Factors that hindered or facilitated the success of solar dryer promotion included seasonality, potential for dual use, sales structure, and cash flow. Factors that hindered or facilitated the success of activities that promoted institutional linkages with programs that could use fish products included acceptance by recipients, and comparative cost and authorization to procure such products by state officials. Increased Consumption - Most activities that aimed to increase consumption of fish and fish products centered on SBBC. Others included diverse institutional linkages that aimed to disseminate the approaches promoted by the project. Improved Food and Nutrition Security in Odisha and Scaling - The key factor determining the success of activities that aimed to increase food and nutritional security and scale up approaches was how well the project leveraged the alignment of objectives between agencies. An additional factor was the way in which WorldFish was able to engage and utilize existing systems. EQ 3: Are project beneficiaries and stakeholders continuing to use promoted behaviors related to use of innovative, nutrition-sensitive fisheries technologies and integrated approaches to improve food and nutrition security? If so, how? If not, what factors hinder continuation of these practices? Increased Availability - The degree of beneficiaries’ market orientation is the primary factor related to their continued use of behaviors that the project promoted to improve the availability of fish from household and community fishponds. The team estimated that less than 10 percent of fish farmers are sufficiently market-oriented to understand the value of purchasing inputs with their own funds to improve productivity; most obtain some quantity of fish from their ponds for household consumption, but do not purchase inputs beyond seed for re-stocking, instead allocating their Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 12 resources to other livelihood activities. An exception to this is SHGs that farm fish on community ponds. In these cases, lease terms are most likely to be the determining factor in the continued application of promoted technologies. Across both types of fish farmers, none changed their cultivation of vegetables as a result of receiving vegetable seeds, and none continued cultivation of orange sweet potato. Increased Accessibility - WSHGs using solar dryers for fish drying face multiple cash flow-related issues that determine the financial viability of continued use of these units, including seasonality, potential for dual use, as well as income structure, which can take the form of price premiums for quantity, delaying sales to reduce market gluts, or some combination of these. Given the low business skills of most WSHG members, determining profitability is often confusing, and group members lack the ability to expand market linkages. The relatively high costs of repairs and maintenance of the solar dryer units is the key determining factor in continued use of the dryers. Following project close-out, the WSHGs have not demonstrated willingness to invest their own funds to this end. EQ 4: Have the implementing partners/stakeholders strengthened by USAID funding continued to implement project components? What is stakeholder perspective/feedback about project impact? Although mostly still in pre-implementation phase, implementing partners have continued to pursue project objectives through additional activities funded by GOI entities and external donors. Although WorldFish initially launched its activities through an MOU with the State Department of F&ARD, the agency has established robust linkages with a number of additional official entities through leveraging joint objectives. This convergence of agency objectives strengthened the ability of WorldFish to continue pursuing project components. EQ 5: What are the factors that contributed to or impaired the long-term sustainability of project outcomes and outputs? Once established, at a subsistence level, systems to increase consumption of fresh and dried whole small fish are sustainable, as they present limited to no additional costs to household and community fishpond farmers. This is largely due to the natural regeneration of these fish through prolific breeding, including their re-establishment in perennial farms. In the case of more market-oriented farmers, including SHG-managed community ponds, increased production through recommended investments has the potential for long-term sustainability based on the significantly positive return on investment generated. However, most of these farmers are choosing to invest their capital resources into alternative livelihoods rather than fishponds, which puts sustainability of fish farming as a commercial enterprise that requires capital investment into question. LESSONS LEARNED Disaggregating beneficiary selection criteria by objective can improve project effectiveness and efficiency. The level of adoption of promoted mola-carp production techniques was underwhelming. Most beneficiaries simply consumed increased productivity that resulted from distributed inputs without adopting the techniques that would lead to post-project sustainability of these changes (this is particularly the case with vegetable seeds and planting materials). A key limiting factor in this regard is the degree of market orientation of beneficiaries, with the majority instead focusing their limited financial resources on alternative livelihood activities. Meanwhile, the majority of participating WSHGs reported the continued use of solar dryers. However, they expressed Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 13 challenges in creating market linkages and hesitated to invest their own funds in maintenance and repair of the units in order to keep them functional following project closeout. Pilot institutional linkages with SNPs also met with mixed success, with SPN administrators reporting strong rejection of dried fish powder consumption by children and lack of continuity of the programs due to cost and authorization associated with the use of fish and fish products. The SBCC effort did successfully build on the pre-existing high level of consumption of small fish and induced beneficiaries to shift consumption to whole small fish, continuing this practice following the project close out in 2021. Lessons learned from these conclusions suggest that tailoring beneficiary selection criteria by objective can improve the effectiveness and efficiency of certain activities included in subsequent initiatives implemented through the inter-agency linkages built during the project to disseminate its models. First, beneficiary selection needs to consider the willingness of specific beneficiaries to sustain the cost of continued implementation, which in this case largely corresponds to their level of market orientation. Projects that select only those beneficiaries that demonstrate higher levels of market orientation are more likely to engage beneficiaries who will sustain these activities post￾project, which in turn can foster improved sustainability over a wider geographic area. Likewise, the sustainability of solar dryer use is more likely through selection of beneficiary groups (in this case, WSHGs) that understand the market linkages and business models underpinning the sustainable use of these units, especially the cash flow required to maintain and repair them. In a less tangible way, successfully linking to institutional markets requires engaging beneficiary institutions that understand the potential costs and benefits of adopting new products (in this case, for fish and fish products with SNPs), especially in the pilot phase. On the other hand, the successful implementation of SBCC around whole fresh and dried fish consumption built on pre-existing consumption patterns and demonstrates that new behaviors with no cost to beneficiaries can be promoted with success by targeting broad sectors of society, a strategy requiring fewer resources than generalized input distributions. Commercial technology adoption can be enhanced by focusing on commercial market linkages and business plans. Building on the previous lesson regarding limited market sophistication of project beneficiaries, projects can improve sustainability by including a focus on business planning and market linkages. In the case of mola-carp polyculture, this means developing and disseminating models to demonstrate the improved income from investing in fish farms, including how to optimally market increased surpluses into existing market systems. In the case of solar dryers, WSHGs operating these units would be more likely to invest in needed maintenance and repairs if they better understood the business models underpinning their use and how to leverage existing markets for sale of increased outputs. Direct implementation can undermine sustainability. The subset of project activities to increase availability and accessibility of fish and fish products generally relied on project partners to deliver training and post-project TA. However, many beneficiaries reported limited access to post￾project TA due to the limited capacity of partner staff, such as F&ARD extension officers. Interviews with field staff at these institutions confirmed their limited capacity. As an alternative, numerous studies have shown that projects have leveraged ongoing post-project TA from embedded market actors, such as buyers and input providers, with a commercial interest in their customers and suppliers. These studies have also stressed that “timebound” methodologies, such as a training event, do not provide farmers with on-demand access to required technical assistance as needs arise; yet, this is often critical for beneficiaries for whom years may elapse between a training event and the need for specific assistance. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 14 Changes in consumption are possible, but local tastes need to be considered carefully. The success of the SBCC effort to increase consumption of whole fresh and dried small fish demonstrates that effectiveness of building on existing consumption habits. However, children’s strong rejection of dried fish powder demonstrates that efforts to influence changes in consumption must carefully consider local tastes. Leveraging the convergence of agency objectives and activities can facilitate effective and efficient project implementation. The project’s effective and efficient implementation was facilitated by leveraging the convergence of pre-existing agency objectives and activities of project partners. This included the rapid scale-up of mola-carp polyculture promotion and solar dryer use by building on the existing MOU between WorldFish and the Department of F&ARD for implementation of the Odisha Fisheries Policy. Additionally, the pilot implementation of institutional linkages with SNPs at Anganwadi centers and schools leveraged the pre-existing ICDS implemented by the Department of WCD&MS. Finally, WorldFish built on the pre-existing objectives of multiple agency partners to implement the successful SBCC to increase consumption of whole small fish. RECOMMENDATIONS Tailor Beneficiary Selection. Avoid generalized (or geographically defined) distributions and consider ways of integrating beneficiary self-identification into activities that will require beneficiaries to invest their own resources after post closeout. This may include the following mechanisms: • Require a co-pay from beneficiaries of distributed inputs. • Require beneficiaries to work with pre-approved advisors to develop a business plan and/or plan for market linkages in order to qualify as recipients of inputs or funds. In addition to tailoring beneficiary selection to more market-oriented individuals, this approach may present an opportunity for initial provision of TA, as well as introducing beneficiaries to potential sources of post-project TA. • Avoid direct implementation of input distributions and instead provide qualifying beneficiaries with vouchers that are redeemable at pre-qualified commercial suppliers. This approach may have the additional benefit of establishing market linkages between fish farmers and input vendors who may also present sustainable sources of inputs and TA following project closeout. Finally, if coupled with project-supplied training of suppliers, this approach may promote market linkage development for recommended inputs. Integrate Support for Business Plans and Market Linkages. Through the preceding recommendation or additional adjunct activities, consider ways to integrate beneficiary business planning and market linkage expansion into activities. This may include: • Linking beneficiaries to advisory services through vouchers for business and marketing planning services that can be redeemed at pre-approved suppliers. These suppliers may be provided by projects which offer specialized training in related business and marketing activities. • Assessing market opportunities to determine where value-added activities can improve the profitability of product sales and developing relevant sales strategies. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 15 • Working with input suppliers and buyers to develop mechanisms for forward investment into business activities. This may include quasi-contract farming mechanisms through which suppliers and buyers provide inputs on credit against future purchases from beneficiaries. Prioritize a Strategy of Leveraging Market Systems Versus Direct Implementation. Consider ways to integrate input suppliers, buyers, and other upstream value chain actors into projects as sustainable commercially interested TA sources. This could include focusing on addressing weaknesses in the supply and input chains that impede the adoption of improved production and processing practices and technologies through also addressing linkage to feed seed and feed suppliers (and other inputs), including wholesale suppliers. Continue Behavior Change Efforts around Consumption Habits. Continue to implement SBCC activities around fish and fish product consumption by integrating these into the existing activities and mechanisms of partner entities. However, carefully consider ways to build on local tastes and consumption patterns based on established local diets and products. Also, carefully assess the cost implications of new practices, especially linkages with institutional markets. Leverage Convergence of Agency Objectives. Continue to leverage the convergence of agency objectives and activities to facilitate effective and efficient project implementation. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 16 1. PROJECT OVERVIEW USAID/India has a long history of support for food and nutritional security and adaptation programs. These have focused on sharing and transferring innovative, cost-effective solutions to farming challenges globally, in partnership with the Government of India (GOI), civil society organizations, and the private sector. In 2017, in response to the need to improve food and nutrition security in the Indian state of Odisha, USAID partnered with Public International Organization (PIO) WorldFish to implement the Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha project. WorldFish is one of 15 Consultative Group of International Agricultural Research (CGIAR) centers and implements the Aquaculture CGIAR Research Program. WorldFish is headquartered in Penang, Malaysia, but implemented the project through its facilities in Odisha. 1.1 PROJECT BACKGROUND In Odisha, WorldFish is embedded with the State Government of Odisha Department of Fisheries and Animal Resources (F&ARD) through a six-year memorandum of agreement, signed in July 2016, to provide technical assistance (TA) for implementation of the Odisha Fisheries Policy. This policy focuses on sustainable fish production systems and technologies.1 In October 2017, USAID provided WorldFish additional funding through the Indian Partnership Program (IPP) Annual Program Statement (APS) to demonstrate approaches for nutrition-sensitive technologies, including hygienic solar drying of marine small fish and use of community and village bodies of water for rearing small fish across three to four districts of Odisha. USAID/India funding was $1.5 million.2 In August 2019, USAID was forced to delay a final tranche of funding due to budget limitations, and WorldFish scaled down some activities. This funding was restored in 2020.3 Originally a three-year project, on September 30, 2020, USAID/India approved a six-month no cost extension to March 31, 2021, allowing WorldFish to resume activities affected by the budget delay and COVID-19 restrictions.4 1.2 PROJECT PARTNERS In addition to WorldFish and the State Government of Odisha Department of F&ARD, key project partners included the State Government of Odisha Department of Women and Child Development and Mission Shakti (WCD&MS), which manages supplemental nutrition programs (SNPs) through Integrated Child Development Services (ICDS), a program GOI launched throughout India in 1975 to provide nutritional meals, preschool education, and other services to children under six and their mothers. WCD&MS implements ICDS in Odisha through schools and a network of Anganwadi centers, rural childcare centers also established by the GOI in 1975 to combat malnutrition though provision of nutritious meals to children under six. WCD&MS also sponsors women’s self-help groups (WSHGs), which implement internal lending programs through a village savings and loan approach to foster economic development. Other partners included the State Micro, Small, & Medium Enterprises (MSME) Department, the Indian Council of Agricultural Research-Central Institute of Fisheries Technology (ICAR-CIFT) based in Cochin, Kerala, and Odisha Livelihood 1 Locally Led Development: Partnering for Improved Nutrition - Lessons from Odisha, India. JSI Research & Training Institute, Inc. for USAID Feed the Future, August 2021. 2 Hamzaoui, Mustapha. Project Concept Note on “Scaling innovative and nutrition-sensitive aquaculture and wild fisheries production technologies through partnership in the Indian states of Odisha and Assam,” submitted by WorldFish in February 2017 in response to India Partnership APS. USAID for WorldFish, April 3, 2017. 3 Financial Report – Period from 01/10/2017 to 31/03/2021. WorldFish for USAID/India, April 14, 2021. 4 Reddy, Vamsidhar. Action Memo: Approval to extend the period of performance of the Scaling nutrition sensitive fisheries technologies and integrated approaches through partnership in Odisha project. PIO Grant No. AID-BFS-IO-17-00005 by six months from September 2020 to March 31, 2021. USAID, July 15. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 17 Mission (OLM).5 MS Swaminathan Research Foundation (MSSRF) conducted the baseline study and provided ongoing project monitoring services. 1.3 PROJECT GOAL AND OBJECTIVES The USAID-funded project contributed to USAID/India’s then-existing Country Development Cooperation Strategy (CDCS) Sub-Goal 2: “Innovations accelerate development outcomes in India and globally.” The project also aimed to advance the USAID/India objective of “CDCS role in contributing to global development enhanced” and IR 4.1, “Indian innovations for development impact shared with other countries.”6 The stated goal of the project was to “improve food and nutrition security in the Indian state of Odisha.” As stated in project reports, “this project will lead to the introduction of nutrition-sensitive production technologies for nutrient-rich fish and vegetables in selected districts of Odisha as well as increased production of high quality fresh small fish and dried fish for making fish-based products.”7 The project aimed to accomplish this goal through the following two strategies: 1. Introduce nutrition-sensitive production technologies and integrated approaches, including combining fish and vegetables within farming systems in the Indian state of Odisha, with a focus on children’s nutrition (adoption of hygienic solar drying was added later). 2. Establish strengthened and expanded partnerships with a range of partners including local government, private sector, research institutions, NGOs, and CBOs, to increase the sustainability of interventions after completion of the project and foster scaling. The proposed project was constructed around an overarching theory of change, which envisioned deploying the two strategies to implement four integrated components (and corresponding objectives): 1) increased availability; 2) increased accessibility; 3) increased consumption of nutritious foods; and 4) scaling regional and national. Furthermore, each component included several intermediate results (IRs), depicted in the orange boxes in Figure 1 below.8 5 Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha - Project Brief. WorldFish, April 2020. 6 Scaling innovative, nutrition-sensitive fisheries technologies and integrated approaches through partnerships in Odisha, India can improve food and nutrition security – A project proposal. WorldFish submitted to USAID, India, May 2017. 7 Annual Report: Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha: USAID IPP Funded Project (Oct 2017-Sept 2018). WorldFish for USAID/India, November 9, 2018. 8 Scaling innovative, nutrition-sensitive fisheries technologies and integrated approaches through partnerships in Odisha, India can improve food and nutrition security – A project proposal. WorldFish submitted to USAID, India, May 2017. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 18 Figure 1: Project Theory of Change Source: Scaling innovative, nutrition-sensitive fisheries technologies and integrated approaches through partnerships in Odisha, India can improve food and nutrition security - A project proposal. WorldFish for USAID/India, May 2017. As expressed in project literature, the two strategies would be deployed to implement the four components supporting the project goal in an integrated fashion as illustrated in the following table.9 Table 1: Project Components and Description COMPONENT DESCRIPTION Increased Availability Increased production, productivity, and diversity of nutrient-rich fish from innovative technologies of aquaculture and capture fisheries, as applied to household and community ponds, and inland freshwater reservoirs. Increased Accessibility Development, marketing and distribution of affordable, safe, nutritious fresh fish and fish-based products using production from aquaculture and capture fisheries, including Chilika Lake. Increased and expanded distribution to reach multiple population groups, in partnerships with the private sector and research institutes. Increased Consumption Social behavior-change approaches, targeting the whole family, to promote increased nutrient- rich fish and vegetable consumption, especially in women and young children; and increase knowledge and practice of essential nutrition and essential hygiene actions, in partnerships with relevant local government programs, NGOs and CBOs. Improved Nutrition and Scaling Further scaling of innovative, nutrition-sensitive fisheries technologies and integrated approaches, through expanding and strengthening regional and national partnerships and collaborations in West Bengal and Terai, Nepal. Source: Annual Report: Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha: USAID IPP Funded Project (Oct 2017-Sept 2018). WorldFish for USAID/India, Penang, Malaysia, November 9, 2018. 9 Annual Report: Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha: USAID IPP Funded Project (Oct 2017-Sept 2018). WorldFish for USAID/India, Penang, Malaysia, November 9, 2018. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 19 In operational terms, as initially proposed, WorldFish aimed to implement these four integrated components of the project through multiple corresponding activities (or “interventions”), each with discrete targets, as shown in the following table.10 Table 2: Proposed Project Components and Activities COMPONENTS AND ACTIVITIES 1. Increased Availability • 450 households practicing carp-mola polyculture in homestead ponds. • Local community organizations begin culturing mola and other small fish in six community ponds/reservoirs. • Micronutrient-rich vegetables and orange sweet potato produced in at least 85% of households practicing carp￾mola pond polyculture. 2. Increased Accessibility • Five branded, nutritious, safe, fresh fish and fish-based products available in rural and urban markets. • 20 women’s groups producing high quality dried fish from Chilika Lake and reservoirs. 3. Increased Consumption of Nutritious Foods • Quantity and frequency of small fish intake increased in the diet of at least 5,000 children and 5,000 women. • 25 schools with fish in midday meals. • 25 Anganwadi centers distributing fish products for the first 1,000 days of life. 4. Improved Food and Nutrition Security in Odisha and Scaling • Two states in India and two countries in the region adopt nutrition-sensitive technologies and integrated approaches in the fisheries sector. Source: Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha – Fact Sheet. WorldFish. 10 Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha – Fact Sheet. WorldFish. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 20 1.4 PROJECT GEOGRAPHIC FOCUS Table 3: Project Zone - Districts and Blocks Following award of the USAID/India-funded project in 2017, WorldFish and the Government of Odisha Department of F&ARD identified six blocks in three districts to serve as mola-carp polyculture demonstration sites (two neighboring blocks per target district located adjacent to ongoing F&ARD project pilot blocks). In Cuttack district, an additional pond belonging to a government school was selected for carp-mola polyculture demonstration. Subsequently, the partners carried out a sensitization program in 21 neighboring villages in 14 Gram Panchayats (village clusters) in each target block and identified 345 households with backyard ponds and six community ponds at the village cluster level to serve as on-site demonstration sites.11 In addition, WorldFish and F&ARD identified ten coastal sites for construction of demonstration solar dryers for marine fish, including two each in the USAID-funded districts of Baleshwar and Jagatsinghpur, as well as two each the districts of Bhadrak and Puri, and one each in Khurda (Khorda) and Ganjam districts. 11 Annual Report: Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha: USAID IPP Funded Project (Oct 2017-Sept 2018). WorldFish for USAID/India, Penang, Malaysia, November 9, 2018. DISTRICT BLOCKS 1. Baleshwar 1. Khaira 2. Soro 2. Mayurbhanj 3. Gopabandhu (GB) Nagar 4. Kaptipada/a 5. Khunta 3. Jagatsinghpur 6. Jagatsinghpur 7. Naugaon 4. Cuttack 7. Cuttack Sadarb 5. Puri 9. Kanas/a a. Added to project zone in 2018. b. Included due to proximity to F&ARD site. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 21 Figure 2: Project Zone Map Source: Annual Report: Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha: USAID IPP Funded Project (Oct 2017-Sept 2018). WorldFish for USAID/India, Penang, Malaysia, November 9, 2018. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 22 2. EVALUATION OVERVIEW Through the Collaborating, Learning and Adapting in India Mechanism (CLAIM) Activity, USAID/India contracted Panagora Group to provide monitoring, evaluation, and learning services over a period of five years. In 2022, through the CLAIM Activity, USAID India asked Panagora to conduct this ex-post facto evaluation of the “Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha” project, which ended in March 2021 (see Annex 1: Scope of Work). 2.1 EVALUATION PURPOSE The purpose of this evaluation of the Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha project was to assess its sustainability, and to learn how outcomes and impacts evolved after completion of the project. Evaluation findings and conclusions will also be used to guide future programs in similar sectors. 2.2 EVALUATION TIMELINE Between November 21, 2022–March 31, 2023, Panagora Group deployed a three-person evaluation team to conduct an evaluation of the Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha project, which ended on March 21, 2021. Remote fieldwork with key informants outside of the project zone took place over a period of four weeks in December 2022–January 2023. Subsequently, the team undertook fieldwork in Odisha over a period of four weeks between January–February 2023, focusing on the geographic zones where implementing partners used USAID funding to implement activities, which included the districts of Baleshwar, Jagatsinghpur and Mayurbhanj, as well as limited activities involving solar dryers in the districts of Bhadrak, Cuttack, Ganjam, and Puri. Data analysis and reporting continued until March 31, 2023 (see Annex 2: Evaluation Timeline). 2.3 EVALUATION QUESTIONS The evaluation focused on five topics and corresponding evaluation questions (EQs), as follows. Table 4: Evaluation Questions by Topic TOPIC EVALUATION QUESTION Project Components 1. To what extent and how did the project components reach their objectives during implementation? Intervention Success Factors 2. What factors hindered interventions, and what factors contributed to their success? Continued Use and Hindering Factors 3. Are project beneficiaries and stakeholders continuing to use promoted behaviors related to use of innovative, nutrition-sensitive fisheries technologies and integrated approaches to improve food and nutrition security? If so, how? If not, what factors hinder continuation of these practices? Continued Implementation 4. Have the implementing partners/stakeholders strengthened by USAID funding continued to implement project components? What is stakeholder perspective/feedback about project impact? Sustainability Factors 5. What are the factors that contributed to or impaired the long-term sustainability of project outcomes and outputs? Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 23 2.4 EVALUATION TEAM The evaluation team included three members (see Annex 3: Evaluation Team): • Team Leader: The Team Leader was responsible for overall design and implementation of the evaluation and ensuring that all tasks and deliverables were achieved on time and of high quality. He led the literature review and gap analysis and managed other data collection activities, including remote key informant interviews (KIIs), as well as KIIs and focus group discussions (FGDs) in Odisha, and supervised the mini survey and site visits. The Team Leader led data analysis with input from the Agriculture/Evaluation Specialist, led the formulation of conclusions, lessons learned, and recommendations based on findings, and led development of the initial presentation of findings and draft and final evaluation reports. • Agriculture/Evaluation Specialist: The Agriculture/Evaluation Specialist participated in the literature review and gap analysis and led the development of the plan for field-based KIIs and FGDs under the supervision of the Team Leader. In coordination with the Team Leader, he also led implementation of the mini survey and site visits. The Agriculture/Evaluation Specialist provided input, including contextual background information, into all data analysis activities, and into the initial presentation of findings and draft and final evaluation reports. • Logistics Coordinator: Under the direction of the Team Leader, the Logistics Coordinator led the scheduling of data collection activities, including the remote KIIs, and supported the logistics of the evaluation team, including lodging and transport to data collection sites in and around Odisha. Four translators also supported the team in data collection (two Oriya and Santal speakers in Baleshwar and Mayurbhanj, and two Oriya speakers based in Bhubaneshwar), as well as two drivers. 2.5 DATA COLLECTION The evaluation team collected data through the following means: 1) an initial literature review and gap analysis; 2) KIIs, and FGDs with beneficiary groups; 3) a quantitative “mini survey” of beneficiaries; and 4) site visits to project-related sites. LITERATURE REVIEW AND GAP ANALYSIS From November 21–December 7, 2022, the evaluation team conducted a literature review, assessing available project documents to understand the context and underlying concept of the project, as well as to understand how WorldFish and partners implemented the project. Documents reviewed included the project proposal, Monitoring, Evaluation and Learning Plan, Action Plan for Phase 1, three Annual Reports, the Fact Sheet and Project Brief, the USAID Action Memo for the World Fish extension, and others. The team also reviewed background research documents on topics related to project themes and context (see Annex 4: Reference Citations). Based on this preliminary review, the team produced a literature review gap analysis to verify their initial Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 24 understanding of the project and identify areas for further document collection and research during subsequent data collection phases of the evaluation (see Annex 5: Gap Analysis).12 KEY INFORMANT INTERVIEWS AND FOCUS GROUP DISCUSSIONS Key Informant Interviews. The purpose of the KIIs was to probe results of the literature review and gap analysis for findings related to the EQs. KIIs consisted of in-depth facilitated discussions with individuals or small functional groups of related individuals (up to three participants) using a semi￾structured “evolving subject-driven” approach. KII participants were selected according to the likelihood of significant knowledge of project activities and convenience of access to enable the team to access the largest number of informants possible over the course of data collection within the limited time and personnel resources available. The team used an interview guide and an iterative process to assemble information across successive interviews so that it can be aggregated and analyzed in a cohesive and consistent manner.13 During literature review, the team identified three stakeholder groups as potential informants, as illustrated in Table 5. Table 5: Project Stakeholder Groups STAKEHOLDER GROUP DESCRIPTON Donor Staff • USAID/India Mission staff Remote and Odisha-based Implementing Partner Staff • WorldFish • Government of Odisha Department of Food and Animal Resource Development (F&ARD) • Government of Odisha Department of Women and Child Development and Mission Shakti (WCD&MS) (including Anganwadi center staff) • Indian Council of Agricultural Research-Central Institute of Fisheries Technology (ICAR-CIFT) • Micro, Small, & Medium Enterprises (MSME) Department • Odisha Livelihood Mission Project Beneficiaries • Farmers practicing carp-mola polyculture and vegetable production on household fishponds. • Farmers practicing carp-mola polyculture on community fishponds. • Women’s self-help group members involved in solar fish drying and marketing. • Recipients of dried fish at SNPs in Anganwadi centers. • Recipients of dried fish powder at SNPs in schools. Following document review and gap analysis, the team developed a provisional list of respondents, which was expanded following a kick-off meeting with USAID/India. Also, based on the literature review and gap analysis, the team developed data collection protocols for the KIIs (see Annex 6: Data Collection Protocols). Subsequently, congruently with the quantitative mini survey, the team implemented the KIIs in two overlapping phases, as follows: • Initial remote KIIs with donor staff and implementing partner staff, which included requests for additional data and contacts to continue identifying potential informants through a “snowball” approach. 12 USAID/India informed Panagora that no external mid-term or final evaluation of the project was implemented. Therefore, the gap analysis was critical to the final design of data collection tools, ensuring that data collected presented a holistic picture of the project design and implementation. 13 King, Gary, Robert Keohane, and Sydney Verba. Designing Social Inquiry: Scientific Inference in Qualitative Research’ Princeton University Press. Princeton University Press, 2016. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 25 • In-person, on-site KIIs with Odisha-based implementing partner staff and project beneficiaries, which also included requests for additional contacts to continue identifying informants. In cases where respondents were not fluent in English or Hindi, translators provided Oriya- and Santal-to-English translation. Over the course of data collection, the team leader and agriculture/evaluation specialist jointly implemented 56 KIIs with purposively selected samples of the three project stakeholder groups. These included two donor staff and 13 implementing partner staff (see Annex 7: KII Contact List), as well the former Maharaja of Mayurbhanj.14 The KIIs also included 40 mola-carp polyculture beneficiaries operating household ponds. To the degree possible, the KIIs reflected the diversity of the beneficiary population (e.g., gender, caste, and ethnic minorities such as tribal affiliations). Focus Group Discussions. As with the KIIs, the purpose of the FGDs was to better understand topics related to the EQs. The FGDs consisted of semi-structured moderated discussions with groups of approximately four to ten individuals. Purposive sampling was used to select participants according to project knowledge and experience, as well as convenience in access to ensure inclusion of as many participants as possible for the evaluation. As with the KIIs, the evaluation team facilitated the FGDs in a semi-structured format, using a discussion guide that was refined following initial (Phase 1) interviews (see Annex 6: Data Collection Protocols). Facilitators also used probing questions related to topics and emerging findings from previous KIIs and FGDs and from the concurrent ongoing mini survey analysis. To the degree possible, the team held FGDs in settings where participants feel comfortable, such as near homes and worksites, and provided refreshments for participants. As possible, the FGDs also reflected the diversity of the project’s beneficiary population. During Phase 2 of the KIIs, during on-site fieldwork in Odisha, the evaluation team conducted 18 FGDs with approximately 118 project beneficiaries stratified to reflect beneficiary types and geographic diversity across the project zones. These included 13 FGDs with approximately 78 community fishpond farmers, and five FGDs with approximately 40 WSHG members using project￾supplied solar dryers in the districts of Baleshwar, Bhadrak, Jagatsinghpur and Mayurbhanj (see Annex 8: FGD Summary). QUANTITATIVE MINI SURVEY To complement qualitative data collection, the evaluation team conducted a quantitative “mini￾survey” of household and community fishpond farmer project beneficiaries, stratified by project district. The survey was “mini” in that the sample size was limited to an illustrative, non-statistically significant sample. Thus, analysis of survey results was limited to simple tabulation, frequencies, and cross tabulations, without the use of descriptive or inferential statistics. Survey topics focused on perceived degree of changes in practices before and since the project ended, as well as on enabling and hindering factors related to project activities. The survey contained closed-ended questions related to individuals’ experience with specific project activities and used a Likert scale of 0-5, with “1” indicating a low score and “5” a high score (“0” indicated “not exposed to the activity”). The survey also gathered a limited amount of basic data related to household and farm characteristics, and previous experience (see Annex 6: Data Collection Protocols). 14 Contact data related to project beneficiaries interviewed was redacted form this contact list to protect informant anonymity in line with Panagora Group’s ERB practices. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 26 The Oriya- and Santal-speaking translators implemented the survey immediately following KIIs and FGDs. In total, the evaluation team collected mini survey data from 97 household fishpond farmer beneficiaries in the districts of Baleshwar, Jagatsinghpur, and Mayurbhanj, representing approximately 12 percent of the total household fishpond farmer beneficiary population of 789.15 To the degree possible, the survey selection process also reflected the diversity of the project’s beneficiary population (see Annex 9: Mini Survey Data Set). SITE VISITS AND DIRECT OBSERVATIONS The evaluation team visited project sites to better understand the context and experiences of project beneficiaries, assess enabling and hindering factors related to project activities, continued use of practices and sustainability, and to identify lessons learned related to the EQs. The site visits consisted of a short tour of a purposively selected sample of project-related sites and were conducted together with translators and beneficiaries to facilitate probing questions. Purposive sampling was used to identify sites on the same basis as that of KIIs and FGDs (accounting for likelihood of significant knowledge of beneficiaries present and accessibility). The team used data obtained from the site visits to inform and support emerging findings and conclusions and provide illustrative cases studies. During site visits, the team collected data using a structured site visit guide finalized following Phase 1 KIIs (see Annex 6: Data Collection Protocols). Table 6: Site Visits by Type SITE TYPE NO. Household fishpond 40 Community fishpond 16 Solar dryer 5 Anganwadi center with SNP 2 School with SNP 3 Markets 6 Mola stocking pond 1 The team leader and agriculture/evaluation specialist visited 73 project-related sites, including 40 household fishponds and 16 community fishponds, five solar dryer sites operated by WSHG members, two Anganwadi centers, three schools that provided fish products, and six markets where fish products are sold. The team also visited the mola stocking pond in Udala block in Mayurbhanj district, where seed fish was procured to supply seed distributions. GEOGRAPHIC DISTRIBUTION OF DATA COLLECTION The evaluation team conducted Phase 1 KIIs remotely with donor and implementing partner staff, as well as about 40 percent of Odisha-based partners. Phase 2 on-site data collection was distributed between all blocks in the districts where USAID-funded activities occurred according to density of beneficiary populations, or approximately 30 percent in Baleshwar, 40 percent in Mayurbhanj, and 30 percent in Jagatsinghpur districts (see Table 7). The team also visited and/or contacted WSHGs operating project-supplied solar dryers in the districts of Bhadrak, Cuttack, Ganjam, and Puri. 15 Ratha, B.C., P.A. Padiyar, N. Shenoy, C.V. Mohan Scaling nutrition-sensitive fisheries technologies and integrated approaches through partnership in Odisha: USAID-IPP Funded Project (Oct 2017–Mar 2021). Project Final Report for the period October 2017 to March 2021. WorldFish for USAID, April 2021. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 27 Table 7: Geographic Distribution of Data Collection Activities RESPONDENT/ZONE/A REMOTE BALASWAR MAYURBHANJ JAGATSINGHPUR Key Informant Interviews 1. Donor Staff 100% 0% 0% 0% 2. Remote Implementing Partner Staff 100% 0% 0% 0% 3. Odisha -Based Implementing Partner Staff 40% 20% 20% 20% 4. Project Beneficiaries 0% 30% 40% 30% Focus Group Discussions and Mini-Survey Project Beneficiaries 0% 30% 40% 30% Site Visits Project-related Sites 0% 30% 40% 30% a. The team also visited and/or contacted WSHGs operating project-supplied solar dryers in the districts of Bhadrak, Cuttack, Ganjam, and Puri. 2.6 ETHICAL CONSIDERATIONS Prior to Phase 2 (on-site) data collection, Panagora Group’s Ethical Review Board (ERB) reviewed and approved all data collection instruments related to project beneficiaries in line with its Panagora Group Human Research Policy.16 Panagora Group developed this policy for human research in compliance with U.S. federal requirements and to advance the ethical standard and rigor of Panagora research and evaluations. Phase 2 data collection commenced after ERB approval was obtained. All Panagora evaluation personnel, including headquarters, staff, and in-country and external consultants, also followed Panagora professional and ethical guidelines to ensure this evaluation was carried out with honesty and integrity, respondents were protected, and data security was ensured. Prior to data collection, the evaluation team leader held a team meeting to review ethical standards laid out in the ERB policy and ensure compliance related to interviewing techniques, including standards regarding probing questions and engagement with vulnerable populations. This included ensuring that respondent confidentiality and privacy was protected during data collection by obtaining informed consent to collect data and informing respondents about potential uses and any potential sharing or publication of data. 2.7 DATA ANALYSIS METHODS Throughout KIIs and FGDs, interviewers transcribed key notes into MS Word doc-based forms in real time to identify emerging trends to aggregate findings around common themes and generate further probing questions for subsequent KIIs, FGDs and site visits. As needed to ensure accuracy, the interviewers also recorded data directly into audio recording software. These recordings were destroyed following transcription to ensure data privacy. For the quantitative mini survey, the agriculture/evaluation specialist input data into a database and conducted basic analysis throughout the data collection period to identify emerging trends, such as frequency distribution and sub-group comparison via cross-tabulation. No statistical analyses were conducted. Upon completion of the 16 Panagora Group Human Research Policy. Panagora Group, June 9, 2022. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 28 mini survey, the agriculture/ evaluation specialist aggregated data into an Excel-based spreadsheet and presented the data with a series of visualizations determined in coordination with the Team Leader. Data analysis methods used by the team over the course of this evaluation included: • Content Analysis—Content analysis entailed the evaluation team’s intensive review of KII and FGD data to identify and highlight notable implementing partner and beneficiary perspectives on project design and components, enabling or hindering factors related to specific project activities, and post-project sustainability. • Trend Analysis—Trend analysis enabled the evaluation team to further examine project progress toward targets over time, and factors related to continuance and sustainability since project close-out, to identify anticipated impacts and outcomes and how specific exogenous and endogenous events may have contributed to these outcomes. • Triangulation—Subsequent to fieldwork, triangulation enabled the evaluation team to cross-verify and cross-validate findings that emerged from distinct data sources. Methodological triangulation also enabled the evaluation team to strengthen potential linkages and accuracy of data in cases where results obtained through one method were less conclusive than another method. Data analysis was conducted throughout the course of this evaluation to identify initial findings and conclusions for a consultative presentation (out-briefing) with USAID/India following fieldwork. Based on feedback during the presentation, analysis continued until submission of the draft evaluation report. Following submission of the final evaluation report, all interview transcripts, as well as survey datasets collected by the evaluation team, will be scrubbed of identifying text to protect respondent confidentiality. 2.8 LIMITATIONS AND POTENTIAL BIASES The methodology used in this evaluation had a number of potential biases and limitations. These, and the steps the evaluation team took to mitigate them, included: • Positive response (“halo”) bias: Probing questions regarding livelihood and development outcomes may result in positive response bias, the tendency of respondents to subjectively focus on positive outcomes. To the degree possible, the evaluation team mitigated this bias by probing for both successes and challenges to develop the most holistic picture possible of project achievements and challenges relative to the EQs. The team also triangulated responses against data collected from the literature review, including documents produced by the implementing partners. • Selection bias: Selection bias is an inherent risk when implementers help to facilitate contact with members of some stakeholder groups. The team worked closely with USAID and WorldFish staff to organize KIIs and FGDs and with project stakeholders and beneficiaries. However there remains a risk that implementing partner staff selected the most active, responsive, or engaged individuals, meaning that the team only heard from informants that were likely to report positive experiences. To mitigate the risk of selection bias, prior to launching on-site data collection, the team requested that donor and implementing partner staff provide a universal list of stakeholders in advance. Subsequently, the team identified individuals from this list to contact for interviews. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 29 • Sampling limitations: Due to time and resource constraints, the evaluation team was only able to conduct KIIs and FGDs with a selected sample of each stakeholder group. In particular, the mini survey sample contained a non-statistically significant set of respondents. To mitigate potential biases, the team worked closely with USAID and implementing partner staff to identify a reasonably representative sample set of interviewees from each stakeholder group and also stratified data collection according to beneficiary population distribution over the project zone. • Subjective measurements: Qualitative approaches can result in project analysis rooted in professional opinions and experience of the evaluation team, which will inevitably carry some bias. To mitigate this, the team triangulated findings across stakeholder groups and methods and relied on data to draw evidence-based conclusions and recommendations. In addition, where possible the team sought out the professional opinions of relevant personnel to corroborate and review findings and conclusions to improve their accuracy and soundness. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 30 3. FINDINGS AND CONCLUSIONS Over the course of the project’s implementation, adaptive management resulted in significant evolution of the key project activities and related targets. By the time of publication of the Year 3 annual report, WorldFish had amended and updated the performance reporting format as illustrated in Table 8 below. Results presented in this report match those reported in the project final report published in April 2021 following project closeout on March 31, 2021.17 Table 8: Reported Project Components and Key Activities with Targets and Results COMPONENTS AND ACTIVITIES TARGET RESULT RATEa Objective 1: Increased Availability No. of households practicing carp-mola polyculture (including vegetable and orange sweet potato) 2,000 789 39.5% No. of community ponds stocking mola and other 24 22 91.6% Objective 2: Increased Accessibility No. of women’s groups engaged in fish drying 40 10 25.0%b No. of Anganwadi centers using fish products in mother and child health care (ICDS) 50 50 100% No. of schools using fish in mid-day meal program 50 24 48.0% Objective 3: Increased Consumption of Nutritious Foods SBCC materials 0 19 n/a Objective 4: Improved Food and Nutrition Security in Odisha and Scaling MOUs 0 5 n/a Partnerships 0 11 n/a Policy influence 0 4 n/a Cross-country visits 0 3 n/a Training to other government functionaries 0 3,120 n/a a. Calculated by the evaluation team. Activities that lacked targets are labeled “n/a.” b. WorldFish-ICAR-CIFT MOU states a target of 10 WSHGs, which is also reported in project documentation. Source: Ratha, B.C., P.A. Padiyar, N. Shenoy, C.V. Mohan. Scaling nutrition-sensitive fisheries technologies and integrated approaches through partnership in Odisha: USAID-IPP Funded Project (Oct 2017—Mar 2021). Project Final Report for the period October 2017 to March 2021. WorldFish for USAID, April 2021. EQ 1: PROJECT COMPONENTS - FINDINGS This section presents findings related to EQ 1: To what extent and how did the project components reach their objectives during implementation? The team focused on the evolution of project strategies, 17 Ratha, B.C., P.A. Padiyar, N. Shenoy, C.V. Mohan. Scaling nutrition-sensitive fisheries technologies and integrated approaches through partnership in Odisha: USAID-IPP Funded Project (Oct 2017—Mar 2021). Project Final Report for the period October 2017 to March 2021. WorldFish for USAID, April 2021. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 31 components and objectives, and activities and targets, setting the stage for findings related to post￾project sustainability of outcomes and impacts after completion of the project as described in subsequent sections of this report. Specifically, this section references the reporting table above adopted by WorldFish in the second annual report and subsequently utilized. INCREASED AVAILABILITY The project promoted mola-carp polyculture to “increase the availability of small fish (e.g., mola) among backward and vulnerable communities in target districts through increased production, productivity and diversity of nutrient-rich fish from innovative technologies of aquaculture and capture fisheries, as applied to household and community ponds, and inland freshwater reservoirs.”18 To achieve this, block-level F&ARD field staff identified beneficiary households and self-help groups (SHGs) in each village cluster on a rolling basis based on field staff’s pre-existing knowledge of farmer “motivation.”19 Subsequently, the partners and other experts held “mass/group awareness” meetings (trainings) with households and SHG representatives to discuss topics including “pre-stocking pond management, fish seed selection, feed and health management practices for sustainable production of carp and mola, and nutritional education.”20 Beneficiaries described the trainings as taking place in nearby auditoriums and sports facilities and attended by approximately 40 individuals.21 Subsequently, through a one-time free distribution, the project distributed mola and carp seed, initial feed supplies, and 50 percent of operational set-up costs to individual (household) farmers and 60 percent to women managing community ponds. The Government of Odisha Department of WCD&MS also aimed to provide all beneficiaries with seasonal vegetable plants and kitchen garden kits to promote vegetable gardening and provide additional income and nutrition to community members. To this end, WorldFish, in collaboration with the ICAR-Central Tuber Crops Research Institute, also selected orange sweet potato (OSP) as a complementary crop with high nutritional value and market potential.22 The final project annual report states that by March 2021, the project had piloted carp-mola polyculture in 789 household ponds and in 22 community ponds, benefiting 1,307 households (in addition, 226 beneficiaries surveyed reported harvesting approximately 2,366 kg of OSP in the third year of the project).23 18 Annual Report: Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha: USAID IPP Funded Project (Oct 2017–Sept 2018). WorldFish for USAID/India, Penang, Malaysia, November 9, 2018. 19 Implementing partner KIIs, Baripada, Mayurbhanj, January 20, 2023. 20 Annual Report: Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha: USAID IPP Funded Project (Oct 2017–Sept 2018). WorldFish for USAID/India, Penang, Malaysia, November 9, 2018. 21 Mola-carp polyculture beneficiary KIIs. 22 Annual Report: Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha: USAID IPP Funded Project (Oct 2019–September 2020). WorldFish for USAID India, Penang, Malaysia, October 31, 2020. 23 Ratha, B.C., P.A. Padiyar, N. Shenoy, C.V. Mohan. Scaling nutrition-sensitive fisheries technologies and integrated approaches through partnership in Odisha: USAID-IPP Funded Project (Oct 2017–Mar 2021). Project Final Report for the period October 2017 to March 2021. WorldFish for USAID, April 2021. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 32 Table 9: Evolution of Carp-Mola Polyculture Beneficiaries UNIT PERIOD RESULT TARGET Pond type Oct 2017 to Sept 2018 Oct 2018 to Sept 2019 Oct 2019 to Sept 2020 Oct 2020 to Mar 2021 Cumulative to March 2021 Household 167 588 34 0 789 2,000 Community 1 21 0 0 22 24 Source: Ratha, B.C., P.A. Padiyar, N. Shenoy, C.V. Mohan. Scaling nutrition-sensitive fisheries technologies and integrated approaches through partnership in Odisha: USAID-IPP Funded Project (Oct 2017–Mar 2021). Project Final Report for the period October 2017 to March 2021. WorldFish for USAID, April 2021. These figures exceed the initial proposed targets of 450 households practicing carp-mola polyculture in homestead ponds and six local community organizations beginning to culture mola and other small fish in six community ponds/reservoirs.24 However, they fall short of WorldFish revised targets set in 2018 of 2,000 household ponds and 24 community ponds, reaching 39.5 percent and 91.6 percent of the target, respectively. In KIIs, donor and implementing partner staff described these shortfalls as the result of a beneficiary recruitment drop and limitations on activity implementation following the 2019 budget delay and subsequent COVID-19 restrictions on movement and beneficiary contact.25 INCREASED ACCESSIBILITY Project documentation describes the purpose of this component of the project was “to develop, market and distribute affordable, safe, nutritious fresh fish and fish-based products using production from aquaculture and capture fisheries, including Chilika Lake. Also, to increase and expand the distribution to reach multiple population groups, in partnerships with the private sector and research institutes.”26 WorldFish envisioned accomplishing this objective through creating private sector partnerships to develop new fish-based products well as institutional linkages with the WCD&MS ICDS program SNPs at Anganwadi centers and in schools. Following the project launch in 2017, WorldFish and the Department of F&RD initially surveyed the availability of captured small, indigenous fish species in Odisha, focusing on three sectors: 1) brackish water (in Chilika lake); 2) fresh water; and 3) coastal water, to “investigate the availability of fish in terms of quality and quantity, their usage/disposal methods/patterns, pricing (fresh and dried) and seasonality, and to identify key stakeholders in the supply chain and explore ways to link the producers to aggregators and processors.”27 The survey found “communities practiced sun drying in open spaces resulting in considerable quality loss and little to no knowledge of sanitary practices.”28 These practices render products unattractive to potential partners, and focus on private sector linkages was subsequently reduced.29 24 Scaling innovative, nutrition-sensitive fisheries technologies and integrated approaches through partnerships in Odisha, India can improve food and nutrition security—A project proposal. WorldFish submitted to USAID, India, May 2017. 25 Donor and implementing partner KIIs. 26 Ratha, B.C., P.A. Padiyar, N. Shenoy, C.V. Mohan. Scaling nutrition-sensitive fisheries technologies and integrated approaches through partnership in Odisha: USAID-IPP Funded Project (Oct 2017–Mar 2021). Project Final Report for the period October 2017 to March 2021. WorldFish for USAID, April 2021. 27 Annual Report: Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha: USAID IPP Funded Project (Oct 2017–Sept 2018). WorldFish for USAID/India, November 9, 2018. 28 Ibid. 29 Donor KII. Notably, the proposed project target of “Five branded, nutritious, safe, fresh fish and fish-based products available in rural and urban markets” was removed in the amended 2019 reporting table. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 33 Nonetheless, WorldFish did pursue the institutional linkage approach to improve accessibility, signing signed a memorandum of understanding (MOU) in 2018 with M/S Kalinga Marine Exports in Bhubaneswar to package and process hygienic dried fish for use in SNPs.30 In addition, between October 2018 and August 2019, WorldFish established a pilot program with Falcon Marine Ltd. to supply dried marine fish to 2,336 children at WCD&MS-supported schools, encouraging consumption of whole fish to take advantage of nutritional components found in fish heads. However, in 2019, this pilot was curtailed due to USAID concerns with sustainability and private sector role.31 In response to the product quality and sanitation-related challenges, in 2019 F&ARD, WCD&MS, and OLM identified 10 WSHGs from coastal villages for production of hygienic dried fish using solar dryers and signed an MOU with ICAR-CIFT in Cochin, Kerala to supply and oversee the installation of 10 solar dryers for the production of dried small marine fish. ICAR-CIFT also provided training to the WSHGs on a rolling basis in groups of three (the final close-out report stated that, although solar dryer installation was completed by the project end date, training was delayed due to COVID￾19 restrictions and was only completed by June 2021).32 In addition, in 2021, recognizing the challenge of producing a sanitary fish product using locally sourced fish during the remaining life of the project, under the same MOU WorldFish and WCD&MS sourced fish powder manufactured in Kerala from ICAR-CIFT to implement a six-month pilot feeding program at 50 Anganwadi centers in Kaptipada Block, Mayurbhanj.33 Table 10: Evolution of WSHGs Drying Fish, Anganwadi Centers/Schools Using Fish in SNPs UNIT PERIOD RESULT TARGET Oct 2017 to Sept 2018 Oct 2018 to Sept 2019 Oct 2019 to Sept 2020 Oct 2020 to Mar 2021 Cumulative to March 2021 WSHG 0 0 0 10 10 40a Anganwadi Centers 0 0 0 50 50 50 Schools 3 21 0 0 24 50 a. WorldFish-ICAR-CIFT MOU states a target of 10 WSHGs, which is also reported in project documentation. Source: Ratha, B.C., P.A. Padiyar, N. Shenoy, C.V. Mohan “Scaling nutrition-sensitive fisheries technologies and integrated approaches through partnership in Odisha: USAID-IPP Funded Project (Oct 2017- Mar 2021) - Project Final Report for the period October 2017 to March 2021” WorldFish for USAID, April 2021. Although the final project annual report and close-out report cite a target of 40 WSHGs groups engaged in fish drying, other project documentation lists the target as 10, which is also the number included in the MOU with ICAR-CIFT. Based on the latter figure, the project reached 100 percent of 30 Ratha, B.C., P.A. Padiyar, N. Shenoy, C.V. Mohan. Scaling nutrition-sensitive fisheries technologies and integrated approaches through partnership in Odisha: USAID-IPP Funded Project (Oct 2017–Mar 2021). Project Final Report for the period October 2017 to March 2021. WorldFish for USAID, April 2021. 31 Reddy, Vamsidhar. Key observations and recommendations for the project on Scaling Nutrition-sensitive fisheries technologies and approaches through partnerships from the Mid-term assessment conducted during March 25-29, 2019. USAID, March 2019. 32 Annual Report: Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha: USAID IPP Funded Project (Oct 2019–September 2020). WorldFish for USAID India, October 31, 2020. 33 Ratha, B.C., P.A. Padiyar, N. Shenoy, C.V. Mohan. Scaling nutrition-sensitive fisheries technologies and integrated approaches through partnership in Odisha: USAID-IPP Funded Project (Oct 2017–Mar 2021). Project Final Report for the period October 2017 to March 2021. WorldFish for USAID, April 2021. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 34 its target (or 25 percent based on the former figure). Likewise, utilizing the fish products supplied through the WorldFish-ICAR-CIFT MOU, the project reached 100 percent of its target of 50 Anganwadi centers using fish products in the mother and child health care ICDS program. The collaboration with ICAR-CIFT, together with the initial collaboration with Falcon Marine Ltd. to supply dried marine fish products to schools, also facilitated the project reaching 48 percent of its target of 50 schools using fish in mid-day meal programs. INCREASED CONSUMPTION OF NUTRITIOUS FOODS To increase consumption of nutrient-rich fish and vegetables, especially among women and young children, and increase knowledge and practice of essential nutrition, WorldFish collaborated with the New Delhi-based NGO MAMTA-Health Institute for Mother and Child to develop a set of 22 social behavior change communication (SBCC) materials related to the project activities. These materials included leaflets, posters, radio spots, videos, brochures, wall paintings, and a calendar. MAMTA also created a short cartoon video on the benefits of small fish-based nutrition during the first 1,000 days of life featuring Tikki Mausi, a mascot specially developed for the project, which the Department of WCD&MS used to promote project messaging. These materials targeted the general public in the project zone, and WCD&MS also provided them to Anganwadi centers and schools. WorldFish and WCD&MS sensitized state and district staff on use of the materials and organized FGDs with Anganwadi center and school staff on the benefits of whole small fish consumption.34 In project reports, the SBBC activity listed as the key intervention related to increased consumption lacked a quantitative target. However, the final project report lists 19 as a final result.35 It is unclear what unit of measurement is implied by this target. Nonetheless, the final project close-out report describes a number of institutional activities and linkages related to this component. These include:36 • On October 18, 2019, through the recommendation of WCD&MS, WorldFish attended the “International Consultation Promoting Nutrition-Sensitive Approaches and Linking Agriculture, Health and Nutrition Towards Malnutrition-Free Odisha” to deliver a presentation on linking fisheries with nutrition. • In November 2019, WorldFish organized a visit by a delegation of senior-level Odisha state government officials to WorldFish headquarters in Penang, Malaysia for a workshop on “Maximizing Nutritional Outcomes in Odisha through Fish,” followed by an exposure visit to mola-carp polyculture sites linked to nutrition activities in Cambodia. • Between January 30 and February 4, 2020, WorldFish led a delegation of F&ARD and WCD&MS officials on an exposure visit to Bangladesh to observe research activities related to mola-carp polyculture and vegetable production, and its impact on nutrition in ultra-poor communities. • In 2020, WorldFish was adopted as a member of the Technical Expert Advisory Group on Nutrition and became a partner on state nutrition initiatives, including the Strategy for Odisha’s Pathway to Accelerated Nutrition, targeting 125 nutritionally challenged hard-to￾reach blocks. 34 Ibid. 35 Ibid. 36 Ibid. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 35 • On February 25, 2021, WorldFish and the Department of WSD&MS signed an MOU for inclusion of small fish in SNPs and subsequently jointly developed standard operating procedures for piloting and evaluating fish products used in SNPs. • Based on the project pilot utilizing ten solar dryers operated by WSHGs, the State Department of MSME requested a proposal for inclusion of approximately 100 additional solar dryers in its One District One Product Programme to be implemented in six coastal districts of Odisha. IMPROVED FOOD AND NUTRITION SECURITY IN ODISHA AND SCALING Initially, WorldFish envisioned scaling up the nutrition-sensitive technologies developed through the project through adoption in two states in India and two countries in the region, provisionally identified as Bangladesh and Nepal.37 However, as project implementation proceeded, the project reporting table evolved to focus on a range of “collaboration arrangements” with government and non-government agencies to promote scaling throughout Odisha and in diverse regions beyond.38 WorldFish implemented these linkages through an array of mechanisms, which included: • Collaboration arrangements with other government/non-government agencies for scaling. • A corporate social responsibility event through the Indo-American Chamber of Commerce to connect investors’ goals to WorldFish project activities.39 • Meetings with other regional, national, and international organizations for scaling up the program. • Capacity building activities for government functionaries. Accordingly, project target units were revised to reflect various outcomes of these mechanisms, including execution of MOUs, establishment of partnerships, influence on policies, cross-country visits, and training of officials. Table 11: Evolution of Improved Food and Nutrition Security and Scaling Activities UNIT PERIOD RESULT TARGET Oct 2017 to Sept 2018 Oct 2018 to Sept 2019 Oct 2019 to Sept 2020 Oct 2020 to Mar 2021 Cumulative to March 2021 MOU 0 1 3 1 5 0 Partnership 4 3 4 0 11 0 Policy 0 1 3 0 4 0 Visit 0 0 0 3 3 0 Training 179 1,604 759 578 3,120 0 37 Scaling innovative, nutrition-sensitive fisheries technologies and integrated approaches through partnerships in Odisha, India can improve food and nutrition security--A project proposal. WorldFish for USAID/India, May 2017. 38 Annual Report: Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha: USAID IPP Funded Project (Oct 2019–September 2020). WorldFish for USAID India, October 31, 2020. 39 This event was delayed due to COVID-19 restrictions and held virtually in February 2021. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 36 Source: Ratha, B.C., P.A. Padiyar, N. Shenoy, C.V. Mohan. Scaling nutrition-sensitive fisheries technologies and integrated approaches through partnership in Odisha: USAID-IPP Funded Project (Oct 2017–Mar 2021). Project Final Report for the period October 2017 to March 2021. WorldFish for USAID, April 2021. Although none of these outcomes were assigned discrete targets, WorldFish achieved a degree of progress under each type as illustrated in the table above. The effect of these achievements is discussed in detail below (see EQ 4: Continued Implementation -Findings). EQ 1: PROJECT COMPONENTS - CONCLUSIONS This section presents the conclusions of the evaluation team related to EQ 1: To what extent and how did the project components reach their objectives during implementation? INCREASED AVAILABILITY The project surpassed initial proposed targets of establishing mola-carp polyculture on 450 household fishponds and six community fishponds. However, the project fell short of a revised target of 2,000 household fishponds, reaching only 789 ponds, or 39.5 percent of the revised target. It also fell slightly short of the revised target for community fishponds, reaching 22 ponds, or 91.6 percent of the revised target of 24. Donors and implementing partner staff attribute these shortfalls to a beneficiary recruitment drop and limitations on activity implementation following the 2019 budget delay and subsequent COVID-19 restrictions on movement and beneficiary contact. INCREASED ACCESSIBILITY WorldFish met its objective of establishing 10 solar dryers in coastal areas managed by WSHGs through collaboration with ICAR-CIFT, which supplied the dryers and trained the group members on their use. Additionally, ICAR-CIFT provided dried fish powder to facilitate WorldFish meetings its objective of 50 Anganwadi centers using fish products in the mother and child health care ICDS. On the other hand, although WorldFish was able to launch a pilot program for distribution of fish through WCD&MS-managed schools, the program was curtailed in 2019 due to USAID concerns, and the project subsequently fell short of its target of 50 schools using fish in mid-day meal program, reaching only 24 schools, or 48 percent of its objective. INCREASED CONSUMPTION Though the project undertook an array of SBCC activities, it is unclear from project documentation whether this yielded the desired changes in consumptions patterns, particularly among women and young children, the primary targets of the SBCC intervention. IMPROVED FOOD AND NUTRITION SECURITY IN ODISHA AND SCALING The project aimed to improve food and nutrition security through a range of “collaboration arrangements” with government and non-government agencies to promote scaling throughout Odisha and beyond. Project target units were revised to reflect execution of MOUs, establishment of partnerships, influence on policies, cross-country visits, and training of officials. However, the project did not set quantitative targets for these items. EQ 2: INTERVENTION SUCCESS FACTORS - FINDINGS This section presents the findings of the evaluation team related to EQ2: What factors hindered interventions, and what factors contributed to their success? Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 37 INCREASED AVAILABILITY To understand the context of fish farming in the project zone, in both its current form and historical development, the evaluation team consulting historical references. In KIIs with mola-carp polyculture beneficiaries, farmers throughout the project zones indicated that fish is a key component of local cuisine and is usually consumed on a daily basis. In the mini survey, just 1 percent of respondents ranked the importance of fish in household diets as “very low” (1), while the majority ranked it at 5, or “very high” (65 percent). In contrast, respondents did not feel as strongly about the importance of fish to household incomes, with 15 percent ranking it as “very low” and 37 percent ranking it as “very high.” See Figure 3. Figure 3: Pre-Project Importance of Fish in Household Diets (left) and Incomes (right) Source: Panagora mini-survey, January-February 2023. This qualitative data can be interpreted as confirming the relatively high to very high importance of fish in household diets, while contrasting this with the less high importance of sales of surplus fish to household incomes. Notably, this may also reflect a gradual transition away from historical household self-sufficiency in fish underway in the region as described in KIIs with mola-carp polyculture beneficiaries. KII respondents stated that populations in the region had historically accessed fish, including undifferentiated small fishes, through collection in the region’s numerous ponds and rivers. However, unfettered access to these sources began to diminish in the past 30-40 years as high levels of in-migration to Odisha put population pressure on natural resources. As one KII respondent explained, “my father’s father just gathered fish freely in ponds like everyone. My father was the first one that farmed fish in a pond he owned.”40 To increase availability of fish and fish products, WorldFish promoted mola-carp polyculture in household and community ponds through training, provision of a portion of start-up costs, and distribution of inputs including pond preparation material (zeolite), mola and carp seed, and an initial tranche of fish feed. These activities engaged a wide variety of households at different stages in the evolution away from historical subsistence self-sufficiency described here. Figure 4: Household Fishpond Owner Typologies Likewise, farmers can be arrayed along a continuum ranging from “household oriented” to “market￾oriented,” along which the evaluation team identified four illustrative typologies: 1) “household 40 Mola-carp polyculture beneficiary KII. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 38 suppliers,” or farmers that utilize their fishponds to supply household consumption, but do not sell fish and do not invest in productivity enhancing inputs, such as zeolite and fish feed, usually because they are primarily focused on alternate livelihood activities; 2) “dual use” farmers, producing sufficient fish to supply both home consumption and some surplus that can be sold; 3) “semi￾commercial” farmers that produce both for household consumption and outside sales through limited investments in productivity-enhancing inputs; and 4) “commercial farmers” who seek to optimize productivity of their fishponds to maximize sales and income.41 The validity of these typologies is strengthened by the close proximity to each other in which they occur. That is, in target village clusters, within a radius of a few hundred yards, the evaluation team encountered very distinct types of fishpond production and very low levels of technology transfer between farmers. For example, at one village, a “commercial farmer” achieving very high productivity directed the team to a second farmer (described as a “family friend”) located a five-minute walk away. This second farmer was a “semi-commercial” fish farmer in that, as he was primarily engaged in poultry farming, he applied poultry feed to his fishpond, resulting in sub-optimal productivity. When questioned why he didn’t use fish feed to obtain the higher productivity level achieved by the nearby commercial farmer, reportedly a close acquaintance from whom acquiring this information should have been relatively easy, he stated he didn’t believe the type of feed used impacts productivity and that, as he was primarily a poultry farmer, he didn’t consider it important to investigate alternative inputs. Although anecdotal, the dynamics of these typologies, especially the significant variation in technology adoption resulting from livelihood-related prioritization of resources, had important consequences for post-project sustainability of outcomes and impacts. KIIs respondents also noted that technology adoption corresponds to two typologies of ponds found in the region: 1) seasonal ponds, which run dry each year, requiring farmers to invest in seed to restock on a seasonal basis; and 2) perennial ponds, which retain water throughout the year, allowing highly reproductive mola to sustain themselves naturally. However, it is unclear to what degree seasonal ponds could be modified to make them perennial through amendments to dikes and bunds. In numerous KIIs and FGDs, farmers made clear that virtually no one purchased mola seed for re-stocking, but rather replied on the fish’s prolific breeding to maintain populations in perennial ponds. In addition, many expressed the belief that seasonal fishponds “naturally” restocked themselves with small fish following rainy seasons. In almost all cases, farmers made very little differentiation between mola and other small fish, such chuna and korondi, often referring to them interchangeably.42 Few communal ponds are seasonal, and therefore KII respondents report that mola restocking is not an issue, as breeding maintains populations. A more significant factor that may discourage ongoing use of community-managed fishponds is lease status. At the time of project implementation, KII respondents stated that leases were commonly only for two-year terms, reducing the attractiveness of SHG investment into production. However, in KIIs and FGDs, respondents stated that in 2022, terms of pond leases had been extended to five years.43 INCREASED ACCESSIBILITY The project aimed to increase accessibility to fish and fish-based products through pilot use of solar dryers by WSHGs. In KIIs and FGDs, WSHG members universally recognized solar dryers as 41 Mola-carp polyculture beneficiary KIIs and FGDs. 42 Mola-carp polyculture beneficiary KIIs and FGDs. 43 Mola-carp polyculture beneficiary KIIs and FGDs and implementing partner KIIs. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 39 producing higher quality dried fish. The key factors influencing the success of the pilot use of solar dryers were finance related. In KIIs with implementing partner staff, respondents quoted the cost of the solar dryer utilized in the pilot program at about INR 150,000 (~US$1,821) per unit, including plastic sheeting, a solar panel and fans, transport of materials from Kochi, Kerala, and materials for unit foundation (“basement”).44 Although these unit costs may result in positive cash flow under certain business models, every WSHG interviewed by the evaluation team stated that the cost of the unit was “too expensive,” and not something they would invest in using their own funds.45 In addition, WSHGs using the dryers may require investment capital for purchasing fresh fish for drying, though in many cases, the groups are able to dry unsold “surplus” fish, often caught by male family members, and thus do not incur a cost for stock. Additional factors that can affect potential income include seasonality (i.e., the length of time during which viable fish for drying is available), the potential for dual uses (using the dryers for other potentially income generating products, though respondents noted that dual use is constrained by residual fish odor following fish drying). Ongoing costs can also include repairs and maintenance (the evaluation team encountered at least two of the pilot units that required replacement of damaged parts, including fans and solar panels that had been vandalized, and plastic sheeting that had been damaged).46 Finally, net income varies considerably between sites and scenarios, with WSHGs at some sites reporting higher prices for the better quality, more hygienic dried fish that resulted from use of the solar dryers.47 At other sites, these groups reported no difference in prices for better drying techniques, but rather that increased income is solely the result of being able to sell unsold surplus fish over a longer period of, in some cases, allowing them to dispose of otherwise unsold fish, and in other cases allowing them to avoid a market glut following a good catch that would depress prices. Notably, in conjunction with the perception of WSHGs that the cost of the solar dryer was very high, these complicated and multifaceted variations led WSHGs to cite assistance with “market planning” (or business models) as their most urgent need. In addition, WorldFish aimed to increase accessibility through building linkages with institutional users for utilization of fish and fish products made from mola including through SNPs at Anganwadi centers and schools providing mid-day meals. For these activities, taste and acceptance were the most important factors that hindered or facilitated the success of these interventions, though cost and authorization also played a role. Throughout the project zones in Odisha, the evaluation team noted the widespread practice of drying marine fish (often on roads or at other public areas), and of the presence of whole dried fish, both small and large, marine and riverine, at markets, indicating good acceptance of these products. Likewise, in KIIs and FGDs, beneficiaries described liking the taste of dried fish products made from mola, and 78 percent of mini survey respondents described the taste as “very good” (5), as illustrated in Figure 5. 44 Implementing partner KIIs. 45 WSHG solar dryer beneficiary FGDs. 46 WSHG solar dryer beneficiary FGDs and Implementing partner KIIs. 47 WSHG solar dryer beneficiary FGDs. One WSHG reported receiving a margin of INR 20 over poorer quality unhygienic traditional dried fish, which sold for INR 120 per kg. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 40 Figure 5: Survey Respondents’ Rankings of Taste of Products Made from Whole Dried Mola Source: Panagora mini-survey, January-February 2023. On the other hand, staff at Anganwadi centers stated that children rejected dried fish powder manufactured from marine fish at mid-day meals. Staff at centers in Kaptipada described children vomiting and refusing to eat meals prepared with the fish powder, and its use was subsequently discontinued.48 In subsequent KIIs with ICAR-CIFT staff, respondents stated that they recommended reducing the quantity of powder by 50 percent, but Anganwadi center staff did not implement this recommendation.49 In addition, the comparably higher cost of fish and fish products in SNPs made these products less attractive than traditional protein sources. According to Department of MDC&MS guidelines, the authorized per person cost of a take home ration is Indian Rupees (INP) 9.50 for pregnant and lactating mothers, INR 8 for normal children under three years, and IRP 12 for severely malnourished children.50 The authorized cost of on-site feeding is IRP 8 per day (comprising a meal at IRP 6.76 and a snack at IRP 1.24). According to Anganwadi staff, eggs procured through bulk purchases cost only IRP 5.50 per serving, while a 30-gram serving of fish, which is nutritionally comparable, costs INR 9, rendering the dried fish economically unviable.51 Furthermore, the same respondent stated that dried fish was not an authorized item for use as of January 2023 (despite the February 25, 2021 MOU signed by WorldFish and MSD&MS for inclusion of small fish in SNPs).52 INCREASED CONSUMPTION The SBBC promotion launched in 2019 focused on consumption of whole small fish to leverage nutritional resources located in the head. In KIIs and FGDs with beneficiaries aware of this campaign, respondents indicated acceptability of consuming the entire fish.53 Other institutional mechanisms for promoting increased consumption are discussed in subsequent sections (see below: EQ 4: Continued Implementation – Findings and EQ 4: Continued Implementation – Conclusions). IMPROVED FOOD AND NUTRITION SECURITY IN ODISHA AND SCALING The diverse collaboration arrangements between government/non-government agencies implemented by WorldFish for scaling the project largely leveraged alignment of objectives between agencies, which was the key factor in success of activities under this objective. Examples of this 48 Anganwadi center staff KIIs. 49 Implementing partner staff KIIs. 50 Revised Guidelines for Implementation of Take-Home Ration 2018 (Supplementary Nutrition Programme of ICDS and Scheme for Adolescent Girls). Department of MDC&MS. 51 Anganwadi center staff KIIs. 52 Ibid. 53 Mola-carp polyculture beneficiary KIIs and FGDs. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 41 factor can be found in the WorldFish–F&ARD relationship through which mola-carp polyculture was promoted, the Department of F&ARD–WD&MS collaborations to foster institutional linkages with SNPs, as well as the collaboration with ICAR-CIFT and Central Tuber Crops Research Institute to provide solar dryers, fish products, and OSP planting material.54 On a related note, the use of existing organizations and systems, such as WSHGs, was an additional factor in hindering or determining success. These factors, and the specific outcomes related to them in discrete activities, are further discussed in subsequent sections (see below: EQ 4: Continued Implementation – Findings and EQ 4: Continued Implementation – Conclusions). EQ 2: INTERVENTION SUCCESS FACTORS - CONCLUSIONS This section presents the conclusions of the evaluation team related to EQ2: What factors hindered interventions, and what factors contributed to their success? INCREASED AVAILABILITY Activities that aimed to increase availability included promotion of mola-carp polyculture through training, provision of start-up costs, and distribution of inputs to farmers and SHGs using household and communal ponds. The key factor determining the success of these activities was the degree of market orientation of beneficiaries, which ranged from very low to very high, and determined the degree to which beneficiaries are willing to allocate their own resources into fish production following project-funded input distributions. Additional factors included type of pond (seasonal or perennial), and, in the case of communal ponds, the terms of leases, previously two years, now extended to five years. INCREASED ACCESSIBILITY Activities that aimed to increase accessibility focused on improving the quality and hygiene of fish products through promoting the use of solar dryers, as well as fostering linkages with institutional systems that could use fish and fish products, especially SNPs at Anganwadi centers and schools providing mid-day meals. Factors that hindered or facilitated the success of solar dryer promotion were mostly related to finance and cash flow, and included seasonality, potential for dual use, sales structure, and cash flow. Factors that hindered or facilitated the success of activities that promoted institutional linkages with SNPs to utilize fish products consumer preference/acceptance, as well as comparative cost and authorization to procure such products. INCREASED CONSUMPTION While activities that aimed to increase consumption of fish and fish products were centered on SBCC, they also included diverse institutional linkages that aimed to disseminate the approaches promoted by the project. These activities feature diverse success factors and are discussed in subsequent sections of this report. IMPROVED FOOD AND NUTRITION SECURITY IN ODISHA AND SCALING The key factor determining the success of activities that aimed to increase food and nutritional security and scale up approaches was how well the project was able to align its objectives with other agencies, and there are numerous examples of how WorldFish staff fostered and developed these collaborations. An additional factor was the way in which WorldFish was able to leverage existing 54 Locally Led Development: Partnering for Improved Nutrition—Lessons from Odisha, India. JSI Research & Training Institute, Inc. for USAID Feed the Future, August 2021 and implementing partner staff KIIs. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 42 organizations and systems. These factors, and the specific outcomes related to them in discrete activities, are further discussed in subsequent sections of this report. EQ 3: CONTINUED USE AND HINDERING FACTORS - FINDINGS This section presents the findings of the evaluation team related EQ 3: Are project beneficiaries and stakeholders continuing to use promoted behaviors related to use of innovative, nutrition-sensitive fisheries technologies and integrated approaches to improve food and nutrition security? If so, how? If not, what factors hinder continuation of these practices? As such, the section presents findings related to continued use of TA and inputs provided through three activities: 1) promotion of carp-mola polyculture in household fishponds; 2) promoting stocking mola and other fishes in community fishponds; and 3) promotion of WSHGs’ use of solar dryers to produce hygienic dried fish. INCREASED AVAILABILITY As noted, the final project annual report states that by March 2021, the project had piloted carp￾mola polyculture in 789 household ponds and in 22 community ponds, benefiting 1,307 households (see EQ 1: Project Components – Findings). In addition, a final study of 784 farmers conducted by WorldFish and partners in 2021 found that this led to 657 out of 784 households surveyed harvesting fish from their ponds, resulting in increased consumption of fish. In addition, the same report states that 226 beneficiaries surveyed reported harvesting approximately 2,366 kg of OSP in the third year of the project.55 In addition, in the mini survey carried out by the evaluation team, the majority (72 percent) of beneficiary respondents rated the training they received as “very useful” (5), while only 1 percent and two percent ranked the training as “very un-useful” (1) or “un-useful” (2), respectively.56 Similarly, the largest proportion of responses to the question about post-project change in income, 39 percent, assigned a score of “very high” (5), though 24 percent ranked this change as “minor” (3) and 13 percent indicated “no change” (1). In addition, six percent indicated they “did not receive inputs” (0). Figure 6: Usefulness of Technical Training (left) and Post-Project Change in Incomes (right) Source: Panagora mini-survey, January-February 2023. However, these findings do not necessarily correlate to farmers’ continued practice of new behaviors that the project promoted, in part by providing farmers the resources needed to adopt these new behaviors. This has implications for sustainability, which would depend on beneficiaries using their own funds to purchase these inputs, such as pond preparation material (zeolite), fish feed, 55 Ratha, B.C., P.A. Padiyar, N. Shenoy, C.V. Mohan. Scaling nutrition-sensitive fisheries technologies and integrated approaches through partnership in Odisha: USAID-IPP Funded Project (Oct 2017–Mar 2021). Project Final Report for the period October 2017 to March 2021. WorldFish for USAID, April 2021. 56 Panagora mini-survey, January– February 2023. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 43 and re-stocking seed. KIIs and FGDs with mola-carp beneficiaries suggest sustainability is linked to the extent to which farmers are motivated by a commercial interest (see EQ 2: Intervention Success Factors – Findings).57 • “Household suppliers” did not report purchasing any inputs following use of project￾provided inputs, instead resuming previous subsistence production. • “Dual use” farmers, producing sufficient fish for home consumption with limited sale of surplus, also did not purchase inputs, with the exception of carp seed to re-stock ponds. • “Semi-commercial” farmers that produce both for household consumption and outside sales purchased seed for restocking as well as some fish feed to increase productivity. • “Commercial farmers” purchased the full suite of inputs to maximize productivity. Although the evaluation team did not carry out a quantitative evaluation of this classification system, KIIs suggested that approximately one-third of farmers fell into the “dual use” category.” which also includes all farmers with seasonal fishponds. On the other hand, of 40 farmers interviewed, the team estimates that three would fall into the category of “commercial,” though two of these seemed to be in this category prior to project implementation. On communal ponds, virtually all SHGs continued to harvest small fish that maintain populations through natural replication, and they are also undertaking some drying for later sales. Lease terms was the most significant factor in continued use of these ponds. Of eight community ponds no longer in use one and a half years after project end, for five, SHGs reported refusal of lessors to renew leases, or leasing to an alternate party, as the cause of discontinuing use as illustrated in Table 12.58 57 Mola-carp polyculture beneficiary KIIs and FGDs. 58 Ibid. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 44 Table 12: Status of Community Fishponds in February 2023 LOCATION PROJECT ACTIVITIES STATUS No. District Block Mola/a Carp/a Training/a Notes 1 Balasore Soro No Yes Yes Lease ended - no longer in use. 2 Balasore Soro No Yes Yes Lease ended - no longer in use. 3 Balasore Soro No Yes Yes Continued use. 4 Balasore Khaira Yes Yes Yes Continued use. 5 Balasore Khaira Yes Yes Yes Continued use. 6 Jagatsinghpur Sadar Yes Yes No Continued use. 7 Jagatsinghpur Sadar Yes Yes No Continued multipurpose use. 8 Jagatsinghpur Naugaon No No Yes No longer in use. 9 Jagatsinghpur Naugaon No Yes No Continued use. 10 Jagatsinghpur Sadar No Yes No Continued use. 11 Jagatsinghpur Sadar No Yes Yes No longer in use. 12 Jagatsinghpur Sadar No Yes Yes Leased to other party. 13 Jagatsinghpur Sadar No Yes Yes Continued use. 14 Jagatsinghpur Sadar No Yes No Continued use. 15 Jagatsinghpur Sadar No Yes No Continued use. 16 Jagatsinghpur Sadar No Yes Yes Continued use. 17 Mayurbhanj GB Nagar Yes Yes Yes Continued use. 18 Mayurbhanj GB Nagar Yes Yes Yes Leased to other party. 19 Mayurbhanj GB Nagar Yes Yes Yes Seasonal pond - leased to other party. 20 Mayurbhanj Kaptipada Yes Yes Yes Reservoir - annual community harvesting. 21 Puri Kanas Yes Yes Yes No longer in use. 22 Cuttack Cuttack Sadar Yes Yes Yes Continued use. a. Indicates SHG received relevant fish seed or training from project. Source: Panagora KIIs/FGDs and site visits, January–February 2023. KIIs suggest that perceptions of access to TA constitutes an additional defining factor with regard to fish farmer typologies and post-project behavior. In KIIs with mola-carp polyculture beneficiaries on both household and community fishponds, when asked about post-project sources of TA, most respondents stated that they lacked access to TA. They specifically stated that visits by Department of F&ARD extension staff were “extremely limited,” and that local agro-input dealers, including fish feed vendors, were unable to provide them advice or TA on input use or support to address occasional problems, such as disease outbreaks. On the other hand, all “commercial” farmers, and Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 45 some “semi-commercial” farmers, stated that they could easily access TA through online video channels such as YouTube, and that these sources met their TA needs.59 All mola-carp polyculture beneficiaries who received vegetable seeds reported consuming the vegetables they produced. Given that these were hybrid seeds, no farmers saved seeds, and all resumed their previous planting patterns the following season (though a few continue to tend banana trees provided by the project). On the other hand, although the final project close-out report states that “sweet potato is an important crop planted and consumed in Odisha,”60 the Year 3 annual report contradictorily states that “acceptance of OSP by the communities was a great challenge due to (a) lack of awareness on nutritional value of OSP in the community, (b) OSP is not included in their regular food habit, (c) lack of suitable land for cultivating in some of the project area.”61 Accordingly, no respondents reported continuing production of OSP following receipt of planting material during the project.62 INCREASED ACCESSIBILITY Despite the multiplicity of cash flow and profitability factors related to solar dryer use, repair and maintenance expenses were the most common reason cited for discontinuing their use after the project closed out, with three of the ten units no longer in use due to these problems: see Table 13. No WSHG members reported paying for post-project maintenance or replacement parts from their own funds, further reinforcing the impression of poor understanding of business models.63 59 Ibid. 60 Ratha, B.C., P.A. Padiyar, N. Shenoy, C.V. Mohan. Scaling nutrition-sensitive fisheries technologies and integrated approaches through partnership in Odisha: USAID-IPP Funded Project (Oct 2017– Mar 2021). Project Final Report for the period October 2017 to March 2021. WorldFish for USAID, April 2021. 61 Annual Report: Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha: USAID IPP Funded Project (Oct 2018-Sept 2019). WorldFish for USAID/India, Penang, Malaysia, October 30, 2019. 62 Ibid. 63 WSHG solar dryer beneficiary KIIs and FGDs, and site visits. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 46 Table 13 Status of Solar Dryers (February 2023_ LOCATION ACTIVITIES AND USE STATUS No. District Block Training/a Fish Dual/b Notes 1 Jagatsinghpur Kujanga Yes Yes Yes Solar dryer stolen, but in dual use. 2 Jagatsinghpur Kujanga Yes Yes Yes In dual use. 3 Puri Astaranga Yes Yes Yes In dual use 4 Puri Astaranga Yes No No Fan stolen; unit not functional. 5 Khurda Tangi No No No Insufficient funds for stock, unit not in use. 6 Ganjam Sadar Yes No No Unit not functional. 7 Balasore Sadar Yes Yes Yes Some water access problems, but in dual use. 8 Balasore Bahanaga Yes Yes No Used for fish only. 9 Bhadrak Basudevpur Yes Yes No Used for fish only. 10 Bhadrak Basudevpur Yes Yes No Used for fish only. a. Indicates WSHG received training from project. b. Indicates WSHG currently using solar dryer for fish and other products simultaneously. Source: Panagora KIIs/FGDs and site visits, January–February 2023. As noted, WSHG members cited improved market linkages and assistance with business plans as their most pressing needs, which is unsurprising given the multiplicity of factors that determine cash flow for solar dryer operations. In addition, in FGDs, WSHG members stated that the need to situate the dryers on commonly accessible land often meant that they were located at remote sites that are difficult to supervise, and therefore especially vulnerable to vandalism and theft, as occurred in at least two cases. As noted, respondents in FGDs also stated that the strong fish odor that remained in the solar dryers following use was the key reason they did not improve incomes through alternative use in seasons when fish was not available for drying (see EQ 2: Intervention Success Factors – Findings).64 EQ 3: CONTINUED USE AND HINDERING FACTORS – CONCLUSIONS This section presents the findings of the evaluation team related EQ 3: Are project beneficiaries and stakeholders continuing to use promoted behaviors related to use of innovative, nutrition-sensitive fisheries technologies and integrated approaches to improve food and nutrition security? If so, how? If not, what factors hinder continuation of these practices? INCREASED AVAILABILITY The primary factor related to continued use of behaviors promoted to improve the availability of fish from household and community fishponds is the degree of market orientation of beneficiaries. Given the evolution of fish production over the past two to three decades, from an essentially subsistence activity in which households readily obtained fish from accessible rivers and ponds to a more specialized activity carried out on specific household and community-managed ponds, it is 64 WSHG solar dryer beneficiary FGDs. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 47 unsurprising that a narrow population has chosen to specialize in this sector to supply markets, while the majority of the population focuses their resources on other livelihoods strategies. The degree of market orientation determines the willingness of farmers to invest their own resources in fish production. Reported access to TA seems to be a proxy for market orientation, with less market￾oriented farmers describing a dearth of TA, while more market-oriented farmers reported that they were able to easily access required TA through online resources such as YouTube. Although typologies in this regard are only illustrative, it is likely that much less than ten percent of fish farmers are sufficiently market oriented to purchase inputs with their own funds to improve productivity, while the remainder do not purchase inputs beyond seed for re-stocking and allocate their resources to other livelihoods activities. The exception to this generalization is SHGs that farm fish on community ponds. In these cases, lease terms are most likely to be the determining factor in the continued application of promoted behaviors. Across both types of fish farmers, none changed their cultivation of vegetables as a result of receiving vegetable seeds, and none continued cultivation of OSP. On the other hand, a few continue to harvest project-supplied trees. INCREASED ACCESSIBILITY WSHGs utilizing solar dryers for fish drying face a multiplicity of cash flow-related issues that determine the financial viability of continued use of these units, including seasonality, potential for dual use, as well as income structure, which can take the form of price premiums for hygiene and quality, extending quantity or duration to increase sales or reduce market glut, or some combination of all of these. Given the low business skills of WSHG members, determining profitability is often confusing, and the groups lack the ability to expand market linkages. The relatively high costs of repairs and maintenance of the solar dryer units was found to be the key determining factor in continued solar dryer use and following the project close-out, the WSHGs have not demonstrated willingness to invest their own funds in maintenance. EQ 4: CONTINUED IMPLEMENTATION - FINDINGS This section presents the findings of the evaluation team related to EQ 4: Have the implementing partners/stakeholders strengthened by USAID funding continued to implement project components? What is stakeholder perspective/feedback about project impact? As such, this section considers potential post￾project support and expansion mola-carp polyculture, as well as solar drying and inclusion of fish and fish products in SNPs, especially through partnerships with Odisha state government entities as well as through findings from additional external donors. INCREASED AVAILABILITY In March 2022, WorldFish and the Odisha State Government Department of F&ARD announced that an assessment study of the collaborative agreement signed between them in June 2016 had resulted in significant progress toward several Government of India objectives, including “fisheries and aquaculture productivity increase, poverty alleviation, food security and nutrition gains, household resilience and women’s empowerment.”65 Likewise, in KIIs, Department of F&ARD officials described their partnership with WorldFish positively and indicated that they planned further support for scale-up of activities.66 Similarly, WorldFish continues to implement a number of activities launched through the initial WorldFish–F&ARD MOU in zones covered by this agreement, 65 Cullhaj, Megi. The technical collaboration between WorldFish and the Government of Odisha (India) reaches nutrition and income goals for over 151,407 people. WorldFish Website, April 14, 2022: mel.cgiar.org 66 Implementing partner KIIs. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 48 included ongoing support to over 6,000 WSHGs to adopt mola-carp polyculture in Gram Panchayat tanks in collaboration with the Department of WCD&MS. In addition, as noted above, WorldFish was included in the Odisha State Nutrition Secretariat as a member of the Technical Expert Advisory Group on Nutrition through which it would support the Government of Odisha in implementing the state Strategy for Odisha’s Pathway to Accelerated Nutrition, targeting 125 nutritionally challenged hard to reach blocks, and “ODISHA 2020–25” to promote nutrition￾sensitive approaches for linking agriculture, health and nutrition (see above EQ 1: Project Components – Findings).67 Accordingly, although actual launch was still pending at the time of this evaluation, WorldFish has entered into a number of new project agreements to expand availability of small fish. Notably, these include a pilot in Kamrup District of Assam funded by the Indian Oil Corporation and the GIZ￾funded “Taking Nutrition-Sensitive Carp-Small Fish Polyculture Technology to Scale” project, which aims to establish hatchery technologies and seed dissemination models for small indigenous species in Odisha.68 INCREASED ACCESSIBILITY As noted, in 2020 the Department of WCD&MS approved a pilot for inclusion of small fish in SNPs at its Anganwadi centers in Mayurbhanj District, and jointly with WorldFish developed standard operating procedures for inclusion of small fish in other SNPs in Odisha; this has since been submitted and approved (this program was scheduled for launch in November 2020).69 The decision to launch this pilot was partially based on an “exposure visit” to Cambodia in 2019 where state GOI staff witnessed similar fish consumption. Likewise, in 2022 the State Departments of F&ARD and MSME jointly decided to scale up the solar dryer pilot project launched under the USAID-funded project under its One-District One-Product program. Through this program, WSHGs will receive financial assistance, credit linkages, and market linkage support for at least 100 solar fish dryers over the next two years.70 EQ 4: CONTINUED IMPLEMENTATION – CONCLUSIONS This section presents the conclusions of the evaluation team related to EQ 4: Have the implementing partners/stakeholders strengthened by USAID funding continued to implement project components? What is stakeholder perspective/feedback about project impact? Although mostly still in pre-implementation phase, key partners have continued to pursue project objectives through additional activities funded by GOI entities as well as by external donors. Although WorldFish initially launched its activities through an MOU with the State Department of F&ARD, the agency has established robust linkages with a number of additional official entities through leveraging joint objectives. This convergence of agency objectives has strengthened the ability of WorldFish to continue pursuing project components. EQ 5: SUSTAINABILITY FACTORS - FINDINGS 67 Ratha, B.C., P.A. Padiyar, N. Shenoy, C.V. Mohan. Scaling nutrition-sensitive fisheries technologies and integrated approaches through partnership in Odisha: USAID-IPP Funded Project (Oct 2017–Mar 2021). Project Final Report for the period October 2017 to March 2021. WorldFish for USAID, April 2021. 68 Ibid. 69 Annual Report: Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha: USAID IPP Funded Project (Oct 2019-September 2020). WorldFish for USAID India, October 31, 2020. 70 Ibid. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 49 This section presents the findings of the evaluation team related to EQ 5: What are the factors that contributed to or impaired the long-term sustainability of project outcomes and outputs? Specifically, it identifies the factors that determine sustainability of mola-carp polyculture to improve availability, and for less and more market-oriented farmers. There are several reasons to believe the project fostered a sustainable increase in demand for whole fresh and dried small fish among mola-carp polyculture beneficiaries. Beneficiaries like the taste of the fish (see above EQ 2: Intervention Success Factors – Findings), and WorldFish and its partners helped reinforce this through nutrition training focusing on the benefits of increased consumption of fish and fish products, especially eating whole small fish. In the mini survey, the majority of beneficiaries, or 66 percent, described this training as “very useful” (5). Likewise, 35 percent of mini survey respondents rated the post-project change in their diets as “significant” (3) while 43 percent rated this change as “extremely significant” (5), as illustrated in Figure 7. Figure 7: Usefulness of Nutrition Training (left) and Post-Project Change in Diet (right) Source: Panagora mini-survey, January–February 2023. In KIIs with household fishpond farmers and in FGDs with SHGs farming community ponds, respondents described consuming increased quantities of whole small fish, both fresh and dried. Given that mola and other small fish replicate naturally without farmers incurring costs, this change in consumption to larger quantities of whole small fish is likely to be sustainable regardless of the market orientation typology of farmers (even in cases of perennial ponds, small fish reappear in household ponds through waterflow between rivers and ponds).71 In cases where farmers are more market oriented i.e., semi-commercial” and “commercial,” increased mola sales represent a net increase with virtually no cost, and increased sales and hence availability is therefore likely to be sustainable. In addition, KIIs with more market-oriented fish farmers suggest application of recommended inputs generates a significant positive net income based on prevailing market prices. Taking SHG-managed community ponds as an illustrative mid-point in market orientation and based on data collected through KIIs and FGDs, expenses including lease cost, labor for pond preparation, fish feed, and carp fingerling and mola fry, against revenues from mola and carp sales, suggest a positive return on investment of approximately 1.40 percent as illustrated in Table 14. 71 Mola-carp polyculture beneficiary KIIs and FGDs. 11% 8% 2% 3% 9% 66% 0% 20% 40% 60% 80% 100% 0 1 2 3 4 5 7% 7% 2% 35% 4% 43% 0% 20% 40% 60% 80% 100% 0 1 2 3 4 5 Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 50 Table 14: Mola-Carp Polyculture on Communal Pond Illustrative Cost of Production ITEM UNIT NO. RATE (INR) COST (INR) Expenses Lease Year 1 1,500 1,500 Pond preparation Labor Day 2 250 500 Fish feed Bag 1 7,000 7,000 Fingerling (Carp) Kg. 40 300 12,000 Fry (Mola) Kg. 12 600 7,200 Pond maintenance/a n/a n/a n/a n/a Harvest/a n/a n/a n/a n/a Total Expenses 28,200 Revenue Sale (Mola) Kg. 25 150 3750 Sale (Carp) Kg. 350 180 63,000 Total Revenue 66,750 Net income 38,550 a. Costs to buyer Source: Mola-carp polyculture beneficiary KIIs and FGDs. Of course, actual expenses and revenues will vary significantly based on specific context and market orientation of individual farmers, which in turn determines their willingness to invest in fish farming as opposed to other livelihoods activities. As noted, there are a wide array of both cost and revenue factors that determine the economic sustainability of solar dryer use (see above EQ 2: Intervention Success Factors – Findings). EQ 5: SUSTAINABILITY FACTORS - CONCLUSIONS This section presents the conclusions of the evaluation team related to EQ 5: What are the factors that contributed to or impaired the long-term sustainability of project outcomes and outputs? Once established, increased consumption of both fresh and dried whole small fish presents limited or no additional costs to household and community fishpond farmers. This is largely due to the natural regeneration of these fish due to prolific breeding, including their re-establishment in perennial farms. These factors make increased long-term consumption of fresh and dried whole small fish for the purpose of improved nutrition sustainable. In the case of more market-oriented farmers, including SHG-managed community ponds, increased production through recommended investments is sustainable over the long term based on the significantly positive return on investment generated. However, the majority of the population Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 51 choses to invest their capital resources into alternative livelihoods, and therefore will not invest these resources into their fishponds to continue best practices promoted by this project. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 52 4. LESSONS LEARNED The “Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha” project deployed the two strategies of introducing nutrition-sensitive production technologies and integrated approaches including combining fish and vegetables within farming systems in the state of Odisha and establishing strengthened and expanded partnerships with a range of partners to address four integrated objectives: 1) increased availability; 2) increased accessibility; 3) increased consumption of nutritious foods; and 4) scaling regional and national. Table 15: Reported Project Components and Key Activities with Targets and Results (revisited) COMPONENTS AND ACTIVITIES TARGET RESULT RATEa Objective 1: Increased Availability No. of households practicing carp-mola polyculture (including vegetable and OSP) 2,000 789 39.5% No. of community ponds stocking mola and other 24 22 91.6% Objective 2: Increased Accessibility No. of women’s groups engaged in fish drying 40 10 25.0%b No. of Anganwadi centers using fish products in mother and child health care (ICDS) 50 50 100% No. of schools using fish in mid-day meal program 50 24 48.0% Objective 3: Increased Consumption of Nutritious Foods SBCC materials 0 19 n/a Objective 4: Improved Food and Nutrition Security in Odisha and Scaling MOUs 0 5 n/a Partnerships 0 11 n/a Policy influence 0 4 n/a Cross-country visits 0 3 n/a Training to other government functionaries 0 3,120 n/a a. Calculated here by the evaluation team. Activities lacking specific non-zero quantitative targets are labeled “n/a”. b. WorldFish-ICAR-CIFT MOU states a target of ten WSHGs, which is also reported in project documentation. Source: Ratha, B.C., P.A. Padiyar, N. Shenoy, C.V. Mohan “Scaling nutrition-sensitive fisheries technologies and integrated approaches through partnership in Odisha: USAID-IPP Funded Project (Oct 2017- Mar 2021) - Project Final Report for the period October 2017 to March 2021” WorldFish for USAID, April 2021. Key activities to achieve the objective of increased availability included promotion of mola-carp polyculture on home and community-managed fishponds, in partnership with the State Department of F&ARD, through training and distribution of inputs and partial funding to cover some of the costs of this enterprise. The project also promoted solar dryer use through a pilot program with WSHGs and through piloting institutional linkages with SNPs through Anganwadi centers and schools. To improve consumption, the project used SBCC to encourage consumption of whole fresh and dried Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 53 small fish. Finally, to achieve regional and national scaling, the project developed a wide range of inter-agency linkages building to disseminate its models. This section presents lessons learned based on the evaluation findings. 4.1 TAILORING BENEFICIARY SELECTION DISAGGREGATING BENEFICIARY SELECTION CRITERIA BY OBJECTIVE CAN IMPROVE PROJECT EFFECTIVENESS AND EFFICIENCY. The evaluation team found limited beneficiary adoption of promoted mola-carp production techniques that lead to increased availability. Instead, most beneficiaries simply consume extra fish resulting from increased productivity from distributed inputs rather than shifting to a more commercial model as envisioned by the project. A key limiting factor in this regard is the degree of market orientation of specific beneficiaries, with the majority choosing to focus their limited financial resources on alternative livelihoods activities rather than investing them in continued enhanced productivity. With regard to activities that aimed to improve accessibility, the promotion of solar dryers was relatively successful, with the majority of participating WSHGs reporting the continued use of dryers. However, these groups also expressed challenges in creating market linkages and were reluctant to invest their own funds in maintenance and repair of the units following project close￾out. The pilot institutional linkages with SNPs also met with mixed success, with SNP administrators reporting strong rejection of dried fish powder consumption by children and lack of continuity of the programs due to cost and authorization associated with the use of fish and fish products. On the other hand, the project led SBCC, which aimed to foster increased consumption of whole dried and fresh small fish, seems to have been highly successful. The evaluation team found that these nutritional messages built on the high level of pre-existing consumption of small fish and induced a shift among beneficiaries to consumption of whole small fish; therefore, this practice continues following the project close-out in 2021. The mixed results with respect to increasing fish production among farmers and uptake of solar dryer use among WSHGs offer important lessons learned related to the need to tailor selection criteria by objective. First, beneficiary selection needs to consider the willingness of specific beneficiaries to sustain the cost of continued implementation, which in this case largely corresponds to their level of market orientation. By selecting only those beneficiaries that demonstrate an interest in entrepreneurship, projects are likely to reach higher numbers of beneficiaries that will sustain market-oriented activities post project. In turn, achieving improved cost efficiency through better targeting of distributions will allow implementors to facilitate improved sustainability of availability over a wider geographic area. Likewise, the sustainability of solar dryer use is likely to be increased by selection of beneficiary groups (in this case, WSHGs), that demonstrate an understanding of the market linkages and business models underpinning the sustainable use of these units, especially the cash flow required to maintain and repair them. In a less tangible way, linkages with institutional markets also requires engaging beneficiary institutions that have a demonstrated understanding of the potential cost– benefit of adopting new products (in this case, for fish and fish products with SNPs), especially in the pilot phase. On the other hand, the successful SBCC effort around whole fresh and dried fish consumption built on existing consumption patterns. This demonstrates the feasibility of promoting new behaviors that can be adopted with no cost to beneficiaries. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 54 4.2 BUSINESS PLANS AND MARKET LINKAGES COMMERCIAL TECHNOLOGY ADOPTION CAN BE ENHANCED BY COMMERCIAL MARKET LINKAGES AND BUSINESS PLANS. For activities that require beneficiaries to invest their own resources beyond the life of the project, projects can improve the likelihood of sustainability by including a focus on business planning and market linkages. In the case of promoting mola-carp polyculture, this means developing and disseminating models to demonstrate the improved income derived from investment into fish farms, including how to optimally market increased surpluses into existing market systems. In the case of solar dryers, WSHGs operating these units would be more likely to invest in maintenance and repairs required for their continued operation if they had a better understanding of the business models underpinning their use and the ways to leverage existing markets to expand sales. 4.3 DIRECT IMPLEMENTATION VERSUS LEVERAGING MARKET SYSTEMS DIRECT IMPLEMENTATION CAN UNDERMINE SUSTAINABILITY. Some of the project activities relied on project partners to deliver training and post-project TA. However, numerous beneficiaries reported limited access to partner staff, such as F&ARD extension officers, who would provide this TA post-project—a chronic challenge to official extension services in most countries. Interviews with field staff at these institutions confirmed their limited capacity. As an alternative, numerous studies have found that projects can tap embedded market actors, such as buyers and input providers with a vested interest in the project’s venture, as resources for post￾project TA. Furthermore, these studies have also stressed that “timebound” methodologies, such as a training event, do not provide farmers with on-demand access to required technical assistance as needs arise. For instance, coaching is often critical for beneficiaries for whom years may elapse between a training event and the need for specific assistance.72 4.4 CHANGING CONSUMPTION AND TASTE CHANGES IN CONSUMPTION ARE POSSIBLE, BUT LOCAL TASTES NEED TO BE CONSIDERED CAREFULLY. The successful outcomes that resulted from the project- ed SBCC to increase consumption of whole fresh and dried small fish demonstrate that changes in consumption are possible, especially in cases where these changes build on existing consumption habits. However, the strong rejection of dried fish powder by children demonstrates that efforts to change consumption habits must be considered carefully. Specifically in this case, an “exposure visit” to Cambodia was instrumental in convincing officials of the viability of this consumption change. However, the evaluation team notes that fish sauce is an established staple of Cambodian diets that is not present in the diets of the population of Odisha. 72 For example, see Mars Indonesia – Service Delivery Model, Case Study Report. The Sustainable Trade Initiative (IDH), June 2015 and Rinck, David, Leah Ghoston, Hariyadi Hariyadi and Cininta Pertiwi. MCC Indonesia Green Prosperity Project Sustainable Cacao Partnership Grants Performance Evaluation—Final Evaluation Report. Social Impact, Inc. for Millennium Challenge Corporation, April 2020. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 55 4.5 LEVERAGE CONVERGENCE OF AGENCY OBJECTIVES LEVERAGING THE CONVERGENCE OF AGENCY OBJECTIVES AND ACTIVITIES CAN FACILITATE EFFECTIVE AND EFFICIENT PROJECT IMPLEMENTATION. Throughout project implementation, WorldFish enhanced the efficiency and effectiveness of all activities by leveraging the convergence of project and current government agency objectives and activities of project partners. This included the rapid scale-up of mola-carp polyculture promotion and solar dryer use by building on the existing MOU between WorldFish and the State Department of F&ARD for implementation of the Odisha Fisheries Policy to demonstrate sustainable fish production systems and technologies. It also included the pilot implementation of institutional linkages with SNPs at Anganwadi centers and schools through leveraging of the pre-existing ICDS implemented by the State Department of WCD&MS. In the case of the successful SBCC aimed at increasing consumption of whole dried and fresh small fish, WorldFish built on the existing objectives of multiple agency partners. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 56 5. RECOMMENDATIONS This section presents the recommendations developed by the evaluation team based on the preceding lessons learned. 5.1 TAILORING BENEFICIARY SELECTION Avoid generalized (or geographically defined) distributions and consider ways of integrating beneficiary self-identification into activities for which beneficiaries will need to invest their own resources to sustain activities after project close-out. Strategies to do this may include: a. Require a co-pay from beneficiaries of distributed inputs. b. Require beneficiaries to work with pre-approved advisors to develop a business plan and/or plan for market linkages to qualify as recipients of inputs or funds. In addition to tailoring beneficiary selection to more market-oriented individuals, this approach may also present an opportunity for initial provision of TA, as well as introducing beneficiaries to potential sources of post-project TA. c. Avoid direct input distributions and instead provide qualifying beneficiaries with vouchers that are redeemable at pre-qualified commercial suppliers. This approach may have the additional benefit of establishing market linkages between fish farmers and input vendors, who may also present sustainable sources of inputs and TA following project close-out. Finally, if coupled with project-supplied training of suppliers, this approach may promote market linkage development for recommended inputs. 5.2 BUSINESS PLANS AND MARKET LINKAGES Consider ways to integrate beneficiary business planning and market linkage expansion into activities. Some of the recommendations above may also contribute to this; other options include: 1. Linking beneficiaries to advisory services through vouchers for business and marketing planning services that can be redeemed at pre-approved suppliers. These suppliers may be provided by projects with specializing training in related business and marketing activities. 2. Assessing market opportunities to determine where value-added activities can improve the profitability of product sales and developing relevant sales strategies. 3. Working with input suppliers and buyers to develop mechanisms for forward investment into business activities. This may include quasi-contract farming mechanisms through which suppliers and buyers provide inputs on credit against future purchases from beneficiaries. 5.3 DIRECT IMPLEMENTATION VERSUS LEVERAGING MARKET SYSTEMS Consider ways to integrate input suppliers, buyers, and other upstream value chain actors into projects as sustainable commercially interested TA sources. This could include focusing on addressing supply chain weaknesses in the supply and input chains that impede the adoption of improved production and processing practices and technologies through also addressing linkage to feed seed and feed suppliers (and other inputs), including wholesale suppliers. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 57 In effect, this enhancement may lead to the development of a “hub and spoke” service delivery system similar to those employed by many commodity companies to ensure supply chain integrity. This results in an input and TA service delivery model that would comprise fish seed breeders and wholesale inputs suppliers linked to local input dealers who may be supported by co-funded grants extend input sales and TA delivery and would alleviate challenges related to official extension service capacity. A key strength of this model is efficiency in reaching large numbers of farmers for technology transfer and TA, in addition to provision of inputs and supplies. For example, with one breeder or input wholesaler serving 20-30 local input dealers, and each retail service provider serving 20-30 SHGs of 20-30 farmers each with inputs and TA, this model can effectively serve between 8,000- 27,000 farmers per wholesaler. 5.4 CHANGING CONSUMPTION AND TASTE Continue to implement nutritional SBCC activities around fish and fish product consumption through synergistically integrating these into the existing activities and mechanisms of partner entities. However, carefully consider ways to build on existing local tastes and consumption patterns based on established local diets and products. Also, carefully assess the cost implications of adopting new consumption practices, especially linkages with institutional markets. 5.5 LEVERAGE CONVERGENCE OF AGENCY OBJECTIVES Continue to leverage the convergence of government agency objectives and activities to facilitate effective and efficient project implementation. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 58 ANNEX 1: SCOPE OF WORK EQ1. Evaluation Purpose Projects may meet their objectives by improving economic, health, or social conditions while they are operating, but genuine success is achieved only through sustained change that does not depend on continued external resources. The overall purpose of this ex-post evaluation is to assess sustainability, and to know how the outcomes and impacts evolved after two years of completing the project. Additionally, the evaluation findings will be used to guide the future programs in the sector. EQ2. Evaluation Questions The contractor must, at a minimum, address the following evaluation questions. • To what extent and how did the project reach its objectives during implementation? • What interventions were more successful and/or had a greater contribution to accomplish the objectives? • Are the promoted behaviors about using innovative, nutrition-sensitive fisheries technologies and integrated approaches to improve food and nutrition security continuing to be practiced by the stakeholders and beneficiaries? If so, how? If not, what are the factors that have hindered the practice to continue? • Have the Implementing partners / stakeholders strengthened by USAID funding continued to implement the action? What is the beneficiaries’ perspective / feedback about the impact of the project? • What are the factors that contributed to or impaired the long-term sustainability of the project outcomes and outputs? EQ3. Introduction and Background: Despite the good economic performance, with over 200 million people who are food insecure, India is home to the largest number of hungry people in the world. In the ranking of the Global Hunger Index 2019, India is ranked 117. On the other hand, India is second in global fish production, producing over 12 million metric tons of fish. But utilizing fish to fight hidden hunger and malnutrition has not been attempted in any significant way in India. In the view of the need to improve food and nutrition security in the state of Odisha, India. The United States Agency for International Development (USAID) partnered with Public International Organization (PIO) WorldFish is a premier research and development organization and is one of the 15 Consultative Group of International Agricultural Research (CGIAR) Centers. The goal of the project namely, ‘Scaling nutrition- sensitive fisheries technologies and integrated approaches through partnership in Odisha” was to improve food and nutrition security in the Indian state of Odisha through increasing availability, accessibility and consumption of fish and fish products. This project led to the introduction of nutrition-sensitive production technologies for nutrient-rich fish and vegetables in selected districts of Odisha as well as increased production of high quality fresh small fish and dried fish for making fish-based products. The project was implemented in close partnership with the Fisheries and Animal Resources Development (F&ARD), Department of Government of Odisha and several other public and private partner institutions in Odisha. The original life of the project was three years (October 2017 – September 2020) that was extended Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 59 by six months from September 30, 2020, to March 31, 2021. The brief descriptions of the activities are as follows. Description of ‘Scaling nutrition- sensitive fisheries technologies and integrated approaches through partnership in Odisha” project The overall goal of this project was to improve food and nutrition security in the state of Odisha, India. The development objective of this project was to increase the supply of and access to affordable, safe, nutrient-rich fish and fish products for increased consumption, through innovative, nutrition-sensitive fisheries technologies and integrated approaches. The project strategically focused on for components as discussed below: • Increased availability - Increased production, productivity, and diversity of nutrient-rich fish from innovative technologies of aquaculture and capture fisheries, as applied to household and community ponds, and inland freshwater reservoirs. • Increased accessibility - Development, marketing and distribution of affordable, safe, nutritious fresh fish and fish-based products using production from aquaculture and capture fisheries, including Chilika Lake. Increased and expanded distribution to reach multiple population groups, in partnerships with the private sector and research institutes. • Increased consumption - Social behavior change approaches, targeting the whole family, to promote increased nutrient-rich fish and vegetable consumption, especially in women and young children; and increase knowledge and practice of essential nutrition and essential hygiene actions, in partnerships with relevant local government programs, non- government organizations (NGOs) and community-based organizations (CBOs). • Improved food and nutrition security and scaling - Further scaling of innovative, nutrition-sensitive fisheries technologies and integrated approaches, through expanding and strengthening regional and national partnerships and collaborations, in particular, in West Bengal and Terai, Nepal. EQ4. Evaluation Design / Methods The evaluation will use both quantitative and qualitative methods for data collection and analysis. The design will include a sample survey to assess the program impact on the beneficiaries and Key Information Interviews will also be conducted to gain insights of perceived triggers and barriers in increasing the supply of and access to affordable, safe, nutrient-rich fish and fish products for increased consumption. Prior to conducting field visits for primary data collection, the evaluation team will conduct an extensive review of documents, including, annual work plans, project reports (annual reports and quarterly progress reports), activity monitoring and evaluation plan, and other related documents. The contractor conducting this assessment will gather a wide range of background information from USAID and the Implementing Partner to ensure that the findings and recommendations are based on an accurate understanding of the program, Key Informants’, and beneficiaries’ feedback. The contractor must describe the evaluation design and methodology in detail. EQ5. Deliverables and Timeline The key deliverables of the evaluation are as follows. The contractor must adhere to the timeline of deliverables. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 60 Inception report: The inception report must describe the conceptual framework the evaluator will use to undertake the evaluation and the justification for selecting this approach. It must detail the evaluation methodology (i.e., how each question will be answered by way of data collection methods, data sources and sampling). Detailed Assessment Methodology: The contractor must describe the methodological approach in detail. The description of the proposed methodology must detail the methods of data collection, sampling strategy / plan, and data analysis plan. Debriefing Meeting: The Contractor must debrief USAID on the preliminary findings of the assessment. This meeting must provide a summary of any analytical results; and discuss challenges, successes, and way forward. The Contractor must deliver an oral presentation of the findings. Final Assessment Report: The contractor must submit a final report that is based on analyzed facts and evidence and fully addresses all the assessment questions. EQ6. Composition of the Evaluation Team The contractor must propose a diverse evaluation team expert in different key components of the evaluation. As a means of building local capacity to undertake evaluations, the team composition should have at least one local expert as a key member of the evaluation team. EQ7. Scheduling and Logistics The Contractor will be responsible for all logistics including coordinating all travel throughout field work, data analysis and report submission to USAID. EQ8. Budget The contractor must propose the detailed budget aligned with each technical component such as evaluation design, sampling plan, data collection and analysis, evaluation team composing etc. Note: USAID/India will share the necessary documents, reports with the contractor required to conduct the ex-post evaluation. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 61 ANNEX 2: EVALUATION TIMEFRAME Table 16: Evaluation Timeline and Deliverable Dates a. Bold indicates deliverable to USAID/India. ACTIVITY / DELIVERABLE /A DATE (2022/2023) Begin Literature Review and Gap Analysis November 21-December 9 DRAFT Inception Report submitted to Panagora December 16 Kick-Off Meeting with USAID/India TBD DRAFT Inception Report submitted to USAID/India December 23 FINAL Inception Report submitted to USAID/India TBD FINAL Inception Report approved by USAID/India TBD Begin weekly check-in meetings with USAID/India TBD Provisional List of Specific KII Respondents submitted to USAID/India TBD Phase 1 (remote) Data collection and analysis December 26-January 15 Data Collection Protocols finalized and summitted to USAID/India January 14 Phase 2 (on-site) Data collection and analysis in Odisha January 14-February 6 Consultative Presentation ppt submitted to USAID/India February 6 Consultative Presentation with USAID/India February 10 Produce Draft Evaluation Report February 8-March 11 Draft Evaluation Report submitted to USAID/India March 21 USAID/India reviews DRAFT report and provides comments March 12-March 21 Evaluation team reviews comments and finalizes report March 22-March 30 Final Evaluation Report submitted to USAID/India March 31 Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 62 ANNEX 3: EVALUATION TEAM SUMMARY Table 17: Summary of Evaluation Team LOE and Roles TEAM MEMBER/a LOE ROLES Team Leader – David Rinck 60 • Oversee implementation of the evaluation. • Oversee the overall design of the evaluation framework. • Oversee scheduling of fieldwork. • Lead the literature review and gap analysis. • Supervise KIIs, FGDs, mini survey and site visits. • Manage other data collection activities as needed. • Lead data analysis. • Lead development and delivery of the presentation of initial findings. • Lead formulation of conclusions, lessons learned and recommendations. • Lead development of the draft and final evaluation reports. Soumik Kundu - Agriculture/Evaluation Specialist 45 • Participate in the literature review and gap analysis. • Maintain schedule of data collection activities and lead design of the logistics plan under the supervision of the Team Leader. • Lead implementation of the mini survey and site visits in coordination with the Team Leader. • Provide regular updates on emerging findings derived from all data collection sources. • Provide input into data analysis activities. • Provide input into the presentation of initial findings. • Provide input into draft and final evaluation reports. Ankita Babbar - Logistics Coordinator 20 • Lead scheduling of data collection activities, including KIIs, FGDs and site visits under supervision of the Team Leader. • Support logistics of the evaluation team in line with the logistics plan to data collection sites in and around Odisha under the direction of the Team Leader. • Maintain regular contact with the team to ensure efficiency of fieldwork. a. In addition, the team contracted two drivers and four Oriya-speaking translators. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 63 ANNEX 4: REFERENCE CITATIONS Cullhaj, Megi “The technical collaboration between WorldFish and the Government of Odisha (India) reaches nutrition and income goals for over 151,407 people” WorldFish Website, April 14, 2022: mel.cgiar.org “Draft Monitoring and Evaluation Plan prepared by MSSRF for Project Proposal on Scaling innovative, nutrition-sensitive fisheries technologies and integrated approaches through partnerships in Odisha, India can improve food and nutrition security by WorldFish funded by USAID” MSSRF for WorldFish. “Financial Report – Period from 01/10/2017 to 31/12/2018” WorldFish for USAID/India, February 13, 2019. “Financial Report – Period from 01/10/2017 to 31/03/2021” WorldFish for USAID/India, April 14, 2021. Hamzaoui, Mustapha “Project Concept Note on “Scaling innovative and nutrition-sensitive aquaculture and wild fisheries production technologies through partnership in the Indian states of Odisha and Assam” submitted by WorldFish in February 2017 in response to India Partnership APS” USAID for WorldFish, April 3, 2017. “Hygienic dried fish production using polyhouse solar dryers - Odisha, India” WorldFish and Mission Shakti, March 2022. “Inclusion of Dried Small Fish in Supplementary Nutrition Programme: Pilot Study in Odisha, India – Baseline Report” MSSRF for WorldFish, May 2021. “Inclusion of dried small fish in the ICDS Supplementary Nutrition Programme - Pilot Study in Odisha, India” WorldFish. “Inclusion of Dried Small Fish in Supplementary Nutrition Programme and Nutritional Analysis of the Meals Along with Fish Powders and Dried Fish: Pilot Study in Odisha, India - Report” MSSRF for WorldFish, December 2021. King, Gary, Robert Keohane, and Sydney Verba “Designing Social Inquiry: Scientific Inference in Qualitative Research’ Princeton University Press” Princeton University Press, 2016. “Locally Led Development: Partnering for Improved Nutrition - Lessons from Odisha, India” JSI Research & Training Institute, Inc. for USAID Feed the Future, August 2021. “Mars Indonesia – Service Delivery Model, Case Study Report” The Sustainable Trade Initiative (IDH), June 2015. “Mid-term assessment of the project Scaling nutrition-sensitive fisheries technologies and integrated approaches through partnerships” USAID, 2018. Nayak, Rashmi Ranjan “Feasibility of inclusion of nutritious Fish in Supplementary Nutrition Programme (SPN) in the state of Odisha” Government of Odisha, Department of Women and Child Health and Mission Shakti to WorldFish, August 20, 2019. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 64 “Notes for USAID/India team assessing the project – Scaling nutrition-sensitive fisheries technologies and integrated approaches through partnership in Odisha, India can improve food and nutrition security” WorldFish for USAID/India. “Odisha-WorldFish Project - Action Plan for Phase 1 (April 2017-March 2019)” WorldFish for Fisheries and Animal Resources Development Department, Government. of Odisha, September 26, 2017. “Panagora Group Human Research Policy” Panagora Group, June 9, 2022. Ratha, B.C., P.A. Padiyar, N. Shenoy, C.V. Mohan “Scaling nutrition-sensitive fisheries technologies and integrated approaches through partnership in Odisha: USAID-IPP Funded Project (Oct 2017- Mar 2021) - Project Final Report for the period October 2017 to March 2021” WorldFish for USAID, April 2021. Reddy, Vamsidhar “Action Memo: Approval to extend the period of performance of the Scaling nutrition sensitive fisheries technologies and integrated approaches through partnership in Odisha project” PIO Grant No. AID-BFS-IO-17-00005 by six months from September 2020 to March 31, 2021” USAID, July 15, 2020. Reddy, Vamsidhar “Key observations and recommendations for the project on Scaling Nutrition-sensitive fisheries technologies and approaches through partnerships from the Mid-term assessment conducted during March 25-29, 2019” USAID, March 2019. Reddy, Vamsidhar “Performance Narrative - Scaling innovative, nutrition sensitive fisheries technologies and integrated approaches through partnerships in Odisha, India – WorldFish” USAID/India, November 16, 2018. “Revised Guidelines for Implementation of Take-Home Ration 2018 (Supplementary Nutrition Programme of ICDS and Scheme for Adolescent Girls)” Department of MDC&MS. Rinck, David, Leah Ghoston, Hariyadi and Cininta Pertiwi “MCC Indonesia Green Prosperity Project Sustainable Cacao Partnership Grants Performance Evaluation - Final Evaluation Report” Social Impact, Inc. for Millennium Challenge Corporation, April 2020. Rinck, David “USAID-India Agriculture and Food Security Strategy Assessment” Panagora Group for USAID/India, November 9, 2021. “Scaling innovative, nutrition-sensitive fisheries technologies and integrated approaches through partnerships in Odisha, India can improve food and nutrition security - A project proposal” WorldFish submitted to USAID, India, May 2017. “Scaling innovative, nutrition-sensitive fisheries technologies and integrated approaches through partnerships in Odisha, India can improve food and nutrition security – Baseline Survey (Nov 2018-Jan2019)” MSSRF for WorldFish, March 2019. “Scaling innovative, nutrition-sensitive fisheries technologies and integrated approaches through partnerships in Odisha, India can improve food and nutrition security – Final Monitoring Report” MSSRF for WorldFish, January 2021. “Scaling innovative, nutrition-sensitive fisheries technologies and integrated approaches through partnerships in Odisha, India can improve food and nutrition security – Project Monitoring Report (April-September 2019)” MSSRF for WorldFish, 2019. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 65 “Scaling innovative, nutrition-sensitive fisheries technologies and integrated approaches through partnerships in Odisha, India can improve food and nutrition security – Project Monitoring Report (October 2019-March 2020)” MSSRF for WorldFish, 2020. “Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha – Fact Sheet” WorldFish. “Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha - Project Brief” WorldFish, April 2020. Annual Reports “Annual Report: Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha: USAID IPP Funded Project (Oct 2017-Sept 2018)” WorldFish for USAID/India, Penang, Malaysia, November 9, 2018. “Annual Report: Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha: USAID IPP Funded Project (Oct 2018-Sept 2019)” WorldFish for USAID/India, Penang, Malaysia, October 30, 2019. “Annual Report: Scaling Nutrition-Sensitive Fisheries Technologies and Integrated Approaches Through Partnership in Odisha: USAID IPP Funded Project (Oct 2019-Sept 2020)” WorldFish for USAID/India, Penang, Malaysia, October 31, 2020. “Supporting Documents for the Semi-Annual Report for the Period Oct 2019-Sept 2020” WorldFish for USAID/India, Penang, Malaysia. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 66 ANNEX 5: GAP ANALYSIS PROJECT COMPONENTS This sub-section presents gaps in available literature related to EQ 1. To what extent and how did the project components reach their objectives during implementation? These gaps are presented using the latest available data as presented in the final project annual report submitted on October 31, 2020. Literature Review Gaps by Available Indicator According to data presented in the final project annual report submitted on October 31, 2020, as of September 2020, 789 households were practicing carp-mola polyculture, falling short of the project target of 2,000 households by 1,211 households. • How many additional households were practicing carp-mola polyculture by the end of the extended project on to March 31, 2021? • What were the reasons for the shortfall in the number of households producing carp-mola polyculture? According to the same data, 22 community ponds were stocking mola and other fish, falling short of the project target of 24 ponds by two ponds. • How many additional ponds were stocking mola and other fish by the end of the extended project on to March 31, 2021? • According to project data, by September 2019, 22 ponds had begun stocking mola, but this number did not increase during the remainder of the reporting period. What were the reasons for which no additional ponds began stocking mola in this period? According to the same data, only one women’s group was engaged in fish drying by September 2020, and this group began fish drying between October 2019-September 2020, falling short of the project target of 40 women’s groups by 39. • How many additional women’s groups engaged in fish drying by the end of the extended project on to March 31, 2021? • What were the reasons for the significant shortfall in number of women’s groups engaged in fish drying? According to the same data, zero Anganwadi centers using fish products in mother and child health care by September 2020, falling short of the project target of 50 groups. • Did any Anganwadi groups begin using fish products in mother and child health care by the end of the extended project on to March 31, 2021? • What were the reasons that the Anganwadi groups did not begin using fish products in mother and child health care? According to the same data, 24 schools were using fish in mid-day meal programs, falling short of the project target of 50 schools by 26. • How many additional schools were using fish in mid-day meal programs by the end of the extended project on to March 31, 2021? • What were the reasons for the shortfall in number of schools using fish in mid-day meal programs? Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 67 Data Gaps • Document request: Is there data available related to additional project objectives as laid out in the project proposal? INTERVENTION SUCCESS FACTORS This sub-section presents gaps in available literature related to EQ 2. What interventions were more successful and/or had a greater contribution to accomplishing project goal(s) by project component (see Table 1: Project Components and Objectives)? This section considers the stated project goal, as laid out in the initial proposal, as well as USAID/India sub-goals, objectives and intermediate results (IRs) as laid out in the mission CDCS, as follows: • Project goal: “improve food and nutrition security in the Indian state of Odisha through increasing availability, accessibility and consumption of fish and fish products.” • CDCS Sub-Goal 2: “Innovations accelerate development outcomes in India and globally.” • CDSC Objective: “CDCS role in contributing to global development enhanced.” • CDSC IR 4.1 “Indian innovations for development impact shared with other countries.” Potential sources of data are noted in parenthesis. Gaps by Project Component and Objective Component 1: Increased Availability • 450 households practicing carp-mola polyculture in homestead ponds. • Local community organizations begin culturing mola and other small fish in six community ponds/reservoirs. • Micronutrient-rich vegetables and orange sweet potato produced in at least 85 percent of households practicing carp-mola pond polyculture. Component 1 Gaps • What changes (expansion/reduction) in carp-mola polyculture occurred on homestead farms between the last reporting period and the end of the extended project timeframe and currently (USAID/India, WorldFish and partners, beneficiary farmers)? • What progress has occurred as a result of government of Odisha funding committed for further development of ponds or village bodies of water, and has this extended carp-mola polyculture (Department of F&ARD)? • What changes (expansion/reduction) in mola and other fish culturing in the six community farms occurred between the end of the extended project timeframe and currently (USAID/India, WorldFish and partners, beneficiary farmers)? • What progress has occurred as a result of government of Odisha funding committed for further development of ponds or village bodies of water and have additional ponds been developed (Department of F&ARD)? • What changes (expansion/reduction) in micronutrient-rich vegetables and orange sweet potato production have occurred in households between the end of the extended project timeframe and currently (USAID/India, WorldFish and partners, beneficiary farmers)? • What changes in micronutrient-rich vegetables and orange sweet potato production have occurred as a result of introduction of these products into SNPs in Mayurbhanj District and Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 68 other sites in Odisha, school feeding programs and sales of OSP vines to Centro Internacional de la Papa (the International Potato Center) (Department of WCH, OLM, Mission Shakti)? Component 2: Increased Accessibility • Five branded, nutritious, safe, fresh fish and fish-based products available in rural and urban markets. • 20 women’s groups producing high quality dried fish from Chilika Lake and reservoirs. Component 2 Gaps • How has availability of nutritious, safe, fresh fish and fish-based products in rural and urban markets changed between the end of the extended project timeframe and currently (USAID/India, WorldFish and partners, beneficiary farmers, WSHGs)? • What progress has been occurred in the adoption of solar dryers by WSHGs since signing of the MOU with to provide these with ICAR-CIFT in Cochin (USAID/India, WorldFish and partners, Department of F&ARD, Department of MSMEs, WSHGs)? • What progress has occurred in scale-up of solar dryer use under the One-District One￾Product program through financial assistance, credit linkages and market linkage support to the WSHGs for at least 100 solar fish dryers over the next two years (USAID/India, WorldFish and partners, Department of F&ARD, Department of MSMEs, WSHGs)? • What is the status of market linkages proposed with private sector companies for marketing of fish-based products, which WorldFish and partners surveyed during the first year of project implementation (USAID/India, WorldFish and partners, Department of F&ARD)? o Document request: Final copy of market survey. Component 3: Increased Consumption of Nutritious Foods • Quantity and frequency of small fish intake increased in the diet of at least 5,000 children and 5,000 women. • 25 schools with fish in midday meals. • 25 Anganwadi centers distributing fish products for the first 1000 days of life. Component 3 Gaps • What changes have occurred in the quantity and frequency of small fish intake between the end of the extended project timeframe and currently (USAID/India, WorldFish and partners, Department of F&ARD, Department of WCH, OLM, Mission Shakti, beneficiary farmers)? • What changes have occurred at schools offering fish in midday meal between the end of the extended project timeframe and currently (Department of F&ARD, Department of WCH, OLM, Mission Shakti, beneficiary farmers)? • What changes have occurred at Anganwadi centers distributing fish between the end of the extended project timeframe and currently (Department of F&ARD, Department of WCH, OLM, Mission Shakti, beneficiary farmers)? Component 4: Improved Food and Nutrition Security in Odisha and Scaling • Two states in India and two countries in the region adopt nutrition-sensitive technologies and integrated approaches in the fisheries sector. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 69 Component 4 Gaps • What progress has occurred in the adoption of nutrition-sensitive technologies and integrated approaches in the fisheries sector in other Indian states and in Nepal and Bangladesh since the end of the project period (USAID/India, Department of F&ARD in target Indian states, USAID/Nepal, USAID/Bangladesh, WorldFish and partners, partner government agencies and PIOs in Nepal and Bangladesh)? CONTINUED USE AND HINDERING FACTORS This sub-section addresses gaps in available literature related to EQ 3. Are project beneficiaries and stakeholders continuing to use promoted behaviors related to the use of innovative, nutrition￾sensitive fisheries technologies and integrated approaches to improve food and nutrition security? If so, how? If not, what factors hinder continuation of these practices? Potential sources of data are noted in parenthesis. • Since the project close-out, what alternative sources of TA are available to support on-going activities (Department of F&ARD, Department of WCH, OLM, Mission Shakti, beneficiary farmers, beneficiary farmer survey, WSHGs, WSHG survey)? • Have beneficiary farmers continued to carp-mola polyculture and vegetable production in homestead ponds? If so, how? If not, what factors hinder continued production (Department of F&ARD, Department of WCH, beneficiary farmers, beneficiary farmer survey)? • Have beneficiary farmers continued to cultivate mola in community ponds/reservoirs? If so, how? If not, what factors hinder continued cultivation (Department of F&ARD, Department of WCH, beneficiary farmers, beneficiary farmer survey)? • Have beneficiary WSHGs continued to use solar dryers supplied by ICAR-CIFT in Cochin? If so, how? If not, what factors hinder continued use (Department of F&ARD, Department of WCH, WSHGs, WSHG survey)? • Have beneficiary households continued to consume fish and fish-based products? If so, how? If not, what factors hinder continued consumption (Department of F&ARD, Department of WCH, beneficiary farmers, beneficiary farmer survey, WSHGs, WSHG survey)? CONTINUED IMPLEMENTATION This sub-section addresses gaps in available literature related to EQ 4. Have the implementing partners / stakeholders strengthened by USAID funding continued to implement project components? What is the beneficiaries’ perspective / feedback about project impact? Potential sources of data are noted in parenthesis. • This EQ will be addressed during field research. SUSTAINABILITY FACTORS This sub-section addresses gaps in available literature related to EQ 5. What are the factors that contributed to or impaired the long-term sustainability of project outcomes and outputs? Potential sources of data are noted in parenthesis. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 70 • Economic and logistic feasibility of carp-mola polyculture and vegetable production in homestead ponds and farmer perspectives (Department of F&ARD, Department of WCH, beneficiary farmers, beneficiary farmer survey)? • Economic and logistic feasibility of mola cultivation in community ponds/reservoirs and farmer perspectives (Department of F&ARD, Department of WCH, beneficiary farmers, beneficiary farmer survey)? • Economic and logistic feasibility of solar dryer use and WSHG perspectives (Department of F&ARD, Department of WCH, WSHGs, WSHG survey)? • Economic and logistic feasibility of fish distribution in SNPs in Mayurbhanj District and other sites in Odisha, school feeding programs and Anganwadi centers (Department of F&ARD, Department of WCH, OLM, Mission Shakti)? Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 71 ANNEX 6: DATA COLLECTION PROTOCOLS INFORMED CONSENT Thank you for taking the time to meet with us today. My name is [NAME]. I am a researcher from a company called Panagora Group, which is based in the United States. Our team is speaking with people to evaluate a project about fish and vegetable farming that WorldFish implemented until March 2021. The project has now ended, and we are interested in learning about how outcomes and impacts have evolved since completion of the project. We would like to conduct a brief discussion with you today to learn about your experience with this project. Your responses and responses from other participants will be compiled into recommendations for a report to help improve future projects in similar areas. The report will be publicly available once it is complete, but it will not include your name or other identifying information. Readers will not be able to identify the specific individuals we spoke to from any specific quotes or data in the report. It is important to understand that while we would like your help in this study, you do not have to participate if you do not want to, and you do not have to answer any of our questions if you feel uncomfortable doing so. Please note that we plan to record this interview. The recordings will be used to transliterate the interview so that we can review the content later. The recordings will not be shared with any third party and will be erased following completion of our report. This interview is expected to take about 60 minutes. You may ask questions at any time during our discussion. If you have questions or concerns about the research after we leave today, you can contact me at [EMAIL] or [PHONE NUMBER]. Do you have any questions before we start? By saying “yes,” and participating in this study, you are indicating that you have heard this consent statement, had an opportunity to ask any questions about your participation, and voluntarily consent to participate. Will you participate in this interview? You may answer yes or no. ◻ Yes, I will participate ◻ No, I will not participate Are you okay with us recording the interview? You may answer yes or no. ◻ Yes, I am okay with recording the interview ◻ No, I am not okay with recording the interview Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 72 KII GUIDE – DONOR STAFF Interview date: Interviewer: Respondent name: Respondent organization: Respondent job title: Background (project role and experience) EQ 1. Project Components - To what extent and how did the project components reach their objectives during implementation? • According to data presented in the project close out report submitted on April 2021, as of the end of the project, 789 households were practicing carp-mola polyculture, falling short of the project target of 2,000 households by 1,211 households. o What were the reasons for the shortfall in the number of households producing carp-mola polyculture? o How effective were the fishponds in improving the availability of nutrient rich fish? • According to the same data, 22 community ponds were culturing mola and other fish, falling short of the project target of 24 ponds by two ponds. o What were the reasons for the shortfall in the number of ponds stocking mola? o How effective were the community ponds in improving the availability of nutrient rich fish? • According to the same data, 11 WSHGs were engaged in fish drying using solar dryers by the end of the project period, falling short of the project target of 40 women’s groups by 29. o What were the reasons for the shortfall in number of WSHGs engaged in fish drying? o How effective were the solar dryers in improving the accessibility of nutrient rich fish? • According to the same data, 50 Anganwadi centers using fish products in mother and child health care by the end of the project, meeting the target of 50. o How effective was the use of fish products by the Anganwadi centers in mother and child health care in improving consumption of fish? • According to the same data, 24 schools were using fish in mid-day meal programs, falling short of the project target of 50 schools by 26. o What were the reasons for the shortfall in number of schools using fish in mid-day meal programs? o How effective was the use of fish products by the school feeding programs in improving the consumption of fish? • According to the same data, 19 SBCC materials were developed to promote consumption of fish. o How effective were the SBCC materials in promoting the consumption of fish? • According to the same data, the project signed 5 MOUs, entered into 11 partnerships, undertook 4 activities to influence policy, made 3 cross-country visits, and trained 3,120 government officials. o How effective were these activities in promoting consumption of fish and fish products? o How effective were these activities in promoting scaling of approaches to other states of India and regionally? EQ 2. Intervention Success Factors - What interventions were more successful and/or had a greater contribution to accomplishing project goal(s)? Availability • What progress has occurred as a result of government of Odisha funding committed for further development of carp-mola polyculture? Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 73 • What progress has occurred as a result of government of Odisha funding committed for further development of ponds or village bodies of water and have additional ponds been developed? • What changes in production of micronutrient-rich vegetables and OSP production have occurred as a result of introduction of these products sites in Odisha? Accessibility • How has accessibility to nutritious, safe, fresh fish and fish-based products in rural and urban markets changed due to project activities? • What progress has occurred, and is likely to occur in next two years, in scale-up of solar dryer use through financial assistance, credit linkages and market linkage support? Increased Consumption of Nutritious Foods • What changes have occurred, and is likely to occur in the next two years, in the quantity and frequency of small fish consumption? SUPPLEMENT – STAFF WITH KNOWLEDGE OF SCALING TO ADDITIONAL AREAS Improved Food and Nutrition Security in Odisha and Scaling • What progress has occurred, and is likely to occur in the next two years, in the adoption of nutrition￾sensitive technologies and integrated approaches in the fisheries sector in other Indian states and in Nepal and Bangladesh? EQ 3. Continued Use and Hindering Factors - Are project beneficiaries and stakeholders continuing to use promoted behaviors related to use of innovative, nutrition-sensitive fisheries technologies and integrated approaches to improve food and nutrition security? If so, how? If not, what factors hinder continuation of these practices? • Have beneficiary farmers continued to carp-mola polyculture and vegetable production in homestead ponds? If so, how? If not, what factors hinder continued production? • Have beneficiary farmers continued to cultivate mola in community ponds/reservoirs? If so, how? If not, what factors hinder continued cultivation? • Have beneficiary WSHGs continued to use solar dryers supplied by ICAR-CIFT in Cochin? If so, how? If not, what factors hinder continued use? • Have beneficiary households continued to consume fish and fish-based products? If so, how? If not, what factors hinder continued consumption? EQ 4. Continued Implementation - Have the implementing partners / stakeholders strengthened by USAID funding continued to implement project components? What is the beneficiaries’ opinion / feedback about project impact? • Since the project close-out, what alternative sources of TA are available to support on-going activities? • Which organization(s) has continued to support beneficiary farmers in carp-mola polyculture and vegetable production in homestead ponds, and how? • Which organization(s) has continued to support beneficiary farmers in cultivating mola in community ponds/reservoirs, and how? • Which organization(s) has continued to support beneficiary WSHGs to use solar dryers, and how? • Which organization(s) has continued to support beneficiary households to consume fish and fish￾based products, and how? Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 74 o Probe for Anganwadi Centers, SNPs, school feeding. EQ 5. Sustainability Factors - What are the factors that contributed to or impaired the long￾term sustainability of project outcomes and outputs? a. Economic and logistic feasibility of carp-mola polyculture and vegetable production in homestead ponds and farmer opinion? b. Economic and logistic feasibility of mola cultivation in community ponds/reservoirs and farmer opinion? c. Economic and logistic feasibility of solar dryer use and WSHG opinion? Economic and logistic feasibility of fish distribution in SNPs in Mayurbhanj District and other sites in Odisha, school feeding programs and Anganwadi centers? Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 75 KII GUIDE – IMPLEMENTING PARTNER STAFF Interview date: Interviewer: Respondent name: Respondent organization: Respondent job title: Background (project role and experience) SELECT ONLY QUESTIONS RELEVENT TO SPECIFIC ACTIVITES OF INTERVIEWEE(S) • Project Components - To what extent and how did the project components reach their objectives during implementation? • Household carp-mola polyculture and vegetable gardens, including OSP: o What were the challenges faced by your organization in promoting adoption of household carp￾mola polyculture, vegetable gardens and OSP? o What were the challenges faced by beneficiaries in adopting household carp-mola polyculture, vegetable gardens and OSP? o How effective were the fishponds, vegetable gardens in improving the availability of nutrient rich fish and vegetables? • Community ponds culturing mola and other fish: o What were the challenges faced by your organization in promoting adoption of fish culturing? o What were the challenges faced by beneficiaries in adopting fish culturing? o How effective were the community ponds in improving the availability of nutrient rich fish? • WSHGs fish drying using solar dryers: o What were the challenges faced by your organization in promoting use of solar dryers for fish products? o What were the challenges faced by beneficiaries in adopting solar dryers for production of fish products? o How effective were the solar dryers in improving the accessibility of nutrient rich fish? • Anganwadi centers using fish products in mother and child health care: o What were the challenges faced by your organization in promoting use of fish products at Anganwadi centers? o What were the challenges faced by beneficiaries in adopting fish products at Anganwadi centers? o How effective was the use of fish products by the Anganwadi centers in mother and child health care in improving consumption of fish? • School using fish in mid-day meal programs: o What were the challenges faced by your organization in promoting use of fish products in school feeding programs? o What were the challenges faced by beneficiaries in fish products in school feeding programs? o How effective was the use of fish products by the school feeding programs in improving the consumption of fish? • SBCC materials to promote consumption of fish. o How effective were the SBCC materials in promoting the consumption of fish, which products? • MOUs, partnerships, policy influence, 3 cross-country visits, government training. o What were the challenges faced by your organization in promoting use of fish products through outreach activities? o What were the challenges faced by trainees in adopting the use of fish products through outreach activities? Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 76 • Intervention Success Factors - What interventions were more successful and/or had a greater contribution to accomplishing project goal(s)? Availability • What progress has occurred, and are likely to occur in the next two years, as a result of government of Odisha funding committed for further development of carp-mola polyculture? • What progress has occurred, and are likely to occur in the next two years, as a result of government of Odisha funding committed for further development of ponds or village bodies of water and have additional ponds been developed? • What changes in production of micronutrient-rich vegetables and OSP production have occurred, and are likely to occur in the next two years, as a result of introduction of these products in Odisha? Accessibility • How has accessibility to nutritious, safe, fresh fish and fish-based products in rural and urban markets changed due to project activities? • What progress has occurred, and is likely to occur in the next two years, in scale-up of solar dryers? Increased Consumption of Nutritious Foods • What changes have occurred, and are likely to occur in over the next two years, in the quantity and frequency of small fish consumption, and why? SUPPLEMENT – STAFF WITH KNOWLEDGE OF SCALING TO ADDITIONAL AREAS Improved Food and Nutrition Security in Odisha and Scaling • What progress has occurred, and is likely to occur in the next two years, in the adoption of nutrition￾sensitive technologies and integrated approaches in the fisheries sector in other Indian states and in Nepal and Bangladesh? Continued Use and Hindering Factors - Are project beneficiaries and stakeholders continuing to use promoted behaviors related to use of innovative, nutrition-sensitive fisheries technologies and integrated approaches to improve food and nutrition security? If so, how? If not, what factors hinder continuation of these practices? • Have beneficiary farmers continued to carp-mola polyculture and vegetable production in homestead ponds? If so, how? If not, what factors hinder continued production? • Have beneficiary farmers continued to cultivate mola in community ponds/reservoirs? If so, how? If not, what factors hinder continued cultivation? • Have beneficiary WSHGs continued to use solar dryers? If so, how? If not, what factors hinder continued use? • Have beneficiary households continued to consume fish and fish-based products? If so, how? If not, what factors hinder continued consumption? • Continued Implementation - Have the implementing partners / stakeholders strengthened by USAID funding continued to implement project components? What is the beneficiaries’ opinion / feedback about project impact? Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 77 • Since project close-out, what alternative sources of TA are available to support on-going activities? • How has your organization continued to support beneficiary farmers in carp-mola polyculture and vegetable production in homestead ponds, and how? • How has your organization continued to support beneficiary farmers in cultivating mola in community ponds/reservoirs, and how? • How has your organization continued to support beneficiary WSHGs to use solar dryers, and how? • How has your organization continued to support beneficiaries to consume fish and fish-based products, and how? • Sustainability Factors - What are the factors that contributed to or impaired the long-term sustainability of project outcomes and outputs? • Economic and logistic feasibility of carp-mola polyculture and vegetable production in homestead ponds and farmer opinions? • Economic and logistic feasibility of mola cultivation in community ponds/reservoirs? • Economic and logistic feasibility of solar dryer use and WSHG opinions? Economic and logistic feasibility of fish distribution in SNPs in Mayurbhanj District and other sites in Odisha, school feeding programs and Anganwadi centers? Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 78 KII / FGD GUIDE – PROJECT BENEFICIARIES Interview date: Interviewer: Respondent name(s): Respondent(s) location: Beneficiary type (SELECT - household fishpond/garden owners, community fishpond staff, stocking pond staff, solar dryer WSHG members) Background (previous experience prior to project) • Prior to your engagement in project activities, how important were fish/vegetables to your household diet, and if/how were they consumed (probe around sources, types and quantities)? • Prior to your engagement in project activities, how important were fish/vegetables to your household income, and if/how were they sold (probe around types and quantities)? • Prior to your engagement in project activities, what sources of technical assistance did you have access to (probe around types and quality)? • Prior to your engagement in project activities, did your household experience periods over the year during which you were forced to change your diet due to scarcity or expense? 1. Availability • How did your household’s income change as a result of project assistance (probe around changes in quantity sold and how)? • What was the most important assistance you received through the project? • What challenges did you encounter when trying to implement the assistance you received? • Since the end of the project, what alternative source of technical assistance do you have? 2. Accessibility • How likely are you to continue to practice the activities the project promoted, and why (probe around access to tastes, inputs, costs, etc.)? • Which activities have you changed since the end of the project assistance (probe around expansion, adaptation, innovations, cessation, changes in quantities)? 3. Consumption • How did your household’s consumption of fish/vegetables change as a result of project assistance (probe around changes in quantity consumed and how)? 4. Nutrition and Food Security • What new nutrition information did you receive about consumption of fish/vegetables and how useful was this information (probe around if/how this resulted in diet changes)? Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 79 • What were the sources of new information about nutrition you were exposed to (probe around trainings, exposure to SBCC materials)? • (If beneficiaries reported seasonal challenges in food access and/or availability) How has your household’s challenges in maintaining their diet over the year changed as a result of the project activities? Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 80 MINI SURVEY – PROJECT BENEFICIARIES Interview date: Interviewer: Group size and gender (male/female): Group Location: Beneficiary type (SELECT - household fishpond/garden owners, community fishpond staff, stocking pond staff, solar dryer WSHG members) Background (previous experience prior to project) • On a scale of 1-5, please rate the importance of fish to your household diet prior to the project (where 1=not important, 3 = somewhat important, 5 = very important). • On a scale of 1-5, please rate the importance of fish to your household income prior to the project (where 1=not important, 3 = somewhat important, 5 = very important). • On a scale of 1-5, please rate the difficulty your household had in maintaining their diet over the course of a year prior to the project (where 1=not difficult, 3 = somewhat difficult, 5 = very difficult). 1. Availability • On a scale of 1-5, please rate how much your household’s income from fish changed as a result of assistance you received from the project (where 1= income did not change, 3 = income changed a little difficult, 5 = income changed a lot). • On a scale of 1-5, please rate the usefulness of the assistance you received from the project (where 1= not useful, 3 = somewhat useful, 5 = very useful). • On a scale of 1-5, please rate how difficult the assistance you received from the project was to use (where 1= very difficult, 3 = somewhat difficult, 5 = not difficult). 2. Accessibility • On a scale of 1-5, please rate how likely you are to continue to use the advice and resources you received from the project (where 1= not likely, 3 = somewhat likely, 5 = very likely). • On a scale of 1-5, please rate how difficult it is for you to obtain the inputs you require to continue activities you learned through the project (where 1= very difficult, 3 = somewhat difficult, 5 = not difficult). 3. Consumption • On a scale of 1-5, please rate how much your household’s consumption of fish changed as a result of assistance you received from the project (where 1= consumption did not change, 3 = consumption changed a little difficult, 5 = consumption changed a lot). • On a scale of 1-5, please rate the taste of the fish products your household consumes as a result of the project (where 1= poor taste, 3 = average taste, 5 = very good taste). 4. Nutrition and Food Security • On a scale of 1-5, please rate the usefulness of the assistance you received from the project about nutrition (where 1= not useful, 3 = somewhat useful, 5 = very useful). • On a scale of 1-5, please rate how much your diet over the year has changed as a result of the project (where 1= no change, 3 = some change, 5 = very significant change). Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 81 SITE VISIT GUIDE Visit date: Interviewer: Site location: Site type (SELECT - household fishpond/garden, community fishpond, stocking pond, solar dryer site, Anganwadi center, school, etc.): Instructions: Ask for consent to visit site. Let owner(s) know that you wish to take notes and photographs to document your observations. Household fishpond/garden, community fishpond, stocking pond, solar dryer site Note whether the following are present. If yes, ask how use has changed over the course project activities (probe around home consumption and sales). If item is pre-existing, has it been upgraded? Has area and/or productivity expanded? Has income expanded? Has consumption expanded? Items No Yes (change in use) Pond / Dryer Veg. garden Stock (fish, veg.) Tools Inputs (feed, fertilizer, etc.) Storage (room, sacks, etc.) Notes (probe around sources of TA, consumer tastes, income potential): Anganwadi center, School Note whether the following are present. If yes, ask how use has changed over the course project activities (probe around consumption by beneficiaries and sourcing). Items No Yes (Change) Stock (fish, veg.) Tools (cooking equipment, plates, utensils, etc.) Storage (room, sacks, etc.) Notes (probe around sources of TA, consumer tastes): Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 82 ANNEX 7: KII CONTACT LIST Table 18: Donor and Implementing Partner Staff KII Contact List DATE/A NAME ORGANIZATION TITLE Donor Staff (USAID) 01/07/23 Chandan Samal USAID India/Food Security Office Senior M&E Specialist 01/07/23 Vamsidhar Reddy USAID India/Food Security Office Development Assistance Specialist (former) Implementing Partner Staff 01/18/23 Saraswati Patti Government of Odisha Department of WCD&MS, Integrated Child Development Scheme (ICDS) Anganwadi staff, Kaptipada, Mayurbhanj 01/18/23 Nirupama Naik Government of Odisha Department of WCD&MS, Integrated Child Development Scheme (ICDS) Anganwadi staff, Kaptipada, Mayurbhanj 01/18/23 Rina Naik Government of Odisha Department of WCD&MS, Integrated Child Development Scheme (ICDS) Anganwadi staff, Kaptipada, Mayurbhanj 01/18/23 Rukmini Sahu Government of Odisha Department of WCD&MS, Integrated Child Development Scheme (ICDS) Supervisor, Kaptipada, Mayurbhanj 01/19/23 Praveen Bhanjdeo Government of Mayurbhanj Maharaja (former) 01/20/23 Alok Kumar Nath Government of Odisha Department of F&ARD Additional Fisheries Officer, Baripada, Mayurbhanj 01/20/23 Subrata Kumar Das Government of Odisha Department of F&ARD Assistant Fisheries Officer, Baripada, Mayurbhanj 01/22/23 Kiran Biswal Bishnupriya Hatcheries, Udala Mayurbhanj Owner 01/24/23 Dr. Suseela Mathew ICAR-CIFT, Cochin, Kerala Principal Scientist, Biochemistry & Nutrition Division 01/24/23 Mr. Chaudhury Shripati Mishra Government of Odisha Department of WCD&MS, Directorate Team Leader, State Project Management Unit, Bhubaneswar 01/24/23 Arun Padiyar WorldFish WorldFish Lead, India 01/24/23 Baishnaba Ratha WorldFish Program Manager 01/24/23 Neetha Shenoy WorldFish M&E Officer 01/24/23 Mr. Debananda Bhanja Government of Odisha Department of F&ARD Additional Director of Fisheries (Technical), Cuttack a. Contact data related to project beneficiaries interviewed was redacted form this contact list to protect informant anonymity in line with Panagora Group’s Institutional Review Board (IRB) practices. Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 83 ANNEX 8: FGD SUMMARY Table 19: Focus Group Discussion Summary NO. DATE DISTRICT BLOCK VILLAGE FGD SIZE BENEFICIARY TYPE 1 1/17/23 Mayurbhanj GB Nagar Saikula 08 Community pond 2 1/17/23 Mayurbhanj GB Nagar Bhangachatu 07 Community pond 3 1/18/23 Mayurbhanj Kaptipada Bhalubasa 07 Community pond 4 1/19/23 Mayurbhanj GB Nagar Kantisahi 05 Community pond 5 1/23/23 Puri Astaranga Nuagarh 08 Solar Dryer 6 1/23/23 Puri Astaranga Kaliakana 09 Solar Dryer 7 1/25/23 Jagatsinghpur Sadar Gajarajpur 05 Community pond 8 1/26/23 Jagatsinghpur Sadar Taradapada 06 Community pond 9 1/26/23 Jagatsinghpur Sadar Mallapur 06 Community pond 10 1/26/23 Jagatsinghpur Sadar Ugalapur 05 Community pond 11 1/27/23 Jagatsinghpur Sadar Siddhala 08 Community pond 12 1/27/23 Jagatsinghpur Sadar Nimapada 05 Community pond 13 1/30/23 Bhadrak Basudevpur Chudamani 08 Solar Dryer 14 1/30/23 Bhadrak Basudevpur Chudamani 06 Solar Dryer 15 1/31/23 Balasore Sadar Mirjapur 09 Solar Dryer 16 1/31/23 Balasore Soro Antapur 06 Community pond 17 2/1/23 Balasore Khaira Sundiadihi 05 Community pond 18 2/1/23 Balasore Khaira Krushanpur 05 Community pond Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 84 ANNEX 9: MINI SURVEY DATA SET Table 20: Mini Survey Data Set DATA QUESTION No. District Block Gender Date 0.A 0.B 0.C 1.A 1.B 1.C 2.A 2.B 3.A 3.B 4.A 4.B 1 Mayurbhanj GB Nagar Male 1/17/23 5 1 1 5 5 5 5 5 4 5 5 3 2 Mayurbhanj GB Nagar Male 1/17/23 5 3 1 3 5 5 5 5 5 5 5 3 3 Mayurbhanj GB Nagar Male 1/17/23 5 4 1 5 5 5 5 5 3 5 5 4 4 Mayurbhanj GB Nagar Male 1/17/23 5 1 1 1 3 5 5 5 4 3 4 3 5 Mayurbhanj Kaptipada Male 1/18/23 2 3 5 5 5 5 5 5 5 5 - - 6 Mayurbhanj Kaptipada Male 1/18/23 4 5 4 3 5 5 5 5 5 5 - - 7 Mayurbhanj Kaptipada Male 1/18/23 5 3 1 2 5 5 5 5 4 5 - 5 8 Mayurbhanj Kaptipada Male 1/18/23 5 4 3 3 5 5 5 5 5 5 - 5 9 Mayurbhanj GB Nagar Male 1/19/23 5 1 1 1 5 5 5 5 5 5 5 5 10 Mayurbhanj GB Nagar Female 1/19/23 5 1 1 1 5 5 5 5 1 5 5 4 11 Mayurbhanj GB Nagar Male 1/17/23 5 5 5 5 3 5 5 5 3 5 5 5 12 Mayurbhanj GB Nagar Male 1/17/23 5 5 5 5 5 5 5 5 5 5 5 5 13 Mayurbhanj GB Nagar Male 1/17/23 5 5 5 5 5 5 3 4 3 5 5 14 Mayurbhanj GB Nagar Male 1/19/23 5 4 3 4 5 5 3 4 5 5 5 5 15 Mayurbhanj GB Nagar Female 1/19/23 5 4 3 5 5 5 5 5 5 5 5 5 16 Mayurbhanj GB Nagar Female 1/19/23 5 5 5 4 3 1 3 3 5 5 5 5 17 Mayurbhanj GB Nagar Female 1/19/23 5 5 5 3 5 2 2 2 2 3 4 5 18 Mayurbhanj Khunta Female 1/21/23 5 5 4 3 5 5 5 5 5 3 2 5 19 Mayurbhanj Khunta Male 1/21/23 5 4 4 3 5 5 5 5 5 3 5 3 20 Mayurbhanj Khunta Male 1/21/23 5 5 5 5 5 5 3 4 1 2 2 2 21 Mayurbhanj Khunta Male 1/21/23 5 5 5 3 2 2 5 5 5 5 5 5 22 Mayurbhanj Khunta Male 1/21/23 5 4 2 3 5 5 3 4 5 5 5 5 Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 85 DATA QUESTION No. District Block Gender Date 0.A 0.B 0.C 1.A 1.B 1.C 2.A 2.B 3.A 3.B 4.A 4.B 23 Mayurbhanj GB Nagar Male 1/17/23 5 5 2 5 3 3 5 5 5 5 5 3 24 Mayurbhanj GB Nagar Male 1/17/23 5 5 1 5 5 3 5 5 3 5 5 3 25 Mayurbhanj GB Nagar Male 1/17/23 5 3 1 5 3 1 5 3 5 5 5 3 26 Mayurbhanj Khunta Female 1/19/23 5 5 5 5 5 3 5 3 5 5 5 3 27 Mayurbhanj Khunta Female 1/19/23 5 3 5 5 5 3 5 3 5 5 5 3 28 Mayurbhanj Khunta Male 1/19/23 5 3 1 5 5 5 5 3 5 5 5 5 29 Mayurbhanj Khunta Male 1/19/23 5 5 3 5 5 3 5 3 5 5 3 3 30 Mayurbhanj Khunta Male 1/19/23 5 5 3 5 5 1 5 5 5 5 5 5 31 Jagatsinghpur Naugaon Male 1/25/23 3 4 1 1 3 5 5 5 1 5 1 1 32 Jagatsinghpur Naugaon Male 1/25/23 3 2 1 0 3 5 0 0 0 0 0 0 33 Jagatsinghpur Naugaon Male 1/25/23 4 0 1 1 3 5 3 5 1 0 0 1 34 Jagatsinghpur Naugaon Male 1/25/23 3 0 1 0 5 5 5 5 1 5 5 1 35 Jagatsinghpur Naugaon Male 1/25/23 3 4 1 0 1 5 5 5 0 0 0 0 36 Jagatsinghpur Sadar Male 1/26/23 3 2 1 0 5 1 1 1 0 5 5 3 37 Jagatsinghpur Sadar Male 1/26/23 4 3 1 1 3 5 5 5 5 5 1 1 38 Jagatsinghpur Sadar Male 1/26/23 2 2 1 1 5 5 1 1 2 5 1 1 39 Jagatsinghpur Sadar Male 1/26/23 3 3 1 0 0 0 1 3 0 0 0 0 40 Jagatsinghpur Sadar Male 1/26/23 2 1 3 0 0 0 0 0 0 0 0 0 41 Jagatsinghpur Sadar Male 1/26/23 4 4 1 5 5 5 3 3 5 5 5 3 42 Jagatsinghpur Sadar Male 1/26/23 3 3 4 4 5 1 5 3 5 5 4 3 43 Jagatsinghpur Sadar Male 1/26/23 3 1 1 5 4 5 5 5 5 5 0 5 44 Jagatsinghpur Sadar Male 1/26/23 3 1 3 4 2 5 1 1 5 5 1 3 45 Jagatsinghpur Sadar Male 1/26/23 4 3 1 5 3 5 2 5 5 5 1 3 46 Jagatsinghpur Sadar Male 1/26/23 3 4 1 1 3 5 0 0 0 0 0 0 Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 86 DATA QUESTION No. District Block Gender Date 0.A 0.B 0.C 1.A 1.B 1.C 2.A 2.B 3.A 3.B 4.A 4.B 47 Jagatsinghpur Sadar Male 1/27/23 3 1 1 5 5 5 1 3 5 5 5 3 48 Jagatsinghpur Sadar Male 1/27/23 3 1 1 1 5 5 1 5 3 5 1 1 49 Jagatsinghpur Sadar Male 1/27/23 3 1 1 5 5 3 3 5 3 5 5 3 50 Jagatsinghpur Sadar Male 1/27/23 3 3 1 5 5 5 5 5 5 5 3 3 51 Jagatsinghpur Sadar Female 1/27/23 3 3 1 5 5 5 5 5 5 5 3 3 52 Jagatsinghpur Sadar Female 1/27/23 3 1 1 3 5 5 1 1 1 5 1 1 53 Jagatsinghpur Sadar Male 1/27/23 3 1 1 1 5 5 3 1 5 5 1 3 54 Baleshwar Soro Female 1/31/23 5 5 4 4 3 4 4 5 3 4 5 5 55 Baleshwar Soro Female 1/31/23 3 5 3 3 5 3 5 3 5 5 5 5 56 Baleshwar Soro Female 1/31/23 1 5 3 4 5 3 4 3 5 5 5 5 57 Baleshwar Soro Female 1/31/23 5 5 5 3 5 5 5 5 5 5 5 5 58 Baleshwar Soro Female 1/31/23 5 5 4 5 5 3 4 4 5 5 5 4 59 Baleshwar Soro Female 1/31/23 5 5 4 4 4 5 5 4 5 5 5 3 60 Baleshwar Soro Female 1/31/23 5 5 5 5 5 5 5 5 5 5 5 5 61 Baleshwar Khaira Male 2/1/23 5 5 5 5 4 5 5 3 3 4 5 5 62 Baleshwar Khaira Male 2/1/23 3 4 5 3 5 4 5 5 5 4 5 3 63 Baleshwar Khaira Male 2/1/23 5 4 3 4 3 4 5 3 3 4 4 3 64 Baleshwar Khaira Male 2/1/23 5 5 4 3 4 4 5 5 5 4 4 5 65 Baleshwar Khaira Male 2/1/23 5 3 3 5 5 5 5 5 5 5 5 5 66 Baleshwar Khaira Male 2/1/23 3 3 5 5 4 3 3 3 2 3 5 5 67 Baleshwar Khaira Male 2/1/23 5 4 5 1 5 5 5 3 5 5 5 5 68 Baleshwar Khaira Male 2/1/23 5 5 5 5 5 5 5 5 5 5 4 2 69 Baleshwar Khaira Male 2/1/23 5 5 5 5 5 4 4 4 5 5 4 5 70 Baleshwar Khaira Male 2/1/23 5 5 5 4 5 3 5 4 3 4 4 4 Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 87 DATA QUESTION No. District Block Gender Date 0.A 0.B 0.C 1.A 1.B 1.C 2.A 2.B 3.A 3.B 4.A 4.B 71 Baleshwar Khaira Male 2/1/23 5 4 5 4 5 4 3 3 5 5 5 5 72 Baleshwar Khaira Male 2/1/23 5 4 5 4 5 4 3 3 5 5 5 5 73 Baleshwar Khaira Male 2/1/23 5 4 5 4 5 4 3 3 5 5 5 5 74 Baleshwar Khaira Male 2/1/23 5 4 5 4 5 4 3 3 5 5 5 5 75 Baleshwar Khaira Male 2/1/23 5 4 4 5 3 5 5 4 5 4 5 5 76 Baleshwar Khaira Male 2/1/23 3 5 4 2 5 3 5 5 5 5 5 5 77 Baleshwar Khaira Male 2/1/23 5 5 5 5 5 4 3 3 3 3 5 5 78 Baleshwar Khaira Male 2/1/23 2 4 5 2 3 5 4 4 5 5 5 5 79 Baleshwar Khaira Male 2/1/23 5 5 4 4 5 3 2 3 4 5 4 5 80 Baleshwar Khaira female 1/31/23 5 3 3 1 5 5 5 3 3 5 5 5 81 Baleshwar Khaira female 1/31/23 3 5 3 3 5 3 5 3 3 5 5 3 82 Baleshwar Khaira Female 1/31/23 3 1 3 3 5 3 5 3 3 5 5 5 83 Baleshwar Khaira female 1/31/23 5 5 3 3 5 5 5 5 3 5 5 3 84 Baleshwar Khaira Female 1/31/23 5 3 1 3 5 5 5 5 5 3 5 3 85 Baleshwar Khaira female 2/1/23 5 5 1 5 5 3 5 3 3 5 5 3 86 Baleshwar Khaira Female 2/1/23 5 3 1 1 3 5 5 5 3 5 5 5 87 Baleshwar Khaira Female 2/1/23 5 3 3 3 5 5 5 5 3 5 5 3 88 Baleshwar Khaira Female 2/1/23 5 3 1 5 3 5 5 5 3 5 5 5 89 Baleshwar Khaira Female 2/1/23 5 5 1 3 5 5 5 3 5 5 5 5 90 Baleshwar Khaira Female 2/1/23 5 3 1 3 5 5 5 3 3 5 5 3 91 Baleshwar Khaira Female 2/1/23 3 3 1 3 5 5 5 5 3 5 5 3 92 Baleshwar Khaira Female 2/1/23 5 5 1 3 5 5 5 5 5 5 5 3 93 Baleshwar Khaira Female 2/1/23 5 5 1 3 5 5 5 5 5 5 5 3 94 Baleshwar Khaira Female 2/1/23 5 5 1 5 5 5 5 5 5 5 5 5 Scaling Nutrition-Sensitive Fisheries Technologies – Ex-Post Facto Evaluation USAID.GOV 88 DATA QUESTION No. District Block Gender Date 0.A 0.B 0.C 1.A 1.B 1.C 2.A 2.B 3.A 3.B 4.A 4.B 95 Baleshwar Khaira Female 2/1/23 5 3 1 5 5 5 5 5 5 5 5 5 96 Baleshwar Khaira Female 2/1/23 5 3 3 5 5 5 5 5 5 5 5 3 97 Baleshwar Khaira Female 2/1/23 5 3 1 5 5 5 5 5 3 5 5 3