PERFORMANCE EVALUATION OF USAID/NEPAL’S KNOWLEDGE￾BASED INTEGRATED SUSTAINABLE AGRICULTURE AND NUTRITION (KISAN) PROJECT FINAL REPORT February 2018 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by CAMRIS International. i PERFORMANCE EVALUATION OF USAID/NEPAL’S KNOWLEDGE-BASED INTEGRATED SUSTAINABLE AGRICULTURE AND NUTRITION (KISAN) PROJECT FINAL REPORT February 2018 Evaluation Mechanism Number: AID-367-C-15-00001 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ii TABLE OF CONTENTS Table of Contents.......................................................................................................................................................iii Acronyms and Abbreviations....................................................................................................................................v Evaluation Team Members .......................................................................................................................................vi Executive Summary ...................................................................................................................................................vii Evaluation Purpose................................................................................................................................................vii Project Background ..............................................................................................................................................vii Methodology ..........................................................................................................................................................vii Evaluation Question 1.........................................................................................................................................viii Evaluation Question 2.........................................................................................................................................viii Evaluation Question 3........................................................................................................................................... x Evaluation Question 4.......................................................................................................................................... xi Evaluation Question 5......................................................................................................................................... xii Key Recommendations...................................................................................................................................... xiii 1.0) Evaluation Purpose............................................................................................................................................. 1 2.0) Activity Background........................................................................................................................................... 1 2.1) KISAN Activities ............................................................................................................................................ 1 2.2) KISAN Implementation Strategy................................................................................................................ 2 3.0) Evaluation Questions and Methodology ....................................................................................................... 2 4.0) Findings and Conclusions.................................................................................................................................. 3 4.1) Evaluation Question 1................................................................................................................................... 3 4.2) Evaluation Question 2................................................................................................................................... 9 4.3) Evaluation Question 3................................................................................................................................. 17 4.4) Evaluation Question 4................................................................................................................................. 21 4.5) Evaluation Question 5................................................................................................................................. 25 5.0) Recommendations............................................................................................................................................ 28 5.1) Design and Management ............................................................................................................................ 28 5.2) Poverty, Disadvantaged Groups, and Women ..................................................................................... 28 5.3) Markets, Marketing, and Extension Service........................................................................................... 29 5.5) Capacity Development............................................................................................................................... 30 Appendix A: Implementation Model and Results Framework.......................................................................31 Appendix B: Evaluation Questions ....................................................................................................................... 33 Appendix C: Detailed Methodology..................................................................................................................... 34 iii Appendix C: Assessment of Secondary Quantitative Data Sources, Final..................................................37 Appendix D: Detailed Tables of FGDs and KII Completed by District ......................................................53 Appendix E: Gross Margin Report ....................................................................................................................... 58 Appendix F: Evaluation Instruments..................................................................................................................... 64 Appendix G: Limitations ......................................................................................................................................... 87 Appendix H: Statement of Work ......................................................................................................................... 88 Appendix I: Summary of KISAN Indicators......................................................................................................100 iv USAID/Nepal Monitoring, Evaluation and Learning Project ACRONYMS AND ABBREVIATIONS AICL Agriculture Input Company Limited AMT Agriculture Marketing Technician BLP Business Literacy Program BMI Body Mass Index BOP Business Opportunity Program CSISA Cereal System Initiative for South Asia DADO District Agriculture Development Offices DSA Daily Subsistence Allowance FGD Focus Group Discussion FTF Feed the Future GESI Gender and Social Inclusion Specialist GoN Government of Nepal IPM Integrated Pest Management JTA Junior Technical Assistant KII Key Informant Interview KISAN Knowledge-based Integrated Sustainable Agriculture and Nutrition LSP Local Service Provider MFI Micro Finance Institution MOAD Ministry of Agriculture Development MPC Market Planning Committees NWCSC Nepal Women’s Community Service Center PAHAL Promoting Agriculture, Health and Alternative Livelihoods SACCOs Savings and Credit Cooperatives STCL Salt Trading Corporation USAID United States Agency for International Development VDC Village Development Committee ZOI Zone of Influence Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition v USAID/Nepal Monitoring, Evaluation and Learning Project EVALUATION TEAM MEMBERS Peter Jarvis, Team Leader Aruna Palikhe, National Sector Specialist Rita Khatiwada, Gender Empowerment and Social Inclusion (GESI) Specialist Bikash Acharya, Field Researcher Maya K. C., Field Researcher Thakur Amgai, Translator/Interpreter Peter Appleton, Marc Shapiro, Lauren Else, Editors Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition vi USAID/Nepal Monitoring, Evaluation and Learning Project EXECUTIVE SUMMARY EVALUATION PURPOSE The purpose of this final performance evaluation of the first Knowledge-based Integrated Sustainable Agriculture and Nutrition (KISAN) activity1 is to assess USAID/Nepal’s effort to increase agricultural productivity and private sector investments in agriculture in the Feed the Future (FTF) Zone of Influence (ZOI). Findings from the evaluation will be used by USAID and other partners to determine strategies for implementing follow-on FTF activities and provide evidence of best practices. PROJECT BACKGROUND KISAN assisted over 100,000 subsistence smallholder farmers in 4,300 groups to graduate to more commercial agriculture by improving on-farm production and facilitating market development. Over 70 percent of beneficiaries were women. The project focused on target commodities that are important for food security (rice, maize, and lentils), and high-value (off-season vegetables), which are nutrient-rich. The project focused on building the capacities of private sector and community-based service providers to improve the supply of quality inputs, credit, and other services such as land preparation and equipment rental. In parallel, the project facilitated market linkages to improve farmers’ access to service providers and buyers. In more remote districts, as the private sector is less active, the project delivered services directly to farmers. By delivering services to farmers in partnership with private sector service providers, the project aimed to build sustainability. METHODOLOGY The evaluation team reviewed documents provided by USAID as well as secondary data sources, and a separate review of these secondary sources was submitted 2 to USAID: “Assessment of Secondary Quantitative Data Sources for KISAN Project Performance Comparisons”. The team conducted field work in eight districts as well as in Kathmandu, completing 32 Focus Group Discussions (FGDs) with farmer beneficiaries and over 100 key informant interviews (KIIs) with beneficiary leaders, value chain actors, implementing partners and other stakeholders including partner projects (including some in Kathmandu). Qualitative data collection instruments were developed based on the five evaluation questions provided by USAID. The team also conducted key informant interviews, without pre-structured questions, with partner projects, KISAN staff including the technicians that worked with farmers, and with Government of Nepal (GoN) staff in District Agriculture Development Offices (DADO). 1 This report follows ADS guidelines in referring to KISAN as an activity rather than a project. All references, unless otherwise stated, are to the first KISAN, or KISAN-I, activity and not the subsequent KISAN-II. 2 Submitted July 2017 Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition vii USAID/Nepal Monitoring, Evaluation and Learning Project EVALUATION QUESTION 1 How do the main outcome-level results achieved in Nepal’s FTF ZOI districts compare to existing national statistics on agricultural productivity/yield, sales, consumption of inputs, and market efficiency, among others? FINDINGS BY OUTCOME Agricultural Production/Productivity and Income A comparison of national statistics with KISAN-reported data shows that the yield per hectare for all but one crop is higher for KISAN beneficiaries than the reported district averages, except for lentils, which were slightly lower. Beneficiaries targeted by KISAN reported increases in income. Nutrition KISAN reports that vegetable consumption is higher across all districts and that there is no evidence of malnutrition. National statistics do not map directly to Feed the Future (FTF) Zone of Influence (ZOI) areas and so do not speak directly to KISAN. These national statistics indicate that the percentage of children under five years who are severely malnourished has increased in the Terai and Western regions but decreased in the Hill ecological region and Mid-Western and Far-Western regions. The percentage of women 15-49 years of age with acute under-nutrition (body mass index less than 18.5) has increased in the Terai but decreased in the Hill areas, Western, Mid-Western and Far-Western regions. Agricultural Inputs and Technology While no national statistics were available to speak to agricultural inputs in KISAN areas, agricultural credit did increase in KISAN intervention areas at considerably higher rates than found generally through the limited data available. Market Efficiency While little evidence was found to demonstrate KISAN contribution to market efficiency, almost all FGDs reported that they had been introduced to traders by KISAN, which may help facilitate some market efficiencies. CONCLUSIONS There is evidence that the activity improved household food security, as most beneficiaries reported higher income to buy food, higher consumption of vegetables of a greater variety than previously, and a greater self-sufficiency in rice and maize production. There was no evidence of severe undernutrition or undernutrition reported by beneficiaries. The project consistently met most of its intended annual agriculture yield and sales targets. In addition, yields for KISAN crops are significantly higher than yields from non-KISAN farms. Irrigation technology as well as training in vegetable and cereal production methods increased significantly in KISAN areas helped to increase farm incomes. EVALUATION QUESTION 2 In what way has the interaction between different activities (i.e. strategy to integrate agriculture activities of KISAN with IPM, CSISA and BLP) contributed to achieving or hindering outcomes? FINDINGS, BY OUTCOME KISAN was linked with partnerships with other USAID activities, IDE Nepal’s Integrated Pest Management (IPM), Cereal System Initiative for South Asia (CSISA) and the Business Literacy Program (BLP). Specific benefits to KISAN by partnering with other USAID projects follow. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition viii USAID/Nepal Monitoring, Evaluation and Learning Project Improved Access to and Use of Quality Inputs The CSISA activity helped to support advanced production technologies for cereal cropping and mechanization, but demand did not increase in part due to unaffordability. CSISA distributed new and improved varieties of seeds to lead farmers for demonstration plots, although distribution of seeds as inputs was not a key programmatic area for KISAN. Through association with IPM, KISAN built capacity of agrovets’ in the technical use of farm inputs and agricultural chemicals and IPM technologies. The DADO grant was effective in partnering with KISAN to raise sufficient capital to install a number of irrigation systems, particularly in the Hill areas, and supported activities such as plastic for tunnel houses and infrastructure for Market Collection Centers. Improved Capacity of Agricultural Extension Workers, Service Providers, and Farmers, to Increase Agricultural Productivity or Yield Extension workers increased in number and capacity through the grantees sub-activity because grantees used the grant to hire technicians and local service providers (LSPs), but they could not fulfil demands due to generally low skill levels, particularly in pest management, and the burden of too many groups to service. The sub-activity participants were also better able to participate in KISAN activities and crop marketing activities. IDE Nepal trained 91 KISAN staff in Integrated Pest Management (IPM) practices and technologies, which were widely adopted by farmers. As indicated with respect to inputs, above, although not an explicit target KISAN training, KISAN built slightly the capacity of agrovets. The overall model of support, agrovets as an embedded extension service, did not appear fully sustainable and scalable post-intervention. The evidence for this conclusion is two￾fold: Sixty percent staff recruited by the private sector would be laid off after KISAN ended for lack of financial compensation. In addition, agrovet service did not reach remote communities because of their dependency on sales of products to make field visits financially viable, limiting its continued viability in those areas. Farmer capacity is explained in the next sub-section. Improved Agricultural Production, Post-harvest Technologies, and Practices Adopted at Farm Level KISAN beneficiaries received training in vegetable and cereal production, farm mechanization and farm management from IPM, CSISA and BLP. This resulted in substantially higher yields than those reported by the Ministry of Agriculture as typical district averages. However, some of the integrated efforts did not result in adoption of planned improved practices at the farm level. Few farmers or groups purchased the mechanization demonstrated. According to monitoring data, the partnership with BLP improved the literacy and numeracy skills of 54,000 farmers, including 22,000 KISAN direct beneficiary farmers, increasing their capacity somewhat to learn new techniques though even many of those trained remain functionally illiterate. Improved Market Efficiency KISAN partner activities were not designed to impact market efficiency (IPM, CSISA, BLP), so no additional efficiencies were introduced by the intersection of activities. Improved Capacity of GoN and Local Organizations KISAN planned and coordinated activities with the DADO staff, which were not an explicit focus of KISAN’s capacity-building interventions and who generally are considered still to have out-of-date technical skills. The activity did not provide capacity development for the GoN staff. CONCLUSIONS Cooperation with CSISA, IDE and BLP enabled KISAN to expand its abilities to offer support to beneficiaries as well as expand the area in which they work, although results varied. Overall, the combined assistance from USAID resulted in production increases. The capacity of agrovets greatly increased, allowing them to provide higher-quality extension services to farmers, in less-central areas. The model for continuing their embedded extension services was not sustainable in more￾distant rural areas, though. While the integration with BLP improved literacy and numeracy skills of Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition ix USAID/Nepal Monitoring, Evaluation and Learning Project targeted farmers, many of the production and post-harvest technologies and practices were not implemented at the farm level and the value of the demonstration of mechanization was minimized by low uptake. The KISAN activity and its partners played a minor role in improving market efficiency or expanding post-harvest technologies. The activity and its integrated work did not change the current market channels where farmers close to market prefer to sell individually, and farmers in remote areas with no vehicle access remain both severely disadvantaged and no less connected to markets. The greatest financial benefits from increases in agricultural production were for those farmers who had good market access and sufficient land and resources, including capital and water. KISAN also undertook few post-harvest interventions regarding infrastructure, training or technology applications, also potential routes towards improving efficiency. The grantee sub-activity had a significant effect on bringing new technologies to farmers, DADO staff, and local organizations. However, this intervention is not fully sustainable, because over 60 percent of the staff recruited by the private sector specifically for the KISAN activity will be laid off after KISAN ends. EVALUATION QUESTION 3 How do project beneficiaries perceive impacts of the interventions on household farm income, agricultural productivity/yield, food security and nutrition? What project interventions/strategies do they find as helping or hindering increased agricultural productivity/income and improved nutritional status? FINDINGS BY OUTCOME Perceived impacts on household farm income, agricultural productivity/yield, food security and nutrition Beneficiaries in FGDs reported that they had benefited from working with KISAN with respect to income, food security, and nutrition, mainly from promoting sales of a diverse set of vegetables. Key interventions/strategies identified included improved varieties, systematic planning, irrigation in the Hill areas, and improved nursery management for rice. Although vegetable production was one of the focus commodities of the project, KISAN-supported groups reported greater food security in terms of cereal crops for consumption. Beneficiaries also point to the BLP module on nutrition and health. However, farmers also noted the negative effect discussed further in evaluation question four of the unintended consequence of falling prices and profitability. A factor that farmers noted as hindering income and market functioning is limited or dated information on market prices. There are no mobile apps in Nepali, farmers generally do not have smartphones, and there is no long-term push SMS service providing rate information that could help offset market imbalances. Change in household composition, including migration. Change by individuals to replace migrant work with agricultural work as a result of improved technologies, market access, higher returns, etc. There has been little evidence of impact on migration outside the country to seek work as a result of the KISAN activity. Impact on women’s incomes and livelihoods Seventy percent of project beneficiaries were women farmers, but only 2 women were grantees, and 37 were men. Women reported increases in household incomes due to the KISAN activity and the BLP. KISAN did not focus on the most marginalized women (landless), as it was designed as an agricultural activity. However, some marginalized women participated in the BLP. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition x USAID/Nepal Monitoring, Evaluation and Learning Project Changes in income affecting household gender dynamics Many female beneficiaries reported increased confidence and changes in behavior. For example, women reported that they now make financial decisions jointly with men. Women reported that there are programs against gender-based violence (GBV) and there is more awareness of gender￾based violence being a problem. However, the KISAN activity did not include GBV-related interventions, and in the public forum of FGDs, women were reluctant to speak about gender-based violence issues. CONCLUSIONS The strategy of focus on increasing vegetable production and marketing as a result of KISAN increased on-farm production and household consumption, and nutrition thus has improved. If income from farming remains at levels generally found in the field, farmers indicate that income remains insufficient to discourage migration. Women’s empowerment through KISAN training and the BLP, though, has contributed to the change in household incomes for even the most marginalized families and has allowed women to support their families. KISAN and BLP training enhanced women’s confidence. Men respected women more, as women increased their ability to contribute to family income through KISAN and BLP training. Although participating women are now more confident and respected, most of the high-income earning farmers are still men, who continue to have access to greater resources and are better able to participate in marketing activities. Men continued to be the dominant participants in the KISAN interaction workshops that introduced farmers to other stakeholders such as traders and agrovets. EVALUATION QUESTION 4 In what way have KISAN interventions (including engaging in specific value chains) contributed to changes in the market systems and private sector investment in agriculture? FINDINGS BY OUTCOME Interventions that Changed Market Systems in KISAN Districts KISAN improved farmers’ linkages to traders and expanded farmers’ marketing horizons through introductions to local traders and through providing contacts to traders. Traders participated in “exposure days” where they met farmers in groups. Another positive change emanated from KISAN linking savings groups to higher-level monetary financial institutions via networking activities, especially in urban centers. This expanded credit options for beneficiary farmers, with the potential for both greater investment and savings from additional income. As discussed further under the sub-section on unintended consequences, as KISAN is not the lone activity promoting vegetable production, farmers now complain that the price of vegetables is decreasing and making vegetable production unprofitable. This has changed market dynamics, if not systems. Satisfaction with Changes to Service Provision and Access KISAN beneficiaries did not perceive that DADO staff provided them with training, nor have the DADO staff received capacity building from the KISAN activity. Beneficiaries perceived DADO staff as being under-resourced and understaffed. DADO technical staff are considered to have outdated technical skills, and remote communities rarely see a DADO officer or any agrovet technician because agrovets still must depend on sales of products to make field visits financially viable. Combined contributions by DADO and KISAN in funding irrigation scheme improvements were viewed favorably but were largely limited to existing DADO-registered groups. The level of private sector involvement in providing services to farmers are not expected to continue at the levels found Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition xi USAID/Nepal Monitoring, Evaluation and Learning Project during KISAN, as travel allowances would disappear and grantees were expected to reduce by over 60 percent of the technical staff employed. Unanticipated Negative Impact on Functioning of Market Systems Many projects, including KISAN, promoted vegetable production, and substantially increased the supply of vegetables. This reduced the price of vegetables and, in some cases, made vegetable production unprofitable. Further, the increased area in and geographically dispersion of vegetable production by various actors has facilitated the spread of pest and diseases, including new ones they do not know how to control. Some poor-quality seed provided by agrovets did not germinate and resulted in crop and income loss to farmers. Rapid expansion of vegetable production by KISAN and other projects resulted in reduced prices and has created problems with pests and diseases. Some farmers reported irritated eyes and skin due to use of agricultural chemicals. CONCLUSIONS While KISAN activities did have a small impact on market systems, some of the elements required for an efficient market system were not included in KISAN interventions. KISAN did improve farmers’ linkage and expanded marketing horizons. Vegetable production became less profitable for farmers, perhaps as an unexpected outcome of programming from KISAN and other donor-funded interventions. DADO was not expected by KISAN to fully provide services as a substitute for KISAN by the end of KISAN’s programming, and, indeed, DADO remains perceived under-resourced and understaffed. Most farmer beneficiaries contacted have not noticed an improvement in the services being provided by DADO as a result of the KISAN activity, neither in quality nor quantity of service. The exception was for their co-funding of limited irrigation schemes in areas in which they already operated. The private sector service to farmers is not expected to continue at similar levels barring follow-on programming, as grantees were expected to heavily reduce their staffing. Without a continuation of KISAN’s funding of technicians’ private services, farmers will continue to struggle with issues involving increased problems with pests and diseases and misuse of, or negative reactions to, agricultural chemicals. EVALUATION QUESTION 5 Based on findings from effectiveness questions, what factors, interventions or changes to systems or institutions (public and private) may contribute to sustainability in follow-on activities? FINDINGS AND CONCLUSIONS The evaluation defines the term sustainable in terms of continuity of programming and programmatic effects after the end of the activity in the absence of follow-on programming. In terms of public- and private-sector support, the current institutional organization of DADO and the current technical level of many DADO staff and under-resourcing suggests low levels of sustainability. The stakeholders and agrovets, who are motivated by profit, will be capable of providing quality support services but only to those farmers and organizations who can pay for these services, which favors those closer to urban areas. For greater sustainability, KISAN-II would need to offer more and more-consistent training to both private and sector staff, along with appropriate incentives to ensure that the extension needs of activity beneficiaries are met. The market system for vegetables and cereals in the FTF ZOI is highly inefficient and provides the greatest benefit to farmers close to the market. In remote areas, stronger cooperatives are required Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition xii USAID/Nepal Monitoring, Evaluation and Learning Project to aggregate and encourage traders to visit. There also are few services providing equipment for loan, as too few farmers are capitalized sufficiently to purchase equipment themselves. More careful analysis of market dynamics, unintended consequences in terms of price and pests, and structural constraints is necessary to strengthen commercial viability of value chains. Lack of focus on post-harvest infrastructure, including cold storage, and greater variation in products threatens gains made along vegetable value chains. District markets are poorly resourced and managed. Access to fertilizer is uncertain and slow. Irrigation systems in the hill areas were inadequately funded by activity ($450 per scheme) plus user fees/individual investments, also reducing their sustainability and the sustainability of KISAN outputs. KEY RECOMMENDATIONS DESIGN AND MANAGEMENT 1. To increase the extent to which interventions are relevant and cost-effective for targeted communities, implementers of USAID agricultural activities should undertake strong needs assessments for beneficiary groups and analyze carefully costs and benefits to select investments that offer the best return on capital. 2. Since area farmers appreciated participating in cooperatives and they appear to work well in Nepal, especially in more-rural areas, USAID and its activities should consider supporting agricultural cooperative development and agricultural service developments (market aggregation and input supply) through these cooperatives. 3. Since irrigation is perhaps the most limiting constraint of many farmers’ success, USAID’s implementers should include a focus on irrigation and design irrigation schemes to be adequately funded by activity plus user fees/individual investments, in order to ensure sustainability. 4. USAID should consider funding components of agricultural activities that focus on livelihood diversification to test it as a strategy for reducing out-migration from agricultural areas, increasing resilience, and targeting poverty reduction. POVERTY, DISADVANTAGED GROUPS, AND WOMEN 5. To the extent that the Mission’s goal is prioritizing helping the poorest, most remote, the landless, and under-represented groups, it should encourage KISAN-II to achieve poverty targets. Such programming would consider a range of carefully-targeted options from promoting push strategies to encourage the poorest to participate in markets, building capacity in ancillary activities within the farming value chain, livelihoods diversification, kitchen gardens to improve food security, and water storage where there is no irrigation. 6. Since the BLP was one of the most well-received interventions by women and to the extent that similar outcomes are still a priority for the Mission, KISAN-II should look for linkages with BLP and similar interventions to improve GESI-related outcomes. 7. KISAN-II and other future activities could emphasize women entrepreneurs and focus on creating new opportunities to develop female entrepreneurship. Consideration should be given to establishing a separate fund specifically for women, to help offset the financial market imbalance against women. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition xiii USAID/Nepal Monitoring, Evaluation and Learning Project MARKETS, MARKETING, AND EXTENSION SERVICE 8. Future USAID activities like KISAN should either undertake interventions to make the market system more efficient and provide access to all KISAN beneficiaries or undertake activities that do not rely so strongly on market access and market efficiency. 9. USAID can deepen and broaden the impact of its activities by supporting the GoN to improve market information systems that will allow farmers to access price information along the supply chain from farm to final market. 10. Since the lack of cold storage in district markets limits product shelf-life and impacts prices, USAID, KISAN-II, and future activities should consider including post-harvest value chain interventions including infrastructure, such as storage, grading, and sorting, as well as capacity building for marketing executive committees, as a way to reduce production losses. 11. Given the absence of an efficient and effective DADO extension service, KISAN-II and future USAID activities should consider incorporating an incentive-based agreement with private agrovets and other service providers to ensure that extension services be provided to poorer farmers. CAPACITY DEVELOPMENT 12. USAID’s implementing partners should conduct training needs assessments and training plans for all agrovets when they are recruited as grantees or project partners. 13. To increase sustainability of activity outputs and outcomes, KISAN-II and other USAID￾funded activities should include among its capacity-building plans, interventions that provide specific training programs in support of DADO staff to increase their capacities and ability to provide support over the longer term. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition xiv USAID/Nepal Monitoring, Evaluation and Learning Project 1.0) EVALUATION PURPOSE The purpose of this final performance evaluation of the Knowledge-based Integrated Sustainable Agriculture and Nutrition (KISAN) activity is to assess USAID/Nepal’s effort to increase agricultural productivity and private sector investments in agriculture in the Feed the Future (FTF) Zone of Influence (ZOI). First, the evaluation included a desk review of existing data to analyze differences in agricultural data between the FTF ZOI and the national statistics, as well as within ZOI districts between direct FTF beneficiaries and non-FTF beneficiaries, to better understand the performance data. Second, the evaluation examined the effectiveness of integrating various FTF activities in the ZOI, and what changes were seen in market systems as a result of FTF interventions. Findings from the evaluation will be used by USAID, KISAN and other implementing partners to determine strategies for implementing follow-on FTF activities, and provide evidence of best practices3. 2.0) ACTIVITY BACKGROUND 2.1) KISAN ACTIVITIES KISAN assisted subsistence smallholder farmers to graduate to commercial agriculture by improving on-farm production and facilitating market development. The activity focused on target commodities that are important for food security (rice, maize, and lentils), and high-value (off-season vegetables), which are nutrient-rich. Market opportunities vary across KISAN’s target area. In areas with access to markets, such as the Terai and low-lying Hill areas, KISAN focused on building the capacities of private sector and community-based service providers to improve the supply of quality inputs, credit, and other services such as land preparation and equipment rental. In parallel, the project facilitated market linkages to improve farmers’ access to service providers and buyers. Opportunities to attract buyers and to engage private sector service providers were fewer in more remote regions. Here, KISAN activity staff worked directly with farmers to achieve higher yields and increase household consumption of vegetables. Coordination and collaboration with the Government of Nepal (GoN) was intended to ensure unified messages on recommended agricultural inputs and techniques, and assist in coordinating investments in irrigation and collection center infrastructure. The first two and a half years of the activity focused on forming over 4,000 farmer groups with over 80,000 beneficiary farmers4 across the 20 districts. Village Development Committees (VDCs) were selected in consultation with the GoN. Farmers received training in good agricultural practices and market information, and the project promoted farmer-to-farmer training through lead farmers who established demonstration plots. Farmers were linked to buyers and input suppliers attracted by greater volumes of production to be traded. By delivering services to farmers in partnership with private sector service providers, the project aimed to build sustainability. By working with women (70 percent of beneficiary farmers were women), the activity addressed gender inclusion and sought to economically empower women. During the last two years, the activity focused on deepening the capacity of farmers and building sustainability. This included working with the Business Literacy Project (USAID) to build the literacy and business skills of 54,000 vulnerable farmers, including 22,000 KISAN direct beneficiaries.5 3 Purpose taken from Statement of Work. 4 Actual numbers are reported by KISAN to be over 100,000 beneficiaries. 5 DEPROSC reported 54,000 beneficiaries in the project areas of which 22,000 were direct KISAN beneficiaries. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 1 USAID/Nepal Monitoring, Evaluation and Learning Project More respected and successful farmers graduated to become lead farmers (for farmer-to-farmer learning) and local service providers or agrovets (input suppliers). The private sector partners strengthened by KISAN included collection centers and Market Planning Committees (MPCs), buyers, input suppliers, and microfinance institutions. Some of these private sector beneficiaries were also recipients of KISAN Business Opportunity Grants which help leverage private sector efforts and linkages to farmers to support value chain development. Food security was supported by improving agricultural productivity and incomes, and through skills development. The activity’s implementation model and results framework are shown in Annex A. 2.2) KISAN IMPLEMENTATION STRATEGY The initial implementation strategy as specified in the original contract (February 2013) was to work with 160,000 beneficiary households (equivalent to one million people) over five years. The original plan was to work with selected households for just one year and then move on to new households. However, USAID agreed that working with households for just one year was not sufficient to ensure beneficiary integration in the market system. Because KISAN was required to maintain a presence in all 20 districts, the budget and targets were adjusted. This reduced the targeted number of households to 100,000 and of these, 85,000 would be engaged directly by the KISAN activity. The initial contract included the nutrition component for 160,000 farm households -equivalent to 1 million people, but this component was transferred to the Suaahara project. 3.0) EVALUATION QUESTIONS AND METHODOLOGY USAID posed five evaluation questions by DEC categories, with 14 sub-questions. These are presented below and grouped by category of question type. A “Getting to answers matrix” was completed for each evaluation question as a first step in assessing measures to be used, data sources, and selection criteria. Table 1: The Five Evaluation Questions Posed by USAID Relevance 1. How do the main outcome level results achieved in Nepal’s FTF ZOI Districts compare to existing national statistics on agricultural productivity/yield, sales, consumption of inputs, and market efficiency, among others? Within the ZOI, how does data compare between FTF direct beneficiaries and non-FTF beneficiaries? Effectiveness 2. In what way has the interaction between different activities (i.e., strategy to integrate agriculture activities KISAN with IPM, CSISA and BLP) contributed to achieving outcomes in: (i) Improved access to increased quality inputs for farmers? (ii) Improved capacity of agricultural extension workers, services providers and farmers to increase agricultural productivity/yield? (iii) Improved agricultural production and post-harvest technologies and practices adopted at farm level? (iv) Improved market efficiency? (v) Increased capacity of GoN and local organizations? Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 2 USAID/Nepal Monitoring, Evaluation and Learning Project a. What are the particular features of the integrated USAID strategy that have made a difference in outcomes? 3. How do project beneficiaries (disaggregated by sex and marginalization) perceive impacts of the interventions on household farm income, agricultural productivity/yield, food security and nutrition? What project interventions/strategies do they find as helping or hindering increased agricultural productivity/income and improved nutritional status? a. What changes have taken place in the household income, food security, and nutrition of target populations as a result of the interventions? b. What explains such changes, particularly with respect to the most marginalized households? c. To what extent have the interventions had an impact on women’s incomes and livelihoods, particularly among the most marginalized? d. To what extent have the interventions had an impact on the nutritional status of women and children, particularly those under age 2? 4. In what way have KISAN interventions (including engaging in specific value chains) contributed to changes in the market systems and private sector investment in agriculture? a. What interventions changed the functioning of market systems in KISAN districts (i.e., expanded and inclusive markets – small-holder farmers, small enterprises; collection centers/aggregation of products; improved sales/prices; access to markets, inputs/services, financial services; and, investment)? b. What interventions have had unanticipated negative impact on functioning of market systems in KISAN districts? Sustainability 5. Based on findings from effectiveness questions, what factors, interventions or changes to systems or institutions (public and private) may contribute to sustainability in follow-on activities? The evaluation team reviewed documents provided by USAID, undertook 105 KIIs and 32 FDGs. Details on methodology are found in Annex C. 4.0) FINDINGS AND CONCLUSIONS Section 4.0 presents the principal finding of this evaluation and the resulting conclusions from these findings. While an effort has been made to link conclusions directly to findings, not all findings map to conclusions. Additionally, some conclusions are crosscutting and do not map directly to a specific finding, but are based on several findings. 4.1) EVALUATION QUESTION 1 How do the main outcome level results achieved in Nepal’s FTF ZOI districts compare to existing national statistics on agricultural productivity/yield, sales, consumption of inputs, and market efficiency, among others? Within the ZOI, how does data compare between FTF direct beneficiaries and non-FTF beneficiaries? 4.1.1) AGRICULTURAL PRODUCTION/PRODUCTIVITY AND INCOME FINDINGS Figure 1 shows the comparison of the average yield per hectare by crop type for KISAN beneficiaries compared to the Ministry of Agriculture Development (MOAD) data for 2015 (consistent years for both the KISAN data and the MOAD data). The yield per hectare is higher for Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 3 USAID/Nepal Monitoring, Evaluation and Learning Project KISAN beneficiaries than the reported district averages6, by between 30 and 60 percent. The exception is lentils, for which KISAN areas are slightly lower. Figure 1 – Average Yields by Crop Type Rice Maize Lentil (Pulses) Bitter Gourd Cabbage Cauliflowe r Cucumber Tomatoes Yield 2015 MOAD 3.13 2.43 0.96 12.3 14.3 12.5 14.7 12.8 KISAN 2015 4.51 3.24 0.91 18.68 26.44 18.49 23.06 30.22 0.00 5.00 10.00 15.00 20.00 25.00 30.00 35.00 MT/Ha Avg. Yield (MT/Ha) by Crop Type: MOAD Data Compared with KISAN Benficiaries across all 20 Project Districts (2015) Yield 2015 MOAD KISAN 2015 The activity consistently met most of its intended annual agriculture and sales targets. Reported average yields for KISAN crops are substantially higher than yields from non-KISAN farms. KISAN also reported an increase in income of $1.87 million by 2016 for the project beneficiaries; Forty-one percent of this amount was investment by beneficiaries who were also grantees under the business opportunities sub-activity (the grantee share). Overall agricultural production potential was negatively impacted due to some farmers dropping out of groups. Reasons provided during FGDs were that they were too busy to attend training or because they did not feel that the activity interventions were to be relevant to them. The extent of dropouts and effect on productivity is unclear, although many households rely on remittances rather than solely on agriculture7. Conclusion The KISAN interventions met their targets and appear to have increased crop yields, evident especially when compared to broader Ministry of Agriculture data for the same districts. 4.1.2) NUTRITION For nutrition, this evaluation report compares data from the Nepal Demographic Health Survey 2011 and 2016. There are two relevant nutrition indicators compared: the first is the percentage of “children less than five years, with weight for age below two SD of Median”, or who are malnourished8, and the second, for women, is body mass index9. The percentage of children under 6 Please refer to the full assessment report for a more complete representation and discussion of the data. 7 Lali Guras Women Farmers Group, Salyan 8 Weight for age is a composite index of both the weight for height and height for age indicators. If below - 2 SD from the median, children are considered malnourished. 9 For women, the BMI seems to be the most appropriate measurement for comparison; if less than 18.5 then women are considered acutely undernourished. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 4 USAID/Nepal Monitoring, Evaluation and Learning Project five years who are severely malnourished has increased in the Terai and Western regions, but decreased in the Hill ecological region and Mid-Western and Far-Western regions. For body mass index less than 18.5, the percentage of women 15-49 years of age with acute undernutrition has increased in the Terai, but decreased in the Hill areas, Western, Mid-Western and Far-Western regions. This secondary data cannot confirm that KISAN impacted change, as the FTF ZOI does not intersect with either the ecological zones (Terai and Hill) or the regions for which the data is available. However, these are relevant indicators for nutrition in areas where the FTF zone of influence is situated and indicates the possibility of some impact by KISAN, suggesting the need for further investigation. Table 4 A: Percentage Children under five years malnourished (by weight) by region and year Ecological Zone 2011 2016 Mountain 35.9 28.5 Hill 26.6 18.1 Terai 29.5 32.5 Development Region Eastern 25.4 26.5 Central 29.5 28.0 Western 23.2 23.7 Mid-Western 36.9 29.1 Far-Western 32.6 28.1 Table 4 B Percentage Women (15-49 years) acute under nutrition (by weight) by region and year Ecological Zone 2011 2016 Mountain 16.5 12.1 Hill 12.4 11.5 Terai 22.7 22.9 Development Region Eastern 16.2 16.6 Central 20.2 18.4 Western 14.0 13.8 Mid-Western 19.3 16.7 Far-Western 23.9 21.7 According to activity monitoring data, all farmers reported improvement related to food security from growing vegetables for consumption; better quality and more diversity; gaining income from selling vegetables; and, where cereal crops were supported, improved production of rice and maize. Conclusion There is evidence that the KISAN activity improved household food security at some level, because most beneficiaries reported having a higher income to buy food, higher consumption of vegetables of a greater variety than previously, and a greater self-sufficiency in rice and maize production. Supporting this conclusion, there was no evidence of undernutrition reported by KISAN beneficiaries, in line with FTF goals. National statistics were not available in a geographical mapping that provide additional clarity with respect to nutrition and food security. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 5 USAID/Nepal Monitoring, Evaluation and Learning Project 4.1.3) AGRICULTURAL INPUTS AND CREDIT The evaluation team examined relevant secondary data to triangulate, although the ability to do so is limited. The Ministry of Agriculture Statistical Year Book does not provide data on sales of crops by farmers. The limited data on agricultural inputs, which is reported for fertilizers and improved seeds, is only for state-owned companies (AICL and STCL) and does not include data for the private sector. The data on improved seeds is only for the National Seed Company Ltd and is not reported by district. There is also a lack of data for new capital investment in agriculture. The Ministry of Finance reports the status of credit flows of banks and financial institutions in the agricultural sector but does not report capital investment levels for the sector10. The Nepal Rastra Bank (Central Bank) reports gross fixed capital formation, but this does not appear to be broken down for agricultural capital investments11. The Nepal Rastra Bank reports that agricultural credit increased from 50.9 billion rupees in 2013/14 to 71.0 billion in 2015/16, an increase of 40 percent. KISAN reports that the value of rural loans for activity beneficiaries increased from a baseline (2014) of $667,615 to $7.448 million (2016 actual), an eleven-fold increase over the baseline value. Conclusion Agricultural credit, frequently required to purchase inputs and technology, increased in KISAN intervention areas, and the increase in agricultural credit was far higher in KISAN than found through the limited national-scope data available, suggesting sizable effects of KISAN where it operated. 4.1.4) AGRICULTURAL TECHNOLOGY – CROPS AND CAPITAL The activity was relevant to beneficiaries in increasing agricultural productivity, as it improved their agricultural skills as well as technologies for growing cereal crops. KISAN did not introduce new crops but introduced crops familiar to farmers but not grown before. The beneficiaries reported that these interventions were highly valuable, as the additional production of cereals improved food security, as did the greater amount and variety of vegetables for home consumption. Out of 32 focus groups, five groups (unprompted) specifically mentioned that the interventions for cereal crops were valuable, but that they also needed more support on cereal production, especially rice.12 The main interventions mentioned by beneficiaries in all FGDs was for vegetable production. When questioned further, most communities indicated that they benefited from interventions to help with maize production and rice production (where grown). In just over half of women-only FGDs, women mentioned that livelihoods training, such as tailoring, would have been more relevant and that they would have preferred training for livestock enterprises. The initial results of vegetable production under irrigation confirmed beneficiaries’ comments in FGDs, who consistently reported that irrigation is their most important need and is crucial for success in vegetable farming. The KISAN capital budget for irrigation was not large enough for many schemes, especially in Hill districts, to get past the proposal stage. Many farmers indicated that they are reluctant to join with others to borrow money for joint schemes. Where KISAN had been able 10 Economic Survey Fiscal Year 2015/16, Government of Nepal, Ministry of Finance 2016. 11 Monetary Policy for 2016/17, Nepal Rastra Bank, July 2016. 12 For example, Ram Nagar Krishak Samuha, in Kapilvastu district. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 6 USAID/Nepal Monitoring, Evaluation and Learning Project to support irrigation independently or with the support of the DADO, farmers have done well from vegetable farming.13 Farmers in the Hill areas and in the Terai grow cereals -mainly rice, wheat and maize. Rice generally requires irrigation, although there are rainfed rice systems/varieties. In FGDs, farmers indicated that cereal production was mainly for home consumption, although those with a larger land area may have surplus for sale, and local millers do buy rice and maize from some of the KISAN groups. DADO staff emphasized the importance of mechanization for cultivation, but also for planting, weeding, harvesting and cereal processing (KISAN introduced maize shellers). KISAN demonstrated mechanization options (tillers, seeders) to farmers with the support of CSISA. DADO staff wanted more emphasis on mechanization14,15 and indicated that this will be one focus of the Prime Minister’s Agricultural Modernization Project. Nevertheless, most beneficiaries reported that buying equipment was too expensive, with few exceptions. Conclusions There was insufficient needs assessments (i.e. of land ownership, irrigation, landless individuals, potential for cropping, etc.) during the design process for individual communities. For example, for farmers in remote areas without road access (most particularly in Hill districts where bicycle transport is not possible), growing vegetables, which perish quickly, was less relevant due to market access problems. Livelihoods training would have been more appropriate than vegetable production in these remote locations. Additionally, irrigation technology and efforts was often reported as important for increasing farm incomes under KISAN. 4.1.5) MARKET EFFICIENCY Almost all FGDs reported that they had been introduced to traders by KISAN and that they had been given the small book of trader contact information provided by KISAN, although some individuals reported that did not receive it. In Surkhet, a grantee trader distributed 20 bicycles to small traders so that they can collect vegetables from farmers.16 In some locations, KISAN17 also supported the formation of collection centers through Market Planning Committees (MPCs)18, and provided MPCs with some training on marketing, bookkeeping, and how to work in a group, as well as with a limited supply of crates for transporting vegetables to market. Not all of the collection centers are operational19 and some do not have any infrastructure, even though DADO (and other projects) had provided some grant support for physical infrastructure. Farmers cooperate best when they have road access and must aggregate vegetables for sale. Traders are not interested in making collections if the volume is less than 500 kilograms. Road access was challenging, and individual farmers were not able to easily access the market. Poor road conditions in 13 Kopila Mahila Krishak Samuha, Kailali Income Rs 10000-15000/ Kattha. Babai farmers group Dang. Most of the farmers make over Rs 100,000/year 14 Five out of Seven DADO informants mentioned this. 15 Mainly for cultivation, but also for planting, weeding, harvesting and cereal processing (KISAN introduced maize shellers). 16 Babu and Shahi trader in Surkhet 17 However, some collection centers are initiated through KISAN’s support e.g. in Dang and Salyan. KISAN helped to link to DADO who supported in construction of collection center and KISAN provided logistic support worth to Rs. 60000. In Dang the group members were encouraged to initiate collection center. 18 Tunibot Agricultural Produce Collection Centre, Salyan; Satbariya Agricultural Produce Collection Centre, Dang. 19 Suthabari Biu Tatha Taja Tarkari Samuha, Surkhet ( No collection center); Upakar Farmers Group, Palpa (No collection center); Yakta Farmer Group (Collection Center is far away). Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 7 USAID/Nepal Monitoring, Evaluation and Learning Project more remote locations are a severe constraint and limit the incentive for traders, as well as commercial agribusinesses such as agrovets, to visit villages.20 Kisan Shara farmers groups (Palungmainac VDC, Palpa) were successful, despite being seven kilometers from the main road with challenging dirt road access that is prone to landslides. The group is classed by KISAN as a competitive farmers group (“A” classification), with an active cooperative where farmers collaborate sales. Farmers reported an income of $8,000, mainly from the sale of tomatoes. The cooperative in this community was also a recipient of the BOP grant. In the main districts’ markets, there was no capacity-building for the executive committee provided by KISAN, and no evidence of any changes to practices as a result of KISAN.21 There are more diverse and greater volumes of produce on the market, especially vegetables. However, the greater volume has reduced market prices especially in peak-season, which is proving to be a disincentive for farmers. Break-even prices can be above the market prices quoted by farmers, as costs do not include the value of the farmers’ labor for production and marketing. KISAN data collection was not designed to capture this level of detail. Conclusions Farmers close to market prefer to sell individually. Farmers in remote areas with no vehicle access are severely disadvantaged. Farmers with vehicle access but with difficult road conditions tend to cooperate out of necessity to aggregate crops for sale. The greatest financial benefits were for those farmers who had good market access and sufficient land and resources (including capital and water). There were few post-harvest interventions undertaken, such as infrastructure, training, and technology applications. 4.1.6) BUSINESS LITERACY PROGRAM Findings Business Literacy Program (BLP) interventions benefited 54,000 farmers, including 22,000 KISAN direct beneficiaries, of which 91 percent were women. Women indicated they benefited by gaining more respect in households due to their ability to contribute to household income and affairs, more self-confidence, and from earning their own incomes from selling vegetables.22 BLP consisted of 40 lessons that could be customized for different enterprises -- e.g. basket-making, trading, baking, etc. Per DEPROSC, beneficiaries and beneficiary farmers would have preferred additional sessions after the last (fifth) module in order to practice skills, consolidate knowledge, and develop opportunities. Conclusion The BLP, with its customizable modules, was one of the most appreciated interventions, particularly for women. 20 Dadeldhura Kailpal Taja Tarkari Samuha, Saraswati Tarakari Utapadan Samuha, Palpa. 21 Ranitalau Market, Banke; Khajura Market, Kohalpur, Banke. 22 Out of total of 32 surveyed FGD 19 groups reported they received BLP training. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 8 USAID/Nepal Monitoring, Evaluation and Learning Project 4.2) EVALUATION QUESTION 2 In what way has the interaction between different activities (i.e. strategy to integrate agriculture activities KISAN with IPM, CSISA and BLP) contributed to achieving or hindering outcomes? i. Improved access to increased quality inputs for farmers? ii. Improved capacity of agricultural extension workers, services providers and farmers to increase agricultural productivity/yield? iii. Improved agricultural production and post-harvest technologies and practices adopted at farm level? iv. Improved market efficiency? v. Increased capacity of GoN and local organizations 4.2.1) INTEGRATION OF AGRICULTURE ACTIVITIES (KISAN WITH IPM, CSISA AND BLP) OVERALL CSISA is a technology generator for cereals and legumes, namely rice, maize and lentils, but not vegetables. It worked with KISAN for the entire period of the activity, starting in 2013. Working with KISAN, CSISA was able to expand its area of work from six to 20 districts. CSISA provided training to KISAN technicians and provided technical backstopping for seed companies working with KISAN.23 Technologies supported by CSISA were seed varieties tested and evaluated, seeding methods, fertilizer and weed management, harvesting, mechanization and post-harvest management. According to CSISA information, KISAN needed more technical skills -- e.g. agronomists and mechanization experts, to support the less technically specialized KISAN Agriculture Marketing Technicians (AMTs). CSISA staff in Dang suggested that coordination between CSISA and KISAN needed improvement, and that the local relationship between the staff among the collaborating partners also affected the on-ground activity performance. Mechanization was demonstrated by KISAN staff, in association with CSISA, but success was limited because most farmers could not afford the capital investment, maintenance and operation costs. KISAN was not able to provide grants24. IDE Nepal trained 91 KISAN staff in Integrated Pest Management (IPM) practices and technologies, which were widely adopted by farmers. However, farmers are still challenged by pests and disease issues and report this as their most challenging technical problem. Technologies promoted by the IPM project included bio-agents for pest and disease control, pesticides, NEEM (Azadirachta indica), pheromone traps, netting in nurseries, peat as a medium for seedlings, and plastic trays for growing out seedlings. The pheromone traps were observed in most locations visited by the evaluation team, as were bio-pesticides, and most of the other technologies associated with IPM were mentioned by beneficiaries in FGDs. The Business Literacy Program provided training to 54,000 people in KISAN beneficiary households of which 22,000 were direct KISAN beneficiaries. Over 90 percent of the BLP beneficiaries were women. The BLP included five modules: i. Literacy & numeracy (basic) – four months. ii. Nutrition & health – links to the USAID Suaahara project. iii. Life skills including Disaster Risk Reduction (DRR) (decision-making). iv. Entrepreneurship includes KISAN activities and non-farm activities. 23 Unique Seed Company buy breeder seed from CSISA (CYIMMT). 24 Ghansingh Dada Farmers group, Palpa; in fact this was true for most of the groups surveyed e.g.: Laliguran Krishak Samuha, Dang. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 9 USAID/Nepal Monitoring, Evaluation and Learning Project v. Access to finance. The retention rate in the BLP was very high (over 80 percent). The sub-activity participants reported greatly increased confidence and, along with increased literacy and numeracy skills, were better able to participate in group meetings and speak up for themselves, and consequently were more able to participate in KISAN activities and crop marketing activities. Conclusions The interaction between different activities increased access to improved cereal and legume seeds and crop management technologies in extended geographical areas. Farmers knew and practiced an integrated approach to pest management, although they still contest with pests and diseases. Farmers involved with BLP increased their knowledge, self-confidence, and crop marketing activities. 4.2.2) ACCESS TO INCREASED QUALITY INPUTS FOR FARMERS Training provided to local service providers (LSPs) in technical topics was insufficient25, due to lack of capacity of trainers. In response, the activity supported outreach by agrovets, introducing them to KISAN farmers groups, providing them with technical training, and covering their costs for some extension activities with farmers. KISAN provided grantees funds to cover part of the costs of recruiting technical staff. As an example, Shree Swargadwari Agri Nepal, Dang, provided extension services to farmers in Dang, Salyan and Rukum, supporting 888 KISAN farmers and 1,700 other farmers. In Salyan, the Unnat Krishi Kendra Agrovet Shop is working with 28 farmers groups, and in Kailali, the Niglasaini Agrio￾Centre and Supply is working with 2,000 farmers. In remote villages, far from the main road, beneficiaries reported that they are rarely visited by private sector participants (agrovets) or by the DADO staff. KISAN technicians have been the primary extension agents in these remote areas.26 IDE Nepal confirmed that the lack of agrovets and other private sector participants in the rural areas severely limits business development in communities. Farmers must normally visit the urban area to seek technical advice from the agrovet shops situated in town. IPM training sites are in only two districts (Banke and Surkhet); however, on six sites (including one established by the DADO), capacity-building was still provided to the KISAN technicians from all districts. For most agrovets, the pheromone traps are a new innovation and they are now selling them. Overall, 49,000 KISAN beneficiary farmers were trained in IPM technologies27 and of these, 21,000 farmers (42 percent) adopted IPM technologies. The IPM training was also provided to four DADO plant protection officers from the Mid-Western region, and two of the DADO plant protection officers participated in a one-week exposure visit to South India with the KISAN staff (two staff members). KISAN notes that its programming was not focused on delivering seed aside from demonstration purposes. CSISA did provide free seeds28 to KISAN lead farmers for the demonstration plots established to conduct the trainings. Providing free seeds did not create ownership by farmers, and 25 They had a five-day training initially, and then a three-day follow up training after a year or so. 26 Specifically, Dadeldhura Kailpal Taja Tarkari Samuha. 27 KISAN data quoted by IDE Nepal. 28 Cereal sees were produced by CSISA. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 10 USAID/Nepal Monitoring, Evaluation and Learning Project some farmers resented not receiving them29 even though providing seeds was not planned as part of KISAN programming. Participating farmers and agrovets both reported problems with the germination rates for seeds30 suggesting that that seed germination rates should be tested prior to any free distribution. For future programming considerations, KISAN should note that some farmers complained about the steep hike of prices in seeds by the agrovets during the activity cycle.31 Conclusions KISAN training improved agrovets’ capacity in the technical use of farm inputs and agricultural chemicals, IPM technologies, and improved their ability to relate to farmers. Some KISAN beneficiary farmers have graduated to become agrovets32 who supply inputs to farmers.33 The CSISA activity helped to support advanced production technologies for cereal cropping and mechanization, while the IPM intervention built the capacity of KISAN technical staff. The production technology was exposed to the farmers, but demand, supply and farmers’ affordability were not well taken. The DADO grant was effective in partnering with KISAN to raise sufficient capital to install a number of irrigation systems, particularly in the Hill areas where the KISAN amount (45,000 NPR) was generally inadequate. The DADO grant also supported other activities, such as plastic for tunnel houses and infrastructure for Market Collection Centers. 4.2.3) CAPACITY OF AGRICULTURAL EXTENSION WORKERS, SERVICE PROVIDERS AND FARMERS TO INCREASE AGRICULTURAL PRODUCTIVITY/YIELD The number of extension workers increased through the grantees program because grantees used the grant to hire technicians and LSPs. Sustainability of this increased service provision is in question because many grantees will dismiss some of the technicians and LSPs recruited for KISAN at the end of the activity.34 Out of 17 grantees interviewed, seven explicitly mentioned that they would lay off the staff hired with the support of KISAN. However, Dinesh Shiva Enterprises in Kapilvastu said it would retain the AMT supported by KISAN, and Rapti Feed in Dang would retain eight of ten staff.35 KISAN extension workers or field technicians often could not fulfil farmers’ demands for quality service because of a lack of skills36, particularly in pest management, and due to the burden of having to service too many groups. Some technicians reported that they had to work with more than 60 groups, while the KISAN-prescribed standard number of groups was 45, with 15 groups per VDC and three VDCs per technician (15 x 3 = 45). KISAN technicians in five districts37 reported that they did not receive adequate training, particularly in pest management. Grantee agrovets reported receiving some technical training from KISAN, but this was not substantive and was three to four days, including orientation, plus a one-day private sector business meeting. Agrovets did report that they received technical training on IPM practices including the use 29 Pragatisil Bahuudesiya Tatha Bakhra Palan Samuha, Salyan 30 Pragatisil Bahuudesiya Tatha Bakhra Palan Samuha, Salyan (sometimes seed is provided after planting season which cannot be used on time). Kisan Shahara Group, Palpa. 31 Sirjansheel Farmers Group, Triveni, Salyan 32 Deu Mani Chaudhari, Dang, Chairman of collection center. 33 Arjun PD Joshi said “I worked as an LSP in KISAN for 2 years and supported group formation, nursery management, demonstration of different crops and vegetables, and making liquid manure; later I opened my own agrovet business”. 34 GATE (Banke); UNIQUE Seed Co. (Kailali). Dev Bhar Rice Mill (Banke). 35 The Cooperative grantee in Surkhet, will retain one staff even after KISAN ends (Shree Pabitra Janakalyan Krishak Sahakari Sasth). 36 Lead farmer, Kapilbastu. 37 Saraswati KC-AMT, Dang; Deva Kunwar-AMT, Kapilbastu; Lumkala Ghimire- AMT-Palpa; AMT Nepalgunj, Banke; and District Coordinator, Surkhet Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 11 USAID/Nepal Monitoring, Evaluation and Learning Project of bio-pesticides and pheromone traps, which was the direct result of KISAN’s cooperation with the IPM intervention. There is evidence that agrovets tended to provide higher quality services to farmers who purchased inputs from them. The most effective support provided to grantees (agrovets, millers, seed companies) was the introduction of grantees to the KISAN farmer groups, which significantly expanded the business network for the agribusinesses and improved participation and a sense of “community38”. Typical increases in turnover were reported to be from 27 percent to 50 percent, but one seed company reported an increase in profits of 300 percent after the company was connected to another 500 farmers through the activity. Grantees noted that before KISAN they barely had any interaction with farmers and did not go to farmers’ homes. Working with KISAN, they reported, provided new insights into the lives of farmers and a greater sense of social responsibility. For grantees, the activity grant could be as much as 80 percent of the total grantee activity costs. While this seems like a high proportion of the grant, the grantees were expected to provide significant services to farmers as part of the grant in line with developing best practices. KISAN reported that the grantees invested $675,000, equal to more than half of the $1.2 million grant provided by USAID funds. In some cases, grantees discontinued work with KISAN because the demands for activity support were too onerous.39 The KISAN grant intervention was implemented late in the activity period. Of the 39 beneficiaries, only four grantees were approved by 2015 and 35 (90 percent) were approved in the year ending September 2016. Of the total $1.2 million grant, records available to the evaluation team indicate that only $14,000 was spent in 2015, and the balance of the funds was spent in 2016. There is still a lack of agribusiness activity in more remote rural communities, which severely limits business development. One agrovet40 has produced a low-cost video to demonstrate technologies to farmers, and reportedly established a plant clinic to give farmers advice on pest and disease identification. Given the relatively low cost now of producing audio-visual materials for demonstrating technological innovation, there appears to be more opportunity to make use of such tools to promote new technologies to farmers. KISAN activity documents indicated that by the end of the activity, 75 percent of all technical support was provided by the private sector. Every KISAN field technician/extension worker was supported by three LSPs (75 percent of a total of four staff). When the activity concludes, much of the technical support from the LSPs will cease with the end of grant support. Farmers who went on exposure visits did not sufficiently share their learning when returning to their groups, and they had little incentive or training to do this. Women were frequently unable to attend cross-visits or study tours due to cultural mores and family commitments. Instead, male members attended, thereby undermining the capacity-building of female members.41 The BLP intervention improved the literacy and numeracy skills of 54,000 farmers, including 22,000 KISAN direct beneficiary farmers. For most beneficiaries of the program, this was not sufficient for them to be able to complete the farmer logbook on their own, and many had to be assisted by family members (including children). Most farmers who attempted to keep the logbook did so haphazardly, with little idea as to the purpose of calculating profit per enterprise or per unit area. 38 For example, the grantee Shree Pabitra Janakalyan Krishak Sahakari Sastha Mehelkuna, Surkhet is working with 40 farmers groups. Similarly, Rapti Feed is working with 4000 farmers. 39 The team interviewed one inactive miller in the Dang district who did not wish to commit to the BOP. 40 Swargadwari Agrovet, Dang. 41 Palpa, Gansinghdada Mixed group. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 12 USAID/Nepal Monitoring, Evaluation and Learning Project The logbooks frequently had income and expenditure for several crops confused on the same page or section of the book (which means that it would be inconsistent with keeping records by crop type, or enterprise). Almost no farmers who did keep the logbook could report profit or gross margin per Kattha or Ropani (1 kattha = 339 square meters. 1 Ropani i= 509 square meters.)42. Further, there was no printed logbook which might have assisted farmers to better understand its purpose and to maintain it correctly. Many beneficiaries did benefit from the BLP and increased their capacity to learn new techniques, but many illiterate farmers still struggle. The BLP was well received by beneficiaries and most want follow-up activities. After the activity, public free extension will revert largely to the DADO staff, who are not capacitated to support KISAN activities (lack of resources, technical skills, transport and lack of motivation and management). The DADO staff are generally considered to have out-of￾date technical skills, even by their senior managers.43 Conclusions Extension workers increased in number and capacity through the grantees sub-activity. This is because grantees used the grants to hire technicians and LSPs. However, they could not fulfil demands due to generally low levels of skill, especially in pest management and the burden of too many groups to service. The grant sub-activity participants also were better able to participate in KISAN activities and crop marketing activities. IDE Nepal trained 91 KISAN staff in Integrated Pest Management (IPM) practices and technologies, which were widely adopted by farmers. Agrovet capacity were exposed to new IPM technologies (e.g. Pheromone trap), and their capacity with respect to these technologies increased, allowing them to provide higher quality extension services to farmers. Agrovets do not want to dedicate their time unless compensated financially by KISAN. At the close of the activity, KISAN will no longer provide compensation, and there may be few sales in accessible farm communities. In the current situation, agrovets cannot provide what is needed -- an effective and embedded extension service. The BLP intervention improved the literacy and numeracy skills of 54,000 farmers, including 22,000 direct beneficiary farmers, increasing their capacity to learn new techniques. However, many of those trained remain illiterate and still struggle. 4.2.4) AGRICULTURAL PRODUCTION AND POST-HARVEST TECHNOLOGIES AND PRACTICES ADOPTED AT FARM LEVEL The yield data comparisons (refer to evaluation question 1) show that the yields achieved by project beneficiaries are substantially higher than those reported by the Ministry of Agriculture as typical district averages. The income changes reported demonstrate that farmers are now growing vegetables commercially. Where lead farmers did receive assistance for cereal crops as a part of demonstration plots, the improved seeds, more systematic plant spacing, and improved nursery management for rice increased yields substantially. All groups reported receiving new technologies from KISAN, although not all groups received training in all the technologies. KISAN advocated eight categories of technology with 23 individual 42 By KISANs own reporting, very few farmers kept the so-called log book effectively. 43 DADO Kailali and Dadeldhura. DADO require capacity building support to be more effective. Salary for DADO staff is only18,000 NPR per month. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 13 USAID/Nepal Monitoring, Evaluation and Learning Project technologies. The broad categories are crop genetics, pest management, soil conservation, irrigation, water management, climate mitigation, marketing44 and post-harvest handling technology. The evaluation team found that farmers had been introduced to the technologies, including more systematic planting, raised beds, plant spacing, improved nursery management, the use of polybags for seedlings, IPM technology including pheromone traps, the use of bio-pesticides, and, to a limited extent, improved seeds. Soil solarization, crop staking, and mulching were identified by about three quarters of groups. Climate mitigation and marketing were identified by less than half, with post￾harvest handling adopted by over half of group members. Not all groups were able to benefit from irrigation support, because the amount of the KISAN grant was often insufficient to cover the cost of a pump or bore. Where there was support for irrigation, the funds were often insufficient to cover all members in the group.45 In Hill districts, soil conservation technologies are already practiced widely (terraces and contour planting, for example)46. Many beneficiaries complained that they were unable to afford the technologies advocated by KISAN, particularly the plastic tunnel houses and agricultural equipment. A plastic tunnel-house costs $150 to $250, which many farmers cannot afford. A small hand tiller costs at least $1,000. Many farmers and private sector beneficiaries wanted greater subsidies for farm inputs, particularly in the early years of the activity. KISAN did not generally provide subsidies, but a grant was provided for pumps, bores and pipes up to a limited amount (about $450). Post-harvest technologies were limited to basic grading and the provision of a limited number of crates47 for transport to market. There are no formal grading requirements with payments for grades, but payments are higher for larger sizes (especially tomatoes and potatoes). Other than some basic grading (by size and for color and cleanliness), there was minimal adoption or promotion of post-harvest technologies and none for produce processing48. KISAN did not introduce value addition processing technologies, packaging or branding initiatives. Maize shellers were introduced and sometimes donated to or purchased by farmers. Some groups received one grain polybag for storing seeds, to be used by several group members. There were no cold storage initiatives by KISAN. Interviewees indicate that a lack of cold storage in district markets is a major constraint that limits the shelf life of fresh produce such as green, leafy vegetables. The few markets visited by the team49 were dirty, fly-ridden and lacked hard standing, with the consequence that during the rains the market becomes wet and muddy. The lack of cold storage means that produce quickly deteriorates and is wasted. Currently, unquantified losses occur from wastage, decreased quality, and declining prices to producers and traders. Farmers indicated a reluctance to pay for cold storage, believing that this would not benefit them but would be a marketing chain cost, and benefit the distributors. 44 Marketing is broadly described and includes contract and lease farming, use of mobile phones, marketing information, and audio-visual materials. 45 The team were informed that to be implemented, irrigation support needed to be able to serve at least 8 to 10 households in the group and most groups consisted of at least 20 households. 46 It was the wrong time of year to see all the technologies demonstrated in the field, and if groups mention using the technologies the team cannot confirm that all farmers in the group are using them. 47 Around 40 crates supplied to market collection centers (MPCs). 48 MPC: Ramghat collection Centre Kapilbastu and Saatbariya Krishi Upaj Kankalan Kendra , report farmers bring vegetables without grading; MPC themselves have to hire labor for grading. 49 Visited five different collection centers in Banke, Kapilbastu Salyan and Dang. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 14 USAID/Nepal Monitoring, Evaluation and Learning Project Conclusions Most groups received mechanization demonstrations, and a few groups received machines, but few farmers or groups purchased the mechanization demonstrated, minimizing the valor of this training. While the grantee intervention had a significant impact on bringing new technologies to farmers, this sub-activity is largely unsustainable because the staff recruited by the private sector specifically for this purpose will mostly be laid off post-KISAN, as potentially over 60 percent will not continue after KISAN-I ends. While most KISAN technicians received adequate technical training, some KISAN technicians did not, particularly in pest management. The DADO technical staff are generally considered to have out-of-date technical skills. Evidence suggests KISAN did not improve capacity of DADO extension workers measurably. 4.2.5) MARKET EFFICIENCY KISAN partner activities were not designed to impact market efficiency (IPM, CSISA, BLP). However, most FGDs reported that they had been introduced to traders by KISAN, which helps facilitate some market efficiencies. 4.2.6) CAPACITY OF GON AND LOCAL ORGANIZATIONS KISAN did plan and coordinate activities with the DADO staff, but the activity did not provide capacity development for the GoN staff. KISAN planned and coordinated activities with DADO, although DADO technical capacity-building was not an objective. There are no monitoring indicators to report capacity-building for DADO staff, and the annual report for Year 4 talks about coordination activities with DADO and joint monitoring activities, but not about technical capacity building. The DADO chiefs reported that KISAN did not provide capacity-building for DADO staff. Farmer beneficiaries reported that they did see DADO staff more when they visited activity sites and events with KISAN staff, but the informants did not report any increase in the number of independent visits by DADO. In remote locations, DADO staff rarely visit villages (e.g. Saraswati Tarakari Utapadan Samuha, Palpa). In less remote locations, farmers can visit the DADO Agricultural Service Centers for advice, but active extension services by DADO staff appears to be lacking. The most effective support provided by DADO were the grants that supported KISAN in providing irrigation support, which was particularly important in the Hill districts where the KISAN funds for irrigation were insufficient. Some individual farmers also reported receiving grants from DADO for tunnel houses and collection centers,50,51 and the evaluation team also saw plastic tarpaulins provided by DADO that served as linings for water storage ponds (although these were not in groups where FGDs were carried out). By working with savings and loans groups, and through linking MFIs to these groups, the MFI networks were also expanded. This is evidenced by the increase in membership of the multipurpose cooperatives and the additional demand reported by MFIs for loans to KISAN members. Because of the additional incomes reported by farmers, the level of savings increased, and consequently the loan amount available to members of SACCOs increased as well. The distinction between MFIs and SACCOs was not clear to many KISAN beneficiaries. However, they understood that SACCOs generally provide small loans to farmers, often with no collateral or 50 In Salyan, DADO supported the construction of a collection center. 51 In Salyan, one lead farmer had 21 tunnels and for some he had received a DADO grant or subsidy. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 15 USAID/Nepal Monitoring, Evaluation and Learning Project by using group guarantee. The MFIs are generally approached by farmers who require larger loans that are not available from SACCOs. For small loans, therefore, farmers use their own savings, loans groups and SACCOs; and for larger loans they use the MFIs. The Central Bank of Nepal has criteria that must be met by SACCOs and MFIs. KISAN supported SACCOs in technical training, especially in bookkeeping for their leaders (chair, secretary and treasurer), and provided ledgers, passbooks and stationary support. KISAN provided accounting software to some MFIs, but at least one of the SACCOs that the evaluation team met reported that the software had not been installed and that they continued to use the existing paper￾based system. Their staff also said that they did not have the capacity to use the software.52 Conclusions KISAN linked the savings groups to higher-level MFIs via networking activities, especially in urban centers. This expanded credit options for beneficiary farmers, with the potential for both greater investment and savings from additional income. Despite KISAN’s investment in training, little to no improvement was apparent in DADO technical or managerial performance. 4.2.7) FEATURES OF THE INTEGRATED USAID STRATEGY THAT HAVE MADE A DIFFERENCE IN OUTCOMES The BLP improved the literacy and numeracy of a significant number of farmers, mainly women. Farmers reported that it would have been more effective if it was implemented prior to KISAN activities. Beneficiaries likely would also have benefitted if the intervention is extended, with capacity-building continued for one day each week.53 KISAN was able to take advantage of the fact that the BLP was already developed and thus not difficult to introduce into activity communities. The Promoting Agriculture, Health and Alternative Livelihoods (PAHAL) activity adopted technologies previously tested by KISAN and used their training approaches and guides, although the extent of improvement was not evident through evaluating KISAN alone.54 The IPM intervention built the capacity of 91 KISAN staff (34 in year two, and 57 in year three) using already developed technologies; however, the IPM sub-activities were an add-on and not integrated into the KISAN activity from the start. The IPM intervention staff mentioned that the contractual arrangements for KISAN made it difficult to make adaptations to the KISAN workplan in order to integrate IPM activities.55 IPM technologies are widely implemented by farmers, but farmers are still very concerned about the increase in pest and disease problems and are worried about whether they will be able to continue to control pests and disease after the intervention ends. KISAN adopted tried and tested technologies from CSISA for rice, maize and lentils. According to the CSISA staff, however, the project would have been more effective if technical specialists – agronomists and mechanization experts – were appointed in every district to support the more generalist KISAN AMTs. Conclusion The activity gained somewhat from partnerships with other USAID activities, IPM, CSISA and BLP. As they were not closely designed for easy integration, activities supported each other modestly through capacity development and building off information and lessons learned. 52 Sahaj Savings and Credit Cooperative in Kaplivastu. 53 13 groups did not mention the BLP in their community. 54 Mark Pommerville, PAHAL, COP, Mercy Corps, Kathmandu. 55 IDE Nepal, Kathmandu. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 16 USAID/Nepal Monitoring, Evaluation and Learning Project 4.3) EVALUATION QUESTION 3 How do activity beneficiaries (disaggregated by sex and marginalization) perceive impacts of the interventions on household farm income, agricultural productivity/yield, food security and nutrition? What activity interventions/strategies do they find as helping or hindering increased agricultural productivity/income and improved nutritional status? a) What changes have taken place in the household income, food security, and nutrition of target populations as a result of the interventions? b) What explains such changes, particularly with respect to the most marginalized households? c) To what extent have the interventions had an impact on women’s incomes and livelihoods, particularly among the most marginalized? d) To what extent have the interventions had an impact on the nutritional status of women and children, particularly those under age 2? 4.3.1) CHANGES IN THE HOUSEHOLD INCOME, FOOD SECURITY, AND NUTRITION OF TARGET POPULATIONS AS A RESULT OF THE INTERVENTIONS AND FACILITATING AND HINDERING FACTORS All beneficiaries in FGDs reported that their incomes, food security and nutrition benefited when working with KISAN. Benefits varied widely between groups and between individuals within groups. Beneficiaries reported increased incomes, mainly from vegetable sales. A few farmers noted the equivalent of $5,000 to $7,000 increases when they used irrigation technologies and were able to grow vegetables out of season. More typically, the range of additional seasonal income was $60 to $80 per month, to $250 to $600 for the more successful farmers. In the Terai, farmers supported by KISAN were able to double their income, even without irrigation. Saraswati farmers groups in Banke reported that their income increased to the equivalent of around $70 (7,000 NPR) per kattha (1 kattha = 339 square meters, 1 ropani i= 509 square meters). Dalit women farmers now sell their surplus vegetables which amount to 10 percent to 15 percent of production. In the past, they had to buy vegetables for their own use. Laliguras women’s group in remote area of Daduldhera, who do not have irrigation, reported that they sold six to seven kilograms of cucumbers in season and earned around $0.25 (25 NPR) per kilogram (180 NPR, or $1.75 for 7 kg). Women reported lower incomes than men, with the exception of female lead farmers who really embraced vegetable farming, including plasticulture (use of plastic sheets to form and shape vegetable beds). In some cases, these female farmers had access to remittances which helped them to purchase farm inputs and construct plastic tunnel houses which are a relatively high cost ($180 to $250 each). In many cases, the women who sold in lower volumes were less commercially motivated and relied on remittances from family members working outside of Nepal or in urban areas within Nepal. The Dibyajyoti women farmers group in Kapilvastu, the Laligurash farmers group in Daduldhara and the Bishwa Khola Farmers group in Kapilvastu all reported relying on remittances for food security. Dibyajyoti women farmers also cited problems with land availability and sales. In all groups, when men were working away from home, including overseas, remittances were the primary income source, and KISAN activity incomes were secondary. Exceptions were women whose incomes exceeded those of some males due to successfully adopting KISAN-advanced technologies. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 17 USAID/Nepal Monitoring, Evaluation and Learning Project The most successful Terai farmers in remote areas were either close to the market where they were able to sell individually or had road access and cooperated with other farmers. Traders do not visit small villages where produce volume is too small to be worthwhile. Irrigation is key to extend the growing season and make production worthwhile for famers and traders. Most beneficiaries in FGDs reported that they did not grow vegetables for sale before working with KISAN. Following KISAN interventions, families now consume and sell a greater diversity of vegetables. FGD members reported that growing one kattha56 of vegetables was equivalent, in terms of income, to four kattha of cereal crops. Households now sell vegetables to buy rice and maize that may have been displaced by vegetable crops.57 Although vegetable production was the focus of the project, KISAN-supported groups reported larger cereal crops for consumption, but not for sale. They reported that they now had sufficient cereal crops for consumption for six months, an increase from the three months cereal crops they had prior to working with KISAN. Interventions included improved varieties, systematic planning and improved nursery management for rice.58 A factor that farmers noted as hindering income is limited or dated information on market prices. Many farmers do not know what margins are made between the local markets and the larger urban markets. Farmers access price information through communications with traders by phone or in person. There are no mobile apps and farmers generally do not have smartphones and no push SMS service providing rate information that could help offset market imbalances. This is seen as a factor that hinders agricultural income and smoothly-functioning markets. Beneficiaries (primarily women) who received BLP literacy and numeracy training also received a module on nutrition and health and are now better informed about nutrition59. 54,000 farmers, including 22,000 KISAN direct beneficiaries attended the BLP training. Most of the FGDs conducted with women confirmed that they are better informed about nutrition through BLP trainings. Women reported eating more vegetables and feeling healthier. Nutrition and health was one of five BLP modules. Conclusions All beneficiaries in FGDs reported that they had benefited from working with KISAN with respect to income, food security, and nutrition, mainly from promoting sales of a diverse set of vegetables. Key interventions included improved varieties, systematic planning, and improved nursery management for rice. Although vegetable production was the focus of the project, KISAN-supported groups reported greater food security in terms of cereal crops for consumption. Beneficiaries also point to the BLP module on nutrition and health. 4.3.2) CHANGE IN HOUSEHOLD COMPOSITION, INCLUDING MIGRATION. CHANGE BY INDIVIDUALS TO REPLACE MIGRANT WORK WITH AGRICULTURAL WORK RESULTING FROM IMPROVED TECHNOLOGIES, MARKET ACCESS, HIGHER RETURNS, ETC. The KISAN activity has had very little impact on those migrating out of the country to seek work. The team did come across some instances where men returned from working abroad and decided to stay at home rather than return abroad. 60 As an example, beneficiaries mentioned that if someone could earn a net income equivalent to $1,500 from growing vegetables, it would be 56 1 kattha 339 square meters or 0.0339 hectares. 1 Ropani 509 sq. m or 0.0509 ha. 57 Kisan Sahara Farmers Group, Palpa, Laliguras Women’s group Dadeldhura 58 Suthabari Biu Tatha Taja Tarkari Utpadan Samuah, Surkhet, Kisan Sahara, Palpa, Laliguras women farmers group, Dadeldhura 59 Nutrition and health was one module of 5 BLP modules. 60 Ghansingh Dada Farmers group, Palpa, Makhamali Krisak Samuha, Kapilvastu. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 18 USAID/Nepal Monitoring, Evaluation and Learning Project worthwhile to remain in the village instead of migrating abroad for work. This would require earning between 300,000 to 500,000 NPR from farming ($3,000 to $5,000) to reach $1,500 net income. Much of the motivation for staying at home is the hardship of working abroad rather than the income that can be earned from working in agriculture at home61. Although there were few cases in which beneficiaries mentioned family members stayed home rather than migrate to work overseas, beneficiaries did report that there was less need to work as a day laborer as a result of working with the KISAN activity62. Men still needed to take time off from farm work to earn income as day laborers (e.g. driving rickshaws, laboring or working in brick kilns), but dependency and time off was reduced and it is possible resilience might be increased. Some households have very small plots that are not sufficient for adequate income generation. In these situations, male household members still migrate, and the women manage the plots at home. Remittances from migrant workers also facilitated the affordability of new technologies like plastic tunnels, which some farmers say are unaffordable. In Tharu communities there has not been significant change, and few family members migrate, instead continuing established practices in their communities. Most Tharu community family members (Bhajani VDC) work on the farm to grow cereal crops and vegetables.63 Migration for work can also involve large costs when commissions must be paid to agents to find jobs overseas, particularly for work and travel to Gulf countries. In these situations, additional farm income can make it more possible to meet migration costs rather than prevent migration. Conclusions When net income from farming is more than $1,500 per annum, it is sufficient to encourage migrant workers to stay at home and undertake agricultural activities. This would require a gross income of around $3,000 or more. Remittances from migrant workers also facilitate the affordability of new technologies like plastic tunnels and mechanization. 4.3.3) EXPLANATION OF SUCH CHANGES, PARTICULARLY WITH RESPECT TO THE MOST MARGINALIZED HOUSEHOLDS Groups such as Dalits, Janajatis and Muslims are included in the category of “disadvantaged” people, but most participated fully in KISAN activities. There were some exceptions, such as Dalit members from the Laliguras farmers group and the Lataganyap group. The most marginalized have very little or no land, so there is little benefit to be gained from participating in KISAN activities. Participation in livelihoods training that does not require land would have been relevant for these community members, though very few participated.64 In one FGD, eight Dalit members from Lataganyap, Taja, Tarkari, Utpdan, Mahila, Krisak, and Samuha indicated they were excluded in Dadeldhura65 and reportedly did not receive ongoing KISAN support. The Lataganyap women group reported that these members were removed from the group in consultation with KISAN staff because “they never did the savings and repaid the loan” and “did not show interest in attending the group meeting and growing vegetables.” However, KISAN staff indicated they were not consulted. Conclusions 61 BABAI FGD, an irrigation Group in Dang (2 people mentioned); Jana Jagriti Vegetable Production Group (one farmer came back and remained to grow vegetables). 64 Two families in Janchahana Farmers Groups, Dadheldura district, who had migrated to the village and were landless. 65 In Jananchana Taja Tarkari Utpadan Samuha in Dadheldura, Dalits also mentioned they had no land and did not have any benefits form the project. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 19 USAID/Nepal Monitoring, Evaluation and Learning Project Although most groups of beneficiaries fully participated in KISAN, those considered the most marginalized, who owned little or no land, did not appear to benefit as consistently from activities. 4.3.4) EXTENT THE INTERVENTIONS HAD AN IMPACT ON WOMEN’S INCOMES AND LIVELIHOODS, PARTICULARLY AMONG THE MOST MARGINALIZED Seventy percent of project beneficiaries were women. Women reported that they received agricultural training and that there were few barriers to participation. Especially for those with no or little land, other training such as tailoring and livestock farming would be appropriate. Where women were lead farmers committed to vegetable production, and especially when using new plastic tunnel technology, incomes rose to equal to or better than men’s incomes. Other women’s incomes increased $60 to $80 in the season. Women’s traditional roles have changed in the households where husbands migrated or where they have become involved in KISAN. They now make decisions in consultation with their husbands and are engaged both at the farm and household level. The majority of women reported that their workload has increased. Although 70 percent of activity farmer beneficiaries were women, all the grantees the team interviewed were men, as KISAN confirmed that there were only 2 female grantees out of 39. The evaluation team did not find any female grantees in the 8 evaluation districts. Women did not participate in business activities off the farm due to their household responsibilities and limited access to credit. A guarantee is required to get a loan from the bank. Cultural constraints also limit women, for example, perceived travel restrictions away from home. Additional incomes enhanced women’s access to credit because the savings levels in the KISAN-supported groups improved, resulting in more funds available for lending by groups to beneficiary members. KISAN activities also enhanced women’s awareness of saving and loans, and one of the BLP modules included access to finance. Conclusion The BLP intervention enhanced women’s confidence and the respect they have from other family members in their households. 4.3.5) CHANGES IN INCOME AFFECTING HOUSEHOLD GENDER DYNAMICS, INCLUDING GENDER-BASED VIOLENCE Women report that there are activities against gender-based violence and there is more awareness of gender-based violence as a problem. However, the KISAN activity did not include such activities, and in the public forum of a FGD, women are reluctant to speak about GBV issues. The relationships in households have improved but there is no evidence that domestic violence has changed because of KISAN. In one group, the beneficiaries mentioned that no change had occurred66. Women reported that through earning their own income they felt more independent and were more respected by other members of their households. Most of the BLP participants were women, and the literacy and numeracy training had improved their confidence and enabled them to speak up in situations where they had previously remained quiet. Stakeholders report their daughters-in-law are now more respected by their mothers-in-law and other members of the household. Most women reported that they make decisions with their husbands and that their husbands now have more respect for them as they are able to earn income independently. There were almost no FGD members who reported that they were not able to contribute to household decision-making. Where husbands are working abroad, women may be the main decision-makers in the household, but in these circumstances, they try to collaborate with their husbands when making important household decisions. 66 Laliguras Women’s group, in Salyan. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 20 USAID/Nepal Monitoring, Evaluation and Learning Project The additional spending priorities for women were education for children, food and general household supplies. Purchase of rice and maize would have increased if the KISAN activity did not support improvements in cereal production (since vegetables have displaced an area of cereals), but in areas where KISAN did support improvements in cereal cropping, the production of cereal crops has also improved. Even those with very little land have improved food security and nutrition through growing vegetables. Small surpluses for sale help with home expenses and children’s education. Conclusions Most of the high income earning farmers are still men. Men continue to have access to greater resources and are better able to participate in marketing activities. Men continued to be the dominant participants in the KISAN interaction workshops that introduced farmers to other stakeholders such as traders and agrovets. The respect for women in the family improved as they increased their ability to contribute to family income through KISAN and BLP training. Many female beneficiaries now make financial decisions jointly with men. Income for women has allowed women to support the family. In line with findings across countries, women place a higher priority on household expenditure than men, giving high priority to children’s education and family food supply. Although the KISAN activity did not include GBV-related interventions, there is more awareness of GBV as a problem within the family. 4.4) EVALUATION QUESTION 4 In what way have KISAN interventions (including engaging in specific value chains) contributed to changes in the market systems and private sector investment in agriculture? a. What interventions changed the functioning of market systems in KISAN districts (i.e., expanded and inclusive markets – small-holder farmers, small enterprises; collection centers/aggregation of products; improved sales/prices; access to markets, inputs/services, financial services; and, investment)? b. What interventions have had unanticipated negative impact on functioning of market systems in KISAN districts? 4.4.1) INTERVENTIONS THAT CHANGED THE FUNCTIONING OF MARKET SYSTEMS IN KISAN DISTRICTS Farmers’ linkages to traders were improved by KISAN through introductions to local traders and through the small book of trader contacts produced by KISAN, which most groups had access to. This expanded farmers’ marketing horizons. Traders participated in exposure days where they met farmers in groups. Grantees, including agrovets, millers, seed companies and MFIs were supported to provide additional services to farmers and were linked to farmers to expand their purchasing network. However, the grantee intervention started quite late in the activity life cycle and there were only a total of 39 grantees. Grantees were supported by the grants to recruit technicians and LPSs to support KISAN activities. These staff did provide additional outreach to farmer groups and expanded trading networks for the grantees, but in most cases these staff will not be retained by grantees after the end of the activity. Leading farmers have become traders because of encouragement from KISAN, and these new farmer-traders are assisting other farmers in the groups to sell their produce67. 67 Lead Farmer in Dang now listed as trader, Chaibair Chaudhary. Chanak Chaudhary in Kailali District. An LSP in Salyan, Chet Badhur Oli. Lead farmer in Dadeldhura, Arjun PD Joshi. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 21 USAID/Nepal Monitoring, Evaluation and Learning Project Vegetable production is more profitable in Hill districts with good market access and irrigation. Farmers in Hill districts report no problems selling vegetables during peak production from around May until October. By contrast, farmers in the Terai report problems with marketing smaller volumes of vegetables and with falling prices because of the additional production.68 Farmers complain that traders have too much negotiating power because farmers are not informed of the margins that traders make between farm and final market. Farmers said that they need to know the prices in the main markets to assess what margins the traders can make after purchasing in the village. Traders stress the risk in making purchases because prices in district markets can fluctuate significantly, even during a single day depending on supply and demand. Communication with traders is still mostly by phone or in person. There are no mobile apps and farmers generally do not have smartphones. Some farmers69 mentioned that they had benefitted previously from an SMS market price service70 provided by a different organization. While the service was a weekly update, farmers thought that daily price updates are necessary as the price of vegetables fluctuates every day. The daily prices of vegetables from bigger markets throughout the country is updated on a government run website71 and displayed online. It is not feasible for farmers to go online to check the website. There are no cool or refrigerated stores in the district markets (with some exceptions noted for potatoes, for example72). Thus, perishable goods, like fresh vegetables, have a very short shelf life that can be exacerbated in the heat of the Terai, which can exceed 45 degrees Celsius. Some groups reported that farmers do not cooperate effectively to jointly market produce in the volumes required by traders. These groups tend to be those with good market access where farmers can easily sell their own produce. In places that do have vehicle access but that are not close to the market or highway, farmers are forced to cooperate to sell commercial volumes of vegetables. Traders are often not interested in traveling to villages to collect from less commercially￾oriented groups selling in small volumes. Traders require that farmers provide at least 500 kilograms worth of goods to make collection worthwhile. The worst marketing situation is for remote locations in the Hill districts because farmers cannot use bicycles and must carry produce to market. These locations do not have any vehicle road access73. The activity provided training to savings and loans groups in accounting and provided ledgers and passbooks. Strengthening savings, combined with the additional income the activity generated for farmers, expanded membership of the KISAN-supported savings and loans groups. Increased saving also increased the amount available for borrowing within groups. For example, Mahila Uhhan Bahu Udesya Sahakari Santha Ltd., a women’s multipurpose savings cooperative in the Hirminiya, Banke district, doubled its membership by 197 members with recruits from KISAN beneficiaries to a total of 388 Members across 52 groups. Some SACCOs reported that they find it difficult to compete with MFIs that have field staff in communities. This is because MFIs can more conveniently mobilize savings from community members and offer immediate credit rather than insist on a three-month membership period before advancing loans. KISAN was effective in linking the savings groups to higher-level MFIs because of their networking activities, especially in interaction meetings with stakeholders in urban centers. MFIs can advance 68 Makhamali Farmers Group, Kapilbastu 69 Babai Jaivik Bibidhta Group 70 Provided under Community Smart Village by Li-bird 71 www.agribiz.gov.np 72 Kohalpur Agricultural Producer Wholesale Market. 73 Laligurash Farmers Group, Dadeldhura, Lataganyap Farmers Group, Dadeldhura, Namuna Tarkari Utpdan Krisi Samuha, Surkhet. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 22 USAID/Nepal Monitoring, Evaluation and Learning Project larger loans to farmers than the savings and loans groups. The Nepal Women’s Community Service Center MFI in Dang was effective in maintaining a good working relationship with the KISAN activity, stating that 65 percent of KISAN group members across 50 groups had now taken out loans with them. Conclusions Farmers access price information through communications with traders by phone or in person. There are no mobile apps and farmers generally do not have smartphones and no push SMS service providing rate information that could help offset market imbalances. Many farmers do not know what margins are made between the local markets and the larger urban markets. KISAN linked groups to MFIs via networking activities. This expanded credit options for beneficiary farmers, which resulted in both greater investment and savings due to additional income from KISAN activities. 4.4.2) SATISFACTION OF MARKET ACTORS WITH CHANGES TO SERVICE￾PROVISION, ACCESS AND OTHER MARKET SYSTEMS. CHANGES NOTICED BY BENEFICIARIES IN SERVICES RECEIVED FROM GON (DADO) AS A RESULT OF TRAINING In more accessible communities, beneficiaries reported seeing more of the DADO staff, and they frequently came to the communities with the KISAN staff. In remote communities however, beneficiaries rarely, if ever, see a DADO officer or any agrovet technician. Out of 32 groups interviewed, 12 noted that they were never visited by DADO or an agrovet technician, and that they only received extension services from KISAN technicians. Farmers also said that the DADO technicians are not technically up-to-date and do not have the resources to undertake regular monitoring. Farmers also can possess more knowledge than DADO staff74. Both KISAN staff and DADO staff confirmed that the DADO staff participated with KISAN in planning and coordinating activities. However, the DADO staff were not perceived by beneficiaries as having provided training to the KISAN beneficiaries, nor have the DADO staff received capacity building from the KISAN activity. DADO staff are perceived as being under-resourced and understaffed, and the DADO technical staff are considered to have outdated technical skills. DADO did contribute capital funds to partner with KISAN in funding irrigation scheme improvements, especially where the KISAN funds were insufficient and limited to around $350 to $450 (35,000 to 45,000 NPR) per scheme. Most of the schemes are in existing DADO registered groups. In some cases, these are new registrations facilitated by the KISAN activity. In Hill areas, the amount of capital for irrigation improvements were inadequate, but interventions were possible when KISAN funds were partnered with DADO grants as well as some contribution of labor or money from the farmers. In more remote locations, private sector initiatives may not be viable because agrovets depend on sales of products to make field visits financially viable, and the time and cost involved in visiting remote communities is not viable. The intervention funded grantees to participate in activities (to provide training and services to farmers in the groups), and the activity paid LSPs to participate in and provide training to farmers. Most grantees the evaluation team interviewed said that they would not continue to employ some of the technicians they had recruited as part of the intervention. (Overall, at least 60 percent of the recruited staff will be let go at the end of the activity.) Further, most LSPs will cease to provide services, as they will no longer get paid after the KISAN ends.75 Some of the LSPs that worked with 74 Unnat Krishi Kendar Agrovet Shop. 75 LSP1 were expected to work for the project for at least 10 days a month and were paid 4,000 Rs a month for their services. LSP2 were called as needed to provide services and were paid 400 Rs per day. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 23 USAID/Nepal Monitoring, Evaluation and Learning Project KISAN (or who were employed by the grantees) have graduated to agrovets in their own right, and in some cases they have become farmer-traders. These individuals are exceptions.76 Because of grantees’ inability to continue to employ many of the technical staff they employed while KISAN beneficiaries (the exact number varied by grantee from one or two, to the largest milling business employing around ten); the future lack of travel allowances provided to KISAN private sector participants for visiting farmers; and due to payments ending for services to LSPs; the level of private sector involvement in providing services to farmers would not continue at the levels found during KISAN. Conclusions KISAN beneficiaries did not perceive that DADO staff provided them with training, nor have the DADO staff received capacity building from the KISAN activity. Beneficiaries perceived DADO staff as being under-resourced and understaffed. DADO technical staff are considered to have outdated technical skills, and remote communities rarely see a DADO officer or any agrovet technician because agrovets still must depend on sales of products to make field visits financially viable. Combined contributions by DADO and KISAN in funding irrigation scheme improvements were viewed favorably but were largely limited to existing DADO-registered groups. The level of private sector involvement in providing services to farmers are not expected to continue at the levels found during KISAN, as travel allowances would disappear and grantees were expected to reduce by 60 pecent the technical staff employed. 4.4.3) INTERVENTIONS THAT HAVE HAD UNANTICIPATED NEGATIVE IMPACT ON FUNCTIONING OF MARKET SYSTEMS IN KISAN DISTRICTS Many activities, including KISAN, have been promoting vegetable production. KISAN supported over 80,000 farmers directly and reports assisting even more indirectly. The result of greater supply was lower prices, especially at peak production periods. Most of the farmers who were engaged in vegetable farming as a business reported falling prices in peak season due to high production. They report that vegetable production is profitable only if they can produce in the off-season. There was insufficient variation in type and timing across seasons to offset this unintended negative consequence. The evaluation team encountered situations where the farmers were feeding cabbages to livestock because the price was so low that the crop was not worth selling.77 Additional production of vegetables without profitability is frustrating to beneficiaries. This is particularly a problem in the Terai, where production coincides with traditional production periods that are dependent on weather, rather than using technology that extends the season, such as irrigation and plasticulture. KISAN models do not include labor inputs by the farm family, as USAID FTF gross margins require only direct cash costs to be collected. Direct cash costs are low, so inclusive of family labor and local opportunity costs, along with the low prices, means that farming may not be profitable. The KISAN activity demonstrated mechanization options to farmers. Farmers are skeptical about replacing animals with mechanized equipment, however, and indicate doing so also would mean a loss of animal manure which is advocated as an organic fertilizer,78 increasing costs of production. The KISAN activity has increased the area in vegetable production and the locations for vegetable production are now more geographically dispersed. This means it is now easier for pest and diseases to spread. Thus, an unintended effect is that farmers now report an increase in pests and diseases, including new ones they do not know how to control.79 The ability to manage diseases and pests 76 Chet Bd Oli, LSP Salyan, Arjun PD Joshi-LSP, Trader, Lead Farmer and Agrovet, Dadeldhura 77 This feedback was provided by farmers both in FGDs and in KIIs. 78 Kisan Sahara farmers group, Palpa, Babai Jaivikbibidhika farmers group, Dang 79 Kisan Saraha, Palpa, Shrijanshil Farmers Group, Salyan. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 24 USAID/Nepal Monitoring, Evaluation and Learning Project was the challenge most frequently cited by farmers, in addition to water availability for irrigation. Most agricultural chemicals have labelling in English, Chinese, and other languages. Farmers, who are often illiterate in their own languages, are unable to read product labels, including toxicity directions. Agrovets do explain safe chemical application practices and application rates, but not having easy-to￾understand instructions in an appropriate language is a hazard. Difficulty sourcing and unaffordability of safety gear (such as aprons, gloves goggles, etc.) magnifies hazards, even when farmers are educated on proper use. The evaluation team did find some of this equipment in some agrovet shops.80 Despite the activity focus on bio-pesticides, the evidence of increasing pests and diseases makes the use of agricultural chemicals increasingly in demand. Many farmer beneficiaries complained about the adverse effects of using agricultural chemicals, and the team met one lead farmer who appeared to be very sick.81 This farmer blamed the use of chemicals as the problem, and in other cases, farmers complained about irritation to the eyes and skin due to chemical use. Spraying tomatoes and other climbing crops in tunnel houses is especially hazardous because of the enclosed space. Agrovet-provided seed (and KISAN-provided seeds to lead farmers on demonstration plots) were not always of high quality, and there were frequent beneficiary reports of poor germination or lack of germination for seeds (see previous discussion, 4.2.1).82 Agrovets individually buy small quantities of products from bigger suppliers, particularly for vegetable seed, and much of the seed is imported. In the case of rice and some other cereal crops, local companies and NARC are trying to fulfil farmers’ demands. There were few complaints about rice seed, but farmers were requesting hybrid or high-yielding varieties for both maize and rice. Some agrovets reported that they stock seed until its expiration date and then cannot sell it, although others reported that they could exchange expired seed. The fact that they stock seed until the expiration date may mean that they receive old imported stock. Conclusions Several different unintended consequences arose from KISAN directly or indirectly. Key among them, rapid expansion of vegetable production by KISAN and other activities has reduced the price and profitability for vegetables, especially in the main harvest season. There was insufficient variation in type and timing across seasons to offset this unintended negative consequence. Further, he expansion of vegetable production has helped spread pests and diseases among vegetable farmers, who will struggle to continue to control them without the continued services of the KISAN technicians. Although KISAN emphasized bio-pesticides, some farmers found an increased need to use chemicals without the literacy to understand proper use with the potential for suffering adverse effects of using agricultural chemicals (irritated eyes and skin). There was insufficient focus on the BLP and other training interventions involving literacy especially for use in agriculture technology, as many farmers continue to be functionally illiterate and challenged by new technologies they likely will not continue after KISAN support ends. There were instances of agrovets providing poor quality seed that sometimes did not germinate, leaving farmers to lose the crop and suffer a loss of income. 4.5) EVALUATION QUESTION 5 Based on findings from effectiveness questions, what factors, interventions or changes to systems or institutions (public and private) may contribute to sustainability in follow-on activities? 80 The evaluation team found the appropriate safety equipment in Unnat Krishi Kendar Agrovet shop in Salyan. 81 Lila Kunwar, Lead Farmer, Palpa. 82 In Banke, Palpa, Dadheldura, and Dang districts. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 25 USAID/Nepal Monitoring, Evaluation and Learning Project Farmers close to the markets are more accessible to the private sector and are better situated to sustain activities in the future, as are those farmers who graduated to competitive status during the life of KISAN. The rest will “continue with what KISAN taught us as best we can.” Those in remote areas will continue to receive little technical support due to the lack of access to their communities and agrovets’ unwillingness to collect from places where they can make little profit. Farmers interviewed called for an improved source of same-day market information to decrease imbalances in information that allow traders to dominate negotiations and prevent closer to real￾time adjustments in supply where demand is low. Farmer groups indicated they have a higher probability to be sustainable when they are serviced by an active cooperative that is effective in aggregating crops for sale and providing inputs through an agrovet shop in the community. This is also the case if they are relatively remote but have road access, even a poor one. Lack of market aggregation for traders to deal with farmers in remote areas leads to inefficiency and prevents KISAN’s interventions from being spread to and sustained in more-remote locations. The market executive committees in the district market centers did not receive much-needed capacity development through the project. There is minimal change to grading and post-harvest management. More careful analysis of market dynamics, unintended consequences, and structural constraints is necessary to strengthen commercial viability of value chains. Lack of focus on post￾harvest infrastructure and greater variation in products threatens gains made along the vegetable value chain. The district markets are poorly resourced, and management could be greatly improved. There is a need for cold storage to increase the shelf-life of fresh produce, and very little cold storage is now in place. KISAN reports that at the end of the activity, 75 percent of technical services were supplied by the private sector and only 25 percent by KISAN. The evaluation team did not see clear evidence of this proportion supplied by the private sector, even though this was the KISAN approach in the Terai rather than the Hill areas. Agrovets must profit from selling products, so it is not viable for agrovets to make visits to remote communities. Traders cannot visit villages with no road access and are unwilling to make collections from communities with less than 500 kilograms of vegetables to collect. Because most grantees who hired technical staff indicated they will lay off about 60 percent of these staff members after the end of the activity, the evaluation team considers that many KISAN farmers will struggle to continue with the promoted interventions. Out of the 12 agrovets interviewed, all will continue to work with farmers to some extent but not at the level they did with support from KISAN (which includes travel cost reimbursements, as well as grants used to employ staff). They will work with their contact farmers because it is beneficial for their business turnover, but the issue is whether they will continue to work with groups and undertake field visits to farmers’ fields. The most viable business appears to be the large maize miller in Dang which hired 10 Junior Technical Assistants (JTAs), an accountant and a manager with KISAN support ($93,000). It works with 202 farmer groups with about 20 members each (4000 farmers). They said they will retain eight of the ten JTAs. This is a notable exception, as it is by far the largest business supported that was interviewed. Although it is difficult to be precise, of the 12 agrovets interviewed, three seemed to indicate that they will continue as before (under KISAN). However, it is difficult to imagine they will operate at the same level, because they will no longer receive the grant support, travel cost reimbursement or motivation provided by KISAN staff. DADO staff will still provide services to farmers after the project ends, but the DADO staff are under-resourced and the DADO technical staff are considered by farmers, as well as by their senior managers, to have outdated skills. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 26 USAID/Nepal Monitoring, Evaluation and Learning Project Pests and diseases are a major problem for vegetable farmers, and they will struggle to continue to control pests and diseases without the continued services of KISAN technicians. While agrovets are available to help, the majority of agrovets are based in the urban areas and farmers must visit their shops to obtain advice and to purchase products. Farmers reported that fertilizer was difficult to receive on time and frequently is not applied at all because of the lack of timely supplies. As shortages of chemical fertilizer occur every year, farmers have asked the government to shorten the lengthy consignment process. In case of a crisis, the government can fast-track shipments by making a government-to-government deal with India, but the process needs to be fair and transparent. Currently, chemical fertilisers are imported through Birgunj customs only. The lawmakers suggested reviewing the existing Nepal-India transit agreement so that they can be shipped through other border points by signing a letter of exchange. It is the evaluation team’s understanding that Nepal has an option to import fertilizers through Visakhapatnam port, as well as through the Kolkata and Haldia ports as is being done currently. Farmers asked the government to open the fertilizer trade to the private sector by fixing their import quota and making a rational adjustment to the subsidy policy to induce it to come on board. Farmers involved in this study indicated they believe that the limited distribution system prevents farmers from getting fertilizer on time, and that many are forced to pay extra. Other factors that would increase sustainability of interventions or augment their effects noted in other sections of the report include irrigation schemes that were inadequately funded by activity plus user fees/individual investments, reducing sustainability. Low capitalization and insufficient numbers of individuals, groups, or companies engaged in reducing costs of mechanization by loan services reduced changes in planned changes in farmer movement toward mechanization. DADO staff do not have the resources to take over from KISAN after the activity ends. DADO staff are too stretched to service the needs of most farmers, and DADO staff skills are outdated with insufficient budget for travel. Agrovets will focus on the most successful farmers who provide a market for their products but will likely not help the neediest farmers when KISAN support is terminated. Farmer groups that are serviced by an active cooperative that effectively aggregates crops for sale and provides inputs through an agrovet shop in the community show a higher probability of sustainability. Farmers who live closer to markets have more access to private sector resources, and therefore are better situated to sustain KISAN activities. Low market prices for vegetables demotivate a proportion of farmers, especially in the Terai. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 27 USAID/Nepal Monitoring, Evaluation and Learning Project 5.0) RECOMMENDATIONS 5.1) DESIGN AND MANAGEMENT 1. To increase the extent to which interventions are relevant and cost-effective for targeted communities, implementers of USAID agricultural activities should undertake strong needs assessments for beneficiary groups and analyze carefully costs and benefits to select investments that offer the best return on capital. Consideration should be given though to varying value chains or components of the value chains targeted. 2. Since area farmers appreciated participating in cooperatives and they appear to work well in Nepal, USAID and its activities should consider supporting agricultural cooperative development and agricultural service developments (market aggregation and input supply) through these cooperatives. Their benefits may be particularly high in more-remote areas, in which they can provide a market aggregating function. 3. Since irrigation is perhaps the most limiting constraint of many farmers’ success, USAID’s implementers should include a focus on irrigation and design irrigation schemes to be adequately funded by activity plus user fees/individual investments, in order to ensure sustainability. Activities should undertake proper analysis of costs and benefits to increase the extent to which those investments that offer the best return on capital will be selected.83 4. USAID should carefully determine its desired tradeoff between improving the return and sustainability of investments versus a focus on poverty reduction for the poorest. A more rigorous vetting process would select more motivated and competent farmers, improving the return on investment, particularly for interventions attempting to build a sustainable competitive value chain. This approach, however, likely would not benefit many of the poorest and most disadvantaged. 5. Programming that helps farmers diversify income sources can have many benefits. USAID should consider funding components of agricultural activities that focus on livelihood diversification to test it as a strategy for reducing out-migration from agricultural areas, increasing resilience, and targeting poverty reduction. 5.2) POVERTY, DISADVANTAGED GROUPS, AND WOMEN 6. Poverty targets should be addressed separately from beneficiaries of whom there is an anticipation of commercial success and sustainability. To the extent that USAID wants to prioritize helping the poorest segment of the FTF ZOI rural inhabitants, activity programming should invest in interventions specifically for meeting poverty targets, in addition to lifting average yields or income or creating sustainable value chains. Priority targets for such programming include those who are landless, under-represented ethnic groups, or those geographically most isolated from markets. Programming should consider a range of options but not only promoting push strategies to encourage the poorest to participate in markets, as evidence suggests these programmatic targets may benefit only peripherally from farmer training. 7. To address food security for the poor and disadvantaged in locations with poor market access, KISAN-II and other implementers could increase their focus on kitchen gardens for 83 Irrigation supports out-of-season production which is normally of higher value. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 28 USAID/Nepal Monitoring, Evaluation and Learning Project household consumption rather than on commercial vegetable production. If there is no irrigation, this should be accompanied by more emphasis on water storage, collection and systems to reduce water use (e.g., drought tolerant varieties, mulching cereal crops, and improving soil organic matter to improve water holding capacity). Programming to target the poorest also would include livelihoods diversification. 8. Since landless individuals are some of the most disadvantaged, future activities should include the provision of livelihoods training, including ancillary activities within the farming value chain. This could include skills in agricultural mechanization, agriculture value addition and ancillary activities within the farming value chain (with credit access). Although it is more labor-intensive to provide such programming, the needs of the disadvantaged should be assessed and addressed separately from others with a smaller and more targeted approach. 9. Since the BLP was one of the most well-received interventions by women and to the extent that similar outcomes are still a priority for the Mission, an intervention like BLP should be continued and built upon to improve GESI-related outcomes. 10. KISAN-II and other future activities could emphasize women entrepreneurs and focus on creating new opportunities to develop female entrepreneurship. Consideration should be given to establishing a separate fund specifically for women, to help offset the financial market imbalance against women. 5.3) MARKETS, MARKETING, AND EXTENSION SERVICE 11. Future USAID activities like KISAN should either undertake interventions to make the market system more efficient and provide access to all KISAN beneficiaries or undertake activities that do not rely so strongly on market access and market efficiency. The existing marketing system was a major impairment to KISAN success, which market reforms can broaden an activity’s impact. 12. USAID can deepen and broaden the impact of its activities by supporting the GoN to improve market information systems that will allow farmers to access price information along the supply chain from farm to final market. Since most farmers do not yet have smartphones, it will need to accessible through multiple sources, local telephone service, public announcements or local publications, and online for the future. 13. Since the lack of cold storage in district markets limits product shelf-life, impacts prices, and reduces market efficiency, USAID should consider including post-harvest value chain interventions including infrastructure, such as storage, grading, and sorting, as well as capacity building for marketing executive committees within KISAN-II or future activities, as a way to reduce production losses and hits on farmer income. 14. KISAN-II and other IPs need to target carefully promotion of assistance with crops. For farmers in remote locations without road access, IPs should prioritize crops that are less perishable first and high value second, because they will require carrying or bicycle transport. 15. Given the absence of an efficient and effective DADO extension service, KISAN-II and future USAID activities should consider incorporating an incentive-based agreement with private agrovets and other service providers to ensure that extension services be provided to poorer farmers. For example, the activity could guarantee a return to the extension service based on specific targets being achieved by the farmer clients. Once targets are achieved, the relationship would become a farmer-service provider relationship only. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 29 USAID/Nepal Monitoring, Evaluation and Learning Project 5.5) CAPACITY DEVELOPMENT 16. USAID’s implementing partners should conduct training needs assessments and training plans for all agrovets when they are recruited as grantees or project partners. 17. Implementers should provide stronger, more consistent training of its technicians at the start of their employment and constant refresher training, because it is this staff that train the beneficiaries. 18. To increase sustainability of activity outputs and outcomes, KISAN-II and other USAID￾funded activities should include interventions that provide specific training programs in support of DADO staff to increase their capacities and ability to provide support over the longer term. 19. The danger of inappropriate use of agricultural chemical to farmers, particularly those lacking literacy and numeracy, is real and a potential negative unintended effect of donor-sponsored interventions that spread new products and potentially pests and diseases alongside them, even when attention is paid to the issue. KISAN-II and other interventions may need to consider additional supply- and demand-side approaches to helping illiterate farmers avoid these perils. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 30 USAID/Nepal Monitoring, Evaluation and Learning Project APPENDIX A: IMPLEMENTATION MODEL AND RESULTS FRAMEWORK Figure A1. The KISAN implementation model Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 31 USAID/Nepal Monitoring, Evaluation and Learning Project Figure A2. The KISAN Results Framework Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 32 USAID/Nepal Monitoring, Evaluation and Learning Project APPENDIX B: EVALUATION QUESTIONS Evaluation Questions Relevance 2. How do the main outcome level results achieved in Nepal’s FTF ZOI Districts compare to existing national statistics on agricultural productivity/yield, sales, consumption of inputs, and market efficiency, among others? Within the ZOI, how does data compare between FTF direct beneficiaries and non-FTF beneficiaries? Effectiveness 3. In what way has the interaction between different activities (i.e. strategy to integrate agriculture activities KISAN with IPM, CSISA and BLP) contributed to achieving or hindering outcomes in: (vi) Improved access to increased quality inputs for farmers? (vii) Improved capacity of agricultural extension workers, services providers and farmers to increase agricultural productivity/yield? (viii) Improved agricultural production and post-harvest technologies and practices adopted at farm level? (ix) Improved market efficiency? (x) Increased capacity of GoN and local organizations? b. What are the particular features of the integrated USAID strategy that have made a difference in outcomes? 4. How do project beneficiaries (disaggregated by sex and marginalization) perceive impacts of the interventions on household farm income, agricultural productivity/yield, food security and nutrition? What project interventions/strategies do they find as helping or hindering increased agricultural productivity/income? e. What changes have taken place in the household income, food security, and nutrition of target populations as a result of the interventions? f. How has household composition changed, including migration? Have individuals replaced migrant work with agricultural work as a result of improved technologies, market access, higher returns, etc? g. What explains such changes, particularly with respect to the most marginalized households? h. To what extent have the interventions had an impact on women’s incomes and livelihoods, particularly among the most marginalized? i. How have changes in income affected household gender dynamics, including gender-based violence? 5. In what way have KISAN interventions (including engaging in specific value chains) contributed to changes in the market systems and private sector investment in agriculture? a. What interventions changed the functioning of market systems in KISAN districts (i.e., expanded and inclusive markets – small-holder farmers, small enterprises; collection centers/aggregation of products; improved sales/prices; access to markets, inputs/services, financial services; and, investment)? b. How satisfied are market actors with changes to service-provision, access and other market systems? Specifically, have beneficiaries noticed a change in the services received from GoN (DADO) as a result of their (GoN) training? c. What interventions have had unanticipated negative impact on functioning of market systems in KISAN districts? Sustainability 6. Based on findings from effectiveness questions, what factors, interventions or changes to systems or institutions (public and private) may contribute to sustainability in follow-on activities? Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 33 USAID/Nepal Monitoring, Evaluation and Learning Project APPENDIX C: DETAILED METHODOLOGY The evaluation team reviewed documents provided by USAID, undertook 105 KIIs (details Appendix D) and 32 FGDs. These documents included the project M&E Plan, Annual Reports, the final project work plan, cost benefit analysis completed for rice, maize, and cauliflower, a market channels assessment report, FTF surveys and assessment, the mid-term evaluation of KISAN, and an analysis report for lentils, amongst others. The team reviewed the secondary data sources, and a separate review of these secondary sources was submitted to USAID: “Assessment of secondary quantitative data sources for KISAN activity performance comparisons” (submitted July 2017) (Appendix C). This report discusses the problems in comparing the secondary data sources with KISAN data; these include the small number of KISAN households within the targeted districts in comparison to the total numbers of households, the lack of comparable datasets, the fact that the years/periods represented by the data are different, and the fact that the KISAN districts are not necessarily represented by data that is available for regions and ecological zones (KISAN districts being only a subset). There is Ministry of Agriculture production data at the District level, but this data is not based on robust sampling methodology. According to the Ministry, it is representative of typical data for districts. At the time of this analysis, the latest publication of the Statistical Information on Nepalese Agriculture is for 2015 which presents typical yield data by district. In addition to the secondary data review and analysis, the team conducted field work in eight districts (four each from the Hill and Terai) as well as in Kathmandu (Table 3). The team completed 32 focus group discussions (FGDs) with farmer beneficiaries84 and 105 key informant interviews (KIIs) with beneficiary leaders, value chain actors, implementing partners and other stakeholders including partner projects (including some in Kathmandu). Key informants were selected randomly by the team in sufficient numbers to be representative of the beneficiary categories. A detailed breakdown of FGDs and KII completed is shown in Appendix D. For the KIIs there were 41 female participants and 104 male participants (Table 2). Table 2: Participants in Key Informant Interviews by Type of Key Informant, and by Gender Type of Key Informant No. Men No. Women Agribusiness 7 1 Agrovet 11 0 Collaborating 13 1 Cooperative 5 0 GoN 7 1 KISAN Staff 11 9 Lead Farmer 14 5 MFI 4 5 Miller 5 4 MPCC 5 4 SACCO 9 14 Seed Company 6 0 Trader 7 1 Grand Total 104 41 Data collection instruments were primarily qualitative in nature as USAID did not require a quantitative survey of farm households. The data collection instruments were: 84 40 were planned but the team decided on fewer when it was realized that no new information was being presented by beneficiaries in FGDs. USAID was informed of this revision. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 34 USAID/Nepal Monitoring, Evaluation and Learning Project ● Focus group discussion framework for mixed male and female farmer beneficiaries ● Focus group discussion framework for women only farmer beneficiaries ● Key informant interview (KII) for beneficiaries of the Business Opportunities grant scheme ● KII for input suppliers ● KII for market actors ● KII for micro-finance organizations The team also conducted key informant interviews, without pre-structured questions, with partner projects, KISAN staff including the technicians that worked with farmers, and with Government of Nepal (GoN) staff in District Agriculture Development Offices (DADO). Table 3: Number of FGDs and KII Conducted by District Districts No. FDGs No. KIIs Banke 3 17 Dadeldhura 5 11 Dang 3 17 Kailali 5 16 Kapilvastu 5 14 Kathmandu 0 5 Palpa 5 8 Salyan 3 8 Surkhet 3 9 Total 32 105 While the KISAN activity formed 4,300 farmer groups, the evaluation team conducted FGDs with 32 farmer groups, which is less than one percent of all KISAN groups. Of the FGDs conducted, 19 were with women only in the discussions and 13 were mixed groups of men and women participants. Thirteen groups were classified as having irrigation and 19 without irrigation. There were 19 groups classified as “disadvantaged” and 13 “non-disadvantaged” groups. A total of 363 beneficiary farmers participated in FGDs of which 190 were classified as being disadvantaged individuals85. KISAN classified farmer groups on a continuum from “A” being competitive to “D” being vulnerable. The number of groups included by classification was as follows: A Competitive 9 B Market orientated 13 C Developing 7 D Vulnerable 3 FGD groups were selected randomly by the Evaluation Team. FDGs were selected for 8 districts from the KISAN listings. In the selection of the 40 FGDs (later reduced to 32 because of repetition of findings from FGDs) the Evaluation Team included a sample from each of the classifications A, B, C and D roughly in proportion to the numbers by classification across all 4,300 KISAN groups, but with a minimum of 5 per category. For the vulnerable category it was difficult to identify 5 groups in the locations visited as classification D represented only five percent of all KISAN groups. Initially, the Evaluation Team selected five for classification D, but the actual number of three groups were determined by the availability of groups in the districts and locations visited by the team. However, these three account for almost five percent of the category D groups participating in the FGDs. 85 Dalits, Janajati, or Muslims as classified by KISAN. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 35 USAID/Nepal Monitoring, Evaluation and Learning Project During the progress of the field work the team regularly debriefed, and completed the preliminary findings, conclusions and recommendations matrix as the basis for the first draft of the report. In order to synthesize the data from the KII and the FGDs, the team summarized the data in an Excel matrix for each FGD and for KIIs (the primary types of key informant86) completed.87 In all FGDs, the discussions were led by one of the three evaluation team experts (the Team Leader, the Agricultural Sector Specialist, or the GESI Specialist). They were assisted by a Researcher/Facilitator who facilitated the guideline questions. The discussion leaders also asked follow-up or probe questions as required during the facilitation process. The Team Leader (international) was assisted by an interpreter. The discussion leaders made comprehensive notes at all meetings (FGDs and KII) and were responsible for transcribing these notes to a Word Document. Where groups did not speak Nepali, one of the research assistants was a fluent local language speaker. The field work was conducted in the eight districts between May 17, 2017 and June 23, 2017, in three field trips. Pretesting of the instruments was completed by 17th May, before the main field work was conducted. Limitations were stated in the evaluation design document (refer Appendix H). However, it should be noted that the KISAN activity formed 4,300 farmer groups and the evaluation team conducted FGDs in 32 farmer groups, which is less than one percent (1 percent) of all KISAN groups. Therefore, whilst the information from the FGDs conducted by the evaluation team is useful for obtaining information from beneficiaries for lessons learned, the number of groups making similar comments cannot be used to estimate the percentage of groups over all groups making such comments, because of the small number of FGD groups conducted over the total number of groups. 86 Input suppliers, market actors, GoN staff, and grantees. 87 It was envisaged that the Dedoose software would be used but it proved to be too time consuming to use this software and on the recommendation of the CAMRIS/MEL COP the approach adopted was changed. USAID was informed of this change. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 36 USAID/Nepal Monitoring, Evaluation and Learning Project APPENDIX C: ASSESSMENT OF SECONDARY QUANTITATIVE DATA SOURCES, FINAL Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 37 USAID/Nepal Monitoring, Evaluation and Learning Project FINAL PERFORMANCE EVALUATION OF USAID/NEPAL’S KNOWLEDGE-BASED INTEGRATED SUSTAINABLE AGRICULTURE AND NUTRITION (KISAN) PROJECT ASSESSMENT OF SECONDARY QUANTITATIVE DATA SOURCES FOR KISAN ACTIVITY PERFORMANCE COMPARISONS – FINAL AFTER REVISION July 2017 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by CAMRIS International. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 38 USAID/Nepal Monitoring, Evaluation and Learning Project ASSESSMENT OF SECONDARY QUANTITATIVE DATA SOURCES USAID requested the evaluation team to assess USAID/Nepal’s effort to increase agricultural productivity and private sector investments in agriculture in the FTF Zone of Influence (ZOI). The first step is to conduct a desk review of existing data to analyze differences in agricultural data between the FTF ZOI and the national statistics, as well as within ZOI districts between direct FTF beneficiaries and non-FTF beneficiaries. GENERAL PROBLEMS WITH SECONDARY DATA SOURCES 1. COMPARING KISAN ACTIVITY DATA IN THE FTF ZONE OF INFLUENCE WITH NATIONAL STATISTICS The KISAN activity worked with around 100,000 direct beneficiaries in 20 districts. This represents between seven percent and nine percent of all households in the project districts. Even adding other indirect beneficiaries who may have benefited from KISAN through services provided by the private sector service providers who worked with KISAN (reported to be another 20,000 to 30,000 indirect beneficiary farmers) only brings the percentage of beneficiaries in the KISAN districts to 9.8 percent of farm households. Table 1: Number of Beneficiary Farmers by District, and Number of Farmers in the Annual Farmer Survey by District Based on the relatively small percentage of project beneficiaries in districts (an average of only 7.9 percent), it is unlikely that any measurable88 district-wide or FTF ZOI impacts would have occurred as a result of KISAN. Even if 88 For example, an increase in incomes for KISAN beneficiaries of 50% would equate to an increase of only 3.95% across the entire population of the districts (assuming KISAN beneficiaries are similar to non-beneficiaries), which is within the margin of error for most surveys. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 39 USAID/Nepal Monitoring, Evaluation and Learning Project there were a small measurable change, it would be difficult to ascribe attribution to the KISAN activity for any positive change because of other factors such as weather, work being done by other projects, which may also have affected agricultural production and food security. Data from the KISAN activity beneficiaries may be compared to national statistics (district data which would include KISAN beneficiaries); however, data reported by the Ministry of Agriculture (refer section 6A of this report) is not based on robust survey methodology and the latest published agricultural census data available is for 2011 (the KISAN activity started in 2013). The project also worked with farmers in clusters around marketing channels, so the distribution of the KISAN beneficiaries is not randomly distributed across the districts in which other secondary data may have been collected in a more random sampling approach. Figure 1: KISAN Intervention Clusters 2. COMPARING DATA ACROSS ECOLOGICAL ZONES KISAN’s annual agricultural survey collects data form a sample of 740 farm households, of which 360 are in the Hill areas and 380 in the Terai. Data in National statistics for the Terai and Hill ecological zones includes these zones in the Eastern and Central regions, whereas the KISAN activity worked only in the Western, Mid-Western and Far-Western regions. 3. COMPARING DATA ACROSS REGIONS KISAN worked in three regions, but did not work in all districts in each of these regions. Neither did KISAN work in all VDCs in a district. Therefore, any data for the KISAN activity cannot be expected to be representative of the Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 40 USAID/Nepal Monitoring, Evaluation and Learning Project entire districts in which KISAN worked and will not be representative of other districts within the regions (Table 2). Table 2: Regions and number of districts in which KISAN implemented activities, compared to the total number of districts in these regions Whilst the project worked in all Hill and Terai districts in the Western and Mid-Western Regions, it did not work in the mountain districts, yet regional statistics include data for mountain districts which are likely to vary from lower altitudes particularly with respect to agricultural data. In the Western region, the project did not work in districts in all ecological zones. 4. PROBLEMS IN DISAGGREGATING DATA National survey data is frequently not disaggregated89 to districts, and certainly is not reported at a VDC level which may be comparable since KISAN activity activities were targeted to VDCs. Data for VDCs is likely to represent households with similar agro-ecological environments, whereas districtwide data may not be comparable to KISAN locations which are a sub-sample of district populations. 5. DATA COLLECTED FOR NON-COMPARABLE TIME PERIODS Data may be collected in National level surveys which are repeated only every five years, or less frequently. The Central Bureau of Statistics undertakes a large agricultural sector survey, every 10 years. The last survey was conducted in 2010/11 and the next survey will not be conducted until 2020, after the project ends. The Demographic Health Survey was conducted in 2011 and 2016, but the project started in 2013, two years after the dataset collected in 2011, so there is no guarantee that the 2011 data will remain representative of data for 2013. 6. COMPARING DATA BETWEEN DIRECT BENEFICIARIES AND NON-BENEFICIARIES KISAN collects monitoring data using four monitoring forms. These forms are for beneficiary farm households, farmer groups, non-farmer organizations supported by KISAN (e.g. agrovets, seed companies, etc.) and for training. These data are entered into the project’s WIKISAN database. These forms are primarily a register of all project beneficiaries, but some outcome data is also recorded. For example, the farmer level data includes monthly income and expenditure. The training form records all the KISAN supported training, by beneficiary of training, training provider and topic of the training. 89 The MOAD Statistical Information on Nepalese Agriculture (the so called “green book”) is not based on survey data. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 41 USAID/Nepal Monitoring, Evaluation and Learning Project KISAN also conducts an annual survey of beneficiaries. This survey includes both farmers and non-farmer beneficiary organizations. Data collection forms are used for farmers, change agents (local service providers), finance organizations, agrovets, seed companies, trading organizations, output processors, and cooperatives. For farmers, the data include crop gross margins, which provides the KISAN farm level data on crop production, crop area, yield per hectare, sales, and retentions for home consumption (the main food security measure in addition to income). The gross margin data also provides the value of cash inputs purchased by farmers, the data form also includes information on access to services and constraints faced by farmers. The forms for non-farmer beneficiaries includes data on services offered by businesses, technologies adopted as a result of KISAN, new capital investments, loans received, sales and market growth, and constraints faced by beneficiaries. For farmers, the KISAN dataset provides the best source of data for beneficiaries. For non-beneficiaries, the existing data source that may be most useful, in the absence of other data, is the 2014 Baseline Survey (as the base from which the non-beneficiary farmers started with respect to crop production, yield and sales). Although some of the farmers participating in the baseline survey may have become project beneficiaries, these data at least represent the base from which any improvements may be measured. Strictly speaking, however, this dataset may not represent the status of non-beneficiaries whose situation may have changes since 2013 (when the project started).90 For non-beneficiary farmers, the MOAD district statistics might be considered a benchmark for farmers not participating in KISAN because only a very small percentage of farmers within districts are project beneficiaries. However, the distribution of project beneficiaries is clustered around value chain routes, so it would be an heroic assumption to say that district averages for production type variables would be representative of the non￾beneficiaries in the same locations as KISAN farmers. The only credible comparison between beneficiary farmers and non-beneficiary farmers would be for comparisons to be made in the same villages as the KISAN beneficiaries, because there would be a high probability that farm size and farm type, soils, agro-climatic and market conditions would be similar for non-beneficiaries as for KISAN beneficiaries. To our knowledge, and at the time of writing this evaluation design, it is believed that there are no relevant data available for non-beneficiaries, without collecting new data. The project does not collect farmer group data. There are no records at the group level for the area of crops grown by group members, sales by members, market destination, or price for sales. The groups are primarily used for training the KISAN famer beneficiaries who receive training using the demonstration sites established by the lead farmers. 7. LACK OF DATA FOR NEW CAPITAL INVESTMENT IN AGRICULTURE KISAN reports data for new private sector investment in agriculture as a result of project activities, the baseline by definition is zero (as the measure is new investment, not the change to existing investment). With a baseline of zero and no measure of capital investment by beneficiaries at the start of the project, it is not possible to calculate any percentage change to capital investment levels for beneficiaries or the KISAN activity. The Ministry of Finance reports the status of credit flows of banks and financial institutions in the agricultural sector but do not appear to report capital investment levels for the sector91. The Nepal Rastra Bank (Central Bank) reports gross fixed capital formation, but this does not appear to be broken down for agricultural capital investments92. The value of agricultural loans may also be used as a surrogate for investment levels; however, loans may be used for working capital, living expenses, and debt repayment, as well as for capital investment. 90 The BL survey was conducted by KISAN in 2014. 91 Economic Survey Fiscal Year 2015/16, Government of Nepal, Ministry of Finance 2016. 92 Monetary Policy for 2016/17, Nepal Rastra Bank, July 2016. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 42 USAID/Nepal Monitoring, Evaluation and Learning Project ASSESSMENT OF PARTICULAR, SECONDARY DATA SOURCES AGRICULTURAL DATA A. The Ministry of Agricultural Development also produce agricultural statistics in the so-called “green book”: Statistical Information on Nepalese Agriculture. These data are published annually. Data is available for total crop production, yield per hectare and total area in production by various types of crops. Crops include those which KISAN supported being: Paddy rice, maize, lentils, cauliflower, cabbage, tomato, okra, cucumber, and bitter gourd. The district level data also includes the sales of fertilizer. However, this data does not include data for fertilizer sales by the private sector, so in this respect it is incomplete. The Central Bureau of Statistics advised the team that these data are compiled from the practical field knowledge of the DADO staff working in the districts; the data is not based on a robust statistical survey. Food security is primarily measured by the KISAN activity in terms of increased retention of crops on the farm for consumption by the farm household. This is a component of the availability of food. The project will not have impacted biometric measurements (which measure the utilization component of food security, such as stunting, wasting, underweight) because the benefits of the project have not have had sufficient time to be expressed by these indicators. More productive irrigated land for beneficiaries would contribute to the stability of food production, as would use of improved agricultural technologies. The team members have summarized the available data from the MOA and this data is presented in Appendix A. However, due to the limitations stated, it is not possible to draw any conclusions on the impact of the KISAN activity on the FTF ZOI using this data. B. The District Agriculture Development Office publish annually the Annual Agriculture Development Program and Achievement Book. This provides information on geographical and demographic status of the district, introduction to DADO office, the DADO’s program and annual progress. It also gives some information on districts agricultural statistics for the year such as the area of the district, production status of other sub sectors of agriculture, livestock service provided in the district, detail on some seed producers in district, rainfall status of the previous year etc. However, production data provided in this book does not match with published national statistics. Representatives from MOAD, suggest the production and yield data are not usable because the data is not collected in scientific manner. There is no VDC level data in this book. C. Data from Agriculture and Food Security Project (AFSP). AFSP works in 19 districts but only in the Hill ecological zone. 11 districts are the same as KISAN districts (seven in the Mid-Western Region and four in the Far￾Western region). The project has four components: Research on new technologies, technology dissemination (with livestock, agriculture and health offices of Government), nutrition (through behavior change and food preservation), and project management. The main crops targeted by the project are paddy, maize, wheat, and potatoes. In the 2016 Beneficiary Results Assessment Report (conducted in nine project districts) the following average crop yields are reported for rice and maize (the two crops targeted by both the KISAN activity and AFSP). Table 3: Average Yields of Rice and Maize Reported by the AFSP Project for Beneficiary and Non-Beneficiary Farmers (2013 Baseline, and 2015) 2013 baseline 2015 Crop Control Control Beneficiaries (Non-beneficiaries) Paddy Rice (MT/Ha) 2.2 1.6 2.8 Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 43 USAID/Nepal Monitoring, Evaluation and Learning Project Winter Maize (MT/HA) 1.7 1.8 2.33 Note: Yields of paddy were affected in 2015 by late monsoon rains which resulted in late transplanting of seedlings In 2015, KISAN average yields for paddy and maize in the Hill districts were 4.1 and 3.24 MT/Ha respectively93. FOOD SECURITY AND NUTRITION DATA D. The Feed the Future Nepal 2015 Zone of Influence Interim Assessment Report, July 2016 reports data for 2010/11, 2013 and 2014 based on various sources. This data corresponds to the start of the KISAN activity in 2013 and to the baseline survey which was conducted by KISAN in 2014. The data is reported for the FTF ZOI not by district, and the sample size used in the 2014 survey (880) would be insufficient for disaggregation by district, although disaggregation by agroecological zone (Terai and Hill areas) would likely be possible from the raw data. Any measurement of change in food security would require new data for 2017 and, as far as the evaluation team is aware, this data will not be available. The indicators in the FTF Interim Assessment Report are different to the indicators reported by KISAN. Further, the fact that KISAN beneficiaries are only 8% of total households would make any attribution of measured change to the KISAN activity very tenuous. E. The Multiple Indicator Cluster Survey 2014, primarily reports data on women and children in Nepal. The survey covers child mortality, nutrition, child health, water and sanitation, water quality, child development, literacy and education, child protection, knowledge of HIV/AIDs and reproductive health, etc. The indicators are different to those reported by KISAN and the same limitations for data interpretation and the attribution of causality that apply to the FTF ZOI study also apply to data from this survey. F. Nepal Food Security Monitoring System (NeKSAP) supported by UN (WFP) and EU reports food security monitoring information at the district level and for some VDCs. Line agencies enter data into the system every four months and the data reflects the contribution of projects. The Senior Agricultural Development Officer from DADO is the district focal point. NeKSAP reports on 17 indicators which measure food security across the dimensions of availability, access, utilization and stability. Three outcome level indicators (food consumption, livelihoods, and child malnutrition) are reported only infrequently. The other 14 indicators are reported every four months, albeit in a qualitative manner using subjective assessment or inference (Table 4). Table 4: NeKSAP Acute Food Security Phase Classification indicators Outcome Level Indicators (measured infrequently) 1. Food Consumption food and dietary diversity, food groups); 2. Livelihood Change (and Coping strategies); 3. Acute child malnutrition (6-59m), wasting as per the prevalence of Global Acute Malnutrition (GAM); Indicators measured every four months 4. Crop production situation; 5. Food stock at household level; 6. Stock of main staples in key markets (food availability in the market); 7. Employment opportunities within and neighboring districts; 8. Income through sales: NTFP, cash/high value crops, and small enterprises; 9. Income: meat, milk, egg, fish, honey 10. Market price of main staple including rice, wheat flour, and others. 11. Remittances 12. Human disease incidence, and epidemics 13. Water supply for drinking, sanitation (ODF) 14. Out-migration (stress induced) 15. Civil security (social violence, and bandh/blockade) 93 The evaluation team do not have disaggregated data for 2015 Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 44 USAID/Nepal Monitoring, Evaluation and Learning Project 16. Climatic hazards: floods, landslides, dry spell, snowfall, hailstorm, and strong wind 17. Disaster: Earthquake, fire Food security monitoring is reported to be conducted down to the VDC level in some instances; however, at this level, the reports are mostly qualitative and do not report against quantitative monitoring indicators. The usefulness of the NeKSAP monitoring might be that VDCs supported by the KISAN activity, and also monitored by NeKSAP, may show improved food security as measured by improved livelihoods perceptions, crop production, food stocks, and incomes. However, such data would be based on the perceptions of the District Food Security Networks (DFSNs)94 and not based on quantitative indicator measurement. Data limitations are that the outcome level indicators (which are quantitative) are based on irregular national level surveys (and frequently not disaggregated down to district level), and the local level data is based on perceptions or inference because it is not possible to collect quantitative data every four months. Food security monitoring is primarily designed to identify food insecure locations requiring assistance and uses five classifications for food insecurity at the local level - Minimally food insecure, Moderately (Stressed), Highly (Crisis), Severely (Emergency), Humanitarian Emergency (Declared famine). G. Nepal Demographic Health Survey (DHS). The DHS was conducted in 2011 and 2016. The key indicators report for 2016 has just been released. The purpose of the 2016 NDHS is to generate reliable information on fertility; family planning; adult and child mortality; maternal and child health; nutrition; women’s empowerment and domestic violence; knowledge of HIV and AIDS; and other health issues. Relevant nutrition indicators that may be compared are the percentage of Children less than five years with weight for age below two SD of Median, or who are malnourished95 and for women Body Mass Index96. It is doubtful whether the KISAN activitys could be expected to impact these indicators within the last few years of the project period (when activities really started to take off). DHS data is not disaggregated to the district level, so district level data cannot be compared to KISAN districts. Further, the Terai and Hill ecological zones reported by the DHS do not correspond to the same ecological zones reported by KISAN because the DHS ecological zones include all regions of Nepal within these zones (including Central and Eastern), whereas the KISAN zones only include the Western, Mid-Western, and Far-Western regions. Another problem is that the KISAN activity started in 2013 and the previous DHS data is for 2011, 2 years prior to the start of the KISAN activity. Appendix B does show the comparative data for the two nutrition indicators of weight for age for children under five years old and BMI for women 19-49 years of age, although not much inference can be taken from them regarding the performance of the KISAN activity. H. Data from Suaahara Project. The Suaahara project targets different beneficiaries to KISAN. Suaahara targets children under five years old and pregnant and lactating mothers, not farmers. Interventions are primarily to encourage primarily behavior change to improve nutrition outcomes. For agriculture, the project encourages disadvantaged groups to grow small plots of vegetables for consumption by the household, with some surplus for sale, rather than more commercial production on larger plots. The Suaahara project targets 16 of the same districts as KISAN in Phase 2, but this phase has only been going for 10 months – too early for any impact. In Phase 1 of the project, only four districts were the same as the KISAN districts. Suaahara indicators are not the same as the KISAN indicators. 94 District food security networks (DFSNs) operate in 74 of 75 districts. Chaired by the Chief District Officer (CDO), co-chaired by the Local Development Officer (LDO), and secretariat by the District Agricultural Development Officer (DADO). Also includes district-level line agencies, I/NGOs, civil society organizations, and the private sector. 95 Weight for age is a composite index of both the weight for height and height for age indicators. If below - 2 SD from the median, children are considered malnourished. 96 For women, the BMI seems to be the most appropriate measurement for comparison; if less than 18.5 then women are considered acutely undernourished. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 45 USAID/Nepal Monitoring, Evaluation and Learning Project The synergy of Suaahara with KISAN is that Suaahara linked their village model farmers with the KISAN local service providers and KISAN supported market centers. Appendix A Comparing KISAN Agricultural Data with National Statistics Comparing yield per hectare for KISAN beneficiaries with MOAD Government information97, shows that the yield for almost all commodities supported by the KISAN activity is nearly double across Hill districts. In the Terai, with the exception of lentils, crop yields are higher for KISAN beneficiaries when compared to the reported averages for districts as represented by the MOAD data (Figure A1). Figure A1 – Comparison of Yield per Ha for MOAD Data and KISAN Data for the 20 KISAN Intervention Districts 97 Statistical Information on Nepalese Agriculture, the so called “green book”, which is not based on robust survey methodology Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 46 USAID/Nepal Monitoring, Evaluation and Learning Project Note: MOAD data is not available for 2016, hence the 2015 MOAD data is presented with the 2015 data for the KISAN activity. Figure A2 shows the comparison of the average yield per hectare by crop type for KISAN beneficiaries compared to the MOAD data for 2015 (consistent years for both the KISAN data and the MOAD data). Except for lentils, the yield per hectare is significantly higher for KISAN beneficiaries than the reported district averages. Figure A2 – Average Yields by Crop Type Rice Maize Lentil (Pulses) Bitter Gourd Cabbage Cauliflower Cucumber Tomatoes Yield 2015 MOAD 3.13 2.43 0.96 12.3 14.3 12.5 14.7 12.8 KISAN 2015 4.51 3.24 0.91 18.68 26.44 18.49 23.06 30.22 0.00 5.00 10.00 15.00 20.00 25.00 30.00 35.00 MT/Ha Avg. Yield (MT/Ha) by Crop Type: MOAD Data Compared with KISAN Benficiaries acrosss all 20 Project Districts (2015) Yield 2015 MOAD KISAN 2015 Table A1 shows the ranked yields for rice (paddy) and maize across the 20 districts supported by the KISAN activity, using the MOAD reported averages. The KISAN average for beneficiaries is also shown. The KISAN yields outperform the district averages with the exception of maize yield in Bardiya (refer the note for this data point). Table A1 – Ranked Yield Data for Paddy and Maize for KISAN Districts shown with the Average Yield per Hectare for KISAN Beneficiaries MOAD Data for Yield per Ha: Rice and Maize by District 2015 Paddy Maize RANK District MT/Ha District MT/Ha Lowest Achham 2.11 Kailai 1.04 Doti 2.11 jajarkot 1.55 Baitadi 2.24 Baitadi 1.62 Rolpa 2.60 Pyuthan 1.75 Jajarkot 2.68 Doti 1.90 Rukum 2.75 Kanchanpur 1.90 Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 47 USAID/Nepal Monitoring, Evaluation and Learning Project Gulmi 2.78 Dailekh 1.95 Kailai 2.98 Dang 1.99 Kapilbastu 3.00 Rukum 2.00 Arghakhanchi 3.10 Palpa 2.10 Dailekh 3.15 Banke 2.15 Banke 3.15 Gulmi 2.20 Kanchanpur 3.15 Salyan 2.24 Pyuthan 3.36 Rolpa 2.30 Salyan 3.55 Surkhet 2.60 Dadeldhura 3.64 Achham 2.78 Surkhet 3.80 Dadeldhura 2.91 Palpa 3.84 Arghakhanchi 3.00 Bardiya 4.23 Kapilbastu 3.05 >> Highest Dang 4.30 > Avg. KISAN 3.24 >>> Avg. KISAN 4.51 Bardiya 7.50 Note: For Bardiya 7.5 MT/Ha for maize seems an incorrect figure, but this is the data reported in the "Green Book" Chemical Fertilizer Application: The MOAD district data includes data on sales of fertilizer by type of fertilizer. However, this data does not capture sales by the private sector so cannot be used to compare the use of fertilizer by KISAN beneficiaries with district wide data. KISAN beneficiaries were connected to private sector input suppliers to increase the availability and use of inputs. Total Area in Crops and Total Production: It is not possible to compare change in area planted in crops and total production across KISAN districts using MOAD data and data for KISAN beneficiaries because the baseline survey does not report the area in KISAN supported crops or the total production by crop type. Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 48 USAID/Nepal Monitoring, Evaluation and Learning Project Appendix B Data on Nutritional Status Table B1 - Indicator: Children less than five years malnourished Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 49 USAID/Nepal Monitoring, Evaluation and Learning Project Table B2 - Indicator: BMI for women 15 to 49 years of age Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 50 USAID/Nepal Monitoring, Evaluation and Learning Project Figure B1 – Selected indicators for nutrition Appendix C Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 51 USAID/Nepal Monitoring, Evaluation and Learning Project Data on New Private Sector Investment in Agriculture There appears to be no other comparable data for this indicator. The KISAN baseline is zero and the project reports an increase of $1.866 million by 2016 for the project beneficiaries; 41% of this amount was investment by beneficiaries who were also grantees under the Business Opportunities Program (the grantee share). Because the baseline is zero, it is not possible to calculate the percentage change from the baseline for this indicator. The Nepal Rastra Bank reports agricultural credit increased from 50909 million rupees in 2013/14 to 70999 million in 2015/16, and increase of 40%. By contrast, the KISAN activity reports that the value of rural loans for project beneficiaries increased from a baseline (2014) of $667,615 to $7.448 million (2016 actual), an eleven-fold increase over the baseline value. Table C1: KISAN data on the value of rural loans from the year 4 annual report Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 52 USAID/Nepal Monitoring, Evaluation and Learning Project APPENDIX D: DETAILED TABLES OF FGDS AND KII COMPLETED BY DISTRICT Summary of FGDs and KIIs FGDs by District and Classification of Commercialization District A B C D TOTAL Banke 2 1 3 Dadeldhura 3 2 5 Dang 1 2 3 Kailali 2 2 1 5 Kapilvastu 1 3 1 5 Palpa 2 2 1 5 Salyan 2 1 3 Surkhet 1 2 3 9 13 7 3 32 28% 41% 22% 9% 100% FGDs by Irrigation and Disadvantaged Category Irrigation Disadvantaged Yes No Yes No Banke 2 1 2 1 Dadeldhura 1 4 1 4 Dang 3 0 3 0 Kailali 1 4 4 1 Kapilvastu 2 3 3 2 Palpa 1 4 2 3 Salyan 2 1 2 1 Surkhet 1 2 2 1 13 19 19 13 41% 59% 59% 41% Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 53 USAID/Nepal Monitoring, Evaluation and Learning Project Number of Male and Female Participants in FGDs Male Female Totals Banke 6 26 32 Dadeldhura 17 38 55 Dang 8 28 36 Kailali 1 56 57 Kapilvastu 15 48 63 Palpa 22 25 47 Salyan 19 21 40 Surkhet 1 32 33 89 274 363 25% 75% Number of Disadvantaged Individuals Participating in FGDs Madhes Non￾Dalits Janajati i Disadvantaged Banke 23 0 9 Dadeldhura 15 0 40 Dang 0 34 2 Kailali 2 0 55 Kapilvastu 2 27 13 21 Palpa 0 22 25 Salyan 1 17 22 Surkhet 1 10 22 44 110 13 196 Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 54 USAID/Nepal Monitoring, Evaluation and Learning Project Final Evaluation of Knowledge-based Integrated Sustainable Agriculture and Nutrition 55 Number of Key Informants by Gender and Type of Informant Sum of Row Labels Sum of No_Men No_Women Agribusiness 7 1 Agrovet 11 0 Collaborating 13 1 Cooperative 5 0 GON 7 1 KISAN staff 11 9 Lead Farmer 14 5 MFI 4 5 Miller 5 0 MPCC 5 4 SACCO 9 14 Seed Company 6 0 Trader 7 1 Grand Total 104 41 56 Type of Grantee by Business Business Number Agribusiness Agrovet Cooperative SACCO Seed Company 5 5 1 1 4 16 57 APPENDIX E: GROSS MARGIN REPORT Table 1 below shows the gross margin reported by KISAN in its 4th annual report. It was based on KISAN field survey data across 740 households collected in the year 2016. Table 1. Gross Margins reported by KISAN 2016 – $or MT per Hectare equivalent Rice Maize Lentils Bitter Gourd Cabbage Cauliflower Cucumber Tomatoes Gross Margin USD/Ha 574 611 476 5,404 5,075 4,611 6,230 9,692 Yield MT/Ha 4.24 3.46 0.55 17.06 25.86 16.74 22.99 31.15 Price USD/M T 184 213 987 343 209 296 287 329 Source: KISAN 4th Annual Report for 2016 for gross margins and personal communication with KISAN staff for the average yields and prices for the reported gross margins Although the evaluation team is assigned to conduct the qualitative survey, as requested by mission, we have made interviewed some of the lead farmers on gross margin data to validate the gross margins reported by KISAN. Most of the farmers’ interviewed by the evaluation team were some of the best farmers among beneficiaries. In total 37 farmers were interviewed from eight different districts for the vegetable crops they produced in the year 2016. No data were collected for cereal crops. Table 2 shows number of sample collected from each districts and type of vegetables sampled. Out of total 37 samples, largest sample was for tomatoes, followed by cauliflower, cucumber and cabbage respectively. There were just two samples for bitter gourd. For bitter gourd, the sample might be too small to be meaningful for comparison purposes. Table 2. Number of Collected sample by Districts and type of Vegetables Name of the Districts Bitter Gourd Cabbage Cauliflower Cucumber Tomatoes Banke 2 2 2 Dadeldhura 1 1 2 Dang 1 1 1 1 1 Kailali 1 1 2 Kapilvastu 1 2 2 Palpa 1 Salyan 1 2 2 2 Surkhet 1 1 1 1 1 Sub total 2 7 10 7 11 58 Tables 3 and 4 show yield and price of different vegetable type. Comparing the surveyed data with KISAN’s data, shows there is not much difference in the yields except for the yield of tomato. Evaluation team found the yield of tomato is nearly 65% higher than KISAN’s yield. This was because one or two of the farmers sample were making very high yield from tomato production (60MT/ha for outdoor tomatoes and 250MT/Ha98 for production in tunnel house). While comparing prices, KISAN’s prices are higher than the evaluation team’s data. This might be due to decrease in price of vegetable crops in the years 2017. This price difference has certainly impacted on the gross margin result. Table 3. Crop Yields by Crop Type MT/Ha Mean Min Max Median KISAN Yield Bitter Gourd 21.6 19.7 23.5 21.6 17.06 Cabbage 36.8 9.8 59.1 37.4 25.86 Cauliflower 14.3 3.4 31.8 13.3 16.74 Cucumber 28.4 14.1 49.1 26.6 22.99 Tomato 51.3 15.2 250.0 29.5 31.15 Table 4. Price for Sales by Crop Type USD/MT Mean Min Max Median KISAN Price Bitter Gourd 599.7 599.5 600.0 599.7 343 Cabbage 181.4 60.0 300.0 180.0 209 Cauliflower 301.0 70.0 702.5 250.0 296 Cucumber 226.8 150.0 350.0 200.0 287 Tomato 252.5 83.3 350.0 280.0 329 In the table 5 the team has calculated gross margin following the same method of USAID and compared with KISAN’s data. Table 5. Gross Margin Comparison Evaluation Team and KISAN's data Crop KISAN Evaluation Differences Difference Reported teams GM between GM Evaluation Team data and KISAN Data Bitter 5,404 11,129 - -106% KISAN GM is lower Gourd 5,725 Cabbage 5,075 3,305 1,770 35% KISAN GM is higher Cauliflowe 4,611 2,148 2,463 53% KISAN GM is r higher 98 250 MT/Ha equivalent is possible for tunnel houses, but none of the beneficiaries will have this area under plastic. KISAN do not report the number of gross margin records in the data set under tunnel houses. 59 Cucumber 6,230 4,549 1,681 27% KISAN GM is higher Tomato 9,692 5,996 3,696 38% KISAN GM is higher KISAN present gross margin data to demonstrate net income from growing the crops the project supported, including for 5 vegetable crops. Though KISAN’s gross margin is higher for all the crops except Bitter gourd (only two in our sample), compared to the evaluation team data, the gross margins can be validated as reasonable because the price of each product has decreased in the year 2017 compared to 2016 (the KISAN data year). The lower gross margins calculated by the team are primarily due to lower prices; this is consistent with farmers complaining about falling prices. And there is not so much of difference in final gross margin. There are also some additional observations on how the KISAN gross margins are calculated and presented (in line with USAID practice): a. Presenting gross margin per hectare provides a misleading representation of benefits for project beneficiaries because farmers grow very small areas in vegetables, far less than one hectare. For example, 0.5 to 1.0 Kattha (339 sq. m) is a typical area of vegetables for project beneficiaries. This means the actual gross margin for beneficiaries is likely to be 2 to 4% of the per hectare figures. 14. b. Gross margin data for farmers is skewed to the right, with relatively few farmers earning high incomes and most farmers earning low incomes (the distribution is non-normal). The skew is more pronounced for data per farmer than per hectare because there is a large variability in the areas of crops being grown by farmers. Expressing the data per hectare tends to normalize the data. This means that the average value for gross margin per farmer is significantly higher than the median, with the median value being a better reflection of what are typical net incomes for vegetable enterprises for small scale vegetable production. KISAN data sums the gross margin data across crop areas rather than calculating averages per farmer and then calculating the distribution of gross margin across farmers. This means that the data is weighted by crop area rather than each farmer contributing equally to the reported average gross margin per hectare. Due to the way that KISAN data is calculated, there is no distribution reported for gross margin and no median calculated. For example, for tomatoes the KISAN gross margin is reported as $9,692 per hectare calculated with the data weighted in proportion to the area grown by each farmer; whereas when calculated across farmers (with each farmer contributing equally to the weighting) the gross margin per farmer has a mean of $242 and a median value of $152 per farmer growing tomatoes (Figure 1). This is true for all other four vegetable crops. Table 6 below shows KISAN’s mean and median of gross margin per farmers calculated by evaluation team. Table 6. Mean and Median of Gross margin calculated per farmer (KISAN’s data) Bitter Gourd Cabbag e Cauliflower Cucumber Tomato Mean USD 98 128 153 150 242 60 Median USD 75 101 103 79 152 The mean was $150 for cucumber, 153 for cauliflower, $128 for cabbage and 98 for bitter gourd whereas median for the KISAN data based on their dataset are $152 for tomatoes, $79 for cucumber, $103 for cauliflower, $101 for cabbage and $75 for Bitter gourd. The following figures show actual distribution of gross margin for vegetables Figure 1. Distribution of Gross Margin per Farmer for Tomatoes Figure 2. Distribution of Gross Margin per Farmer for Cucumber 61 Figure 3. Distribution of Gross Margin per Farmer for Cauliflower Figure 4. Distribution of Gross Margin per Farmer for Cabbage Figure 5. Distribution of Gross Margin per Farmer for Bitter Gourd Also of note, is that the method used method to calculate gross margins only considers cash costs and incomes, and not the value of family labour and return per labour day. Return per labour day may be more important for farmers than net cash return per unit area. On-farm capital costs for the enterprise are also not included. For example, if a farmer owns own cultivation equipment, instead of hiring equipment, these costs are not captured in the gross margin. Similarly, the capital costs of tunnel houses are not captured because tunnel houses are a capital costs where the costs would be depreciated over the assets life. 62 Table 7. Example of GM with family labour included 15. 63 APPENDIX F: EVALUATION INSTRUMENTS Final after Field Testing Focus Group Discussion for KISAN Farmer Group Members Group Information 1 Unique FGD ID Number ___ ___ ___ ___ To be filled by evaluation team 2 Interviewer / Facilitator Name(s) 3 Notes on this form taken by (name) 4 Date of FGD Day: ___ ___ Month: ___ ___ Year: 2017 5 Location: Name of Region 6 Location: Name of District 7 Location: VDC 8 Location: Group Name Introductions ● Facilitator introduces her/himself and the note taker: ● Working for CAMRIS International, conducting the evaluation on behalf of USAID - ● Welcome the participants and thanks them for taking part in the discussion, reaffirm how important their views are. ● Inform participants that information collected from individuals will not be attributed to them, and their names will not be used. ● Describe purpose of evaluation: To learn lessons from the KISAN activity that may make the new projects more effective - ● Time for discussion about 1 to 1.5 Hours ● No right or wrong answers. Everyone’s views are important. Please feel free to ask questions. 64 Participant List # Participant Name Gender Male Femal e Ethnicity Signature 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Total 65 Farmer Group FGD Questions (KISAN) A. Group Information 1. When did this group start working with KISAN? 2. How many members are in this group? Total: ____ Men: ____ Women: ____ 3. Are there elected leaders? If Yes: Do women have equal opportunity to be elected as leaders? If not why? Elected leaders: Name Position Gender M / F B. Working with KISAN 4. What were the main activities you participated in under KISAN? For training activities, list topics of the training… 5. What were the most effective KISAN activities, and why? Check: Did this include the Business Literacy program? 6. Are there any activities that should not be repeated, and why? 7. Were there any activities that were needed, which were not provided? 8. Have there been any negative consequences of working with KISAN? C. Marketing Produce 9. What were the main crops you sold, when working with KISAN? 10. Are there any that you will cease to grow after KISAN? And why? 11. Are there any crops, not supported by KISAN, which you plan to grow in future, and why? 66 12. By how much has your income increased from selling crops supported through the KISAN activity? 13. How has the selling and marketing of crops changed as a result of working with KISAN? 14. Do you think these selling arrangements will continue in future? If not, why not? D. Provision of Services 15. Do you have access to agricultural technical advice (e.g. farm extension services)? 16. How would you say access to technical services has changed, whist working with KISAN? e.g. Number of visits, providers of services, quality of service 17. Has the services provided by MOA/DADO changed in any way whilst working with KISAN? 18 Do you pay for agricultural technical services? If so, who charges for services? 19. After the project ends, do you think these services will continue, if not why not? 20. How do you communicate with farm input suppliers and traders? And has this changed since you started to work with KISAN? (e.g. mobile phone, mobile app, meet in person, etc.) E. Access to farm Inputs 21. Do you use more farm inputs because of working with KISAN? - If yes, which ones? 22. Are there any farm inputs which you are still not able to purchase? 23. Are there currently any problems with the quality of farm inputs? 24. Do you now have the technical skills to use farm inputs effectively (fertiliser, agricultural chemicals, etc.)? F. Labour / Mechanisation 25. Are there any periods you are short of labour for farming activities? If yes, what periods and what activities? 26. Is there any farm equipment that you now use that you did not use before working with KISAN? If yes, what type of equipment and purpose? 27. Have you used any new post-harvest technologies as a result of working with KISAN? If yes, how have these changed things? 67 28. As a result KISAN, has the opportunity for agricultural work replaced the need to migrate to seek work away from the farm? G. Having enough food and access to food 29. Has your access to food and the amount of food you have to eat changed as a result of working with KISAN? 30. If yes, can you describe how this has changed? e.g. More food to hold on the farm, more types of food, better understanding of nutrition, etc. H. Microfinance 31. What credit activities has KISAN supported? 32. How has your access to credit improved as a result of working with KISAN? 33. What was the most useful credit related activities supported by KISAN, and why? I. Future Plans 34. What support does this group now need? Or can you now manage independently? Final question: Is there anything else you would like to tell us about your involvement with KISAN that we didn’t ask you about? Closing Remarks Thank everyone for participating, their opinions have been useful and will contribute to making improving future USAID projects. 68 Women Only Focus Group Discussion for KISAN Members Group Information 1 Unique FGD ID Number ___ ___ ___ ___ To be filled by evaluation team 2 Interviewer / Facilitator Name(s) 3 Notes on this form taken by (name) 4 Date of FGD Day: ___ ___ Month: ___ ___ Year: 2017 5 Location: Name of Region 6 Location: Name of District 7 Location: VDC 8 Location: Group Name Introductions ● Facilitator introduces her/himself and the note taker: ● Working for CAMRIS International, conducting the evaluation on behalf of USAID - ● Welcome the participants and thanks them for taking part in the discussion, reaffirm how important their views are. ● Inform participants that information collected from individuals will not be attributed to them, and their names will not be used. ● Describe purpose of evaluation: To learn lessons from the KISAN activity that may make the new projects more effective - ● Time for discussion about 1 to 1.5 Hours ● No right or wrong answers. Everyone’s views are important. Please feel free to ask questions. 69 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Women Participant List # Participant Name Gender Female Ethnicity Signature Total 70 Questions for Women Farmer Groups Women’s, Dalit and Janajati Inclusion: 1. Has KISAN improved the lives of women, Dalit, and Janajati? How? If not, why? 2. Are you members of other groups other than KISAN? If yes, how is the KISAN group different from other groups you may belong to. 3. In your farmer group do women, Dalit and Janajati members speak up? Are they listened to? Are their opinions considered in making group decisions? 4. What are the key barriers that women, Dalit and Janajati face in fully participating in your group? Are they able to get assistance from KISAN in production and marketing? 5. Are women, Dalit and Janajati elected to leadership positions within this KISAN group? If yes, what are their positions? 6. Are there any women’s only activities within the KISAN groups? If yes, what are they? 7. Are the interests of women and considered and served by the group? If so, how is this done? Bank Accounts and Credit: 8. Who has a bank account in your house? 9. If you have a loan(s), how did you access the loan? From where did you get the loan? What was the interest rate? 10. Do women face any barriers for accessing credit that are not face by men? Training: 11. Did women and marginalized community receive the training for the activities they participate in? If not, for what skills did they not receive the training? 12. Did you face any problems or obstacles in participating KISAN’s training or learning? If yes, what were they? Were you listened to in resolving these obstacles? Workload and Responsibilities: 13. Did your workload as women increase with the project’s activities? If yes, is this a problem? 14. Have the KISAN activities resulted in any changes in responsibilities between you and your husband(s) either on the farm or in the household? If yes, what are the changes? 15. Has KISAN activity activities increased your household income, if yes how? 16. Have changes affected family relationships? If yes, how? 71 17. As a result of the project, has the opportunity for agricultural work replaced the need for members of your household to migrate to seek work away from the farm? Decisions about Money/Income: 18. Do you receive a fair share of the income from the vegetable and crop production? If not, why not? 19. Do women face any problems marketing vegetables and negotiating market prices, compared to men? 20. Who keeps the money, and who makes decisions about the money you earn from vegetable or crop growing? 21. Are women’s priorities considered equally with the men, and other HH members, when making decisions about the use of money? 22. On what did you spend the extra money which you received from vegetable or crop growing? List… Of these, which is the highest priority? Other: 23. Are you aware of any needs of women and disadvantaged groups that should have been addressed by the KISAN activity but were omitted? 24. Would you share if there was any evidence (experienced, manifested, or heard) of discrimination/exclusion of poorest and marginalized in services from value chain actors? This question is for both victims and observers of discrimination. 72 Key Informant Interview: Input Suppliers (Agrovets, Seed Company, Other Input Supplier e.g. Mechanisation providers) 1 Informant Name(s): Phone number: 2 Company / Institution Name: 2 Interviewer Name(s) 4 Date of Interview Day: ___ ___ Month: ___ ___ Year: 2017 5 Name of Region 6 Name of District 7 VDC or Municipality 1. Do you provide farm inputs to farmers; if yes what are they? 2. Do you get these inputs locally, or are they imported by your company? 3. Are there any problems with quality and availability of inputs? 4. Do you have a permit for importing seeds, or other agricultural inputs? 5. What kind of assistance did you receive from the KISAN activity? 6. What assistance was provided? E.g.: ● Training (Type and amount or times) ● Materials (seed, fertilizer, ag chemicals etc.) ● Information on source of supplies, markets, or marketing, etc. 7. Did this support help you to provide better service to famers? If yes, how? 8. As a result of KISAN’s support, are you selling more inputs? Serving more farmers? Enlarging the geographic areas that you serve? 9. To how many groups of KISAN farmers do you sell farm inputs? 73 10. What percentage of your total market would the KISAN groups represent? 11. Do you provide trade credit to farmers? If so please explain the type of credit, purpose, and repayment terms? 12. If yes, has this credit to farmers increased with KISAN support? 13. What was the most effective support that KISAN provided, and why? 14. Do your clients include socially disadvantaged groups (such as Dalit, Janajati, or others) women, or women-headed households? 15. As a result of KISAN, have you accessed new agricultural products, or new technologies? 16. Do you provide extension services to your farmer customers/clients? - If yes, what are these services? - Do farmers pay for these services? 17. Were you providing these extensions services before KISAN supported you? 18. How did KISAN activity help improve these services? 19. How many staff do you have providing extension service to farmers? 20. Were there any issues or problems working with KISAN? 21. What are the biggest challenges working with farmers? 22. Do you believe your business practices promoted by KISAN will continue after KISAN ends? 74 Key Informant Interview with Grantees of Business Opportunities Program 1 Informant Name(s): Phone number: 2 Company / Institution Name: 2 Interviewer Name(s) 4 Date of Interview Day: ___ ___ Month: ___ ___ Year: 2017 5 Name of Region 6 Name of District 7 VDC or Municipality 1. Please describe your business activity (e.g. input supply, trader, produce buyer, farm mechanization, processor, etc.) 2. Approximately what would be your gross annual turnover? 3. How many farmers do you serve? 4. What project did the grant from KISAN support? 5. How much was the KISAN grant? 6. What percentage of the total project investment was this? 7. What other assistance did you receive from KISAN, other than the grant funds? E.g. ● Technical training ● Materials ● Information on source of supplies, markets, or marketing, etc. ● Business planning 8. Since receiving assistance from KISAN, how has your business improved? 9. Can you provide examples of by how much your business has improved and in what products / services? 75 10. Do you provide trade credit to farmers? If so please explain the type of credit, purpose, and repayment terms? 11. If yes, has this credit to farmers increased with KISAN support? 12. What was the most effective support that KISAN provided, and why? 13. As a result of KISAN, have you accessed new agricultural products, or new technologies? 14. Do you provide extension services to your farmer customers/clients? - If yes, what are these services? - Do farmers pay for these services? 15. Were you providing these extensions services before KISAN supported you? 16. How did KISAN activity help improve these services? 17. How many staff do you have providing extension service to farmers? 18. Were there any issues or problems working with KISAN? 19. What are the biggest challenges you now face? 20. Do you believe your business practices promoted by KISAN will continue after KISAN ends? 76 Key Informant Interviews with Market Actors (MPCs, Traders, Collection Center, Millers) 1 Informant Name(s): Phone number: 2 Company / Institution Name: 2 Interviewer Name(s) 4 Date of Interview Day: ___ ___ Month: ___ ___ Year: 2017 5 Name of Region 6 Name of District 7 VDC or Municipality 1. What support have you received whilst working with KISAN? 2. What was the most effective support, and were there any issues or problems? 3. How many groups do you work with now, which ones are they, and where? 4. Do you collect produce from the farmer groups? If yes, what produce and how often? 5. What type of records do you keep regarding crop purchases and sales? 6. What are the main challenges when working with farmers? Do NOT Prompt: e.g.: such as crop quality, complying with contract agreements, timing of supply, repayments by farmers, etc. 7. Do you provide extension/advisory services to farmers? If yes, what are these services? 8. How many field staff do you have working with farmers? 77 Skip next if no field staff: 9. What technical support are you able to provide to your field staff? 10. What access do you have to technical information and resources? 11. Do you provide trade credit to farmers? If so please explain the type of credit, purpose, and repayment terms? … If yes, has this credit to farmers increased with KISAN support? 12. Are there any crops where there is now over production? … If yes, which ones? 13. Are there opportunities for new crops, not now grown? … If yes, which ones? 14. What effect has the KISAN activity had on prices received by farmers? 15. What are the main issues for crop quality? Do NOT prompt: e.g. farmer practices, wastage, or non-sales. 16. What are the current local and export market challenges? 17. How confident are you about your current business, and what plans do you have for the future? 18. Do you believe what KISAN has achieved will be able to be continued after the project ends? 78 KII (FGD)with GON Staff Working with KISAN activity 1 Informant Name(s): Phone number: 2 Company / Institution Name: 2 Interviewer Name(s) 4 Date of Interview Day: ___ ___ Month: ___ ___ Year: 2017 5 Name of Region 6 Name of District 7 VDC or Municipality 1. What activities did you participate in with the KISAN activity? And when? 2. Did the KISAN activity provide you with any training? - If yes, what was it? - Was it relevant? - Was it effective? - How could it have been improved? 3. What support did KISAN provide to farmers and farmer groups? 4. What technical training did you yourself provide to farmers, as part of KISAN activities? 5. Did women receive the same training as men? 6. Were there any special KISAN strategies to encourage equal participation by women in KISAN activities? 7. What were the main problems and issues when working with farmers? 8. Were the KISAN approaches the right ones? 9. In your opinion, are the groups now sustainable on their own? 10. Which crops were most successful, and why? 11. Were there any issues related to the availability of credit for farmers? 79 12. Do farmers accept commercial farming arrangements; for example, do they abide by quality and contract arrangements made with buyers? 13 What are the main technical problems still facing farmers, such as pests and diseases, water, labour shortages, mechanization, etc.? 14. Do farmers know about calculating profitability for their enterprises, or business? 15. If there were to be a new project, what would you suggest anything should be done differently? 16. In your perception, has the KISAN activity improved the food security situation for farm households? If yes, in what way? 17. In your perception, has the KISAN activity improved the situation for disadvantaged groups, and youth? If yes, in what way? 18. Are there any KISAN activities that the GON plans to continue after the project ends? If yes, what are they and why? 80 KII Micro-Finance Institutions 1 Informant Name(s): Phone number: 2 Company / Institution Name: 2 Interviewer Name(s) 4 Date of Interview Day: ___ ___ Month: ___ ___ Year: 2017 5 Name of Region 6 Name of District 7 VDC or Municipality 1. How many farmer savings and loans groups does this institution support? 2. How many farmer members would this represent? 3. What support have you had from the KISAN activity? - Training - Capital injection - Links to banks or other credit institutions - Other? 4. How could this have been improved, were there any problems? 5. What type of loans do you provide to farmers? Loan Type Maximum Value Term Interest Rate No. of Loans Collateral Required (Y/N) 6. What is the total amount of capital that this institution has invested? a. Loans to borrowers? b. Investments? e.g. Deposits 81 7. Do you have any products designed for women, disadvantages people, or youth? If yes, please describe. 8. What percentage of clients are in arrears? 9. What is the value of bad debts? 10. Do you have clients who are not farmers? - If yes, what percentage of clients would this represent? - And what value of the total loan portfolio? 11. What services do you now provide to group members, other than credit? (e.g. business planning, client training, insurance, etc.) - How could these services be improved? 12. Do you provide any training to you own staff, or savings and loan group leaders? If yes, what type of training? 13. How is your management committee selected? How many are women? 14. Do you employ any professional staff, such as accountants or administrators? 15. What are the main challenges in managing institutional groups and members? 16. When the KISAN activity ends, will you be able to continue providing the current services you offer to clients? 17. What other support is available to you after KISAN ends? 18. In your opinion, what are the main challenges still facing your farmer clients? 82 Questions for Lead Farmers (KISAN) 1 Informant Name(s): Phone number: 2 Affiliated Group Name: 2 Interviewer Name(s) 4 Date of Interview Day: ___ ___ Month: ___ ___ Year: 2017 5 Name of Region 6 Name of District 7 VDC or Municipality A. Working with KISAN 1. What were the main activities you participated in under KISAN? For training activities, list topics of the training… 2. Do you keep the farmer log book advocated by KISAN? If not why not? 3. What were the most effective KISAN activities, and why? Check: Did this include the Business Literacy program? 4. Are there any activities that should not be repeated, and why? 5. Were there any activities that were needed, which were not provided? 6. Have there been any negative consequences of working with KISAN? B. Marketing Produce 7. What were the main crops you produced, when working with KISAN? 8. Are there any that you will cease to grow after KISAN? And why? 9. Are there any crops, not supported by KISAN, which you plan to grow in future, and why? 10. By how much has your income increased from selling crops supported through the KISAN activity? 11. How has the selling and marketing of crops changed as a result of working with KISAN? 83 12. Do you have any other suggestions for improving marketing system in your village? C. Provision of Services 13. Do you have access to agricultural technical advice (e.g. farm extension services)? 14. How would you say access to technical services has changed, whist working with KISAN? e.g. Number of visits, providers of services, quality of service 15. Has the services provided by MOA/DADO changed in any way whilst working with KISAN? 16 Do you pay for agricultural technical services? If so, who charges for services? 17. After the project ends, do you think these services will continue, if not why not? 18. How do you communicate with farm input suppliers and traders? And has this changed since you started to work with KISAN? (e.g. mobile phone, mobile app, meet in person, etc.) D. Access to farm Inputs 19. Do you use more farm inputs because of working with KISAN? - If yes, which ones? 20. Are there any farm inputs which you are still not able to purchase? 21. Are there currently any problems with the quality of farm inputs? 22. Do you now have the technical skills to use farm inputs effectively (fertiliser, agricultural chemicals, etc.)? E. Labour / Mechanisation 23. Are there any periods you are short of labour for farming activities? If yes, what periods and what activities? 24. What recommendations do you have for mechanization to solve the problem of labour for production, harvesting and post harvesting? 25. As a result KISAN, has the opportunity for agricultural work replaced the need to migrate to seek work away from the farm? F. Microfinance 26. What credit activities has KISAN supported? 27. How has your access to credit improved as a result of working with KISAN? 28. What was the most useful credit related activities supported by KISAN, and why? G. Future Plans 29. What are the main challenges you face as a farmer? 84 30. Do you have any suggestions to KISAN for future? Final question: Is there anything else you would like to tell us about your involvement with KISAN that we didn’t ask you about? 85 ▪ Does farmer use own machinery for cultivation, or other cropping activities, that is not shown in the above expenditure? If yes, please list the equipment used. (e.g. tractor, hand tractor, mechanized weeding, harvesting equipment, etc.) 86 APPENDIX G: LIMITATIONS LIMITATIONS (FROM THE EVALUATION DESIGN) Time: The project formed and supports over 4,000 farmer groups. Time and resource limitations for the evaluation will mean that focus groups discussions can be held with around 40 beneficiary groups which is only 1% of all beneficiary groups. Statistical Data: The Agricultural Development Strategy 2015-3035 refers to 2010/11 agricultural statistics. The FTF interim survey was conducted during 2015 and combined this data with data from 2014. The team has started to investigate what agricultural statistical data is available in districts and the usefulness of this data may be quite limited. Data from national level surveys frequently cannot be disaggregated by district (or VDC) and is not reported by districts (mainly because of insufficient sample size). Most beneficiaries do not keep written farm records, so they will only be able to give their best estimates of historical crop sales and incomes (based on recall). Because of this, data from the individual interviews with beneficiaries will only approximate their true values. However, this information will provide another source of data which can be triangulated against project data. Secondary Data: National surveys, if relevant, may not report data by districts and almost certainly will not be able to be disaggregated to the famer beneficiary group or village level. This means that even if national level survey data is available, it will be unlikely this data can be disaggregated to make it comparable to the data from the beneficiary farmer groups. Topography: Nepal’s topography will be challenging to navigate, particularly in Hill districts. Breakdowns are likely. Sufficient time will be required for travel to more remote locations. Difficulty arranging meetings at GON office: In the first weeks of the assignment the team have attempted to meet GON staff in the Ministry compound. This has proved to be quite difficult involving several attempts at arranging meetings with the correct people and some non-productive visits to GON offices. The team will use the Dedoose software to analyze the qualitative response from FGDs and KIIs. However, the numbers of groups and informants will be limited (40 or so groups, around 81 KII but with around 16 to 24 informants classified as input supply companies, market actors, finance institutions, etc.). These relatively small numbers of informants, by classification, will limit the quantitative possibilities for categorizing qualitative responses. 87 APPENDIX H: STATEMENT OF WORK FINAL PERFORMANCE EVALUATION OF USAID/NEPAL’S KNOWLEDGE-BASED INTEGRATED SUSTAINABLE AGRICULTURE AND NUTRITION (KISAN) PROJECT Draft STATEMENT OF WORK 1) BACKGROUND With a population of 27 million, Nepal is a severely food deficit country struggling to recover from a 10-year civil war. With a GDP per capita of $524 (estimated FY 2010), Nepal remains the poorest country in South Asia and the 13th poorest country in the world behind Rwanda. About 55% of Nepalese live below the international poverty line of $1.25/day. Nearly 80% of the population works in the agriculture sector, accounting for 38% of the GDP. Recent declining agricultural production has depressed rural economies and increased widespread hunger and urban migration in Nepal. This situation is compounded by a population growth rate of over 2% per year and one of the highest ratios of population to arable land in the world. In 2009, 43 of 75 districts were reported to be food deficient, with 23 districts in Nepal chronically food insecure. Two out of three Nepalese suffer food insecurity at some point during the year. Based on these high levels of hunger and poverty, on April 24, 2010, Nepal was officially selected as one of twenty focus countries worldwide for the USG Feed the Future (FTF) Presidential Initiative. According to the Nepal Demographic Health Survey (NDHS 2011), the rate of stunting among under-five children is 41% (surpassing Sudan), wasting is 11% and underweight is 29% (exceeding Ethiopia), reflecting widespread chronic malnutrition. Nearly half of children six months to five years of age are anemic. Although the problem of under-nutrition is pervasive across all of Nepal, national aggregates mask wide disparities across socioeconomic groups and ecological regions. Agriculture plays a crucial role in Nepal’s economy. Due to increasing population and declining rates of agricultural growth, Nepal is considered a structurally food deficit country. Crop production, like poverty rates, varies significantly by region and district. ● 66% of households in Nepal are experiencing food shortages ● 43% of households are skipping or reducing meals ● 30% of households in the Hill and mountain regions are forced to consume seed stock ● 73% of households in the mountain regions send at least one member out for work Nepal’s terrain, 64% of which is covered with mountains, is only about 22% arable, and is therefore not easily suited for efficient food grain production. The hills and mountains in the Mid-Western and Far-Western Regions typically have the highest rates of food insecurity and hunger, since crop production is often poor and transportation and other infrastructure is lacking. In this context, farmers have limited exposure to improved agriculture practices and technology and the government extension system needs to be strengthened. Furthermore, there are weak linkages between producers and consumers and poor access to markets for farmers. The rugged terrain increases transport costs, impedes links of products to markets, and makes service provision difficult for large parts of the country. Nepal is also particularly vulnerable to climate change and has recently experienced increased drought and flooding. Due to extremely low levels of income and agricultural production, the poorest households must allocate almost three quarters of their income to food. Additionally, many of these people were 88 highly affected by the decade-long conflict and increasingly common drought and other natural disasters exacerbated by climate change, leaving them with few coping mechanisms to deal with rising food prices. USAID/Nepal and other donors joined in a Government of Nepal (GON)-led process to identify nutrition needs in Nepal. The problem analysis and knowledge management review reveals that the main underlying causes of hunger, poverty, and under-nutrition in Nepal include low agricultural productivity, poor household decisions about food distribution and consumption, limited livelihood opportunities, weak market linkages, poor enabling environment, poor hygiene and sanitation practices, and inadequate production, availability and consumption of nutritious, locally-available foods. 2) DESCRIPTION OF THE PROJECT Activity Name KISAN Implementer Winrock International Cooperative Agreement/Contract # AID-367-C-13-00004 Total Estimated Ceiling of the Evaluated Project/Activity(TEC) $20,414,809 Life of Strategy, Project, or Activity February 14, 2013 – August 13, 2017 Active Geographic Regions USAID/Nepal FTF 20 Districts Development Objective(s) (DOs) DO2. Inclusive and sustainable economic growth to reduce extreme poverty IR2.1. Agriculture-based income increased IR2.2. Small enterprise opportunities expanded USAID Office Social, Environmental and Economic Development Office, USAID/Nepal The KISAN activity is part of USAID’s global Feed the Future (FTF) initiative, a $20 million five-year program, working to advance food security objectives by increasing agricultural productivity. Nepal was one of 19 countries chosen for the Presidential FTF Initiative in 2010. The overall goal of the FTF initiative is to reduce hunger and poverty while improving nutrition in Nepal. The KISAN activity purpose, which contributes to the achievement of this goal, is to increase agricultural productivity and income, improve nutritional status of women and children under five, and increase resilience of vulnerable communities and households. FTF Nepal’s geographic zone of influence (ZOI) is centered on 20 Terai and lower Hill districts in the Far-Western, Mid-Western, and Western Regions. The Project has three primary and integrated components which focus on: a) Agricultural productivity, b) Improved Nutrition, and c) Skills development (literacy, numeracy, and business/ entrepreneurial skills). KISAN works in close coordination with the GoN’s Ministry of Agricultural Development (MOAD) and is aligned with the other FTF projects in Nepal, including the Cereal System Initiative for South 89 Asia (CSISA) and the Innovation Lab (IL) programs. KISAN collaborates with USAID’s Business Literacy Program – which is increasing the business literacy of 54,000 KISAN beneficiaries – and with USAID’s Suaahara project, which is improving the health and nutrition of Nepalis within KISAN’s focus districts. Agriculture and nutrition interventions of KISAN are complementary. Improved livelihoods enable households to indirectly benefit through various ways, such as increased food quantity, quality and variety, purchase of livestock and other productive assets, lower seasonal migration, increased savings and loan repayments, and improved school enrollment. KISAN focuses its interventions on high value vegetables and horticultural crops (such as tomatoes, cauliflower, cabbage, eggplant, peppers, and beans, among others) that can improve local nutrition as well as generate incomes through sales and marketing. Additionally, the Project works with traditional cropping systems, specifically lentils, maize, and rice, and develop activities to attract risk adverse smallholders, such as intercropping with vegetables, crop rotation, conservation tillage, early harvest and climate resilient rice seeds, and protein-rich maize varieties. The choice in types of foods produced will impact consumption patterns and nutritional status of women and children. Although KISAN works in selected value chains to help develop markets, KISAN is not a classic “value chain” project. This is a reflection of its focus on food insecure areas and specific commodities. In reaching the objective of improved nutrition, KISAN directly links Project activities to behavior change communication (BCC) activities at the community level to provide a comprehensive set of actions aimed at improved nutrition, water sanitation, and hygiene practices. Additionally, it provides households and communities with the basic resources and knowledge to improve their availability and access to nutritious and diverse foods, as well as providing training to change agents. The nutrition and hygiene component will have a special focus on households with pregnant and lactating women and children under five years of age, with a particular focus on children under two years of age. KISAN mobilizes change agents that can provide improved technologies, quality inputs, technical know-how, knowledge, and other back stopping support for production and marketing of vegetables and agricultural crops. The Project builds the capacity of private sector and community-based organizations to improve the availability of quality farm inputs; increase access to credit, extension and other services; and improve the competitiveness and efficiency of processors and other buyers. In parallel, KISAN facilitates market linkages between farmers and input providers, service providers, and buyers. It also engages with a wide range of public, private, and civil society stakeholders including farmers, NGOs, academic and research institutions, businesses and training centers. The Project incorporates female-friendly farming practices in its interventions, such as the adoption of multi-use water systems or appropriate tools. KISAN ACTIVITY OUTCOMES 1. Improved access to increased quality inputs for farmers 2. Improved capacity of agriculture extension workers, service providers, and farmers 3. Improved sustainable agriculture production and post-harvest practices and technologies adopted at the farm level 4. Improved market efficiency 5. Increased capacity of GON and Nepali organizations Figure 3. KISAN Results Framework FY15 – FY17 90 3) PURPOSE OF THE EVALUATION The purpose of this final performance evaluation of the KISAN activity is to assess USAID/Nepal’s effort to increase agricultural productivity and private sector investments in agriculture in the FTF Zone of Influence (ZOI). First, the evaluation will include a desk review of existing data to analyze differences in agricultural data between the FTF ZOI and the national statistics, as well as within ZOI districts between direct FTF beneficiaries and non-FTF beneficiaries, to better understand the performance data. Second, the evaluation will examine the effectiveness of integrating various FTF activities in the ZOI, and what changes were seen in market systems as a result of FTF interventions. Findings from the evaluation will be used to determine strategies for implementing follow-on FTF activities, and provide evidence of best practices. The final performance evaluation will be primarily for the use of USAID, KISAN and other FTF Implementing Partners for identifying best practices and lessons learned to inform future programming decisions. 91 4) EVALUATION QUESTIONS The following evaluation questions will guide the final evaluation process. In answering these questions the evaluation team must analyze how the project has impacted women and other vulnerable groups. Evaluation Criteria Evaluation Question Relevancy 6. How do the main outcome level results achieved in Nepal’s FTF ZOI Districts compare to existing national statistics on agricultural productivity/yield, sales, consumption of inputs, and market efficiency, among others? Within the ZOI, how does data compare between FTF direct beneficiaries and non-FTF beneficiaries? Effectiveness 5. In what way has the interaction between different activities (i.e., strategy to integrate agriculture activities KISAN with IPM, CSISA and BLP) contributed to achieving outcomes in: (xi) Improved access to increased quality inputs for farmers? (xii)Improved capacity of agricultural extension workers, services providers and farmers to increase agricultural productivity/yield? (xiii) Improved agricultural production and post￾harvest technologies and practices adopted at farm level? (xiv) Improved market efficiency? (xv)Increased capacity of GON and local organizations? c. What are the particular features of the integrated USAID strategy that have made a difference in outcomes? 6. How do project beneficiaries (disaggregated by sex and marginalization) perceive impacts of the interventions on household farm income, agricultural productivity/yield, food security and nutrition? What project interventions/strategies do they find as helping or hindering increased agricultural productivity/income and improved nutritional status? j. What changes have taken place in the household income, food security, and nutrition of target populations as a result of the interventions? k. What explains such changes, particularly with respect to the most marginalized households? l. To what extent have the interventions had an impact on women’s incomes and livelihoods, particularly among the most marginalized? m. To what extent have the interventions had an impact on the nutritional status of women and children, particularly those under age 2? 92 7. In what way have KISAN interventions (including engaging in specific value chains) contributed to changes in the market systems and private sector investment in agriculture? d. What interventions changed the functioning of market systems in KISAN districts (i.e., expanded and inclusive markets – small-holder farmers, small enterprises; collection centers/aggregation of products; improved sales/prices; access to markets, inputs/services, financial services; and, investment)? e. What interventions have had unanticipated negative impact on functioning of market systems in KISAN districts? Sustainability 7. Based on findings from effectiveness questions, what factors, interventions or changes to systems or institutions (public and private) may contribute to sustainability in follow-on activities? 5) EVALUATION DESIGN AND METHODOLOGY The final evaluation of KISAN will consist of a rigorous methodology including a sampling strategy that will yield meaningful insights into key project outcomes as specified by the evaluation questions. Data Collection Methods 1) Literature review: A thorough review of project documents including the M&E plan and theory of change, and secondary literature in order to identify key trends in agricultural production, food security, and market systems, among others, in Nepal. 2) Review of national, FTF ZOI districts and beneficiary statistics: A thorough review of existing data in Nepal’s FTF ZOI districts, including beneficiary data collected by KISAN, and national statistics on agricultural productivity/yield, sale of products, consumption of inputs, and market efficiencies, among others. 3) Focus group and key informant interviews: FGDs and KIIs will be conducted with various stakeholders such as value chain associations, including women’s groups, and beneficiary households in order to obtain a deeper insight into the different dimensions of change in capacities, context, attitudes and perceptions of project beneficiaries. Data Analysis Quantitative data analysis: Comparative analysis of existing data in Nepal’s FTF ZOI districts against national statistics on agricultural productivity/yield, sale of products, consumption of inputs, and market efficiencies, among others. Qualitative data analysis: Systematic approach to coding, interpreting, and understanding the data. The focus group and in-depth interviews will be tape-recorded with the consent of the participants. Along with interview notes, all tape-recorded data will be transcribed and translated into English, and coded using Dedoose, a qualitative data analysis software. Disaggregation Data collection and analysis will be disaggregated by sex and marginalization status of target populations to discern differences and similarities of experiences, perceptions, needs, and barriers, among others. 93 6) EVALUATION TEAM The evaluation team will consist of experts comprising a range of skills directly relevant to the purpose of the KISAN final performance evaluation. The evaluation team will be composed of a team leader/evaluation expert, an agriculture sector specialist, a GESI (gender and social inclusion) expert, and three field researchers. Team Leader/Evaluation Expert (1): The team leader must have extensive experience working on or implementing Feed the Future activities. S/he should have a postgraduate degree, in agribusiness, agricultural economics, agronomy or a related field, with at least 10 years of experience leading and/or evaluating value chain development activities for USAID or other donor agencies, preferably in Asia. S/he should have experience leading evaluations and assessments for USAID, and be familiar with current USAID evaluation policy and guidance, and have experience with both quantitative and qualitative research methodologies. This experience should include development of research questions, and critically examining the validity of results frameworks and theories of change within the context that projects are operating within. Experience in systems and sustainability analysis, and assessing the development of agricultural producer groups, trade associations, as well as forward and backward value chain linkages required. The ideal candidate will have excellent oral and written communication skills in English; writing samples from recent evaluations or reports, and three professional references, will be requested. Agriculture Sector Specialist (1): The Agriculture Sector Specialist is a senior expert on promoting agribusiness and agriculture sector in Nepal, should have extensive experience working with USAID or other donor agencies. A Nepali citizen, the Agriculture Sector Specialist should have a postgraduate degree, in agribusiness, agricultural economics, agronomy or a related field, from a recognized national or international university, with at least 10 years of experience working as a senior officer or researcher in agriculture and agribusiness, including with questions relating to agricultural finance in Nepal. Experience leading, or serving as senior technical expert, in agriculture sector evaluations and assessments for USAID or other donor entities, is preferred. The ideal candidate will have excellent oral and written communication skills in English; writing samples, and three professional references, will be requested. GESI expert (1): Gender and Social Inclusion expert must have a minimum of Master’s Degree in social science or relevant subject area, and at least 5 years’ of experience as a member of an evaluation team. The GESI expert must have an extensive knowledge about the GESI issues in Nepal. GESI expert will work with the evaluation team to develop evaluation methodology to assess the how the project has affected women and other vulnerable groups. Field Researchers (3-6): The team will include three Nepali Field Researchers with a Master’s degree or equivalent in social sciences, preferably in women’s studies/ development studies, from a recognized national or international university with at least three years of experience working in the development sector, preferably with a focus on agriculture or a Bachelor’s degree with at least eight years of experience. They must have experience working with international organizations and stakeholders; knowledge of Tharu & Abadhi local languages, excellent oral and written communication skills in English; and demonstrable skills utilizing software applications such as excel, word, power point, etc. 7) KEY DOCUMENTS FOR REVIEW 94 ● KISAN Annual Work Plans ● KISAN and other FTF M&E plans ● KISAN and other FTF Progress Reports (Annual, quarterly) ● FTF Multi-Year Strategy (MYS) ● Mid-term Evaluation Report ● Other studies, reports o Adoption and Diffusion Study (IFPRI) o Lentils study (CSISA) o Lentils CBA (USAID/Nepal/Andrew Nelson) o Market channels study (KISAN) o Population Based Surveys (2013 and 2015) 8) KEY STAKEHOLDERS TO BE CONSULTED ● KISAN beneficiaries ● GON officials (MOAD, DOA, DADOs) ● Implementing partners, sub-partners, and USAID-funded projects in the region ● Other relevant donors ● Relevant NGOs and INGOs ● USAID/Nepal staff ● Market players/actors (retailers, agro vets, wholesalers, traders and processors) 9) TIMELINE FOR EVALUATION The total Level of Effort for the evaluation will be approximately 2.5 months spread over a three month period. The following is a tentative timeline for the evaluation tasks; the detailed timeline will be developed during team planning meeting and as part of finalizing the evaluation design. The evaluation is expected to start in March 2017 and be completed by early June 2017. Timing (Anticipated months or duration) Proposed activities Important considerations/constraints February end, 2017 Sign consulting agreements March 1-8, 2017 (7 days) Desk Review March 10, 2017 Travel of team leader to Nepal March 13 - 14, 2017 (2 days) USAID in-brief, KISAN activity briefing and team planning meetings March 14-22, 2017 (7 days) - Preparing the evaluation design - Preparing the evaluation instruments March 23-24, 2017 (2 days) USAID Review of the evaluation plan and instruments 95 March 27 - 29, 2017 (2 days) Key Informant Interview (KII) with USAID staff, KISAN staff and key stakeholders in Kathmandu March 30 - April 22, 2017 (21 days) Field data collection April 24 - May 12, 2017 (17 days) - Transcription/translatio n of data - Data coding and analysis May 16, 2017 Presentation of preliminary findings to USAID and KISAN May 17 - June 3, 2017 (16 working days) Consultants submit draft report to CAMRIS June 3 - 9, 2017 (5 working days) CAMRIS revision of draft report June 9, 2017 Submission of draft report to USAID June 9 – June 16, 2017 (5 working days) Review of preliminary draft evaluation report by USAID and KISAN June 19-June 22, 2017 (4 working days) - Consultant incorporating USAID feedback - Complete the Comments Matrix June 23-June 29, 2017 (5 working days) Finalization of evaluation report by CAMRIS June 30, 2017 Submission of final evaluation report to USAID 10) DELIVERABLES The following is a set of evaluation deliverables, with the expected timing of each adhering to the above Evaluation Timeline. 96 1. Discussion of plan and design: An in-briefing with USAID/Nepal upon the Team Leader’s arrival in Nepal for introductions and to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan, and/or to adjust the Statement of Work (SOW). 2. Evaluation Design: Following the discussions of plan and design with the Mission, the evaluation team will start work on the evaluation design. The evaluation team must submit an evaluation design (which will become an Appendix to the Evaluation report). The evaluation design will include: (1) Background and Project Description including Project Theory of Change; (2) Key Evaluation Questions; (3) Methodology (including data collection methods, sampling, and data analysis) (4) a detailed evaluation design matrix that links the Evaluation Questions in the SOW to data sources, methods, and the data analysis plan; (5) draft questionnaires and other data collection instruments or their main features; (6) the list of potential interviewees and sites to be visited and proposed selection criteria and/or sampling plan (must include calculations and/or a justification of sample size, plans as to how the sampling frame will be developed, and the sampling methodology, where applicable); (7) Evaluation work plan including the anticipated schedule and logistical arrangements and a list of the members of the evaluation team, delineated by roles and responsibilities; (8) known limitations to the evaluation design; and (9) a dissemination plan. 2. Discussion of plan and design: An in-briefing with USAID/Nepal upon the Team Leader’s arrival in Nepal for introductions and to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan, and/or to adjust the Statement of Work (SOW), and the evaluation design, if necessary. 3. Desk Review Analysis of data and Mid-term Briefing: The evaluation team is expected to hold a mid-term briefing with USAID after completing the desk review component on the findings of the desk review and its implications on the evaluation questions, including potential challenges and emerging opportunities. The team will also provide the evaluation COR/manager with periodic briefings and feedback on the team’s findings, as agreed upon during the in-briefing. If desired or necessary, weekly briefings by phone can be arranged. 4. Final Presentation: The evaluation team is expected to hold a final presentation in person/by virtual conferencing software to discuss the summary of findings and recommendations to USAID. This presentation will be scheduled as agreed upon during the in-briefing. 5. Draft Evaluation Report: , The contractor will submit a draft evaluation report of not more than 25 pages in length, single-spaced in TNR 12 point font, excluding Appendices, with an executive summary of not more than 3 pp. in length, within 9 weeks of the Team Leader’s arrival in Nepal 6. Final Evaluation Report: A final evaluation report of not more than 25 pages in length, single￾spaced in TNR 12 point font, excluding Appendices, with an executive summary of not more than 3 pages in length, within 10 working days of receipt of consolidated comments in electronic format from USAID. A Comments Matrix should also be included with a list of comments from USAID and responses from the evaluation team addressing each comment. All the quantitative data collected as part of this evaluation must be submitted as CSV files to Data Development Library with PII removed as required by ADS 579. The Final approved evaluation report must be submitted to the Development Experience Clearing House (DEC). 11) EVALUATION REPORT CRITERIA A. FINAL REPORT FORMAT 97 The evaluation final report should include an abstract; executive summary; background of the local context and the strategies/projects/activities being evaluated; the evaluation purpose and main evaluation questions; the methodology or methodologies; the limitations to the evaluation; findings, conclusions, and recommendations. For more detail, see “How-To Note: Preparing Evaluation Reports” and ADS 201mah, USAID Evaluation Report Requirements. An optional evaluation report template is available in the Evaluation Toolkit. The executive summary should be 2–5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) The Appendices to the report shall include: ● The Evaluation SOW; ● Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team; ● All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; ● All sources of information, properly identified and listed; and ● Signed disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of. ● Any “statements of difference” regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team. ● Summary information about evaluation team members, including qualifications, experience, and role on the team. In accordance with ADS 201, the contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within three months of the evaluation’s conclusion. B. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report. ● Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity. ● Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. ● The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. ● Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. ● Evaluation methodology should be explained in detail and sources of information properly identified. 98 ● Limitations to the evaluation should be adequately disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). ● Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. ● Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. ● If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. ● If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. 99 APPENDIX I: SUMMARY OF KISAN INDICATORS 100 STATEMENTS OF DIFFERENCE The following Statements of Difference were prepared by Winrock International in response to the Performance Evaluation of USAID/Nepal’s Knowledge-based Integrated Sustainable Agriculture and Nutrition (KISAN) Project (Contract No. AID-367-C-13-00004) dated February 2018. The following is Winrock International’s point-by-point response to findings presented in the evaluation report where there remains significant unresolved difference of opinion. Page viii. Regarding Evaluation Question 1, Outcome Agricultural Inputs and Technology, the evaluation states: While no national statistics were available to speak to agricultural inputs in KISAN areas, agricultural credit did increase in KISAN intervention areas at considerably higher rates than found generally through the limited data available. Winrock response: This statement does not fully capture the extent to which KISAN contributed to increased access to agricultural inputs. Based on KISAN’s Y5 annual survey, which collected data from 357 respondents, agrovet input sales increased by 99%, and agrovet clientele increased by 41%. KISAN worked with approximately 280 agrovets. These results are strong indicator of expanded access to inputs within the KISAN zone of influence. Page viii. Regarding Evaluation Question 1, Outcome Market Efficiency, the evaluation states: While little evidence was found to demonstrate KISAN contribution to market efficiency, almost all FGDs reported that they had been introduced to traders by KISAN, which may help facilitate some market efficiencies. Winrock response: This statement does not fully capture the extent to which KISAN contributed to market efficiency. KISAN annual survey data shows a significant increase in incremental sales of farmers, totaling $169.4M over the life of project (LOP). KISAN data also showed that the percentage of KISAN farmers selling to the market increased from a baseline of 70% to nearly 100% over LOP. Finally, KISAN data shows the percentage of farmers selling rice increased from 40% to 60% over LOP. These results are a strong indicator of KISAN’s contribution to market efficiency. Page ix. Regarding Evaluation Question 2, Outcome Improved Access to and Use of Quality Inputs, the evaluation states: Apart from its interaction with other activities, KISAN distributed new and improved varieties of seeds, some of which experienced issues with germination rates and yields and steep price increases. Winrock response: This statement is incorrect. KISAN did not distribute seeds, except for a very limited and time-bound distribution following the 2014 floods at the request of the Government of Nepal. Page ix. Regarding Evaluation Question 2, Outcome Improved Capacity of Agricultural Extension Workers, Service Providers, and Farmers, to Increase Agricultural Productivity or Yield, the evaluation states: The overall model of support, agrovets as a free extension service, did not appear sustainable post-intervention. 2101 Riverfront Drive | Little Rock, Arkansas 72202 p 1 501 280 3000 | f 1 501 280 3090 winrock.org 2121 Crystal Drive, Ste. 500 | Arlington, Virginia 22202 p 1 703 302 6500 | f 1 703 302 6512 And goes on to state: The capacity of agrovets greatly increased, allowing them to provide higher-quality extension services to farmers. The model for continuing their free extension services was not sustainable, though. Winrock response: Winrock disagrees with these statements regarding agrovet extension services. Agrovets did not provide “free” extension services. Agrovets sold products to farmers at training venues, and provided their clients with embedded extension services. KISAN data indicates that agrovet sales increased by 99%, and clientele increased by 41%. Based on KISAN’s assessment, embedded services contributed directly to the impressive results noted above and constitute a significant economic incentive for their continuation after the project. Winrock would appreciate the opportunity to review the evaluation team’s evidence that these services are not sustainable. Page ix. Regarding Improved Agricultural Production, Post-harvest Technologies, and Practices Adopted at Farm Level, the evaluation states: The partnership with BLP improved the literacy and numeracy skills of about 13,500 direct beneficiary farmers, increasing their capacity somewhat to learn new techniques though even many of those trained remain functionally illiterate. Winrock response: This statement is incorrect. Of the 118,000 HHs benefitting from KISAN activities, 51,000 also benefited from BLP activities. Page ix. Regarding Conclusions, the evaluation states: The KISAN activity and its partners played a minor role in improving market efficiency or expanding post-harvest technologies. Winrock response: As noted above: 1) KISAN annual survey data shows a very significant increase in farmer incremental sales; 2) KISAN data shows that the percentage of farmers selling to the market increased from 70% to nearly 100%; and 3) KISAN data shows the percentage of farmers selling rice increased from 40% to 60%. Page x. Regarding Evaluation Question 2, Conclusions, the evaluation states: However, this intervention is not fully sustainable, because over 60 percent of the staff recruited by the private sector specifically for the KISAN activity will be laid off after KISAN ends. Winrock response: This statement is not sufficiently supported by evidence. Elsewhere in the report the percentage is quoted as 50% (page 24). Regardless of whether the actual figure is 50% or 60%, Winrock contends that this is a positive finding that points to sustainability, whereas it is presented here as negative. This finding would indicate that 40% - 50% of farmers who received extension services from the private sector under KISAN will continue to receive those services after the project ends, whereas previously they would not have had any access to these types of services. These results are self￾reinforcing; as the private sector continues to increase sales due to improved relationships through the provision of embedded services provided to farmers, they will be incentivized to expand these services as time goes on. Winrock only worked with these private sector partners over a period of one year. Working with these partners over multiple years, depending on their specific needs, would likely further strengthen their capacity and increase the retention of more extension staff and agents. Page 2 Page xi. Regarding Evaluation Question 3, Conclusions, the evaluation states: The strategy of focus on increasing vegetable production and marketing as a result of KISAN increased on-farm production and household consumption, and nutrition thus has improved. Winrock response: Winrock wishes to point out that vegetable production was not the only focus of the project. KISAN worked in the rice, maize, lentil and vegetable value chains. Farmers were generally more attracted to produce and sell vegetables due to the profit-earning potential relative to other crops. Indeed, KISAN survey data shows that gross margin per ha is 8 times higher for vegetables than cereals (please see Table 15 of the KISAN Final Report). Page xi. Regarding Evaluation Question 4, Outcome Satisfaction with Changes to Service Provision and Access, the evaluation states: Combined contributions by DADO and KISAN in funding irrigation scheme improvements were viewed favorably but were largely limited to existing DADO-registered groups. The level of private sector involvement in providing services to farmers are not expected to continue at the levels found during KISAN, as travel allowances would disappear and grantees were expected to reduce by roughly half the technical staff employed. Winrock response: Winrock seeks to clarify that irrigation scheme improvements were exclusively limited to DADO registered groups, because this was a precondition for any support from the DADOs. And because of this precondition, KISAN assisted approximately 1,600 farmer groups to register with DADOs so that they would be eligible for support. Regarding the evaluation team finding that technical staff employed by the private sector would be reduced “by roughly half”, this would indicate that approximately 50% of farmers who received extension services from private sector partners will continue to receive those services, which is a positive outcome for nascent private sector firms that only worked with the project over a period of one year. Page xi - xii. Regarding Evaluation Question 4, Outcome Unanticipated Negative Impact on Functioning of Market Systems, the evaluation states: This reduced the price of vegetables and, in some cases, made vegetable production unprofitable. Further, the increased area in and geographically dispersion of vegetable production by various actors has facilitated the spread of pest and diseases, including new ones they do not know how to control. And goes on to state: Rapid expansion of vegetable production by KISAN and other projects resulted in reduced prices and has created problems with pests and diseases. Winrock response: These statements are not sufficiently supported by evidence, and Winrock questions whether gross margin calculations were completed by the evaluation team. KISAN found that when farmers earned less per kilo from a vegetable crop one year from the last they felt as though they were losing money; however, gross margin calculations showed otherwise. This was the perception of many farmers, but in general it did not prove out economically. Despite well documented price fluctuations, farmers still prefer to produce and sell vegetables. KISAN survey data shows that gross margin per ha for vegetables is 8 times higher than cereals. Winrock would appreciate the Page 3 opportunity to review the evaluation team’s evidence regarding the negative impacts of vegetable farming. Page xii. Regarding Evaluation Question 4, Conclusions, the evaluation states: While KISAN activities did have a small impact on market systems, most of the elements required for an efficient market system were not included in KISAN activities. Winrock response: Winrock disagrees with this statement. KISAN’s primary strategy was to build capacity of private sector service providers to increase the provision of extension services, provide quality inputs, and ensure markets for the production. By Y5, approximately 75% of KISAN farmers were receiving support directly from the private sector or local service providers. KISAN worked with more than 600 private sector enterprises in total. By the evaluation team’s own estimates, approximately 50% of those enterprises providing embedded services will continue to do so after the project ends, which is a successful outcome that demonstrates increased market system efficiencies. Page xii. Regarding Evaluation Question 5, Findings and Conclusions, the evaluation states: Many irrigation systems were inadequately funded by activity plus user fees/individual investments, also reducing their sustainability and the sustainability of KISAN outputs. Winrock response: This statement is inconsistent with KISAN findings, and Winrock would be grateful for additional information on the number of irrigation sites visited by the evaluation team and the percentage that were not functional. Winrock agrees that irrigation systems that do not have community or HH contribution are not sustainable, as farmers will not see themselves as the owners of the system. Based on KISAN’s experience, when there is farmer investment, they are very much involved in site selection and ongoing operation and maintenance. This, in turn, creates the foundation for sustainability. At the time of writing, KISAN has not received any information from communities, partners or the GON suggesting that any of the tube well schemes supported by KISAN are not currently operational, even during the dry season. Page xiii. Regarding Key Recommendations, Design and Management, the evaluation states: Since area farmers appreciated participating in cooperatives and they appear to work well in Nepal, especially in more-rural areas, USAID and its activities should consider supporting agricultural cooperative development and agricultural service developments (market aggregation and input supply) through these cooperatives. Winrock response: Winrock notes that this is a good suggestion for future implementation efforts. KISAN worked with only a few cooperatives in KISAN I, as our experience with cooperatives was that few were well managed. Winrock would be grateful to review the evaluation team’s information source backing up the statement that cooperatives “appear to work well in Nepal.” Page 10-11. Regarding Evaluation Question 2, Outcome Access to Increased Quality Inputs for Farmers, the evaluation states: KISAN provided free seeds1 to some farmers, mainly for the demonstration plots established to conduct the trainings. 1 Cereal seeds were produced by CSISA. Page 4 And goes on to state: Bidur Agrovet in Palpa and other farmers complained about the poor quality of cauliflower seeds, and farmers in Ghansing Dada reported poor yield from onion seeds distributed by KISAN. Winrock response: As previously noted, KISAN did not distribute seeds to farmers. In some of the Demo plots, seeds provided by the Cereal Systems Initiative for South Asia (CSISA) may have been used by KISAN farmers. Page 15. Regarding Evaluation Question 2, Conclusions, the evaluation states: While the grantee intervention had a significant impact on bringing new technologies to farmers, this sub￾activity is largely unsustainable because the staff recruited by the private sector specifically for this purpose will mostly be laid off post-KISAN, as potentially over 60 percent will not continue after KISAN-I ends. And goes on to state: Evidence suggests KISAN did not improve capacity of DADO extension workers measurably. Winrock response: As previously noted, this statement is not sufficiently supported by evidence. Elsewhere in the report the percentage is quoted as 50% (page 24). Regardless of whether the actual figure is 50% or 60%, Winrock contends that this is a positive finding that points to sustainability, whereas it is presented here as negative. This finding of the evaluation team would indicate that 40 - 50% of farmers who received extension services from the private sector under KISAN will continue to receive those services after the project ends. Winrock believes that this is a positive outcome from an initial round of one-year support provided to nascent firms for which this was an entirely new model of doing business. Regarding DADO extension workers: KISAN did provide training to approximately 80 GON staff, but our attempts to do more were often rebuffed by service center and DADO staff as they were either not interested or were too busy. KISAN instead worked through private sector organizations. Page 18. Regarding Evaluation Question 3, Outcome Changes in the Household Income, Food Security, and Nutrition of Target Populations as a Result of the Interventions and Facilitating and Hindering Factors, the evaluation states: In all groups, when men were working away from home, including overseas, remittances were the primary income source, and KISAN activity incomes were secondary. And goes on to state: When net income from farming is more than $1,500 per annum, it is sufficient to encourage migrant workers to stay at home and undertake agricultural activities. This would require a gross income of around $3,000 or more. Remittances from migrant workers also facilitate the affordability of new technologies like plastic tunnels and mechanization. Winrock response: These statements are not entirely accurate or contextually correct. Annual income from remittances in Nepal is approximately $1,100 per person on average; however, the upfront investment per individual to travel abroad for employment is approximately $2,000. Annual incremental sales from agriculture among KISAN farmers is around $800 per person. Due to the high investment up front, combined with the social cost of migration, most farmers survey by KISAN indicate that they want to Page 5 increase their income from agriculture. Farmers were encouraged to invest income from remittances into agriculture. Winrock suggests that this practice be continued under future agriculture initiatives. Sincerely, DeAnn McGrew, Director, Agriculture and Volunteer Programs Winrock International Page 6