1 This report was produced for the United States Agency for International Development (USAID) by Optimal Solutions Group, LLC, as part of USAID’s Learning, Evaluation, and Analysis Project (Contract Number: AID-OAA-C-11-00169). Nepal Economic, Agriculture, and Trade Activity Performance Evaluation USAID/NEPAL FINAL REPORT UNITED STATES AGENCY FOR INTERNATIONAL DEVELOPMENT LEARNING, EVALUATION, AND ANALYSIS PROJECT (AID-OAA-C-11-00169) NEPAL ECONOMIC, AGRICULTURE, AND TRADE (NEAT) ACTIVITY PERFORMANCE EVALUATION FINAL REPORT Prepared for United States Agency for International Development/Nepal by Optimal Solutions Group, LLC M Square Research Park 5825 University Research Court, Suite 2800 College Park, MD 20740, USA www.optimalsolutionsgroup.com Prepared by: Dr. Jim Hanson, Team Leader and Agriculture Expert Ms. Christabel Dadzie, LEAP Program Manager and Evaluation Specialist Dr. Pushpa Rajkarnikar, Trade and Economic Policy Expert Mr. Haseeb Payab, CBA and Agriculture Analyst May 30, 2014 Disclaimer The views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States government. i CONTENTS Acknowledgments.......................................................................................................................................... i Acronym List ................................................................................................................................................ ii EXECUTIVE SUMMARY ......................................................................................................................... iii I. INTRODUCTION AND PROJECT DESCRIPTION .......................................................................... 1 Nepal Economic, Agriculture, and Trade Activity ................................................................................... 2 Final Evaluation of the NEAT Activity .................................................................................................... 4 II. METHODOLOGY ............................................................................................................................... 6 Methods of Work ...................................................................................................................................... 6 Research Design.................................................................................................................................... 6 Survey Sample .......................................................................................................................................... 7 III. EVALUATION FINDINGS ........................................................................................................... 10 Characteristics of NEAT beneficiaries and partners surveyed and interviewed..................................... 10 Enabling Policy and Business Environment: ...................................................................................... 10 Enhancing Competitiveness of Selected Sectors Increased and Food Security: ................................. 11 Increasing Access to Financial Services ............................................................................................. 12 Findings Categorized within Key Evaluation Questions ........................................................................ 14 IV. CONCLUSIONS AND RECOMMENDATIONS ......................................................................... 48 Conclusions............................................................................................................................................. 48 Recommendations ................................................................................................................................... 50 V. APPENDICES .................................................................................................................................... 54 Appendix 1: USAID Evaluation Statement of Work for the LEAP Team ............................................. 54 Appendix 2: Sampling Design for Beneficiary Surveys......................................................................... 72 Selection and Sampling of VDCs ....................................................................................................... 72 Methods of Selecting Households ....................................................................................................... 72 Appendix 3: Value Chains and Food Security: Semi-structured Key Informant Interviews Organizations/Agencies Represented ..................................................................................................... 76 Appendix 4: Evaluation Report Reference List ...................................................................................... 78 Appendix 5: Design Matrix for Review of Nepal Economic, Agricultural and Trade (NEAT) Program Evaluation ............................................................................................................................................... 82 Appendix 6: NEAT Districts that Received Intervention ....................................................................... 86 ii Acknowledgments The LEAP evaluation team is grateful for the guidance provided by USAID/Nepal, in particular, Anita Mahat, economic specialist; Murari Adhikari, M&E specialist; and Kishore K.C., program specialist, who were very responsive to the team’s questions and provided advice throughout the evaluation. Special thanks are also extended to the NEAT Activity program staff, particularly Phil Broughton, chief of party, and Catherine Maldonado, M&E specialist, who were also very responsive to the LEAP team’s many questions at the onset of the evaluation and provided valuable guidance in identifying project implementation locations. Finally, the project team leaders are grateful for support provided by Optimal’s supporting analysts Nicola Smith-Kea, research associate, and Claire Everhart and Kristen Schwartz, research analysts, for the many hours they spent providing research support to complete this evaluation. Dr. Mark Turner, chief of party, is also acknowledged for his technical guidance and senior review. i Acronym List ABBS Any Branch Banking System ANROPI Association of Nepalese Rice, Oil, and Pulses Industry CEAPRED Center for Environmental and Agricultural Policy Research, Extension, and Development CEO Chief Executive Officer CMF Centre for Microfinance CNI Confederation of Nepalese Industries CSD Center for Self-Help Development DADO District Agriculture Development Office DAG Disadvantaged Group DOC Department of Commerce E3 Bureau of Economic Growth, Education, and Environment FEVEN Federation of Fruits and Vegetables Entrepreneurs Nepal FNCCI Federation of Nepalese Chambers of Commerce and Industry FNCSI Federation of Nepal Cottage and Small Industries FOPA Forum for People Awareness GON Government of Nepal HIMCOOP Himalayan Orthodox Tea Producers Cooperative Ltd. IEA Industrial Enterprise Act INSAN Institute for Sustainable Agriculture Nepal IRD Inland Revenue Department LEAP Learning, Evaluation, and Analysis Project MADE Multidimensional Agriculture for Development MFIs Microfinance Institutions MOAD Ministry of Agricultural Development MOC [Nepal] Ministry of Commerce and Supply MOF Ministry of Finance MOI Ministry of Industry NEAT Nepal Economic, Agriculture, and Trade [Project] NESTPROL Nepal Small Tea Producing Ltd NRB Nepal Rastra Bank OCR Office of Company Registrar PMP Performance Management Plans PPDs Public-Private Dialogues PPSDC Public Progressive Social Development Center SEAN Seed Entrepreneurs Association of Nepal SOW Statement of Work SSSC Sean Seed Service Centre Ltd. USAID United States Agency for International Development USC Unity Service Cooperation VDC Village Development Committee WTO World Trade Organization ii EXECUTIVE SUMMARY Project Purpose and Background The United States Agency for International Development (USAID)/Nepal contracted Optimal Solutions Group, LLC (Optimal), through its Learning, Evaluation, and Analysis Project (LEAP) contract to conduct a final performance evaluation on the Nepal Economic, Agriculture, and Trade (NEAT) program. The NEAT program was implemented between 2011 and 2013, following Nepal’s being named a country of focus by Feed the Future, Global Health, and Global Climate Change Initiatives. Chemonics International (Chemonics), the implementing partner for conducting the NEAT program, focused on enhancing food security; increasing agricultural production, exports, and sales; improving capacity; increasing government revenue; and improving trade and commerce in 21 districts in the East, Far West, Midwest, and Western regions of Nepal. Purpose of the Evaluation The purpose of the NEAT final performance evaluation is to assess the program’s performance and effectiveness, identify best practices and lessons learned, and offer recommendations for future programming. This evaluation adheres to the USAID Evaluation Policy (2011) by focusing on highlighting the NEAT project’s achievements, progress, and design details to produce descriptive and normative understanding of the program and its ability to achieve the expected results. Key Evaluation Questions The evaluation focuses on the following six guiding evaluation questions, as posed by USAID/Nepal: 1) How appropriate and effective were NEAT’s approaches and interventions in promoting inclusive economic growth? 2) How effective was the NEAT program in improving the economic status of women and disadvantaged populations? 3) How has NEAT’s policy-reform work improved the business-enabling environment for private sector–led growth? 4) How appropriate and effective was the program in improving the government of Nepal’s (GON’s) systems and capacity for economic growth? 5) How has the program enhanced the competitiveness of the selected agriculture value chains (lentil, ginger, vegetables, and tea)? 6) What are the prospects for sustainability of the end results produced by the NEAT program? Methodology The LEAP team collected and analyzed data from a wide range of program documents, surveys, and key stakeholder interviews. The evaluation was initiated with extensive document review of all project iii materials received. Interview guides were developed based on the key evaluation questions and project￾relevant information obtained through document review. Semi-structured interviews were utilized to assess the NEAT program’s macro-level activities, particularly the “enabling policy and business environment” component. A survey of beneficiaries was also used to gather information from a larger sample on micro-level activities of smallholder-farmer households, processors, input suppliers, and traders. Districts surveyed were determined through USAID/Nepal and NEAT implementing partners’ input. Village development committees (VDCs) within districts were then randomly selected for sampling. Whenever possible, households were selected within VDCs, using a sampling interval to reduce error. Qualitative and quantitative analyses of data were conducted during and after the field visit. Qualitative data synthesis of about 30 stakeholder interviews under the “enabling policy and environment” component of NEAT allowed the research team to identify common issues, successes, and challenges of the NEAT program. Surveys of more than 400 respondents across the four value chains provided descriptive statistics, such as measures of central tendency, dispersion, and distribution information. Triangulation was used to verify the information gathered. Key Findings Enabling Policy and Business Environment  Overall, the “enabling policy and business environment” component of the NEAT program was reported to be appropriate and effective. All respondents, with the exception of a few outliers, were pleased with the program’s approaches and interventions.  Public-private dialogues (PPDs) were noted as important activities in successfully promoting stakeholder engagement.  The use of industry experts as NEAT consultants reportedly instilled confidence among stakeholders in their abilities to adequately confront the important projects.  The several GON ministries involved reported success in ministry coordination, stakeholder engagement, and the effectiveness of NEAT’s project implementation.  Forty policies and procedural reforms had been assessed, drafted, or re-drafted by NEAT staff, nine of which have since been passed and moved toward implementation by the completion of the NEAT program. Food Security and Four Agricultural Value Chains  The efforts to improve food security by diversifying from staple crop production to vegetable production were similar to the efforts provided to help commercial vegetable producers. This similarity suggests that programs within USAID that work to bolster food security and those that support commercial value chains could, in certain circumstances, be compatible.  The 2.5-year duration was too short. Many of the participants wanted a longer program more in line with the original 5-year program that was approved. Many farmers claimed they were just grasping how to implement learned skills when the program ended.  Collection centers have been helpful in improving efficiency and saving time in farmers’ marketing efforts, but respondents shared that some collection centers are outside their reach (and therefore difficult to access), and others are incomplete. iv  All major efforts to aid vegetable farmers, which included training in market linkages and improved accessibility, building collection centers, and improving seeds, were reported as successful.  In general, stakeholders within the tea value chain were satisfied with the NEAT intervention. However, their perceptions about the program generally ranked lower than those related to other value chains. Tea growers were primarily dissatisfied with the intervention’s implementation approach and viewed it as a “top-down” approach that lacked true stakeholder participation.  The evaluation found an increase in lentil production from participation in NEAT, despite the challenges of excessive and detrimental rainfall. However, this increase was achieved by male farmers, whose yields increased 85 percent; female farmers’ yields actually declined.  Ginger farmers generally reported satisfaction with the trainings provided, but they wished for more training over a longer period of time, for they found it difficult to implement what they learned in such a short time. Farmers and NEAT instructors reported that a solution has not yet been identified for disease control, which continues to be an overarching problem. However, they did report an increase in yields and overall benefits.  Improved market access and better quality through post-harvest handling of ginger was reported.  A number of respondents, particularly within the ginger and tea value chains, thought that NEAT’s approach was not the most suitable, because the locals’ input on what they actually needed was not sought.  The sustainability of the NEAT program was hindered by its failure to focus on training local agricultural development officers who could continue NEAT’s work in the future. Key Recommendations  Consider splitting large, multi-faceted programs to narrow topic focus and reduce implementation complications. NEAT was a large and diverse program, which could understandably present size￾related complications. It is therefore recommended that future, similar programs be divided into two separate calls for proposal. In the case of NEAT, for instance, the agricultural and food security program could have easily been split from the enabling policy and business environment component. Smaller programs would allow implementers to better focus on the specific components needed to promote success in those areas.  Ensure stakeholder engagement with all key stakeholders and from the onset of programming. Respondents from the business sector shared their dissatisfaction in not being involved in decisionmaking from the onset, stating that the programs were very government-focused. It is recommended that future programs spend time in identifying the key stakeholders from the various sectors and involving them in the decisionmaking processes from the onset, particularly during the planning stages.  Subsidies for seed and fertilizer can be helpful for farmers involved in the food-security program. However, they should be limited to farmers who will have the ability to purchase these inputs in the future.  Encouraging farmers to use new hybrid seeds needs to be carefully considered. Although productive, hybrid seeds usually must be repurchased each year. For cash-strapped, geographically isolated poor farmers, hybrid seeds may not be the best solution. Improved, non￾v                                                              hybrid varieties are needed, but efforts should also be made to work with open-pollinated sources of seed so that farmers can be taught how to harvest and handle seeds for next year’s crop.  Future training programs for farmers should give a higher priority to controlling plant diseases. One of the NEAT program’s goals was to help farmers increase production through the use of better seeds, fertilizer, and improved cropping methods. Production did increase for all crops, but with this increased production came increased disease pressure, for the two are often linked. Plant-disease diagnosis and treatment should be included with future agricultural development programs.  Agricultural development programs that focus on capacity-building initiatives with farmers should ideally occur over a long period of time to ensure effectiveness. Throughout the evaluation process, beneficiaries and key stakeholders expressed the concern that the intervention was too short at 2.5 years. Many of the participants wanted a longer program more in line with the 5-year program that was originally planned.  To improve the sustainability of agricultural development projects, there should be a greater inclusion of all key stakeholders, particularly local district agricultural development officers.  Access to finance initiatives within similar programs should be included, as these activities empower the farmers to participate in other program interventions, such as training, capacity building, and purchasing of appropriate seeds, among others. Both the banks that received training and the beneficiaries who were able to access financial products ranked this component very highly. Ultimately, providing opportunities for financial independence to the NEAT beneficiaries would promote sustainability of the program’s interventions. Overall, the evaluation team found the NEAT program greatly benefited many stakeholders and made a positive impact through its interventions. There is room for improvement in future projects, which are outlined in the report, with some improvements already implemented in programs such as the Agricultural Development Strategy, led by the ABD.1 One main example includes the need for a longer program, which was requested specifically for the agricultural development program. People who are planning future interventions should carefully study the recommendations in this report to increase their programs’ impact. However, the NEAT program’s intervention was generally reported to be positive. 1 “TA 7762-NEP Preparation of the Agricultural Development Strategy (ADS)” vi                                                              I. INTRODUCTION AND PROJECT DESCRIPTION Since signing an assistance agreement with the Government of Nepal (GON) in January 1951, the United States Agency for International Development (USAID) has contributed to some of Nepal’s most innovative development successes, ranging from laying Nepal’s first roads and installing the first telephone exchange to enabling agriculture to flourish in once-uninhabitable regions and facilitating peace and democracy in recent years.2 Although Nepal has achieved remarkable development gains within the last decade, it continues to struggle with high poverty rates. Substantial economic disparities exist among Nepal’s ethnic and caste groups and across different regions of the country. The civil war that occurred between 1996 and 2006 caused additional setbacks in the country’s development process. A comprehensive peace agreement was signed at the end of the decade-long Maoists’ insurgency, which helped usher in historic political and social changes.3 Since then, progress toward fostering political reform and addressing critical development challenges has slowed because of frequent changes in government leadership and special￾interest groups’ jostling for power.4 Currently, Nepal faces many challenges. It is the poorest country in South Asia, with an estimated 55 percent of the population living on less than $1.25 per day.5 The country’s physical infrastructure has been chronically underdeveloped because of climate challenges, its extensive mountainous terrain, and its past political instability. In fact, most of the population does not have access to all-weather roads, which results in an impeded flow of goods across districts and into ports and airports. Additionally, Nepal has a low literacy rate (approximately 57.4 percent) and a very low rate of formal employment (about 4 percent).6 Some of these challenges result from problems being faced in various sectors of Nepal’s economy, such as the agricultural sector. Although the agricultural sector in Nepal accounts for more than one-third of the country’s gross domestic product and involves more than 60 percent of its workforce, Nepal’s increasing dependence on food imports has left the country vulnerable to global price shocks.7 The combination of low crop yields, globally high food-market prices, and damaged government institutions plays a significant role in the serious food-security and nutritional problems that affect the Nepalese population. It is estimated that around 41 percent of Nepalese children under 5 years of age suffers from stunting, which negatively affects their overall health and thus their potential to make future economic contributions.8 Despite these difficulties, many opportunities do exist to help ensure an adequate food supply, strong economic development, and good governance within Nepal. USAID’s assistance to Nepal has focused increasingly on expanding development efforts, a goal that was reaffirmed in 2010 when Nepal was 2 http://www.usaid.gov/nepal 3 http://www.chemonics.com/OurWork/OurProjects/Pages/Nepal%20Economic%20Agriculture%20and%20Trade.aspx 4 http://www.usaid.gov/nepal 5 Nepal FTF webpage (http://www.feedthefuture.gov/country/nepal) 6 http://www.unicef.org/infobycountry/nepal_nepal_statistics.html 7 Food and Agriculture Organization of the UN. “FAO initiative on soaring food prices.” Retrieved from http://www.fao.org/isfp/country-information/nepal/en/ 8 UNICEF. (2011).Statistics by Country - Nepal. Retrieved from http://www.unicef.org/infobycountry/nepal_nepal_statistics.html 1                                                              named a country of focus under all three U.S. government’s presidential-development initiatives— Feed the Future, the Global Health Initiative, and the Global Climate Change Initiative.9 USAID has worked at the macroeconomic level to improve Nepal’s business environment to foster growth in the private sector and has assisted industry sectors with increasing their competitiveness and productivity. USAID also remains committed to the basic foundations of democracy and free markets and has made significant contributions to support civil society, effective local governments, microfinance institutions, free-market systems, and sustainable private-enterprise growth.10 USAID’s overall assistance in Nepal focuses on  deepening support for civil society and good governance;  increasing the agricultural sector’s productivity and improving nutritional status;  providing sustainable, accessible, and quality basic health services to Nepalese citizens;  stimulating improved economic growth and trade and fiscal policies through increased participation with GON;  assisting with improvements to trade data availability, reporting systems, and negotiation capacities;  supporting GON’s education efforts, including promoting higher levels of literacy curriculum and teachers’ skills;  expanding interventions to address critical biodiverse areas of the country; and  increasing awareness and driving implementation of disaster risk-reduction measures.11 9 http://www.usaid.gov/nepal 10 Ibid. 11 Ibid. 12http://www.chemonics.com/OurWork/OurProjects/Pages/Nepal%20Economic%20Agriculture%20and%20Trade.aspx Nepal Economic, Agriculture, and Trade Activity The USAID/Nepal Mission maintains that it is possible to expand the Nepalese economy and generate substantial economic growth despite the difficulties Nepal faces. USAID’s Nepal Economic, Agriculture, and Trade (NEAT) program was designed to provide assistance in building the foundations for rapid, sustained, and inclusive economic growth, which will lessen pressures caused by conflict, reduce poverty, and improve lives.12 Initiated in December 2010, the NEAT program—a 2.5-year project—was implemented by Chemonics International (Chemonics). The project targeted 21 districts in the East, Far West, Midwest, and Western regions of Nepal and focused on promoting positive political and social change by enhancing food security; increasing agricultural production, exports, and sales; improving capacity; increasing government revenue; and improving trade and commerce. The NEAT program focused on the following:5  Economic growth—fostering a business environment conducive to private sector–led growth  Economic policy facilitation—supporting GON efforts to improve trade, fiscal policies, and practices to facilitate trade and increase revenues without distorting the economy 2  Market and export promotion—encouraging competitiveness and exports in selected agricultural sectors by reducing production, processing, and marketing constraints  Food-security improvement—enhancing food security through developing sustainable agricultural production practices, improving input delivery systems, and enhancing market linkages  Financial-service inclusion—strengthening microfinance institutions (MFIs) and services to increase their accessibility to underserved and disadvantaged populations, especially women and the poor, by building the capacity of existing microfinance service providers To achieve these goals, the NEAT program provided technical and managerial expertise to a number of stakeholders, including key government ministries, departments, and agencies; private-sector institutions, enterprises, and support organizations; and smallholder farmers. The program provided various types of support, including a combination of targeted, short-term technical-assistance programs, grants, subcontracts, procurement and distribution of goods and services, and training, within each of the following areas of work: improving capacity, increasing revenue, supporting World Trade Organization requirements, improving competitiveness, and supporting the National Trade Integration Strategy. The NEAT program’s intended results included  strengthening the GON’s economic policy platform through o providing legal framework to facilitate business; o assisting with advocacy, networking, and distribution of information; o developing a “one-stop shop” for new business registration and regulation; o writing business research reports; o streamlining tax collections and modernizing inland revenue administration; o updating and strengthening customs practices; and o strengthening capacity for trade-policy analysis and communication management;  increasing the competitiveness of selected sectors by fostering o a conducive business environment for private sector–led growth; and o more competitive agricultural value chains;  enhancing national food security; and  increasing access to microfinance services by o providing targeted technical assistance with the development of a new institute; and o building capacity within the sector through increased training for providers. Upon the NEAT program’s completion, USAID commissioned an evaluation to assess the program’s performance to gather best practices and lessons learned that could be used for future programs. 3                                                              Final Evaluation of the NEAT Activity Under the USAID Learning, Evaluation, and Analysis Project (LEAP) contract (no. AID-OAA-C-11- 00169), Optimal Solutions Group, LLC (Optimal), was tasked with planning, designing and conducting the final performance evaluation of the NEAT program between July and November 2013. This evaluation adheres to the guidance provided through the USAID Evaluation Policy (2011),13 specifically by focusing on highlighting the NEAT project’s achievements, progress, and design details to produce descriptive and normative understanding of the program and its ability to achieve expected results. The evaluation team utilized quality industry standards during project planning, data collection, and analysis of NEAT data to provide credible evidence that corresponds directly to the evaluation questions as proposed in the statement of work (SOW). In addition, the LEAP team took into consideration local and global contextual information in designing the data-collection tools to ensure that the results of the evaluation would be useful to the identified audience. Specifically, the evaluation team consisted of local and international staff, providing a platform to reinforce local capacity for future strategic evaluations. The results of this evaluation are fact-based and provide detailed evidence of the performance of the NEAT project beyond anecdotal evidence. Therefore, the results of this evaluation can appropriately be used to ensure accountability to NEAT stakeholders and to improve the effectiveness of future USAID projects in this sector. Evaluation Objectives The performance evaluation had the following objectives:  Examine the effectiveness of NEAT interventions, implementation strategies, and approaches to promote rapid, sustained, and inclusive economic growth and enhanced food security  Investigate intended and unintended consequences of the NEAT program  Document lessons learned, good practices, and factors that influenced program effectiveness that can be shared throughout the agency  Assess how the project affected men and women differently Key Evaluation Questions The evaluation focused on the following six guiding evaluation questions, as posed by USAID/Nepal: 1) How appropriate and effective were NEAT’s approaches and interventions in promoting inclusive economic growth?  Appropriateness of NEAT’s five components, its interventions, and its engagement with target group and stakeholders 2) How effective was the NEAT program in improving the economic status of women and disadvantaged populations?  Food security, access to financial services, productive engagement in the selected agriculture value chains, and nutritional outcomes 13 http://www.usaid.gov/sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf 4 3) How has NEAT’s policy-reform work improved the business-enabling environment for private sector–led growth? 4) How appropriate and effective was the program in improving GON systems and capacity for economic growth?  Taxation, revenue, customs, and trade 5) How has the program enhanced the competitiveness of the selected agriculture value chains (lentil, ginger, vegetables, and tea)?  Production, productivity, processing, and exports 6) What are the prospects for sustainability of the end results produced by the NEAT program? The LEAP evaluation team consisted of Dr. James Hanson, team leader and agriculture expert; Christabel Dadzie, program manager and evaluation specialist; Dr. Pushpa Rajkarnikar, local consultant and trade and economic policy expert; and Ahmad Haseeb Payab, cost-benefit analysis and agriculture analyst. Evaluation activities included developing an evaluation design, completing a document review, and conducting site visits to Nepal, which included key informant interviews with and surveys of relevant stakeholders. This report presents the findings of the LEAP team’s evaluation, which are based on analyses of data from project documents received, beneficiary farmer surveys, and key informant interviews. 5 II. METHODOLOGY Methods of Work This section details the methodology used to conduct the evaluation. For this evaluation, the following activities were implemented:  Implementation of a final performance evaluation that focused on the project’s implementation; examined its inputs, outputs, and outcomes; and assessed progress made toward target results and goals  Analysis of the program’s effectiveness in meeting its goals and its ability to promote the sustainability of implemented initiatives after the program’s completion  Quantitative and qualitative evaluation methods for data collection and analysis  Assessment and provision of findings for the program’s macro- and micro-level components  Identification of the project’s best practices and lessons learned within its implementation areas for future programming decision making Research Design The research design of the NEAT performance evaluation involved the collection and analysis of data from a wide set of project materials, surveys, and key informant interviews. The design, which was revised through a detailed document-review process, was submitted to and discussed with USAID/Nepal during the August 2013 In Brief meeting in Kathmandu (refer to Appendix 5 for evaluation design matrix). Data Collection The LEAP team first conducted an extensive document review and used a variety of data-collection methodologies, including surveys, semi-structured key informant interviews, and an observation study. Document review: Prior to the site visit, the evaluation team conducted an extensive document review of the NEAT program’s portfolio. The review aided the team in capturing background information about the program’s goals, stakeholders, inputs, outputs, and outcomes. USAID provided the evaluation team with various program-related documents, including annual work plans (years 1 through 3), annual and quarterly reports, a performance-monitoring plan, success stories, and relevant reports from other sources, including the NEAT implementation team. The evaluation team also reviewed background documentation about the Nepalese economy and government documents related to agricultural and economic policy. The document review was used to assess whether the project activities were implemented as planned and to identify any challenges or problems that delayed or altered their implementation. The full list of documents provided by USAID is included in Appendix 4. Semi-structured interviews: Key informant interviews were utilized to evaluate the NEAT program’s activities. In preparation for the key informant interviews, the evaluation team developed interview protocols that outlined the purpose, structure, and duration of the interviews and included a list of questions. The preliminary interview schedule was drafted prior to the site visits and finalized after the LEAP team arrived in Nepal in August 2013. To determine who should be interviewed, the LEAP team 6 consulted with the USAID/Nepal mission and Chemonics, which provided a list of stakeholders associated with the NEAT program. The list of organizations interviewed is provided in the Evaluation Findings section. In-depth, semi-structured key informant interviews with government agencies were used to analyze the relevance of NEAT’s activities to the government as well as changes in policies or policy decision making as a result of NEAT’s interventions. Interviews conducted with members of business communities were used to establish their perceptions of changes within the business environment that resulted from NEAT interventions. Interviews with beneficiaries of NEAT training activities were used to ascertain the effectiveness of these activities on trade promotion and economic growth. It is important to note that for a program such as NEAT, it is always challenging (and most times impossible) to isolate successes and challenges occurring solely as a result of the program, given that other external factors are always possible, which could also lead to positive (or negative) results affecting NEAT beneficiaries. Therefore, by conducting interviews, the team was only able to identify beneficiaries’ perceptions of progress/regress as a result of their participation in the program. Further, given the limited timeline for program implementation, it would be challenging to realize the full performance of the program, particularly on the policy side, given that a policy’s enactment is only the very beginning of its ability to produce positive results in a society. Survey of beneficiaries: To evaluate micro-level activities and identify detailed information about the effectiveness of the NEAT program for beneficiaries—smallholder-farmer households, processors, input suppliers, and traders—the evaluation team also developed a comprehensive survey and administered it to these beneficiaries. The team employed best practices in developing the survey instrument to ensure that the survey results would be valid and reliable. Survey Sample Due to the large size of the NEAT intervention, a sample of beneficiaries to be surveyed needed to be developed. Through collaboration with USAID/Nepal and NEAT’s implementing partners, the LEAP team identified a sampling of districts and village development committees (VDCs) in which the evaluation team collected beneficiary and stakeholder information on the project’s effectiveness. The sampling strategy and final sampling framework is provided in Appendix 2. Limitations to the Study This evaluation study faced the following main limitations:  Timeline for completion: Given the vastness of the NEAT program (see Recommendations section for information on possibly dividing such projects in the future), the time allotted for the evaluation was relatively minimal. However, USAID/Nepal needed the evaluation to be completed within a certain timeframe (before preparations were made for new programs), and as such the evaluation team endeavored to conduct the study within the period. The restricted timeline also meant that a limited number of stakeholders were interviewed, particularly concerning the “enabling business environment and policy” portion of the evaluation. The team 7 discussed the key stakeholders to be interviewed with USAID/Nepal and agreed on who was most pertinent to the evaluation.  Limited access to NEAT implementation staff: The evaluation began during the last week of the NEAT program, as it was wrapping up. This timing meant that the evaluation team was not able to conduct interviews with most of the NEAT staff members, which presented a challenge in confirming some of the findings in the field, particularly regarding how the initiative was implemented. To mitigate this challenge, the evaluation team worked closely with the NEAT Chief of Party (COP) and monitoring and evaluation (M&E) adviser, who tried to facilitate access to former staff members.  Access to appropriate beneficiaries: The primary part of the evaluation occurred during the rainy season, which meant that some areas where the NEAT intervention had taken place were not accessible. For this type of study, it is always useful to be able to reach as many respondents as possible (particularly given how geographically large the NEAT activity was) to ensure that findings represent all the beneficiaries. The evaluation team discussed this limitation with USAID and agreed on a sampling framework that would be as representative as possible.  Limited interviews among some stakeholder groups regarding the agricultural value chain and competitiveness: The evaluation team reached targets for the quantitative evaluation of the beneficiaries but were not able to complete as many qualitative interviews of stakeholders working within a value chain as desired. Although this situation did not invalidate the conclusions, reaching more stakeholder groups would have made the findings more robust. Data Analysis The LEAP team, in collaboration with USAID/Nepal, determined the analysis techniques that were most suitable for answering the six research questions. Analysis of macro-level activities The data gathered for analysis of the “enabling policy and business environment” component of NEAT were qualitative. Content analysis was used to objectively and systematically identify themes/topics derived from in-depth, semi-structured interviews. After each interview, the evaluation team compiled the interview notes and categorized responses into topic areas for reporting. A synthesis matrix was created that captured all interview responses. This qualitative data-synthesis matrix allowed the evaluation team identify common issues and outliers to responses in an objective manner and based on the compilation of findings. Analysis of micro-level activities A four-step process analyzed the data gathered from the food-security and the four value-chain surveys. This analysis provides descriptive statistics, such as measures of central tendency (mean and median); measures of dispersion, such as variance and inter-percentile ranges; and distributional information (percentiles and frequency).  Average responses (means) were calculated for each response from the five surveys for all farmer households sampled. Surveys were conducted only with members of farm households who participated in the NEAT program. 8  The data were further disaggregated into segments of the target population—males versus females, disadvantaged groups (DAG) versus non-DAG, residents in Terai versus residents in Hills—to provide more detailed information on micro-level activities  In each of the five surveys, respondents were given the opportunity to respond to open-ended questions. In particular, farmers were asked at the end of the survey whether “they experienced challenges with the project” and “what suggestions they had to improve the effectiveness and usefulness of such projects in the future.” These written responses were also analyzed using a thematic approach. This information was combined with the quantitative results to provide a better understanding of the farmers’ evaluations of the NEAT program.  Qualitative interviews were conducted with implementing partners, traders, processers, and input sales people. These results complemented the farmers’ evaluations to give a more holistic view of NEAT’s accomplishments and were analyzed in the same manner described previously for the qualitative macro-level data. Triangulation was used to verify the information gathered. Data collected from the document reviews, semi-structured interviews of various players in the process, and surveys were compared to the extent possible to strengthen the reliability of the analysis results. 9                                                              III. EVALUATION FINDINGS The United States Agency for International Development (USAID)/Nepal requested that the Learning, Evaluation, and Analysis Project (LEAP) evaluation team respond to six key evaluation questions, listed in the Methodology section. This section provides findings gathered through reviews of the Nepal Economic, Agriculture and Trade (NEAT) program’s documentation as well as data collected from the NEAT program’s beneficiaries and key stakeholders regarding its performance. Findings are compiled according to the six key evaluation questions posed by USAID/Nepal, keeping in mind the major expected results of the NEAT program. Characteristics of NEAT Beneficiaries and Partners Surveyed and Interviewed Enabling Policy and Business Environment One of the key goals of the NEAT program was to advance policies in Nepal to “unleash economic growth.” As such, the NEAT program established the following priorities:  Increase revenue  Facilitate trade and investment (World Trade Organization [WTO])  Promote agricultural competitiveness  Build the capacity of the public and private sector The LEAP evaluation team interviewed 34 stakeholders (19 representatives from eight government institutions and 15 representatives from 14 business and private-sector organizations and 1 non￾governmental organization), as shown in exhibit 1, below, to assess the relevance, effectiveness, and sustainability of the NEAT program to enhance the policy and business environment in Nepal.14 The team also visited one custom office to observe the operations of a NEAT-supported activity. Exhibit 1. Government and private-sector organizations interviewed Category Organization/Agency # of interviews conducted Department of Customs 5 Inland Revenue Department 1 Government ministries and departments Ministry of Agricultural Development 4 Ministry of Commerce and Supply 3 Ministry of Finance 1 Ministry of Industry 1 Office of the Company Registrar 2 Trade and Export Promotion Center 1 Private-sector and Confederation of Nepalese Industries (CNI) 1 business Federation of Nepalese Chamber of Commerce and Industry (FNCCI) 6 organizations Federation of Customs Agents Associations Nepal 1 14 Due to the limited evaluation timeline and budget, the LEAP team agreed with USAID/Nepal on the stakeholders to interview for this portion of the evaluation. 10                                                              Category Organization/Agency # of interviews conducted Federation of Nepal Cottage and Small Industries (FNCSI) 1 Freight Forwarder Association of Nepal 1 Seed Entrepreneurs Association of Nepal (SEAN) 1 Laxmi Bank, Ltd. 1 Mega Bank, Ltd 1 Nirdhan Utthan Bank, Ltd. 1 United Youth Community (UNYC) 1 Nongovernmental organizations Center for Self-Help Development (CSD) 1 Enhancing Competitiveness of Selected Sectors Increased and Food Security To boost the competitiveness of selected sectors, the NEAT program focused on four value chains with high export and import potential: orthodox tea, lentils, ginger, and vegetables. A market-driven approach was introduced to producers, and direct linkages were established between producers and buyers. NEAT team members also provided capacity building in technical areas, worked to improve business skills among actors in selected value chains, trained farmers in new production practices, and helped them increase their access to improved inputs. To improve food security, NEAT team members pursued a diversified approach in targeted food-insecure districts, including providing new production practices in districts and for specific commodities. The program also implemented a cost-sharing approach for providing subsidies to beneficiaries that encouraged farmers to access inputs, but over time the farmers received less assistance as they began to invest more in their land. To further assist with improving food security, input delivery systems were improved, and the program worked on enhancing market linkages. New approaches were also implemented, including livestock production for landless farmers and crop diversification for increased income and nutrition. The NEAT program also supported research and field testing for new varieties of disease-resistant seeds. To evaluate the effectiveness of the NEAT program’s ability to improve agricultural competitiveness and food security, surveys were administered to 449 farmers who participated in the NEAT program. Exhibit 2, below, provides the breakdown of beneficiaries interviewed by food-security level and by value chain. Survey participants ranged in number from 62 for vegetable farmers to 174 for food-security targets. Various demographic data were collected at the beginning of each survey to provide context for the respondent type as it related to the intervention received. Among the farmers who were interviewed, 43 percent were women, 48 percent were members of a disadvantaged group (DAG), and 37 percent were located in the Terai (as opposed to the Hill country).15 15 Note that these numbers are categorized per group calculated from the total; therefore, a female can be from a DAG or a male can be from the Terai region. 11                                                              Exhibit 2. NEAT participants, by different categories Food security Vegetables Tea Ginger Lentils Total All 174 62 78 66 69 449 Male 81 37 54 26 39 237 Female 80 22 22 39 30 193 Non-DAG 84 48 40 27 14 213 DAG 81 12 31 38 55 217 Hills 100 39 78 66 0 283 Terai 74 23 0 0 69 166 To gather additional information necessary for an effective evaluation of the NEAT program, semi￾structured interviews were also conducted with key stakeholders who were essential players in interventions related to the competitiveness of selected sectors and the enhancement of food security. These interviews included 17 implementing partners across food security and the four value chains; 5 input suppliers engaged in food security and the lentil and vegetable value chains; and 20 traders/processors engaged in the four value chains. These numbers are provided in exhibit 3, below. Exhibit 3. Semi-structured interviews, by sector Sector Implementing partner Input supplier Trader/processor Total Food security 3 2 0 5 Ginger 3 0 5 8 Lentils 4 1 4 9 Tea 1 0 8 9 Vegetables 6 2 3 11 TOTAL 17 5 20 42 Increasing Access to Financial Services Microfinance is regarded as one of the appropriate mechanisms to combat poverty.16 One of the NEAT program’s main components was the provision of avenues for microfinance institutions (MFIs) to increase access to financial services, in particular among women and DAGs. To do so, NEAT supported Nirdhan Utthan Bank Limited (NUBL) and United Youth Community (UNYC) for geographic expansion and new product development and Mega Bank and Laxmi Bank for branchless banking. The program also provided training opportunities to these (and other) financial institutions to promote sustainability of their products. Although NEAT did not have any program in the hills or mountains exclusively, the program encouraged partner organizations to expand their services to unserved and underserved areas that may 16 Improving Access to Microfinance Services in the Hills and Mountains of Nepal. Prepared by Dr. Purushottam Shrestha and presented at the Microfinance Summit Nepal 2013 12                                                              have included hills and mountain districts.17 Exhibit 4 below lists the financial institutions that the NEAT program engaged with. Exhibit 4. Microfinance and mobile banking organizations Organization/Agency Laxmi Bank, Ltd. Mega Bank, Ltd. Nirdhan Utthan Bank, Ltd. United Youth Community The Centre for Self-Help Development 17 Ibid. 13 Findings Categorized Within Key Evaluation Questions This section provides detailed findings about the NEAT program categorized by the key evaluation questions, listed in the evaluation statement of work (SOW), and within the context of the NEAT program’s targeted components. The findings provided were gathered through document reviews, semi￾structured interviews, surveys, and observation of NEAT-supported activities. It is important to note that the evaluation team made an effort to report the findings from the majority of the stakeholders but also was careful to document the outliers, particularly as some of these suggestions for improvement could be useful to future programs. These findings should not take away from the overall sentiment that the NEAT program was successful. Evaluation Question 1: How appropriate and effective were NEAT’s approaches and interventions in promoting inclusive economic growth?  Appropriateness of NEAT’s five components, interventions, engagement with target group and stakeholders a. Enabling Policy and Business Environment Overall, with regard to the “enabling policy and business environment” component of the NEAT intervention, the program was deemed appropriate and effective. With the exception of a few outliers, all respondents were pleased with the NEAT program’s approaches and interventions in promoting inclusive economic growth. Representatives from the government ministries and departments interviewed expressed satisfaction with the NEAT staffers’ initial approach of first informing the ministries about the program’s pending activities and then receiving input on the work plans and decisions to be made. The NEAT team made a conscious effort to ensure government key stakeholder involvement through advisory committees and work groups, some of which were already in existence within the ministries and others that were set up in collaboration with the ministries and departments. Additionally, representatives from the private sector—particularly, the Federation of Nepalese Chambers of Commerce and Industry (FNCCI) and the Confederation of Nepalese Industries (CNI)—were included in the initial process of deciding how the NEAT program was to be conducted, particularly concerning their work with the Ministry of Industry (MOI) on developing an investment portal. However, other members of the business community who were interviewed regarding their involvement with the NEAT program, particularly through public-private dialogues (PPDs), expressed the desire for more participation during the inception stages, a request that naturally would be difficult to implement, given the vastness of the private sector. One activity that clearly stood out as very positive to government and private-sector respondents were the PPDs, which were noted as important activities in successfully promoting stakeholder engagement; PPDs occurred often throughout the development of studies, guidelines, policies, and acts. Furthermore, some stakeholders stated that NEAT consultants included known industry experts, which therefore instilled confidence among the stakeholders in their abilities to adequately confront important issues. The overall sentiment within this area regarding the NEAT program’s 14 approaches and interventions was positive, and stakeholders, particularly within the government sector, believed that individual interventions would promote economic growth in the short, medium, and long term (should the policies be fully implemented). Public-Sector Stakeholder Engagement: A respondent from the Ministry of Finance (MOF) reported that the NEAT program adequately leveraged the ministry’s role in serving as a liaison between the other ministries and NEAT team members, coordinating activities, and approving implementation plans. This system was implemented through an agreement between the NEAT program and the Government of Nepal (GON). The advisory committee which was formed by the MOF and headed by the joint secretary of finance approved activities, while the individual ministries chose and developed their respective programs. Members of the MOF also shared that they witnessed good relations between representatives from the NEAT program and from other ministries. Implementing ministries agreed that coordination and stakeholder engagement were appropriate and effective. Representatives from the Ministry of Commerce and Supplies (MOC) described NEAT activities as “demand-driven,” with the ministry leading the charge in identifying and setting the stage for the main focus areas and developing work plans in collaboration with the NEAT team. Respondents from MOI, which conducted “in-house” discussions and decided on priorities to be addressed, agreed with this assessment, stating that MOI led in-house discussions to identify priorities and then asked that the NEAT program support the selected programs, after which a series of discussions took place between MOI and NEAT staff members regarding the respective programs and the operating modalities. MOI was involved in the decision making process as well as in preparations of the operational modalities of the interventions. MOI interviewees also shared that they received financial, logistical, and technical support from the NEAT program to prepare policies, acts, and regulations. NEAT consultants contributed to the preparation of the draft documents, proposed revisions, and also worked to build the capacity of GON participants. The drafts developed were then reviewed by MOI, after which PPDs were conducted to further review the updated reports and make changes as needed. NEAT’s involvement also helped ensure that the process continued to move forward. Given the collaborative effort between NEAT, government entities, and the private sector (where applicable), respondents were pleased with the NEAT program’s ability to ensure that these major efforts, as much as possible, did not stall. Policies, acts, and regulations that underwent this process included  the Foreign Investment Policy (FIP) 2069;  the Intellectual Property Right (IPR) Policy;  the Industrial Enterprises Act (IEA);  IEA regulations; and  the Foreign Investment and Technology Transfer Act (FITTA). Private-Sector Stakeholder Engagement: In the private sector, CNI positively rated the effectiveness of NEAT’s project implementation. CNI reported that NEAT sought the organization’s input in developing a work plan for the implementation of an investment portal (directed by MOI). The NEAT program was also acknowledged for providing financial and logistical support for the study and launching of the 15 investment portal and for assisting CNI with facilitating PPDs for drafting FITTA. CNI successfully launched the investment portal, named “Investment Nepal” and owned by GON. CNI credited the NEAT program for its assistance in making this portal a reality. As mentioned previously, the NEAT program partnered with FNCCI from the initiation of the program. Respondents from FNCCI shared that the NEAT program provided financial support for a variety of activities, including drafting the Agri-Business Promotion Act, drafting the Agri-Mechanization Policy, and developing a Business Confidence Index. Also, a NEAT consultant supported the drafting process for the Agri-Business Promotion Act and provided technical support to a group of experts within the Ministry of Agricultural Development (MOAD) as they drafted the Agri-Mechanization Policy. These processes, as noted by respondents, were all very participatory in nature. A private seed company involved with the NEAT program had a different opinion about stakeholder engagement. Representatives from SEAN Seed Service Centre Ltd (SSSC) suggested having more local involvement in the original design of the project. They noted that local partners should include the members of the private sector (e.g., SSSC), nongovernmental organizations (NGOs), and GON. According to a respondent, SSSC was brought on to the project a year after implementation, which limited the company’s ability to be effective. However, SSSC was not a named partner in the grant, so it was not possible to have brought it on any earlier. Ideally, SSSC should have been able to import parent lines and begin a breeding program, but this activity takes much longer than its 1.5 years of participation in the NEAT program. Appropriateness of the NEAT program’s activities in alignment with government priorities: To truly promote economic growth and improve the business environment, the alignment between selected activities and government priorities is important to ensure that guidelines and acts will be implemented to promote sustainability. Most respondents agreed that the NEAT program’s approaches were effective and that its interventions were in line with government priorities. MOAD shared that the NEAT activities implemented within its ministry were strongly aligned with government priorities. For instance, drafting the new Agri-Business Promotion Act, which provides for contract farming, and drafting the Agriculture Mechanization Policy are important components of agricultural development and therefore highly prioritized on the government’s agenda. Further, with the NEAT program’s assistance, the National Seed Policy was also implemented through a consultative process. However, the compensation guideline is pending; as a result of the new policy, one respondent noted that the number of seed registrations has slightly increased when compared to past figures. Similarly, according to MOI, the existing Foreign Direct Investment (FDI) policy was more than 2 decades old and needed updating; the NEAT intervention was timely in spearheading this effort. Respondents noted that additional discussions were still needed to finalize the FDI policy and that GON had yet to move forward with this matter. Similarly, with Nepal’s having made several commitments to WTO yet being unable to fulfill them, new policies or revisions to existing policies were required to make the country compatible with WTO guidelines; GON had already announced the Industrial Policy 2010, but its implementation had been stalled because of the absence of a new Industrial Policy Act. Therefore, in updating and developing the supporting act to enhance implementation of the industrial policy, the NEAT intervention was very timely in this sector and aligned with government priorities. 16 Reactions, however, were not all positive. For instance, one MOAD representative stated that NEAT’s fertilizer study was redundant and did not align with government priorities. It was expressed that other organizations had already conducted similar studies on fertilizer and that MOAD had already submitted a concept note to the Investment Board in response to a Japanese International Cooperation Agency (JICA) comprehensive study on fertilizer covering its demand and the establishment of a manufacturing plant, among other topics. Therefore, it was argued that NEAT’s partial study was not relevant, and had MOAD been consulted from the onset, a different set of priorities would have been identified. It is important to note that the ADB-led Agricultural Development Strategy (ADS), referenced previously which provides the state of agriculture in Nepal as well as a platform for ways forward, incorporated recommendations from the NEAT study. The evaluation team was informed by USAID/ Nepal that in coordination with MOAD, it was agreed that USAID would conduct the demand side assessment and WB will do the supply side. These recommendations are now part of the ADS. This finding is important to explain that although some MOAD staff might have viewed it as a flaw for NEAT not to tackle all aspects of agricultural development (also possibly because they did not have full information about the program’s agreed upon tasks), it is always important for a program, with limited timeframe to be able to focus on a particular aspect of programing that can make its work most effective, which is what was the case for NEAT. Similarly, although approximately 65 percent of GON respondents expressed that selected NEAT interventions were definitely useful, a MOF representative stated that most of the implemented activities were supply-driven and fell outside the priorities of government agencies and departments. Others within MOF expressed that NEAT funding should have been funneled through the government treasury, per usual practices, given that resource management is the primary task of this ministry and that therefore funds should have been transferred to government agencies instead of coming directly from NEAT. It is important to note that this sentiment is a common complaint for USAID programs that do not channel funds through GON, but this process is due to a requirement outside the purview of the contract and therefore does not reflect the contract’s effectiveness. One other respondent expressed concerns about the NEAT program’s focus being mainly on production, arguing that a large portion of programs included only software components and that, although software support is essential, it is not enough for commercialization of agriculture; what is equally important, but missing from the NEAT program, is support regarding hardware components, particularly for post-harvest activities (e.g., cold storage, roadways for transportation of agricultural product, processing plants, establishment of market collection centers, etc.). This aspect of the agricultural process is essential and as such should be included in interventions of future projects. The evaluation team notes that the NEAT program responded to a SOW that was agreed upon and therefore was not able to tackle all the various needs for promoting agriculture, particularly within a limited implementation period. The NEAT program did facilitate the development of collection centers for market linkages, which are an essential component of “hardware” deemed useful to promoting commercialization of agriculture. Further, the NEAT program’s activities were established so that farmers receiving microfinance loans would be able to purchase the hardware necessary to increase their productivity. 17                                                              b. Improving Agricultural Competitiveness and Food Security Overall, the NEAT food-security and value-chain interventions were successful in achieving their goals. Responses from beneficiary surveys on food security and value-chain competitiveness were positive in terms of the approaches and effectiveness of implementation of the NEAT program. Respondents were also pleased with the results they achieved due to their participation in the NEAT program. However, beneficiaries believed improvement was needed in a few areas of the program. Exhibit 5, below, shows survey responses to the overall program. Farmers who participated in the food￾security interventions shared the most positive responses about the NEAT program, which are reflected in an average score of 4.4. Although some differences may exist within a given value-chain or food-security group, simple averages taken across the five groups illustrate that farmers who are males or females, non￾DAG or DAG, and living in the Hills or in Terai all evaluated the program relatively equally. Exhibit 5. Response to survey question: Overall you are able to sell more “products” and earn more income as a result of NEAT’s interventions (max = 5.0) Food security Vegetables Tea Ginger Lentils Simple average All 4.4 4.1 3.8 3.7 3.7 3.9 Male 4.5 3.9 3.9 3.8 3.5 3.9 Female 4.2 4.1 3.4 3.6 4.1 3.9 Non-DAG 4.4 4.0 3.7 3.8 4.1 4.0 DAG 4.4 4.0 3.8 3.6 3.6 3.9 Hills 4.3 4.1 3.8 3.7 n/a 4.0 Terai 4.6 4.0 n/a n/a 3.7 4.1 Although the respondents reacted positively to the program, they had specific suggestions for the future, which are provided below. Concerning the overall interventions, the general consensus was that the NEAT program’s implementation period was too short. Many of the farmers as well as the input suppliers and traders/processors commented that the program would have been more effective if it had spanned a longer period of time. Farmers in particular reported that they were just beginning to grasp the ideas and implement changes on their farms when the program ended, which meant that the amount of guidance they had received dropped. Finally, if a program similar to NEAT were to develop in the future, the farmers would like to be exposed to a broader range of topics. These conclusions are supported by specific responses from farmers, provided below.18 Food security: 170 of 174 respondents (98 percent) offered suggestions for improvement, including the following:  Thirty-three farmers (18 percent) requested more instruction in the areas of innovative/modern farm techniques, seeds, livestock, fruits, vegetables, and cash crops. 18 The specific question asked was “What suggestions would you have to improve the effectiveness and usefulness of such projects in future?” 18  Twenty-nine farmers (17 percent) expressed that new programs such as NEAT should have longer durations. For instance, one farmer stated that the “timespan [of the project needs to be increased] because it takes time to learn and apply the procedures in the field.” Another farmer stated, “The project should be implemented for a longer time than just 18 months.” Vegetable value chain: 60 of 62 respondents (97 percent) offered suggestions for improvement, including the following:  Eighteen farmers (30 percent) wanted more training in such topics as vegetable seeds, disease control, modern farming methods, other cash crops, and off-season vegetable production. In terms of the training sessions, common themes that were expressed included the following: o “Vegetable trainings in the future should be more intensive and frequent.” o “We wish such programs lasted for an effective amount of time—and not just for about a year. This is too short a time to observe real changes in the behavior of farming practices or productivity.” o “There should be continuity [more years] in the project for a longer period.” Ginger value chain: 63 of 66 respondents (95 percent) offered suggestions for improvement, including the following:  In addition to disease management, mentioned in a later section, twenty-one farmers (33 percent) wanted more training in such topics as new seeds, livestock, modern farming methods, and income generation. Common themes regarding the way that training was offered included the following: o “The projects could have stayed longer to teach [us] more modern methods and techniques because [we] wanted to learn more and know more.” o “If the projects want to actually help farmers, they should inquire what the need really is, instead of imposing their project ideas on the villagers.” o “Project was too short; its duration should increase.” Lentils value chain: 65 of 69 respondents (94 percent) offered suggestions for improvement, including the following:  Twenty-one farmers (32 percent) had strong comments regarding the inadequate length of the project and the infrequent training sessions. Common themes from the farmers on this topic included the following: o “They could have stayed a bit longer and taught [us] more new and effective methods for farming.” o “They got training for a day from 10am to 4pm, which was a small duration.” o “The training was incomplete with a small time period. The timing was a bit wrong, the NEAT came at the end of the lentil planting and left at the time of lentil production. So, after production, there were many problems with no solutions.” o “The training was for 1 day and for 2 hours, which was inefficient. There were so many things to be covered, such as about diseases, lentils’ height, gel or medicines, or fertilizers to be used, which was not taught. Therefore these projects might have been longer to cover these areas.”  Sixteen farmers (25 percent) mentioned that they would like training in the areas of home economics, sugar cane, income generation, sustainable development, modern farming methods, livestock, poultry, and vegetable production. 19  It is important to note that an interview with a lentil implementer provided a possible explanation for why the training length was deemed inadequate: The implementer explained that the contract with NEAT started late, in September 2011. As such, because lentils are planted in December, everything had to be rushed. With some of the initiatives, farmers were only able to begin training on such topics as better weed control and marketing techniques. Tea value chain: 76 of 78 respondents (97 percent) offered suggestions for improvement, including the following:  Seventeen farmers (22 percent) would like roads and other improved modes of transportation to their villages.  In terms of subjects for future trainings, 10 farmers (13 percent) would like more help with cowsheds, and 9 farmers (12 percent) would like help with the production of organic tea. Stakeholder engagement: Information gathered from some beneficiaries (actual responses provided below) revealed that, overall, the NEAT program’s goals were predetermined without local input. Respondents noted that the program adopted a top-down approach and was not developed at the grassroots level. The MOAD and the District Agriculture Development Officers (DADO) were critical of this approach. However, information gathered from the interviews suggests that once the NEAT implementers heard of this problem, they changed their implementation methodology to be more inclusive. Similarly, the food-security implementing organization complained that the geographic areas in which its members would work were already decided. But again, once this problem was voiced, they were granted full authority to make their own decisions regarding which farmers they would work with within those regions. The problem of minimal or nonexistent stakeholder engagement seemed to have been most severe within the tea value chain, where the implementing partners were often large-scale tea estates and industries. Respondents reported that these partners did not welcome villagers’ suggestions, especially regarding the construction and handover of collection centers. Project beneficiaries and implementers in many cases seemed to have experienced a disconnect between their needs and the choice of programs selected, which led to low levels of ownership of the infrastructures by local communities. For instance, some beneficiaries expressed the concern that the NEAT program’s implementers did not first conduct an assessment to determine the true needs of the targeted areas and gain input from participants regarding which activities should be implemented. In addition, beneficiaries found some of the implemented activities to be less important than others that would have been more useful to them. One trader noted that the lack of transparency in the NEAT program caused problems for the farmers. Some of the big estates and processors received large financial grants for marketing improvements, leaving the small farmers concerned that this money was not used on behalf of poor growers. One marketer also noted, “[The] Nepal people are getting donor fatigue from being ignored in the planning of these programs.” Some specific responses and the corresponding percentage of respondents are provided below: Tea value chain: 17 farmers (22 percent) suggested a more grassroots approach to the tea project. More specifically, they made the following suggestions:  “Launch the project directly at the local level or at the cooperative level.” 20  “Implement the program by coordinating directly with the farmers as well as cooperatives.”  “Coordinate with the farmers or cooperatives while implementing the program. Priority should be given to local manpower.”  “These projects shouldn’t come to the farmers through traders/companies (e.g., Himalayan Shangri-La). It should reach out to the farmers through their cooperative and understand their problems directly.”  “Prior information about the projects, including their objectives, how they want the villagers to be involved/to contribute, when the projects will be completed, will help [implementers] gain [farmers’] trust and confidence. This is one of the most important steps for the success of any similar projects.” Investments: Investments made by the NEAT program in collections centers, grading and sorting facilities, fertilizers, seeds, and irrigation systems were all positively evaluated by the beneficiaries of the four value-chain interventions and food-security activities. More detail is provided about their value and suggestions to improve these investments are discussed on subsequent pages under the appropriate value chain or food security. Economic Status of Women and DAGs: Farmers who were interviewed and the various groups involved in implementing the program were not aware of any special projects for women and DAGs. However, the majority of beneficiaries of the NEAT intervention were women and people in DAGs, who gained economically and saw their livelihoods improve as a result of the program. Exhibit 6, below, shows the responses by women regarding their improvements as farmers. Food-security interventions were, on average, the most effective in improving women’s farming ability, closely followed by interventions related to lentils, vegetables, and ginger (even with the noted problems that existed with disease-prone seeds). In comparison to activities for the other value chains, the effectiveness of the tea program in improving women’s farming abilities was ranked significantly lower. Exhibit 6. Women’s opinions on how NEAT’s interventions have improved their farming abilities for the following factors (max = 5.0) Efficiency of farm operation Productivity of farm Decisionmaking power Market linkage Income Average Food security 4.5 4.6 4.5 4.5 4.1 4.4 Vegetables 4.3 4.3 4.2 3.9 3.9 4.1 Tea 3.3 3.0 3.3 2.9 3.2 3.2 Ginger 4.1 4.1 4.0 4.3 3.9 4.1 Lentils 4.3 4.4 4.3 4.2 4.1 4.3 Average 4.1 4.1 4.1 3.9 3.9 c. Increasing Access to Financial Services 21                                                              The NEAT program did not directly conduct interventions to increase access to financial services to targeted audiences. However, the program identified financial institutions and provided training and funding so that they would provide needed services to the target areas. Interviews with the financial organizations that participated in the NEAT program indicated that the support provided by the NEAT program was effective in enhancing their ability to work in these new areas, which realized profits to their institutions. These institutions were also very satisfied with trainings provided through the NEAT programs. When beneficiaries, mostly women, were asked about their satisfaction with their ability to access financial institutions, overall they responded positively, as detailed in the subsequent section. Evaluation Question 2: How effective was the NEAT program in improving the economic status of women and disadvantaged populations?  Food security, access to financial services, productive engagement in the selected agriculture value chains, nutritional outcomes a. Food-Security Programming for Women and DAG Populations The NEAT final report lists the following tasks as being used to improve the livelihoods and increase resilience for food-insecure farmers, particularly women, in Nepal:  Facilitating access to markets  Improving financing (discussed in Access to Financial Services section)  Increasing production by introducing sound agricultural practices  Encouraging livestock production by landless farmers  Introducing crop diversification for increased income and nutrition19 The following section represents findings gathered concerning interventions provided to the food-insecure populations, primarily women and the DAG population. Although Evaluation Question 2 was primarily focused on examining the effects on women and DAG populations, data were also disaggregated by Hill versus Terai populations where appropriate. As stated previously, the beneficiaries of the NEAT program did not have knowledge of specific programs targeting women and DAGs, but most of the beneficiaries who received interventions were women and members of DAG populations. As such, the report provides findings disaggregated by gender and DAG, with this section focusing on the interventions provided to food-insecure areas/populations. Overall, women and members of the DAG population who received the NEAT interventions were satisfied with the programs and reported positive changes to their livelihoods and economic statuses as a result. The following sections elaborate on the various NEAT programs introduced to the food-insecure areas and the findings gathered for each program.20 19 NEAT Activity Final Report (June 2013), p. 45. 20 Note that findings on women and DAG populations are also reported for those who participated in the “enhancing competitiveness for selected agricultural value chain” component, reported in subsequent sections. 22 Good Agricultural Practices Trainings: The NEAT program provided support to food-insecure areas by offering several training sessions with the hope that they would lead to improved livelihoods and food security. Training covered a variety of topics focused on using inputs, including seeds and fertilizers, as well as being introduced to improved practices, such as land preparation, nursery management, soil management, and irrigation, among others. (See exhibit 7, below, for the full list of training topics.) When beneficiaries were asked about their satisfaction levels with inputs and training on different agricultural practices, all responses on average were positive, rating 4.0 or above on a 5-point scale. The main concern voiced regarding training was the short length, as elaborated above. Exhibit 7. Farmer satisfaction level on inputs and trainings provided to promote food security Timeliness Usefulness Quality Average Inputs Seeds 4.4 4.5 4.6 4.5 Fertilizers 4.4 4.5 4.4 4.4 Training Land preparation 4.3 4.4 4.3 4.3 Nursery management 4.7 4.6 4.6 4.6 Soils management 4.2 4.2 4.1 4.2 Irrigation 4.0 4.1 3.9 4.0 Post-harvest handling 4.4 4.4 3.8 4.2 Market linkages 4.0 4.1 4.0 4.0 Business skills 4.3 4.3 4.0 4.2 Even with the project’s limited timeframe, through the training and implementation of new practices, increases in sales were realized. Exhibit 8, below, illustrates sales increases for farmers involved in the food-security program. One year includes the fall and spring crops, with fall 2012 and spring 2013 constituting the year for NEAT involvement. Growers were asked about their sales for the year prior to their involvement in the NEAT program in comparison with the year when the NEAT interventions occurred. Overall, farmers participating in the NEAT food-security programs reported a 28 percent increase in sales due to their involvement. Female farmers, DAG farmers, and farmers in the Terai experienced the highest rate of growth in sales after their participation in NEAT. Sales for farmers in the food-security program also varied within the three subgroups: Male farmers had greater sales than female farmers, non-DAG farmers had greater sales than DAG farmers, and farmers in the Terai had greater sales than farmers in the Hills. Exhibit 8. Total farmer sales from the food-security program for 1 year (spring and fall crops) for cereals and vegetables (Nepalese rupee), prior to and during NEAT Year prior to NEAT Year during NEAT1 % increase in sales All 53,996 69,296 28% Male 59,681 74,869 25% Female 36,918 46,808 27% Non-DAG 77,998 88,445 13% DAG 35,287 57,513 63% 23 Year prior to NEAT Year during NEAT1 % increase in sales Hill 38,284 42,087 10% Terai 69,866 89,546 28% Cost-Sharing Program: In addition to training on good farming practices, the NEAT program offered support to this targeted population with a cost-sharing system through which seeds and fertilizers were provided to farmers on a subsidized basis. In the first two crop seasons, farmers paid 40 percent of the cost, but in the fifth crop season, the farmers paid 100 percent of the costs. Beneficiaries were asked whether this cost-sharing approach was useful in terms of timeliness, usefulness, and quality. Overall, the farmers valued these inputs. An indirect subsidy, however, was that because 100 percent of the fertilizer was imported, the NEAT program had to deliver the inputs to central locations for distribution so that the farmers could obtain them. The Center for Environmental and Agricultural Policy Research, Extension, and Development (CEAPRED), the major implementer for food security, shared the opinion that fertilizer use would most likely fall by approximately 45 percent once the NEAT program formally ended. Similarly, conversations with farmers who participated in the food-security program and received subsidies indicated that up to 50 percent of those farmers—especially the very poor, who made the transition from staple crops to vegetable crops—would be returning to staple-crop cultivation, because they could not afford the seeds and other inputs that had been introduced to them during the program without the NEAT-provided subsidies. CEAPRED also shared that future programs should avoid the heavy subsidization of inputs, because it distorts farmers’ choices. Because of this issue, CEAPRED ranked the NEAT program as slightly unsustainable due to its overreliance on subsidies. This subsidization of inputs seemed to generate controversy among program participants:  On the one hand, farmers who had never used these inputs benefited by being exposed to them and realized their value. In addition, the subsidy amounts declined throughout the program, so in the end farmers were paying full price. It can be argued that these farmers will continue to purchase fertilizers and seeds on their own in the future, given that their new appreciation for these inputs already had them purchasing the supplies at full “expensive” price.  On the other hand, this rationale only applies to farmers who are located close to the source of the inputs. Many other farmers are unable to access fertilizer supplies because these inputs are not located within reasonable distances to them (making costs even higher because of transportation, which was formerly covered by the NEAT intervention). Irrigation. Many of the farmers surveyed also extensively discussed the matter of irrigation, saying they valued the irrigation services provided. Irrigation improved yield, total farm production, quality, and total sales (see exhibit 9, below). 24 Exhibit 9. Improvement in the following characteristics, by household, because of assistance with irrigation technologies Yield Total farm production Quality Total sales All farmers 4.1 4.1 4.2 4.0 However, farmers experienced some challenges with installation and assistance related to these technologies (see exhibit 10, below), which lowered their satisfaction ratings. Exhibit 10. Average satisfaction level, by household, after receiving assistance with irrigation technologies for the construction of small-scale irrigation schemes All Male Female Non-DAG DAG Hill Terai Satisfaction level 3.5 3.5 3.5 3.5 3.5 3.4 3.6 For many of the farmers, access to adequate water supplies and on-farm irrigation systems was not sufficient to irrigate their crops. Although it was difficult to ascertain the specific cause of the problems with irrigation, the main problem seemed to be related to the supply of water from wells that were bored. Farmers from the food-security group offered the following suggestions regarding irrigation:  Twenty-two farmers (13 percent) wanted more help with irrigation.  Fifteen of the 22 farmers who wanted more help with irrigation lived in the Terai. Detailed information regarding the specific problems that farmers had with the irrigation technologies was not collected in the survey. Of the total number of respondents, three had problems with artisanal boring. Most made comments similar to the following:  “Water availability is the main concern [in our village], so the focus should be on in irrigation.”  “Irrigation systems, if provided, should benefit the entire community and not just a selected few farmers.”  “Irrigation schemes only worked in my village and failed in the other seven villages where [they were] constructed. Such oversights should be prevented to make the program effective.” Crop Diversification: The NEAT program made a concerted effort to encourage these farmers to diversify from only staple crops to include vegetable production. Overall, farmers were pleased with this program: They reported that their family incomes increased and that the quality and quantity of their families’ food improved as a result. Exhibit 11, below, describes the satisfaction level of farmers who participated in the crop-diversification program. 25            Exhibit 11. Levels of satisfaction with the NEAT program’s efforts to diversify from staple crops to include vegetable production 3.4 3.6 3.8 4.0 4.2 4.4 4.6 Overall Satisfaction Family Income Qty. of Food for Family Qual ‐ Food for Family All Male Female This exhibit clearly demonstrates an increase in family income and improved quantity and quality of food as a result of the crop-diversification program. If “large difference in response” is defined as any response that is at least three-tenths greater, then the results show that men valued the diversification program more than women did (4.2 > 3.8). The men also thought it helped family income more (4.3 > 4.0). Similarly, although their responses are not shown in this exhibit, farmers in the Terai experienced greater overall satisfaction with the diversification program than did farmers in the Hills (4.2 > 3.9), and Terai farmers were more satisfied with their increased income than were farmers in the Hills (4.4 > 4.1). DAG and non￾DAG farmers’ responses to diversifying from staple crops to include vegetables were similar. Reduction in Hunger Levels: A major concern for families in the food-security program is hunger. Before participating in the NEAT program, these families lacked sufficient food for nearly one-third of the year, or 3.9 months, as illustrated in exhibit 12, below. Although these families still face challenges with hunger post-NEAT, currently the shortfall for all families has been reduced to 2.4 months. Exhibit 12. Number of months annually during which a family was short on food All Male Female Non￾DAG DAG Hill Terai Before NEAT 3.9 4.3 3.2 3.8 3.6 3.6 4.2 After NEAT 2.4 2.7 2.1 2.3 2.3 2.3 2.5 The survey also questioned whether beneficiaries were satisfied with the level of improvement in the daily amount of food available to their families as a result of the NEAT intervention. On a 5.0 scale, all participants answered 4.4, on average. Males and females answered 4.4 and 4.3, respectively, indicating a high level of satisfaction. Improvements in Nutritional Outcomes. Improvements in nutritional outcomes are key to promoting food security; as such, beneficiaries were asked whether they had made any changes to their diversity and preparation levels through the NEAT program’s interventions. Among the respondents, 96 percent 26                                                              increased the diversity of food provided to their families, but only 28 percent changed the preparation of food cooked for their families, as detailed in exhibit 13, below. This finding suggests that future programs could possibly include more information regarding food preparation to guarantee its quality, nutrients, vitamins, and safety. Exhibit 13. Changes to nutritional outputs as a result of the NEAT program’s interventions and training on food security Family Food Diversity Preparation All 96% 28% Male 98% 30% Female 94% 26% Non-DAG 99% 27% DAG 94% 27% Hill 97% 24% Terai 95% 32% Block Farming: The block-farming program introduced by NEAT as part of the food-security intervention showed moderate success among the targeted population. Among the 174 farmers surveyed, approximately one-third were involved in block farming and could speak to its effects, a number that aligns with the NEAT final report, which claims that 23 percent of farmers in food-insecure districts are now engaged with block farming.21 During the evaluation, when asked about the positive effects of block farming on collective marketing (as illustrated in Exhibit 14 below), respondents gave average scores of 3.5 regarding the quality of the product, 4.1 for sales, and 3.9 for the average price. These scores indicate that block farming had a positive effect on farmers’ collective-marketing efforts. Exhibit 14. Block-farming practices Block Farming Question Sub-question Number of respondents % of Respondents Average score (1–5) Q38: Getting engaged in block farming for collective-marketing purposes has had a positive effect in the long run on the following: Quality of product 48 28% 3.5 Sale 48 28% 4.1 Average price 48 28% 3.9 21 USAID Nepal. “Strengthening the Foundations for Inclusive Economic Growth.” Final Report: Nepal Economics, Agriculture, and Trade (NEAT) Activity. Prepared by Chemonics International, Inc. (Contract No. EEM-I-00-07- 00008). August 2013. 27                                                              Livestock: The NEAT program provided support to more than 1,500 households with its livestock activities,22 or 5.2 percent of the food-security intervention households. These respondents gave the livestock program above-average scores when asked about their satisfaction with the livestock they received, training sessions, and their increases in income due to the livestock program. The findings are illustrated in Exhibit 15 below. Exhibit 15. Livestock farming Livestock Question Number of respondents % of Respondents Average score (1–5) Q11: To what extent are you satisfied by receiving livestock and livestock-related trainings? 47 27% 3.3 Q24: How has your income changed after receiving livestock and livestock-related assistance? 52 30% 3.5 Q37: Engaging in livestock activities has had a positive effect on your income in the long run. 97 56% 3.9 b. Access to Financial Services for Women and DAGs The NEAT program did not provide direct financial assistance to beneficiaries of its interventions. However, it facilitated access to finance by engaging and providing training to microfinance banks and related financial establishments, which in turn expanded their branches’ presence or provided branchless banking to the communities within which NEAT’s interventions were occurring. The NEAT program worked with four financial institutions to provide regular banking as well as branchless banking. The program also worked with a local financial-training institution to provide information to target audiences to promote access to finance in the areas where NEAT interventions were occurring and in rural areas. The financial institutions that NEAT directly engaged with were Nirdhan Utthan Bank, Mega Bank, Laxmi Bank, United Youth Community (UNYC) Nepal, a financial intermediation nongovernmental organization (FINGO). The NEAT program also engaged with the Centre for Self-Help Development (CSD), a finance-training institution. All financial institutions that worked with the NEAT program found the intervention in this area to be useful and a major source for exposure and expansion. CSD submitted a proposal for conducting training within the financial industry to promote expansion of financial services and provide other tools for encouraging institutions to expand to the rural areas. NEAT experts accepted the CSD proposal and lent support on a cost-sharing basis. With the help of NEAT, CSD was able to execute the following activities:  Training of trainers (organized for local finance trainers)  Training on basics of microfinance (organized for microfinance cooperatives and FINGOs) 22 USAID Nepal. “Strengthening the Foundations for Inclusive Economic Growth.” Final Report: Nepal Economics, Agriculture, and Trade (NEAT) Activity. Prepared by Chemonics International, Inc. (Contract No. EEM-I-00-07- 00008). August 2013. 28  Training on individual/collateral-based lending in microfinance (organized for area branch managers)  An assessment of training needs (organized for microfinance cooperatives and FINGOs) With the NEAT program’s support, the training sessions that CSD organized were successful overall, with a total of 111 participants benefiting from them. CSD reported that its engagement with the NEAT program was quite productive and promoted immediate positive and long-lasting results. CSD representatives shared that they would not have been able to reach such an audience without NEAT’s assistance and believed that, because of the program, the microfinance industry is now better equipped to implement its work and promote economic growth. Services that the NEAT program provided to financial institutions included the following:  Offered financial support to the four financial institutions it engaged with  Served as a bridge between the financial institutions and the Central Bank for those that needed that access  Supported a field study to Pakistan that provided local and international access to and lessons learned from the microfinance implementation system  Provided capacity building for branch managers and other staff at the microfinance banks  Facilitated the Mobile Financial Summit The various trainings and financial support provided to the financial institutions led to the following outcomes, according to the financial institution staff members who were interviewed:  UNYC was able to open two additional branches due to the NEAT program’s provision of funding and training. As of September 2013, these branches have 3,300 customers, more than NEAT’s target of 2,500.  UNYC also shared its ability to introduce new products as a result of training provided by the NEAT program and access to new areas that required innovative products, such as seasonal loans.  Mega Bank opened 3 branchless banks with the financial support provided by the NEAT program, totaling 48 branchless banks in the country.  Nhanirdhan Utthan Bank opened six additional branches. As of September 2013, these branches have 5,393 customers, more than the NEAT-established target of 5,280. The bank also shared its ability to introduce new products as a result of training provided by the NEAT program and access to new areas that required innovative products. In response to the question of whether the NEAT intervention had increased access to financial services for women, rural populations, and DAGs, all the financial institutions responded positively, agreeing that the NEAT intervention made an impact. Most banks indicated that their targeted recipients were women, with most registering nearly 100 percent female customers and the others having a minimum of 30 percent female customers registered. The banks were also able to reach approximately 80 percent of the DAG populations within the targeted regions. Within these areas, one bank noted that 50 percent of the people forming the DAG conducted at least one transaction every month, which is a positive indication that individuals with more access to financial institutions are willing to utilize their services. Another noted that as a result of area farmers’ increased and readily available access to financial institutions, the farmers had also increased their savings with regard to transportation and travel costs, leading to an 29 increase in overall production. Residents within these areas are now involved in different income￾generating activities, which will invariably lead to better social statuses, such as the ability for DAG children to attend schools. The microfinance training institution noted that with the help of NEAT interventions, the presence of microfinance cooperatives and institutions has grown rapidly in these rural areas. The evaluation team also sought input from beneficiaries regarding the services they received from these financial institutions and learned that 49 percent of the farmers interviewed took microfinance loans with institutions in their areas. When asked to rank whether “[t]he process for receiving a loan was easy and reasonable” on a 5-point scale, with 5 being “strongly agree” and 4 being “mildly agree,” farmers within the food-security, vegetables, tea, ginger, and lentils subgroups provided scores of 4.6, 4.5, 4.4, 4.5, and 4.2, respectively. Although participation varied among food-security groups and the four value chains, on average, more females received loans than males, an equal number of non-DAG and DAG participants received loans, and more farmers from the Terai received loans than those from the Hills. It did not appear that women and DAGs faced discrimination when obtaining loans. Exhibit 16 below shows beneficiary groups’ access to microfinance in percentage terms, disaggregated by gender and DAG status. Exhibit 16. Percentage of NEAT beneficiaries accessing microfinance 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Food Security Vegetables Tea Ginger Lentil Males Females DAG Non‐DAG 30                                                              Food Security Vegetables Tea Ginger Lentil All Males Females Non‐DAG DAG Hills Terai 47% 48% 46% 52% 40% 42% 54% 42% 30% 64% 40% 50% 44% 39% 46% 52% 36% 60% 29% 46% n/a 52% 54% 51% 44% 58% 52% n/a 57% 41% 77% 43% 60% n/a 57% Respondents were also asked about their satisfaction levels with the process for receiving loans and whether they would be inclined to take future loans. Exhibit 17, below, provides the findings: The farmers responded positively about the ease and reasonableness of obtaining the loans, and 78 percent of all farmers who had received loans planned to obtain loans in the future. Exhibit 17. Satisfaction with obtaining a loan and likelihood of future participation Food security Vegetables Tea Ginger Lentils Average Agreement with the ease and reasonableness of obtaining the loan (5.0 max) All 4.6 4.5 4.5 4.5 4.6 4.5 If you took a loan, do you plan on taking another loan in the future? All 62% 89% 100% 50% 87% 78% Evaluation Question 3: How has NEAT’s policy-reform work improved the business environment for private sector–led growth? One of the NEAT program’s main purposes was to provide technical guidance in the development of various Nepalese policy studies, guidelines, and acts in an effort to improve the government sector. The NEAT program was also to serve as a catalyst for improving the business environment and inspiring private sector–led growth in Nepal. Overall, the NEAT program was viewed as effective in improving the business environment for private sector–led growth. However, respondents noted that NEAT-related activities cannot be isolated in their ability to improve growth due to external factors, such as government stability and the existence of other donor agencies working within the same space.23 By the completion of the NEAT program, 40 policies and procedural reforms had been assessed, drafted, or re-drafted by NEAT staff, 9 of which had been passed and moved toward implementation. Policy reforms that contributed to enabling a better business environment included the following: 23 Note that Evaluation Questions 1 and 4 provide further examples that relate to this evaluation question, which have not been included here to avoid repetition of responses. 31  Feasibility of four new data links for the Office of Company Registrar (OCR)—Inland Revenue Department (IRD) and OCR data links passed, while the other three were submitted for presentation to legislature/executive consideration  IEA—It was prepared with U.S. government assistance and later passed/approved by the cabinet, and it is currently in the process of enforcement through ordinance  Agricultural-mechanization policy, foreign direct-investment policy, intellectual-property policy, analysis of Nepal-China trade-agreement issues, agribusiness promotion act, export-import management act/regulations, and FITTA—All have been submitted for consideration to legislature/executive groups. In general, members of the private sector found the NEAT program’s interventions to be effective. However, unlike those in the government sector, more respondents within the private sector were critical of the program’s implementation and its ability to appropriately engage members of the private sector. As mentioned previously, the NEAT program provided technical and financial support to CNI to launch an investment portal and organize PPDs on FITTA. PPDs provided the forum for productive interactions between government officials and the business community on the draft policies and acts. The investment portal provides authentic and updated information about government policies, the country’s investment climate, and features of the major project. The activities provided through the NEAT program were all prerequisites for improving the investment-enabling environment in the country, and so, according to CNI, the NEAT intervention was perceived to be highly effective in meeting its objective of creating an enabling environment in the business/industrial sector in Nepal. The FNCCI Industrial Committee shared similar sentiments about the NEAT program: From this committee’s perspective, the NEAT program’s interventions were effective and highly aligned with enabling the business environment in the country. For instance, GON had announced a new industrial policy in 2010, but in the absence of a new act, this policy had not been implemented. Similarly, a 20- year-old foreign-investment policy was not compatible with the current global situation and WTO regulations. Therefore, drafting a new foreign direct-investment policy and IEA was the government’s priority, and doing so would positively promote an enabling business environment. However, the committee shared reservations about outward investment. Similarly, the agriculture-business promotion policy was established in 1962 but was not in use due to the absence of an implementing act. The Agriculture Enterprises Promotion Centre (AEPC) had already been in discussions with MOAD to draft an act, and therefore the NEAT intervention was timely in providing the support needed to make the act a reality. Members of FNCCI’s Tax Revenue Committee also expressed satisfaction with the implementation of the Any Branch Banking System (ABBS) developed with assistance from the NEAT program. The implementation of the ABBS and the reformation of other tax-collection systems through the NEAT program were effective and have simplified the tax system. Respondents did note that ABBS needs to be expanded further, because currently only large-scale taxpayers can use this mechanism, thereby limiting its impact. FNCCI worked closely with MOC and the NEAT program to reform the customs system in Nepal. Because of these reforms, the cargo clearance system has been simplified; however, some valuation￾32 related challenges remain within the Customs Department. FNCCI representatives also believed that PPDs were most effective in enabling the business environment, because they brought business entities to the decision making table and promoted interaction with relevant government entities, thus fostering private-public partnerships. These dialogues also facilitated business growth by providing a platform for advocacy and the transfer of information from the business sector to government. This communication promoted a sense of ownership by the business community in the development and revision of policies, which will in the long run promote the sustainability of such programs. FNCCI participated in more than thirty PPDs during the NEAT intervention period, and FNCCI itself organized fifteen PPDs in different parts of the country. These PPDs, which included more than 50 participants in each session, covered such topics as the foreign direct-investment policy and IEA. Further, all policy-reform work underway was discussed during PPDs. Although the NEAT intervention was seen as useful on the whole, respondents did share some negative feedback. The Federation of Customs Agents Association, for instance, shared that although clearing goods through customs has become easier because of training provided by the NEAT program, the new system is inefficient because it increases the time and cost needed to complete clearance due to the introduction of a system that requires entering each good separately (previously, multiple goods could be entered through a single declaration form). Further, although many actors were involved in the NEAT interventions through the Customs Department, private-sector participation in the reform program was minimal, with interventions focusing mostly on government officials and entities. For instance, the Nepal Freight Forwarders Association, which is the most knowledgeable and experienced organization among private-sector organizations when it comes to customs reform, was not consulted. Respondents shared that excluding the business sector entirely could cause challenges in promoting the sustainability of programming, particularly because the government tends to have high staff turnover rates. The Seed Entrepreneurs Association of Nepal (SEAN) shared that the existing seed policy was restrictive, and, as a result, entrepreneurs in the seed industry faced challenges with importing improved and hybrid seeds into the country. At the same time, due to geographic proximity and open borders, low-quality hybrid seeds were being smuggled into the country, adversely affecting production as well farmers. They were therefore satisfied with the new seed policy, which directly addresses this problem and creates a liberal environment for seed importation from India and other countries. Because the private sector primarily handles the production and importation of seeds, changes in regulations would positively affect the business environment in this sector. 33 Evaluation Question 4: How appropriate and effective was the program in improving GON systems and capacity for economic growth?  Taxation, revenue, customs, and trade The various stakeholders interviewed perceived several components of the NEAT program as effective. The NEAT program provided several training activities for government entities, which yielded opportunities to improve existing practices or introduce participants to new ones within the agriculture, technology, finance and revenue, trade and investments, and customs industries. The overall response to the NEAT program’s effectiveness in building capacity within the organization was positive, with examples of provided activities that had yielded intended and unintended positive outcomes. According to respondents at MOC, the NEAT program provided a variety of capacity-building activities, including studies, training, and restructuring of the Trade Advisory Committee. NEAT consultants also assisted in strengthening the Trade Policy Analysis wing, which works with the Planning Division and has gained valuable insight on how to effectively analyze trade data. Realizing the importance of the Trade Policy Analysis wing, MOCS has committed to funding this unit to ensure its sustainability. The study on the trade and investment framework agreement (TIFA), an agreement with the United States focusing on Nepal-India trade and Nepal-China trade, was particularly useful for the ministry, offering valuable recommendations gathered through the studies. As a result of this exposure, MOCS is preparing a trade￾facilitation master plan to holistically guide activities that will foster trade and economic growth in Nepal. Within this ministry, the trade-negotiation training sessions were viewed as most useful, although a respondent noted that the 2-day session should have been expanded over a longer period of time. MOCS also pointed out that NEAT consultants’ involvement in the preparation of import/export code and a WTO reference booklet in English was quite helpful. In recognition of the document’s usefulness, the ministry also translated it into Nepali. Additionally, study reports gathered through the various training sessions are serving as good reference guides for enhancing sustainability of programming. For instance, based on some of the recommendations in one of the trade studies, the government has asked China for duty-free market access to Nepalese products. Information gathered through interviews suggests that China is ready to provide duty-free access for more than 4,000 Nepalese products, and a full implementation of this agreement is in progress. Essentially, this ministry found the trainings and studies to be very useful and has taken ownership in implementing ideas that were learned. The web-based databank established in the Trade and Export Promotion Centre (TEPC) has been yet another useful intervention from the NEAT program, according to respondents from MOCS, TEPC, and FNCCI’s Trade Committee. It has made available a wide range of trade data within short 2-month timeframe. OCR respondents shared similar sentiments regarding the NEAT program’s interventions. Through the program, this institution was able to strengthen its information technology (IT) infrastructure (increased server space) and technical know-how among its staff. If staff numbers at the institution are maintained, barring the frequent turnover of staff, it is believed that the institution will be able to sustain its lessons learned. Respondents from IRD shared that NEAT interventions covered data automation, institutional capacity building, and preparation of guidelines, which have served as useful reference tools for 34                                                              increasing capacity in the department. Capacity building through the automation of processes at the department and training programs provided to staff have also helped ensure the institution’s sustainability. IRD participants considered the establishment of the Revenue and Forecasting Unit in the department as a one of the most useful NEAT-related interventions. In recognition of the importance of this unit, the department has made budget provisions for this unit from its own finances. Respondents at MOAD shared that the NEAT intervention was particularly effective because the activities took place at the policy (implemented at the central level) and operational levels (implemented at the district level). The NEAT program provided support for several study tours, including one to India and Bangladesh for high-level consultation and observation of a contract-farming system and agriculture mechanization. At the district level, the NEAT program provided many training opportunities to farmers, value-chain actors, and social mobilizers, including insights about seed quality and its importance as well as market information about their products. The program also provided technical support to agricultural offices at the district levels, which enhanced their capacity to identify improved methods for effectively working with farmers. Key Evaluation Question 5: How has the program enhanced the competitiveness of the selected agriculture value chains (lentils, ginger, vegetables, and tea)?  Production, productivity, processing, and exports According to its final report, to enhance the competitiveness of selected agriculture value chains, the NEAT program focused on a “push-and-pull approach,” conducting the following activities:  Working with farmers and enterprise-level producers to push a high-quality product to the market and on the demand side to increase uptake  Creating linkages between producers and buyers  Building the capacity of actors throughout the value chain  Training farmers in improved production practices24 As stated in the final report, one of the program’s priorities was to increase the competitiveness of farmers in the selected agricultural value chains of vegetables, tea, lentils, and ginger. Several programs were provided to farmers within the program’s targeted region to improve production, productivity, processing, and exports. This section reports the findings by program area, such as training in good agricultural practices (GAP), improved market access, better seeds, improved quality through post-harvest handling, and better efficiency with collection centers for each of the four value chains within the interventions instituted by the NEAT program. 24 NEAT Activity Final Report (June 2013), p. 25. 35 Vegetable Value-Chain Findings The major efforts to aid vegetable farmers included training on establishing market linkages and improved accessibility, building collection centers, and improved seed quality. All efforts were reported as being successful and are detailed subsequently. Increased vegetable sales from participation in the NEAT program: The average annual increase in vegetable sales after participating in the NEAT program was 44 percent for all participants. Female and DAG participants had a higher percentage increase, reporting 87 percent and 114 percent, respectively, but their base sales in the year prior to participating in the NEAT program were less than those of their male and non-DAG counterparts. These findings are provided in exhibit 18, below. As for differences within the vegetables value chain among the three subgroups, male farmers had more sales than female farmers, non-DAG farmers had greater sales than DAG farmers, and farmers in the Hills had greater sales than farmers in the Terai. Exhibit 18. Total sales for 1 year (spring and fall crops) for vegetables (Nepalese Rupee), prior to and during the NEAT program Year prior to NEAT Year during NEAT1 % Increase in sales All 56,513 81,245 44% Male 68,838 92,362 34% Female 29,949 56,061 87% Non-DAG 60,097 82,824 38% DAG 29,477 63,025 114% Hill 78,789 109,542 39% Terai 21,311 31,597 48% GAP training for vegetable growers: Vegetable farmers were satisfied with the training they received through the NEAT program. Training topics included the use and planting of seeds; correct use of fertilizer, compost, and pesticides; grading and packaging; market linkage and improved accessibility; and business skills. Each was evaluated separately and scored more than 4.0 on a 5-point scale. Exhibit 19, below, illustrates the effect of these “good management practices” for the vegetables value chain. All five categories received positive evaluations. Even though the differences were slight, the responses did indicate that the greatest challenges to such programs as NEAT are increasing sales and improving price. As previously stated in the report, some farmers had suggestions on how to improve similar programs in the future, but even when these suggestions are taken into account, the farmers’ overall evaluation was still positive. 36 Exhibit 19. Did the trainings on “good agriculture practices” for vegetables improve your practices on any of the following components? All Farmers 4.2 4.1 4.0 3.9 3.8 3.7 3.6 All Farmers Increased market access and improved vegetable quality through post-harvest handling: Assistance and support for market linkage and accessibility helped farmers improve their access to inputs and markets, post-harvest handling, total vegetable sales, and average price (see exhibit 20, below). However, average price was slightly lower. Traders and processors interviewed stated that the quality of vegetables had improved. More specifically, one trader explained that since the intervention, many farmers had begun separating the vegetables into three grades, which is important and a good practice to implement. Another trader shared that the training in fertilizer application had improved the quality of the vegetables because of changes in usage practices; before the intervention, fertilizer was randomly applied, which caused uneven production, but that is no longer the case. Although these are only anecdotal reports from two traders, these statements are presented as example of how vegetable quality improved because of the NEAT program. Exhibit 20. Improvements to vegetable farming due to assistance and support on market linkages and accessibility Access to inputs Access to markets Post-harvest handling Total vegetable sales Average price All 4.1 4.0 4.2 4.0 3.8 Increased market access and efficiency through collection centers for vegetables: The construction of new collection centers and the repair of old ones were perceived quite positively, as shown in exhibit 21, below. Most of the participating farmers had struggled with poor access to markets, and therefore having a common area for receiving their production was very helpful. They ranked highly the savings in time associated with these centers. Additionally, because of having a place to sell, the farmers’ production, productivity, quality, sales, and income all increased. 37 Exhibit 21. Impact of NEAT’s intervention on constructing new collection centers or repairing existing ones Time-saving Productivity Quality Sale Income All 4.7 4.4 4.4 4.1 4.4 For the vegetables value chain, farmers had the following suggestions regarding market linkages:  Four farmers (6 percent) would like sorting and grading machines for their villages.  The collection centers were very helpful for improving marketing, but not all farmers had easy access to them. Nine farmers (15 percent) would like collection centers to be located nearer their villages. Typical of these comments, one farmer said, “Future projects should offer assistance to construct the collection center nearby the village.” Improved vegetable production with improved seeds: Another intervention was the introduction of improved seeds (non-hybrid) and improved seeds (hybrid). The average response to the question “After training on correct use and planting of vegetable seeds, to what extent has the use of improved or hybrid seeds on your farm, compared to non-improved or traditional seeds, changed?” was very positive, scoring 4.7 on a 5-point scale. Improved vegetable production through irrigation: Many farmers received training and assistance in establishing on-farm sources of water and irrigation systems for their vegetable production. Irrigation is important to vegetable farmers, and 13 vegetable farmers (22 percent) suggested that future projects offer more help with irrigation. According to their responses, farmers in the Terai experienced more irrigation problems than Hill farmers did (8 farmers versus 5 farmers). Food-security intervention farmers in the Terai also had more problems with irrigation. Tea Value-Chain Findings In general, stakeholders within the tea value chain were satisfied with the NEAT program’s interventions. However, their perceptions about the program generally ranked lower than those of other value-chain participants. As previously mentioned, the tea growers were dissatisfied with the intervention’s implementation approach, viewing it as “top-down” and lacking true stakeholder participation. Increased tea yields from participation in the NEAT program: Stakeholders in the tea value chain realized increases in yield because of the NEAT program. Exhibit 22, below, shows the increases in yield (kg/hectare) attributed to the NEAT program for organic and non-organic tea. The percentage increases in yield were greater for the organic growers, even though they were starting from a lower base. Nearly all the organic growers were already growing organically before the NEAT program began, and these growers were satisfied in their decision to make the change to organic production (4.0 on a 5-point scale). The organic farmers were also satisfied with the process of obtaining organic certification for tea (4.4 on a 5-point scale). Like producers of other organic crops, growers of organic tea had greater problems with yield and production, but the increased price helped them with improving sales. 38 Exhibit 22. Comparison of yields (kg/Ha) for organic and non-organic tea, before and during participation in the NEAT program Non-organic Organic Average yield (kg/Ha) % Increase in yield Average yield (kg/Ha) % Increase Prior to in yield NEAT years During NEAT years Prior to NEAT years During NEAT years All 349 406 17% 147 235 60% Male 353 397 13% 130 230 77% Female 309 363 17% 188 243 30% Non-DAG 357 386 8% 195 352 80% DAG 215 246 15% 78 131 67% GAP training for tea growers: According to respondents within the tea value chain, training programs were quite effective in helping farmers improve the quality of their organic and orthodox tea. However, according to farmers, one of the weaknesses of the training program was its lack of frequency. In most cases, training programs on improved methodologies, such as plucking and pruning, integrated pest management (IPM), organic fertilizers, nursery trainings, and vermin composting, were given only once during the entire course of the NEAT project’s duration. The tea farmers suggested that training sessions would have been more beneficial had they been provided more frequently, especially taking into consideration the annual cycle of tea-cultivation seasons. In addition, women were less satisfied than men were with these training sessions, as illustrated in exhibit 23, below. Exhibit 23. Tea growers’ satisfaction with usefulness of training for organic and orthodox tea Integrated pest management (IPM) Vermi-compost technology Management of cowsheds Improved pruning and plucking techniques Organic tea training All 4.2 3.9 4.2 4.3 Male 4.1 4.0 4.1 4.3 Female 3.9 3.5 3.3 4.2 Orthodox tea training All 4.0 4.0 4.1 4.2 Male 4.0 3.9 4.0 4.1 Female 3.7 3.8 2.7 4.1 Increased market access and better efficiency with collection centers for tea: Farmers largely viewed the repair of old collection centers and the construction of new collection centers positively. In particular, the savings in labor that farmers experienced by delivering their tea to a common location was significant. However, the value of the market linkage and accessibility programs was viewed less favorably. In both cases, female farmers were less positive than their male counterparts (see exhibit 24, below). On the other hand, the traders agreed that the quality of the raw leaf that they were purchasing had increased because of the NEAT program’s interventions. In the districts where tea production occurs, market infrastructures, such as collection centers, were found to be appropriately located across program areas. The construction 39 5.0 quality for collection centers visited was found to adequate as well. However, not all collection centers were being used as per the NEAT program’s expectations. For instance, in Gopetar Village Development Committee (VDC) in Pancthar and Sankhejung VDC in Ilam, the collection centers were not being used for weighing or collecting tea leaves before sale; one center was locked and deserted, while the other was being used only for meetings and village gatherings. Exhibit 24. Did repairing or building new collection centers for tea have an effect on the following components? Time Saving Productivity Quality Sale Income 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 All Male Female Exhibit 25. The effects of the assistance and support provided on “marketing linkage and accessibility” for tea, by component 2.5 2.6 2.7 2.8 2.9 3.0 3.1 3.2 3.3 Inputs Markets Post‐Harvest Sales Price All Male Female 40 Lentils Value-Chain Findings Increased lentil production because of participation in the NEAT program: Many lentils value-chain beneficiaries commented on the region’s excessive rain and the detrimental effect it had on lentil yields. Yet even with these challenges, an increase in production was realized, with the average increase in production per farm being 23 percent. However, women’s lentil yields actually declined during that period, while men’s yields increased by 85 percent. DAG farmers had considerably lower percent increases in yields than did their non-DAG counterparts, a figure that is detailed in exhibit 26, below. Exhibit 26. Comparison of average farm production (kg) for lentils before and during participation in the NEAT program Lentils Production (kg) % increase in During NEAT yield years Prior to NEAT years All 287 233 23% Male 319 173 85% Female 252 309 -19% Non-DAG 364 176 107% DAG 286 258 11% GAP training for lentil farmers: When lentil farmers were asked about their perceptions of the effectiveness of GAP training sessions for their value chain, they generally responded positively, with an average rating of 3.88 on a 5-point scale; that said, and as mentioned previously, these responses were among the lowest across the participating value chains. Exhibit 27, below, provides the breakdown of responses. Exhibit 27. Improvements in lentil-growing practices due to GAP training Total farm production Total farm sales Post-harvest handling Quality of farm production Average price All 3.75 3.75 3.95 3.95 3.88 Exhibit 28, below, provides the opinions gathered from lentil growers concerning their satisfaction with the NEAT interventions introduced to them. The findings concerning these initiatives are discussed in more detail subsequently. 41 Exhibit 28. Satisfaction level regarding the assistance of the NEAT program’s initiatives NEAT initiative Satisfaction level (max. 5.0) Improvement in usage of improved or hybrid seeds compared to non-improved or traditional seeds 4.38 Selling of bulk products 3.46 Drying and processing, packaging, and low-cost storage construction 3.21 Time savings from the construction of new collection centers 4.58 Improved quality from the construction of new collection centers 3.84 Improved market access for inputs, better post-harvest handling, and more collection centers for lentils: The evaluations for improvements due to market linkage and accessibility were positive among lentil farmers. The highest score was for improved access to inputs, a sentiment shared by an agrovet who sells inputs to lentil farmers. He stated that because of the intervention, the farmers were more aware of the need to use quality fertilizers, such as nitrogen, potash, zinc, and di-ammonium phosphate (DAP), in their fields. He also positively noted that sales had increased for improved seeds and quality pesticides. Lentil growers shared their overall satisfaction with the NEAT program’s interventions, as shown in exhibit 29, below. As with other programs, the lentil growers noted the savings in time associated with the collection centers. However, many of the traders and processors interviewed in the lentils value chain said that the quality of lentils had not improved very much. The responses by farmers concerning such NEAT program initiatives as selling in bulk, drying, processing, packaging, low-cost storage, and improvements in quality from the collection centers supported the traders’ views that these areas require additional work, noting that farmers need more help with sorting and grading lentils to ensure a quality product. Exhibit 29. Improvements due to market linkage and accessibility for lentils All Farmers 4.10 4.00 3.90 3.80 3.70 3.60 All Farmers 3.50 42 Improved lentil production through improved seeds: The lentil farmers noted the value of the improved or hybrid seeds. As mentioned previously, challenges due to excess rain developed during the NEAT program’s duration. Additionally, plant diseases in lentils also had a detrimental effect on yields, with farmers reporting that traditional varieties seemed to be inherently disease-resistant. Farmers were content with the improved seeds but sometimes faced challenges in trying to ensure that the new seeds successfully adapted to the locality in which they were being grown. Diversification of lentil production to include vegetable production: Many of the lentil farmers reported that they would like to diversify their lentil production and grow more vegetables for home consumption and sales. It appears that the distinctions between food-security programs for poor farmers and value-chain programs for commercial farmers may not have been that different. Ginger Value-Chain Findings Ginger farmers generally reported satisfaction with the training provided, as indicated above. However, they wanted more of it and would have liked the NEAT intervention to have lasted for a longer period of time. According to the beneficiaries, it was difficult to incorporate what they had learned in such a short time. Diseases were an overarching challenge within the ginger value chain, with farmers and NEAT instructors reporting that a solution had not been identified for disease control; despite this issue, however, farmers did realize some yield increases and noted overall benefits of the intervention. Increased ginger yields from participation in the NEAT program: Ginger yields increased by 10 percent. Yields for women and for DAG farmers were lower than for their counterparts in years prior to NEAT participation and continued to remain lower after participation in the NEAT program, but the percentage increases were similar across groups, as illustrated in exhibit 30, below. Exhibit 30. Comparison of average farm yield (kg/ropani) for ginger before and during participation in the NEAT program Ginger Production (kg/ropani) % Increase in Prior to NEAT yield years During NEAT years All 273 301 10% Male 327 374 15% Female 238 256 8% Non-DAG 278 314 13% DAG 269 293 9% GAP training for ginger farmers: All the ginger traders and processors interviewed thought that the quality of ginger had increased because of improved grading, sorting, cleaning, drying, and packing. However, the farmers shared numerous complaints about disease-related problems with the ginger crops. Some of the farmers thought that they resulted from inferior seeds, while others suggested the excessive rain caused the disease problems. Ginger farmers rated their new skills in grading and packing ginger and the resulting increase in quality of farm production more highly than improvements in total farm 43 production (see exhibit 31, below). This lower ranking for total farm production may be due to the effect of disease. Like other farmers who participated in the NEAT program, ginger farmers reported that growing more of the product and improving post-harvest handling did not necessary lead to higher prices (or at least the level of price increases that would match farmers’ expectations). Exhibit 31. Improvements in ginger-growing practices due to GAP training All Farmers 3.50 3.60 3.70 3.80 3.90 All Farmers Ginger seeds and disease: A major concern raised by ginger farmers was the fact that much of their crop suffered from diseases. They argued that the seeds given to farmers were of poor quality. Farmers expressed that NEAT experts should have put more time and resources into trying to understand and eliminate the diseases that affected their crops and provide more training on disease management, which would have been beneficial to farmers. Farmers were appreciative of the seeds provided, but if the seeds are not suitable for local production, then eventually, as one farmer said, they will go “back to the local seeds.” Several farmers mentioned that “medicines” were promised to eradicate diseases, but medicines “didn’t arrive in time,” so many ginger farmers seemed to have suffered at the hands of diseases and lack of follow-through. More specifically, ginger farmers offered these suggestions for future programs:  Eleven farmers (17 percent) wanted help in managing disease problems in ginger. Specific comments included the following:  “The seeds provided to [us] were a total waste because they caught diseases. Now [we] have reverted to local seeds.”  “We couldn’t get rid of disease despite trainings.”  “There was no solution to all the diseases. Medicines didn’t arrive in time. The project didn’t complete promised works. They [ginger trainers] left abruptly.” Improved market access and better quality through post-handling for ginger: As with other farmers, who share problems of isolation and poor roads, the farmers raising ginger valued improved access to markets. Gaining inputs was deemed nearly as valuable as selling products more easily due to this increased linkage. The challenge, however, as with changes due to GAP training, was to improve farm 44 sales and ginger prices. Farmers rated their satisfaction with improved ways of grading and packing ginger only modestly, although, as mentioned previously, the traders reported that the quality of ginger was improving. Farmers also gave only modest ratings to their satisfaction with contract-farming lessons. It should also be noted that the majority of the ginger farmers interviewed did not participate in contract farming; 21 percent of the farmers were selling to Annapurna Organic. Exhibit 32, below, provides findings on satisfaction with market linkages and accessibility, while exhibit 33 subsequently reports the overall satisfaction level with the NEAT program’s interventions. Exhibit 32. Improvements to ginger farming due to assistance and support with market linkages and accessibility Total farm production Total farm sales Post-harvest handling Quality of farm production Average price All 3.98 4.13 4.04 3.78 3.69 Exhibit 33. Satisfaction level with the assistance received from the NEAT program’s initiatives NEAT initiative Satisfaction level (max. 5.0) Grading, sorting, drying, and packaging ginger 3.51 Selling ginger through contract farming 3.46 Percentage of farmers selling to Annapurna Organic 21% Evaluation Question 6: What are the prospects for sustainability of the end results produced by the NEAT program? a. Sustainability—Enabling policy and business environment Most stakeholders interviewed agreed that the NEAT program’s approaches were effective and in line with government priorities. This alignment is important to promote sustainability and helps determine whether guidelines and acts will be implemented. To truly promote economic growth and improve the business environment, it was important that the activities promoted by the NEAT program be appropriately aligned with GON priorities. Most respondents from MOAD shared that NEAT program activities implemented within their ministry were highly aligned with government priorities. For instance, drafting a new agri-business promotion act that includes provisions for contract farming and drafting an agriculture mechanization policy are important components for agricultural development and rank highly on the government’s agenda. Similarly, according to MOI, the existing FDI policy was more than 2 decades old and needed updating; as such, the NEAT intervention was timely in spearheading this effort. Similarly, with Nepal having made 45                                                              several commitments to WTO, new policies or revisions to existing policies were required to make the country WTO-compatible; the government had already announced its industrial policy for 2010, but its implementation had been stalled. By helping update and develop the supporting act to enhance implementation for the industrial policy, the NEAT intervention was very timely in this sector and aligned with government priorities. Similarly, although participants agreed that selected areas of the NEAT program were definitely useful, a representative from MOF believed that most of the implemented activities were supply-driven and fell outside the priorities of GON agencies and departments. Another respondent expressed concerns about the NEAT program’s primary focus being mainly on production. It was also argued that a large portion of programs included only software components, and although software support is essential, it is not sufficient on its own to commercialize agriculture. What is equally important—but missing from the NEAT program—is support for hardware components, particularly for post-harvest activities (e.g., cold storage, rope way for transportation of agricultural products, processing plants, establishment of market collection centers, etc.). This element is essential and as such should be included in interventions of future projects.25 b. Sustainability—Food Security and Competitiveness Although the NEAT program benefited its targeted audience in many ways, as indicated throughout the report, farmers and other stakeholders expressed concerns regarding sustainability of the food-security and competitiveness components after the program’s completion. These are discussed below. Stakeholder engagement: Farmers who participated in the NEAT program’s interventions for food security and the four value chains appreciated the overall program. However, they had a number of concerns that could affect the sustainability of a future project of this nature. As mentioned previously, respondents were concerned about the lack of local input in the program’s development, and a major component of sustainability is getting buy-in from the target audience. This buy-in seems to have been missing, based on the responses shared by some beneficiaries. A number of respondents, particularly within the ginger and tea value chains, thought that the NEAT program’s approach was not the most suitable, because the locals’ input on what they actually needed was never sought. This situation affected the overall interest levels among the wider population with regard to the project. A number of respondents expressed that for a project to work, “[planners] must first know what the people want.” A large portion of the surveyed stakeholders shared that other issues were more suitable for interventions, such as poor road conditions, lack of necessary equipment, and lack of transportation, which they would have expressed had they been contacted at the beginning of the project. The NEAT program’s sustainability was also hindered by its lack of focus on training local agricultural development officers who could continue the project’s work in the future, a sentiment was shared by DADOs and program beneficiaries. Most of the extension work was accomplished by 25 The evaluation team notes as previously stated that provision of hardware inputs is outside the NEAT SOW, and moreover not all aspects of promoting agricultural development could be tackled, particularly given the limited timeframe of the contract. However, the sentiment is included since it does respond to sustainability of such programs. 46 subcontractors/implementers, who hired their own teams of professionals to work with farmers. Many of the materials developed by NEAT experts were provided to agricultural extension officers, who valued the information. However, a key distinction is that although the NEAT program may have informed agricultural extension workers, thus earning their support, agricultural extension was not an active component of NEAT’s programs. A designated “train the trainer” program for extension workers would have required extra time and resources, but it could have helped enhance the program’s overall sustainability. For example, most DADO officials interviewed reported that they were not involved with the NEAT project’s implementation; in most cases, NEAT staff members visited DADOs during the initial phases of selecting project sites, VDCs, and villages for program implementation, but after this point, DADOs and their staff were not involved in the project in any shape or form. Although working directly with government agencies has its limitations and hurdles, it is nonetheless important to achieve a balanced engagement with related government institutions, such as MOAD, DADOs, and DDC. The use of government guidelines, registration of program farmer groups with DADOs, joint monitoring visits, and inclusion in DDC annual programs could have been effective in strengthening the NEAT program as well as ensuring the project’s long-term sustainability. Training: As mentioned previously in the Findings section, farmers largely valued the GAP training they received, although they also had suggestions for how to improve the training. However, one significant suggestion they had likely affects sustainability: The NEAT program’s duration was too short, and given the allotted time span, the farmers’ classes were infrequent. Educating farmers about better ways to farm and feed their families is a complex topic. In addition, there is always a lag between training and adopting, because farmers need to learn to first trust the new educators/implementers. Projects that are focused on encouraging agricultural development among poor farmers need to be at least 3 years in length. 47 IV. CONCLUSIONS AND RECOMMENDATIONS Conclusions Enabling Policy and Business Environment Overall, the NEAT program was successful in its attempt to enable a better policy and business environment through its many activities with the public and private sectors.  Regarding stakeholder engagement, given the program’s vastness and the number of stakeholders to be reached, the program was effective in its ability to identify the relevant key stakeholders in the government and private sector, MOF and FNCCI, respectively—to participate in initiation activities. Although business-sector stakeholders made comments regarding their desire for increased participation, the evaluation team notes that identifying key stakeholders who represent relevant communities is the most effective (and sometimes the only) way to move forward with such large-scale projects.  PPDs were unanimously found by members of the public and private sectors to be the best way to encourage stakeholder engagement and gain buy-in for various policy changes.  The duration of the NEAT program was short, which affected the ability to truly measure its performance, particularly on the policy side. It is evident that progress with policy reforms was highly attributed to the NEAT program’s involvement, which ultimately will help foster improved policies and a stronger business environment in Nepal. However, policy reform and its translation into practice tends to take several years, so a short program could mean that efforts made to push for reform during the program’s duration are not always continued upon the program’s completion, even if measures of sustainability (including local stakeholders) are incorporated, because priorities tend to change with time.  Members of the business community noted general improvements within the business environment as a result of the NEAT program’s interventions and resulting policy reforms. They also discussed challenges regarding people’s capacity to translate some of the changes proposed into reality (although this issue is outside the NEAT program’s control); it is possible that a longer intervention period could be helpful in increasing the amount of training participants receive to ensure that changes made can be implemented effectively.  Concerning improvements within the GON, particularly in building capacity toward promoting economic growth, the NEAT program was very effective. GON respondents across participating ministries were pleased with the training sessions provided and shared that they would continue to use the tools provided to them even after the program ended. Food Security and Four Agricultural Value Chains  The NEAT program’s interventions for food security and the tea, ginger, lentils, and vegetables value chains were successful.  The program’s 2.5-year timeframe was too short. Many of the participants wanted a longer program, more similar to the original 5-year program that was approved. Farmers stated that they were just beginning to understand the changes needed to improve their agricultural production 48 when the program ended. More training classes and time were needed to ensure the successful adoption of these new technologies and methods on their farms.  Farmers’ need to improve food security by diversifying from staple-crop production to vegetable production is likely similar to the needs of farmers in the vegetables value chain who want to improve their vegetable production. This similarity suggests that programs designed by USAID to improve food security and those that support the commercial vegetables value chain could, in certain circumstances, be compatible and thus combined to increase the efficiency of program delivery.  Participants highly valued the establishment of collection centers at which farmers could deliver their products for sale. Other market-linkage efforts were equally appreciated, not only in connecting farmers to markets but also in helping them such acquire inputs as seeds and fertilizers.  Traders and processors stated that they observed improvements in the quality of agricultural products during the NEAT program. Understanding how to grade, sort, clean, and package products is important for all farmers.  Participants had problems with the NEAT program’s top-down style of operation and its lack of transparency, although NEAT experts attempted to mitigate this perception and include local people and organizations in the decision making process. The tea value-chain participants experienced the worst problems with this top-down manner of operating.  One of the NEAT program’s goals was to help farmers increase their production through the use of better seeds and fertilizer and improved cropping methods. Production did increase for all targeted crops, but this increased production also brought increased disease pressure. Plant￾disease diagnosis and treatment should be included in future agricultural development programs.  NEAT experts largely bypassed the government infrastructure of DADOs and extension workers, choosing instead to contract with local nongovernmental organizations (NGOs) to provide agricultural extension services and training for farmers. This approach hinders program sustainability. Although the improvements made in local communities will continue to provide benefits, it would have been better for the NEAT program to incorporate local extension workers in a “train the trainer” program so that the educational/extension efforts could continue after the program’s formal conclusion. Increased Access to Financial Services Overall, the microfinance interventions were highly successful. Participating banks and organizations as well as beneficiaries found the NEAT program’s interventions to be highly useful in promoting the expansion of new products and enabling banks to broaden their reach, particularly to women and disadvantaged groups (DAGs) within the intervention areas. Building capacity to help existing microfinance institutions (MFIs) expand their work seems to be a successful way to improve access to finance. 49 Recommendations To conclude the evaluation, the LEAP team has gathered recommendations identified through the data collected. These recommendations include both lessons learned and good practices that can be used to inform future programming.  Consider splitting large multi-faceted programs: The NEAT program was large and diverse, which understandably presented size-related complications. It is therefore recommended that in the future similar programs be divided into two separate calls for proposal. In the case of the NEAT program, for instance, the agricultural and food-security program could have been split from the enabling policy and business environment component. Smaller programs would allow implementers to be more focused on all components needed to promote success in those particular areas. Enhancing Policy and Business Environment  Ensure stakeholder engagement with all key stakeholders from the onset of programming: As mentioned previously, PPDs were viewed as positive initiatives in gathering stakeholder opinions concerning the drafting of various policies, studies, guidelines, and acts. However, particularly within the business sector, some respondents expressed dissatisfaction in not being involved in decision making from the onset, stating that the programs were too government￾focused. It is recommended that planners of future programs spend time identifying key stakeholders from the various sectors and involve them in the decision making processes from the onset, particularly during the early stages. Ensuring that the main stakeholders are involved early on will be key to promoting sustainability upon completion of the program. Improving Competitiveness and Food Security  The efforts to improve food security by diversifying from staple-crop production to vegetable production were similar to the efforts to help commercial vegetable producers: This similarity suggests that other programs within USAID designed to improve food security and those that support commercial value chains could, in certain circumstances, be compatible. For example, the needs of farmers in the food-security program to diversify from staple crops to vegetables are similar to the needs of vegetable farmers to improve their production, and the needs of those two groups are similar to those of lentil farmers, who also requested training in vegetable production. Efficiencies and effectiveness might be achieved in program delivery (e.g., improving vegetable production) by combining efforts across these three groups of farmers in future projects.  Collection centers are very helpful in improving the efficiency of and saving time in farmers’ marketing efforts: Participants highly valued the establishment of collection centers at which farmers could deliver their products for sale. Other market-linkage efforts were equally appreciated, not only in connecting farmers to markets but also in helping them acquire such inputs as seeds and fertilizers. These initiatives should be included in future programming. 50  Subsidies for seeds and fertilizer can be helpful for farmers involved in the food-security program: There is an open question regarding whether these subsidies will positively affect farmers’ continued purchase of seeds and fertilizer after the program ends. It would be useful to survey those farmers who received subsidies to determine whether they have continued making these purchases. In addition, it would be important to determine which criteria, such as access to input markets, income level of farmers, or irrigation, were most critical in making this decision.  Future training programs for farmers should give a higher priority to controlling plant diseases: One of the goals of the NEAT program was to help farmers increase their production through the use of better seeds and fertilizer and improved cropping methods, and production did increase for all crops. However, with this increased production came increased disease pressure, for the two are often linked. Plant-disease diagnosis and treatment should be included in future agricultural development programs.  Agricultural development programs that focus on capacity-building initiatives with farmers should ideally occur over a long period of time to ensure effectiveness: Throughout the evaluation process, beneficiaries and key stakeholders expressed the concern that the intervention was too short at 2.5 years. Many of the participants wanted a longer program more in line with the 5-year program that was originally planned. Farmers stated that they were just beginning to understand the changes needed to improve their agricultural production when the program ended. More training classes and time were needed to ensure the successful adoption of these new methods and technologies. Further, the design, implementation, and installation of irrigation on famers’ fields were difficult. Sufficient time should be allocated in future projects to ensure that the irrigation is working properly and the farmers are instructed on how to use it.  To improve the sustainability of agricultural development projects, there should be a greater inclusion of all key stakeholders, but particularly local DADOs. Findings from this evaluation show that it is not sufficient to provide local agricultural development officers with educational materials that were developed for use by at least some of the stakeholders. In addition, these local agricultural professionals should receive intensive training, possibly being included in farmers’ training workshops with the implementing partners, because these local agricultural professionals need to be able to continue the agricultural training of farmers in the future. There were also problems with the NEAT program’s top-down manner of operation and its lack of transparency. NEAT experts tried to mitigate this perception by including local people and organizations in the decision making process, but this improvement was intermittent, and some value-chain participants reported more problems than others over the life of the project. Overall, program planners should make considerable effort to share program goals and strategies with local partners and seek their input. This is an important first step to gaining trust and buy-in, which helps ensure the sustainability of such programs. 51 Increasing Access to Financial Services  Provide access to finance opportunities through existing establishments: Although the NEAT activity did not directly provide microfinance loans, it facilitated training and capacity development of existing MFIs to expand their services to NEAT program beneficiaries, particularly women and DAGs. A first success of this activity was that the NEAT program did not actually provide loans to beneficiaries, but provided capacity development (both in funding and through trainings) to existing establishments to promote this work. Having already established entities take on the role of providing finances to NEAT beneficiaries meant that the NEAT program did not need to take on this burden; it also signaled the sustainability chances of the programs provided by the banks and MFIs, given that by the time the NEAT program ended, they had already incorporated such efforts into their general activities.  Continue to provide access to finance initiatives for similar programs, as this promotes overall program success and economic growth: Based on the information gathered, the NEAT program offered successful training programs, even to the point that some MFIs stated that they started new program development as a result of the training and access to finance they received. Beneficiaries shared this level of satisfaction. Ultimately, helping the farmers access finance enables them to leverage other aspects of the program, such as the purchase of seeds or additional equipment, which then improves their farming practices. The initiative was also able to promote economic growth overall, because residents within the targeted areas reported current involvement in different income-generating activities, which will invariably lead to better social statuses, such as the ability for DAG children to attend schools. 52 53 V. APPENDICES Appendix 1: USAID Evaluation Statement of Work for the LEAP Team NEAT FINAL EVALUATION SOW 1. INTRODUCTION: This statement of work (SOW) is for the final evaluation of USAID/Nepal’s Nepal Economic Agriculture and Trade (NEAT) Program. NEAT is a two and half year, $22.6 million program designed to: strengthen the foundations for rapid, sustained and inclusive economic growth; enhance food security; reduce poverty; and improve lives. This multifaceted program aims at fostering a conducive business environment for a private sector led growth; encouraging competitiveness and exports in selected agricultural and non -agricultural commodities or services; enhancing food security; improving trade and fiscal policies and practices to facilitate trade and increase revenues; and strengthening the microfinance institutions to increase the access of women, poor and disadvantaged to financial services. NEAT is supporting a wide range of beneficiaries that include small holder farmers, private firms, and government agencies to perform a number of important tasks for enhancing food security, increasing production, exports and sales, improving capacity, increasing government revenue and improving trade and commerce. The project started in December 2010 and is scheduled to end in August 2013. The geographic focus of this program is in 21 districts of East, West, Mid-West and Far-Western Development Region. Chemonics International is implementing the project in partnership with local organizations. This evaluation will examine the effectiveness of the NEAT interventions to promote inclusive economic growth and enhance food security; investigate intended and unintended consequences of the program; and document lessons learned and good practices that can be shared throughout the Agency to improve development learning and future programming. The scope of the evaluation is guided by the evaluation questions in Section 6. USAID/Nepal seeks the services of a qualified, international organization with expertise in monitoring and evaluating development projects to conduct this final evaluation of the NEAT Program. 2. PROJECT CONTEXT: The NEAT program was designed right after the ending of the insurgency, as the Mission realized the need for an economic growth program that addressed the key underlying causes of conflict such as poverty and governance, and thus to prevent Nepal from reentering the conflict. At that time, many conditions were not favorable for economic growth such as exports, industry in general and overall level of activity in the economy was sluggish and trending downward. Productivity levels in both agricultural and non-agricultural sectors were low in comparison to neighboring countries. Serious problems existed in the government policy and governance framework which provide the setting for the growth, and in the enablers, i.e. infrastructure, financial system and human resources. The political uncertainty and disruptive actions (i.e. strikes, extortions) were depressing investment and economic activity. While the macroeconomic setting was stable, the government setting had substantial weaknesses. Critical issues included the need to broaden the tax base; improve trade policies and practices, including complying with World Trade Organization (WTO) requirements; and ensure that the new constitution supports a private sector led market economy. Substantial weaknesses were also found in the business environment. The 54 financial system, despite its diversity and number of institutions, had major problems in policy, capacity, and reach, especially to the hill and mountain areas. Only 8 percent of the population in rural areas has access to financial services. Enterprises were hesitant to expand operations, undertake new ventures, and invest; substantial funds and persons were fleeing overseas, especially to India. The private sector was not willing to take risks associated with unfavorable conditions and political uncertainty. Seriously complicating the situation was the worsening food situation. Hence, the NEAT Program was designed to address pressures of conflict and poverty by strengthening the foundations for rapid, sustained and inclusive economic growth. This would involve substantial improvement in governance related to business. The Mission concluded that despite the difficulties and problems mentioned above, opportunities did exist for expanding the economy and generating substantial economic growth. 3. DEVELOPMENT HYPOTHESIS: The NEAT Program was designed with the hypothesis that a country’s economic growth depends on the “setting” (government microeconomic and macroeconomic policies) and “enablers” such as the infrastructure, finance and human resources and that there is a need to strengthen both in order to get a country on the path of economic growth. NEAT was designed to improve both the setting and the enablers through its five interrelated components. Interventions improving the business environment, trade and fiscal policies and practices were to strengthen the government setting. While its work on Competitiveness and Exports were expected to assist enterprise to respond to opportunities and initiate or accelerate economic growth. Microfinance was expected to improve the inclusiveness of the growth, while the activities under food security were expected to enhance food security with increased domestic supply of food grains and livestock. USAID’s Economic Growth Strategy asserts that economic growth takes place at the level of the enterprise. Increases in the production of enterprises are what drive economic growth. Enterprises produce in response to market opportunities. To take advantage of these opportunities, enterprises must be ‘competitive’, which is profoundly influenced by what government does. The competitiveness of enterprises is defined by their ability to compete in end markets. This, in turn, is affected by the business enabling environment, the capability and incentives of firms to meet end market buyers’ demands, and the productivity of the firms themselves and the industry in which they participate. Enterprise competitiveness is dependent on the ‘setting’ in which they operate and on the ‘enablers’ which provide essential inputs. The government, through its macroeconomic and microeconomic policies and governance, provides the setting -- the parameters and rules of the game which influence incentives for enterprises. Infrastructure, financial system, and human resources – also heavily shaped by government – are the enablers of economic growth; they provide needed inputs to enterprises. Both the setting and enablers are important for boosting economic growth and poverty reduction. They provide the environment for ‘enterprise competitiveness’. 4. PROJECT INTERVENTIONS AND ACHIEVEMENTS: The goal of the NEAT Program is to improve Nepal’s economic foundations to promote rapid, sustained, and inclusive economic growth that will less the potential for conflict, reduce poverty, and improve lives. NEAT aimed to address the challenges described above and to promote positive political and social change through its five components: 1. Fostering a conducive business environment for private sector led growth; 2. Encouraging competitiveness and exports in selected agricultural commodities; 3. Enhancing food security; 55 4. Improving trade and fiscal policies and practices to facilitate trade and increase revenues without distorting the economy; 5. Strengthening microfinance institutions to increase the access of women, poor and disadvantaged to financial services. These interrelated components are implemented through a series of technical activities, including but not limited to: grants, subcontracts, short-term technical assistance, procurement and distribution of goods and services, and training. The planned outcomes of the program were: 1. Improved Environment for Business Growth 2. More Competitive Agriculture Value Chains 3. Enhanced National Food Security 4. Strengthened Government of Nepal Economic Policy Platform 5. Increased Access to Financial Services 6. Fostering a conducive business environment for private sector-led growth NEAT has partnered with the Government of Nepal (GON) in strengthening economic policies. NEAT uses demand driven approaches to supporting the GON, with the main areas being: improving capacity, increasing revenue, supporting WTO requirements, and improving competitiveness and supporting the National Trade Integration Strategy. The GON’s Ministry of Finance (its Department of Custom and Inland Revenue Department); Ministry of Industry; Ministry of Commerce and Supplies; and Ministry of Agriculture Development. NEAT’s Productivity Components supported the products for domestic and export markets in ginger, lentil, tea and vegetables. NEAT adopted a value chain approach supporting input suppliers, farmers, collection center, traders and processors and access to markets. Key Project Achievements The major achievements of the NEAT Program are summarized below:  33 policy and procedural reforms have been analyzed, six of which have been passed and implementation is on the way.  71, 342 households have benefitted from the food security interventions.  US $ 8,544,238 increases in incremental sales of producers  9212 hectares under improved management practices  35,598 households are using improved technologies  US $ 1,463,957 loans disbursed in rural areas 5. THE EVALUATION PURPOSE, AUDIENCE AND USE: The primary purposes of the evaluations are to:  Assess the effectiveness of NEAT’s interventions, implementation strategies and approaches to promote rapid sustained and inclusive economic growth and enhance food security.  Identify and document good (or best) practices and lessons learned and factors that influenced program effectiveness useful for future and ongoing programs.  Examine the intended and unintended consequences of the program.  Investigate how the project impacted men and women differently. 56 With these purposes in mind, the evaluation team must tailor recommendations so that they contribute to development learning and future and existing programs of the Agency. The audience of the evaluation report will be the USAID/Nepal Mission and the Agency as a whole. The evaluation will benefit the Government of Nepal, USAID/Nepal’s implementing partners, other donors and local organizations that are planning and implementing economic growth projects. Learning from the NEAT program should also contribute to the design of the Mission’s future economic growth programs and the implementation of the food security program. The recommendations, lessons and good practices will be instrumental in informing the implementation approaches of the new KISAN project. As USAID/Nepal is developing its Country Development and Cooperation Strategy (CDCS) for the next five years, the learning from this evaluation will help improve the future programming directions. The CDCS will have greater emphasis on local capacity building. This evaluation will look at learning with respect to working with local Nepali organizations and building government capacities. 6. EVALUATION QUESTIONS: The evaluation must be designed to answer the key evaluation questions listed below. 1) How appropriate and effective were the NEAT’s approaches and interventions in promoting inclusive economic growth?  Appropriateness of NEAT’s five components, interventions, engagement with target group and stakeholders 2) How effective was the NEAT Program in improving the economic status of women and disadvantaged population?  food security, access to financial services, productive engagement in the selected agriculture value chains, nutritional outcomes. 3) How has NEAT’s policy reform work improved the business enabling environment for a private sector led growth? 4) How appropriate and effective was the program in improving GON systems and capacity for economic growth?  taxation, revenue, customs and trade 5) How has the program enhanced the competitiveness of the selected agriculture value chains (lentil, ginger, vegetables, and tea)?  production, productivity, processing and exports 6) What are the prospects for sustainability of the end results produced by the NEAT program? 7. EVALUATION METHODS: The offerer is required to propose an evaluation methodology comprising quantitative and qualitative methods. The methods must correspond to the evaluation questions and show how each method will collect information necessary to answer each question. The Offerer must describe the strategy for choosing sample population for data collection and how inherent biases will be minimized. The proposal must include a plan for analysis of both quantitative and qualitative data and explain how these analyses will help answer each evaluation question. For quantitative methods, the process must fulfill adequate statistical rigor such as mean, standard deviation and regression as applicable and data must be disaggregated by gender. Round tables and short workshops might also be appropriate for assessment and learning with implementing partners, 57 USAID/Nepal staff, NGOs, the private sector, relevant donors and the Government of Nepal. Evaluators must collect data from multiple sources. Evaluators must select the project sites, beneficiaries and activities independently for data collection. The evaluation team is required to make a presentation of its evaluation methodology to the technical team in the Social, Environmental, and Economic Development (SEED) Office and Project and Program Development Office of USAID/Nepal and finalize the methodology after incorporating their inputs. 8. PERFORMANCE INFORMATION SOURCE: USAID/Nepal will make relevant documents available to the evaluation team leader for their review which include: Statement of Work, Project PMP and Work plans (Years 1-3), Activity Approval Document, Annual reports, quarterly reports, monthly reports, accrual reports and success stories These documents will be made available at the start of the assignment. Stakeholders include implementers as well as direct and indirect beneficiaries:  Beneficiaries  Community members  District Advisory Committee  VDC Secretaries of selected VDCs  Local Development Office (LDO)  District Agriculture Development Officer  District Officers of related line agencies (e.g. District Agricultural Office, District Development Committee)  Chemonics/NEAT Staff  Business Associations – such as FNCCI, CNI  Government Agencies: Ministry of Finance, Department of Customs, Inland Revenue Department, Ministry of Industry, Ministry of Agriculture Development, and Ministry of Commerce and Supplies.  Commercial Banks – Mega Bank, Laxmi Bank  Microfinance institutions  Private businesses Other stakeholders include the following:  Staff of selected other donor and INGO staff  USAID/Nepal SEED team 9. TIMELINE AND DELIVERABLE(S): (i) Timeline The timeline for this SOW is June 17, 2013 – August 30, 2013. Given the 49-day period for the consultancy, this timeline includes some flexibility for unexpected interruptions or non-working days if needed. Estimated number of Work Days Estimated Number of Actual Work Days Activities 58 Day 1- 5 5 Document review and preparation of evaluation work plan & methodology Day 6-7 2 Travel to the Nepal Day 8 1 In Brief with USAID/Nepal staff Day 9- 11 3 Document review, planning, finalization of evaluation work plan and detailed methodology. Presentation of evaluation work plan and methodology Day 12 - Day 317 10 (Trade/Economic Policy Expert) 20 (Agriculture Expert) Policy Component Kathmandu-based interviews Food Security and Competitiveness Component Field work (including travel to and from field sites) and some Kathmandu based interviews Team Leader will divide the time between the two Day 32-39 8 Evaluation team review of findings and debriefing; prepare and deliver a separate presentation, as scheduled by USAID/Nepal, to outline major findings / recommendations. Drafting of the Preliminary Draft Evaluation Report and submission to USAID/Nepal for feedback Day 40 - 41 2 Travel to US Day 42-49 8 Review of Preliminary Draft Evaluation Report by USAID/Nepal Day 50-58 8 Finalization and Submission of draft report The evaluation timeline provided above is a guide that may need to be refined. USAID/Nepal will provide comments within 8 working days of the submission of the draft report. A revised final draft will be submitted within 8 working days after receipt of comments from USAID/Nepal. The evaluation report will be final only after it is cleared in writing by USAID/Nepal. (ii) Deliverables To make the field time as efficient as possible, preparation must include completing a majority of the documentation review, establishing interview questionnaires and or guides, field mobilization plan, developing team protocol and responsibilities, and establishing the evaluation schedule. The team must complete the following deliverables within the specified period of the assignment: Deliverables include an evaluation methodology, presentation and a final evaluation report with recommendations, as outlined below. 1. A detailed Work Plan submitted to the COR at USAID/Nepal for approval. 2. Evaluation Methodology which describes the overall evaluation design, data collection and analysis methods for each evaluation questions, data collection instruments. 3. Presentation of the evaluation methodology to the technical team in SEED and PPD Office for comments before beginning the evaluation. 59 4. Draft Evaluation Report, submitted to the USAID/Nepal COR, who will provide comments from the internal evaluation report review process in USAID/Nepal within 10 working days of submission. 5. Power Point Presentation on important findings, conclusions and & recommendations to an audience of USAID/Nepal Mission, partners, donors, and GON. USAID/Nepal will invite the audience and organize the venue for this presentation. 6. Final Evaluation Report which clearly describes the findings, conclusions and recommendations submitted to the USAID/Nepal COR. The report must also include best practices, case studies and lessons learned and follow the guidelines in Section 10 and meet the criteria outlined in the Evaluation Report Review Checklist in Annex 3. 7. Two hard copies of the final evaluation report, 20-30 pages, not including graphs, diagrams, tables, annexes, cover pages, and table of contents, with good quality spiral binding. 8. A soft copy of evaluation report, in MS Word and PDF format. 9. Raw data and records of the evaluation report (e.g. interview transcripts, survey responses etc.) in electronic form collected by the evaluation team separately from the report. The evaluation report should demonstrate a clear line of analysis between findings, conclusions and recommendations. The report must be in concise and clear English with visual summaries such as graphics, charts and summary data tables Deliverable Submission Evaluation Work Plan and Methodology Day 11 Presentation on Evaluation Methodology Day 11 Preliminary Draft Report Day 42 Power point Presentation on important findings, conclusions, lessons learnt, and recommendations Day 42 Draft Evaluation Report Day 58 The Team Leader has the final responsibility for prioritizing which conclusions and recommendations are highlighted in the report. If there are additional recommendations or alternatives in addition to those highlighted, they can be included in an annex. Different perspectives or subject matter expertise within an evaluation team will sometimes lead to a different interpretation of facts. Footnotes may be used to draw attention to different interpretations of findings. 10. REPORTING GUIDELINES: USAID/Nepal requires that the team review USAID’s Criteria to Ensure the Quality of the Evaluation Report, which can be accessed online at: http://transition.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf As mentioned above, findings from the evaluation will be presented in a draft report at a full briefing with USAID/Nepal. The format for the evaluation reports is as follows: 1. Executive Summary – concisely state the most salient findings and recommendations 2. Table of Contents 3. Introduction – purpose, audience, and synopsis of task 4. Background 5. Methodology – describe evaluation methods, including constraints and gaps 60 6. Findings/Conclusions 7. Recommendations/Future Directions 8. References (including bibliographical documentation, meetings, interviews and focus group discussions); 9. Annexes – annexes that document the evaluation methods, schedules, interview lists and table – should be succinct, pertinent and readable. 11. TERMS AND CONDIDTIONS OF THE CONSULTANCY: Each member of the evaluation team will be required to sign a Non-Disclosure Agreement (Annex 1) and Disclosure of Real or Potential Conflict of Interest for USAID evaluations (Annex 2). 12. COMPOSITION OF THE EVALUATION TEAM: The evaluation team must be made up of 3 non-USAID development professionals with expertise in project evaluation, agriculture, private sector and trade and fiscal areas. The Team Leader must have extensive knowledge and experience leading and conducting project evaluations and evaluation methodologies. One member must be a trade/ /economic policy expert, and one member must be an agriculture expert. There should be both male and female members in the evaluation team. At least one member of the team must have experience in cost benefit analysis. The evaluation team members should not be employees of any of the organizations that are receiving funds from the NEAT Program. Team Leader: The Team Leader must have a minimum of Master's Degree and at least 10 years of relevant research and evaluation experience as a lead. Proven experience in similar types of evaluation. The Team Leader will have the authority and responsibility to conduct and manage the evaluation and submit deliverables to USAID/Nepal. The responsibilities include: technical leadership for and supervision of team members; quality control and timeliness of all deliverables; preparation or supervision of evaluation methodology, logistical plan, data collection, and report preparation; serve as a primary point of contact for the evaluation team to USAID Contracting Officer and his/her representative. The Team Leader will lead the presentations to USAID/Nepal and other stakeholders on the findings, conclusion, and recommendations of the evaluation and ensure timely submission of all deliverables. Agriculture Expert: Master’s Degree, at least 7 years of experience in implementing agriculture programs. The Agriculture Expert will work with the evaluation team to develop an evaluation methodology and evaluate the appropriateness and effectiveness of agriculture and income generation interventions (business development, market development and the value chain approach). The Agriculture Expert will interact with subcontractors, on various aspects of agriculture program implementation, including planning, design, implementation, sustainability, and best practices. Trade/Economic Policy Expert: Master’s Degree, at least 7 years of experience in implementing economic policy program with a focus on trade & fiscal policy and practices. The Economic Policy Expert will work with the evaluation team to develop an evaluation methodology and evaluate the appropriateness and effectiveness of economic policy and access to finance interventions (trade, fiscal and access to finance) The Trade/Economic Policy Expert will interact with subcontractors, on various aspects of policy program implementation, including planning, design, implementation, sustainability, and best practices. Logistics 61 The evaluation team is responsible for managing all logistics required for completing the evaluation. This includes but is not limited to arranging for transportation, meeting venues and appointments for meetings. Chemonics International or its sub-contractor staff may assist in organizing meetings. USAID/Nepal will provide at least one copy of the Chemonics planning and reporting documents and may provide other reference materials as required. USAID/Nepal Participation USAID/Nepal staff may join the evaluation team as and when necessary. USAID staff may participate as an additional member of the team during primary data collection, specifically during Semi Structured Interviews with focus groups, key informants, implementing partners. The USAID/Nepal team participant will manage his/her own logistics through close coordination with the Team Leader. To ensure against bias or conflict of interest, the USAID/Nepal team member’s role will be limited to participating in the fact-finding phase, and contributing to the analysis. The final responsibility for analysis, conclusions and recommendations will rest with the independent members and Team Leader. 13. REPORTING AND DISSEMINATION: The final, approved report must be entered in the Development Experience Clearinghouse database (DEC). The evaluation team leader is responsible for submitting the final, branded and approved report into the DEC. See website for instructions on how to submit reports into the DEC database. 14. BUDGET: The Offerer is expected to submit a proposed budget in the range of $100,000 – $150,000 along with proposed team members. The items in the proposed budget should include consultancy fees, per diem, in￾country airfare, vehicle rental, group accident insurance and other direct cost such as stationary, photocopy, utilities/venue rental. 15. EVALUATION CRITERIA: The technical proposal will be more important than cost in the best value decision. However, the cost proposal submitted by the Offerer will also be an important factor in determining the best value. The Offerer should note that these criteria: (1) serve as the standard against which the proposal will be evaluated, and (2) serve to identify the significant matters which the Offerer should address in the proposal. 1. Evaluation Methodology/approach: (35%) •Appropriateness of data collection methods to answer each evaluation question •Use of quantitative and qualitative methods for data collection •Plan for data analysis •Sampling design •Approaches for triangulation 2. Technical Competence: (30%) •Qualification and experience of evaluation team members; •Composition of the team with expertise on economic policy, trade & fiscal policies and practices, financial services, and agriculture. •Expertise in evaluation of programs and projects 62 •Legal and professional status of the firm or individual •Organizational strengths of the firm or individual 3. Experience: (30%) •Past experience on evaluation of projects and programs including experience of evaluating economic growth and agriculture and competitiveness programs •Experience managing logistics for conducting such evaluations •Relevance of experiences conducting USAID or other donor implemented programs and projects within the last two years. 4. Mobilization Potential: (5%) •Ability and readiness to conduct the assignment between June 17, 2013 – August 30, 2013; and •Ability to make timely payments for Travel and Daily Allowance and manage logistics support to the evaluation team members Annex 1: Non-Disclosure Agreement PRECLUSION FROM FURNISHING CERTAIN SERVICES AND RESTRICTION ON USE OF INFORMATION With respect to proposal submitted dated XXXXXX in response to solicitation of USAID/Nepal’s evaluation of NEAT dated XXXXX, the undersigned hereby agrees and certifies to the following: (a) This SOW calls for the contractor to furnish important services in support of the evaluation of the NEAT. In accordance with the principles of FAR Subpart 9.5 and USAID policy, the contractor shall be ineligible to furnish, as a prime or subcontractor or otherwise, implementation services under any contract or task order that results in response to findings, proposals, or recommendations in the evaluation report within 18 months of USAID accepting the report, unless the head of the contracting activity, in consultation with USAID’s competition advocate, authorizes a waiver (in accordance FAR 9.503) determining that preclusion of the contractor from the implementation work would not be in the government's interest. (b) In addition, by accepting this contract, the contractor agrees that it will not use or make available any information obtained about another organization under the contract in the preparation of proposals or other documents in response to any solicitation for a contract or task order. (c) If the contractor gains access to proprietary information of any other company in performing this evaluation, the contractor must agree with the other company to protect the information from unauthorized use or disclosure for as long as it remains proprietary, and must refrain from using the information for any purpose other than that for which it was furnished. Contractor must provide a properly executed copy of all such agreements to the contracting officer. Signature: ________________________ Name Typed or Printed: ________________________ Date: ________________________ 63                                                                                                                                              Annex 2: Disclosure of Real or Potential Conflict of Interest for USAID Evaluations Instructions: Evaluations of USAID projects will be undertaken so that they are not subject to the perception or reality of biased measurement or reporting due to conflict of interest. 26 For external evaluations, all evaluation team members will provide a signed statement attesting to a lack of conflict of interest or describing an existing conflict of interest relative to the project being evaluated.27 Evaluators of USAID projects have a responsibility to maintain independence so that opinions, conclusions, judgments, and recommendations will be impartial and will be viewed as impartial by third parties. Evaluators and evaluation team members are to disclose all relevant facts regarding real or potential conflicts of interest that could lead reasonable third parties with knowledge of the relevant facts and circumstances to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Operating Unit leadership, in close consultation with the Contracting Officer, will determine whether the real or potential conflict of interest is one that should disqualify an individual from the evaluation team or require recusal by that individual from evaluating certain aspects of the project(s). In addition, if evaluation team members gain access to proprietary information of other companies in the process of conducting the evaluation, then they must agree with the other companies to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. 28 Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. Disclosure of Conflict of Interest for USAID Evaluation Team Members Name Title Organization Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), 26 USAID Evaluation Policy (p. 8); USAID Contract Information Bulletin 99‐17; and Federal Acquisition Regulations (FAR) Part 9.5, Organizational Conflicts of Interest, and Subpart 3.10, Contractor Code of Business Ethics and Conduct. 27 USAID Evaluation Policy (p. 11) 28 FAR 9.505‐4(b) 64 implementer name(s) and award number(s), if applicable) I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, 65 organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 66                                                              Annex 3: Checklist for Evaluation Report Review Title of study being reviewed: __________________________________ GOOD PRACTICE ELEMENTS OF AN EVALUATION REPORT29 Keyed to USAID’s 2011 Evaluation Policy EVALUATION REVIEW FACTOR 1 2 3 4 5 Comments STRUCTURE OF THE REPORT 1. Does the evaluation report have a cover sheet attached indicating the type of evaluation conducted (e.g. performance evaluation or impact evaluation) and general design? 2. If a performance evaluation, does the evaluation report focus on descriptive and normative evaluation questions? 3. If the evaluation report uses the term “impact evaluation,” is it defined as measuring the change in a development outcome that is attributable to a defined intervention (i.e. impact evaluations are based on models of cause and effect and require a credible and rigorously defined counterfactual)? 4. Regardless of the type of evaluation, does the evaluation report reflect use of sound social science methods? 5. Does the report have a Table of Contents (TOC)? 6. Do Lists of Figures and Tables follow the TOC? 7. Does the report have a Glossary of Terms? 7.1.1 Are abbreviations limited to the essential? 8. Is the date of the report given? 9. Does the body of the report adhere to the 20 page guide? 10. Is the report well-organized (each topic is clearly delineated, subheadings used for easy reading)? 11. Does the report’s presentation highlight important information in ways that capture the reader’s attention? 12. Is the report well written (clear sentences, reasonable length paragraphs, no typos, acceptable for dissemination to potential users)? 13. Does the evaluation report focus on the essential issues concerning the key questions, and eliminate the “nice to know”, but not essential information? 14. Does the evaluation report disclose either lack of a conflict of interest by all evaluation team members and/or describe any conflict of interest that existed relative to the project being evaluated? 15. As applicable, does the evaluation report include statements regarding any significant unresolved differences of opinion on the part of funders, implementers and/or members of the evaluation team? EXECUTIVE SUMMARY 16. Does the evaluation report begin with a 3- to 5-page stand-alone summary of the purpose, background of the project, main evaluation questions, methods, findings, conclusions, recommendations and lessons learned (if applicable) of the evaluation? 17. Does the Executive Summary concisely state the main points of the evaluation? 29 In addition to the USAID 2011 Evaluation Policy, good practices in evaluation reporting have also been drawn from: Morra Imas, Linda and Ray C. Rist. 2009. The Road to Results: Designing and Conducting Effective Development Evaluations. Washington, DC.: The World Bank. Scriven, Michael. 2005. Key Evaluation Checklist. Stufflebeam, Daniel L. 1999. Program Evaluations Metaevaluation Checklist. 67 EVALUATION REVIEW FACTOR 1 2 3 4 5 Comments 18. Does the Executive Summary follow the rule of only saying what the evaluation itself says and not introducing new material? INTRODUCTION 19. Does the report introduction adequately describe the project? 19.1. Does the introduction explain the problem/opportunity the project was trying to address? 19.2. Does the introduction show where the project was implemented (physical location) through a map? 19.3. Does the introduction explain when the project was implemented? 19.4. Are the “theory of change” or development hypotheses that underlie the project explained? (Does the report specify the project’s inputs, direct results (outputs), and higher level outcomes and impacts, so that the reader understands the logical structure of the project and what it was supposed to accomplish?) 19.5. Does the report identify assumptions underlying the project? 19.6. Does the report include sufficient local and global contextual information so that the external validity and relevance of the evaluation can be assessed? 19.7. Does the evaluation report identify and describe any critical competitors to the project that functioned at the same time and in the project’s environment? 19.8. Is USAID’s level of investment in the project stated? 19.9. Does the evaluation report describe the project components funded by implementing partners and the amount of funding? 20. Is the purpose of the evaluation clearly stated? 21. Is the amount of USAID funding for the evaluation indicated? 22. Are all other sources of funding for the evaluation indicated as well as the amounts? 23. Does the report identify the evaluation team members and any partners in the evaluation? 24. Is there a clear statement of how the evaluation will be used and who the intended users are? 25. Are the priority evaluation questions presented in the introduction? 26. Does the evaluation address all evaluation questions included in the Statement of Work (SOW)? 26.1. Are any modifications to the SOW, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline indicated in the report? 26.2. Is the SOW presented as an annex? 26.3. If so, does the annex include the rationale for any change with the written sign-offs on the changes by the technical officer? SCOPE AND METHODOLOGY 27. Does the report provide a clear description of the evaluation’s design? 27.1. Is a design matrix or similar written tool presented in an annex that shows for each question/subquestion the measure(s) or indicator(s) used to address it, the source(s) of the information, the type of evaluation design, type of sampling if used, data collection instrument(s) used, and the data analysis plan? 28. Does the report state the period over which the evaluation was conducted? 29. Does the report state the project time span covered by the evaluation? 68 EVALUATION REVIEW FACTOR 1 2 3 4 5 Comments 30. Does the evaluation report indicate the nature and extent of consultation on the evaluation design with in-country partners and beneficiaries? 31. Does the evaluation report indicate the nature and extent of participation by national counterparts and evaluators in the design and conduct of the evaluation? 32. Does the report address each key question around which the evaluation was designed? 33. Is at least one of the evaluation questions directly related to gender analysis of outcomes and impacts? 34. Are data sex-disaggregated? 35. In answering the questions, does the report appropriately use comparisons made against baseline data? 36. If the evaluation is expected to influence resource allocation, does it include information on the cost structure and scalability of the intervention, as well as its effectiveness? 36.1. As appropriate, does the report include financial data that permits computation of unit costs and analysis of cost structure? 37. Is there a clear description of the evaluation’s data collection methods (summarized in the text with the full description presented in an annex)? 37.1. Are all tools (questionnaires, checklists, discussion guides, and other data collection instruments) used in the evaluation provided in an annex? 37.2. Does the evaluation report include information, as appropriate, on the pilot testing of data collection instruments? 37.3. Does the evaluation report include information, as appropriate, on the training of data collectors? 38. Are all sources of information properly identified and listed in an annex? 39. Does the evaluation report contain an section describing the “strengths” and “limitations” associated with the evaluation methodology (e.g. selection bias, recall bias, unobservable differences between comparator groups, small samples, only went to villages near the road, implementer insisted on picking who the team met with, etc)? 40. Does the evaluation report indicate the evaluation methodology took into account the time, budget, and other practical considerations for the evaluation such as minimizing disruption and data burden? 41. Does the report have sufficient information to determine if the evaluation team had the appropriate methodological and subject matter expertise to conduct the evaluation as designed? 42. If an impact evaluation was designed and conducted, does the evaluation report indicate that experimental methods were used to generate the strongest evidence? Or does the report indicate that alternative methods for assessing impact were utilized and present the reasons why random assignment strategies were not feasible? 43. Does the evaluation report reflect the application and use to the maximum extent possible of social science methods and tools that reduce the need for evaluator-specific judgments? 44. Does the evaluation scope and methodology section address generalizability of the findings? ANALYSIS 45. Are percentages, ratios, cross-tabulations, rather than raw data presented, as appropriate? 46. When percentages are given, does the report always indicate the number of cases used to calculate the percentage? 69 EVALUATION REVIEW FACTOR 1 2 3 4 5 Comments 46.1. Is use of percentages avoided when the number of cases is small (<10)? 47. Are whole numbers used or rounding-off numbers to 1 or 2 digits? 48. Are pictures used to good effect? 48.1. Relevant to the content 48.2. Called out in the text and placed near the call-out 49. Are charts and graphs used to present or summarize data, where relevant? 49.1. Are the graphics easy to read and simple enough to communicate the message without much text? 49.2. Are they consistently numbered and titled? 49.3. Are they clearly labeled (axis, legend, etc.) 49.4. Is the source of the data identified? 49.5. Are they called out in the text and correctly placed near the call-out? 49.6. Are the scales honest (proportional and not misleading by virtue of being “blown-up”)? FINDINGS 50. Are FINDINGS specific, concise and supported by strong quantitative and qualitative evidence? 50.1. As appropriate, does the report indicate confirmatory evidence for FINDINGS from multiple sources, data collection methods, and analytic procedures? 51. Are adequate data provided to address the validity of the “theory of change” or development hypothesis underlying the project, i.e., cause and effect relationships? 52. Are alternative explanations of any observed results discussed, if found? 53. Are unplanned results the team discovered adequately described? 54. Are opinions, conclusions, and recommendations kept out of the description of FINDINGS? CONCLUSIONS 55. Is there a clear distinction between CONCLUSIONS and FINDINGS? 56. Is every CONCLUSION in the report supported by a specific or clearly defined set of FINDINGS? 57. Are the CONCLUSIONS credible, given the FINDINGS the report presents? 58. Can the reader tell what CONCLUSIONS the evaluation team reached on each evaluation question? RECOMMENDATIONS 59. Are RECOMMENDATIONS separated from CONCLUSIONS? (Are they highlighted, presented in a separate section or otherwise marked so that the reader sees them as being distinct?) 60. Are all RECOMMENDATIONS supported by a specific or clearly defined set of FINDINGS and CONCLUSIONS? (Clearly derived from what the evaluation team learned?) 61. Are the RECOMMENDATIONS practical and specific? 62. Are the RECOMMENDATIONS responsive to the purpose of the evaluation? 63. Are the RECOMMENDATIONS action-oriented? 64. Is it clear who is responsible for each action? 65. Are the RECOMMENDATIONS limited/grouped into a reasonable number? LESSONS LEARNED 70 EVALUATION REVIEW FACTOR 1 2 3 4 5 Comments 66. Did this evaluation include lessons that would be useful for future projects or programs, on the same thematic or in the same country, etc.? 67. Are the LESSONS LEARNED highlighted and presented in a clear way? 68. Does the report indicate who the lessons are for? (e.g., project implementation team, future project, USAID and implementing partners, etc.) BOTTOM LINE 69. Does the evaluation report give the appearance of a thoughtful, evidence￾based, and well organized effort to objectively evaluate what worked in the project, what did not and why? 70. As applicable, does the evaluation report include statements regarding any significant unresolved differences of opinion on the part of funders, implementers and/or members of the evaluation team? 71. Is the evaluation report structured in a way that will promote its utilization? 72. Does the evaluation report explicitly link the evaluation questions to specific future decisions to be made by USAID leadership, partner governments and/or other key stakeholders? 73. Does the evaluation report convey the sense that the evaluation was undertaken in a manner to ensure credibility, objectivity, transparency, and the generation of high quality information and knowledge? 71 Appendix 2: Sampling Design for Beneficiary Surveys Through collaboration with USAID/Nepal and NEAT implementing partners, the LEAP team identified a sampling of districts in which the evaluation team would collect beneficiary and stakeholder information on the project’s effectiveness. The evaluation team determined that the final list of districts to be surveyed should include the following:  a minimum of one district from each geographic location (Hills and Terai);  districts that are accessible;  districts that contain the most number of households who participated in the NEAT program (to increase the statistical power of analysis);  districts that have a high density of disadvantaged groups; and  districts that received the maximum number of NEAT interventions. Once the initial sampling framework was developed at the district level, the LEAP team held several meetings with USAID/ Nepal and NEAT implementing partners to verify which areas would be accessible to the interviewers given that the interviews would be conducted during the rainy season. Through this process, the list was revised, as needed. Selection and Sampling of VDCs Once the districts were selected, based on the overall districts where implementation occurred (refer to Appendix 5) the next step was to identify the village development committees (VDCs) in each district that would be sampled. A list of VDCs was constructed, from which the project team randomly selected VDCs to be sampled. The final sample of VDCs to be surveyed was based on:  the accessibility of the VDC (because of road conditions and weather);  survey time limitations (traveling distance to the VDC); and  the maximum number of activities implemented in the VDC. Using the criteria mentioned above, the LEAP team was able to select an appropriate sample size to ensure that sufficient data were obtained from different geographic locations to allow for a representative sample of the targeted population. Methods of Selecting Households Two methods were used to select households for inclusion in the survey. Method 1 was the preferred methodology used, with method 2 being used only as needed. Method 1. To ensure that each household in the selected districts and VDCs had an equal chance of being selected for the survey, a sampling interval was determined to reduce error. This was done by first calculating the total number of households (provided by Chemonics) in each VDC and then computing the cumulative of these households. The sampling interval is equal to the total number of households in all VDCs under each component divided by the sample size of the component. The number of households to be interviewed based on this interval size is equal to the number of households in each VDC divided by the sampling interval, as demonstrated below: 72 Sample size= 160 H.H VDCs= 4 Population Cumulative Population Number of households to be interviewed VDC 1 100 100 27 (100/3.7) VDC 2 290 390 79 (290/3.7) VDC 3 150 540 41 (150/3.7) VDC 4 50 590 14 (50/3.7) Sampling interval= 590/160 = 3.7 160 Method 2. The second method involved purposeful selection of households. This method was utilized when challenging field conditions made households inaccessible. In the case of the tea value chain, there was no compiled universe of households available, so it was necessary to use this method for household selections. Selected Survey Implementation Districts and VDCs The team surveyed the following:  174 food security households in 8 districts and 18 VDCs;  69 lentil households in 3 districts and 6 VDCs;  66 ginger households in 4 districts and 6 VDCs;  78 tea households in 2 districts and 11 VDCs; and  62 vegetable households in 4 districts and 8 VDCs. Food security and competitiveness target districts and VDCs District VDC Components Food Security Ginger Lentils Tea Vegetable Arghankachi Narapani √ Sitapur √ Bardiye Bagnaha √ Deudakala √ Motipur √ Neulapur √ Dang Chailahi √ Lalmatia √ 73 District VDC Components Food Security Ginger Lentils Tea Vegetable Phulbari √ Urahari √ Ilam Fikal √ Kanyam √ Kolbung √ Mangalbare √ Pachakanya √ Pashupatinagar √ Sankhejung √ Shantidanda √ Kapilvastu Dankauli √ Gotihawa √ Kopuwa √ Nigliwahawa √ Taulihawa √ Tilaurakot √ Kathmandu Dharapani √ Indrayani √ Lalitpur Lele √ Manikhel √ Nawalparasi Kushma √ Mithukarma √ Somani √ Palpa Masyam √ Tansen √ √ Telgha √ √ Panchthar Panchami √ Pauwasartap √ Tharpu √ 74 District VDC Components Food Security Ginger Lentils Tea Vegetable Pyuthan Dharmawati √ Dhungegadhi √ Hanksapur √ Maranthana √ Rolpa Liwang √ Salyan Dhanabang √ Dhanwang √ Phalawang √ √ Triveni √ 75 Appendix 3: Value Chains and Food Security: Semi-structured Key Informant Interviews Organizations/Agencies Represented Sector Category Organization/Agency Food Security Implementing Partner District Agriculture Development Office (DADO) Center for Environmental and Agricultural Policy Research, Extension and Development (CEAPRED) Input Supplier Sidhent Agro Center Ginger Implementing Partner Forum for People Awareness (FOPA) Multidimensional Agriculture for Development (MADE) Public Progressive Social Development Center (PPSDC) Trade Processor Anapurna Organic Agricutlure Ltd Khaptar Aroma Mahalaximi Herbal & Spices Pvt. Ltd. Small Trader PASPATI and Galla Suppliers ROHIT and Galla Suppliers Lentil Implementing Partner Forward Jahada Krishi Sewa Kendra DADO Office, Nawalparasi Input Supplier Siddhartha Agrovet Center Trade Processor Agriculture Cooperative Ltd Saurad Agarwal Panur Prasad Agrauri. Tea Implementing Partner Himalayan Orthodox Tea Producers Cooperative Ltd. (HIMCOOP) Trade Processor Nepal small tea producing limited (NESTPROL) Nepal Green & Speciality Tea Private Limited Kanchenjunga Tea Estate & research center pvt. Ltd. Himalayan Shangri-La Tea Producers (P) Ltd Gorkha Tea Estate Pvt Ltd Trishakti Pathivara Tea Industry Paathibhara Lekali Krishi Sahakari Lekali Organic tea Udhyog Vegetable Implementing Partner Maden Pokhara Multipurpose Kids/Local NGO DADO, Kapilvastu Federation of Fruits and Vegetables Entrepreneurs Nepal (FEVEN) 76 Sector Category Organization/Agency Institute for Sustainable agriculture Nepal (INSAN) Unity Service Cooperation (USC0 Input Supplier Sean Seed Service Centre Ltd. (SSSC) Muariya Seed Store (Agrovet) Trade Processor MPC – Marketing Planning Committee Panchera Vegetable Collection Center Grameen Unnati Krishak Samuha 77 Appendix 4: Evaluation Report Reference List List of NEAT Program Documents Received from USAID/Nepal Annual Reports (2)  United State International Development Agency (USAID) Nepal. “Annual Report: Nepal, Economic, Agriculture and Trade Activity (October 2010 – September 2011).” October 2011.  USAID Nepal. “Annual Report: Nepal, Economic, Agriculture and Trade Activity (October 2011 – September 2012).” October 2012. Annual Work Plans (3)  USAID Nepal. “Annual Work Plan (March 1, 2011 – September 30, 2011) Nepal Economic, Agriculture, and Trade Activity.” FY2011. March 18, 2011.  USAID Nepal. “Annual Work Plan (October 1, 2011 – September 30, 2012) Nepal Economic, Agriculture, and Trade Activity.” FY2012. August 30, 2011.  USAID Nepal. “Annual Work Plan (July 1, 2012 – June 30, 2013) Nepal Economic, Agriculture, and Trade Activity.” FY2013. June 30, 2012. Performance Management Plan (2)  USAID Nepal. “Performance Management Plan: Nepal Economic, Agriculture and Trade Activity 2011.” March 2011.  USAID Nepal. “Revised Performance Management Plan: Nepal Economic, Agriculture and Trade Activity 2012.” August 2012. Quarterly Reports (8) 2011  USAID Nepal. “Quarterly Report: Nepal Economic, Agriculture and Trade Activity (January 2011 – March 2011).” Quarter 2, FY 2011. April 2011.  USAID Nepal. “Quarterly Report: Nepal Economic, Agriculture and Trade Activity (April 2011 – June 2011).” Quarter 3, FY 2011. July 2011. 2012  USAID Nepal. “Quarterly Report: Nepal Economic, Agriculture and Trade Activity (October 2011 – December 2011).” Quarter 1, FY 2012. January 2012.  USAID Nepal. “Quarterly Report: Nepal Economic, Agriculture and Trade Activity (January 2012 – March 2012).” Quarter 2, FY 2012. April 2012.  USAID Nepal. “Quarterly Report Addendum: Supplemental M&E Data for the Nepal Economic, Agriculture and Trade Activity (January 2012 – March 2012).” Quarter 2, FY 2012. April 2012. 78 2013  USAID Nepal. “Quarterly Report: Nepal Economic, Agriculture and Trade Activity (April 2012 – June 2012).” Quarter 3, FY 2012. July 2012.  USAID Nepal. “Quarterly Report: Nepal Economic, Agriculture and Trade Activity (October 2012 – December 2012).” Quarter 1, FY 2013. January 2013.  USAID Nepal. “Quarterly Report: Nepal Economic, Agriculture and Trade Activity (January 2013 – March 2013).” Quarter 2, FY 2013. April 2013. Success Stories (1)  USAID Nepal. “Success Story: Saving for a Brighter Future.” Quarter 1, FY 2013. January 2013. Disaggregated Data (3)  USAID Nepal. “FY12 Disaggregated Data.” 2012.  USAID Nepal. “FY13 Q1 Disaggregated Data.” 2013.  USAID Nepal. “FY13 Q2 Disaggregated Data.” 2013. Subsector Assessment Documents (3)  USAID Nepal. “Value Chain/ Market Analysis of the Ginger Sub-Sector in Nepal (Contract No. AID-367-TO-11-00001).” August 2011.  USAID Nepal. “Value Chain/ Market Analysis of the Lentil Sub-Sector in Nepal (Contract No. AID-367-TO-11-00001).” August 2011.  USAID Nepal. “Value Chain Competitiveness Assessment for Selected [Tea and Vegetable] Sub-Sectors: Nepal Economic, Agriculture, and Trade Activity (Contract No. AID-367-TO-11- 00001).” November 2011. Other Desk and Literature Review Documents (5)  USAID Nepal. “NEAT Assessment/Study Report List.” Prepared by Gautam, USAID/SEED.  USAID Nepal. “Food Security and Vulnerability Analysis: Nepal Economic, Agriculture and Trade Activity (Contract No. EEM-I-000-07-00008).” February 2011.  USAID Nepal. “Economic Policy Assessment: Nepal Economic, Agriculture and Trade Activity (Contract No. EEM-I-000-07-00008).” February 2011.  USAID Nepal. “Grants Management Handbook: Nepal Economic, Agriculture and Trade Activity (Contract No. EEM-I-00-07-00008).” February 14, 2011.  USAID Nepal. “Nepal Economic, Agriculture, and Trade: Description/ Specifications/ Statement of Work [for Chemonics].” List of Other Documents Reviewed for the Evaluation (9) 79 Adhikari, Ratnakar, and Kamalesh Adhikari. “Market Access Barriers to Select Nepalese Agricultural Exports.” Prepared by South Asia Watch on Trade, Economics and Environment (SAWTEE) and International Institute for Sustainable Development (IISD). December 2005. Asian Development Bank, June 2013. Agriculutural Development Strategy Final Report. http://www.adb.org/sites/default/files/projdocs/2013/43447-022-nep-tacr-01.pdf Bennett, Lynn, Dilli Ram Dahal, and Pav Govindasamy. “Caste, Ethnic and Regional Identity in Nepal: Further Analysis of the 2006 Nepal Demographic and Health Survey.” September 2008. Centre for International Studies and Cooperation (CECI), Sahakarya Project. “Micro-financing towards Empowerment of Disadvantaged Groups in Nepal: Innovations and Practices.” Prepared by Micro Finance Department and Nepal Rastra Bank. March 2008. Department for International Development (DFID) Nepal, and The World Bank Nepal. “Unequal Citizens: Gender, Caste and Ethnic Exclusion in Nepal.” 2006. Federspiel, Genevieve, Dandi Ram Bishwakarma, Dibya Gurung, and Shanker Pathak. “Disadvantaged Groups (DAG) Mapping: Dolakha, Ramechhap and Okhaldhunga Districts.” Prepared by Swiss Agency for Development and Cooperation (SDC). June 2008. Government of Nepal National Planning Commission Secretariat, Central Bureau of Statistics. “National Population and Housing Census 2011 (National Report).” Volume 01, NPHC 2011. Kathmandu, Nepal. November 2012. United Nations Educational, Scientific and Cultural Organization (UNESCO), including the Education for All 2000 Assessment. Unicef Statistics. http://www.unicef.org/infobycountry/nepal_nepal_statistics.html. Nepal total adult literacy rate, 2008 – 2012. United States Agency for International Development (USAID). “USAID Evaluation Policy.” Retrieved: http://www.usaid.gov/sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf Washington, District of Columbia. January 2011. United States Department of Health and Human Services. “Identifying and Selecting Evidence-Based Interventions: Revised Guidance Document for the Strategic Prevention Framework State Incentive Grant Program.” January 2009. Shrestha, Purushottam. “Improving Access to Microfinance Services in the Hills and Mountains of Nepal.” Chairman: Naya Nepal Laghu Bitta Bikas Bank Limited. Presented at the Microfinance Summit Nepal 2013 organized by Center for Microfinance (Feb. 14-16, 2013) Kathmandu, Nepal. USAID Nepal. “Assessment of Trade Negotiating Skills of Nepal Officials and Recommendations for Future Action. Post-Trip Assessment Report on Advanced Trade Negotiating Techniques Training Kathmandu, Nepal.” Prepared by Alicia D. Greenridge under Contract no. EEM-I-00-07-00008, AID￾367-TO-11-00001. December 2012 USAID Nepal. “A Study on Nepal-India Trade.” Prepared by South Asia Watch on Trade, Economics and Environment (SAWTEE) under Prime Contract No. EEM-I-00-07-0008, AID-367-TO-11-0001). July 2012. 80 USAID Nepal. “Economic Policy Assessment. Nepal Economics, Agriculture, and Trade Activity.” Prepared by CHecmonics International INc. and METCON Consultants under Prime Contract No. EEM￾I-00-07-00008, AID-367-TO-11-00001. February 2011. USAID Nepal. “Foreign Investment Policy 2012 (English and Nepali Versions). Nepal, Economics, Agriculture, and Trade Activity.” Prepared by the Institute for Policy, Research, and Development (IPRAD) for Chemonics International Inc. under Prime Contract No. EEM-I-00-07-00008, AID-367-TO￾11-00001. December 2012. USAID Nepal. “Gender Assessment. Nepal, Economics, Agriculture, and Trade Activity.” Prepared by Meena Bilgi and Jeannette Gurung of WOCAN under Prime Contract No. EEM-I-00-07-00008, AID￾367-TO-11-00001. May 2011. USAID. “Policy Implementation Assessment Tool: Master Interview Guide for Policymakers.” May 2010. USAID Nepal. “Initial Assessment of the Trade Data Availability, Trade Issues Reporting System, and Trade Analytical and Negotiation Capabilities of the Ministry of Commerce and Supplies (MOCS) Government of Nepal. Nepal Economic, Agriculture, and Trade (NEAT) Activity: Trade Policy Technical Report.” Prepared by Chemonics International Inc. May 2011. USAID Nepal. “Microfinance Training Provider Assessment Report. Nepal Economics, Agriculture, and Trade Activity. (Contract No. AID-367-TO-11-00001)” August 2011. USAID Nepal. “Nepal Investment Policy Considerations. Nepal , Economics, Agriculture, and Trade Activity.” Prepared by Peter F. Carr for Chemonics International Inc. in association with the Institute for Policy Research and Development (IPRAD) Kathmandu, Nepal under the auspices of the USAID/NEAT program under Contract No. AID-367-TO-11-00001. January 2013. USAID Nepal. “Strengthening the Foundation for Inclusive Economic Growth: Nepal Economic, Agriculture, and Trade (NEAT) Activity: Final Report.” Prepared by Chemonics International Inc. August 2013. 81 Appendix 5: Design Matrix for Review of Nepal Economic, Agricultural and Trade (NEAT) Program Evaluation Researchable Question(s) Information Required and Source(s) Scope and Methodology Limitations Implications 1) How appropriate and effective were NEAT’s approaches and interventions in promoting inclusive economic growth?  Appropriateness of NEAT’s five components, interventions, engagement with target group and stakeholders Information on NEAT stakeholder and beneficiary involvement during planning and early implementation Criteria to evaluate appropriateness and effectiveness, as defined by NEAT’s goals, performance indicators and target results Assess causal linkages between NEAT inputs, outputs, and outcomes Information on NEAT resources and activities Barriers to NEAT implementation and environmental factors that influenced program outputs and outcomes Perceived benefits, strengths, and weaknesses of the program from the perspective of stakeholders Potential alternative uses of NEAT resources, as suggested in KIIs and qualitative survey responses Unintended consequences of NEAT and mitigation strategies that were implemented Types of and reasons for challenges, bottlenecks, and holdups experienced Complete document review and synthesize NEAT project documents Assess performance metrics for NEAT and compare to similar USAID program metrics Create a NEAT logic model and compare to Chemonics’ logic model, if it exists Compile literature review of documentation not provided by USAID and/or Chemonics Compile literature review of Nepal environmental factors over the NEAT timeline Conduct key informant interviews across NEAT’s five components Complete a survey of sampled beneficiary farmers Benchmark NEAT to best practices in international development, particularly agricultural and economic programs Selected definition(s) of appropriateness and effectiveness may bias results, unless established prior to analysis stage Difficulty in comparing to other interventions with any completeness Inability to generalize survey and KII findings Difficulty in aligning environmental factors with influence on program beyond correlation Overall, work on this question will allow us to discuss the extent to which NEAT’s design and implementation promoted inclusive economic growth, with consideration to environmental factors in Nepal. Specifically, we will be able to discuss stakeholder and beneficiary involvement in NEAT’s planning and design, and satisfaction with the program. Additionally, we will be able to assess the program’s strengths and weaknesses, both objectively (via logic model) and comparatively (benchmarked to best practices and other programs). 82 Researchable Question(s) Information Required and Source(s) Scope and Methodology Limitations Implications 2) How effective was the NEAT Program in improving the economic status of women and disadvantaged populations?  food security, access to financial services, productive engagement in the selected agriculture value chains, nutritional outcomes. Information on women and disadvantaged populations’ involvement during NEAT planning and early implementation Successes and challenges experienced in the food security, microfinance, and competitiveness components of the project Perceived benefits, strengths, and weaknesses of the program from the perspective of women and disadvantaged population stakeholders Complete document review and synthesize NEAT project documents Compile literature review of documentation not provided by USAID and/or Chemonics Conduct key informant interviews across NEAT’s five components Complete a survey of sampled beneficiary farmers Separately assess qualitative survey responses from women and disadvantaged population beneficiaries Difficulty in establishing criteria for improvement of economic status (i.e., what degree of positive change is necessary to be considered improvement?) Inability to generalize survey and KII findings Overall, work on this question will allow us to discuss the extent to which NEAT’s design and implementation improved the economic status of women and disadvantaged populations. Specifically, we will be able to discuss women and disadvantaged populations’ involvement in NEAT’s planning and design, and perceptions of the program. 3) How has NEAT’s policy reform work improved the business enabling environment for a private sector led growth? Information on business and government stakeholders’ involvement during NEAT planning and early implementation Information on activities conducted that targeted enabling the business environment, and their implementation methods. Perceived successes in these activities from the perspective of business leaders and government officials. Established criteria for success in this component. Unintended consequences of NEAT and mitigation strategies that were implemented Types of and reasons for challenges, bottlenecks, and holdups experienced Complete document review and synthesize NEAT project documents Compile literature review of documentation not provided by USAID and/or Chemonics Conduct key informant interviews with stakeholders in the business environment and government officials Compile literature review of Nepal environmental factors over the NEAT timeline Comparison using outside measures (i.e., World Bank “Ease of Doing Business”) Difficulty in assessing impact, due to timeframe of the evaluation Inability to generalize KII findings Potentially limited ability to assess success beyond implementation of policies Inability to attribute causation to NEAT Overall, work on this question will allow us to discuss the extent to which NEAT’s design and implementation improved the business enabling environment in Nepal. Specifically, we will be able to discuss government officials and business leaders’ involvement in NEAT’s planning and design, and perceptions of the program. 83 Researchable Question(s) Information Required and Source(s) Scope and Methodology Limitations Implications 4) How appropriate and effective was the program in improving GON systems and capacity for economic growth?  taxation, revenue, customs and trade Information on government officials’ involvement during NEAT planning and early implementation Information on activities conducted that targeted GON capacity, and their implementation methods. Perceived successes in these activities from the perspective of government officials. Established criteria for success in this component. Unintended consequences of NEAT and mitigation strategies that were implemented Types of and reasons for challenges, bottlenecks, and holdups experienced Complete document review and synthesize NEAT project documents Compile literature review of documentation not provided by USAID and/or Chemonics Conduct key informant interviews with government officials Compile literature review of Nepal environmental factors over the NEAT timeline Comparison using outside measures (i.e. Heritage “Index of Economic Freedom”) Difficulty in assessing impact, due to timeframe of the evaluation Inability to generalize KII findings Potentially limited ability to assess success beyond implementation of policies Overall, work on this question will allow us to discuss the extent to which NEAT’s design and implementation improved GON systems and capacity. Specifically, we will be able to discuss government officials’ involvement in NEAT’s planning and design, and perceptions of the program. 5) How has the program enhanced the competitiveness of the selected agriculture value chains (lentil, ginger, vegetables, and tea)?  production, productivity, processing and exports Information on value chain participants’ involvement during NEAT planning and early implementation Information on activities conducted that targeted value chain competitiveness, and their implementation methods Perceived successes in these activities from the perspective of value chain stakeholders Established criteria for success in this component. Unintended consequences of NEAT and mitigation strategies that were implemented Types of and reasons for challenges, bottlenecks, and holdups experienced Complete document review and synthesize NEAT project documents Compile literature review of documentation not provided by USAID and/or Chemonics Conduct key informant interviews with value chain stakeholders Conduct a survey of farmers in selected value chains Compile literature review of Nepal environmental factors over the NEAT timeline Comparison using outside measures (i.e. export data) Difficulty in assessing impact, due to timeframe of the evaluation Inability to generalize KII and survey findings Potentially limited ability to assess success in measures such as exports, due to scale of intervention Difficulty in establishing causality, because of potential to be affected by outside Overall, work on this question will allow us to discuss the extent to which NEAT’s design and implementation improved the competitiveness of selected value chains. Specifically, we will be able to discuss value chain participants’ involvement in NEAT’s planning and design, and perceptions of the program. 84 Researchable Question(s) Information Required and Source(s) Scope and Methodology Limitations Implications factors 6) What are the Inability to generalize prospects for KII and survey sustainability of the findings end results produced by the NEAT Respondents may be Overall, work on this program? Established criteria for success in sustainability Complete document review and synthesize NEAT project documents unwilling to disclose that they do not intend to continue NEAT’s question will allow us to discuss the extent to which NEAT’s design Progress in the implementation of project activities that are tied to long-term outcomes Successes and challenges in the implementation of Compile literature review of documentation not provided by USAID and/or Chemonics activities Unable to assess long￾term outcomes and and implementation produced sustainable end results. these activities Perceived knowledge transfer, along with Conduct key informant interviews with value chain stakeholders impacts, due to timeline of evaluation Specifically, we will be able to discuss stakeholders and quantifiable measures of knowledge transfer and its quality Conduct a survey of farmers in selected value chains Ideal criteria for knowledge transfer would involve post- beneficiaries’ expectations for continuation of NEAT training and/or post- activities after the activity test of program has ended. knowledge gained, which may not be available 85                                                              Appendix 6: NEAT Districts that Received Intervention Nepal NEAT Districts that Received Intervention30 Target Districts (21) NEAT Non-Target Districts (31) Arghakanchi Baglung Banke Bara Bardiya Bhaktapur Dailekh Chitwan Dang Dhading Dhankuta Dhanusha Ilam Dolakha Jajarkot Ghorka Jhapa Khanchanpur Kailali Kaski Kapilbastu Kathmandu Morang Kavrepalanchok Palpa Lalitpur Panchthar Lamjung Pyuthan Makawanpur Rolpa Nawalparasi Rukum Nuwakot Rupandehi Okhaldhunga Salyan Parbat Surkhet Parsa Terhatum Ramechhap Rasuwa Rautahat Saptari Sarlahi Sindhuli Sindhupalchwok Siraha Syangja Tanahu Udaypur 30 Districts that received interventions were identified using the Disaggregated Data documents provided by USAID/Nepal. 86