Food For Progress Post-Project Sustainability Assessment Multi-Country Evaluation December 2019 This publication was produced at the request of the United States Department of Agriculture. It was prepared independently by The University of Minnesota. Foreign Agricultural Service, United States Department of Agriculture Five pictures from left to right: a picture of tomatoes, a picture of two men that were interviewed, a picture of a man in a field showing the roots of a plant, a picture of a woman holding coffee beans, and picture of a field planted with unidentified row crops. USDA Food for Progress Post-Project Sustainability Assessment This is an assessment of the sustainability of three already-closed United States Department of Agriculture (USDA) Food for Progress (FFPr) projects in Benin, Guatemala, and Sri Lanka. Conducted during 2017 to 2019 by the University of Minnesota (UMN), the purpose of this sustainability assessment is to understand which project activities have continued after USDA’s Food for Progress funded involvement ended, and what impacts are beneficiaries experiencing as a result. This report highlights the preliminary conditions, design elements, implementation strategies and project activities that are likely to facilitate the sustainability of Food for Progress project activities and impacts after a project has closed. Evaluation Authored by: The University of Minnesota DISCLAIMER: The author’s views expressed in this publication do not necessarily reflect the views of the United States Department of Agriculture or the United States Government. Acknowledgments The University of Minnesota thanks all partners and participants of this assessment. The UMN team was led by; James Bradeen PI, Britta Hansen Project Coordinator with research support from faculty members; Kota Minegishi, Dean Current, Julie Grossman, Paul Porter, George Annor, John Vreyens, Dean Malvik and David Wilsey. Graduate student researchers were Megan Butler, Didier Bolanos and Marie Schaedel. The University of Minnesota worked closely with The Improve Group for project design, evaluation methodology and data collection this team consisted of Becky Stewart, Sara S. McGarraugh, Moira Gaidzanwa, Kayla Meyers, Isabel Marsh and Claire Stoscheck. In Sri Lanka a dedicated team from Eco Lanka carried out all data collection and logistics under difficult circumstances this team was led by Sashan Rodrigo (Project Coordinator) and Randika De Mel (Evaluation Consultant/Team Lead) with support from Niroshi Perera (Field Coordinator and Notetaker), Chamara Prabhath (Note taker), Udeshika Jayapali (Researcher), Hasithi Samarasinghe (Researcher), Piroshini Trikawalagoda (Senior Evaluation Consultant), Nishedha Indraratne (Research Assistants), Sanuri Trikawalagoda (Research Assistants), Piyal Chandana (Driver) and W.H.A Rupasinghe (Driver). In Guatemala the project was greatly assisted by Adriana Smith (local field team lead), Rony Arriola (driver / ag specialist), Sydney Nilan (translation) and Marlon Motha (driver).  In Benin our US based team worked with Odin Monnou to set up site visits, coordinate logistics and provide translation. Food For Progress Implementing Partners greatly aided in coordinating field visits, connecting researchers to local experts and stakeholders, providing time and space for interviews and accommodating the research teams requests for documentation and literature. Many thanks to headquarters and field staff at Counterpart International, Partners for Development and World Vision International. And finally, the University of Minnesota would like to acknowledge the generous support of USDA/FAS for soliciting and financing this post project sustainability assessment. I Table of Contents Executive Summary��������������������������������������������������������������������������������������������������������������������������������������������������� 1 Post-Project Sustainability Assessment�������������������������������������������������������������������������������������������������������6 Project Background��������������������������������������������������������������������������������������������������������������������������������������������6 Evaluation Approach, Methods and Limitations�������������������������������������������������������������������������������8 Data Collection Methods and Rationale��������������������������������������������������������������������������������������������� 10 Findings ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������18 Conclusions ����������������������������������������������������������������������������������������������������������������������������������������������������������������27 Recommendations ������������������������������������������������������������������������������������������������������������������������������������������������30 CASE STUDY — Benin�������������������������������������������������������������������������������������������������������������������������������������������33 Summary ���������������������������������������������������������������������������������������������������������������������������������������������������������������34 Evaluation Purpose������������������������������������������������������������������������������������������������������������������������������������������34 Project Background�����������������������������������������������������������������������������������������������������������������������������������������36 Evaluation Methods & Limitations����������������������������������������������������������������������������������������������������������38 Findings �������������������������������������������������������������������������������������������������������������������������������������������������������������������42 CASE STUDY — Guatemala�������������������������������������������������������������������������������������������������������������������������������59 Summary���������������������������������������������������������������������������������������������������������������������������������������������������������������� 60 Evaluation Purpose ������������������������������������������������������������������������������������������������������������������������������������������61 Evaluation Questions ��������������������������������������������������������������������������������������������������������������������������������������62 Project Background�����������������������������������������������������������������������������������������������������������������������������������������63 Findings ��������������������������������������������������������������������������������������������������������������������������������������������������������������������71 Conclusions�����������������������������������������������������������������������������������������������������������������������������������������������������������88 II CASE STUDY — Sri Lanka �����������������������������������������������������������������������������������������������������������������������������������95 Summary ���������������������������������������������������������������������������������������������������������������������������������������������������������������96 Evaluation Purpose������������������������������������������������������������������������������������������������������������������������������������������97 Project Background ����������������������������������������������������������������������������������������������������������������������������������������98 Findings �����������������������������������������������������������������������������������������������������������������������������������������������������������������105 Conclusions��������������������������������������������������������������������������������������������������������������������������������������������������������� 118 Annexes������������������������������������������������������������������������������������������������������������������������������������������������������������������������126 Annex I — Evaluation Statement of Work����������������������������������������������������������������������������������������126 Annex II — Assessment proposal����������������������������������������������������������������������������������������������������������135 Annex III — Round 1 Site selection criteria�������������������������������������������������������������������������������������� 147 Annex IV — Round 2 Site selection criteria ����������������������������������������������������������������������������������� 150 Annex V — Evaluation Approach, Methods and Limitations �����������������������������������������������152 Annex VI — Data Collection Instruments����������������������������������������������������������������������������������������� 161 III ACH Acción contra el Hambre (Spanish acronym for Action Against Hunger) CADER Centros de Aprendizaje para el Desarrollo Rural (Spanish acronym for the Rural Development Learning Center) CCAM Caisses de Crédits Agricole Mutuels (French acronym for Mutual Agricultural Credits) CIA The Central Intelligence Agency from the United States of America CLCAM Caisse Locale de Crédit Agricole Mutuel (French acronym for Crédit Agricole Mutuel Local Fund) COP Chief of Party CRM-MC Regional Council of Market Gardeners of Mono-Couffo DAC Development Assistance Committee ECI Gestión para el Desarrollo Humano (spanish acronym for Management for Human Development) ESOKO Ghanaian internet services FAO Food and Agriculture Organization FECECAM Fédération des Caisses d’Epargne et de Crédit Agricole Mutuel (French acronym for Federation of Savings Banks and Mutual Agricultural Credit) FFPr Food For Progress GA Giribawa, Sri Lanka GAP good agricultural practices GDP Gross Domestic Product GREEN The Growing Resources for the Enhanced Agricultural Enterprises and Nutrition (GREEN) IBTCI International Business & Technical Consultants, Inc IFAD International Fund For Agricultural Development IG Improve group IICEM Integrated Initiatives for Economic Growth in Mali program INRAB Ministry of Agriculture’s National Institute for Agricultural Research IPs implementing partners KDAD Feed the Future Knowledge-Driven Agricultural Development project MAGA Ministerio de Agricultura, Ganadería y Alimentación de Guatemala (Spanish acronym for Ministry of Agriculture, Livestock and Food) Acronyms IV MICOOPE Federación nacional de cooperativas de Guatemala (Spanish acronym for National Federation of Cooperatives in Guatemala) MFI Micro Finance Institution MSU Michigan State University NGO Non-Governmental Organization OCBD The Office of Capacity Building and Development OCBD The Office of Capacity Building and Development OECD The Organization for Economic Cooperation and Development OFC farmers cultivated other field crops OPA Professional Agricultural Organizations PARI Program of Accompanying Research and Agricultural Innovation PfD Partners for Development PTAA The Agriculture and Food Technology Program PVO private voluntary organizations SPSS software package used for interactive, or batched, statistical analysis. SWAT Soil & Water Assessment Tool SWOT Strategic planning technique analysis to identify strengths, weaknesses, opportunities, and threats. TH Thanamalvilla, Sri Lanka UC, Davis University of California, Davis UMN University of Minnesota URP-AL Regional Associations of Farmers of Farmers of the Atlantique-Littoral URP-OP Regional Associations of Farmers of the Ouémé-Plateau USAC Universidad de San Carlos (Spanish acronym for University Saint Carlos) USDA United States Department of Agriculture USDA-FAS United States Department of Agriculture -The Foreign Agricultural Service USDA-FFPr United States Department of Agriculture-Food For Progress VMIS information system on the current price of market garden produce WFP World Food Program WOCCU World Council of Credit Unions WVI World Vision International V Evaluation — “The systematic and objective assessment of an on-going or completed project or Programme, its design, implementation and results. The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact and sustainability”. (OECD/DAC Glossary, 2009) Evaluation Sample — Is the unit to be studied from the broader population of interest. In this case project beneficiaries from 3 specific FFPr projects. Evaluation Team — Stakman-Borlaug Center (SBC) for Sustainable Plant Health, University of Minnesota, The Improve Group. Food for Progress Project — A United States Department of Agriculture, Foreign Agriculture Services, Program that helps developing countries and emerging democracies modernize and strengthen their agricultural sectors. Impacts — “Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended”. (OECD/DAC Glossary, 2009) Implementing Partner — The lead organization contracted by USDA to carry out project activities. Participants — FFPr beneficiaries that were part of the evaluation sample. Preliminary conditions — The environmental, social, political and economic conditions that characterized a region in the time before the implementation of the project. Project Activities — Refers to components of the FFPr project. Including trainings, coordination, distribution of materials, workshops. Project design — refers to the plan of activities selected to meet the specific project objectives. Project impact — It refers to the tangibles and intangible perceived benefits by the participants of the projects after applying practices and lessons from the project. Project site — Specific regions where a project took place. Result — “The output, outcome or impact (intended or unintended, positive and/or negative) of a development intervention. Related terms : outcome, effect, impacts”. (OECD/DAC Glossary, 2009) Terms VI Stakeholders — Agencies, organizations, groups or individuals who have a direct or indirect interest in the development intervention or its evaluation. Sustainability(1) — “Assessment of the likelihood that the benefits of the project will endure over time after the completion of the project. Sustainability should also assess the extent to which the project has planned for the continuation of project activities, developed local ownership for the project, and developed sustainable partnerships.” (USDA-FAS, 2013) Sustainability(2) — “The continuation of benefits from a development intervention after major development assistance has been completed. The probability of continued long￾term benefits. The resilience to risk of the net benefit flows over time”. (OECD/DAC Glossary, 2009) Executive Summary | 1 Executive Summary Evaluation Purpose and Evaluation Questions This report highlights the preliminary conditions, design elements, implementation strategies and project activities that are likely to facilitate the sustainability of Food for Progress, project activities and impacts after a project has closed. Evidence is presented from three case studies; Benin, Guatemala and Sri Lanka. For the purposes of this study we define sustainability as, the continued benefits resulting from the development intervention after project assistance has been completed. In order to pinpoint the components of a project that have been sustainable this assessment will also determine factors that influence the likelihood of sustainability. The factors could include conditions in country before the project started or the design of the project and its implementation. The 5 main evaluation questions, and additional sub questions are as follows: 1. How did existing conditions affect project sustainability? A. What are the preliminary conditions that set up market linkages for success? B. To what extent are results more sustainable in low, or middle-income countries? 2. What components of the project’s design contribute to sustainability? A. To what extent were local stakeholders involved in project activity design? 3. In what ways did the implementation of the project contribute to its sustainability? A. How do local partnerships contribute to sustained project outcomes and impact? B. What factors lead to training results being sustained? C. How does the combination and sequencing of activities influence sustainability? 4. What activities or practices have continued (and to what extent) since the project ended? A. How valuable to communities are the project’s outcomes, including activities and practices? 5. What are the impacts of the project? A. How valuable to communities are the impacts of the project? B. How did beneficiaries and their communities respond to subsequent shocks? C. To what extent have the impacts of the project contributed to equity? Data collection in progress Project Background FFPr sought to undertake a post-project assessment of the sustainability of 3 closed FFPr projects, one in each of these three regions: Latin America and Caribbean region, Asia, and Africa. The 3 projects to be evaluated were identified by the University of Minnesota and consultants with guidance and final decisions by USDA/FAS. After a careful review of closed projects based on criteria, including: the number of years since the project closed, the types of activities carried out during the project, the presence of the implementing partner in the country and the availability of monitoring data from the project. Design and Implementation A 5-phase evaluation strategy was used that allowed for collaboration between partners at every step of the project. The five phases included 1) Inception, 2) Design & Planning, 3) Data Collection, 4) Data Analysis, and 5) Reporting. Methods were chosen based on multiple factors, including: cultural sensitivity, efficacy in answering evaluation questions, availability of data and feedback from implementing partners. These methods were: Image grouping, In-depth interviews, Observations, and Document review. 2 | Executive Summary  Executive Summary | 3 A wide range of sources were included in data collection. Overall approximately 400 individuals participated in data collection, via interviews, workshops, survey or observations. The analysis was completed with a keen eye toward using the data to inform decisions and future work for the program. Qualitative and quantitative analysis methods were employed to identify and categorize themes to identify findings that are: • Prevalent: common across multiple programs and respondents. • Insightful: unique to a few respondents, but share a perspective that is important to your work. • Illustrative: in-depth comments that do a particularly good job of describing a concept or idea. Findings and Conclusions In order to identify the types of activities that have continued across all three cases. Project activities across all three cases were coded into the three categories, with two additional categories containing data primarily from only one country each. Organizational Capacity Development (data from Guatemala only) and Project Provided Inputs and assets (data primarily from Sri Lanka). This categorization allowed analysis across all cases, even when project activities were somewhat different between each case study. Activities that have continued since the projects ended fall in three main categories; • Technical training activities related to general production management practices, training on vegetable and crop production, irrigation, safe and efficient application of inputs, post-harvest practices, coffee production, how to access technical trainers and information, soil conservation, fruit tree production and animal husbandry. • Value chain activities such as improving access to market information, connecting growers to buyers, creating marketing plans, facilitating value chain linkages. • Business development activities include; creating farm planting and marketing calendars, writing a business plan, training in financial management and improving access to loans and credit. The cross site findings highlight these activity categories that have continued, as well as the impacts that have resulted from continuing the activities. The report finds that project beneficiaries are most likely to continue to implement improved agricultural technical practices learned through project training. Table 1 below shows the number of times former project beneficiaries said they learned an activity that fell in the “technical training” category followed by the number of total times they said they are continuing that activity now that the project has ended. Data from all three case studies show that farmer beneficiaries continue to use what they learned from technical training the most often, around 94% of those trained in various technical activities are continuing to use what was learned from the project. 90% of project beneficiaries across all three cases studied are highly likely to continue to use what they learned related to market access and value chains. Less sustainable across all cases studied were business development activities with less than half of participants who learned about improving their farm business practices still using those skills, only 40% say they are. 4 | Executive Summary  General activities, coded the same across countries Combined number of mentions that the activity was learned from the FFPr project. Combined number of mentions that the activity learned from the FFPr project is still used. Percent who continue to use what they learned from the FFPr project. Technical trainings 1344 1267 94% Value chain improvement 270 244 90% Business development 732 295 40% Organizational capacity development (data from Guatemala only) 207 241 116% Project provided inputs and assets (data primarily from Sri Lanka) 150 128 85% Table 1 — Activity categories that have continued since the projects ended. The impacts experienced by project beneficiaries as a result of continuing project activities, across all three cases are shown in the table below. An increases in income was the most common impact across projects, mostly as a result of continued technical training, chart1. Value chains Technical training Business development 0 Increased income Better access to markets Pride More on farm crop diversity Better Health Improved nutrition (more or better food) Time gains Increase in production 100 200 300 400 500 Impacts resulting from value chain, technical, and business development activies in Guatamala, Sri Lanka and Benin Impacts Number of Respondents Reporting Impact Chart 1 — Impacts from continued activities across 3 case studies. Executive Summary | 5 This study identifies which activities have continued after the completion of the project, and their impacts. This analysis of three case studies, which involved a review of project reports, literature and primary data collected, highlights 6 specific findings that when present increase the likelihood of the sustainability of a project. 1. Existing assets — Build a project around existing community assets and needs. 2. Environment — Address environmental and physical constraints 3. Sequencing — USe effective component sequencing and integration. 4. Existing systems — Strengthen existing systems. 5. Value chain development — Balance investment of developing supply, understand demand, and create linkages. 6. Stakeholder involvement — Enable community-informed planning and design. First, projects that support existing community assets such as knowledge and natural resources are more likely to be sustainable, and projects that address existing environmental and resources constraints are more valued and sustainable in target communities. Second, the intentional and logical sequencing of project activities and inputs lead to more sustained practices by beneficiaries. Third, by strengthening existing systems and organizations projects are more likely to have long term impact and beneficiaries are more likely to be able to access these systems into the future. Forth, projects that understand demand and are able to parlay that into improving supply and facilitating market linkages are more likely to have sustainable results, and be valued by beneficiaries. Finally, projects that secured community and stakeholder buy-in via community informed planning show increased likelihood for sustainability, however this finding was not as strong across the board as others. Based on these findings, recommendations to increase the likelihood of sustainability of future Food for Progress projects include the following: 1. Projects should be required to complete a sustainability assessment as a part of their response to the RFP, that would include a comprehensive analysis of the existing social, economic, cultural, environmental, and political environments that the project or intervention will function within, highlighting specific conditions that are conducive to sustainability. 2. Project activities should directly respond to existing market demand, gaps in market linkages and barriers to supply, utilize existing organizations and respond to organizational capacity needs, community needs, and environmental and agricultural assets. 3. Enable the combination of training, access to finance, and connection to markets. This has been demonstrated to be a “triple threat” to poverty alleviation. Allow for financing to support improved production and connecting to an improved market system. 4. Project activities need to be rolled out in a logical order and in combinations that work together and strengthen the impacts of activities. 6 | Post-Project Sustainability Assessment  | Project Background Post-Project Sustainability Assessment Project Background The Office of Capacity Building and Development (OCBD) is a program area of USDA’s Foreign Agricultural Service that is dedicated to advancing global trade, agricultural development and food security. Programs administered by OCBD help to reduce food insecurity, strengthen agricultural economic growth in targeted developing countries along the full value chain of development, build their capacity to integrate into the global economy, and become viable agricultural trading partners with the United States. The objective is to help move these countries along the agricultural market-development continuum from developing economies, where capacity building is the primary focus, to developed economies, where market development and expansion are the primary focus. Authorized by the Food for Progress Act of 1985, the Food for Progress (FFPr) program provides for the donation of U.S. agricultural commodities to developing countries committed to introducing and expanding free enterprise in the agricultural sector. Donated commodities are monetized (sold on the local market) and the proceeds are used to support agricultural development activities. Food for Progress has two principal objectives: improve agricultural productivity and expand trade of agricultural products. Food for Progress programs have worked to train farmers in animal and plant health, improve farming methods, develop infrastructure (roads, electricity, storage, etc.), establish and strengthen producer cooperatives, provide microcredit, and develop agricultural value chains. FFPr Program Participants have included private voluntary organizations (PVO), foreign governments, universities, and intergovernmental organizations. FFPr projects are currently active in 25 countries across Africa, Asia, and Latin America and the Caribbean (figure 1). The Organization for Economic Cooperation and Development (OECD), Development Assistance Committee (DAC) has defined “sustainability”, in the context of evaluation and results-based management as: The continuation of benefits from a development intervention after major development assistance has been completed. The probability of continued long-term benefits. The resilience to risk of the net benefit flows over time. Post-Project Sustainability Assessment | Project Background | 7 The Food Assistance Division, in its Monitoring and Evaluation Policy defines “sustainability” as: Assessment of the likelihood that the benefits of the project will endure over time after the completion of the project. Sustainability should also assess the extent to which the project has planned for the continuation of project activities, developed local ownership for the project, and developed sustainable partnerships. Many FFPr projects have the long-term goal of sustainability. Two key facets of sustainability are relevant to many FFPr projects: 1. Many training and other collaborative project activities ultimately aim to build the capacity of participants to continue these activities, practices or behaviors themselves, beyond the life of the project 2. Positive results experienced by beneficiaries during the life of the project are often intended to continue to be experienced beyond the life of the project As part of the Office of Capacity Building and Development Results Oriented Management Policy, FFPr sought to undertake a post-project assessment of the sustainability of 3 closed FFPr projects, one in each of these three regions: Latin America and Caribbean region, Asia, and Africa. The 3 projects to be evaluated were identified by the University of Minnesota and consultants with USDA after a careful review of closed projects based on a number of criteria, including: the number of years since the project closed, the types of activities carried out during the project, the scope of the relationship between FFPr and the project’s implementer, and the availability of monitoring data from the project. Figure 1 — Map of current USDA food assistance projects from FAS.gov 8 | Post-Project Sustainability Assessment  | Evaluation Approach, Methods and Limitations Project beneficiary in the field Purpose The purpose of this sustainability assessment is to understand which project activities have continued after USDA’s Food For Progress funded involvement ended, and what impacts are beneficiaries experiencing as a result. In order to pinpoint the components of a project that have been sustainable this assessment will also determine factors that influence the likelihood of sustainability. The factors include conditions in country before the project started, the design of the project and its implementation. This is only the second time that FFPr has returned after a project has ended to assess the sustainability of the project’s activities. The purpose of this assessment is to create a base evidence of knowledge for USDA/FAS to design and fund effective and sustainable projects in the future. Evaluation Approach, Methods and Limitations Evaluation Question Selection and Refinement The evaluation team used a participatory approach to designing evaluation questions. In November of 2017 USDA staff participated in a workshop facilitated by the evaluation team to develop evaluation questions and create a site selection rubric that would generate the information they desired from the study. Once sites and projects were selected implementing partners (IPs) of selected projects were then given an opportunity to review and comment on the proposed evaluation questions. Evaluation Questions 1. How did existing conditions affect project sustainability? A. What are the preliminary conditions that set up market linkages for success? B. To what extent are results more sustainable in low, or middle-income countries? Post-Project Sustainability Assessment | Evaluation Approach, Methods and Limitations | 9 2. What components of the project’s design contribute to sustainability? A. To what extent were local stakeholders involved in project activity design? 3. In what ways did the implementation of the project contribute to its sustainability? A. How do local partnerships contribute to sustained project outcomes and impact? B. What factors lead to training results being sustained? C. How does the combination and sequencing of activities influence sustainability? 4. What activities or practices have continued (and to what extent) since the project ended? A. How valuable to communities are the project’s outcomes, including activities and practices? 5. What are the impacts of the project? A. How valuable to communities are the impacts of the project? B. How did beneficiaries and their communities respond to subsequent shocks? C. To what extent have the impacts of the project contributed to equity? Technical Approach A 5-phase evaluation strategy was used that allowed for collaboration at every step of this project. The five phases included 1) Inception, 2) Design & Planning, 3) Data Collection, 4) Data Analysis, and 5) Reporting. See Annex V for additional details on each phase of the approach and site selection. Project Inception During this phase, the guiding frameworks for the evaluation were established, the evaluation questions and priorities were refined and sources of data and methods were identified. Multiple rounds of feedback were gathered between USDA and the implementing partners. Design & Planning for project selection In this phase, the team planned and prepared all of the processes and materials to be used in the evaluation. Using insights from stakeholder interviews and USDA/FAS, the initial evaluation ideas were developed into project selection rubrics and plans of action. USDA/FAS then provided the evaluation team with a list of 13 projects that met these criteria. An iterative process was used to exclude projects; 5 criteria were ultimately narrowed down for the project selection, via exclusion. These criteria were then combined to create a list of projects that could hypothetically provide some comparison against each other. Six projects were determined to meet the requirements through a review of implementing partner award contracts and initial interviews with partner staff. 10 | Post-Project Sustainability Assessment  | Data Collection Methods and Rationale After the final criteria were established, in June 2018, three closed projects – one for each category – were selected by USDA to be a included in the post project Sustainability assessment: • Guatemala 2012 • Sri Lanka 2005 • Benin 2011 • One backup project was also identified in case a mid-course change was needed Evaluation plans were prepared for each case study, the team received and incorporated feedback from USDA/FAS and implementing partners. These plans included timelines, activities, methods and tools, data security procedures, and other needed content. Draft data collection tools were created including an evaluative rubric, interview protocols, and multi-step site visit protocols. Interviews were conducted with stakeholders in country via phone and Skype to further refine the evaluation questions, methods and sources and to begin to narrow down culturally appropriate evaluation methods. Alongside evaluation question development, USDA-FAS staff informed the proposed methodology by articulating desired goals of the final results. The first design phase of data collection further informed the evaluation plans, built relationships with partners in country and facilitated the logistics of data collection. The input received during the design phase helped guide how local enumerators would be used, interpreters and experts; collaborating with local staff and consultants helps ensure that there is an understanding of a fuller picture of results. Data Collection Methods and Rationale Methods were chosen based on multiple factors, including: cultural sensitivity, efficacy in answering evaluation questions, availability of data and feedback from implementing partners. The same methods were used in all sites to maintain data consistency for comparison across countries. These methods were: • Image grouping • In-depth interviews • Observations • Document review Local coordinators and the implementing partner in each country identified and recruited prospective participants. The evaluation team—of UMN faculty, graduate students, and Improve Group consultants—conducted in-person interviews, field observations and facilitated Image Grouping© sessions in Guatemala and Benin. A local Sri Lankan evaluation firm conducted these activities in Sri Lanka following a terrorist attack in May 2019, the time period that was scheduled for data collection. Image Grouping© Image Grouping© is a participatory mixed-methods data collection tool that helps increase engagement in research, evaluation, needs assessments, and other kinds of data collection. Image Grouping© is used to overcome literacy barriers, particularly in locations where local languages are not commonly written or participants likely have a low level of formal education. Post-Project Sustainability Assessment | Data Collection Methods and Rationale | 11 Workshop participant in Huehuetenango Guatemala using the Image Grouping© data collection tool developed for Guatemala. Image Grouping© As a mixed-methods approach, Image Grouping© can provide both qualitative and quantitative data; an initial activity uses an image-based tool whereby participants use stickers to respond to survey-like questions. In-depth interviews As the name implies, in-depth interviews provide rich and deep knowledge on particular topics. Through anecdotes and detailed stories interviews with key informants have been shown to provide the depth of information one might be able to capture through multiple survey respondents. Interviewers can also ask for more detail than they would otherwise on a survey, providing opportunities for clarification of data. Interviewees helped to describe the design and implementation strategies used during the award, providing more context than program documents held. Interviewees further helped answer questions about the extent to which larger systems change occurred. Observations Observations allow for researchers to collect additional evidence of agricultural practices and technology being used, and to have informal conversations about project activities and farmers experiences with the project. Observation protocols (Annex VI) were created by the evaluation team to be used on all three sites. Another benefit to observations is the lower burden on respondents’ time. 12 | Post-Project Sustainability Assessment  | Data Collection Methods and Rationale Interview with project participant in Grand Popo, Benin Literature review A literature review was conducted at two stages in the evaluation; one, prior to primary data collection to identify factors that increase or decrease the likelihood of sustainability. These factors guided data collection question design and served as a framework for initial analysis. A second stage of literature review occurred that further explored pre-existing conditions relating to sustainability, elements of projects design, and implementation practices. The goal of the review was to determine study areas for exploration, orient data collection and analysis, and provide supporting or refuting secondary evidence of sustainability alongside primary data. Secondary data analysis Project reports; including proposal, midterm and final reports as well as success stories and project summaries were provided to the evaluation team to review and were used to inform research methods, data collection, and analysis. Survey The 2011 Guatemala project is the only one where any type of survey was carried out, the evaluation team chose to implement an online survey for one population of beneficiaries who would not participate in other data collection processes. Post-Project Sustainability Assessment | Data Collection Methods and Rationale | 13 • Extension agents • Farmer Association/Cooperative leaders • Financial institution personnel/loan recipients • Government representatives • Implementing partners • Lead farmers • Partner institutions Throughout the three sites, data collection was administered in 13 languages, including French, Spanish, Mina Eww, Fon, Sinhala, Wémè / Goun. Sample Statistics A wide range of stakeholders from the three sites participated in the sustainability assessment. Implementing partners identified participants that fit various types of stakeholders identified by the evaluation team. The evaluation team conducted outreach with the intention of conducting 25 interviews that would cover all of the needed stakeholder types per site. A few participants participated in multiple forms of data collection within their respective site; as a result, the grand total listed in the Overall Sample Table below does not represent unique individuals that participated in data collection at a site. Field observation in Cotonou, Benin Data Sources A wide range of sources were included in data collection. The evaluation team worked with implementing partners at the three sites to help conduct the evaluation. A range of stakeholders from numerous regions in all three countries were included in the evaluation to ensure a wide range of perspectives were gathered and to triangulate evidence. Stakeholder types include: 14 | Post-Project Sustainability Assessment  | Data Collection Methods and Rationale Image Grouping© participants were selected using a stratified convenience sample. This allowed the evaluation team to select groups based on additional hypotheses relevant to sustainability, such as the relationship between physical distance to markets and sustained outcomes. All Image Grouping© participants are farmers. Each site utilized slightly different criteria for the selection but included attention to: diversity of locations, variety of activity exposure during the project, and a balance of genders. Image Grouping sampling operates on a similar principle as focus groups. In a focus group the accepted number of participants is a maximum of 8 and the desired number of focus groups is the point at which no new themes emerge from additional groups (i.e. reach a saturation point). In our Image Grouping sampling the maximum workshop size was determined to be 20 based off of preliminary interviews with beneficiaries, and 3-4 workshops in various locations in each country were considered adequate with likely no new themes emerging with additional groups. This latter characterization was emerging in the workshops as discussion points repeated. Interviewees were selected based on key informant expertise. Individuals who had direct knowledge of the site’s project participated. Site Image Grouping Participants 4 workshops per site, 12 total Interviews eSurvey Total Total beneficiaries served during project Guatemala 2012-2016 Counterpart International 97 34 107 238* 40,000 Benin 2011-2014 Partners for Development 76 23 99 6,000 Sri Lanka 2005-2013 World Vision 59 26 85 3,043 Grand Total 232 83 107 422 Table 2 — Overall Sample Table *The Guatemala sample is much larger due to the emailed survey that was sent out to extension agents, the survey data are primarily incorporated into the general conclusions about the extension systems interventions in that country. These additional data points were not part of assessing continued activities or impact. Analysis Methods The evaluation team conducted qualitative analysis in Dedoose software to analyze the in-depth interviews and Image Grouping© discussion notes. Initially, the evaluation team developed codes using an inductive approach. These codes were developed by identifying themes from literature on conditions needed for sustainability, as well as the evaluation questions. Following that, a Post-Project Sustainability Assessment | Data Collection Methods and Rationale | 15 deductive approach was used to find new themes based on the responses. The evaluation team lastly looked at relationships between themes and compared differences between different types of respondents, such as farmers and other key stakeholders. Findings are based on codes that meet the following criteria: Substantive — There is a high number of mentions associated with the code, relative to other sub-codes under the same parent code. Inter-ocular — One to only a few mentions are associated with the code, but the perspective is highly relevant or notable. For example if only a few women were interviewed and they mentioned a reduction in family violence, while no men mentioned this project impact, these few mentions would be considered highly relevant and thusly highlighted. Quantitative Image Grouping was analyzed using Excel and SPSS. Frequencies and cross tabs were conducted to understand differences between locations, or the activities that resulted in sustained outcomes. Themes across these responses were compared to those found in the qualitative results and integrated to achieve a mixed method understanding of the data. Survey responses analysis was done using Qualtrics cross-tab feature. Of particular interest beyond descriptive statistics was comparison across employment status, gender, and cohort groups when determining the final results of the project several years after the official project end date. The literature review was conducted to both inform our data collection and compare our findings to existing academic literature as well as program evaluations from other development project reports. Relevant academic literature was identified based upon relevance to agricultural extension, community agricultural development, community capacities, and relevant research in Guatemala, Benin and Sri Lanka. Project reports/evaluations were identified based upon their relevance to the region and their focus on related project objectives and activities. The analysis was completed with a keen eye toward using the data to inform decisions and future work for the program. We used qualitative and quantitative analysis methods to identify and categorize themes to identify findings that are: • Prevalent: Common across multiple programs and respondents; • Insightful: unique to a few respondents, but share a perspective that is important to your work; • Illustrative: in-depth comments that do a particularly good job of describing a concept or idea. Limitations With all evaluations, there are limitations to the data that are gathered and the conclusions that can be drawn. Each of the three sites had limitations based on the unique geographies, demographics and circumstances surrounding each site, which are discussed in their respective sections of this report. It is important to note that at the time of application for FFPr funds, none of these projects were designed with this type of post-project sustainability assessment in mind. At 16 | Post-Project Sustainability Assessment  | Data Collection Methods and Rationale the time of project design and implementation projects tracked outcomes in accordance with their own monitoring, evaluation and learning plans in line with USDA/FAS evaluation policy required at the time of implementation. Recall bias may have occurred due to the time that has passed following the closure of projects. This can lead to participants remembering events at a different scale or forgetting what they did at that time. As a result, some farmers may have reported participation in trainings and services mostly in areas that they still remember and use today, positively biasing findings towards a higher percentage of individuals sustaining practices. Image Grouping© methodology helps to reduce this likelihood by having farmers from similar areas participate in the same session and encouraging them to talk among themselves to remind each other whether they participated in certain activities over others. Also related to the amount of time elapsed between the end of the project and this assessment is that projects that have been closed for longer appear less sustainable than projects closed for a shorter period of time. Positive response bias may have influenced farmers’ responses in two ways. First, in asking how participant lives are today as compared to prior to the project start, farmers may have been more likely to exaggerate the benefits of the project. In many data collection environments, participants aim to “please” data enumerators, responding in a way that they believe the enumerator is hoping for or contributes to their benefit. Farmers may have believed that by extolling the hardship they experienced prior to the project and highlighting how much they continue using today, that this would support future work in their area. Second, implementing partners were used to determine which sites to visit and who to recruit. Partners may have desired to show more successful outcomes for their own audiences as well as potential influence on future funding through USDA. The evaluation team worked to compensate for this potential bias by requesting IPs to suggest sites that had varying degrees of success after being involved with the USDA project. The high rates of sustained practices suggest that this bias may be more prevalent in Guatemala and Benin data. These high rates of continued practices could also result from the short amount of time since the project has ended, 2 years in Guatemala and 4 in Benin. Therefore, information on what challenged or reduced sustainability may be even more beneficial in reviewing to understand what to consider for creating an optimal environment for sustainability. Responses exceeding 100% of beneficiaries continuing practices occurred in all three cases, this happens when more respondents say they are continuing a practice than said they learned about it from the project. For example in Benin, 54 participants said they were taught about safe pesticide handling from the GREEN project and 58 said they are continuing to use the safe pesticide handling practice they learned from the GREEN project. This results in over 100% “continued practice” this could be because the question was loosely interpreted by participants or translated to mean “do you use safe pesticide handling practices?” regardless of who they learned it from, participants may not have specifically remembered the training activity but they feel they are currently doing it, they may have learned from another project participant, or another source. Or participants may have simply been responding in a positive way to “please” data enumerators (see positive response bias above). However in most cases responses exceeding 100% indicate in general that the practice or activity has been a success and is being applied by most or all people who learned about it. Post-Project Sustainability Assessment | Data Collection Methods and Rationale | 17 This project does not include control or comparison communities and thus cannot make claims about magnitude of impact or causality. As a result, although we were able to describe if income changed for participants and whether they attribute this to their respective projects, we do not know the extent to which income changes were solely the result of the USDA project. Likewise, secondary data did not exist at a geographic level similar to the implementation areas to demonstrate need or wealth status prior to project start. Therefore, we are unable to make claims about economic changes to regions where USDA projects were implemented. 18 | Findings   | Cross Cutting Findings, Conclusions and Recommendations • Technical training activities in all three projects touched on; general production management practices, training on vegetable and crop production, irrigation, safe and efficient application of inputs, post-harvest practices, coffee production, how to access technical trainers and information, soil conservation, fruit tree production and animal husbandry. • Value chains activities conducted in all three projects include; improving access to market information, connecting growers to buyers, creating marketing plans, facilitating value chain linkages. • Business development activities conducted in all three projects include; creating farm planting and marketing calendars, writing a business plan, training in financial management and improving access to loans and credit. Data from all three case studies show that farmer beneficiaries continue activities under the technical training theme most often, around 94% of those trained in various technical activities are continuing to use what was learned from the project. Training related to improving access to markets and value chains have also continued at 90%. Less sustainable across all cases studied were the business development activities with less than half of participants who learned about improving their farm business practices still using those skills, only 40% say they are. Findings  | Cross Cutting Findings, Conclusions and Recommendations | 19 The Guatemala project was the only one which made specific and articulated efforts towards improving the organizational structures of their co-ops and businesses of which beneficiaries were a part. In Guatemala all activities related to improving the organizational capacity of various cooperatives and businesses seem to have been quite successful and sustained outside of the confines of the project, with more people saying they are using these organizational tools than were even trained, 116% (see the limitations section for a possible explanation of these results). Sri Lanka was the only project that had articulated activities at the design phase related to the provision of goods, the project provided tree seedlings, water and irrigation systems and other inputs of project participants. Those who received these goods were very likely to still be using them, around 85% of those surveyed are still using something that was provided to them by the project. Our qualitative data from both Sri Lanka and Benin (provided some equipment as a part of some sub-activities) show that when participants were provided with goods (seedlings, seeds, inputs, equipment), and trained well in their use they continue to use and value the contribution. When the goods are low cost, functional and accessible to participants they will also purchase more on their own, see the case of backpack sprayers and protective clothing in Benin. Projects that worked through multiple channels to increase knowledge and change behavior around value chains and marketing found that participants continued to access project facilitated market linkages (for sale of agricultural goods). Less successful and not as likely to be continued by beneficiaries were activities related to business planning, business management and financial planning. General activities, coded the same across countries Combined number of mentions that the activity was learned from the FFPr project. Combined number of mentions that the activity learned from the FFPr project is still used Percent who continue to use what they learned from the FFPr project. Technical trainings 1344 1267 94% Value chain improvement 270 244 90% Business development 732 295 40% Organizational capacity development (data from Guatemala only) 207 241 116% Project provided inputs and assets (data from Sri Lanka only) 150 128 85% Table 3 — Activities that have continued since the projects ended The level of activity continuation of project activities is calculated by averaging the percentage of project participants who continue project activities, across all activities in that project. This rough number goes to illustrate in general terms the percentage of people in this post project study that are continuing with some activities they learned from the project. The Sri Lanka project closed 20 | Findings   | Cross Cutting Findings, Conclusions and Recommendations in 2010 and on average has a 72% activity continuation, the Benin project closed in 2015 and has average activity continuation of 94%, while the FFPr project in Guatemala ended in 2016 has 106% activity continuation. These data serve to illustrate that the sooner a post project sustainability assessment is conducted more beneficiaries are likely to be continuing to use what they learned from the project. These data could imply that the activities implemented in Guatemala were more sustainable overall, based on our triangulation between these results and our interviews, the former is a more likely conclusion. It is our conclusion that 2-3 years post-project is too soon to collect accurate data on the sustainability of project activities and 9-10 years is likely too long to reasonably expect that project participants will be able to accurately recall what they learned from a specific project and communicate the impact. Different types of longitudinal evaluation methodologies can certainly be employed to more accurately assess impact at different times post-project, for example an experimental design that is built into project implementation and not designed adhoc. Value chains Technical training Business development 0 Increased income Better access to markets Pride More on farm crop diversity Better Health Improved nutrition (more or better food) Time gains Increase in production 100 200 300 400 500 Impacts resulting from value chain, technical, and business development activies in Guatamala, Sri Lanka and Benin Impacts Number of Respondents Reporting Impact Chart 2 — Impacts from continued activities across 3 case studies The various impacts that have resulted from these three FFPr projects generally fall into the above categories. The clearest finding of the greatest impact across the cases is the increase in income, which most participants cited as an impact of continuing most all project activities. In Benin for example, the improved integration into and connections with local and regional markets has led to increases in income for some beneficiaries. In Guatemala, Project beneficiaries explain that increased income is due to both increased quantity and quality of goods sold as well as money saved through the resources obtained through the project, such as greenhouses and tools, as well as more cost-efficient practices. Farmers also reported that learning about commercialization Findings  | Factors that Facilitate or Increase the likelihood of Project Sustainability  | 21 and increased access to financial services helped lead to increases in income. In Sri Lanka project participants are continuing to grow tree crops which they received technical training in and cite their ability to generate long-term reliable income through those efforts. Combining annual crop production with perennial tree production has provided participants with multiple reliable sources of income occurring throughout the year, improving both the diversity and resilience of their farming systems. Factors that Facilitate or Increase the likelihood of Project Sustainability The three cases studied provide an assessment of the likelihood that the benefits of the project will continue after the completion of the project. Our findings across these three studies were developed through a review of project reports, literature and primary data; and highlight 6 specific factors that when present increase the likelihood of project sustainability: 1. Existing assets: Projects that support existing community assets such as knowledge and natural resources have been more likely to enjoy sustained impacts. Lesson learned: Build a project around existing community assets and needs. 2. Environment: Projects that address existing climate, soil, topography and/or other resource￾related environmental constraints are more valued and sustained in target communities. Lesson learned: Addressing environmental and physical constraints is key to sustainability 3. Sequencing: The intentional and logical sequencing of project activities and inputs has led to more sustained practices by beneficiaries. Lesson learned: Effective component sequencing and integration leads to practices learned during the project being sustained. 4. Existing systems: By strengthening existing systems and organizations, projects are more likely to have long term impacts and beneficiaries are more likely to be able to access these systems into the future. Lesson learned: Strengthen existing systems. 5. Value chain development: Projects that understood market demand and were able to parlay that knowledge into improving supply and facilitating market linkages were more likely to have sustained results, and be valued by beneficiaries. Lesson learned: It is important to invest in developing supply, understanding demand, and creating linkages. 6. Community-informed: Projects that secured community and stakeholder buy-in via community informed planning showed some increased likelihood that project benefits would be sustained, for increased sustainability, however this correlation finding was not as strong across the board as others. Lesson learned: community-informed planning and design may increase sustainability. 1 Existing Assets Projects that were built around existing community assets are more likely to have a sustainable long term impact. In Benin, programming built upon farmers’ agricultural knowledge and worked with existing active farming unions contributed to the sustainability of technical activities such as improved agricultural practices. In Guatemala, the existence of strong community demand for extension services and market linkages facilitated project buy-in and the long-term sustainability of activities. Although 22 | Findings   | Factors that Facilitate or Increase the likelihood of Project Sustainability resources for extension agents were scarce, they were able to serve communities and obtain their feedback, which shaped training and contributed to sustainability. Due to Sri Lanka’s high literacy, community members were able to grasp, retain, and share concepts easily, using and sharing projected provided technical materials and record books. 2 Environment Projects that successfully leveraged environmental assets and addressed environmental constraints were more sustainable. In Benin, project beneficiaries targeted regions where horticulture production was feasible, sail conditions and rainfall were present and this enabled the successful application of training in different crops and contributed to sustainability after project completion. In Guatemala, extension agents and beneficiaries conducted needs assessments with communities to select the types of crops that would both thrive in their environments and have market demand. The vast variety of micro climates in the highlands of Guatemala means that a one size fits all approach to crop selection would not have been successful. Getting local input when designing training helped successfully respond to local environmental conditions and this resulted in appropriate crop selection and increased sustainability for farmer beneficiaries. Project training also increased awareness among beneficiaries of the effects of climate change and the vulnerability of their local ecosystems. Beneficiaries identified these issues as the biggest threats to the sustainability of project activities in Guatemala, and some of them expressed being more prepared to face these thanks to the project. In Sri Lanka, farmers require regular access to water for home gardens. The project was designed to provide water storage and irrigation systems to capture seasonal rainfall, so when rainfall shortages occurred growers would have water. “Training in soil care and conservation has led to a change and improvement of the quality of our plants and harvest, which saves time and contributes immensely to our financial security. The existence of lakes, rainwater harvesting, and water management know-how means year-long farming without the fear of losing our crops to drought conditions, which is a huge psychological relief and yet another factor that gives us financial security.” Project Beneficiary SRI LANKA Water tank provided by the FFPr project to a farmer in Sri Lanka Findings  | Factors that Facilitate or Increase the likelihood of Project Sustainability  | 23 3 Sequencing Effective sequencing and integration of project activities and inputs led to more sustainable results. In Benin, the combination of training and resources was most notable through the design and allocation of a “full package” to beneficiaries who learned about new agricultural practices and were also qualified and supported to obtain loans. However, the loan timelines were not well aligned with the farmers’ planting and harvesting timelines. Beneficiaries shared that they would seek a loan at the beginning of the season when they needed funds to purchase inputs. However, even if approved, funds wouldn’t arrive to them until the ideal planting time had passed. This in turn delayed harvest time and reduced their yield. In addition, producers often needed to start their repayments before the bulk of their harvest and sale had occurred which made it difficult for some farmers to pay on time. With careful training, planning and coordination between project implementers, financial institutions and growers this could be avoided. If the timing of these activities is right and growers understand they need to seek a loan well in advance of planting time, they would have more success in the program and be more likely to secure financing in the future. In Guatemala, some farmers also benefited from a combination of agricultural training and credit access. As in Benin, credit access allowed beneficiaries to purchase inputs and tools that facilitated the implementation of new farming techniques. Through the project MICOOPE (an implementing partner and local credit union) developed a credit assessment tool better aligned with agricultural cycles, farmer’s needs, and their capacity for repayment. This tool allowed MICOOPE to better assess the financial needs of farmers as well as their capacity for repayment and greatly increased the number of loans that the credit union made to agricultural producers while also reducing loan default rates. The Sri Lanka case highlights the importance of first addressing a key community identified need— in this case, access to water, by providing better wells and water systems at the start of the project. This created buy-in and allowed for the successful implementation of agricultural activities, which also required water. Additionally, Sri Lanka showcases how the combination of activities can lead to sustainability, especially when building upon existing community assets, such as high literacy rates. The training and allocation of resources for the cultivation of short, medium-, and long-term crops were evidence of this. “The project initially gave us only short-term crops, and as the water management techniques and resources were introduced, we got crops that required watering daily. This kind of attention to detail and structure surprised us. Crops that lasted three months and one year were grown together, the continuous cash flow through crops changed the mindset we had and motivated us to work harder and commit ourselves to make a change.” — Project Beneficiary Sri Lanka 24 | Findings   | Factors that Facilitate or Increase the likelihood of Project Sustainability Interview with farmers Union staff in Benin 4 Existing Systems Projects that worked within existing systems to strengthen their capacity instead of duplicating efforts are more sustainable. In Benin, GREEN contracted directly with the Regional Producers Unions to implement training activities for their members. These collaborations strengthened existing institutions and expanded their reach with local associations and farmers groups. Because the project worked with these Unions, and farmer beneficiaries were selected based on their membership in a Union or Association, a base level of trust was already established. The Unions knowledge of farmers and their needs greatly increased the sustainability of project activities. In Guatemala, the project focused upon strengthening the existing formal extension system within the country through training extension agents. In addition to improving the capacity of the formal extension system the project used a “training of trainers” model whereby trained extension agents worked with local farmer learning groups. Project implementers worked hard to coordinate with the Ministry of Agriculture and Livestock in Guatemala and while this level of coordination was difficult, interviewees recognized the long term value of improving the existing extension system. This model has helped to ensure the continuation of project activities. However, this case also proves how difficult it can be to strengthen this type of system, as success is reliant on governments’ policies and their continuation after political transitions. Findings  | Factors that Facilitate or Increase the likelihood of Project Sustainability  | 25 For example, the project lacked an official agreement to hire credentialed extension agents, which led to many agents graduating without job opportunities. In Sri Lanka the project worked with local companies—MILCO and Cargills—to purchase goods from beneficiaries. This has led to improvements such as increasing the market for dairy and produce and expanding the reach of these local companies. Additionally, project beneficiaries worked with national associations, specifically the National Coconut Association, to create opportunities. These regions became one of the main national coconut producers after not having any representation with the association. 5 Value Chain Development Projects that balance investment of developing supply, understanding demand, and improving market linkages will be more sustainable. In Benin, GREEN built partnerships with existing farmers’ unions to build their and individuals growers capacity to access markets. GREEN also did an in-depth market study and connected growers directly to market sellers. Working in partnership with these unions was mutually beneficial and helped contribute to the long term sustainability of project activities and capacity building. Access to credit enhanced farmers’ capacity to implement more of what they learned from the project; therefore, they were more likely to report successes than their counterparts. In Guatemala the project sought to facilitate new market linkages between buyers and producers through various interventions. The most successful of these was the organizational development capacity training to farmer organizations which helped them to improve their internal management capacity. These farmer groups were also trained by extension agents and benefited from improved access to agricultural loans. While there was not much evidence that new market linkages were created through the project, together, these interventions allowed farmer groups to tap into existing local and international markets. However, project beneficiaries also explained that they were not always able to take advantage of market opportunities, and the continued lack of trust with the financial systems has proved to be a challenge that has not entirely been overcome. In Sri Lanka, programming attempted to create markets for beneficiary farmers by connecting them with local companies Milco and Cargills. The project offered incentives to these companies in the form of equipment for their processing and commercialization activities. While Cargills and Milco continue their activities, many farmers report they are currently not well linked to the market. Instead, they are more focused on price controls to influence the market through organizing into associations or cooperatives. Based on this mixed evidence across these 3 case studies we cautiously conclude that when farmers and farmer-based organizations are connected to markets and have knowledge of quality standards and pricing, they are much more likely to continue with the agronomic best practices they learned through the project. 26 | Findings   | Factors that Facilitate or Increase the likelihood of Project Sustainability 6 Community Informed Community-informed planning and design increases sustainability. Projects that secured community and stakeholder buy-in via community informed planning showed some increased likelihood that project activities and their benefits would continue leading to increased sustainability, however this correlation finding was not as strong across the board as others. In Benin, the implementing partner worked collaboratively to assess what activities farmers valued, what would be responsive to their needs, and what would align with the political and physical (water and land environments. When implementers did not build activities and content based on a community-led conversation or local market dynamics—such as the case with post-harvest processing and cooking classes—practices were not valued and subsequently discontinued. Interviewees and workshop participants shared that when their input was sought and they saw the results of their feedback trust was build and led to a great faith that what they were being taught would benefit them in the long run. In Sri Lanka, training covered prevalent and known agricultural barriers and incorporated feedback mechanisms allowing the implementing partner to respond to farmer insights. Addressing these barriers made the training applicable and established strong buy-in from communities, leading farmers continuing the activities promoted by the project. In Guatemala, a community asset and needs assessment-informed program design and was considered responsive to the community. Likewise, beneficiaries believed that using community￾identified trainers and promoters increased the relevance and applicability of training. “When we select the group, we do a ‘group plan.’ This identifies the needs that the community have, within that we get them to focus on more than just a ‘lack of resource’ on production, your milpa, what do you really need. They might decide to grow vegetables, or I need help with my coffee plants. They discuss all of this as a group, and at the end we have a formed plan with priorities to all of these things. — Agricultural Extension Agent, Guatemala Conclusions  | 27 Conclusions Understanding how and why project activities and their results are sustained and adopted after the conclusion of a project is important from a project design and management perspective as lessons learned will be used to inform future programming, and USDA/FAS investments. Balancing investment of developing supply, understanding demand, and creating linkages is key. Across the three sites, farmers remembered and were still using programming that focused on agricultural training to improve practices and the quality or quantity of goods. However, beneficiaries were less successful in their efforts to “increase access to markets to sell these agricultural products” (FFPr Results Framework 2.2). However in Guatemala and Sri Lanka beneficiaries attributed the impact of having more money most often to project activities, while in Benin pride was mentioned the most (196 times) followed by more money (184 times). Overall across all three projects participants saw more money as a direct result of continuing project activities. Projects that both address production barriers and integrate smal-lholder farmers into established value chains will find a virtuous cycle of increased income and increased ability to apply improved practices and technologies. Market-based solutions that invest in value chain development through interventions have become increasingly popular to reach rural producers and facilitate their entrance into larger more established markets, providing a means to improve their economic welfare (Staritz, 2012). Value chain projects that have a coordinated systemic approach for sustainability, show positive results at the farm or enterprise level, with large outreach and positive spillover effects (Dunn, 2014). In Sri Lanka, while dairy and marketing interventions weren’t a resounding success, in the one region where a local business created a distribution plant and connected to dairy farmers, demand was present, and supply was built and reinforced over time. In another region a community-based group created their own enterprise to jointly sell produce and set the price to wholesalers. The ability of a community enterprise to successfully generate funds directly influences its success. It also affects community interest in engaging in enterprise development and governance. In this way, market access can directly affect the ability of enterprises to establish and flourish. Factors such as distance to markets, competition, diversification, market knowledge, access to credit and the intensity of external help can enhance or limit a community enterprise’s ability to capture economic benefits for local communities (de Blas, 2009). 28 | Conclusions   Addressing environmental and physical constraints is key to sustainability Often called “natural capital” this includes a community’s environment, rivers, lakes, forests, wildlife, soil, weather, and natural beauty. Projects that can take advantage of existing natural capitals (soil and forests for example) that support agricultural production, and design solutions for capitals that are lacking (water)will be able to sustain intervention activities related to agricultural production or natural resource management. Access to water was a common theme and need across all projects. Irrigation efforts were successful when growers had the ability thorough loans or project donations to install applicable systems and receive training about how to use them. Project benefits were also more likely to be sustained when beneficiaries were able to see improvements in production thanks to the irrigation systems. Farmers who saw increases in income and recognized that irrigation was a key component continued to purchase and repair their systems and to see lasting benefits. However, when the technology gifted was inaccessible, too expensive or too complicated beneficiaries did not repair or purchase additional systems. Community-informed planning and design increases sustainability. Although USDA sets some key parameters for programming in each country, implementing partners are expected to assess community need in order for the design to be responsive. The findings across the sites indicate that this may not be enough to drive sustainability. Across all three countries, program design elements that incorporated community need and voices contributed to sustainability. However, this community-responsiveness occurred mainly during implementation, rather than during design. In Guatemala and Benin, implementers engaged local government and private organizations, which contributed to sustainability. In Guatemala, partnerships with USAC and MICOOPE enabled the continuation of the extension educational program and agricultural credits. In Benin, the presence of the project encouraged the local government to change land tenure policies. Additionally, the project built upon other preexisting initiatives incentivizing community cohesiveness, which facilitated sustainability. The Benin case represents an example of how it is not necessarily enough to include locals in program design as a way to ensure relevance and sustainability, but it is also necessary to include people from the community being served, especially beneficiaries. The cooking classes, for example, were part of a country-wide health initiative, led by the Benin government, yet were not viewed as relevant to community members in the targeted location. Strengthen existing systems, and build projects around existing community assets Evidence from these case studies as well as work by USAID and IFAD, describe the benefits of a systems approach to capacity building. This approach acknowledges and leverages existing links to develop programs that serve partners and their institutions, not finding partners to serve the program (JOHN GILLIES, 2012). In Guatemala, despite many of the extension agents being fired due to political changes, the existing extension system served as the foundation for programming, people now recognize the extension system as a place to go for agriculture support. Additionally Conclusions  | 29 by using the “farmer to farmer” approach to training the project was able to build upon an existing and familiar methodology, this allowed for ease of buy-in and broad support. When it comes to organizations, traditional governance practices and preexisting social norms can bring about administrative cost savings (Antinori and Bray, 2005) and support the success of community enterprise projects by providing cheap sanctions, discouraging free riding, and motivating group￾members to cooperate. However, the existence of these social institutions does not guarantee the long-term success of such community enterprises. Strong local leadership also has a positive effect on the success of community-based projects (Van Laerhoven, 2010; Dolom and Serrano, 2005; Nath and Inoue, 2008) and is especially important during initial phase of setting up initiatives such as community enterprises (Bianco and Bates, 1990). However, communities that lack technical knowledge or have poor book keeping/business management skills are less likely to successfully manage communal resources and establish enterprises. If communities making key decisions about natural resource management do not understand the technical, financial and management issues involved in the enterprise this lack of capacity can result in a failure of the business (Nittler and Tschinkel, 2005). Effective Component Sequencing and Integration Led to More Sustainable Results Building capacity is a common approach to development programming as seen in Guatemala and Benin; combining this with inputs, or financial access in an appropriate order increases the likelihood of sustaining new or different agricultural practices. The timing of input delivery and activities in a way that sets up a farmer for success, especially in the first year, will greatly increase the likelihood that they continue to practice what they find successful. 30 | Recommendations   Recommendations Require a Sustainability Plan It is best practice, for projects to be sustainable, to set metrics and standards from the initial project identification through conception, design, funding, implementation, monitoring, and evaluation phases. Projects fail to see sustained results when there is no clear and appropriate sustainability plan, especially when there is no incentive from the donor to create such a plan. Currently FFPr project proposals are required to include a “sustainability and lasting impact” section which includes, an estimated baseline, timeline, and outcomes to track progress. Applicants also must address “how the proposed project will achieve sustainable, long-lasting results that extend past the project end date.” This section also includes an exit strategy. The evaluation team recommends that this requirement be restructured into a “sustainability plan.” This proposed sustainability plan would include a comprehensive analysis of the existing social, economic, cultural, environmental, and political environments that the project or intervention will function within, highlighting specific conditions that are conducive to sustainability. The project logic, design, goals, objectives and activities should be fully articulated in the plan. The involvement of stakeholders and beneficiaries is of paramount importance to ensure that the logic and design are meeting identified needs and supporting existing institutions. Beneficiary assessments, regulatory framework studies, marketing and competition analysis, partnership development and institutional analysis all provide the foundation for a sustainability plan that can be evaluated at proposal submission, updated and monitored by program staff and donor agencies. Existing Conditions Projects, at the design stage, should be able to articulate market demand, gaps in market linkages and barriers to supply, existing organizations and organizational capacity needs, environmental and agricultural assets and community needs. Project activities should directly respond to these needs while building on existing assets. The majority of existing conditions are fixed; thus project design needs to demonstrate a response to given conditions (weather, markets, knowledge, geography) and strengthen existing assets (organizational capacity, knowledge, need). Projects should cultivate an in-depth understanding of institutional and organizational capacities; this should be documented in design documents along with explicit concepts of change that identify capacity constraints within potential partner organizations. Selecting the “best” organization or Recommendations  | 31 creating the “best” organization will not necessarily lead to sustained results but improving high value high impact organizations such as the Guatemala extension system, or Benin farmers Unions certainly can. • ●Country selection includes adequate rationale and documented catalog of natural resources assets that can support the desired value chains, high value high impact organizations that can support implementation, sufficient existing local infrastructure to support potential project activities. Project activities are designed to leverage existing natural capitals and have a documented plan to account for required resources lacking. Project design and activities should respond to existing gaps in organizational capacity and outline clear expectation and indicators to improve them. ● • Projects should build upon existing initiatives with support of effective policies such as government initiatives to improve access to land for women. Project Design Projects that are designed in consultation with and respond to existing organizations and their development needs will have an increased likelihood of sustainability. Projects that understand and build demand-informed supply will better be able to integrate beneficiaries into existing supply chains. The combination of training, access to finance, and connection to markets has been demonstrated to be a “triple threat” to poverty. Allowing for financing to supply improved production and a ready market for sale. In addition, projects should focus upon helping beneficiaries access value chains for existing markets rather than relying upon the creation of new markets or new value chains. Projects that understand market demand and are able to parlay that knowledge into improving supply and facilitating market linkages are more likely to have sustained results, and be valued by beneficiaries. It is important that projects focus upon developing supply, understanding existing demand, and creating market linkages rather than creating demand or new markets. The Women’s Empowerment in Agriculture Baseline Report (Malapit, et al., 2014), found that “too little credit, too much work, and a lack of membership in social and economic groups are the main constraints to women’s empowerment in agriculture.” This report outlines the negative association between women’s empowerment and per capita expenditure. While this sustainability assessment did not specifically look at women’s income or share of income, labor involvement and off farm work, this trend is important to be aware of when designing projects that will increase household income from agriculture. Increasing women’s access to credit, and labor saving technologies have both been shown to increased women’s empowerment (2014) Involving women and other beneficiary populations in the project scoping and design phase and creating activities that specifically respond to their self-identified needs could help increase the take up of and sustainability of project activities. Design projects with sequencing of project activities and inputs in mind. Project design should demonstrate a strong understanding of activity time lines so as to ensure that project activities are sequenced in away that each individual activity works in tandem to support other project activities. For example, combining technical training with access to credit. Potential sequencing related constraints should be identified and addressed during project design. 32 | Recommendations   Project Implementation Project activities need to be rolled out in a logical order and in combinations that work together to support sustainability. While it may seem obvious that irrigation should be secured before growers’ plant high value vegetable crops, logistical constraints can sometimes make this challenging. Constraints should be anticipated to the best ability in order to implement activities in a sequence that most facilitate sustainability. Project activities that support one another should also be combined whenever possible to increase the likelihood that those activities will continue after the project has ended. Prioritize projects that work within existing systems and work to strengthen the capacity of local partners through project implementation. By strengthening existing systems and organizations, projects are more likely to have long term impacts and beneficiaries are more likely to be able to access these systems into the future. Summary Projects that successfully generated income for participants had four types of combined activities: established commercial linkages; improved productive capacity of farmers; improved organizational capacity of farmers, farmer groups; and facilitated access to finance. The key to sustainability across the board has been to align these factors with existing financial institutions, firms, organizations and government initiatives. Evidence from a Feed The Future study of over 196 evaluations came to similar conclusions related to value chain development (USAID, Feed the Future Knowledge-Driven Agricultural Development project (KDAD), 2016). CASE STUDY — Benin | 33 CASE STUDY — Benin 34 | CASE STUDY — Benin  | Summary Summary The Growing Resources for the Enhanced Agricultural Enterprises and Nutrition (GREEN) project, implemented by Partners For Development worked with Beninese farmers’ associations to train vegetable growers on how to increase income through better growing techniques, strategies to decrease post-harvest losses, and through improving beneficiaries understanding of market￾driven production. The project objectives were 1. Increase vegetable production and improve marketing skills among small-holder farmers through training and extension. 2. Improve value chain and post-harvest skills by building the capacity of farmer associations to provide improved services to their members. 3. Increase financial services provided to targeted smallholders by increasing access to loans and training in business/financial management. From November 2011 to July 2016, PfD implemented the GREEN Project in six southern states in Benin: Atlantique, Littoral, Couffo, Mono, Oueme and Plateau. Through GREEN, PfD provided valuable training to over 7,300 agricultural producers and through increasing the farmers’ net income, benefited 36,000 family members. This post project sustainability assessment conducted workshops and interviews with 99 individuals to determine what project lessons and activities have been continued since the project closed, and what have the impacts of continuing these activities been on beneficiaries and communities. The assessment collected data from; direct beneficiaries, implementing partners, trainers, loan officers, and community leaders. Evaluation Purpose The Office of Capacity Building and Development (OCBD) is a program area of USDA’s Foreign Agricultural Service that is dedicated to advancing global trade, agricultural development and food security. Programs administered by OCBD help to reduce food insecurity, strengthen agricultural economic growth in targeted developing countries along the full value chain of development, build their capacity to integrate into the global economy, and become viable agricultural trading partners with the United States. The objective is to help move these countries along the agricultural market-development continuum from developing economies, where capacity building is the primary focus, to developed economies, where market development and expansion are the primary focus. Authorized by the Food for Progress Act of 1985, the Food for Progress (FFPr) program provides for the donation of U.S. agricultural commodities to developing countries committed to introducing and expanding free enterprise in the agricultural sector. Donated commodities are monetized (sold on the local market) and the proceeds are used to support agricultural development activities. CASE STUDY — Benin | Evaluation Purpose | 35 Food for Progress has two principal objectives: improve agricultural productivity and expand trade of agricultural products. Food for Progress programs have worked to train farmers in animal and plant health, improve farming methods, develop infrastructure (roads, electricity, storage, etc.), establish and strengthen producer cooperatives, provide microcredit, and develop agricultural value chains. FFPr Program Participants have included private voluntary organizations (PVO), foreign governments, universities, and intergovernmental organizations. FFPr projects are currently active in 25 countries across Africa, Asia, and Latin America and the Caribbean. The University of Minnesota was contracted by the USDA FAS FFPr office to assess the ongoing impacts of 3 closed Food for Progress Projects: 1 each in Africa, Asia, and Latin America and Caribbean. The final countries and projects were selected in join consultation with USDA. The specific questions were designed in close collaboration with USDA, including a workshop conducted with USDA in November 2017 and over a period of months to refine questions and methods. This evaluation was not expected to include experimentation or trials on human subjects, as the programs to be evaluated have all concluded. Each closed project is presented as a case study, and the three projects together are a part of the full “Post project sustainability assessment” report. The Organization for Economic Cooperation and Development (OECD), Development Assistance Committee (DAC) has defined “sustainability”, in the context of evaluation and results-based management as: The continuation of benefits from a development intervention after major development assistance has been completed. The probability of continued long-term benefits. The resilience to risk of the net benefit flows over time. The Food Assistance Division, in its Monitoring and Evaluation Policy defines “sustainability” as: Assessment of the likelihood that the benefits of the project will endure over time after the completion of the project. Sustainability should also assess the extent to which the project has planned for the continuation of project activities, developed local ownership for the project, and developed sustainable partnerships. The case study that follows is one of three studies that are a part of a full report provided to USDA FAS on Dec 31 2019. Evaluation Questions 1. How did existing conditions affect project sustainability? A. What are the preliminary conditions that set up market linkages for success? B. To what extent are results more sustainable in low, or middle-income countries? 2. What components of the project’s design contribute to sustainability? A. To what extent were local stakeholders involved in project activity design? 3. In what ways did the implementation of the project contribute to its sustainability? A. How do local partnerships contribute to sustained project outcomes and impact? B. What factors lead to training results being sustained? C. How does the combination and sequencing of activities influence sustainability? 36 | CASE STUDY — Benin  | Project Background 4. What activities or practices have continued (and to what extent) since the project ended? A. How valuable to communities are the project’s outcomes, including activities and practices? 5. What are the impacts of the project? A. How valuable to communities are the impacts of the project? B. How did beneficiaries and their communities respond to subsequent shocks? 6. To what extent have the impacts of the project contributed to equity? Project Background Benin has one of the most stable democracies on continental Africa, yet ranks 166 out of 187 on the 2012 Human Development Index (USDA-FAS). Benin’s GDP is pinned on the fluctuations of global market prices for its exports, principally cotton. The high cost of agricultural inputs and low rate of mechanization severely limit Benin’s agricultural productivity. An estimated 34% of households in Benin are food insecure (USDA-FAS). Only 15% of households are food self-sufficient, the rest relying on local markets to acquire food during times of scarcity (USDA FAS). The government of Benin has focused its efforts on calorie production rather than nutritional quality to address national food insecurity. However, food scarcity is less of a problem than is malnutrition, especially during the ‘hungry seasons’ between harvests (USDA FAS). Benin is not alone in this struggle, as many of its regional neighbors in West Africa are also challenged by nutritional deficiencies. In Benin and other parts of the region, starches (in the form of cereals and tubers) are the dietary mainstay, often accompanied by an oil-rich sauce and various leafy vegetables. Animal protein is not commonly consumed (USDA FAS). There is a need to focus development efforts on fortifying Benin’s principally grain-based diet with vitamin A, iron, and iodine. The region’s most in need of these interventions are Atacora, Donga, and Mono (USDA FAS). Vegetable farms in Benin are mostly small, with less than a hectare cultivated during any given season. Typically manual tools are used to prepare plots and watering is mainly by hand. Often farmers have limited knowledge about improved farming methods, and usually choose the variety of vegetables and their practices according to family or local traditions. As they are generally poor, vegetable farmers tend to use the lowest cost, and often least effective fertilizers and insecticides, further reducing crop quantity and quality. Fewer women are vegetable farmers due to added barriers to owning land and even less access to financial and human resources. To address issues related to diet diversification for improved nutrition, as well as overall agricultural productivity, the government of Benin has secured the assistance of multiple international actors. USDA FAS has been working in Benin since The Growing Resources for the Enhanced Agricultural Enterprises and Nutrition (GREEN) project worked with Beninese farmers’ associations to train vegetable growers on how to increase the money they take home from their harvests through better growing techniques, through strategies to decrease spoilage between the field and the market, and through improving their understanding of market-driven production. CASE STUDY — Benin | Project Background | 37 Project stakeholders included: 1. Market Gardeners also called producers or farmers were the main project beneficiaries. 2. Regional Farmers Associations including the Regional Associations of Farmers of the Ouémé-Plateau (URP-OP), Atlantique-Littoral (URP-AL) and the Regional Council of Market Gardeners of Mono Couffo (CRM-MC) also called “farmers associations” served as Sib awarded implementing partners. 3. ESOKO a Ghanaian internet services provided facilitated the establishment of an information system on the current price of market garden produce (VMIS) by mobile phone messages; 4. The microfinance lending institution, the Federation of Agricultural Savings and Credit Mutual (in French, Fédération des Caisses d’Epargne et de Crédit Agricole Mutuel, FECECAM) funded small loans to the market gardeners; 5. The Agriculture and Food Technology Program (PTAA) of the Ministry of Agriculture’s National Institute for Agricultural Research (INRAB) trained facilitators and gardeners on processing technologies of vegetables. From November 2011 to July 2016, PfD implemented the USDA-funded Growing Resources for the Enhanced Agricultural Enterprises and Nutrition (GREEN) Project in six southern states in Benin: Atlantique, Littoral, Couffo, Mono, Oueme and Plateau. Through GREEN, PfD provided valuable training to over 7,300 agricultural producers and through increasing the farmers’ net income, benefited 36,000 family members. The GREEN project worked with Beninese farmers’ associations to train vegetable growers to increase the money they take home from their harvests through better growing techniques, through strategies to decrease spoilage between the field and the market, and through improving their understanding of market-driven production. Centered on tomatoes, peppers, and leafy greens, the trainings GREEN provided focused on improving water and soil resources, appropriate use of pesticides, dry season cultivation, and how to handle vegetables immediately after harvesting. GREEN worked with growers’ associations on marketing and business strategies to help them maximize their earnings potential as a group. Summary of final evaluation, 2016 GREEN trained 7,390 agricultural producers in vegetable value chain theory, market assessments, and modern methods to improve agricultural production. By the end of the GREEN project, 97% of participating GREEN farmers produce in response to what the market wants, and 95% of GREEN farmers increased their vegetable production by an average of over 160% by increasing the number of cultivation cycles from two to three and four times per year. GREEN’s final evaluation also showed, farmers increased their net income by over 220% and created over 8,600 seasonal jobs. In summary the final report stated that; In relation to objective 1 — the project achieved 169% increase in crop production of the stakeholders, resulting from improved yields, increased cultivated area and increased number of production cycles in a year. This result was achieved for beneficiaries who received the total package of GREEN services. In relation to objective 2 — for the acquisition of market skills and reducing post-harvest losses, there is a greater formalization of commercial contracts between producers and traders, with a total of 51 contracts created (the goal was 6), while post-harvest losses were reduced by 64.3% (the target was 30%). 38 | CASE STUDY — Benin  | Evaluation Methods & Limitations As for objective 3 — addressing the facilitation of access to financial services, the GREEN project was successful in helping 2,012 growers obtain loans from FECECAM for a 101% achievement rate. The reimbursement rate as of 30 September 2015 is around 93% which is slightly In the final evaluation GREEN’s sustainability was observed through (i) the beneficiaries’ increased knowledge of and integration into local markets; (ii) the significant improvement in productivity resulting from improved cultivation skills and knowledge and the equipment purchased through micro loans such as irrigation systems. However, the final evaluation noted that there were doubts about the lasting impacts in the absence of the GREEN project. This included whether technical support of community based volunteer expert farmers and the field agents would continue to be supported by the Professional Agricultural Organizations (OPA), as well as whether the Esoko SMS messages of market prices would continue without external funding. Evaluation Methods & Limitations Sample Selection A wide range of stakeholders participated in the evaluation. Implementing partners identified participants that fit various types of stakeholders identified by the evaluation team. The evaluation team conducted outreach with the intention of conducting 25 interviews spanning across all stakeholder types per site. Specific data collection participation counts and stakeholder information can be found in table 5. A few participants participated in multiple forms of data collection within their respective site; as a result, the grand total in the Data Collection presented in Table 5 does not represent unique individuals that participated in data collection at a site. Image Grouping© participants were selected using a stratified convenience sample. This allowed the evaluation team to select groups based on additional hypotheses relevant to sustainability. All Image Grouping© participants are farmers. Criteria for the selection included attention to: diversity of locations, a balance of genders, participation in financial package activities or not, and believed sustained practices in present day. Specifically, to address gender-balance, one group of women-only participants was recruited to encourage greater openness about the impacts of the project. A few men insisted on being included; the facilitator thus promoted discussion from the women. Specific gender breakdowns can be found in the Image Grouping© Sample, By Sex and Region( table 6). Data collection was administered in six different languages, which are specified in the Languages Spoken to Conduct Data Collection (table 7). The activities that participants had to choose between came directly from a review of project documents and consultation with the implementing partner. All farmer beneficiaries received some training on farming practices; full package beneficiaries received assistance getting credit with a partner microfinance institution while partial package farmers did not. Two workshops were conducted in regions with the largest markets in Cotonou and Grand Popo, using full and partial package beneficiaries (table 4). This helped test the effect of market proximity on sustained outcomes. Another two workshops were hosted in locations further from a major market; these also disaggregated full and partial package recipients. CASE STUDY — Benin | Evaluation Methods & Limitations | 39 Regions Image Grouping© Workshop Interview Package beneficiaries and location Oueme 1, n=20 4 4 Full package, Far market Couffo 1, n=19 4 Partial package, Far market Mono 1, n=17 7 Full package, Close market Littoral 1, n=20 8 Partial package, Close market Grand Total 4 23 85% Table 4 — Data Collection Counts Stakeholder type Oueme Couffo Mono Littoral Total Financial institution 1 1 2 1 5 Implementing partner staff 3 3 Lead farmer 1 1 5 7 Partner institution 1 1 3 5 Farmer association/ cooperative 1 1 2 Grand Total 4 3 7 8 22 Table 5 — Interview Sample Market Distance Region Village/City Female Male Total Near Littoral Cotonou 6 14 20 Far Mono Ewe-Condji 11 11 Far Mono Yodo Condji 2 2 Near Couffo Klouekanme 6 5 11 Near Couffo Lalo 3 3 Near Couffo Toviklin 2 3 5 Far 6Oueme Oueme 6 14 20 Grand Total 34 42 76 Table 6 — Image Grouping© Sample, by Sex and Region 40 | CASE STUDY — Benin  | Evaluation Methods & Limitations Row Labels Number of Participants Adja 21 English 1 Fon 20 French 9 Mina, Ewe 17 Wémè / Goun 20 Grand Total 88 Table 7a — Languages Spoken to Conduct Data Collection Data Collection The evaluation team conducted interviews with local key stakeholders to understand the local context of each site and gather insights to inform culturally appropriate evaluation design. Findings from this report come from primary data collected through: • Image Grouping© • Interviews • Observations A local coordinator and the implementing partner identified and recruited prospective participants. The evaluation team—of UMN faculty, graduate students, and an Improve Group consultant— conducted in-person interviews, field observations and facilitated Image Grouping© sessions. The number of participants that engaged in each data collection method can be seen in the second column from table 7. Figure 2 — Colored area indicates visited regions by the evaluation team in Southern Benin. CASE STUDY — Benin | Evaluation Methods & Limitations | 41 Data Collection Method Number of Participants Image Grouping© 76 Interviews 23 Grand Total 99 Table 7b — Data collection Methods by number of participants. Analysis The evaluation team conducted qualitative analysis in Dedoose software to analyze interview data and Image Grouping© discussion notes. Initially, the evaluation team developed codes using an inductive approach. These codes were developed by identifying themes from literature on conditions needed for sustainability, as well as the evaluation questions. Following that, a deductive approach was used to find new themes based on the responses. The evaluation team lastly looked at relationships between themes and compared the differences between different types of respondents, such as farmers and other key stakeholders as well as differences based on location. Quantitative Image Grouping was analyzed using Excel and SPSS. Frequencies and cross tabs were conducted to understand differences between locations, and the activities that resulted in sustained outcomes. Themes across these responses were compared to those found in the qualitative results and integrated to achieve a mixed method understanding of the data. In addition to data collected in the field, a literature review was conducted to both inform our data collection and compare our findings to existing academic literature as well as program evaluations from other development project reports. Relevant academic literature was identified based upon relevance to agricultural extension, community agricultural development, community capacities, and relevant research in Benin. Project reports/evaluations were identified based upon their relevance to the region and their focus on agricultural extension. Limitations With all evaluations, there are limitations to the data that are gathered and the conclusions that can be drawn. Recall and response bias may be present in the data. Recall bias may have occurred due to the time that has passed following the closure of projects. This can lead to participants remembering events at a different scale or forgetting what they did at that time. As a result, some farmers may have reported participation in trainings and services mostly in areas that they still remember and use today, positively biasing findings towards a higher percentage of individuals sustaining practices. Image Grouping© methodology helps to reduce this likelihood by having farmers from similar locations participate in the same session and encouraging them to talk among themselves to remind each other whether they participated in certain activities over others. Positive response bias may have influenced farmers’ responses in two ways. First, in asking how participant lives are today as compared to prior to the project start, farmers may have been more 42 | CASE STUDY — Benin  | Findings likely to exaggerate the benefits of the project. In many data collection environments, participants aim to “please” data enumerators, responding in a way that they believe the enumerator is hoping for or contributes to their benefit. Farmers may have believed that by extolling the hardship they experienced prior to the project and highlighting how much they continue using today, that this would support future work in their area. Second, implementing partners were used to determine which sites to visit and who to recruit. Partners may have desired to show more successful outcomes for their own audiences as well as potential influence on future funding through USDA. The evaluation team worked to compensate for this potential bias by requesting IPs to suggest sites that had varying degrees of success after being involved with the USDA project. The high rates of sustained practices suggest that this bias may be more prevalent in Benin’s data. Therefore, information on what has hindered sustained practices, or challenges farmers face may be even more beneficial in reviewing to understand what to consider for creating an optimal environment for sustainability. Responses exceeding 100% of beneficiaries continuing practices occurred in all three cases, this happens when more respondents say they are continuing a practice than said they learned about it from the project. For example in Benin, 54 participants said they were taught about safe pesticide handling from the GREEN project and 58 said they are continuing to use the safe pesticide handling practice they learned from the GREEN project. This results in over 100% “continued practice” this could be because the question was loosely interpreted by participants or translated to mean “do you use safe pesticide handling practices?” regardless of who they learned it from, participants may not have specifically remembered the training activity but they feel they are currently doing it, they may have learned from another project participant, or another source. Or participants may have simply been responding in a positive way to “please” data enumerators (see positive response bias above). However in most cases responses exceeding 100% indicate in general that the practice or activity has been a success and is being applied by most or all people who learned about it. This project does not include control or comparison communities and thus cannot make claims about magnitude of impact or causality. As a result, although we were able to describe if income changed for participants and whether they attribute this to their respective projects, we do not know the extent to which income changes were solely the result of the USDA project. Likewise, secondary data did not exist at a regional level to demonstrate need or wealth status prior to project start; therefore, we are unable to make claims about economic changes to regions where the project was implemented. Findings These findings are organized using the FFPr GREEN projects 3 objectives. Under each objective we will share the findings that correspond to our high level evaluation questions, found in full on page 34 in the Project Background section. Participants responded to a five question Image grouping survey asking participants – (the first two questions correspond to table 8). What activities or practices do you remember learning about through the project? And what activities or practices do you continue to do today? The numerical CASE STUDY — Benin | Findings | 43 counts reflected in table 8 are the number of workshops participants who recalled learning about, followed by the number who said they are continuing the same practice today. This was followed by a discussion of if they were doing the activity at or around the time the project ended and if there were continuing to do it today. The qualitative discussion is reflected in our analysis. Impact was measured in a similar way; workshop participants were asked about what impacts they experienced as a result of continuing with specific project activities and technologies. Impact in this case was measured by asking workshop participants “What impacts do you experience, and as a result of which activities?” this was then calculated by tabulating the number of linked responses between activities and impacts. Overall no difference in responses based on distance to markets was observed. With few (noted) exceptions, location did not affect responses. In certain cases, participants indicated that they continue a practice today, even though they could not recall learning about the practice during the GREEN project implementation (percent continue is over %100). We assume that in these instances, other sustaining factors after the project led to this behavior change. For instance, one explanation is that farmers who received GREEN training could then have served as trainers for other farmers. The time scale of the questions could have been misinterpreted or alternatively, it is possible that farmers who never received the training are doing these things of their own accord for other reasons. Project Beneficiaries wash and sort vegetables in the shade, a practiced learned and continued form the GREEN project. 44 | CASE STUDY — Benin  | Findings Project activities Number who learned about the activity during the GREEN project. Number who continued to use the knowledge 2019 Percent who continue to use what they learned during project implementation All vegetable production activities 227 221 97% Pesticide use 68 67 99% Manufacturing pesticides and natural fertilizers 68 65 96% Crop production/ management 60 63 105% Irrigation systems 31 26 84% All Post-harvest related activities 153 158 103% Pesticide safety 54 58 107% Post-harvest hygiene 50 59 118% Post-harvest processing and preservation 48 41 85% All finance and marketing activities 133 118 89% Entrepreneurship 54 54 100% Market information data 43 37 86% Savings planning/ financial management 36 27 75% All health related activities 104 100 96% Food preparation and eating 63 59 94% Household health 41 41 100% All value chain related activities 91 90 99% Access to markets 58 53 91% Value chains 33 37 112% All business planning related activities 71 54 76% Accessing loans 40 27 68% Business planning 31 27 87% Table 8 — Project activities and the percentage who continue to use what they learned from the project. CASE STUDY — Benin | Findings | 45 Pride, individual and community, 196 Finance, 36 Value Chains, 31 Health, 17 Business planning, 9 Improved crop diversity, 146 Improved nutrition, 139 Value Chains, 46 Health, 46 Better health, 125 Access to markets, 119 Finance, 16 Finance, 13 Finance, 14 Better connections, 109 Finance, 38 Finance, 8 Value Chains, 33 Health, 5 Time gains, 94 Health, 6 Health, 3 Increased production, 76 Impacts associated with GREEN project activities Vegetable Production, 57 Vegetable Production, 25 Vegetable Production, 41 Vegetable Production, 40 Vegetable Production, 18 Vegetable Production, 27 Vegetable Production, 58 Vegetable Production, 37 Post-harvest, 55 Post-harvest, 14 Post-harvest, 16 Post-harvest, 6 Post-harvest, 6 Post-harvest, 16 Post-harvest, 33 Post￾harvest, 17 Post-harvest, 24 Business Planning, 9 Finance, 38 Finance, 20 Finance, 4 Health, 35 Health, 13 Health, 39 Healt, 5 Business Planning, 3 Business Planning, 8 Business Planning, 9 Business Planning, 6 Business Planning, 3 Value Chains, 30 Value Chains, 2 Value Chains, 8 Value Chains, 4 Vegetable Production, 51 Value Chains, 42 Value Chains, 42 More money/income, 184 Business Planning, 6 Numbers indicate numbers of respondents reporting impact Chart 3 — Impacts associated with the GREEN Project activities. OBJECTIVE 1 — Increase vegetable production and improve marketing skills Activities under this objective included technical training in vegetable production, safe use of inputs, water management, soil conservation, and marketing. Project activities centered around tomatoes, peppers, and leafy greens because of strong market demand and the potential to reduce post-harvest losses of these crops. Trainings focused on improving water and soil resources, pesticide education, dry season cultivation, and field based post-harvest techniques (such holding products in shady conditions while picking and packing). These training (and loans provided through activities under objective 3) sought to extend the growing seasons, increase production and income and improve post-harvest quality of produce. Additional activities designed to meet this objective included the training of extension agents, farmer training and electronic communications training. 46 | CASE STUDY — Benin  | Findings What activities or practices have continued since the project ended? Technical trainings on horticultural production (crop production and management) were found to be the most sustainable compared to other project activities. Of the 60 farmers in our sample who learned about crop production and management, 63 (105%) are continuing to use this information and practice what they learned (table 9). GREEN employed a participatory approach that incorporated producer feedback in order to understand the trainings’ effects and perceptions among the community. The collaborative nature of the relationship between GREEN technical trainers and farmers likely contributed significantly to sustained activities. Technical trainings on improved pesticide use and creating natural pesticides and fertilizers were also found to be very likely to have continued. Of the 68 farmers in our sample who learned about improved pesticide use, 67 (99%) are still using better practices as they apply chemical inputs. GREEN also trained participants on fabricating their own, organic pesticides(table 9). 65 of the 68 farmers who recall receiving this training continue to create their own inputs (96%)(Table 9). While most agricultural-related trainings were viewed favorably, the implementation of irrigation technology was determined to be one of the least likely to have continued. Project beneficiaries reported they could not afford the high cost of the pumps on other needed equipment or could not or we unwilling to repair the ones provided by the project. During field observation visits to beneficiaries’ farms, it was clear that irrigation equipment was in disuse or disrepair. Farmers complained about their inability to secure loans to replace or repair the irrigation equipment. In some cases, farmers received training on irrigation use, but lacked the means to implement the technology on their own. One workshop participant reported: “Before GREEN, we did not have knowledge of irrigation practices. GREEN took us on a field visit to Seme -Podje where we saw irrigation techniques. We learned and requested that the system be built in our community. But until now, we have not gotten anything.” Project activities under objective 1 Number who learned about the activity during the GREEN project Number who continued to use the knowledge/ inputs 2019 Percent who continue to use what they learned during project implementation Pesticide use 68 67 99% Manufacturing pesticides and natural fertilizers 68 65 96% Crop production/ management 60 63 105% Irrigation systems 31 26 84% Table 9 — Project activities under objective 1 and the percentage who continue to use what they learned from the project. CASE STUDY — Benin | Findings | 47 What are the impacts related to Objective 1? Project beneficiaries outlined multiple impacts experienced as a result of continuing with specific project activities and technologies related to objective 1. Overall as a result of crop production, irrigation, and improved use of inputs participants in workshops described impacts such as increased personal and community pride, better nutrition and more money. Despite widespread discussion of challenges farmers continue to face, especially financial, when asked about impacts farmers see today, more money was most frequently mentioned (chart 4). In comparing whether or not those who were closer to markets and those who were further had a different experience of having more money today, no differences were found. One participant explained that “The GREEN techniques improved the way of doing things like planting crops, treating crops, and preparing seeds. This helped us to earn more money and limit the pest attacks that we had before.” Another said “Now, we know how to keep track of what we sell and can subtract what we spend on production to derive net profit.” Crop Diversification, was described as an impact of these activities 57 times, participants talked about growing new crops, diversifying away from the low value staple crops they previously grew, and growing more variety of the higher value horticultural crops. This on-farm diversity was valued for providing access to different types of food, increasing income streams, and providing a harvest a multiple times of year. Pride, both personal and community was an impact participants experienced as a result of these activities. This was followed by improved nutrition and more money. Farmers typically felt pride as a result of having more money, although a few discussed it in terms of having greater knowledge. One workshop participant explained that “She is proud because she earned a lot of money in selling her crop.” And another said “He now understands that now his harvest is bigger than before, and that makes him proud.” Impacts Number of Respondents Reporting Impact 0 Pride More Money Crop Diversification Access to Markets Time Gain 10 20 30 40 50 60 Chart 4 — Impacts as a result of activities under objective 1, vegetable production. 48 | CASE STUDY — Benin  | Findings OBJECTIVE 2 — Improve value chain and post-harvest skills The GREEN project conducted technical trainings in post-harvest handling and processing, connecting growers to buyers, and pesticide safety also covered in objective 1. The outcomes were expected to be: more business relationships, reduced post-harvest loss, and create new seasonal jobs. In additional to the trainings, GREEN organized three regional meetings where market buyers met with vegetable producers to share their expectations (such as appropriate period of production for each crop, vegetable quality standards, types of vegetables demanded, etc.) and to establish trust between buyers and growers. Activities under objective 2 Improved value chains and post-harvest skills Number who learned about the activity during the GREEN project Number who continued to use the knowledge/ inputs 2019 Access to markets 58 53 91% Value chains 33 37 112% Post-harvest hygiene 50 59 118% Pesticide safety 54 58 107% Table 10 — Project activities under objective 2 and the percentage who continue to use what they learned from the project. Activities implemented under Objective 2 have continued, some are being used by more than recalled learning about it (results over 100%). This could be that they learned about post-harvest hygiene or value chain linkages from other sources, other project participants, or they didn’t recall learning about but do know it is something they currently engage in. Of the 58 farmers in our sample who learned about access to markets 53 reported that as a result of GREEN, they continue to experience increased access to markets. GREEN introduced participating farmers to wholesalers in other communities and markets. Several farmers who were surveyed reported that since GREEN ended, these relationships have continued to be a key part of their business. Results from the workshops suggested that most farmers felt that GREEN had opened the door to new markets. Indeed, over 100% of beneficiaries are using what they learned related to improving value chains. GREEN’s successful agronomic trainings may be partly responsible, as farmers reported an increase in the quality, quantity, and diversity of their horticultural products. Several farmers attested that wholesalers greatly appreciated the superior quality of their produce. One participant noted: “I continue to cultivate more diverse farm products. At first, I used only to farm one agricultural product, now mainly crops more products. This opens up new markets and brings in more revenue. I sell outside of the community.” In general, workshop participants reported continuing activities and technologies that were low￾cost, easy to implement, and had clear positive impacts. One participant noted, “What is good is good.” Post-harvest practices that improved health and hygiene, particularly hand-washing, and wearing protective clothing when applying chemicals, were particularly impactful. Of the 54 farmers in our sample who learned about safe handling of pesticides, 58 are continuing. Practices that participants are still doing included wearing full-body protective clothing, face masks, CASE STUDY — Benin | Findings | 49 washing vegetables before consumption, and washing hands after spraying pesticides or working in the field. Interviewees and growers who the evaluation team visited were clearly very proud and grateful to the support related to pesticide safety. Many showed us the protective masks, backpack sprayers, and protective clothing provided by the project. Many also pointed out that they have continued to repair or replace items as they used them frequently. The investment of ones own funds is a very good sign that the practices learned have resulted in sustainable behavior change. While beneficiaries continued to apply what they learned about pesticide safety and hygiene, this was not the case for post-harvest food transformation trainings. Farmers noted that they did not like how certain food preservation methods changed the taste of their food. It is also possible that farmers were the wrong target audience for these trainings. Many GREEN participants were full-time or aspiring full-time vegetable growers who did not want to invest the significant time and material resources required for practices such as canning and drying. One farmer from the Cuoffo region suggested that market women would be a better audience for these trainings. What are the Impacts related to Objective 2? Project beneficiaries outlined multiple impacts experienced as a result of continuing with specific project activities and technologies within objective 2. Overall as a result of value chain, post￾harvest and health related activities under this objective participants in workshops described impacts such as increased personal and community pride, better nutrition, better health and more money. Pride was an impact most often of improved connection to markets, and better health. One workshop participant said that “Actime (village) would produce mostly cabbage, but the market didn’t know that the cabbage came from here. The market ladies came here to buy them; with GREEN, they took us to the actual market, and we saw everyplace that sold cabbage, and we were able to sell the cabbage to more people, and the price went up due to higher demand.” Because participants visited markets and connected with buyers they built relationships, saw more value in their work, and the positive feedback from buyers led to increased pride. Improved nutrition was the next most common impact mentioned by participants. One interviewee said referring to beneficiaries that “Green taught them that they should eat their own produce and since starting this practice their health improved.” “Even if we don’t have money, they can eat very well with what they produce.” and “They can identify leafy vegetables that are very good for the nutrition of their children. They feed their children with a variety of healthy vegetables.” Nutrition was described as an impact of a better diet and from being able to purchase more and different food items. Better health. Beneficiaries spoke a lot about how their health had improved, not specifically related to nutrition but as a result of better and safer pesticide and fertilizer handling practice. The health related impacts are much more a result of the specific health and nutrition trainings as well as the continued use of protective clothing, and improved pesticide and chemical handling. Participants noted fewer visits to the health clinic and reduced spending on medication for treating illnesses. Safe pesticide handling in particular was a recurring topic that came up in the workshops, farm visits, and farmer interviews. 50 | CASE STUDY — Benin  | Findings One interviewee mentioned about a participant, “She started to use inputs like fertilizers and pesticides. She may have used them before but didn’t get the harvest that she wanted. Now she gets good yields. Now she also knows that it is the pesticide that causes her to feel sick, not sorcery. GREEN taught them how to use the mask and protective clothing, and now she doesn’t feel bad at all. She also now tells her children to wash their hands before touching food. She also knows not to use dirty water to wash her hands.” More money was a result of both post-harvest activities and improved value chain linkages. While the post-harvest processing such as drying and canning did not take off, other basic practices were adopted. Storing produce in shade, washing it, and sorting it did seem to have an impact on beneficiary selling higher quality produce. The improved integration into and connections with local and regional markets has led to increases in income for some beneficiaries. “GREEN taught us how to use manure as fertilizer. Now I no longer have to worry about buying fertilizers. Formerly, I used to spend money on food, drinks, social situations. GREEN taught me how to better manage funds and how to save for the future.” While benefits from accessed markets is clear and well described by participants, there continue to be logistical and structural challenges to getting product to market. One participant said, “poor infrastructure and poor ability to bring products to market. Especially when it rains.” Impacts 0 Pride More Money Crop Diversification Improved Nutrition Better Health Access to Markets Connections Time Gain Increase in Production Access to Loans Access to Market information Food Security 20 20 60 80 100 120 Number of Respondents Reporting Impact Chart 5 — Impacts as a result of activities under objective 2. CASE STUDY — Benin | Findings | 51 Objective 3 — Increase Financial Services Provided to Targeted Smallholders The GREEN project conducted technical training on business planning, savings, financial management, applying for loans and accessing market information. The project facilitated access to loan packages which were provided by a local finance institution FECICAM/CLCAM, a core project partner. The project also worked with ESOKO to collect and distribute market price information to beneficiaries. Improving credit access was a primary objective of the GREEN project and assessment supported that those interviewed agreed that it was a key factor in sustaining increases in income. Providing credit to farmers who did not have access to this before the project, especially women, increased the likelihood of sustainability of other practices. For example, farmers accessing credit were able to improve the productivity of crops promoted by the project by not only purchasing seeds etc at the start of the growing season, but also by having income available to meet mid-season needs, such as repair of irrigation, or purchase of pesticides during pest outbreaks. Without immediate access to credit or existing relationships with credit agencies, such as CLCAM or FECECAM, the safety net of loans to meet mid-season needs was removed. For example, one interviewed farmer remarked, “When we had credit, we could respond to crop problems immediately (i.e., pests). Fertilizers, fuel for the water pump, replace pipes. We were also able to increase the number of crops they tomato, onion, beet, carrots. Before we only grew tomatoes and other vegetables.” Trainings related to finance Recall learning the topic Continue to use the information today Of those that recall learning the topic percentage that continue the practice Entrepreneurship 54 54 100% Market information 43 37 86% Savings planning/ financial management 36 27 75% Access to loans 40 27 60% Business plans 31 27 87% Table 11 —Project activities under objective 3 and the percentage who continue to use what they learned from the project. The GREEN project included a farmer financial training component where farmers were assisted in creating a financial plan and supported through the loan application process. Despite all farmers receiving financial training and assistance accessing loans, not all farmers continued to use this knowledge to access credit. Potential explanations heard by evaluators for why farmers did not put this new knowledge to work beyond the project included complexity of the application process, discouragement following subsequent loan application, or gender discrimination. However, having 52 | CASE STUDY — Benin  | Findings the opportunity to learn and practice financial literacy and having a credit history did enable some motivated farmers capacity to continue accessing credits. One lending agency commented in interviews that “They [farmers] would not have had these same lending practices without GREEN. For some of the people who came to GREEN, they developed a good credit habit and acquired things they could use for collateral. Where I used to be (before GREEN) [farmers] would come individually, During GREEN, [farmers] were considered as groups in terms of responsibility, but loans were given individually. It is a moral guarantee. After GREEN they began to come as individuals again, and they get the loans under our conditions (not the conditions during GREEN).” Compared to horticultural production and post-harvest trainings, finance and business planning were less likely to have continued since the end of the project. Nevertheless, results from farmer workshops indicate high levels of continuity for entrepreneurship, business planning, and leveraging market information. Approximately 86% (37/43) of beneficiaries surveyed still utilize market information data, while 87% (27/31) continue to practice savings planning or financial management. Farmers’ ability to access loans after the project ended is relatively limited compared to the successful continuation of other practices. Only 60% (27/40) of farmers report that they continue to access loans after receiving this training. PfD staff who were involved in the project’s implementation reported that all farmers received the same set of financial trainings, regardless of whether they were ultimately successful in procuring loans during the project. It is likely that only the subset of participants that were initially successful in obtaining loans have continued to access these services after the project ended. Confusion among farmers about whether they were actually responsible for repaying loans, or whether repayment was guaranteed by PfD likely contributed to this outcome. One farmer interviewed noted: “It seems that the banks did not give second loans to even those who repaid the loans since they didn’t have the ‘guarantee’ that they had with PfD. With no collateral, no one was able to get a second loan after GREEN ended.” Workshop respondents also expressed discontent with how the project ended without creating alternatives to access credit for agricultural purposes. Farmers reported that the loan repayment terms were not conducive to allowing them to correctly time their purchase of inputs and equipment for planting. Participants reported that financial institutions were not understanding of their vulnerability to environmental events like flooding and pest damage: “Because some [farmers] didn’t pay back their loan, they couldn’t get more credit. Some farmers had lots of floods and couldn’t pay back the loan.” In addition, some participants felt that the loan repayment schedule was unachievable since it required payments before farmers were able to earn income through produce sales. For their part, leadership within the project’s partner organization FECICAM/CLCAM noted that while they hoped that growers would continue to access their loan services many were not eligible for future loans or were not able to supply the needed collateral to qualify. Banking institutions ultimately considered farmers too great of a financial risk to continue to dispense loans after the project. CASE STUDY — Benin | Findings | 53 What are the impacts related to Objective 3? Project beneficiaries outlined multiple impacts experienced as a result of continuing with specific project activities and technologies within objective 3. Overall as a result of training in finance and business planning, and facilitated access to loans and better market information participants in workshops described impacts such as better connections, networks and relationships, more money, pride, and continued access to loans. Impacts 0 Pride More Money Crop Diversification Improved Nutrition Better Health Access to Markets Connections Time Gain Increase in Production Access to Loans Access to Market information Food Security 51 10 5 20 25 30 35 40 45 Number of Respondents Reporting Impact Chart 6 — Impacts associated with activities under objective 3. The impact of improved connections was mostly a result of the facilitated access to finance and market information, while the impact of access to loans was more likely a result of people’s continued (although limited) use of business plans, since these plans are a requisite to receiving a loan. For example beneficiaries who now manage their farm with a business plan are also more likely to say that access to loans have been an impact of the project. So while not all beneficiaries had successes with the loan program those who benefited during the project seem to be more likely to still apply for loans. Again more money and pride are impacts of objective three, for many of the same reason as previously mentioned. Conclusions The most successful project components, that is the most continued by participants were those related to crop management, the introduction of high value horticultural crops, safe pesticide handling, including composting and “organic” plant protection methods. The impacts experienced 54 | CASE STUDY — Benin  | Findings by beneficiaries who are continuing to implement these project activities are pride in their work and in their communities and more money from the sale of on farm production. Other impacts experienced are improved nutrition, better health, and better access to markets. Interestingly the impacts that participants mentioned the least were related to two key project components; access to loans and market information. Finally, food security was the impact least likely to be mentioned. Food security is very difficult to measure, especially with this type of data collection, beneficiaries described having more money and better nutrition, two things that alleviate food insecurity, the low results could be related to translation, the framing of the question or other factors. Why were these project components more likely to have continued and to be valued by project participants? A number of factors facilitated the continuation of practices, including; existing policies and positive environmental conditions, the combination of training, finance, and market integration, the use of existing structure and implementing partners were the factors in place that most likely contributed to the successful components of the project. These are explored in more detail below. Conditions that increased the likelihood of sustainability Use of existing organizational partners. Farmer Associations provided critical project support and access to farmers during project implementation. Training based on needs identified by farmers’ associations helped create relevant content, although not all felt PfD was successful at translating need into the right content. Responding to feedback helped farmers learn better. The GREEN implementing partner, PfD, selected topics based on each farmer association, with the assumption that the farmer organization correctly identified farmers needs. Farmers stated that “Technical itineraries brought to us by PfD after identifying the need for farmers to be trained on how to better farm vegetables.” Similar to GREEN, USAID’s successful Integrated Initiatives for Economic Growth in Mali program (IICEM) used partner farmer organizations, unions, and community￾based organizations to achieve project goals of: 1) improving production and productivity via crop intensification and diversification; 2) enhancing financial services; 3) increasing access to markets and trade (Abt Associates, Inc., 2011 and International Business & Technical Consultants, Inc. (IBTCI), 2013). Site selection — Project GREEN focused on specific locations in southern Benin and specific crops, mainly vegetables. While the specific locations in southern Benin never changed over the life of GREEN, there was flexibility, by region, in the importance targeted and emphasized crops. This sustained focus on farmers in specific locations, coupled with the targeted yet site-specific crop emphasis, most likely contributed to more lasting project outcomes. Monitoring and learning — Midway through Project GREEN there was a change in Chief of Party (COP). While this can cause disruption, it appears that mid-management personnel were highly capable in their capacities with the project. Bringing in a new COP who listened closely to the advice of the staff allowed for a re-evaluation of key components of the project with probably resulted in significant improvements, and undoubtedly influenced and improved overall project success. Existing policies — Changes in land tenure policies and other preexisting initiatives aimed at incentivizing community cohesiveness was important in the sustainability of project practices. Our understanding was that the land tenure reforms allowed more women access to land and for CASE STUDY — Benin | Findings | 55 farming. This was significant because the majority of the beneficiaries of the GREEN project were women. Interviewees mentioned that cohesiveness in community organizations, including farmer groups and cooperatives before the project, provided a solid foundation for the sustainability of project activities “One thing the government did was that the government reorganized us into local, regional, and national level cooperatives. This made us able to access these types of projects. At that time, they did codification of farmer groups, and each group had to register.” “Around this time the government passed a law that allowed women to inherit the land, so he can’t say that GREEN is the result, what was the result from GREEN is that women had access to credit.” one of the farmers said. Responded to need — The project was designed and implemented in such a way that activities addressed specific needs of growers and the growers associations, to connect to markets and have access to current commodity prices. Respondents with greater access to (physical) markets (Cotonou and Grand Popo) noted their asymmetric access to commodity price information. Learning about how to access market information and the importance of current market data, increased respondents’ continued practice of market information. Respondents in Couffo and Oueme with less access to (i.e. more remote access from) markets before the project, also valued project efforts to create new market access, motivating them to continue the practices they learned from the project. “Actime would produce mostly cabbage, but the market didn’t know that the cabbages came from here. The market ladies came here to buy them; with GREEN, they took us to the actual market, and we saw everyplace that sold cabbages, and we were able to sell the cabbage to more people, and the price went up due to higher demand.” (From farmers in Cotonou and Grand Popo) “When they started using GREEN practices, they had peace at home. They did not have access to the market. Now the buyers are coming to them. From being connected with buyers, they are making it more of a habit. This is also helping them to lobby for better road infrastructure now that people know that they are producing crops.” (Farmers from Couffo and Oueme) Responded to and leveraged existing natural resources — Interviewees and farmers believed that physical resources like access to water and land facilitated the adoption and sustainability of project outcomes. One of the farmers said, “We had our own practices, but the green project hired facilitators to teach us about the use of pesticides and fertilizers and how to improve production. We did have water, but the project helped us improve the use of water for production and helping set up irrigation systems. The project introduced us to improved seeds” Combination and sequencing of activities — Most interviewees expressed that GREEN’s training content and delivery, in combination with the provision of credit, had no precedence in Benin. These respondents believed that this combination contributed to the continuation of practices beyond the life of the GREEN project itself. “Everything, health nutrition, training and safety for working, protection (safe agricultural practices), how to eat healthy, and financial training. This [financial training] is what distinguished PfD from the other partners; this was the most important thing they did, the financial part. If you ask us, this was the most important part of the project. If you graded all the partners, PfD would come first because they trained us in everything.” Throughout the project evaluation process, it became clear that market links were more valuable than initially understood at the start of project implementation. The new PfD Chief of Party hired at GREEN’s mid-point, made important changes to the way the project connected farmers to existing markets for their crops. Respondents commented that market connections were not part of the initial 56 | CASE STUDY — Benin  | Findings stages of the project. Multiple staff expressed that connecting farmers to market buyers prior to other trainings may have helped farmers to better understand the relationship to demand. Despite a USDA market assessment of in-demand crops prior to project implementation, a few respondents expressed that some promoted crops did not have a market outlet, and that hindered sustainability. One respondent stated “I would have introduced connections with the markets sooner. This happened towards the middle. The market women usually go to the farm to buy the vegetables, so the farmers do not know what is actually needed at the market.” Conditions that decreased the likelihood of sustainability Poor and inconsistent infrastructure- According to a report by the Program of Accompanying Research and Agricultural Innovation (PARI) in 2017, factors such as degraded roads; volatility of international prices of key agricultural products; land insecurity, especially in the south of the country; low involvement of small-holder farmers in decision making; limited access to credit for agricultural activities; malnutrition and lack of infrastructure amongst others significantly impacts Benin’s agricultural activities. Observations made from our assessment of this project showed that these factors still exist and existed prior to the project, and did impact the sustainability of the project. Interviewees expressed that before the project, some farmers had limited capacity to physically bring produce to markets due to poor road infrastructure, reports indicate this barrier continues to exist. “Poor infrastructure and poor ability to bring products to market. Especially when it rains.” Response to market demand and preference — The project offered training in post-harvest practices that would reduce food loss and create new products. However, some farmers felt there was no market for some of the processed products. “For vegetable leaves, people do not like the processed ones. They prefer fresh ones. For tomatoes and peppers, they show reluctance in this area; our markets are in town.” Another participant states that “We also learned how to make tomato paste. However, we don’t have the materials, and it is too hard. You need to have many things, and we know how to do it, but it is too difficult.” Increasing access to market systems information was managed by GREEN and a contractor, ESOKO based in Ghana. While growers were able to describe the benefits from having access to this immediate price information, allowing them to get better prices from the market of their choosing. ESOKO set up an internet based platform for sending real-time information on market prices of a variety of fresh vegetables. Prices reached the producers through SMS messages on their mobile phones. Agents within the markets are responsible for collecting market prices and forwarding them to the PfD manager who validates the prices and then sends them to ESOKO for distribution through the platform. Growers were able to sign up for a subset of alerts, for specific crops, specific markets and a combination of both. Some beneficiaries complained about the discontinuation of ESOKO and the lack of an alternative, however overall growers were not willing to pay for these services. One interviewee said that “With the use of cell phones to access market information, now that ESOKO is not working they don’t have information. So now they have to speculate on which market to sell. They incur losses this way.” Others also described that while they were not using the formal SMS system, they now understood the value of getting that information, and would sometimes seek it out in other ways, calling family or market women in different regions. CASE STUDY — Benin | Findings | 57 The case of credit Responses were mixed when it came to the sustainability of the credit access program. In the 1970’s the government of Benin established the Caisses de Crédits Agricole Mutuels (CCAM) to mobilize savings to fund agricultural activities (Mylène Kherallah 2001). Agricultural reforms under the Structural Adjustment program by the World Bank in the 1990’s resulted in the liberalization of food marketing; reforms in inputs markets where the distribution of farm inputs was privatized; agricultural finance where CCAM was restructured to promote independence at the local level resulting in CCAM renamed into Caisse Locale de Crédit Agricole Mutuel (CLCAM). Even though the CLCAM are to facilitate the access of credit to farmers, farmers believe the CLCAMS provided loans to them only when the loans were guaranteed by the government or non￾governmental organizations. Respondents observed that credit institutions still perceive farming as a risky economic activity, causing the disruption of most credit services to GREEN beneficiaries after the project ended. According to the farmers, amongst the factors that has contributed to the CLCAMS perceiving farming as a risky activity are the unreliability of weather patterns that affect productivity, and people’s histories of not repaying loans provided by foreign sponsors. Interviewees mentioned that cohesiveness in community organizations, including farmers groups and cooperatives before the project, provided a solid foundation for the sustainability of project activities “The problem now continues to be the conditions and the time required to repay the loan. With farming, sometimes the rain doesn’t come, and the dry season comes later. Sometimes it rains heavier, and they lose their harvest. When these problems occur, it is hard to repay.” according to one of the farmers. “Anyone can get credit; the challenge is the repayment. If there is a project that wants to accompany the producers, and then they follow them around until they pay back the loan. So, if a project will do that, then they will get the credits. Now when a project comes, and people know that then the people won’t pay. (if people know that a foreign project is sponsoring, the people don’t think they need to pay).” After GREEN ended, some farmers reporting losing access to credit and a souring relationship with the MFI. Some shared that without program staff to connect farmers to loan institutions, they did not feel confident accessing loans on their own. “When the project ended, people were not happy. In the end, it seems like everybody dropped them; they didn’t continue a relationship with the financial institutions. Now there isn’t any more these types of credit. Because the financial institutions were being backed by the project.” In interviews with CLCAMS we were assured that this was not the case, their policies have remained the same and anyone meeting those conditions can apply for a loan. In areas where loan officers had a positive relationship with their clients both were in agreement that getting loans and making timely repayment can be difficult but when they worked together and shared trust and confidence beneficiaries were more likely to return after the project to access a loan. 58 | CASE STUDY — Benin  | Findings SOURCES African Development Fund (2005). Republic of Benin Results-Based 2005-2009 Country Strategy Paper. Retrieved from https://www.afdb.org/fileadmin/uploads/afdb/Documents/Project-and￾Operations/ADF-BD-WP-2005-63-EN-BENIN-2005-2009-CSP.PDF https://www.usaid.gov/benin/our-work USDA Foreign Agricultural Service (2014). Global Agricultural Information Network: Benin Agricultural Situation. Benin-Geography. Retrieved from https://www.globalsecurity.org/military/world/africa/bn￾geography.htm World Atlas (2018). Geography Statistics of Benin. Retrieved from https://www.worldatlas.com/ webimage/countrys/africa/benin/bjlandst.htm FAO (2018). FAO Country Profile: Benin. Retrieved from http://www.fao.org/countryprofiles/index/ en/?iso3=BEN Innovation forSustainable Agricultural Growth in Benin (2017). Research and Agricultural Innovation (PARI). https://research4agrinnovation.org/wp-content/uploads/2017/10/BeninDossier2017.pdf CASE STUDY — Guatemala | 59 CASE STUDY — Guatemala 60 | CASE STUDY — Guatemala  | Summary Summary The Counterpart FFPr (Food for Progress) project in Guatemala, implemented between September 2012 and January 2017, supported the USDA FFPr objectives of increased agricultural production and trade by increasing the use of improved agricultural techniques and technologies among small￾holder producers and improving linkages between buyers and sellers. The project was designed based on the assumption that the strengthened organizational capacities of producers, increased access to agricultural finance, and expanded agricultural infrastructure are the building blocks necessary for increasing the agricultural production and trade. The three main objectives were:: 1. Improve the capacity of the Guatemalan Ministry of Agriculture’s extension system. 2. Expand the financial services available to agricultural producers by developing financial products tailored to their needs. 3. Increase the organizational capacity of local cooperatives. This post project sustainability assessment conducted workshops and interviews with 238 individuals to determine what project lessons and activities have been continued since the project closed, and what the impacts of continuing these activities has been on beneficiaries and communities. The assessment collected data from; direct beneficiaries, implementing partners, trainers, loan officers, and community leaders. A mixed methods approach was used, including interviews, desk research and workshops with project participants. Findings Most activities project participants were trained in continue today, especially activities related to agricultural practices for improved grain and horticulture production. By increasing the knowledge and technical know-how of farmers, farmer groups, extension agents, and promoters, the project provided communities with knowledge and skill sets that could sustain beyond the support of the project. Farmers attributed this believe their increased capacity has from trainings contributed to greater income (more money) improved community and personal pride and better health. All four new loan products remain on offer from the MICOOPE books and lending in these product categories continues within the project area and, increasingly, beyond. While beneficiaries who participated in data collection did say they continue to access these loans, they didn’t not feel they were directly related to helping them experience project impacts such as more money or improved production. Overall the small grants provided to groups helped farmer associations to overcome previous barriers created by lack of access to basic inputs. Farmer associations have increased use of improved financial and organizational management practices. This evaluation revealed several factors that helped to increase the likelihood of that project impacts would be sustained. For example, preexisting conditions such as a high demand and need for agricultural extension support as well as existing market linkages helped incentivize local community participation in the project and promote the long-term sustainability of project activities. In order to ensure that training was valuable in local communities, trained extension agents conducted needs assessments to help identify the types of training that would be most CASE STUDY — Guatemala | Evaluation Purpose  | 61 valuable based upon local community environmental conditions and markets for agricultural products. Due to the vast variety of micro climates in the highlands of Guatemala, it is like that a standardized approach to training would not have been as successful. Getting local input when designing training helped successfully respond to local environmental conditions and increased sustainability. The sequencing of project activities to combine technical training for communities and extension agents, organizational capacity building training, and access to micro-financing also proved effective. Working with local credit unions such as MICOOPE helped increase the accessibility of credit to many farmers. Several factors also served as impediments to the long-term sustainability of project impacts. For example, farmer aversion to credit continues to be a factor that prevents small-scale producers from taking advantage of the agricultural credit products developed by MICOOPE with the support of the project. In addition, the high-turnaround of extension agents within the Ministry of Agriculture and Ranching meant that many of the extension agents trained through the project no longer work for MAGA. Finally, while implementation of the project allowed for significant stakeholder input and flexibility, the actual design if the project did not engage in much stakeholder feedback. If key partners such as MAGA had been engaged earlier in the project, it is possible that some roadblocks, such as temporary extension agent contracts, might have been avoided. Evaluation Purpose The Office of Capacity Building and Development (OCBD) is a program area of USDA’s Foreign Agricultural Service that is dedicated to advancing global trade, agricultural development and food security. Programs administered by OCBD help to reduce food insecurity, strengthen agricultural economic growth in targeted developing countries along the full value chain of development, build their capacity to integrate into the global economy, and become viable agricultural trading partners with the United States. The objective is to help move these countries along the agricultural market-development continuum from developing economies, where capacity building is the primary focus, to developed economies, where market development and expansion are the primary focus. Authorized by the Food for Progress Act of 1985, the Food for Progress (FFPr) program provides for the donation of U.S. agricultural commodities to developing countries committed to introducing and expanding free enterprise in the agricultural sector. Donated commodities are monetized (sold on the local market) and the proceeds are used to support agricultural development activities. Food for Progress has two principal objectives: improve agricultural productivity and expand trade of agricultural products. Food for Progress programs have worked to train farmers in animal and plant health, improve farming methods, develop infrastructure (roads, electricity, storage, etc.), establish and strengthen producer cooperatives, provide micro-credit, and develop agricultural value chains. FFPr Program Participants have included private voluntary organizations (PVO), foreign governments, universities, and intergovernmental organizations. FFPr projects are currently active in 25 countries across Africa, Asia, and Latin America and the Caribbean. The University of Minnesota was contracted by the USDA FAS FFPr office to assess the ongoing impacts of 3 closed Food for Progress Projects: 1 each in Africa, Asia, and Latin America and 62 | CASE STUDY — Guatemala  | Evaluation Questions Caribbean. The final countries and projects were selected in join consultation with USDA. The specific questions were designed in close collaboration with USDA, including a workshop conducted with USDA in November 2017 and over a period of months to refine questions and methods. This evaluation was not expected to include experimentation or trials on human subjects, as the programs to be evaluated have all concluded. Each closed project is presented as a case study, and the three projects together are a part of the full “Post project sustainability assessment” report. The Organization for Economic Cooperation and Development (OECD), Development Assistance Committee (DAC) has defined “sustainability”, in the context of evaluation and results-based management as: The continuation of benefits from a development intervention after major development assistance has been completed. The probability of continued long-term benefits. The resilience to risk of the net benefit flows over time. The Food Assistance Division, in its Monitoring and Evaluation Policy defines “sustainability” as: Assessment of the likelihood that the benefits of the project will endure over time after the completion of the project. Sustainability should also assess the extent to which the project has planned for the continuation of project activities, developed local ownership for the project, and developed sustainable partnerships. The case study that follows is one of three studies that are a part of a full report provided to USDA FAS on Dec 31 2019. Evaluation Questions 1. How did existing conditions affect project sustainability? A. What are the preliminary conditions that set up market linkages for success? B. To what extent are results more sustainable in low, or middle-income countries? 2. What components of the project’s design contribute to sustainability? A. To what extent were local stakeholders involved in project activity design? 3. In what ways did the implementation of the project contribute to its sustainability? A. How do local partnerships contribute to sustained project outcomes and impact? B. What factors lead to training results being sustained? C. How does the combination and sequencing of activities influence sustainability? 4. What activities or practices have continued (and to what extent) since the project ended? A. How valuable to communities are the project’s outcomes, including activities and practices? 5. What are the impacts of the project? A. How valuable to communities are the impacts of the project? B. How did beneficiaries and their communities respond to subsequent shocks? C. To what extent have the impacts of the project contributed to equity? CASE STUDY — Guatemala | Project Background | 63 Project Background As of 2016 Guatemala accounted for nearly ¼ of Central America’s GDP making it the largest economy in the region (USAID, 2017).The country’s GDP is $75.62 billion with a GDP per capita of $8,200 (2017 est.). This is half the average GDP for countries in Latin America and the Caribbean (CIA, 2018). Income inequality is a problem in Guatemala. Over half of the country lives below the national poverty line and 25% of the population lives in extreme poverty. 79% of the indigenous population is living below the poverty line and 40% is living in extreme poverty (CIA, 2018). About 31% of the country is employed in agriculture, 13% is employed in industry, and 56% if employed in the service industry. Guatemala’s main industries include sugar, textiles and clothing, furniture, chemicals, petroleum, metals, rubber, and tourism (CIA, 2018) The Western Highlands (excluding Solola department) have a large population and an extremely high level of vulnerability. According to the “Diagnostico de la Region de Occidente de Guatemala” (Diagnostic Report for Western Guatemala) produced by MAGA, 75% of municipalities are considered highly vulnerable with an additional 13% considered in mid-vulnerability. Approximately 2.1 million people of the 2.8 million living in this area currently live under the poverty line. For this reason, the region has also been singled out by the Guatemalan Government with their “Plan Occidental” and by the US Government through the Feed the Future initiative. Project have traditionally focused on two populations: those with land and production that allow them to immediately enter an export market and the extreme poor who receive humanitarian food assistance. Agriculture is one of the primary economic drivers in Guatemala. The sector employs one out of three Guatemalans (IMF, 2017). The agricultural sector in Guatemala accounts for 13.5% of the country’s GDP. The crops with the highest cumulative gross production value in Guatemala include bananas, sugar cane, rubber, coffee and corn (USAID, 2017). Agriculture accounted for 45% of Guatemala’s exports in 2015. However this percentage has actually been decreasing since 2001. Guatemala’s largest export market for agricultural products is the United States followed by El Salvador and the Netherlands (USAID, 2018). Guatemala’s main exports include bananas, raw sugar, coffee, palm oil, knit sweaters. Guatemala’s top export destinations include the United States, El Salvador, Honduras, Canada, Mexico and Nicaragua. The country’s main imports include refined petroleum, cars, packaged medicines, broadcasting equipment and delivery trucks. These imports primarily come from the United States, Mexico, China, El Salvador, and Costa Rica (OEC, 2018; CIA, 2018). Guatemala also receives the most remittances out of all countries in Central America due to the country’s large community in the United States. Remittances are equivalent to ⅔ of Guatemala’s exports and account for about 1/10 of its GDP (CIA, 2018). Figure 3 —Ccolor green indicates visited regions by the evaluation team in Benin 64 | CASE STUDY — Guatemala  | Project Background Households in Guatemala suffer from the fourth highest malnutrition rate in the world and the highest malnutrition rate in Latin America. Currently, 50% (up to 65% in indigenous communities) of Guatemalan children are physically stunted due to lack of access to nutritious food. Unstable weather patterns (exacerbated by global climate change) intensify food insecurity by harming subsistence crop production and limiting household nutritional options (USAID, 2016). As is the case in much of the global south, poor and marginalized populations suffer disproportionately from food insecurity (WFP, 2017) Some of the major challenges that Guatemala faces related to its agricultural sector include access to appropriate infrastructure, access to finance, access to agricultural insurance, and customs bottlenecks for international trade (USAID, 2017). Major environmental issues faced by Guatemala include rapid deforestation in the North, soil erosion and water pollution (CIA, 2018). Counterpart’s 2012 Food for Progress project in Guatemala responded to these needs and focused on three main objectives: 1) Improve the capacity of the Guatemalan Ministry of Agriculture’s formal extension agents and certified non-governmental agricultural advisers to provide technical support to agricultural producers for increased productivity and expanded trade; 2) Expand the financial services available to agricultural producers by developing financial products tailored to their needs, training credit unions and agricultural extension agents and facilitating new market linkages between buyers and sellers; and 3) Increase the organizational capacity of local cooperatives representing smallholder producers through the identification of areas of critical weaknesses and executing beneficiary-owned organizational development action plans. The project supported the USDA FFPr objectives of increased use of improved agricultural techniques and technologies among smallholder producers and improving linkages between buyers and sellers. The project was designed to strengthen organizational capacities of producers, increase access to agricultural finance, and expand agricultural infrastructure, all prerequisites for increasing the agricultural production and trade. The project was designed to meet these objectives through the following project activities: • Coordinate with the Ministry of Agriculture’s (MAGA) extension services to strengthen agricultural extension system and the technical support provided to agricultural producers through the Government of Guatemala’s “Sistema Nacional de Extensión Rural” (SNER) • Introduce new and expand existing financial services for cooperatives by linking them to buyers and financial services, provide training on financial needs and issues of agricultural activities, and by providing technical support to extension agents to train potential buyers of local agricultural products • Train farmers and cooperatives through Agricultural Extension Agents on the use of new techniques and technologies to improve agricultural productivity • Build the organizational development capacity of cooperatives through technical and financial support. The project was focused on the municipalities of Quiché, Huehuetenango, San Marcos, Quetzaltenango, and Totonicapán. The project “targeted specific areas of the country and specific farmers with the highest likelihood of success in export crops” (Final evaluation report, 2017). CASE STUDY — Guatemala | Project Background | 65 Summary of Evaluation, 2017 The project’s final evaluation report (2017) summarized the achievements of the project upon completion based upon the projects three main objectives: extension agent training, provision of financial services and organizational development. The report outlines a list of 37 indicators utilized to evaluate project success and reports that the team found evidence that 35 of those indicators were fully achieved. For the two remaining indicators, both focused upon the increased use of financial services, the report explains that while the evaluators saw improvement there was not sufficient evidence to report that those indicators had been achieved. The project report concluded that the overall sustainability of these results would ultimately depend upon the degree to which these processes and changes are institutionalized. The project achieved or exceeded all targets related to the agricultural extension component and evaluators pointed out that the approach of training MAGA extension agents and technicians who would then train farmers and farmer associations/cooperatives was effective. Allowing farmers and farmer groups to select topics of the trainings helped ensure that the training being provided by extension agents was applicable, relevant and up-to-date. In addition, this component of the project was being institutionalized through a partnership with the University of San Carlos. However, the final project report explained that one key assumption of the extension training component of the project was that the Guatemalan Ministry of Education would continue to support trained extension agents and the extension training program. In reality, turnover within the Guatemalan Ministry of Agriculture resulted in less continuation of trained agents in extension positions than had been originally anticipated. While the multi-level approach was seen as a strong design element, lack of continued support from the Guatemalan government proved to be a limitation. The financial services component of the project was considered to be one of the most successful at project close out due to the generation of four new agricultural loan products and the loan analysis tool that was developed. Key results from this objective included the creation of products tailored for agricultural producers, improved understanding of agricultural client needs and constraints, reduced loan processing time, interest rates, and overall portfolio risk as well as more capital available to agricultural producers. However, evaluators at project close out could not find evidence to support increased use of these financial services by beneficiary farmers. Nonetheless, due to the benefits of these results, evaluators expressed confidence that they would be institutionalized. The project also exceeded all targets set for the organizational development component and evaluators praised the beneficiary-driven approach the project took of improving organizational development through self-evaluations and the strategic development of action plans. The project worked with a large number of producer groups dispersed over a large geographic region. However, evaluators also added that while there was a high demand and appreciation for organizational development support, Counterpart was the only organization providing this type of support in the Western Highlands and were unable to identify a core institution that could build off the achieved results of the project. The large number of beneficiaries also limited the ability of the project to achieve better economies of scale and collaboration between beneficiary groups resulting in weaker linkages between the farmer training and financial services components of the project as well as underdeveloped market linkage services. While the facilitation of new market linkages was not widely achieved, the project had more success strengthening existing market linkages 66 | CASE STUDY — Guatemala  | Project Background Evaluation Methods and Limitations Sample Selection A wide range of stakeholders participated in the evaluation. Implementing partners identified participants that fit various types of stakeholders identified by the evaluation team as well as where to conduct Image Grouping©. Interviewees were selected based on their knowledge and expertise of sustainability in the region. The implementing partner chose San Marcos and Huehuetenango as sites for data collection. These sites were chosen by the implementing partner based on ease of accessibility for farmers to participate, as well as proximity to farmers who had a variety of dosage levels with different project activities. The data collection team also administered interviews with the implementing partner in Guatemala City and Quetzaltenango. The evaluation team conducted outreach with the intention of conducting 25 interviews across all stakeholder types at each location. Specific participation counts and stakeholder information can be found in the Interview Population table 13. A few select beneficiaries participated in multiple forms of data collection within their respective site; as a result, the grand total in the Data Collection Counts table 14 does not represent unique individuals that participated in data collection at a site. Image Grouping© participants were selected using a stratified convenience sample. This allowed the evaluation team to select groups based on additional hypotheses relevant to sustainability. All Image Grouping© participants are farmers. The criteria for the selection included attention to: diversity of locations, a balance of genders, and variety and dosage of activities farmers participated in. Specific gender breakdowns can be found in the Image Grouping© Sample, By Sex and Region (table 12). Data collection was administered in Spanish, however the range of languages participants speak, as shown in the Languages Participants Speak table 15, demonstrates the variety of demographics present in the sample. The activities that participants had to choose between came directly from a review of project documents and consultation with the implementing partner. Location Female Male Number of Participants Huehuetenango 16 33 49 San Marcos 15 33 48 Grand Total 31 66 97 Table 12 — Workshop sample, by sex and location Interviewees were selected in close consultation with Counterpart International field staff. Selection was based on key informant expertise, knowledge of project activities, level of participation in various project components, and their current level of involvement with Counterpart International. Key informants were purposely selected to represent a variety of experiences with project participants and across project components. CASE STUDY — Guatemala | Project Background | 67 Stakeholder type Huehuetenango San Marcos Guatemala City Quetzaltenango Total Extension agent 5 5 1 11 Financial institution 2 1 3 Implementing partner staff 2 4 1 7 Farmer association/ cooperative 2 5 1 8 Financial Institution loan recipient 2 2 Government 3 3 Grand Total 13 10 10 1 34 Table 13 — Interview population For the e-survey, the team followed the approach used in the final evaluation with the extension agents. Counterpart International provided access to the email addresses of the extension agents trained in the certificate program through all seven cohorts during the project implementation. The team designed a survey instrument to be accessed using Qualtrics as the tool. The survey questions focused not on evaluating the training but collected demographics, employment status, ongoing use of the topics in which they were trained and how they worked with people in the community, cohort group number, specific content information and the application of the knowledge used since the project ended whether still employed by the Ministry of Agriculture or working in a related field. Region Image Grouping© Workshop Interview Huehuetenango 2 13 San Marcos 2 10 Guatemala City 10 Quetzaltenango 1 Grand Total 4 34 Table 14 — Data collection 68 | CASE STUDY — Guatemala  | Project Background Language Number of Participants Aguacateo 2 Cancobal 5 Inan 5 Mam 1 Popti 1 Spanish 113 Table 15 — Languages used in data collection Data Collection The evaluation team conducted interviews with key stakeholders to understand the local context and gather insights to inform culturally appropriate evaluation design. Findings from this report come from primary data collected through: • Image Grouping© • Interviews • Observations • Surveys Local coordinators and the implementing partner helped identify and recruit prospective participants. The evaluation team—of UMN faculty, graduate students, and Improve Group consultants—conducted in-person interviews, field observations and facilitated Image Grouping© sessions. The team also administered an online survey to a list of government extension agents from the project. The number of participants that engaged in each data collection method can be seen in table 16. Data Collection Method Number of Participants Image Grouping© 97 Interviews 34 ESurvey 107 Table 16 — Data collection method by number of participants CASE STUDY — Guatemala | Project Background | 69 The survey consisted of 23 questions. The survey was translated into Spanish prior to setting up the tool online. An email inviting 502 people to access the survey was distributed in May 2019 remaining accessible for ten weeks. Of this total, 4 emails bounced; 43 were duplicate email addresses. Initial logins of 170 people were documented; one reminder was sent to encourage individuals to complete the survey. The completed responses totaled 107 for a response rate of 23.5%. Every cohort had respondents who contributed to the survey. Analysis Methods The evaluation team conducted qualitative analysis in Dedoose software to analyze the in-depth interviews and Image Grouping© discussion notes. Initially, the evaluation team developed codes using an inductive approach. These codes were developed by identifying themes from literature on conditions needed for sustainability, as well as the evaluation questions. Following that, a deductive approach was used to find new themes based on the responses. The evaluation team lastly looked at relationships between themes and compared differences between different types of respondents, such as farmers and other key stakeholders. Quantitative Image Grouping was analyzed using Excel. Frequencies and cross tabs were conducted to understand differences between locations, or the activities that resulted in sustained outcomes. Themes across these responses were compared to those found in the qualitative results and integrated to achieve a mixed method understanding of the data. The e-survey data analysis was done using Qualtrics cross-tab feature. Of particular interest beyond descriptive statistics was comparison across employment status, gender, and cohort groups when determining the final results of the project several years after the official project end date. The respondents were 63.9% men and 36.1% women. The respondents were trained in 7 cohort groups. The respondents broke out into the following groups by percentage: Cohort 1 – 13% (14 people) Cohort 2 – 13% (14 people) Cohort 3 – 10% (11 people) Cohort 4 –13% (14 people) Cohort 5 –16% (17 people) Cohort 6 – 5% (5 people) Cohort 7 – 28% (30 people) The data analysis of the emailed survey was done using Qualtrics cross-tab feature. Of particular interest beyond descriptive statistics was comparison across employment status, gender, and cohort groups when determining the final results of the project several years after the official project end date. In addition to data collected in the field, a literature review was conducted to both inform our data collection and compare our findings to existing academic literature as well as program evaluations from other development project reports. Relevant academic literature was identified based upon relevance to agricultural extension, community agricultural development, community capacities, and relevant research in Guatemala’s western Highlands. Project reports/evaluations were identified based upon their relevance to the region and their focus on agricultural extension. 70 | CASE STUDY — Guatemala  | Project Background Limitations With all evaluations, there are limitations to the data that are gathered and the conclusions that can be drawn. Recall and response bias may be present in the data. The evaluation team also recognizes that this study did not use an experimental design which means there are limits to conclusions about impact. Recall bias may have occurred due to the time that has passed following the closure of projects. This can lead to participants remembering events at a different scale or forgetting what they did at that time. As a result, some farmers may have reported participation in trainings and services mostly in areas that they still remember and use today, positively biasing findings towards a higher percentage of individuals sustaining practices. Image Grouping© methodology helps to reduce this likelihood by having farmers from similar areas participate in the same session and encouraging them to talk among themselves to remind each other whether they participated in certain activities over others. Positive response bias may have influenced farmers’ responses in two ways. First, in asking how participant lives are today as compared to prior to the project start, farmers may have been more likely to exaggerate the benefits of the project. In many data collection environments, participants aim to “please” data enumerators, responding in a way that they believe the enumerator is hoping for or contributes to their benefit. Farmers may have believed that by extolling the hardship they experienced prior to the project and highlighting how much they continue using today, that this would support future work in their area. Second, implementing partners were used to determine which sites to visit and who to recruit. Partners may have desired to show more successful outcomes for their own audiences as well as potential influence on future funding through USDA. The evaluation team worked to compensate for this potential bias by requesting IPs to suggest sites that had varying degrees of success after being involved with the USDA project. The high rates of sustained practices suggest that this bias may be more prevalent. Therefore, information on what has hindered sustained practices, or challenges farmers face may be even more beneficial in reviewing to understand what to consider for creating an optimal environment for sustainability. This project does not include control or comparison communities and thus cannot make claims about magnitude of impact or causality. As a result, although we were able to describe if income changed for participants and whether they attribute this to their respective projects, we do not know the extent to which income changes were solely the result of the USDA project. Likewise, secondary data did not exist at a regional level to demonstrate need or wealth status prior to project start; therefore, we are unable to make claims about economic changes to regions where the project was implemented. CASE STUDY — Guatemala | Findings | 71 Findings These findings are organized using the FFPr projects 3 objectives. Under each objective we will share the findings that correspond to our high level evaluation questions, found in full on page 62. Most activities project participants were trained in continue today, especially activities related to agricultural practices for improved grain and horticulture production. By increasing the knowledge and technical know-how of farmers, farmer groups, extension agents, and promoters, the project provided communities with knowledge and skill sets that could sustain beyond the support of the project. Farmers attributed this increased capacity contributed to greater income (more money) improved community and personal pride and better health. All four new loan products remain on offer from MICOOPE and lending in these product categories continues within the project area and, increasingly, beyond. While beneficiaries who participated in data collection did say they continue to access these loans, they didn’t not feel they were directly related to helping them experience project impacts such as more money or improved production. Overall the small grants provided to groups helped farmer associations to overcome previous barriers created by lack of access to basic inputs. Farmer associations have increased use of improved financial and organizational management practices. 72 | CASE STUDY — Guatemala  | Findings Activities Number who learned about the activity in 2015 Number who continued to use the knowledge/ inputs 2019 Percent who continue to use what they learned during project implementation Accessing technical experts (Extension Agents) 29 42 145% Individual capacity development 28 40 143% Conflict management within the household 20 25 125% Training in grain and vegetable production 41 51 124% Financial documentation 21 25 119% Org structure and documentation 38 45 118% Relationship building 35 40 114% Conflict management within organizations 16 18 113% Training in creating an annual calendar for production 47 52 111% Improving access to credit 23 25 109% Animal husbandry training 47 51 109% Conducting market analysis 42 44 105% Training on nutrition and diet 35 36 103% Climate change adaptation technology and training 43 44 102% Coffee production training 22 22 100% Training on post-harvest practices 35 35 100% Processing food crops 37 37 100% Improving relationships and connections to local organizations 49 48 98% Food security 39 38 97% Basic production and farm management 58 55 95% Training on agricultural insurance 14 13 93% Washing practices and Household hygiene 54 50 93% Improving value chain 34 31 91% Training on creating a marketing plan 36 31 86% Creating a Business plan 30 17 57% Table 7 — Activities continued from Guatemala project. CASE STUDY — Guatemala | Findings | 73 BETTER RELATIONSHIPS, 128 Improved production and farm managment, 51 Value chain and markteing, 33 Nutrition and food security, 22 Community connections, 16 Organizational capacity development, 7 Loans and financial services, 18 Improved production and farm managment, 32 Value chain and markteing, 8 Nutrition and food security, 32 Community connections, 22 Organizational capacity development, 22 Loans and financial services, 12 Improved production and farm managment, 14 Value chain and markteing, 21 Nutrition and food security, 4 Community connections, 48 Organizational capacity development, 31 Loans and financial services, 10 HEALTH, 125 Value chain and markteing, 12 Community connections, 9 Organizational capacity development, 7 Loans and financial services, 10 Nutrition and food security, 55 Improved production and farm managment, 32 Value chain and markteing, 11 Community connections, 17 Loans/financial, 3 Nutrition and food security, 18 Improved production and farm managment, 42 Value chain and markteing, 23 Community connections, 17 Organizational capacity development, 11 Loan/financial, 3 Nutrition and food security, 12 Improved production and farm managment, 17 Value chain and markteing, 24 Community connections, 20 Organizational capacity development, 6 Loans and financial services, 10 Nutrition and food security, 5 Improved production and farm managment, 10 BETTER NUTRITION, 71 Value chain and markteing, 11 Community connections, 12 Loans and financial services, 10 Improved production and farm managment, 9 Organizational capacity development, 9 Nutrition, 1 Org. ..., 1 Value chain and markteing, 11 Community, 4 Loans, 1 Nutrition and food security, 33 Improved production and farm managment, 21 Value chain, 4 Community connections, 12 Org. capacity devel..., 8 Loans and financial services, 16 Improved production and farm managment, 10 Value..., 1 Community connections, 24 Organizational capacity development, 31 Loans and financial services, 7 Nutrition..., 1 Improved production and farm managment, 6 Value chain and markteing, 5 Community connections, 16 Organizational capacity development, 4 Loans and financial services, 4 Nutrition and food security, 11 Improved production and farm managment, 46 Value chain and markteing, 16 Community connections, 13 Loans..., 2 Org. ..., 1 Nutrition and food security, 43 Improved production and farm managment, 36 MORE MONEY/INCOME, 184 PRIDE—Individual and Commuity, 128 BETTER RELATIONSHIPS, 128 HEALTH, 125 IMPROVED CROP DIVERSITY, 91 MORE TIME, 83 ACCESS TO MARKETS, 75 BETTER NUTRITION, 71 IMPROVED BUSINESS, 56 BETTER ACCESS TO CREDIT, 50 ORGANIZATIONAL CAPACITY, 70 AGRICULTURAL PRODUCTIVITY, 86 FOOD SECURITY, 111 Impacts associated with Guatemala FFPR project activities Numbers indicate numbers of respondents reporting impact Chart 7 — Impacts from Guatemala project 74 | CASE STUDY — Guatemala  | Findings OBJECTIVE 1 The first object of the Counterpart International project was to improve the capacity of the Guatemala Ministry of Agriculture’s formal extension agents and certified non-governmental agricultural advisors to provide technical support to agricultural producers for increased productivity and expanded trade. Through the Food for Progress program, Counterpart International worked with the University of San Carlos, Ministry of Agriculture and UC Davis (initially Michigan State University) to develop a certificate program in extension education. The project was designed based on the assumption that strengthened organizational capacities of the producers, increased access to agricultural finance, and expanded agricultural infrastructure are also necessary for increasing the agricultural production and trade. In order to meet the goals of increasing production the project sought to improve the capacities of extension agents who in turn would train farmers and cooperatives in new and improved agricultural practices and improved technologies. The model adopted by MAGA uses local promoters within CADERs (Spanish acronym for the Rural Development Learning Center) to lead CADER members, set up demonstration plots What activities or practices have continued since the project ended? Workshop participants totaled 97, about one third of whom were women. Participants responded to a five question Image grouping survey asking participants – (the first two questions correspond to Table 14).What activities or practices do you remember learning about through the project? And what activities or practices do you continue to do today? The numerical counts reflected below are the number of workshops participants who recalled learning about, followed by the number who said they are continuing the same practice today. This was followed by a discussion of if they were doing the activity at or around the time the project ended and if there were continuing to do it today why. The qualitative discussion is reflected in our analysis (Table 18). Activities corresponding with Objective 1 Number who learned about the activity in 2015 Number who continued to use the knowledge/ inputs 2019 Percent who continue to use what they learned during project implementation Training in grain and vegetable production 41 51 124% Animal husbandry training 47 51 109% Climate change adaptation technology and training 43 44 102% Coffee production training 22 22 100% Training on post-harvest practices 35 35 100% Basic production and farm management 58 55 95% CASE STUDY — Guatemala | Findings | 75 Conducting market analysis 42 44 105% Processing food crops 37 37 100% Training on agricultural insurance 14 13 93% Training on creating a marketing plan 36 31 86% Accessing technical experts (Extension Agents) 29 42 145% Relationship building 35 40 114% Conflict management within organizations 16 18 113% Improving relationships and connections to local organizations 49 48 98% Conflict management within the household 20 25 125% Training on nutrition and diet 35 36 103% Food security 39 38 97% Household hygiene 54 50 93% Table 18 — Activities that have continued corresponding with Objective 1. Continued Extension Activities As of the time of the extension agent emailed survey, 35% of the men and 33% of the women trained in the diploma program were still employed by MAGA; 26% of the men and 33% of the women were working in the field of extension but with other organizations. Other agents although no longer gainfully employed by MAGA, they are members of a CADER and volunteer time for the sake of the community. Those who still serve as extension agents today report that their main clients are promoters (29.1%), farmers (37.7%), associations (9.7%), or other clients (23.3%) such as students, other extension agents, vulnerable families, entrepreneurs or youth. A series of questions was asked to determine if the extension agents continue to interact with CADERS which play a key role in demonstrating best practices. Among the respondents, 70.0% said they still work with CADERs. The CADERs are located in the communities so they play a key role for dissemination using this approach. Most extension agents participate in CADERs as paid employees (52.1%) or volunteers (16.7%). Extension agents were also asked to rate how effective they think they have been based on applying their knowledge and skills from the diploma program. Most extension agents rated themselves as very successful (38.5%) or successful (55.8%). At the conception of this project, the administration was supportive of the program and it was a priority of the Ministry of Agriculture to build this network. However it has struggled to meet the initial vision for transformation. The initial curriculum developed was well received. However, there were differences of opinion on the best approach to integrate into the training program. While two cohorts were trained under the USAC-MSU curriculum, the Ministry of Agriculture officials became 76 | CASE STUDY — Guatemala  | Findings disenchanted with the approach. The MAGA team wanted a program based on a farmer-to-farmer dissemination process. The Ministry felt strongly that the trickle-down effect of farmers sharing knowledge and practice was the better fit for Guatemala. Eventually Counterpart International reached an agreement with the withdrawal of MSU from the project. The University of California/ Davis then began to work on the training modules for the certificate program using a more farmer￾to-farmer approach supported by the MAGA. Continuation of Project Activities in Communities Workshops with project beneficiaries revealed that most activities continue today, especially activities related to improved agricultural practices, ranging from grains, to animals, to post￾harvest. By increasing the knowledge and technical know-how of farmers, farmer groups, extension agents, and promoters, the project provided communities with knowledge and skill sets that could sustain beyond the support of the project. Farmers believe their increased capacity from training contributed to their ability to maintain the practices they learned about the project as well as teach and learn from others. The capacity-building focus of the project also increased beneficiaries’ understanding on how to access technical experts, connect with other organizations, build relationships, and manage conflict. Farmers also report that relationship building and communication has helped them sustain coordination and maintain networks of technical experts, other farmers and market actors established throughout the project. These relationships have helped farmers exchange ideas, obtain support, and market their products. The most important motivator for continuing to use project activities and practices was experiencing the project’s outcomes. Not only have project activities continued beyond the end of the project, but some practices have been adopted by those who did not originally participate in the training as benefits of the project activities were observed in the community. “People have continued to reproduce what they learned in our trainings. We weren’t the only one doing trainings, but we supported the cascade training approach, based in replication, we gave the inputs, so they could carry out these trainings in their community. This would be easier for communities to understand and pick up the practices and technologies. We feel this is sustainable also because other organizations are using that same methodology. The exchanges on field days to learn about new technologies. We would take them to model farms, so they could see what we were training them on and to see if it was working. This is what we call farmer to farmer.” “The trainings helped us a lot and we told them that we didn’t know, I only got to middle school, this is what it was like for me. We learned to communicate more and better. We would show our neighbors, come look at how this is going, and we tell each other what we are doing and learn from each other; we exchange, ie. seeds and coffee, knowledge, we look, try, see how this is going.” CASE STUDY — Guatemala | Findings | 77 What are the impacts of Objective 1? Impacts of Extension Training Guatemala is benefiting from the cadre of individuals trained in extension education. While full time employment with MAGA has dropped, individuals are being hired by other organizations continuing to work in agriculture and development in rural Guatemala. The certified agents recognize through the training in the modules how to enter in communities, the need to communicate and have regular contact with clients in the community, and the need for ongoing professional development. The respondents to the survey rated themselves as contributing to the MAGA policy priorities for addressing the needs of rural women, access to native and traditional foods, nutrition and assistance through community food production, strengthening leadership capacity of women and empowering women while responding appropriately to the needs of the rural population. 84.3 % of MAGA employed agents say they are very successful or successful at making change in the communities that they work with; 100% of those doing extension for other organizations say they are very successful or successful at making change. There has also been a positive continued impact from the extension training component of the project as USAC has now institutionalized a post-graduate program. In addition, the certificate program will continue based on an entrepreneurial approach. USAC continues to work with MAGA to adapt the curriculum and training programs. USAC, using feedback from the participants, has also changed modules to move from the historical and theoretical to the more practical approaches and skills identified by the cohorts. Many non-governmental organizations want people with skills to enter into communities to implement development projects. As a result, USAC will now offer training for anyone and charge 1,500 quetzales to participate in the training. This will cover their costs and contribute to the ongoing need for professionals with extension education skills. In addition, USAC is planning for a cohort of 100 students in a bachelor-level program. They are hoping to have a rolling enrollment of 25 students at a time. Under an initial agreement with MAGA, enrollment will require an offer for full time employment to be guaranteed by the Ministry. Community-scale impacts According to the workshops with beneficiary farmers and associations, activities related to agricultural practices have created the greatest number of impacts for project beneficiaries. Impact was measured by asking workshop participants “What impacts do you experience, and as a result of which activities?” This was calculated by tabulating the number of linked responses between activities and impacts (chart 9). In chart 8 money was linked as an impact to various activities by 147 times. Also, money as an impact from the project was linked from 51 times for agricultural practices to 7 times for organizational capacity activities. The most common impact is “money” described as “increased household income” and also as money saved. Project beneficiaries explain that increased income is due to both increased quantity and quality of goods sold as well as money saved through the resources provided from the project, such as the provision of green houses and tools, as well as more cost-efficient practices. Farmers also reported that learning about commercialization and increased access to financial services helped lead to increases in income. 78 | CASE STUDY — Guatemala  | Findings 0 Agricultural Practices Commercialization Coordination and Connection Financial Services for Individuals Household and Nutrition Organizational Capacity 1020304050 Activities Number of Respondents Reporting Impact Chart 8 — Summary of Activities that led to more money for participants. Due to improved income generation opportunities, participants also identified a decrease in migration stemming from agricultural advancements from the project. “They are not going to the coast to look for employment. They pay for their children to study. They have economic resources for their homes. Their agriculture is sustainable. He has visited the farmers he worked with previously. He can see them making money from agriculture. He perceives a change in attitude.” “We now don’t need to immigrate because we have enough to eat and we have access to market. We don’t have to leave we can take care of ourselves. We don’t have a lot of land, but we can do a lot with our resources if we are open to learning and trying new things. And we are now replicating these teachings- we have extra food, we have sheep, cows, chickens, and we have learned how to take care of them and have extra food” “The Project brought us family wellbeing. Before, I immigrated to Mexico and sometimes to the USA and I’m sorry for doing it- I entered illegally- but now I’m more established in my house with my family- we have enough food and I don’t have to do what I was doing before. We are together with the family and we have what we need for our daily sustenance which is the most important thing. The project has given us wellbeing and happiness” CASE STUDY — Guatemala | Findings | 79 0 Access to Market Pride Organizational Capacity Agricultural Productivity Health Food Sufficiency Relationships More Money Time Nutrition Credit Business Planning 20 40 60 80 100 120 140160 Impacts Number of Respondents Reporting Impact Chart 9 — Summary of Impacts experienced from project objective 1. Other common impacts include increased individual and community pride (128 linked mentions), improved interpersonal relationships (128 linked mentions), as well as improved health (125 linked mentions) and food security (111 linked mentions). Farmers said that their health and food security (in terms of the quantity and diversity of food choices) continues to improve as a result of project activities related to household food security and nutrition. Some explained that women’s inclusion in project nutrition and income-generating activities has especially resulted in increased nutrition for their children. Project activities focused upon increasing coordination and connection with others helped contribute to improved relationships (within the community between farmers, with technical experts, and with value chain actors) and communication with fellow farmers which had previously been challenging due to the civil conflict within the region. Beneficiary farmers also said they gained an increased sense of individual and community pride from project activities especially those related to relationship building and organizational capacity building. “The work really depends on communication; we don’t just tell other people what to do but rather we learn from others. And with these people, since we know them then it is easier to more formally say, hey, I need information about such and such thing…if someone wants to start up with one of these practices I am personally willing to help and I have done it.” 80 | CASE STUDY — Guatemala  | Findings OBJECTIVE 2 Project objective two focuses on expansion of financial services available to agricultural producers (outcome) via three, related strategies: development of financial products tailored to the needs of agricultural producers; training of credit union staff and agricultural extension agents in new products and, more generally, in consumer finance products; and by facilitating new market linkages between buyers and sellers. The final evaluation of the project (ECI&ACH 2017) indicated that objective two was, in many respects, one of the most successful of the overall project. Four new loan products were developed and offered. Loan originations increased over the project cycle, as did overall portfolio value. For this reason, the financial services component was “Considered to be one of the most successful components of the project, given its achieved results and its value for money. The MICOOPE (Spanish acronym for National Federation of Cooperatives in Guatemala) project component has also been recognized by the USDA representatives at the US Embassy in Guatemala City as one of the biggest success stories of the project” With regard to financial inclusion, the final report states “The project ́s contribution towards ‘financial inclusion’ and ‘access to finance’ was also significant. In a direct contribution, over 4,000 new MICOOPE members gained access to demand-oriented agricultural loans. Indirectly, many more people benefited from the intervention, as the seven participating institutions added over 78,000 new members during the project phase, representing an exceptional double-digit growth (11%), while the peer group grew only between 5-8%.” Success within the project’s training component were likewise acknowledged, but qualified. “However, in terms of project tasks that were outside their core competencies (market access, market information), achieved results were less impressive than in the loan business. Compared to the scope and structure of the certification course for training extension agents implemented by USAC, MICOOPE has given a rather low priority to training activities (extension officers, producers organizations etc.), thus limiting the potential sustainability of positive agricultural growth effects to be reinforced through ‘financial literacy’ and cooperation with rural extension officers.” What activities or practices have continued since the project ended? Development of consumer finance (loan) products and training of staff and intermediaries The project sought to develop new consumer finance products suited to meet farmer needs with the expectation that these products would be utilized if supported by increased training on the supply (credit union) and demand (cooperative and farmer) side, and by market linkage activity throughout the system. MICOOPE created four new loan products to support financing of equipment, livestock, (land) preparation and refinancing. These new products contributed to CASE STUDY — Guatemala | Findings | 81 the increased relevance in 2014 and since of the agricultural loan portfolio (R. Jordan, personal communication). Trainings and market linkage likewise occurred, although end-of-project evaluation reports suggest with lesser fidelity to the original plan. All four new loan products remain on the MICOOPE books and lending in these product categories continues within the project area and, increasingly, beyond. Loan origination and portfolio values from over the project period (2014-2016) presented with data form 2018 (Table 19) demonstrate impressive growth over the project life cycle with some subsequent drop off in aggregate originations. However, the overall loan value and corresponding portfolio value increased steadily throughout the project and into the present reporting period. Establishing direct linkages between project beneficiaries and loans originating within the MICOOPE portfolio is not possible. For this reason it is not possible to definitively state that X beneficiaries or even that any particular individual engaged the MICOOPE system. However, it is possible to state that over the three years of the project, approximately 12,000 loans originated in new product categories within the target region. Data from 2018 provides evidence that this activity continues unabated. 2014 2015 2016 2018 Total loans # 1,687 4,062 6,853 6,018 Portfolio value $ (000) 2,963 7,503 15,427 19,444 Average loan value $ 1,756 1,847 2,251 3,276 Table 19 — MICOOPE agricultural loan portfolio originations and value, 2014-2018* *Note: MICOOPE data for 34 branch offices within the project area. Workshop image grouping results Activities corresponding to Objective 2 Number who learned about the activity in 2015 Number who continued to use the knowledge/ inputs 2019 Percent who continue to use what they learned during project implementation Using an annual calendar for production 47 52 111% Credit access 23 25 109% Value chain linkages 34 31 91% Table 20 — Number and percent change of beneficiaries depending on the following activities between 2015 and 2019. 82 | CASE STUDY — Guatemala  | Findings What are the impacts related to objective 2? Project objective two sought to expand financial services to agricultural producers through the development of new financial products (loans), training of loan officers and extension agents in these products, and by facilitating market linkages for producers. The initial assumption being that finance represented a barrier to commercially oriented farming activity, and that improvements across the lending market system would serve as a catalyst. Evidence gathered in the final evaluation suggests this is exactly what happened and with anticipated results. Data from 2018 suggests that the outcomes observed during the project continue. Nevertheless, we assert that these indicators may not tell the whole story about project impact and sustainability The institutional impact of the project is clear: four new loan products were created by MICOOPE, introduced successfully into target markets, and remain part of the MICOOPE product portfolio. The development of the new products and training (even if limited) in their use precipitated institutional and market system changes (impacts) that are likely to persist. Moreover, analysis of the current market context suggests that current lending activity may understate future potential as loan originations are a function of numerous factors, including, but not limited to, market conditions and alternative financing options. Three observations made by our team during this evaluation contribute to our assertion that preset loan originations may understate future potential. The first relates to markets, and the second and third relate to alternative financing options. Guatemala’s western highlands is an important coffee production region and many of the farmers in the region are coffee producers. The “bottoming out” of global coffee price in early 2019 concluded a precipitous slide that began in mid-2016, shortly after the midpoint of the CI project (Figure 4). Some reports indicate a breakeven price for green coffee in Latin America of around US$1.24, a price met or exceeded for much of the project time frame but not attained in subsequent years. With coffee price below the breakeven point for most of the post-project era, it makes little sense to use debt to finance production. Additionally, remittances to Guatemala climbed steadily throughout the project period and since, with 2018 values nearly double those in 2014, when the project began (Figure 5). We know from conversations with farmers that remitted funds provide an important source of operating and investment capital and that these funds are often perceived as preferable to loans due to their lower risk profile. Finally, a conversation with leadership in one of the project’s coffee producer associations revealed that capacity building efforts of this and other projects have enabled them to create and manage micro- loan portfolios financed, at least in part, by loans from formal financial institutions. If the single anecdote reveals a trend rather than an isolated case, better organized associations and cooperatives may well be leveraging a limited number of formal loans to serve a broader pool of member producers. CASE STUDY — Guatemala | Findings | 83 2015 2016 PROJECT POST-PROJECT 2017 2018 2019 $1 $0.80 $1.20 $1.40 $1.60 $1.80 $2 $2.20 $2.40 $2.50 $2.80 $3 Figure 4 — Coffee prices. 2015-2019 USD. PROJECT POST-PROJECT 2014 2015 2016 PROJECT 5105 5544 6285 7160 8192.2 9287.8 2017 2018 2019 10000 8000 6000 4000 2000 Figure 5 — Remittances to Guatemala 2014-2019. 84 | CASE STUDY — Guatemala  | Findings How Valuable to Communities are the Impacts from objective 2. Evidence of community use/value of new loan products is limited. On one hand, 23 of 97 workshop participants indicated that they learned about credit access through the project and 25 indicated that they continued to access credit in early 2019, a slight increase. Anecdotally, one loan customer interviewed highlighted that the terms and process associated with MICOOPE’s new agricultural loans led him to join the credit union and to take advantage of the new loan products to support his coffee production system. Important value that this farmer articulated, value difficult to capture through quantitative indicators, was the increased ease of process and improved loan terms. A sense that the new loan products were developed for his type of production system and that they loan officers understood it as well registered and increased his comfort level with the use of credit. Objective Activity Number who learned about the activity in 2015 Number who continued to use the knowledge/ inputs, 2019 Percent Change, 2015-2019 Individual finance Annual calendar 47 52 111% Individual finance Credit access 23 25 109% Individual finance Value chain 34 31 91% Table 21 — Summary of activities continued under objective 2. The Final Evaluation Report for the project detailed the overall success of the loan product differentiation strategy implemented by the CI project and MICOOPE. The number of loans increased over the project, as did the overall value of the loan portfolio. Although it is not possible to link new lending activity with project beneficiaries, it is reasonable to assume that new individuals were brought into the banking system due to the credit union’s increased ability to identify and price risk, features enhanced through product differentiation and staff training. CASE STUDY — Guatemala | Findings | 85 Impacts 0 Access to Markets Crop Diversification Pride Organizational Capacity Agricultural Productivity Health Food Sufficieny Realationships More Money Time Nutrition Credit 5101520 Chart 10a — Impacts from objective 2. OBJECTIVE 3 Counterpart International worked with agricultural extension agents and other local stakeholders to Increase the organizational capacity of local cooperatives representing small-holder producers through the identification of areas of critical weaknesses and executing beneficiary-owned organizational development action plans. These activities took place in the second and third year of project implementation. Counterpart performed an initial assessment of each cooperative using Counterpart International’s Organizational Development Toolkit (OD) and Organizational Certification Process (OCP) in order to facilitate an in-depth and participatory assessment of the areas of weakness and strength within six functional areas: (1) governance and strategic management; (2) financial sustainability; (3) external relations; (4) management and quality control; (5) human and material resources; and (6) accounting, grant, and financial management. Based on the assessment scores, cooperatives worked with consultants to develop an action plan for addressing the identified weaknesses using a combination of the following tools: 1) training modules; 2) direct technical assistance; 3) mentoring relationship; or 4) peer exchanges among the targeted organizations. To support participating cooperative Counterpart provided technical and financial support, training, and assistance in mobilizing other resources. Specifically, Counterpart provided trainings on strategic planning, financial management, human resource management, facilitation skills development, performance based management, and team building and conflict resolution. Counterpart provided financial support in the form of competitively selected small grants to cooperatives to carry out elements of their capacity building action plans. 86 | CASE STUDY — Guatemala  | Findings The organizational development activities sought to improve what interviewees described chaotic organizations, with little oversight or structure prior to their connection with the Counterpart project. “There was a lack of “togetherness” [we didn’t know how] to communicate with other people and groups.” Some cooperatives had more structure, formal documents and processes, but described challenges when it came to meeting international standards, or partnering with government and the private sector. Women in these organizations described feeling marginalized and not taken seriously as professional actors in their organizations and communities. What activities or practices have continued since the project ended? Overall beneficiaries have continued doing much of what they learned about during the project. Not only have project activities continued beyond the end of the project, but project beneficiaries are doing more now than when the project first ended. Beneficiaries explain that this is in part due to seeing the benefits other beneficiaries experienced in using those practices, as well as learning the practices from one another. For example one group of beekeepers describe streamlining a process for payments, which in turn help them be more transparent and eliminate corruption. Other organizations and members have seen this and are more open to implementing these types of reforms. Activities corresponding to Objective 3 Number who learned about the activity in 2015 Number who continued to use the knowledge/ inputs 2019 Percent who continue to use what they learned during project implementation Individual capacity development 28 40 143% Financial documentation 21 25 119% Organizational structure and documentation 38 45 118% Business planning 30 17 57% Table 22 — Number and percentage change of beneficiaries based on activities between 2015 and 2019. The small grants helped farmer associations access basic inputs. These grants provided needed capital and tools, such as farming equipment, processing machines, as well as office equipment such as printers and computers to support office management. Participants describe the benefits in two ways; 1. To provided needed infrastructure and office materials and 2. Allowed them to save the money they would have needed to spend on these items. These grants supported their needed credentialing, this was a very valuable benefit which allowed group to continue to operate and as a formal organization access additional resources. One participant said that “They did get computers and the transformer to eliminate the poor quality of electricity they were receiving. They obtained shelving to store the samples of honey for traceability purposes. They also obtained a file CASE STUDY — Guatemala | Findings | 87 cabinet.” Another interviewee responded that “they learned about contracting and hiring people. They improved their accounting. They received computers which was an activity. They are more advanced now and have moved away from paper for keeping their records.” Farmer associations have increased the use of financial and organizational management practices. Farmer associations have increased organizational and financial capacity, allowing them to better serve farmers in their region and members of their organizations. Farmer organizations mentioned that there has been an increase in the number of farmers joining, which in turn continues to improve the organizations capacity. One participant shared that now they have a “better organization, they can now join groups and have official titles and apply for other projects or to work formally.” This sentiment was reflected by others as an important impact of the organizational capacity development activities, that by using formal organizational structures and obtaining proper credentials they are able to access additional resources for their membership. Impacts Associated with organizational capacity development training The main impacts associated with organizational capacity development were improved relationships and not surprisingly, improved organizational capacity. These are impacts associated specifically with continuing with activities such as financial documentation, clarifying organizational structures and individual roles. Improved relationships was important to help sustain the organizational capacity. However, creating business plans within organizations saw a major decline in use; 43% of beneficiaries participating in data collection say they no longer use business plans. Impacts 0 Access to Markets Crop Diversification Pride Organizational Capacity Apgrcultural Productivity Health Food Sufficiency Relationships More Money Time Credit Business Planning 51 10 5 20 25 30 Nutrition Number of Respondents Reporting Impact Chart 10b — Impacts associated with organizational capacity development activities. 88 | CASE STUDY — Guatemala  | Conclusions Conclusions Sustainability for this case study is defined as the likelihood that the benefits of the project will endure over time after the completion of the project. We additionally looked at beneficiaries’ value of those continued benefits. Combined we get a sense of what has continued and why. The findings section above outlined what activities and practices have continued since the project closed, in this section we will explore how existing conditions (political, social and environmental), the project’s design and implementation have contributed to sustainability. This case study also looked at the variable of having a shock post-project and how this specific shock may have affected the likelihood of sustainability. Finally our research assessed beneficiaries value of the impacts experienced and how those impacts may or may not have contributed to equity in the project region. How did existing conditions affect project sustainability? Certain conditions that were present in the Western Highlands of Guatemala at the time of the projects design, and at the start of the project facilitated the continuation of activities and practices as well as helped to perpetuate the impacts experienced from the project. In contrast other conditions acted as barriers that made continuing activities and practices difficult or impossible, or contributed to an environment in which the impacts were less likely to be experiences, thus discouraging participants from continuing specific activities. Community assets Existing community assets specifically existing farmer knowledge was most often identified as having a major influence on the continuation of project activities. Farmer beneficiaries had a base knowledge of highland agriculture, individuals had been producers at various scales for much of their lives. The project worked with existing growers and CADERs which made learning and applying new and improved practices much more likely than if the project had been introducing farming and production concepts to non-producer beneficiaries. “We already knew how to plant corn etc. before the project. And this helped us to continue activities after the project” The lack of certain community assets such as quality roads and facilities were barriers to producers who otherwise would have been more successful at continuing to apply what they learned during the project. These beneficiaries who were typically more remote faced additional challenges like less access to water, inputs and networks that also restricted their ability to reap the benefits of the project. Demand for services It was clear from our interviews with project staff, the Ministry of Agriculture MAGA and Extension agents that there was a strong existing demand for an extension system in Guatemala. Efforts were in-place to create an extension system at multiple levels throughout the country; nongovernmental organizations had begun using the extension system and local universities sought to create an extension credentialing program. Our research shows that the demand that the start of the project was high thus the timing of the project filled a need for those in the Western Highlands. CASE STUDY — Guatemala | Conclusions | 89 Market conditions Key informant interviewees and workshop participants described a rural environment with poor market conditions such as low prices or demand for products, combined with some distrust for lending agencies and the formal banking sector. The conditions observed and described in Guatemala and, more specifically, the Western Highlands in the years before the project were less amenable to the use of commercial finance due largely to limited products and limited knowledge and readiness on the part of the credit unions (supply side) and the cooperatives and farmers (demand side). Much of the project’s success related to access to financial services seems to have derived from resolving some of those initial conditions, through the creation of products and additional training of supply and demand side actors on products and processes. Natural resources Beneficiaries who had little to no access to water or arable land, or experienced unpredictable drought were less likely to be able to sustain project activities. Those who faced issues with insect pests, and diseases at the outset were also less likely to continue to benefit from the project. Beneficiaries were more likely to mention lack of water as a challenge compared to other stakeholders (80% of mentions by beneficiary farmers 20% by extension agents). The inverse is also true, especially for those who has access to water. These beneficiaries were able to take full advantage of what they learned from Extension agents and promoters trained by the project. One grower said “My potatoes have a watering system. And now we have a market for our product.” “We also have to protect what we plant- for example before when we started we didn’t know what a watershed is but thanks to God this has helped us a lot for example we have natural wells and we have a nursery group and we plant trees. And if a stream does not have water we plant trees all around it. These are the things we do. And with the group we have to develop an agreement so that the group gets motivated and is patient and this year we are thinking about doing meetings with the group again. Now the climate is very hot and now that it is getting up to 30 [Celsius] we have to take care of it.” A history of paternalistic development projects created expectations by community members that the project design did not address. This was mentioned by extension agents, Counterpart International staff and various farmer associations. One barrier to project participation and the adoption of practices is the culture of paternalism within the development work in this region. Past governmental and nongovernmental development work often took an approach of provision of goods without investment of the local communities. This project’s approach of requiring stakeholder involvement and dedication was a drastic shift from past experiences, which not all community members bought into. “We still have the challenge of paternalism, that the government uses to provide services to rural populations. We work under a system of proposals, and the group proposes a project to us, many of the projects they request are paternalistic in nature, 90 | CASE STUDY — Guatemala  | Conclusions like they ask for us to give them pigs, or plastic, so this is individual not as a group. The national extension system is trying to change this so people forget this old way of working. There is still a lot to do.” Overall those farmer beneficiaries who had existing access to water, were already knowledgeable in basic agriculture and production systems were more likely to have continued to apply what they learned from the project, and to associate value and impact from doing so. For those without initial access to water, but who received or invested in improved water systems also experienced benefits and value from the project. One participant described not having access to water before the project and how this change has impacted them, they explained that “{With} vegetables- the most important is water – where we live there is no water and the water pumping system really helped us to plant new plants. Because before we couldn’t plant anything because we didn’t have water and now we can. So a variety of cultivars, more money, pride, and food security are all results of this project.” Those who started off in deep poverty at what is locally called “sub subsistence” level have been able to benefit from project activities, moving into more of a subsistence level, they are more able to meet their basic needs now. While those who started off at the subsistence level are slightly more market oriented then before, fewer overall sustained improvements have been seen. How did beneficiaries and their communities respond to subsequent shocks? Extension agents interviewed pointed out that the lack of extension employment opportunities for credentialed extension agents served as a shock to the continuation of project activities. One of the primary components of the project was the creation and support of extension agent training programs through USAC, with the intention that these agents would be hired under contract with MAGA. Due to a change in political office and extension education no longer being a priority, these contracts did not occur, and the trained extension agents were left without employment opportunities. Communities lacked the needed technical assistance that extension agents were intended to provide. “The biggest challenge was the impermanence of the extension agents. They are contracted by the government, then by the end of the year they are changed and made it, so the process count continues. Even though we were still training agents we had to keep the new agents updated. In many municipalities they haven’t contracted agents.” In response to this shock, ex-extension agents often obtained work with other organizations working on rural development in the region. USAC has also taken over the diploma program and has implemented requirements that participants have a guarantee of full-time employment in order to obtain funding. What components of the project’s design contribute to sustainability? Partnership with and leveraging of existing, established institutions stands out as the design feature most associated with success and sustainability. Four main institutions were engaged: The University of San Carlos, the MICOOPE nation network of credit unions, the national MAGA CASE STUDY — Guatemala | Conclusions | 91 extension network and, thirdly, various producer associations and cooperatives throughout the project region. In each case, project investments within the existing institutions were designed to create conditions favorable to desired outcomes: increased use of loans to support and expand commercial farming activities. Project activities and methodology were designed with an understanding of the community needs and contexts and built upon community infrastructure and strengths. The resources and activities provided through the project were designed to meet current gaps and be relevant to community needs and cultural contexts. Additionally, promoters and other staff working in communities were selected with an eye to having an established understanding or relationship with the target communities, creating greater opportunities for trust. The relevance of project activities incentivized community participation and increased the likelihood that project activities would be continued over time. However, the project design lacked articulated agreements that would ensure hiring of credentialed extension agents. The program design included informal agreements and expectations that the Ministry of Agriculture would hire extension agents as they became credentialed. Due to a change in political standing, this agreement was not honored and (at the time of our field work) many extension agents received training provided by the project were unable to apply these skills due to a lack of job opportunities, especially in the formal extension system. In turn, communities suffered from a lack of technical support due to a shortage of extension agents. One interviewee explained that, “The Ministry promised us that they were going to contract over 300 extension agents permanently. But not yet, the process goes slowly, and we are worried because now in the new course- it has been established so that for someone from MAGA to participate they need to have a permanent contract. We have been trying to change the rules of the games to be able to participate in the diploma program to be more efficient and sustainable.” In addition, the project lacked the establishment of market linkages and financial capacity building to help farmers overcome barriers related to the cost of inputs for commercialization. While agricultural productivity often increased due to the implementing project led activities, farmers did not always have access to markets or market information this prevented them from selling their products or obtaining the best price. In addition, farmers often did not have the financial capacity to overcome barriers that would allow them to market their products at a larger scale. Two growers explained that, “The primordial point is commercialization. We already know how to process potatoes and we know how to make a calendar and process the crops, but we have problems because the market does not exist- there is nowhere to go to export our products.” “The costs of production of coffee and potatoes are really high, with the fertilizer and all the work, but when we harvest it doesn’t work out, we end up poorer than we started, we end up poorer and there is no help from the government.” 92 | CASE STUDY — Guatemala  | Conclusions Engaging MICOOPE and MAGA agents in the planning process allowed for the specific loan products developed (the supply) to be well suited to both existing and emerging demand on the part of farmers and farmer associations and cooperatives. A collaborative design phase increased the chances of getting the financial products and associated processes right. The successes achieved by the project in the areas of institution and product development specifically at MICOOPE and increased lending (by MICOOPE) have been sustained in large part due to the selection by Counterpart International of an institutional partner already established in the region. Because commercial lending is a formal activity, it is best to work with existing and known institutions; uncertainty and unfamiliarity at any place in the system complicate the process. In contrast, the project sought participation by organized producer groups, many organized around commodities such as coffee, potatoes or tomatoes. This preference, reasonable as it was, imbued the project with a vulnerability to market conditions, which later presented challenges to the sustainability of outcomes and impacts. The coffee cooperatives that we visited shared anecdotally that project training enhanced their successes, but that coffee market challenges limit their use of consumer financial products in the direct and formal way originally envisioned. Other producers, such as the potato growers and the school vegetable producers experienced negative changes to their markets that effectively took them out of their markets. In all cases, the conditions (organized, commodity-driven) that favored inclusion in and success during the project contributed directly or indirectly to the sustained use of new credit products beyond the life of the project. Finally, the selection of MICOOPE was auspicious as the organization had recently concluded a USAID-funded project with the World Council of Credit Unions (WOCCU) - the Cooperative Development Program (2010-2017, with direct implementation in Guatemala ending in 2013) that developed and refined a set of agricultural finance tools to guide agricultural lending for credit unions [link]. MICOOPE itself had therefore recently developed its institutional capacity to develop and deploy new consumer finance products. This general capacity supported the specific objectives of the Counterpart project. To what extent were local stakeholders involved in project activity design Community stakeholders helped conduct SWOT analysis, identify community assets and needs, and informed what project activities and resources might include which increased the likelihood of sustainability. “They have us a series of tools and contacts and opened up the opportunity for us to provide ideas for projects. We made suggestions based on the needs of the communities. That is how they selected ideas for the project and passed these ideas on as six projects (I think six) with financing for these activities.” In what ways did the implementation of the project contribute to its sustainability? The training design relied on the assumption that extension agents would be equipped to train promoters who would then provide training to their local communities. Utilizing local leaders as promoters trained by extension agents aimed to build upon established community relationships, CASE STUDY — Guatemala | Conclusions | 93 and equipping farmers with a variety of technical resources. However, extension agents struggled with high turnover and lack of funding, making it difficult to provide cohesive training over a span of time. Promoters were not adequately trained or provided sufficient resources to successfully recreate trainings in their communities. “For example, [name redacted], he started with us, and then someone replaced him. And he didn’t know what we were working on and he left too. [Name redacted] was another one, and [name redacted] came too, he was working really well with us he started a network that we were a part of. And then he didn’t have a contract, so that was that. “There are so many people and few agents, the agents train the promoters, then these promoters go to the CADER, and learn from their own promoter. This is the difficult part, often the promoter isn’t able (resources, not trained etc.) to transfer this knowledge to the rest of the farmers. As MAGA we won’t have the ability to track this. Our monitoring system is really nonexistent.” How do local partnerships contribute to sustained project outcomes and impact? Establishing a strong partnership with USAC was integral to the sustainability of the extension agent training and certification. USAC served as a strong partner in training extension agents due to an already existing demand for the program coupled with the University’s capacity to implement the program. “Another impact of the diploma program is the interest in academia. Now academia is being involved with extension. Now it is not just on the government or NGOs to do extension. The formalization of curriculum helped to achieve the permanent posting for extension positions” What factor lead to training results being sustained? When promoters had sufficient support and training, the farmer to farmer style implementation was very effective. Communities identified promoters they respected to administer their trainings and training content directly reflected local contexts making content applicable and relevant. When promoters were well-trained, they were able to train communities in activities that spanned across the value chain, from production to processing and marketing. This training program while viewed as valuable by Extension Agent beneficiaries has been challenged by the unforeseen issues learned only after the startup of the project. While there appeared to be support to train a cadre of extension agents in the Ministry as well as with farmer associations, a failure to understand the human resource policies of the government was an issue. It was initially thought that everyone who was trained through the certificate program would be permanently hired, the HR policy required to have a full time position required a bachelor-level university degree. While trainees were hired, they were hired on a short-term, annual basis. The contract needed to be renewed annually. As administrations changed, the support for developing the network of extension agents waned. 94 | CASE STUDY — Guatemala  | Conclusions The sustainability though of the training program has resulted in other outcomes. The project can do little when administrations change or the priorities of an administration changes. While individuals who received training were expecting full-time employment with the Ministry, the reality was that position requirements of the civil service policies were not explored in developing the initial project. Currently, the nuance of the differences in job classes is understood now. How does the combination and sequencing of activities influence sustainability? The evaluation revealed that the combination of both capacity building and resource provision was an effective model that helped increase the likelihood of sustainability. Image grouping results indicate that farmers most often reported that resources (such as access to land and water which already existed and inputs provided by the project) and capacity (the training received) supported project activity sustainability. Specifically, agricultural practices, which had the greatest number of total outcomes associated with it, were most supported by the capacity and resources provided in the project. To what extent have the impacts of the project contributed to equity? The inclusion of women in the project trainings and provision of resources provided women with the opportunity to earn an income independent of their male partners. Prior to participating in the project, women were often dependent on the income of their husbands, and often held little decision making power over household financial resources. With the training and resources provided through the project, women were able to generate their own income which was a source of empowerment and financial independence. One participant stated “I feel proud that the women now are participating now. Now it’s not just the men. Before the project arrived, women were afraid of their husband and they did everything in the house and he did everything outside and the project trained us so that we were able to do things. And then it wasn’t just the men doing things. And we feel very proud about this. And our organization is 100% women and we have gone really well. Our Association helped to reduce machismo- that only the man did things and the woman stayed in the house. Now the man has to prepare his own food sometimes because we have to go to meetings and we have to be on time and he can’t say anything to us about it.” CASE STUDY — Sri Lanka | 95 CASE STUDY — Sri Lanka 96 | CASE STUDY — Sri Lanka  | Summary Summary World Vision International (WVI) in Sri Lanka implemented a Food for Progress project from 2006- 2010. Using the Integrated Farming and Sustainable Agriculture Practices (IFSA) methodology, this Project sought to achieve three objectives: capacity building for farmers, transfer of agricultural production technology and marketing, and business development. This Project included 3,043 farmers from eight provinces in the eastern regions of the country (Table 29). Activities included training in sustainable agriculture production and livestock, water storage, and commercialization. This sustainability assessment was conducted to evaluate the ongoing impacts, if any, experienced by former project participants. A mixed methods approach was used, including interviews, desk research, and workshops with project participants. Nine years after the project closure, most beneficiaries continue practicing lessons learned in the WVI project. The Project implemented initial activities, or practices, responding to three objectives (1) to introduce and improve agricultural crops, (2) to transfer agricultural technology, and (3) to promote water management and training in business and marketing. The main practices used to satisfy the objectives were: use of medium- and long-term tree production and short-term crops, training in water storage using water tanks and the provisioning of the tanks, and soil stewardship and conservation that have continued since the Project’s completion. Of the beneficiaries who learned about medium- and long-term tree production in 2015, 97% responded that they continue to use this practice on their farms or at least the knowledge that they acquired (Table 28). Of the farmers who received trees, 95% still have those trees on their farms (Table 30). Of the beneficiaries who mentioned learning about water tanks and storage, 89% still used this knowledge on their farm in 2019 (Table 28). Of the farmers who received a water storage tank, 96% still used the tank for its intended purpose in 2019 (Table 30). Of the beneficiaries who mentioned learning about short-term crop production, 87% continued using this knowledge on their farms and in their activities in 2019 (Table 28). Of the people who received training in animal husbandry production, 50% still have their animals or continued to use the acquired knowledge (Table 28). Activities also included workshops to develop training in marketing; 83% of the beneficiaries who received group training on marketing still used this knowledge in 2019. The farmers who learned the practices achieved year-long production and depended less on rainfall, a major production barrier in the eastern regions of Sri Lanka. In addition, follow up training in marketing strengthened this knowledge and stabilized individual farm economy. The overall long term impacts of the project on the lives and livelihoods of the farmer beneficiaries are more money from greater production of short- and long-term crops throughout the year. Beneficiaries also recognized that the introduction of water storage increased resilience in the dry seasons, and helped to improve year round production. Greater on farm diversity increased food security, time savings, and added nutritional value to household diets. Three vegetable processing and post-harvest centers were built or improved upon, these facilities continue to function and benefit both beneficiaries and non-beneficiaries. The sustainability of the project has been supported in large part by the new and improved ability of the farmers to store water and apply improved agricultural practices. Project limitations identified were continued challenges to adopt new cultural practices like raising animals for human consumption, as well as elephant-human conflict which destroyed crops. CASE STUDY — Sri Lanka | Evaluation Purpose | 97 Evaluation Purpose The Office of Capacity Building and Development (OCBD) is a program area of USDA’s Foreign Agricultural Service that is dedicated to advancing global trade, agricultural development and food security. Programs administered by OCBD help to reduce food insecurity, strengthen agricultural economic growth in targeted developing countries along the full value chain of development, build their capacity to integrate into the global economy, and become viable agricultural trading partners with the United States. The objective is to help move these countries along the agricultural market-development continuum from developing economies, where capacity building is the primary focus, to developed economies, where market development and expansion are the primary focus. Authorized by the Food for Progress Act of 1985, the Food for Progress (FFPr) program provides for the donation of U.S. agricultural commodities to developing countries committed to introducing and expanding free enterprise in the agricultural sector. Donated commodities are monetized (sold on the local market) and the proceeds are used to support agricultural development activities. Food for Progress has two principal objectives: improve agricultural productivity and expand trade of agricultural products. Food for Progress programs have worked to train farmers in animal and plant health, improve farming methods, develop infrastructure (roads, electricity, storage, etc.), establish and strengthen producer cooperatives, provide microcredit, and develop agricultural value chains. FFPr Program Participants have included private voluntary organizations (PVO), foreign governments, universities, and intergovernmental organizations. FFPr projects are currently active in 25 countries across Africa, Asia, and Latin America and the Caribbean. The University of Minnesota was contracted by the USDA FAS FFPr office to assess the ongoing impacts of 3 closed Food for Progress Projects: 1 each in Africa, Asia, and Latin America and Caribbean. The final countries and projects were selected in join consultation with USDA. The specific questions were designed in close collaboration with USDA, including a workshop conducted with USDA in November 2017 and over a period of months to refine questions and methods. This evaluation was not expected to include experimentation or trials on human subjects, as the programs to be evaluated have all concluded. Each closed project is presented as a case study, and the three projects together are a part of the full “Post project sustainability assessment” report. The Organization for Economic Cooperation and Development (OECD), Development Assistance Committee (DAC) has defined “sustainability”, in the context of evaluation and results-based management as: The continuation of benefits from a development intervention after major development assistance has been completed. The probability of continued long-term benefits. The resilience to risk of the net benefit flows over time. The Food Assistance Division, in its Monitoring and Evaluation Policy defines “sustainability” as: Assessment of the likelihood that the benefits of the project will endure over time after the completion of the project. Sustainability should also assess the extent to which the project has planned for the continuation of project activities, developed local ownership for the project, and developed sustainable partnerships. The case study that follows is one of three studies that are a part of a full report provided to USDA FAS on Dec 31 2019. 98 | CASE STUDY — Sri Lanka  | Project Background Evaluation Questions 1. How did existing conditions affect project sustainability? A. What are the preliminary conditions that set up market linkages for success? B. To what extent are results more sustainable in low, or middle-income countries? 2. What components of the project’s design contribute to sustainability? A. To what extent were local stakeholders involved in project activity design? 3. In what ways did the implementation of the project contribute to its sustainability? A. How do local partnerships contribute to sustained project outcomes and impact? B. What factors lead to training results being sustained? C. How does the combination and sequencing of activities influence sustainability? 4. What activities or practices have continued (and to what extent) since the project ended? A. How valuable to communities are the project’s outcomes, including activities and practices? 5. What are the impacts of the project? A. How valuable to communities are the impacts of the project? B. How did beneficiaries and their communities respond to subsequent shocks? C. To what extent have the impacts of the project contributed to equity? Project Background Sri Lanka is an island nation, divided into 25 districts, located along major maritime trade routes in the Indian Ocean. Following democratic elections of 2015, the end of a 26-year conflict in 2009, and a decade of recovery after a devastating tsunami in 2004, Sri Lanka now focuses on strengthening its democracy, promoting reconciliation, and fostering sustainable economic growth. Sri Lanka is a tropical island with a mean temperature of 27.5 degree Celsius and is predominantly dry, with annual rainfall of less than 1750 mm spanning 4.2 million ha in the north, east and south of the country. An intermediate dry zone covers another 0.8 million ha with rainfall from 1750 mm to 2500 mm per year. A wet zone of 1.5 million ha in the southwest receives annual rainfall of more than 2500 mm. Rainfall is the primary source of water for agriculture in Sri Lanka, which is delivered, in part, by monsoon rains from May to August (Yala season) and October to late January (Maha season). Rainfall also occurs during inter-monsoon periods (Weeraratna & Weerasighe, 2009). The population of Sri Lanka is multi-ethnic and primarily rural. A majority of Sri Lankans are Sinhalese but other groups include Sri Lanka Tamils, Indian Tamils, and moors (Fladby, 1983). Seventy-seven percent of the population lives in the rural sector where farming is practiced extensively (Menike et al, 2015). During 2007 and 2010, the agricultural sector in Sri Lanka represented around 12% and 8.5% of the national gross domestic product (World Bank, 2009). Ancient civilizations occupied the southeastern, north-central and eastern parts of Sri Lanka (Sinhalese Kingdom). The economic basis of this society was paddy cultivation based on large irrigations works (Fladby, 1983). Plantation agriculture was later introduced by the Portuguese in CASE STUDY — Sri Lanka | Project Background  | 99 the early 16th century and then by the Dutch and British. Tea and rubber were introduced to the country in the 1870s by the British. Sri Lanka was a main exporter of commodities during colonial times. In 1948, Sri Lanka gained independence from England and has since made reforms to increase agricultural production and national food security (Weeraratna & Weerasighe, 2009). Sri Lanka shifted its agricultural production methods from an indigenous form of agriculture to adopting the technological package of the Green Revolution in the 1960s (Zubair, 2005). The agricultural sector in Sri Lanka is the main source of livelihood for rural communities and employs 31% of the population. Agriculture in Sri Lanka consists mainly of plantation crops and subsidiary food crops. In 2007, the main plantation crops occupied around 871,000 ha and included: tea, rubber, coconut, minor export crops like cinnamon, coffee, cocoa, pepper, cardamom, cloves, and nutmeg; cashews, and sugarcane. Subsidiary food crops occupied 878,000 ha and included crops such as: paddy (rice) and other cereals (e.g. maize, finger millet), legumes, onions, chili, oil crops, yams and vegetables (e.g. beans, carrots, cabbages, beets, leeks etc.) (Central Bank Annual Report, 2007). Rice obtained from paddy is the staple food of Sri Lankans and it accounts for around 65% of the total cereals consumed in Sri Lanka (Weeraratna & Weerasighe, 2009). Dairy farming is also a traditional non-commercial practice that has ensured the nutritional requirements of the population. Households in Sri Lanka from ancient time had the habit of rearing at least one cow in each house. However, dairy farming is a small segment of the national economy of Sri Lanka and it is predominantly in the hands of small-scale farmers (Jayaweera et al, 2007). There were a total cattle and buffalo population of approximately 1.5 million animals in 2005. (Sri Lanka’s Department of Animal Production & Health, 2005). The total milk production of the country was 192.7 million liters that just supplied 15% of the total increasing domestic demand for 2005 (Central Bank, 2005). The USDA’s Foreign Agriculture Service, through the program Food for Progress, supported developing countries and emerging markets to improve and strengthen their agricultural sectors. Sri Lanka among other countries received, implemented, and completed a Food for Progress project. World Vision International (WVI) in Sri Lanka was the partner organization that began implementing the project in 2006 using the Integrated Farming and Sustainable Agriculture Practices (IFSA) methodology and concluded in 2010. WVI began its work in Sri Lanka in 1977 implementing relief and development programs mainly targeting children and rural communities. UN and humanitarian communities partnered with WV to provide support to those affected by the internal civil war that continued until 2009. One of WV core programs is the establishment of Area Development of Programmes (ADP) in 29 locations across 18 districts. The ADP’s allowed WV to implement programs in education, economic development, health and nutrition and water and sanitation throughout the country. Agriculture has not been the main focus of WV programs in the past, however its presence and understanding of rural regions was important to develop new projects and agricultural practices that could potentially increase farm production and income. Especially in those areas with food insecurity and susceptible to several droughts in the north eastern regions. 100 | CASE STUDY — Sri Lanka  | Project Background After establishing partnerships with several sectors, WV through the IFSA technology aimed to include 3,080 beneficiaries, including women and men farmer leaders from Horowpathana, Kabithigillewa, Galenbindunuwewa (Anuradhapura district) in the north central region; Giribawa (Kurunegala District) in the north western region; Thanamalvila (Moneragala District) in the southeast region; and Weeraketiya (Hambantota District) in the southern region of the country. The IFSA technology outlined three leading objectives of the project: capacity building for farmers, transfer of agricultural production technology and marketing, and business development. Figure 6 — Sri Lanka regions where the project was implemented. (Red dots) regions represented in this study Capacity building for farmers included 280 farmer leaders and 2,800 farmers in two groups organized by short and long-term crop seasons. Capacity building focused on the cultivation of short, medium, and long-term crops using innovative organic farming techniques to improve soil fertility and pest management and incorporating improved animal husbandry and fisheries practices. Activities with farmers were conducted quarterly, monthly, and bi-weekly in which WVI’s technicians and engineers accompanied the process. In addition, part of the training included management of timber and fruit tree production, water harvesting, micro-irrigation, and aquaculture as well as processing of agricultural products and general crop marketing. Farmer leaders had to conduct monthly group meetings with other farmers to demonstrate and discuss the IFSA agricultural technology. CASE STUDY — Sri Lanka | Project Background  | 101 2007/2008 Maha (rainy season) 2008 Yala (dry season) Long-term plant types Medium-term plant types Short-term plant types Halmilla plants Ambaralla (Dwarf) plants Big Onion seeds Mahogany plants Banana plants Bitter Guard seeds Teak plants Cashew (budded) plants Brinjal (Egg Plant) seeds Coconut (potted) plants Bushita seeds Drumstick plants Capsicum seeds Guava (budded) plants Chillie seeds Guava seedlings Mea (Long Bean) Lime seedlings Okra (Ladies Fingers) seeds Mango (budded) plants Radish seeds Orange (budded) plants Red onion bulbs Papaya plants Tomato seeds Pomegranate plants Tibbutu plants Table 23 — Seeds annual crops and perennial crops including trees (medium and long-term) distributed by the WV project. Summary of Final Evaluation Under the objective of stimulating marketing and business development, WVI organized 160 farmer marketing groups and provided marketing training and market information. WVI established eight Product Marketing Centers for the marketing, shipping, and receiving of commodities. In addition, 10 value-added processing units for agriculture commodities were established in rural areas and 490 females were trained to manage cottage-based industries and urban sales marketing outlets. Evaluation Methods and Limitation Sample Selection A wide range of stakeholders participated in the evaluation. Implementing partners identified participants that fit various types of stakeholders identified by the evaluation team. The implementing partner provided recommendations of who to select for an interview; interviewees were selected based on their knowledge and expertise of sustainability in the region. The implementing partner further recommended Galenbindunuwewa, Thanamalvila, and Giribawa as regions for Image Grouping sessions. These sites were chosen based on communities that would likely still be practicing activities, were feasible to travel to, and accessible for workshop participants. The data collection team also administered interviews with the implementing partner in Colombo. 102 | CASE STUDY — Sri Lanka  | Project Background The evaluation team conducted outreach with the intention of conducting 25 interviews spanning across all stakeholder types per site. Specific data collection participation counts and stakeholder information can be found in the Interview Sample table 26. Image Grouping© participants were selected using a stratified convenience sample. This allowed the evaluation team to select groups based on additional hypotheses relevant to sustainability. All Image Grouping© participants are farmers. Criteria for the selection included attention to: diversity of locations, gender, religion, geographic conditions, the IP’s continued presence and programming, and believed sustained practices in present day. Two of these workshops were conducted in Galanbindunuwewa -- one with Muslim women and one with males. One workshop was held each in Thanamalvila and Giribawa; the participants of both of those workshops were a mixed gender group of Buddhist farmer beneficiaries. Specific gender breakdowns can be found in the Image Grouping© Sample, By Sex and Region table 25. Data collection was primarily conducted in Sinhala, although a few were done in English. The activities that participants had to choose between came directly from a review of project documents and consultation with the implementing partner Interviewees were selected based on key informant expertise. Individuals who had direct knowledge of the site’s project were asked to participate. Region Image Grouping© Workshop Interview Galenbindunawewa 2 7 Giribawa 1 5 Thanamalvilla 1 8 Colombo 6 Grand Total 4 n=59 26 Table 24 — Data collection counts. Location F M Total Galenbindunuwewa 10 12 22 Giribawa 13 7 20 Thanamalvila 6 11 17 Grand Total 29 30 59 Table 25 — Image Grouping© sample, by sex and location. CASE STUDY — Sri Lanka | Project Background  | 103 Stakeholder type Galenbindunawewa Giribawa Thanamalvilla Colombo Total Implementing Partner Staff 1 1 3 6 11 Lead Farmer 1 1 1 3 Partner institution 2 1 1 4 Government 2 1 2 5 Mobilizer 1 1 1 3 Grand Total 7 5 8 6 26 Table 26 — Interview sample. Data Collection The evaluation team conducted interviews with local key stakeholders of each award to understand the local context of each site and gather insights to inform culturally appropriate evaluation design. Findings from this report come from primary data collected through: • Image Grouping© • Interviews • Observations Under the U.S. team’s partnership, a local evaluation firm and the implementing partner in each country were used to identify and recruit prospective participants. Originally, the U.S. evaluation team—of UMN faculty, graduate students, and Improve Group consultants—was to conduct data collection activities in partnership with the local firm. However, with the terrorist attack in Colombo in April 2019, the U.S. team had to change plans. The local evaluation team conducted in-person interviews, field observations and facilitated Image Grouping© sessions. Data Collection Method Number of Participants Image Grouping 59 Interviews 26 Field observations 19 Grand Total 104 Table 27 — Number of participants per data collection method Analysis Methods The evaluation team conducted qualitative analysis in Dedoose software to analyze the in-depth interviews and Image Grouping© discussion notes. Initially, the evaluation team developed codes using an inductive approach. These codes were developed by identifying themes from literature on conditions needed for sustainability, as well as the evaluation questions. Following that, a 104 | CASE STUDY — Sri Lanka  | Project Background deductive approach was used to find new themes based on the responses. The evaluation team lastly looked at relationships between themes and compared the differences between different types of respondents, such as farmers and other key stakeholders. Quantitative Image Grouping was analyzed using Excel and SPSS. Frequencies and crosstabs were conducted to understand differences between locations, or the activities that resulted in sustained outcomes. Themes across these responses were compared to those found in the qualitative results and integrated to achieve a mixed method understanding of the data. The literature review was conducted by reviewing previous reports written by World Vision International in Sri Lanka, USDA reports, and academic articles and books about agriculture in Sri Lanka between the period and the regions of the project implementation. Limitations With all evaluations, there are limitations to the data that are gathered and the conclusions that can be drawn. Recall and response bias may be present in the data. The evaluation team also recognizes that this study did not use an experimental design which means there are limits to conclusions about impact. A terrorist attack delayed some activities and influenced participation rates. It was not believed to have influenced responses to a substantial degree. Recall bias may have occurred due to the time that has passed following the closure of projects. This can lead to participants remembering events at a different scale or forgetting what they did at that time. As a result, some farmers may have reported participation in training and services mostly in areas that they still remember and use today, positively biasing findings towards a higher percentage of individuals sustaining practices. Image Grouping© methodology helps to reduce this likelihood by having farmers from similar areas participate in the same session and encouraging them to talk among themselves to remind each other whether they participated in certain activities over others. Likewise, the use of individual farmer manuals during project implementation may have helped farmers be more precise in their memories of what they learned about as these guides were still referenced during the workshops. Positive response bias may have influenced farmers’ responses in two ways. First, in asking how participant lives are today as compared to prior to the project start, farmers may have been more likely to exaggerate the benefits of the project. In many data collection environments, participants aim to “please” data enumerators, responding in a way that they believe the enumerator is hoping for or contributes to their benefit. Image Grouping has helped to minimize this bias by asking questions in different ways and providing a unique method of data collection. Nevertheless, the results from this country in comparison to the other sites suggest that this bias was less likely to exist. Second, implementing partners were used to determine which sites to visit and who to recruit. Partners may have desired to show more successful outcomes for their own audiences as well as potential influence on future funding through USDA. The evaluation team worked to compensate for this potential bias by requesting IPs to suggest sites that had varying degrees of success after being involved with the USDA project. CASE STUDY — Sri Lanka | Findings | 105 This project does not include control or comparison communities and thus cannot make claims about magnitude of impact or causality. As a result, although we were able to describe if income changed for participants and whether they attribute this to their respective projects, we do not know the extent to which income changes were solely the result of the USDA project. Likewise, secondary data did not exist at a regional level to demonstrate need or wealth status prior to project start; therefore, we are unable to make claims about economic changes to regions where USDA projects were implemented. Due to a terrorist attack in Sri Lanka’s capital city two weeks prior to the scheduled start of data collection, data collection was delayed by one month. Despite this dramatic shock to local Sri Lankans, the local evaluation firm believes that responses were not substantially different as a result. Findings In the 9 years since the project closed many beneficiaries continue to use the lessons and practices gained by the World Vision project. Former participants are continuing to use practices and technologies from 12 of the 17 activity areas that our assessment measured. These findings are organized using the FFPr projects 3 objectives. Under each objective we will share the findings that correspond to our high-level evaluation questions, found in full on page 96. • Objective 1 — To introduce and provide improved agricultural seed varieties, crops, improved livestock and fish breeds • Objective 2 — agriculture Production technology to improve small farmers’ agricultural productivity • Objective 3 — To provide better market access by establishing farmer marketing groups and training in improved processing storage and marketing approaches. OBJECTIVE 1 — To introduce and provide improved agricultural seed varieties, crops, improved livestock and fish breeds The capacity building component focused on developing workshops and training for a diverse group of farmers that included women and men of diverse religious backgrounds and the farmer leaders selected by the project. The main training topics included the implementation of new techniques such as the cultivation of short, medium and long-term crops, soil care and conservation including compost, water tanks, storage and irrigation, animal husbandry, (including bee keeping and fisheries). In addition, capacity building included harvesting, processing, post￾harvest practices, and marketing plans to better connect farmers with local value chains. 106 | CASE STUDY — Sri Lanka  | Findings Activities corresponding with Objective 1 Number who learned about the activity in 2015 Number who continued to use the knowledge/ inputs 2019 Percent who continue to use what they learned during project implementation Short-term crop production 60 52 87% Animal husbandry production 42 21 50% Beekeeping production 13 2 15% Soil care and conservation, production 60 46 77% Long-term and medium-term tree production 60 58 97% Training in water tanks and storage 57 51 89% Table 28 — What activities or practices have continued since the project ended? Long-term and medium-term crop production Overall 88% of beneficiaries who participated in our data collection workshop are continuing to use and implement what they learned related to long- and medium-term crop production. Related training included: long term and medium-term tree production; irrigation using water tanks and water storage ponds; and soil conservation. It is important to note that the project integrated training in long- and medium-term crop production and each individual training reinforced the other s. Introduction of irrigation using water tanks and storage ponds stabilized the farming systems creating opportunities for the introduction and success of the long- and medium-term tree production. All productive activities were enhanced by the soil conservation practices introduced. Participants who learned about long-term and medium-term tree production through the USDA project described how they applied what they learned and the benefits they have received, and continue to receive, from continuing to practice what they have learned. Of those that received the training who attended workshops, 97% indicated they continue the practice. They are continuing to grow long-term and medium-term tree crops and cite their ability to generate long-term reliable income through those efforts. Combining short term crop production with the medium and long￾term tree production has provided participants with multiple reliable sources of income occurring throughout the year, improving both the diversity and resilience of their farming systems. “People in these areas used to cultivate short term crops such as grains and vegetables, and luckily for them, during these projects they had the opportunity to cultivate long term products such as coconut, mangoes. Now, they get an income from that.” CASE STUDY — Sri Lanka | Findings | 107 Of the participants in our workshops who received training on irrigation using water tanks and water storage ponds, 89% continue to use those practices. The use of the water tanks and ponds, combined with the introduction of medium and long-term tree crops, has permitted farmers to continue agricultural practices throughout the calendar year regardless of rainfall. Before the introduction of the water storage practices, agricultural production was limited to the rainy season and subject to weather patterns. Introduction of water storage options were a direct response to water issues identified during project planning. “For run-off water collection, we had small tanks that we made from the project, and with that, crop cultivation was solved.” Interview This practice, supported by training, was key to the success of the project providing opportunities to extend the cropping season year-round, stabilize individual farm economies, and provide opportunities for new tree crops. Of the participants in our workshops who received training on soil conservation, 77% continue to use those practices. The training provided focused on water harvesting and management, soil care and fertilization, as well as crop production and animal husbandry. The design of the training responded directly to local context and the challenges many were experiencing in agriculture, such as water management and soil conservation. These training sessions were likely complementary to the water tanks, water storage and irrigation training. In the observational field visits four of 7 farmers mentioned receiving water conservation training while six of seven mentioned receiving soil conservation training. Agricultural capacity building Overall 80% of beneficiaries who participated in our data collection workshop are continuing to use what they learned in these general agricultural capacity building training: soil fertilization and irrigation; post-harvest; and processing training. Of those receiving the soil fertilization and irrigation training, 97% stated they continue to use the training and the practices they learned to maintain soil quality and crop productivity. They were trained on soil fertilization practices and established irrigation systems for their farms. Participants stated that prior to the project the soil in the region was depleted and they did not have access to reliable water. The training addressed a constraint they faced in improving crop productivity. “Training in soil care and conservation has led to a change and improvement of the quality of our plants and harvest, which saves time and contributes immensely to our financial security. The existence of lakes, rainwater harvesting, and water management knowhow means year-long farming without the fear of losing our crops to drought conditions, which is a huge psychological relief and yet another factor that gives us financial security.” From 38 responses in the workshops that mentioned post-harvest training, 32 people (84%) continue implementing post-harvest practices. The continued use of processing practices was less frequent with 12 (48%) of the 25 workshop participants who had received processing training indicating that they continue to use those practices. 108 | CASE STUDY — Sri Lanka  | Findings Overall, the trainings more directly related to improving and expanding crop production through improved soil conservation, fertilization, irrigation and post-harvest activities were well received and they continue to practice what they learned. As reported by workshop attendees, the training in processing reached fewer farmers with less of a continuing impact. Short term annual crop production Of the 60 workshop attendees who received training in short term annual crop production, 52 (87%) indicated that they continue to use the practices they were trained in. Participants were introduced to new types of short-term crop production and continue those practices. The integration of soil and water enhancement with new cropping options have led to some lasting impacts. As was previously discussed, in addition to improving short term annual crop production, participants have added long-term and medium-term crops into their rotation, increasing the variety of products and their financial stability. “We were introduced to new short-term crops (mushroom, watermelon, mung beans, bitter gourd, cassava, long beans, sweet potato, banana); with the addition of medium-term crops (local orange, June plums, moringa, papaya, pomegranate, guava, lime) and long-term crops (coconut, teak, mahogany, mango).” It is important to note that significant changes to farmers cropping systems were introduced accompanied by training to accompany those changes which has been taken up and lead to improvements in local livelihood security. Identification of key constraints to productivity with appropriate interventions backed by training have helped farmers improve their cropping systems and livelihoods. Beekeeping and animal husbandry 15%-50% One area in which the introduction of new practices accompanied by training had limited success and long-term sustainability was beekeeping and animal husbandry with between 15% and 50% of workshop participants who reported receiving training continuing the practices. There were some specific cultural and technical issues related to this lower level of acceptance and sustainability. Beekeeping was discussed in terms of cultural incompatibility of project activities for farmers who practiced the Buddhist religion. Buddhists objected to keeping animals in cages (human-made beehives). Workshop participants describe the lack of sustained animal husbandry practices for the following reasons: lack of resources to maintain healthy livestock (primarily water); lack of labor resources, either due to decreased capacity as participants age or due to less labor availability (often described as children growing up and leaving the home); and misalignment with Buddhist beliefs about animal captivity. The majority of those who adopted animal husbandry resided in Galenbindunawewa (43% of participants doing animal husbandry) and Thanamalvila (33%). Rates of adoption were similar across the three sites; Giribawa had an adoption rate of 50%, Galenbindunawewa was 53%, and Thanamalvila was 47%. CASE STUDY — Sri Lanka | Findings | 109 For context, the majority of those who indicated having received livestock and fish at the close of the project were primarily residing in Galenbindunawewa (64% of respondents indicating receiving livestock or fish) and Thanamalvila (33%). Only one participant in Giribawa indicated receiving livestock or fish. Adoption rates were similar across the three sites, with the exception of Giribawa which had an adoption rate of 100% however only 1 personal recalled learning about animal husbandry and they are still canoeing. (Galenbindunawewa was 43%, Thanamalvila was 45%). “Livestock farming is something only a few of us are continuing. This is due to difficulty in handling the animals owing to our aging or lack of labor power. Livestock farming is labor intensive and time consuming. When the project started, we were much younger men and our children still lived in our household with us; they helped out with the animals and the farm. But today, most of my kids no longer live at home and one of my sons joined the military. My wife and I find it difficult to manage with the routine. We still have one cow, but that’s so that we have milk for personal use.” What are the impacts of objective 1? Workshop participants were asked to describe the impacts of project activities on different aspects of their lives and livelihoods. This section presents those results that are most related to training although it is important to recognize that the impacts are a result of the training integrated with other project activities. 0 More Money Better Health Improved Nutrition/Food Increase in Production Access to Markets Time Gains Crop Diversification 20 40 60 80 100 120 140 Number of Respondents Reporting Impact Impacts Chart 11 — Impacts associated with objective 1. Training in long- and medium-term crop production resulted in a number of impacts on project participants. Participants described the increased income as a result of increased crop production from crop diversification. This enabled participants to have a variety of income sources throughout the calendar year. In addition to crop production, participants described qualitatively increased crop production due to crop diversification. Crop diversification has enabled participants to have a variety of income sources throughout the calendar year, securing an increased likelihood of 110 | CASE STUDY — Sri Lanka  | Findings gaining continual income from agricultural practices. throughout the year. Increased production and crop diversification lead to increased food sufficiency and nutrition (adding greater nutritional value to their household diets) as well as increased income to purchase a greater variety of nutritious foods. “The training and education provided to us completely changed how we cultivate. It increased our production capacity as well as the produce and gave us better financial stability. Despite some of us disagreeing with crop rotation and crop variation, today, we are using completely different farming practices than we were doing before this project. Hybrid seeds have shown a 500% increase in production, which is a life changing increase for any farmer. Crops with different time-frames for growth and harvesting have given us the ability to farm and harvest all year long; the water resource provided that we can actually continue these practices and make a success of it.” Project participants associated agricultural capacity building through training with a number of impacts. Training in production activities were most frequently associated with results in improved nutrition while the specific activity most frequently associated with this outcome was short￾term crop production (29 mentions). The targeted regions have high literacy rates. The training was coupled with training packets which detailed each of the project activities and resources. Project beneficiaries continued to reference training packets long after the project’s end, passing packets on to non-beneficiaries in the community to increase their training as well thus providing sustainable impacts but also expansion of the benefits beyond the initial project participants. An important aspect of the success of the project were the observable and tangible benefits of the project. Participants described being motivated to continue the project activities and practices as a result of observing and experiencing the impacts of the activities. Practices commonly discussed included use of water harvesting techniques, practicing crop rotation and growing a variety of crops, and using agriculture as a means for year-round sustained income. “Training in soil care and conservation has led to a change and improvement of the quality of our plants and harvest, which saves time and contributes immensely to our financial security. The existence of lakes, rain water harvesting, and water management knowhow means year-long farming without the fear of losing our crops to drought conditions, which is a huge psychological relief and yet another factor that gives us financial security.” Short term annual crop production, beekeeping and animal husbandry were also credited with impacts on livelihoods by workshop participants. The activity most often associated with increased health (25 mentions) and increased production (49 mentions) was short-term crop production. In qualitative discussions, farmers most frequently associated work in short, long, and medium￾term crop production training allowing them to have greater production today. People saw less connection between increased production and receiving water, or business training. Although beekeeping and animal husbandry had lower success rates than other activities, in areas where they were adopted, they did have positive impacts. CASE STUDY — Sri Lanka | Findings | 111 “The introduction of livestock gave the farmers of our community an opportunity to further improve on integrated farming the project introduced. Many farmers went into goat, cattle and poultry farming. Some were even introduced to farming fish (Tilapia) in their lakes. This also added a rich source of nutrition to the lives of our families. Protein in milk, and eggs meant a change in how our children were nourished and were valuable additions to our daily lives.” OBJECTIVE 2: Agriculture production technology The provision of goods, especially (1) water storage devices, (2) long term and medium term fruit trees such as orange, June plums, moringa, papaya, pomegranate, guava, lime, coconut, teak, mahogany, and mango, and (3) short term crops such as watermelon, mung beans, bitter gourd, cassava, long beans, sweet potato, and banana were important assets contributing to project sustainability. Activities corresponding with Objective 2. Number who learned about the activity in 2015 Number who continued to use the knowledge/ inputs 2019 Percent who continue to use what they learned during project implementation Providing Goods and Products % Provision of water tanks and storage 55 53 96% Provision of trees 59 56 95% Provision of Livestock and fish 33 16 48% Provision of bees 3 3 100% Total 150 129 86% Table 30 — Activities corresponding with objective 2 and percentage change between beneficiaries that learned and continue using practices introduced by the project. What activities or practices have continued since the project ended? Of 55 workshop respondents who received water tanks or other water storage devices, 53 (96%) indicated they still use this resource. Nine unprompted workshop participants specifically mentioned water and water usage as factors influencing continued project impacts, especially noting that water storage has made farm productivity possible even in the dry season. Similarly, of 59 workshop respondents who received long term and medium-term trees, 56 (95%) report still growing them today. Participants commented on the ability of tree crops to provide reliable income and to integrate well with the production of short-term vegetable crops. Short term crop production was also sustained at high levels, with 60 workshop respondents indicating they received access to short term crops through this project, of which 52 (87%) still grow them. Thirty-eight workshop participants indicated short term crops, often integrated with long term or 112 | CASE STUDY — Sri Lanka  | Findings medium-term tree production, improved crop diversity on their farms and overall crop production. Crop diversity was often linked to diversified farm outputs and income streams, leading to year￾round income potential. “I can make sure my family and my wife are taken care of when I am gone. I may not be here, but all the tools and infrastructure provided will make sure that the practices I learned from the project will help my family through to the next generation. To this day, my coconut plants survive because of the material such as water pipes etc provided for the drip irrigation method we were taught. With 2 clay pots provided to keep water on either side of the plants we learnt that the moisture of the soil can last through the dry season even with little water. Also, I should mention that not only do my agricultural wells help me; they also help others in the area during the dry season to sustain themselves and their crops.” Less impactful for project sustainability was the provision of livestock in general and dairy cows in particular. Thirty-three workshop participants reported receiving livestock with only 16 (48%) continuing husbandry today. Participants report challenges in maintaining healthy livestock with consistent water availability and diseases being common issues. Other participants commented on the amount of labor required to keep cattle, noting decreasing labor capacity as they have aged and children frequently leaving agriculture to pursue other career paths. Some workshop participants indicated that their Buddhist beliefs were incompatible with animal husbandry. Where farmers have continued animal husbandry, production of buffalo (instead of dairy cattle) is common, suggesting that dairy cattle are poorly adapted to local conditions. Hardier buffalo appears to correlate with sustained animal husbandry in this project. One interviewee commented: Some of the animals provided through the project either were not compliant with the local environment or were not responsive to the local culture. Cattle and fish provided through the project were not compatible with the frequent droughts experienced in the region. Cattle suffered from severe dehydration which resulted in less milk supply, lower fat content in the milk supply (leading to decreased prices), as well as death. Fish provided in lakes and ponds suffered due to the decreasing water supply of the lakes and ponds, either being captured by wild cats as the pond become shallower or perishing once the water supply had fully disappeared. While many farmers have divested of cattle and no longer pursue dairy farming for practical or religious reasons, at least one farmer was observed to have specialized in dairy production and another has established a dairy farmers’ organization in his community, leading to sustained localized dairy production that one farmer directly attributed to this project. “Although many of the project beneficiaries have given up their livestock (cows), now the number of farmer families who are engaged in animal husbandry in the village have increased to 35 from 11 and benefit from the initial stocks provided by the project. This has given us a sense of pride and stability that we have never experienced before.” Processing and Post-Harvest Centers A key component of this project was the facilitation or creation of processing and post-harvest centers for partner associations and companies. These facilities prolong the shelf-life of agricultural products and contribute to the market success of farmers. Continued operation of CASE STUDY — Sri Lanka | Findings | 113 these facilities is a vital component of sustained project impacts. MILCO Chilling Center — The MILCO Chilling Center opened in November 2007 with the project’s provision of equipment including a dump tank, milk lines, and two freezers. MILCO engineers maintain and repair equipment as needed, and the equipment is still in good condition. The center has four employees and serves about 400 farmers, collecting about 2,600 liters of milk each day. According to the MILCO Dairy Instructor, a majority of farmers that were involved in the project still supply milk to the Center. Photo 6 — Milco Chilling Center Built by USDA Funds. Storage facility at Agrarian Service Center: The storage facility was built by the project in February 2008. In a typical year, this facility stores 20,000,000 kg of paddy (rice) harvested by 300 to 400 farmers, including project beneficiaries. The facility remains in fair condition with the Divisional Officer indicating that steps towards needed repairs have been taken. This same individual estimate the facility will be useful for at least another 20 years. The facility requires farmers to be registered and the registration is open to all farmers. Besides paddy, the facility is used to store seeds and fertilizers for registered farmers and provides loans. Photo 7 — Giribawa Sri Lanka, Agrarian Services Center that manages the storage facility built by USDA funds Cargills Vegetables and Fruits Collection Center: The Cargills center was built in September 2009 with the project donating the land and the building. No equipment was provided by the project. The center is in good condition and requires routine renovations every 10 years. The center currently employs two supervisors, one agriculture field officer, and 10 service assistants focused on packing and quality assurance of fruits and vegetables. The center is the only collection center for fruits and vegetables in the area of Thanamalvila, serving over 400 farmers, including project beneficiaries. The center receives about 10,000 kg of vegetables and fruits per day, with produce arriving by truck, tractor, bicycle, motorbikes, three wheelers, and by foot. Produce is then transported to the Cargills collection center in Colombo daily. To register with the Center, farmers sign a loyalty agreement with Cargills, belong to a farmer organization, and have high 114 | CASE STUDY — Sri Lanka  | Findings quality produce. The center provides training on crop cultivation, integrated pest management, drip irrigation, good agricultural practices (GAP) certification, and proper usage of agricultural chemicals. Cargill’s keeps aside 50 cents per kilo of produce for a farmer welfare fund in order to provide scholarships for farmer children. Photo 8. Cargills Vegetables and Fruits Collection Center in Thanamalvila What are the impacts of objective 2? Workshop participants most commonly credited the project’s contribution of new agricultural technologies including tree seedling, livestock and fish, and water storage with increased income; increasing farm productivity overall; improving diversity of agricultural products leading to more resilient yields; and improved nutrition and saving time. Less frequently, participants indicated new agricultural technologies provided better access to markets. Interestingly the provision of water tanks and other irrigation technologies were the most strongly associated with impacts related to improved production and more money. Number of Respondents Reporting Impact Impacts 0 More Money Better Health Improved Nutrition Access to Markets Time Gain Crop Diversification 102030405060 Chart 14 — Impacts associated with objective 2 CASE STUDY — Sri Lanka | Findings | 115 OBJECTIVE 3: To provide better market access by establishing farmer marketing groups and training in improved processing storage and marketing approaches. Activities corresponding with Objective 3. Number who learned about the activity in 2015 Number who continued to use the knowledge/ inputs 2019 Percent who continue to use what they learned during project implementation Training in business Training in Business plans 60 16 27% Training in Marketing 24 11 46% Training in Farmer marketing groups 24 20 83% Total 108 47 44% Table 31 — Activities corresponding with objective 3 and percentage change between beneficiaries that learned and continue using practices introduced by the project. Training in business Of the 59 workshop participants, 24 respondents reported they received training in marketing and farmer marketing groups, and all 59 respondents said they received training in business plans. The respondents’ continued practice of the knowledge varied: 46% for marketing, 83% for farmer marketing groups, and 27% for business plans. The participants who adopted marketing practices were primarily in Galenbindunawewa and in Thanamalvila. Participants described the challenges in marketing as a lack of access to markets and a lack of guaranteed prices for produce. Some participants perceived that merchants and middlemen paid them below market values. In cases where farmer marketing groups were practiced, the collective bargaining practice was seen to benefit the farmers secure better prices. “Training in business planning and links to marketing groups and networks has played a key role in how successful our collective has become. Many of our farming community leaders are key players in making sure our crops fetch a competitive price for our produce. We prefer not to sell our individual produce as we are unable to get a reasonable price for small quantities. We have organized ourselves into a collective and that has bargaining strength with the ability to make demands from distributors and sales networks. At times, we do not harvest the crops until we get the right price.” Workshop participants explained that business management practices were found to be cumbersome, and many participants have adapted practices to be components of what they were trained to do or have dropped them altogether. 116 | CASE STUDY — Sri Lanka  | Findings “Most farmers were uncomfortable with business planning and such practices which they felt were time consuming. They collectively agreed that they prefer to farm through practical knowledge and experience rather than pre-planning on paper.” Access to markets Interviewees and workshop participants state that the project’s implementation increased the access to and income from markets for project beneficiaries. While none of the farmers visited by the local evaluation firm explicitly mentioned that the USDA project improved their access to markets, some mentioned that they are selling directly to a market or to wholesalers who come to their farms. Respondents in the workshops describe this in a few ways. The increased crop production and variety increased the amount of produce to be sold at the market, increasing farmer incomes. The increased variety of products also opened up access to new markets and sources of income throughout the year. Additionally, farmers were trained on post-harvest processing techniques that increased the quality of the products and thus their market value. Interviewees and workshop participants state that the project’s design did not include a strong effort to build market connections for project beneficiaries. One farmer mentioned that the Cargill’s vegetable center, located 30 kilometers away, was too far to bring products. Another farmer mentioned that the USDA project did not help him to find a market, while he was delivering his milk to Milk Chilling Center trucks. The centers brought by the USDA project said they are offering either guaranteed prices, fairly negotiated prices, or competitive market rates to farmers. • ●MILCO Chilling Center: The Center offers a “guaranteed price” according to a predetermined formula set by MILCO based on fat and nonfat solids (e.g., 12.5 grams of total solids/liter fetches Rs 65/liter) as well as dairy production trainings to farmers. Some farmers identified increased connections to markets, especially those who began selling milk at Milco. These farmers did not have prior relationships established, or previously relied on middlemen who offered below market rate prices for goods. • ●Storage facility at Agrarian Service Center: The facility allows farmers to market traditional varieties of paddy at premiums (e.g., Rs 60 to 80) over the common varieties (Rs 39 to 41). Without the facility, it is believed that farmers would not be able to market their paddy effectively. Farmers receive guaranteed prices maintained by the facility and the Agrarian Service Center and may be invited to discuss appropriate prices. • ●Cargills Vegetables and Fruits Collection Center: The Center established the supply chain by collecting vegetables and fruits from farmers in Thanamalvila and transporting to a hub collection center in Colombo, from which produce is shipped to Cargills supermarkets throughout the country on a daily basis. Before the Center was built, the farmers were selling to wholesalers directly or via farmer organizations. At the Center, the price of vegetables and fruits are determined each day by supply and demand and the competitor prices at the Cargills Processing Center in Colombo, and it is believed that the Center offers better prices to the farmers in Thanamalvila compared to what the wholesalers would offer. In addition to data collected in the workshops the field visits found some signs of increased income. Six farmers visited mentioned an increase in income due to increased production of their crops or animals. A female farmer mentioned that she had doubled her income by selling crops and buffalo milk after the project implementation. Another farmer raised some buffaloes from CASE STUDY — Sri Lanka | Findings | 117 the initially provided stock and bought more milking cows to expand milk production. A non￾beneficiary farmer but operating near the Cargills Vegetable Collection Center was said to have increased his production and income, afforded a house and a car, received additional training on GAP certification, planting techniques, and fertilizer techniques from Cargills. He currently supplies between 200kg and 300kg of vegetables (Egg plants, tomatoes, beans) and fruits (Banana) to the Collection Center on a daily basis. What are the impacts of objective 3? The workshop participants who were trained in marketing, creating business plans and farmer group marketing indicated that they experience increased income, having more time, better crop diversification as a result of continuing these practices. Increases in money and income were mostly associated with continued use of business plans, while the trainings on farmer marketing groups resulted in more time available. Number of Mentions Impact 0 More Money Better Health Improved nutrition (more or better food) Increase in Production Access to Markets Time gains Crop Diverificaion 10 20 30 40 50 Training Type: Business Plan Farmer Marketing Group Marketing Chart 15 — Impacts from objective 3. Beneficiaries from Galenbindunawewa reported that the training has led to the development of successful farmer marketing groups, which help set the cost of goods in their local region and secure fair market values for their goods when selling to merchants and middlemen. All of the participants in Galenbindunawewa indicated that they have experienced increased market connections and sales as a result of the project’s resources and activities, while none of the participants in the other two communities indicated this impact. Overall, the workshop participants reported the increased sense of connections to markets. Cargill’s donated a portion of their purchases back to the target communities to increase their available financial resources. This practice has continued to this date along with occasional field training and assistance offered to farmers. 118 | CASE STUDY — Sri Lanka  | Conclusions “Companies like Milco and Cargill’s send their field officers to guide farmers from time to time, and that is a big help for them.” The project also partnered with the government and other local actors throughout implementation. The partnerships have increased the demand for the services offered by these actors and built strong, lasting relationships among them. Conclusions Data collection and analysis for the present study was conducted in early 2019, almost nine years since The World Vision Sri Lanka project finished in 2010. Given that 9 years have passed since the project closed it is remarkable that beneficiaries interviewed remembered the details of the project including what they learned from the training workshops and manuals. Also, beneficiaries explained how and why they are continuing to benefit from the project’s activities. Overall the most successful components of the project (excluding beekeeping and fishery) were the training related to soil and irrigation, water storage, tree production including fruits, annual crop production and the provision of water saving/harvesting technologies and tree seedlings. In addition, an increase in income was perceived by most of the beneficiaries through the increase in crop yield and access to markets and infrastructure to sell and store farmers products. The impacts to beneficiaries most associated with continuing these activities are more money and income (252 mentions), increases in production (194 mentions), improved nutrition (176 mentions), more time (150 mentions), and crop diversification (141 mentions). Activity/ Impact More money Better health Improved nutrition (more or better food) Increase in production Access to markets Time gains Crop divers￾ification Grand Total 252 115 176 194 72 150 141 Table 32 — Number of mentions by activity or impact from the project. Activities that are still in practice were also notable in today’s farmers practices. From the total farmers that mentioned learning about different activities during the training and workshops, the most continued activities are training in soil fertilization and irrigation, and Long-term and medium￾term tree production. 97% of farmers that mentioned learned those activities, are still practicing them. Another activity that farmers still use are the water tanks and storage. 96% of farmers that learned how to use them continue using water infrastructure today. Provision of trees, knowledge of the training on water tanks and storage and short-term crops production were mentioned with less frequency as activities that farmers remember using today. CASE STUDY — Sri Lanka | Conclusions | 119 Activities that have been the most likely to continue. Number who learned about the activity in 2015 Number who continued to use the knowledge/ inputs 2019 Percent who continue to use what they learned during project implementation Training in Soil fertilization and irrigation 39 38 97% Long-term and medium-term tree production 60 58 97% Provision of water tanks and storage 55 53 96% Provision of trees 59 56 95% training in water tanks and storage 57 51 89% Short-term crop production 60 52 87% Table 33 — Most likely Activities corresponding with and percentage change between beneficiaries that learned and continue using practices introduced by the project. Below are the impacts from the top six continued activities associated with each of these most continued activities in a greater description. Increase in income is one of the most frequent impacts mentioned by beneficiaries. Also, diversification of the farm crops was often noticed by the farmers. In addition impacts in nutrition, production, time, health and access to markets complement the list of the impacts that were more relevant from the project. 120 | CASE STUDY — Sri Lanka  | Conclusions TIME GAINS, 86 IMPROVED NUTRITION, 83 BETTER HEALTH, 62 INCREASED PRODUCTION, 127 IMPROVED ACCESS TO MARKETS, 39 Impacts associated with Sri Lanka project activities Numbers indicate numbers of respondents reporting impact MORE MONEY/INCOME, 127 Training in soil fertilization and irrigation, 10 Provision of long/med term tree seedings, 16 Training in water storage and irrigation, 8 Training in short term crop production, 40 Provision of water storage and irrigation technology, 22 Training in long/med term tree production, 31 MORE CROP DIVERSIFICATION, 92 Training in long/med term tree production, 39 Provision of water storage and irrigation technology, 6 Provision of long/med term tree seedings, 25 Training in water storage and irrig..., 3 Training in soil fertilization and irrigation, 4 Training in short term crop production, 15 Training in soil fertilization and irrigation, 7 Provision of long/med term tree seedings, 5 Training in short term crop production, 34 Training in water storage and irrigation, 10 Training in long/med term tree production, 9 Provision of water storage and irrigation technology, 21 Training in soil fertilization and..., 7 Training in long/med term tree production, 14 Provision of water storage and irrigation technology, 10 Provision of long/med term tree seedings, 8 Training in water storage and irrigation, 7 Training in short term crop production, 16 Provision of water storage and irrigation technology, 9 Training in short term crop production, 12 Training in soil fertilization and irrigation, 4 Training in long/med term tree production, 4 Training in water storage and irrigation, 5 Provision of long/med term tree seedings, 5 Training in long/med term tree production, 18 Training in short term crop production, 29 Provision of water storage and irrigation technology, 15 Provision of long/med term tree seedings, 13 Training in soil fertilization and irrigation, 5 Training in water storage..., 3 Training in long/ med term tree production, 31 Provision of water storage and irrigation technology, 22 Training in short term crop production, 40 Provision of long/med term tree seedings, 16 Training in soil fertilization and irrigation, 10 Training... water..., 8 Chart 16 — impact from the top six continued activities How did existing conditions affect project sustainability? Various conditions existed in Sri Lanka at the outset of the project, we considered conditions to be anything from rainfall, to literacy, to family structure, some of these conditions made it more likely that the project would be sustainable, and some conditions negatively affected sustainability. This section describes the existing conditions that contributed the most to the sustainability of the project. Natural Environment — The targeted regions had the necessary physical attributes and characteristics to support agriculture. These attributes include fertile soil and seasonal rainfall. However, the seasonal nature of rainfall also imposed a pre-existing limitation. Specifically, prior to this project, water for crops and livestock was not consistently available throughout the year, limiting farm productivity to the short-wet season. This project addressed this limitation, providing water capture and storage capacity and training. As noted above, access to stored water enabled year-round agricultural productivity and a large majority (96%) of recipients of water storage devices still use them today. CASE STUDY — Sri Lanka | Conclusions | 121 Agricultural Environment — Most farmers cultivated a mix of crops or a mix of crops and livestock species (commonly backyard chicken and cattle). Many of them produced for self-consumption, and the prevalence of farmers who marketed agricultural produce varied depending on the type of crop and region. For example, for paddy, the rate of cultivation was about 70% of farmers in Galenbindunawewa, 85% in Giribawa, and 25% in Thanamalvilla and of those the rate of those who marketed was about 30% in Galenbindunawewa, 55% in Giribawa, and 30% in Thanamalvilla. About 70 to 85% of farmers cultivated other field crops (OFC) such as maize, cowpea, gram, and millet, and the rate of those who marketed ranged from about half in Giribawa and two-thirds in Thanamalvilla to most growers in Galenbindunawewa. About half of the growers of OFC processed their produce for marketing. About 60-90% of the farmers grew vegetables, while the reason for growing vegetables differed by the region; the rate of those who market vegetables was about 20% in GA, 10% in GI, and 60% in TH. About 30 to 50% of the farmers grew fruits, mostly for self-consumption. Just over 10% of the farmers in GI grew fruits as market produce. The most common marketing outlet for most types of produce was collecting centers in local villages, with the exception of paddy most commonly sold to contractors based on buyback contractual agreements. The perceived usefulness of collective bargaining was very low in most types of crops and in most regions. “Most of the farmers cultivated only in Maha season before the project started. Then, we developed the water facilities. We improved soil fertility, taught them soil conservation practices and then they were able to cultivate during both seasons – even between Yala and Maha seasons, they cultivated vegetables and other basics for their own consumption.” Economic Environment — According to the Baseline Survey, the most households living under Rs 5,000 per month primarily were earning income from farming. Farmers typically owned 1 to 3 acres of land. Paddy was produced in low land, while other crops were cultivated in uplands or home gardens. Majority of farmers cultivated these crops only in rainy seasons (Maha) due to the lack of water in dry seasons (Yala). Educational Environment — Sri Lanka enjoys high literacy rates (apx 90%), a pre-existing condition leveraged effectively by this project. In particular, written educational materials distributed by this project were retained by beneficiaries as reference materials after the end of the project. In some instances, beneficiaries indicated they memorized the content and then shared the written materials with neighbors and friends. Biotic Stresses — Diseases, insect pests, monkeys, and elephants were among pre-existing biotic stresses that impact Sri Lankan agriculture. This project was unable to address these issues and they continue to negatively impact crop production and project sustainability. Several farmers indicated that wild elephants, in particular, are destructive, especially for coconut production, attributing the challenges to reduced elephant habitat: “This problem was not prevalent in the early days. This problem was initiated during the 1970 – 1975. During those days, villages started to cut down forests to satisfy their need of available lands for agriculture. As a result, the density of natural forests decreased, which leads to the imbalance of the foraging patterns of elephants. Then this conflict started to take place.” 122 | CASE STUDY — Sri Lanka  | Conclusions One farmer reported a particularly innovative approach to preventing crop damage by elephants. She built a tree house in which she sleeps every night. When elephants arrive in search of food, she throws lit firecrackers to scare them off. This dedication and innovation have protected her coconuts and other crops for the past decade. Some of the limiting existing conditions were addressed by the project. For example, training in water management and provision of water tanks, wheels and storage aimed to reduce severity of the droughts, and got farmers out of seasonal only production. Introduction of new short- and long-term crops increase diversity. Training in how to cultivate plants, irrigation and soil fertility complement lacking knowledge that farmers need to maintain their crops all year around. Some existing conditions such as limited access to main roads or preventing the human elephant conflict was not directly reduced. The efforts that went into defining existing constraints (water, soil conditions, etc.) helped guarantee that the trainings would be useful and lead to positive impacts on crop productivity and diversification How did design and the implementation of the project contribute to its sustainability? The project was designed to build upon existing agricultural knowledge and practices in the region. The project specifically targeted seasonal farmers who had access to land and an interest in moving to year-round agricultural practices and a wider variety of crop production. Components of the design which were particularly influential in sustainability included the project’s responsiveness to the assets and deficits of the region; the project built upon existing knowledge and infrastructure while responding to the lack of year-round water resources and agricultural practices. Interviewees and workshop respondents state that the provision of water tanks and other storage systems have increased their capacity for year-round agricultural production, moving them from seasonal farmers to year-round farmers who produce a wide variety of crops. Importantly, provision of these resources early in the project enabled subsequent crop and livestock production and built overall trust in the project on the part of the recipients. The community began to become invested in the other training and resources that the project may provide, after seeing this early success. As noted in the findings, the provision of long term and medium-term tree crops contributed to A farmer’s tree house in which she sleeps every night. When elephants arrive in search of her coconuts she throws lit firecrackers to scare them off. CASE STUDY — Sri Lanka | Conclusions | 123 project sustainability. But a few project beneficiaries explained that the provision of these trees occurred in two cycles. The first distribution cycle depleted the country’s supply of available first quality trees. Consequently, in the second distribution cycle, the project distributed lower quality trees resulting in higher attrition rates due to adverse conditions, in sharp contrast with high survival rates experienced in the first distribution cycle. Similarly, some of the livestock animals provided by the project were not compatible with the local environment. Cattle and fish provided by the project were not compatible with the frequent droughts experienced in the region. Cattle suffered from severe dehydration which resulted in less milk supply, lower fat content in the milk supply (leading to decreased prices), as well as death. Fish suffered due to the decreasing water supply of the lakes and ponds, ultimately being captured by wild cats as the ponds become shallower or perishing once the water supply had fully disappeared. Additionally, several respondents indicated that keeping livestock, specifically cattle and bees, is inconsistent with their religious and cultural values. “Some farmers who were older, opted to give up livestock (cows) management due to its physically demanding nature. Another reason to give up animal husbandry was that some Buddhists prefer not to inflict entrapment and servitude on animals. Due to these reasons, some in our community have given up their livestock to other farmers who are able to dedicate the time and energy to follow through. Bee keeping is currently practiced by only one farming family.” “The introduction of Medium to Long term crops through the trainings and the provision of seedlings was a major boost we received to ensure we had crops that would continue to yield harvest for decades to come. These crops (Mahogany/Teak/Mango /June Plums/Coconut) meant planning for a steady income for the future which as a community we had never thought of before . Medium term crops (Moringa, Local sweet orange, Lime/Papaya) meant an intermediate income as well.” The implementation, staging and sequencing of project activities was key to the success and ultimately the sustainability of those activities. Building upon an understanding of the main issues farmers were facing and initiating with the water storage options created the conditions for the introduction of the new, diversified and year-round cropping systems. Significant changes were introduced and adopted and are continuing. The fact that each introduced change was accompanied by the associated training was important to sustainability. The written materials that accompanied the training continue to be used to maintain and expand project impacts. Several capacity building factors contributed to the success that was achieved: Training and technical assistance were sequenced in a way that created the conditions to move from one activity to the next. Establishing water storage options was key to adding stability and expanding the annual cropping systems and removing the uncertainty that water shortages created. Adopting change is risky with potential for crop loss as farmers learn how to manage new crops. Stabilizing the annual system provided farmers security and stability that allowed them to establish the medium- and longer-term crops that were not able to demonstrate benefits until later but, over time, have demonstrated benefits recognized by project farmers when interviewed. 124 | CASE STUDY — Sri Lanka  | Conclusions The project offered a fairly wide variety of options. Some of the livestock options were not as successful as the crop and tree options for practical and cultural reasons. The variety of options was important and allowed participants to select those that worked out best over time and provided greater opportunities for extending the cropping season and year-round income and income over time as tree crops matured. At the same time that variety of options also represented a training challenge which seems to have been successfully implemented. One example of how the training’s design built upon community strengths is that the training involved written packets which were widely distributed throughout communities. Participants explained that they were able to use the packets for reference long after the project’s end, some even stating that they worked with the packets enough to memorize their teachings and distribute the packets to non-beneficiary farmers for their learning. Marketing poses a huge challenge for many farmers in remote areas. The USDA project sought to Improve farmers’ access to markets and the prices they receive through marketing training and partnerships with existing actors. Farmers’ interests in training in farmer marketing groups appear at odds with the unfavorable perception of collective bargaining among the farmers, as reported in the Baseline Survey. While it is unclear to what extent it was planned, it appears that the increased variety of products opened up farmers’ access to new markets throughout the year. Additionally, farmers were trained on post-harvest processing techniques that increased the quality of the products and thus their market value. Some farmers also identified increased connections to markets, especially those who began selling milk at MILCO chilling center. The partnerships that established MILCO Chilling Center, Cargill’s Vegetables Collection Center, and Agrarian Service’s storage facility were examples of successful design and implementation for helping farmers connect with markets. They demonstrated the importance of efficient logistics and collective bargaining to secure competitive market prices. “World Vision created this as a system, and it buys directly from the farmers and farmer organizations. There was corruption and no guaranteed price at that time. After implementing the center, corruption was eliminated” Respondents describe this in a few ways. The increased crop production and variation increased the amount of produce to be sold at the market, resulting in increased incomes. The increased variety of products opened up access to new markets and sources of income throughout the year. Additionally, farmers were trained on post-harvest processing techniques that increased the quality of the products and thus their market value. Some farmers also identified increased connections to markets, especially those who began selling milk at MILCO chilling center. These farmers did not have prior relationships with MILCO, and for crop produce they relied primarily on middlemen who offered below market rate prices for goods. CASE STUDY — Sri Lanka | Conclusions | 125 Recommendations Future investments in Sri Lanka should learn from the sustainability success and failures of the World Vision Project. While some aspects of the dairy component were a success: poor breed selection, lack of cultural relevancy and buy-in, and a lack of continued and stable market linkages proved to be major barriers to sustainability. Future projects should first assess beneficiary willingness to engage in dairy production, secure buyers, and select improved local breeds. Annual vegetable production and long and medium term tree production were highly sustainable, and provided beneficiaries with additional and year round income, the addition of improving access to water facilitated the sustainability of these activities. From the other case studies these components combined with facilitating access to finance even further increased sustainability. Future investments in Sri Lanka should consider working with an existing and reputable agency to provide tailored financial products to small scale producers. Literature Fladby, B. (1983). Household Viability and Economic Differentiation in Gama, Sri Lanka: An Anthropological Study of Paddy Producing Households Participating in the Green Revolution within an Irrigation Settlement in Sri Lanka. Department of Social Anthropology. The University of Bergen. ISSN 0332-8147 Jayaweera, T., Ruwandeepika, H., Kendaragama, W., Jayarathne, H. & Thotawaththe, T. (2007). Analysis of Cost of Milk Production in Ratnapura District. The Journal of Agricultural Sciences, 2007, vol. 3, no. 1. Department of Livestock Production, Faculty of Agricultural Sciences, Sabaragamuwa University of Sri Lanka, Belihuloya, Sri Lanka Menike, L, Arachchi, K. (2015). Adaptation to Climate Change by Smallholders Farmers in Rural Communities: Evidence from Sri Lanka. Science Direct: Procedia Food Science 6 (2016) 288-292 Sri Lanka’s Department of Animal Production & Health, (2005). Kye Statistics of Dairy Industry 2005-2017. Retrieved from http://www.daph.gov.lk/web/images/content_image/Livestock_stat/ key_stat/2018/cattle. Weeraratna C. & Weerasighe, P., (2009). Agriculture of Sri Lanka. First EditionAugust 2009. Sridevi Printers Ltd. ISBN 978-955-51646-0-3 World Bank. ( 2019). Sri Lanka: Share of economic sectors in the gross domestic product (GDP) from 2007 to 2017 [Chart]. In Statista. Retrieved July 23, 2019, from https://www.statista.com/ statistics/728539/share-of-economic-sectors-in-the-gdp-in-sri-lanka/ Zubair, L. (2005). Modernization of Sri Lanka’s Traditional Irrigation System and Sustainability, Science, Technology and Society, 10(2), 161-195. 126 | Annexes  | Annex I — Evaluation Statement of Work Page 1 of 9 NOFO Template – Version FY16-01 The Department of Agriculture (USDA) – Foreign Agricultural Service (FAS) Office of Capacity Building and Development Notice of Funding Opportunity (NOFO) Food for Progress Post-Project Sustainability Assessment A. Program Description Program Overview, Objectives, and Priorities The post-project evaluation will determine to what extent activities and results of the selected projects in several regions have been sustained after USDA’s involvement ended. This evaluation will contribute to answering one or more research questions in the FFPr Learning Agenda. It will incorporate both existing data and the collection of primary data. The evaluation report will detail evidence of sustainability of project activities/practices and results, evidence of unanticipated sustained effects where applicable, and provide recommendations on what project implementers could have done to enhance sustainability. The assessment will enhance USDA’s and stakeholders’ understanding of what is and isn’t realistically sustainable after USDA’s exit. This original research is anticipated to influence future project funding, design, implementation and evaluation. Additional information is described in the appendix. Issued By Foreign Agricultural Service, Office of Capacity Building and Development, Food Assistance Division, Food for Development Branch Catalog of Federal Domestic Assistance (CFDA) Number and Title 10.960 Technical Agricultural Assistance. Notice of Funding Opportunity Title Food for Progress Post-Project Sustainability Assessment NOFO Number USDA-FAS-10960-0700-10.-17-0004 Authorizing Authority for Program National Agricultural Research, Extension, and Teaching Policy Act of 1977, PL 95-113, as amended, 7 USC §§ 3291(a)(1)(B), 3291(a)(3), 3291(a)(7)(A), 3317, & 3319a Appropriation Authority for Program Agricultural Act of 2017, PL 113-79, § 3201 Program Type New, One-Time Annexes Annex I — Evaluation Statement of Work Annexes | Annex I — Evaluation Statement of Work | 127 Page 2 of 9 NOFO Template – Version FY16-01 B. Federal Award Information Award Amounts, Important Dates, and Extensions Estimated Award Amount: Up to $721,526.00 (Subject to the availability of funding) Projected number of Awards: 1 Period of Performance: Approximately two (2) years Projected Period of Performance Start Date(s): 09/29/2017 Projected Period of Performance End Date(s): 09/30/2019 Extensions may be considered; please see Section H. Pre-Award costs: Not authorized. Cost Share or Match requirements: None/Not required Funding Instrument Cost-Reimbursable Agreement Note that indirect costs, by law, cannot exceed 10% of direct costs under this agreement instrument. FAS has no authority to waive this ceiling. Subawards will be considered on a case-by-case basis, if included in the awardee’s proposal. C. Eligibility Information Eligible Applicants Public & State controlled institutions of higher education State cooperative institutions (as defined by 7 USC § 3103) All applicants must have an active registration in the SAM database at https://www.sam.gov – pending or expired registrants are not eligible. This requirement must be met by the closing date of the announcement and will not be waived. Please contact the program officer listed if you have questions about this requirement. Other Submission Requirements and Information Pursuant to the Consolidated Appropriations Act, 2017, PL 115-31, § 746, all applicants must certify that they have not been convicted of a felony criminal violation under any Federal law within the preceding 24 months. Maintenance of Effort (MOE) MOE is not allowable. 128 | Annexes  | Annex I — Evaluation Statement of Work Page 3 of 9 NOFO Template – Version FY16-01 D. Application and Submission Information Key Dates and Times Application Start Date: Immediately Application Submission Deadline: Wednesday, September 13, 2017 11:59 PM EDT Anticipated Funding Selection Date: Monday, September 25, 2017 Anticipated Award Date: Friday, September 29, 2017 Other Key Dates FAS will hold an open conference call to discuss this notice of funding opportunities and answer questions. All interested parties are encouraged to join on Wednesday, August 30, 2017, at 11:00 AM EDT. Participants will not be required to identify themselves or their institutions, and participation in this call is not required or tracked. (Note: The conference call system may prompt callers to “please state your name.” Just press the pound key to bypass.) The conference line is: +1 888 844 9904 Conference ID: 3243079# Address to Request Application Package This document contains the entire application package. Applications are preferred to be received through the ezFedGrants portal at https://grants.fms.usda.gov – prospective applicants are encouraged to register for this portal. Applicants that are unable to access the ezFedGrants portal should contact the program manager for alternative submission instructions. Note that if selected, registration is a requirement of performance. Applicants, who have submitted their applications through the ezFedGrants portal, are encouraged to contact the program manager to confirm receipt. Content and Form of Application Submission All applications must be in Adobe PDF format, accompanied by the following documents in Adobe PDF format. Please ensure that the most current version of each required form is used by following the links provided.  Standard Form 424, signed by the applicant  Standard Form 424A  Agriculture Department Form 3030, signed by the applicant  Agriculture Department Form 3031, signed by the applicant  A detailed budget and budget narrative, of sufficient detail to justify the expenditure of federal funds Page 3 of 9 NOFO Template – Version FY16-01 D. Application and Submission Information Key Dates and Times Application Start Date: Immediately Application Submission Deadline: Wednesday, September 13, 2017 11:59 PM EDT Anticipated Funding Selection Date: Monday, September 25, 2017 Anticipated Award Date: Friday, September 29, 2017 Other Key Dates FAS will hold an open conference call to discuss this notice of funding opportunities and answer questions. All interested parties are encouraged to join on Wednesday, August 30, 2017, at 11:00 AM EDT. Participants will not be required to identify themselves or their institutions, and participation in this call is not required or tracked. (Note: The conference call system may prompt callers to “please state your name.” Just press the pound key to bypass.) The conference line is: +1 888 844 9904 Conference ID: 3243079# Address to Request Application Package This document contains the entire application package. Applications are preferred to be received through the ezFedGrants portal at https://grants.fms.usda.gov – prospective applicants are encouraged to register for this portal. Applicants that are unable to access the ezFedGrants portal should contact the program manager for alternative submission instructions. Note that if selected, registration is a requirement of performance. Applicants, who have submitted their applications through the ezFedGrants portal, are encouraged to contact the program manager to confirm receipt. Content and Form of Application Submission All applications must be in Adobe PDF format, accompanied by the following documents in Adobe PDF format. Please ensure that the most current version of each required form is used by following the links provided.  Standard Form 424, signed by the applicant  Standard Form 424A  Agriculture Department Form 3030, signed by the applicant  Agriculture Department Form 3031, signed by the applicant  A detailed budget and budget narrative, of sufficient detail to justify the expenditure of federal funds Annexes | Annex I — Evaluation Statement of Work | 129 Page 4 of 9 NOFO Template – Version FY16-01 The proposal narrative should not exceed 12 pages and should describe how the scope will be approached, and include clear timelines for field work and deliverables. Electronic Signature. FAS accepts electronic signatures on all proposal and award documents. FAS will use only electronic signatures on award documents unless the proposal specifically states that institutional policy precludes the use of electronic signatures. Intergovernmental Review This program is not subject to E.O. 12372. Funding Restrictions Generally, funds may not be used in any manner that is prohibited by 2 CFR Part 200 and 2 CFR Part 400. For this program, funds may not be used for general-purpose equipment. All air travel must be in economy class, and compliant with the Fly America Act (49 USC § 40118). Management and Administration (M&A) Costs: M&A costs are not allowable. Indirect Facilities & Administrative (F&A) Costs. Pursuant to statutory restriction in 7 USC § 3319a, indirect costs are limited to 10% of direct costs. FAS has no authority to waive this limitation. E. Application Review Information Application Evaluation Criteria Prior to making a Federal award, the Federal awarding agency is required by 31 U.S.C. 3321 and 41 U.S.C. 2313 to review information available through any OMB-designated repositories of government-wide eligibility qualification or financial integrity information. Therefore application evaluation criteria may include the following risk based considerations of the applicant: (1) financial stability; (2) quality of management systems and ability to meet management standards; (3) history of performance in managing federal award; (4) reports and findings from audits; and (5) ability to effectively implement statutory, regulatory, or other requirements. Factor 1: Technical Approach i. The applicant’s proposal describes a realistic, effective and sound approach, including a description of methods that is of high quality and analytical rigor for achieving the project’s requirements. 130 | Annexes  | Annex I — Evaluation Statement of Work Page 3 of 9 NOFO Template – Version FY16-01 D. Application and Submission Information Key Dates and Times Application Start Date: Immediately Application Submission Deadline: Wednesday, September 13, 2017 11:59 PM EDT Anticipated Funding Selection Date: Monday, September 25, 2017 Anticipated Award Date: Friday, September 29, 2017 Other Key Dates FAS will hold an open conference call to discuss this notice of funding opportunities and answer questions. All interested parties are encouraged to join on Wednesday, August 30, 2017, at 11:00 AM EDT. Participants will not be required to identify themselves or their institutions, and participation in this call is not required or tracked. (Note: The conference call system may prompt callers to “please state your name.” Just press the pound key to bypass.) The conference line is: +1 888 844 9904 Conference ID: 3243079# Address to Request Application Package This document contains the entire application package. Applications are preferred to be received through the ezFedGrants portal at https://grants.fms.usda.gov – prospective applicants are encouraged to register for this portal. Applicants that are unable to access the ezFedGrants portal should contact the program manager for alternative submission instructions. Note that if selected, registration is a requirement of performance. Applicants, who have submitted their applications through the ezFedGrants portal, are encouraged to contact the program manager to confirm receipt. Content and Form of Application Submission All applications must be in Adobe PDF format, accompanied by the following documents in Adobe PDF format. Please ensure that the most current version of each required form is used by following the links provided.  Standard Form 424, signed by the applicant  Standard Form 424A  Agriculture Department Form 3030, signed by the applicant  Agriculture Department Form 3031, signed by the applicant  A detailed budget and budget narrative, of sufficient detail to justify the expenditure of federal funds Page 5 of 9 NOFO Template – Version FY16-01 ii. The applicant’s proposal contains specific deliverables and proposed timeline. iii. The applicant’s proposal includes all of the sections in the statement of work. Factor 2: Experience and Expertise i. The applicant’s proposal contains evidence of experience in designing and implementing evaluations for agricultural programs in the context of developing countries. ii. The applicant’s proposal contains a staffing plan that identifies how the evaluation will be staffed, including the role of key personnel and subawardees (if any). It is not strictly necessary to identify individuals, other than the principal investigator and other key personnel, by name, so long as the staffing plan includes Factor 3: Past Performance i. Past programs, with similar objectives, described by the applicant in their proposal, will be positively reviewed. ii. Prior to making a Federal award with a total amount of Federal share greater than the simplified acquisition threshold as defined by 48 CFR § 2.101, is required to review and consider any information about the applicant that is in the designated integrity and performance system accessible through SAM (currently FAPIIS). iii. An applicant, at its option, may review information in the designated integrity and performance systems accessible through SAM and comment on any information about itself that a Federal awarding agency previously entered and is currently in the designated integrity and performance system accessible through SAM. iv. will consider any comments by the applicant, in addition to the other information in the designated integrity and performance system, in making a judgment about the applicant’s integrity, business ethics, and record of performance under Federal awards when completing the review of risk posed by applicants as described in 2 CFR §200.205 Federal awarding agency review of risk posed by applicants. Factor 4: Cost effectiveness The agency will consider the “value for money” in its evaluation of the program. This factor does not consider the cost alone, but rather the cost relative to the value for the work to be performed. Note that the agency’s available funding may not exceed the amount shown in part B. Annexes | Annex I — Evaluation Statement of Work | 131 Page 6 of 9 NOFO Template – Version FY16-01 Review and Selection Process A panel of personnel with expertise in the program will be retained to review applications received, scoring on the criteria shown above, and selecting based on the proposal that the panel feels has the potential to have the greatest contribution to the field. The panel reserves the right to ‘split’ the award if it sees fit, and applicants should be prepared to modify their proposals to accept such a split on short notice. In the interest of transparency, the agency will (upon request) discuss with non￾selected applicants the reasons for their non-selection. Non-selected applicants should be aware that, pursuant to statutory restriction in 7 USC § 3319a, selections cannot be contested. F. Federal Award Administration Information Notice of Award Notice of Selection (which may be conditional) is expected not later than Monday, September 25, 2017. Awards will be issued note later than Thursday, September 28, 2017. Awardees must have the capability to secure the necessary signatures back to the agency by the close of business on September 29. A post￾award discussion will be scheduled to take place in the first week of October at a time of mutual availability. Administrative and National Policy Requirements All successful applicants may be required to comply with Standard Administrative Terms and Conditions, which are available online at: https://www.fas.usda.gov/grants/general_terms_and_conditions/default.asp The applicable Standard Administrative Terms and Conditions will be those in effect for Fiscal Year 2017, unless specifically amended at a later time. Before accepting the award the Recipient should carefully read the award package for instructions on administering the grant award and the terms and conditions associated with responsibilities under Federal Awards. Recipients must accept all conditions in this NOFO as well as any Special Terms and Conditions in the Notice of Award to receive an award under this program. Reporting Federal Financial Reporting Requirements. The Federal Financial Reporting Form (FFR), as known as the SF-425, must be filed within 30 days after the end of each quarter and within 90 days of the end of the agreement, and is available online at: https://apply07.grants.gov/apply/forms/sample/SF425-V1.0.pdf 132 | Annexes  | Annex I — Evaluation Statement of Work Page 7 of 9 NOFO Template – Version FY16-01 Program Performance Reporting Requirements. Performance Progress Reporting must be filed within 30 days after the end of each quarter and within 90 days of the end of the agreement; this is in addition to any agreed-upon deliverables under the scope of the award, and details the activities undertaken and progress made during the reporting period. In generally will not include the technical findings detailed in the deliverable reports. This may be filed using form SF-PPR or in any other format mutually agreeable to the recipient and the agency. Close Out Reporting Requirements. Within 90 days after the end of the period of performance, recipients must submit a final financial report and final progress report detailing all accomplishments and a qualitative summary of the impact of those accomplishments throughout the period of performance. Acceptance of final reports by the agency constitutes a closeout of the award with no further notice or obligation to either party. G. Awarding Agency Contact Information Contact and Resource Information Prospective applicants with questions are encouraged to join the conference call described in section D. The designated agency point of contact for this agreement is: Caitlin Cackoski (Caitlyn.Cackoski@fas.usda.gov) Kaj Goss (Kaj.Gass@fas.usda.gov) Inquiries will be returned within 24 hours. All questions must be received no later than 48 hours prior to submission deadline, except the point of contact will confirm receipt of proposals upon request. H. Additional Information 1. Extensions Extensions to this program will be considered on a case-by-case basis. Requests for extensions must be submitted 60 days prior to the expiration of the performance period with a strong accompanying justification. Requests for extensions are subject to approval by FAS. 2. Pre-Award Costs Pre-Award Costs are not authorized. 3. Budget Revisions a. Transfers of funds between direct cost categories in the approved budget when such cumulative transfers among those direct cost categories exceed ten percent of the total budget approved in this Award require prior written approval by FAS. Annexes | Annex I — Evaluation Statement of Work | 133 Page 8 of 9 NOFO Template – Version FY16-01 b. The Recipient is not authorized at any time to transfer amounts budgeted for direct costs to the indirect costs line item or vice versa. 4. Turnaround Time USDA apologizes for the short turnaround time on this NOFO. This is due to the inconveniently-scheduled end of the fiscal year. 134 | Annexes  | Annex I — Evaluation Statement of Work Page 9 of 9 NOFO Template – Version FY16-01 Appendix: Full Scope The recipient shall conduct an evaluation assessing the ongoing impacts of a minimum of 3 closed Food for Progress Projects: 1 each in Africa, Asia, and Latin America and Caribbean; proposals that account for additional projects are encouraged, provided suitable rigor is maintained. Specific countries/projects will be selected by mutual consent during the first stages of implementation. The evaluation should include the collection of both quantitative and qualitative primary data in each country. Significant interaction with local participants, beneficiaries, and collaborators in the project is expected. This evaluation is not expected to include experimentation or trials on human subjects, as the programs to be evaluated have all concluded. The quality and amount of secondary data available for each project is likely to vary, and to be generally limited. USDA’s Food Aid Monitoring and Evaluation Policy was administered beginning in 2012, and the projects ultimately selected for evaluation will not necessarily have adhered to standard evaluation practices, nor were aware of the possibility of this exercise, so may have limited records still available today. USDA will make all available data available to the recipient. The recipient should submit data collection instruments (i.e. interview guides and questionnaires, observation, and/or field guides, etc.) to USDA for review before finalizing. USDA will be expected to approve or request changes within 10 business days of receipt. Field visits are anticipated to be an important and necessary component of this activity. FAS is particularly interested in answers to the following questions, along with such others that the recipient may propose:  To what extent have activities or practices that were expected to be implemented locally after the project’s end been implemented?  To what extent have benefits of the project continued after the project’s conclusion?  What evidence is there of unanticipated sustained results or effects (whether positive or negative) of the project?  What further activities, or anticipated benefits, do project participants and beneficiaries indicate they will continue to implement or experience in the future?  Looking across projects by region, what are the sustainability lessons learned, best practices, and success stories? A draft of the final report shall be submitted to USDA for comment; USDA will return such draft within 30 business days after receipt. Upon review and comments, further iterations may be necessary before the final document is accepted. The final report shall be submitted electronically in a PDF format. The team is expected to present the final report to USDA staff in Washington, DC. The findings, or a portion of them, may be presented to other professional audiences (including conferences, journal articles, publications, etc.), however costs of doing so generally may not be charged to this award. Annexes | Annex II — Assessment proposal | 135 Annex II — Assessment proposal University of Minnesota Proposal to USDA-FAS-10960-0700-10.-17-0004 1 Food for Progress Post-Project Sustainability Assessment Introduction The University of Minnesota’s Stakman-Borlaug Center (SBC) for Sustainable Plant Health builds on the interdisciplinary, problem-solving vision of University of Minnesota scientists E.C. Stakman and Norman E. Borlaug, to ensure a food-secure future by creating sustainable and interdisciplinary research-, outreach-, and education-based solutions to complex problems that threaten global crop production. Our content area experts have deep agricultural, economic and nutrition expertise having conducted research and development projects in over 16 countries. Our partners at the Improve Group have served as an external evaluator of complex intervention programs for more than 17 years. They bring expertise in evaluation, including identifying and developing the appropriate methods needed to assess project sustainability. Their experience is global having conducted hundreds of multi-site, complex evaluations in over 15 countries and in diverse communities around the world. Combined, the Stakman-Borlaug Center and The Improve Group bring a unique blend of deep topical area expertise and rigor in evaluation. For this project we propose a five-phase approach that includes (1) Inception, (2) Design & Planning, (3) Data Collection, (4) Data Analysis, and (5) Reporting. We look forward to working collaboratively with our partners at USDA/FAS, their colleagues in-country and project participants that have spent years implementing and benefiting from Food For Progress initiatives. Background Food for Progress supports developing countries and emerging markets to improve and strengthen their agricultural sectors. Food for Progress has two main objectives: to improve agricultural productivity and to expand trade of agricultural products. Past FFPr project range from providing training in labor saving technologies and agricultural conservation to rural households in Kenya (US International Food Assistance Report 2006), to improving feedlot practices and strengthening feed distribution channels in Ethiopia (US International Food Assistance Report 2008). Our post-project evaluation will use a five-phase approach that includes (1) Inception, (2) Design & Planning, (3) Data Collection, (4) Data Analysis, and (5) Reporting. The final Sustainability Assessment will determine if and to what extent Food for Progress development activities and recommended practices have been sustained since project close-out, and what practices beneficiaries plan to continue. This evaluation will specifically seek to understand if and to what extent practices promoted by the projects have been sustained over time, and if beneficiaries report continued benefits as a result of the project. Expected project outcomes will be compared to actual outcomes and we will determine if any unanticipated results, positive or negative have resulted. As indicated in the FFPr Learning Agenda, improvements to market functionality can only be sustained if their effects outlive the program, this evaluation will look across projects by region to identify best practices to achieving lasting sustainability of project results and continued utilization of improved practices by beneficiary populations. Technical Approach A Five-Phase Approach A 5-phase evaluation strategy will be used that allows for collaboration at every step will be used in this project. The five phases include (1) Inception, (2) Design & Planning, (3) Data Collection, (4) Data Analysis, and (5) Reporting. 136 | Annexes  | Annex II — Assessment proposal University of Minnesota Proposal to USDA-FAS-10960-0700-10.-17-0004 2 Methodology A mixed-methods approach will be used to gather data from a wide variety of stakeholders in multiple contexts. In the proposed methodology for this project, we will use our research-based1 data-seeking￾data2 methodology to provide broad information about all FFPr sites, supplemental information about many sites, and in-depth impact information about 3 strategically selected sites. We will do this by reviewing a wide variety of project reports, end line evaluations and results frameworks to determine which three projects should undergo a deep-dive sustainability assessment. By using this methodology USDA/FAS will get high level results information from many projects and an actionable sustainability assessment for 3 FFPr projects. The main actors that will conduct this assessment are the UMN PI and Collaborative Research Specialist, we propose to contract some evaluation services to the Improve Group whose analysts will collect data with input from UMN content area experts. Together this group is referred to as a “working group”. We have outlined the proposed activities for this project within the five phases below. Phase 1 – Project Inception During this phase, the guiding frameworks for the evaluation are established; we refine evaluation questions and priorities, identify sources of data and methods, and determine what is and isn’t in scope for the evaluation. During this phase: • We will facilitate an inception meeting with USDA/FAS to clarify and refine the scope of work, understand roles and responsibilities, establish key contacts, and confirm a communication plan and schedule. We will identify and request background information, and articulate stakeholders that should be involved to guide the remainder of the evaluation. • Following the inception meeting, we will host a retreat for the evaluation. In this half-day retreat, USDA/FAS, stakeholders, and the working group will collaborate to define evaluation roles, goals, and next steps. Although we will finalize details during the inception meeting, participants at the retreat should include all decision makers and influencers involved in the planned evaluations, including key advisors from sites, academic content experts, , those with insight on how the evaluation will be used (e.g., policy makers), etc. • During this phase, we will also draft, get feedback on, and finalize a FAS logic model that describes goals, activities, inputs, outputs, and outcomes. We will gather feedback from key stakeholders and our USDA/FAS liaison, and refine the logic model accordingly. • Once the retreat is complete, we will review existing information (such as prior evaluation reports) to understand what existing data can be used in the evaluation. • Following the retreat and review of data, we will develop a more detailed evaluation approach and workplan, including a more specific timeline of activities. 1 Cohen D, Crabtree B. "Qualitative Research Guidelines Project." July 2006. http://www.qualres.org/HomeIter￾3827.html; Srivastava, P., & Hopwood, N. (2009). A Practical Iterative Framework for Qualitative Data Analysis. International Journal of Qualitative Methods, 8(1), 76-84. doi:10.1177/160940690900800107; Beebe, J. (2005). Rapid Assessment Process. Encyclopedia of Social Measurement, 285-291. doi:10.1016/b0-12-369398-5/00562-4; Wilson￾Grau, R., & Britt, H. (2012). Outcome Harvesting. Retrieved September 8, 2017, from https://usaidlearninglab.org/sites/default/files/resource/files/Outome%20Harvesting%20Brief%20FINAL%202012- 05-2-1.pdf.; Brinkerhoff, R. O. (2003). The Success Case Method: Finding Out What Works. San Francisco, CA: Berrett￾Koehler , Inc. 2 Johnson, S., & Connell, C. (n.d.). Data Seeking Data. Retrieved September 8, 2017, from http://comm.eval.org/HigherLogic/System/DownloadDocumentFile.ashx?DocumentFileKey=e2c3965d-f9b0-4689- 8a7b-a983cde5b9dc&forceDialog=0 Annexes | Annex II — Assessment proposal | 137 University of Minnesota Proposal to USDA-FAS-10960-0700-10.-17-0004 3 Phase 2 – Design & Planning In this phase, we will plan and prepare all of the processes and materials to be used in the evaluation. Using insights from the working group and USDA/FAS, we will take the initial evaluation ideas and turn them into evaluation tools, agreements, and plans of action. These tools will be shared with USDA/FAS and adjusted based on their feedback. The evaluation approach will use mixed-methods to tell an aggregate story about the overall impact and sustainability of the selected programs. • Work with USDA/FAS (and other stakeholders as appropriate) to use the evaluation priorities established in Phase 1 to determine data sources and methods, criteria and factors that will be analyzed and recorded in the rest of the evaluation. • We will prepare and share with USDA/FAS any data collection tools that are needed. Although we will refine our plans during the inception phase, we anticipate that these tools will include an evaluative rubric3, interview protocols, and multi-step site visit protocols. Some aspects of the tools – for example specific agricultural information to gather onsite – will be left open for further development during our data-seeking-data evaluation process (see below). We will get input from USDA/FAS and other local partners to determine factors such as the best ways to phrase questions, timing of data collection, and roles for sites and our evaluation team in collecting data. We will then finalize the tools using the input provided. • Prepare data sharing agreements, if needed; these could be agreements between USDA, USAID, implementing partners, and others to share data for the evaluation and secure it in ways that meet everyone’s needs. • During this phase, we will also prepare the evaluation plan. This will include timelines, activities, methods and tools, data security procedures, and other needed content. Much of the evaluation plan content will also be used in evaluation resource guides that will be available online in real time for staff so that they can easily reference them when needed. For example, for a recent project, we created brief videos to assist providers as they completed a detailed, online data packet used in the evaluation. Phase 3 – Data Collection During the data collection phase, we will use the tools and methodologies defined in the evaluation plan. We propose a three-part data-seeking-data data collection phase: STEP 1: We will gather existing data and information about all of previously funded FFPr projects. This will be data that already exists and that USDA already has, such as evaluation reports from sites. We will use the evaluative rubric to organize the data by site, and analyze the data against criteria decided in the design phase. At the conclusion of this phase, each of the sites will have clear classifications. We will prepare a report for USDA/FAS at the end of this phase that highlights each of the projects’ focus and successes. This will enable a conversation with USDA/FAS about whether interviews in Step 2 need to be targeted in any way, such as on sites where we have gaps in information, or on sites that are classified in a way that makes them a higher priority. STEP 2: We will conduct phone/web interviews with as many implementing partners as possible. During the phone/web interviews, we will ask questions that allow us to reflect with key liaisons about the classifications we made on the rubric, and gather additional data about: ▪ Process information – what happened during the project 3 Davidson, J., Wehipeihana, N., McKegg, K., & Rogers, P. (n.d.). The Rubric Revolution. Retrieved September 8, 2017, from http://www.betterevaluation.org/sites/default/files/AES-2011-Rubric-Revolution-Davidson-Wehipeihana￾McKegg-xx.pdf 138 | Annexes  | Annex II — Assessment proposal University of Minnesota Proposal to USDA-FAS-10960-0700-10.-17-0004 4 ▪ Outcome information – what outcomes happened that aren’t easily identifiable in documented data, or verify outcomes that have been reported ▪ Impact information – determine how communities or environments have changed as a result of the project ▪ Additional data – what additional documentation is available to help us understand process, outcomes and impact Data shared during the interviews and any additional documentation or data provided by implementing will be cleaned, organized, and analyzed using the evaluative rubric. This will allow us to provide additional classifications about sites. At the conclusion of Step 2, we will provide a summary report to USDA/FAS and facilitate a collaborative meeting to use the information to select three sites for in-depth field visits. This summary report will be formatted in a way that it could be shared with other stakeholders as appropriate. STEP 3: Using the analysis completed in the first two steps, we will select sites for field visits where we will collect in-depth process, outcome and impact data about three sites where programs were implemented, one from each region. During these field visits, we will blend: ▪ Agricultural research methods to determine if production gains have been sustained over time, if value chain actors are still able to access resources the intervention made available and other key questions that will be determined in phase 2. ▪ Evaluative data collection4: Surveys, interviews, focus groups, or participatory methods will be used to gather data from beneficiaries, community members, community leaders, and key experts. The input we receive during the design phase will help guide how we are using local enumerators, interpreters and experts; we have estimated costs for working with these partners in our budget. We have found that collaborating with local staff and consultants helps ensure that we are understanding a fuller picture of results; they bring expertise about local context, factors that affected the level of success experienced at each site, and how to interpret data we are seeing. ▪ We will select interview and survey participants using a purposeful sampling5 methodology to assure broad representation of beneficiary group and to avoid bias in reporting. We will use this method as opposed to random sampling because it ensures that people providing data actually have insights about how the FFPr was implemented and its ongoing sustainability. ▪ Before each field visit is concluded, we will facilitate a local interpretation workshop. At this workshop, we will present preliminary findings and engage local leaders in helping us to validate findings, understand local factors, and interpret results.6 4 Northwest Center for Public Health Practice. (n.d.). Data Collection for Program Evaluation. Retrieved September 8, 2017, from https://www.nwcphp.org/docs/data_collection/data_collection_toolkit.pdf; Zukoski, A., & Luluquisen, M. (2002, April). Participatory Evaluation. Retrieved September 8, 2017, from https://depts.washington.edu/ccph/pdf_files/Evaluation.pdf; Guijt, I. (2014). Participatory Approaches, Methodological Briefs: Impact Evaluation 5, UNICEF Office of Research, Florence. 5 Patton, M. Q. (2015, February 23). QUAL Eval Week: Michael Quinn Patton on Purposeful Qualitative Sampling. Retrieved September 11, 2017, from http://aea365.org/blog/qual-eval-week-michael-quinn-patton-on-purposeful￾qualitative-sampling/ 6 Gerstl-Pepin, C. I., & Gunzenhauser, M. G. (2002). Collaborative team ethnography and the paradoxes of interpretation. International Journal of Qualitative Studies in Education, 15(2), 137-154. doi:10.1080/09518390110111884 Annexes | Annex II — Assessment proposal | 139 University of Minnesota Proposal to USDA-FAS-10960-0700-10.-17-0004 5 Phase 4 – Analysis Analysis will occur throughout the project using the evaluative rubric to classify and interpret data. After field visits have been concluded, we will do a further, in-depth analysis of all available data using qualitative and quantitative analysis. Quantitative data will be possible because we will have gathered data from all sites and may gather quantitative data from surveys from field visit sites. Analysis strategies will include: • Analysis of variance to determine how similar sites are across years, focus and region. (for example, if sites funded starting in Year 1 of the program differ significantly from those starting in Year 2) • Descriptive statistics. These will include frequencies, percentages, means and standard deviations to describe outcomes across all participants. In addition, we will conduct cross tabs with significance tests to compare results across focus areas (i.e., industry development, agricultural practices, animal health, etc.), implementing partners, and demographic characteristics (i.e., focus on gender, marginalized groups, etc.) • Regression. To estimate the impacts of the USDA/FAS efforts, we will use regression analysis, which allows us to control for specific variables. Pearson’s correlations and ordinary least squares will be used to assess impact for continuous variables (such as income levels). For categorical variables, such as gender and program completion, we will use logistic regression and cross tabs. Spearman correlations applied to understand the relationship between both categorical and continuous variables. Multiple regression will be used to determine which variables are significant relevant to various outcomes, such as population focus, project focus, etc. Qualitative data will come from interviews, documents, and local field-visit workshops. Qualitative data will be analyzed using a common approach called Constant Comparison/Grounded Theory. This process uses four steps: (1) prepare the data by organizing items as responses to the key questions asked, usually in an Excel spreadsheet or a Word document; (2) review a sample of the data (e.g., notes from 3 of 15 interviews) and start generating a list of themes to watch for in the full dataset; (3) review all of the data and use color coding or hatch marks to note where the data align with themes identified in Step 2, and note what additional themes emerge. When Step 3 is complete, all of the data are assigned to one or more theme(s); (4) determine if any of the themes can be aggregated into a master theme or finding. A software called Dedoose will help track themes and analysis. As we analyze the data, we will synthesize and assess what we’ve learned from our data sources, with a keen eye toward using the data to inform decisions and future work for the program. We will use qualitative and quantitative analysis methods to identify and categorize themes so we can identify findings that are: • Prevalent: Common across multiple programs and respondents; • Insightful: unique to a few respondents, but share a perspective that is important to your work; • Illustrative: in-depth comments that do a particularly good job of describing a concept or idea. As analysis is completed, we will facilitate an emerging findings meeting with USDA/FAS and other stakeholders as appropriate. At this meeting, we will present results and guide a discussion to help uncover important issues and implications for further analysis, recommendations, and policy responses. Following the emerging findings meeting, we will complete our analysis, including addressing any questions that emerged during the meeting. 140 | Annexes  | Annex II — Assessment proposal University of Minnesota Proposal to USDA-FAS-10960-0700-10.-17-0004 6 Phase 5 – Reporting In this phase, we will draft and then finalize the required reports.7 Report formats will be defined in the early phases of the evaluation to ensure they effectively tell the story of USDA/FAS programs, outcomes and key evaluation findings to target audiences. This report will identify evidence of sustainability of project results, activities and practices, identify unanticipated results, and provide recommendation on what implementers could have done to increase sustainability. Results will be presented in a way that are easy for multiple stakeholders to interpret. The final report will also provide actionable recommendations that will help guide future projects working in the same geographies and on the same value chain. We also propose to create a report that is specifically tailored to policy makers and the general public- this will highlight our results and key messages that these audiences will find impactful. Timeline and Deliverables*: Activity Deliverable Due date By whom Phase 1 Inception meeting with USDA/FAS Scope of work 10/31/17 SBC, IG and USDA Retreat with working group members, USDA/FAS and stakholders Defined evaluation roles, goals, and next steps 11/15/17 SBC, IG and USDA Establish workplan and specific timeline Workplan and timeline 12/15/17 SBC and IG Phase 2 Design data collection tools Tools shared with USDA/FAS 04/01/18 SBC and IG Create evaluation plan Plan 04/01/18 SBC and IG Phase 3 Gather secondary data from USDA/FAS and implementing partners on all past FFPr project Summary report 05/01/18 SBC and IG Select sites for in depth sustainability assessment 3 sites selected 07/1/18 SBC and IG Field visits to FFPr sites Trip reports, field notes, summary learning 06/30/19 SBC and IG Phase 4 Data Analysis Data summaries 07/01/19 SBC and IG Phase 5 Draft report Report sent to USDA for review 8/10/19 SBC and IG Final Sustainability Assessment Final Evaluation Report send to USDA 9/30/19 SBC and IG *This timeline is flexible depending on the time and availability of USDA/FAS staff, seasonal and country level time restraints and access to reports and secondary data. Our team is comfortable working in highly fluid environments and will adapt as necessary. 7 Evergreen, S. D. (2018). Presenting data effectively: communicating your findings for maximum impact. Los Angeles: SAGE. Annexes | Annex II — Assessment proposal | 141 University of Minnesota Proposal to USDA-FAS-10960-0700-10.-17-0004 7 Experience and Expertise: University of Minnesota’s Stakman-Borlaug Center for Sustainable Plant Health The University of Minnesota’s Stakman-Borlaug Center (SBC) for Sustainable Plant Health builds on the interdisciplinary, problem-solving vision of University of Minnesota scientists E.C. Stakman and Norman E. Borlaug, to ensure a food-secure future by creating sustainable and interdisciplinary research-, outreach-, and education-based solutions to complex problems that threaten global crop production. SBC maintains collaborative research relationships with 13 disciplines (UMN departments); 6 UMN Colleges (College of Biological Sciences, College of Food Agriculture and Natural Resource Sciences, College of Design, College of Liberal Arts, College of Science and Engineering, and Extension); UMN Libraries, UMN Bee Squad, Minnesota Supercomputing Institute, and the UMN International Science & Technology Practice & Policy Center; 17 external universities; 10 public and nonprofit organizations; and 8 agribusinesses. Through this network the SBC unites diverse expertise to find durable, environmentally-friendly solutions to food security challenges. We lead a wide variety of international projects and have worked closely with partners at USAID, USDA/FAS and the Gates Foundation. The University of Minnesota College of Food, Agricultural and Natural Resource Sciences (CFANS) is a comprehensive school integrating animal- and plant-based agriculture, environmental sciences, and related social sciences to address the grand challenge of feeding a growing world population sustainably. CFANS comprises 12 departments (Agricultural Education, Communication & Marketing; Agronomy & Plant Genetics; Animal Science; Applied Economics; Bioproducts & Biosystems Engineering; Entomology; Fisheries, Wildlife & Conservation Biology; Food Science & Nutrition; Forest Resources; Horticultural Science; Plant Pathology; Soil, Water & Climate) and approximately 250 faculty and 800 staff. CFANS is home to the Bell Museum of Natural History; Minnesota Aquatic Invasive Species Research Center; Minnesota Invasive Terrestrial Plants and Pests Center; Minnesota Landscape Arboretum; the Stakman￾Borlaug Center (SBC) for Sustainable Plant Health, and the University of Minnesota Water Resources Center. CFANS maintains close collaborative ties with other relevant University of Minnesota entities including College of Biological Sciences; College of Veterinary Medicine; Extension; and the Institute on the Environment. CFANS scientists are highly productive and competitive and CFANS maintains an externally￾funded research portfolio of more than $58 million. CFANS scientists have significant global reach with research projects taking place in at least 48 countries and on all 7 continents. The 26,000 living CFANS alumni provide a global network of collaborations and resources for CFANS scientists. Among many productive alumni, CFANS trained Norman Borlaug, Father of the Green Revolution (B.S., M.S., Ph.D.). Today, CFANS offers a wide range of undergraduate, masters, and doctoral degrees and prides itself on providing experiential learning opportunities, including study abroad programing. The Improve Group The Improve Group is a woman-owned, full-service evaluation, research, and strategic planning consulting firm based in Saint Paul, Minnesota. They are committed to helping local, national, and international organizations make the most of information, navigate complexity, and ensure their investments lead to meaningful, sustained impact. That commitment is connected to their vision, that mission-driven organizations will effectively develop more resilient, equitable, and thriving communities. The Improve Group has served as an external evaluator of complex intervention programs for more than 17 years. In that time, they have worked with hundreds of local and state governments, nonprofits, foundations, and businesses—and provided them with crucial and credible information to inform strategic decisions, improve program design and implementation, and deepen their work toward healthy and just communities. 142 | Annexes  | Annex II — Assessment proposal University of Minnesota Proposal to USDA-FAS-10960-0700-10.-17-0004 8 They leverage the power of a project team with exceptional research rigor; of innovative evaluation techniques that explore and explain program outcomes; and an approach that is culturally competent and designed specifically to meet the needs of the most diverse communities. The Improve Group’s experience is global—they have conducted hundreds of multi-site, complex evaluations in over 15 countries and in diverse communities around the world. In addition to their knowledge of a broad range of evaluation methodologies, their team of 17 highly￾qualified evaluators bring an unsurpassed level of cultural competence to their work. In their international projects, including this one, they partner with local evaluators to conduct the highest-quality field research in remote communities. In nearly every project, they design an innovative, mixed-methods evaluation that draws on multiple sources of qualitative and quantitative data to reveal and validate the overall impacts of an initiative. And using their community-responsive approach, they create highly participatory tools and methods to ensure the voices of all community stakeholders are heard and honored. Staffing Plan: This project will be led by PI James Bradeen who will serve as the Project Director for this Post Project Sustainability Assessment. He is a Professor and the Department Head of Plant Pathology at the University of Minnesota and Director of the Stakman-Borlaug Center (SBC), where he oversees an international research portfolio with funding from diverse sources including USDA, CIMMYT, and the Bill and Melinda Gates Foundation. As PI Dr. Bradeen will oversee the technical direction and financial management of the project, assuring that deliverables are on time and within budget, and that the work aligns with all USDA and University of Minnesota policies. Dr. Bradeen will oversee the project to make sure the work is conducted with the highest analytical and ethical rigor. Britta Hansen, the Collaborative Research Specialist for the SBC will lead the day to day coordination, management and implementation of this project. She will be the main point of contact for USDA/FAS and our partners at the Improve Group to contribute to the design and development of work plans and provide technical, financial and administrative support to faculty collaborators, consultants and teams in the field. Ms. Hansen will compile reports from the Improve Group and develop and deliver the final Post Project Sustainability Assessment to USDA/FAS. Additional key personnel on this project include leading faculty in the fields of applied economics, plant pathology, food science and nutrition, veterinary medicine, animal science, integrated pest management, development practice and public policy. Our faculty have a breadth of geographic expertise from across Africa, Latin America and Asia. Specifically, our faculty and partners have worked in Ethiopia, Kenya, Niger, Uganda, Zambia, Ghana, Liberia, Honduras, Guatemala, Bolivia, Nepal, India Bangladesh, India, Indonesia, and Cambodia. A subset of these experts will be selected depending on the specific needs of the projects to be evaluated. Our proposed evaluation analysts from the Improve Group are: Sara Stalland McGarraugh, Kayla Meyers, and Claire Stoscheck who will conduct data collection and analysis in partnership with the Stakman-Borlaug Center. Supporting and overseeing them will be Chief Practice Officer Rebecca Stewart, who ensures excellence and innovation of their work. Additionally, we will determine through the early phases of the project where and how to tap into local expertise for this evaluation. Experience has shown that partnership with local leaders, elders and subject￾matter experts provide invaluable context for producing and interpreting results. Both the SBC and the Improve Group have wide networks in many of the countries where FFPr has worked; we will tap into these and other local content and evaluation experts to support the sustainability assessment at the local level. Annexes | Annex II — Assessment proposal | 143 University of Minnesota Proposal to USDA-FAS-10960-0700-10.-17-0004 9 The Stakman-Borlaug Center Staffing: Principal Investigator and Project Director, James Bradeen is Professor and Department Head of Plant Pathology at the University of Minnesota and the Director for the Stakman-Borlaug Center (SBC), where he manages an international agriculture research portfolio with funding coming from various external funding agencies including USDA, CIMMYT, and the Bill and Melinda Gates Foundation. Professor Bradeen has more than 25 years of research experience in plant genomics, plant health, and food security. He has conducted research in Bangladesh, India, Indonesia, Mexico, and Italy. Professor Bradeen is a skilled administrator experienced in the management of large, complex, multi-stakholder projects. Collaborative Research Specialist Britta Hansen has over ten years of experience in international agriculture development. Ms. Hansen manages the SBC’s international projects, provides direction to the expanding portfolio of global programs, and will be a key manager of the FFPr Post Project Sustainability Assessment. Her background is in the design and implementation of global food security and rural development projects that sustainably improve agricultural production and the incomes of participants. She is fluent in Spanish and Portuguese, having lived in Central and South America for many years. She has led a variety of nutrition and gender sensitive projects and will incorporate a gender sensitive lens to this project evaluation. Prior relevant experiences include her role as a Program Officer with the Feed the Future Innovation Lab for Collaborative Research on Horticulture at UC Davis where she managed over $2 million in research sub awards in the areas of nutrition, gender, technology development and horticulture production. Ms. Hansen’s graduate work centered around evaluating the sustainability of a large-scale water harvesting for agriculture project in the remote Bolivian Andes. The Improve Group Staffing: Sara Stalland McGarraugh, M.P.P., has a wealth of international experience in quantitative and qualitative analytic approaches to evaluation, efficient data analysis, and reporting findings to support client needs and use. Her skills include French fluency, cross-cultural communication, conducting interviews, and creating accessible visual representation of data. Her research and project experience include asset building in agro￾pastoral developing economies, and economic, community, workforce, and youth development. Ms. McGarraugh served as a key evaluation lead on two international agricultural projects highlighted below— the MILK evaluations (I and II) in Niger and our current work with USAID-Senegal and CTI in Senegal. Kayla Meyers, M.D.P., assists her team in accounting for stakeholder needs and concerns in instrument design and analysis. Her work domestically and abroad developed her ability to conduct research and evaluation projects with strong qualitative data collection and analysis capacities. During her time in graduate school, Ms. Meyers focused on economic and social cost benefit analysis, as well as other quantitative data analysis methods, in order to convey stories and images about social programs. Ms. Meyers has worked in Costa Rica and Ecuador. Claire Stoscheck, M.P.P., has served as a leader in multiple nonprofits and managed several programs prior to her work as a researcher and evaluator. She brings deep expertise and experience in food systems to this project: During her graduate program, she conducted extensive research in collaboration with community stakeholders focused on a food procurement tool to make local food system more equitable and sustainable. Ms. Stoscheck also worked at University of Minnesota Extension, where she supported applied research, semi-structured interviews, case study report writing, and graphic design. Rebecca Stewart, M.P.P., provides leadership for scores of projects as The Improve Group’s Chief Practice Officer. She oversees all evaluation, research, planning, and capacity building practices with clients and their stakeholders and understands the unique needs of the organizations she works with. This allows her to 144 | Annexes  | Annex II — Assessment proposal University of Minnesota Proposal to USDA-FAS-10960-0700-10.-17-0004 10 ensure that every project is delivered on time with the highest quality. A consummate generalist, Ms. Stewart has led projects domestically and internationally, in diverse topic areas such as youth leadership, higher education, corrections, and housing Content Area Experts: We will draw on UMN faculty to provide deep technical expertise that aligns with the scope of the final three projects that are selected for in-depth evaluation and sustainability assessment. Our content area experts work in such relevant areas as food policy and agricultural value chains, evaluation of cropping systems, organic horticulture production, health and disease at the human-livestock-wildlife interface, food science and industrial grain processing, the diseases of small-grains, IPM in Africa, global extension, value chain analysis of non-timber forest products and the history and theory of International Development Practice. In addition to these faculty we will draw on students at the masters and doctoral levels to help conduct field work and review and analyze data. Technical experts who have already agreed to be part of this project include: Dr. Marc Bellemare: Associate Professor, UMN Department of Applied Economics . Dr. Bellemare directs the Center for International Food and Agriculture Policy. His research lies at the intersection of agricultural economics, food policy, and international development. A few specific areas in which Dr. Bellemare conducts research include contract farming and agricultural value chains, food prices, and market access. He has conducted research in Peru where he worked with Innovations for Poverty Action and in Madagascar with funds from USAID, the John D. and Catherine T. MacArthur Foundation and NSF. Dr. George Annor: Assistant Professor of Cereal Chemistry and Technology, UMN Department of Food Science and Nutrition. He is also the chair for the General Mills Endowed Professorship in Cereal Chemistry and Technology. George hails from Ghana and has done extensive work there where he was a professor of Food Science and Nutrition at University of Ghana, Legon-Accra. The Department of Horticulture consists of 23 leading experts in improving productivity, value and use of horticultural crops in both Minnesota and around the world, including faculty such as Associate Professor Julie Grossman whose lab explores the ways in which we can better manage plant-soil-microbe relationships in organic systems in order to enhance soil fertility. Dr. Grossman has conducted her research in Central and South America. Dr. Elton Mykeresi: Associate Professor, UMN Department of Applied Economics. His research interests include the study of human capital, causes of poverty, food insecurity and poor nutrition. He has published relevant research such as “Ex-Ante Evaluation of Alternative Strategies to Increase the Stability of Cropping Systems in Eastern and Central Africa” in the Journal of Agricultural Economics. Dr. Jason Kerwin is a development economist based in the Department of Applied Economics at the University of Minnesota, where he is an assistant professor. His research focuses on understanding the choices people in sub-Saharan Africa make about their health, their education, and their employment. He has extensive experience in conducting cross-country and cross-disciplinary research in general, and in running randomized controlled trials in particular. Dr. Bill Hutchison: Professor and Extension Entomologist, UMN Department of Entomology. His research uses a systems approach to Integrated Pest Management (IPM) that support sustainable vegetable and fruit production. In addition to his extension/research responsibilities in Minnesota, he is active in several international activities, including UMN Harvest Choice project. Dr. Hutchison is a Co-PI at the USAID funded Annexes | Annex II — Assessment proposal | 145 University of Minnesota Proposal to USDA-FAS-10960-0700-10.-17-0004 11 IPM Innovation Lab where their team increases production and productivity along the maize, rice, and chickpea value chains by reducing crop losses through dissemination of effective IPM options. He has also been called upon to tackle the recent and devastating spread of Fall Army Worm in Africa. Dr. Katey Pelican: Associate Professor and Head of the Ecosystem Health Division, UMN College of Veterinary Medicine. She is also the Deputy Director and Principal Investigator of the One Health Workforce Project, part of the USAID Emerging Pandemic Threats Program 2. This project is working in 14 countries and has a budget ceiling of $63 million dollars. Dr. Dean Current: Director of the Center for UMN Integrated Natural Resources and Agriculture Management. He works locally and internationally on community-based natural resource management/community forestry, economic and environmental impact assessments. He supervises Master in Development Practice students and leads research projects in Vietnam, Guatemala, Nepal and Ecuador. Dr. David Wilsey: Program Director for the Masters in Development Practice program at the UMN Humphrey School of Public Affairs. His research and practice focus on development of integrated natural resource conservation and livelihood programs, generally focusing on forest- and farm-based livelihood systems. Dr. Ruth Dill-Macky: Professor, UMN Department of Plant Pathology. Her applied research program focuses on economically important diseases of small-grained cereals. She works in close collaboration with wheat, barley and oat breeders, studies plant disease biology, and leads in-country research training programs in Ethiopia. Dr. Matt Rouse: Adjunct Associate Professor, UMN Department of Plant Pathology and Researcher at the USDA/ARS Cereal Disease Lab. Rouse works extensively internationally with significant emphasis on eastern Africa. Past Performance: As we reviewed this project, we carefully considered the variety of skills needed to conduct a successful evaluation. The following are just a sampling of projects that help illustrate our deep experience, knowledge, methods, technical skills and rigor relevant to this project. The Stakman Borlaug Center has a history of designing and implementing a wide variety of international projects dedicated to reducing poverty and improving global food security. In 2015 the center began to ramp up international projects and currently manages a global portfolio of over $1.3 million. Current projects at the SBC include Minimizing a Significant Threat to Global Maize Production- Mitigating the impacts of Maize Lethal Necrosis on food security and human health in Kenya and other East African nations by changing farmer practices to reduce the risk of crop infection. This project is led by CIMMYT with funding from the Bill and Melinda Gates Foundation and USAID. Wheat Rust Surveillance in Ethiopia- This project is enabling Ethiopia to provide more of its farmers with seed of high yielding, rust resistant varieties, sooner and in greater quantities. This project is led by CIMMYT with funds from USAID. Capacity Building for Mitigating Wheat Diseases in Ethiopia- Designing and implementing in-country training programs for public researchers to enhance research capacity for mitigating and managing important diseases of small grains. 146 | Annexes  | Annex II — Assessment proposal University of Minnesota Proposal to USDA-FAS-10960-0700-10.-17-0004 12 Feed the Future South Asia Eggplant Improvement Partnership- Facilitating the late-stage development, deregulation, commercialization and disseminations of Bt eggplant in Bangladesh and the Philippines. This project is led by Cornell University with funds from USAID. Training USDA/FAS Cochran Fellows on phytosanitary system requirements based on U.S. and international standards, providing insights on managing plant products and plant protection products in their home country and the potential for export of their products to the U.S. The Improve Group’s relevant projects include: Compatible Technologies International (CTI): We are currently partnering with CTI on a four-year project in Senegal to evaluate the introduction, distribution and use of a new CTI millet thresher into the local agricultural economy. Funded by USAID-Senegal, the purpose of this supply chain project is to develop the local manufacturing market for the technology and build farmer capacity to efficiently process millet to get higher quality and greater amounts into the market. Mercy Corps: We have partnered with Mercy Corps a number of times to evaluate complex development efforts related to the effects of climate change and food insecurity. For example, we completed an impact evaluation of the MILK Program (Market Innovations and Linkages between the Ranch of Kirkissoye and Urban Niamey), which supports dairy cooperatives to increase the availability of fresh milk while also providing financial support to entrepreneurs to foster economic growth. We believe the MILK II evaluation has particular relevance to the evaluation that the USDA is now embarking on. Our team returned to the region a year and a half after completing the impact evaluation to do a post-program sustainability/resiliency study. In the interim, the region in Niger where the MILK program had been implemented experienced a drought that caused a period of food insecurity. The Improve Group returned to reassess the program to see if it had sustained beneficiaries of the program’s (local dairy farmers, female entrepreneurs and cash-for-work recipients) ability to maintain their resilience through periods of food insecurity. Because the program had ended several years prior, our team employed our Image Grouping© method, which helps respondents recall and reflect on events from an earlier time period, in order to gather quantitative and qualitative data from the local participants. Minnesota Department of Agriculture: The Improve Group is currently working with the Minnesota Department of Agriculture on two different projects. One is to gather input from farmers, cooperatives, distributors and manufacturers to inform the implementation of a complex federal produce safety rule that will impact tens of thousands of produce growers throughout the state; and one is to help implement a new set of standard operating procedures for feeding livestock. Other related projects and clients include: • Rural Information Technology Alliance (RITA): • Center for IDEA Fiscal Reporting (CIFR) Initiative • Physicians for Peace • Oxfam America Annexes | Annex III — Round 1 Site selection criteria | 147 Annex III — Round 1 Site selection criteria Site Selection Criteria for UMN Sustainability Evaluation of FFPr Projects Criteria Weighting 1 Exclude projects awarded before FY2005 or after FY2012 Prior to 2005, FAS did not keep detailed records of project performance, and projects awarded after FY12 have not been closed for a sufficient amount of time to meaningfully determine the extent of sustainability of activities/benefits. High; deleted 2 Exclude projects that will have been closed for less than 1 year by December 2017 After the initial cut described above, there still may be some longer projects that were funded in FY12 or before but haven’t been closed for a long enough time to demonstrate signs of long-term sustainability. The Bangladesh sustainability study done in 2016 looked at a project closed for only 1 year, and USDA has an interest in studying projects that have been closed for a longer period of time. If the project has been closed for 1 year or more by Dec 2017, it will have been closed for more than 1.5 years by the time of projected data collection. High; marked but not currently deleted 3 Exclude Government-to-Government agreements [Code: G2G] FFPr has less control over the design and implementation of G2G awards, and Governments are not required to submit detailed performance reports. High; deleted 4 Exclude projects in Bangladesh USDA already did a sustainability study on a closed project in Bangladesh in 2016; prefer to diversify our learning by focusing elsewhere. High; deleted 5 Exclude projects that focused on types of activities that FFPr no longer funds [Code: Relevance] • FFPr no longer funds research, HIV/AIDS, or humanitarian activities, and focusing on these projects would limit the usability of the evaluation’s findings. • 6 projects focusing on school feeding, humanitarian relief, and/ or HIV/AIDS have already been excluded from the list. Another cut at additional, currently irrelevant activities can be done. • Choose project that were closely related to what FFPr is doing now. That would be the most helpful. • Look to exclude rice etc… High; some deleted Medium; more exclusions might apply 148 | Annexes  | Annex III — Round 1 Site selection criteria 6 Exclude projects considered unsuccessful at closing [Code: Unsuccessful] • The purpose of the evaluation is to learn more about which types of project activities/benefits are sustainable, and to what extent. Unsuccessful projects are a priori not sustainable, and the lessons learned from those projects should already have been documented in performance evaluations. • How to identify what they are… but do exclude the ones that we know are 100% unsuccessful. Medium (should exclude) 7 Exclude projects that were followed by a continuation project [Code: Continuation] If a project were followed by a continuation project, it would be very difficult to attribute sustainable benefits to the original project versus the continuation project. Low (could include only if project is otherwise a good fit) 8 Exclude projects awarded to inactive FFPr recipients [Code: Recipient] • The key audience for this evaluation includes FFPr implementing partners, and evaluating awards to recipients that have not implemented an FFPr project recently may limit the utilization of findings. Choosing closed projects that were implemented by current or recent partners also has the advantage of potentially encouraging the partner to provide their own documentation on the closed project, which may be extremely helpful and more extensive than USDA’s documentation. • The initial categorization in the site selection spreadsheet uses “current” recipient is actively working with FFPr now on another project), “recent” (recipient had a project within last 2 years, but doesn’t have one now), and “past” (recipient hasn’t implemented a project with FFPr in more than 2 years) as a start; can be refined Low (could include only if project is otherwise a good fit) 9 Exclude projects in areas where DOS has an active Travel Warning [Code: Security] While an evaluation clearly can’t succeed if international or local evaluation staff cannot safely reach activity sites, this will vary country to country and may not align with DOS’ warnings. Should be determined with evaluator; may be a factor in narrowing final list. Low Annexes | Annex III — Round 1 Site selection criteria | 149 10 Exclude projects with too little documentation available This criteria can be applied once a “short list” of potential projects is established. FFPr can identify what documentation is available for each potential closed project at USDA, and having very little or no documentation available may be a reason to exclude a project from consideration. Low (could include only if project is otherwise a good fit) 11 Geographic criteria Any others? Does the sub- still exist? Are they current awardees? Having current awardees would make getting old info much easier… they are more willing to work with us. Inclusion criteria? Countries, could be three but not necessarily. Financial criteria (cost/size of project) – anything above 10- 15 over 3 or 5 years. Project duration – continuation vs stand alone? Beneficiary #s/ size and scale of beneficiaries. Expertise of partners/evaluators. 150 | Annexes  | Annex IV — Round 2 Site selection criteria Annex IV — Round 2 Site selection criteria Priority criteria for selecting 7 finalist Agreements Evaluation Question Criteria Rationale for inclusion 1 — What existing conditions are needed for sustainability? IP had previous work in the geographic area Our hypothesis is that without having pre￾existing knowledge of the area or region, an implementing partner would be unable to set up structures, partnerships, and activities that were responsive to local culture and need. This would in turn increase the likelihood of unsustained project outcomes. Scored on a scale of 1-4. 2 — What factors of the design lead to sustainability? Quality of partners Our hypothesis is that if a project had good partners who were heavily involved in design and implementation of activities, then the impacts will be more likely to last and be valued. Partner involvement may also serve as a proxy for cultural competency in a region. Scored on a scale of 1-4. 2 — What factors of the design lead to sustainability? IP uses existing country/regional/ local plan that the activities are built upon Our hypothesis is that if a project used existing local plans from which to design their activities, they would be attending to greater buy-in from the community, achieving more cultural competency, and responding to community/country￾specified need. All of which would serve to promote longer term impacts and sustainability. Scored on a scale of 1-4. 5b — How did beneficiaries respond to subsequent shocks? Is that response attributable (for good or bad) to the project? Did a shock occur post project? Shocks have a dramatic effect on whether or not activities or impact is sustained from a project. In particular, USDA has an interest in supporting projects that are able to build participant resiliency in the face of the increasing climate related shocks. Scored on a scale of 1-4. Description of 3 sites Annexes | Annex IV — Round 2 Site selection criteria | 151 Site 1 This site is one that is based on factors that theoretically support sustainability, but have also experienced a significant climate related shock post-project. The criteria for inclusion in this site were: (1) having quality partners engaged prior and during activity implementation, (2) an existing plan or strategy was used in project design as a guide or confirmation of the regional need and desires for the sector and activities, (3) a climate shock occurred post- project, and (4) the implementing partner had prior experience in the country before receiving the USDA award. The objective of this site is primarily to help answer the USDA evaluation question “How do beneficiaries respond to subsequent (climate) shocks and is it attributable to the project?” By including a site that theoretically should have sustained impacts, and yet also experienced a shock, the evaluation is thus able to test what the response to the shock was. Site 2 This site is one that is based on factors that theoretically support sustainability, and has not experienced a significant climate shock post-project. The same criteria as Site 1 are used in Site 2 for being likely to have achieved some level of sustainability today. The difference between these sites is that a shock has not occurred. Hypothetically, this site will have had the greatest chance of sustainability success of their (2 other) selected peers. The objective of this site is to provide an optimal case for sustainability that can be evaluated for understanding the activities, long term outcomes, and community value that remain in a “best case scenario” environment. Site 3 This site is one that provides a theoretical counterfactual for a sustainable project. The criteria for inclusion as a counterfactual were: (1) quality partners were not involved in project design, (2) an existing plan or strategy was not used as a guide or confirmation of the regional need and desires for the sector and activities, (3) a climate shock has not occurred since the project close, and (4) the implementing partner had prior experience in the country before receiving the USDA award. In applying the aforementioned criteria to the 13 sites available, no projects met all criteria. However, two projects each met 3 of the 4 criteria for inclusion. Each project met a different set of criteria. Therefore, USDA-FAS will need to weigh the benefits and consequences of each site to determine priority learning. 152 | Annexes  | Annex V — Valuation Approach, Methods and Limitations Annex V — Valuation Approach, Methods and Limitations Technical Approach A 5-phase evaluation strategy was used that allowed for collaboration at every step of this project. The five phases included (1) Inception, (2) Design & Planning, (3) Data Collection, (4) Data Analysis, and (5) Reporting. Project Selection Project Inception During this phase, the guiding frameworks for the evaluation were established; we refined evaluation questions and priorities, identify sources of data and methods, and determine what was and wasn’t in scope for the evaluation. During this phase: ● • The evaluation team facilitated an inception meeting with USDA/FAS to clarify and refine the scope of work, understand roles and responsibilities, establish key contacts, and confirmed a communication plan and schedule. The evaluation team identified and requested background information, and articulate stakeholders that should be involved to guide the remainder of the evaluation. ● • During this phase, feedback from USDA/FAS was gathered to refine questions and methods appropriate for each project. ● • Following the review of initial project data, interviews and feedback from stakeholders an evaluation approach and work-plan were developed, including a more specific timeline of activities. Design & Planning for project selection In this phase, the team planed and prepared all of the processes and materials to be used in the evaluation. Using insights from stakeholder interviews and USDA/FAS, the initial evaluation ideas were developed into project selection rubrics and plans of action. ● • The evaluation team and USDA/FAS agreed on an initial set of exclusion criteria to reduce the number of closed project from over 100 down to 13 (detailed summary in Annex X). Those criteria were to exclude projects; – Awarded before FY2005 or after FY2012 – That will have been closed for less than 1 year by December 2017 – Government-to-Government agreements – That focused on types of activities that FFPr no longer funds – That were considered unsuccessful at closing – That were followed by a continuation project – Awarded to inactive FFPr recipients – In areas where DOS has an active Travel Warning – With too little documentation available USDA/FAS then provided the evaluation team with a list of 13 projects that met these criteria. With evaluation team guidance, USDA staff identified additional criteria for selecting closed projects to be part of the sustainability assessment. An iterative process was used to exclude Annexes | Annex V — Valuation Approach, Methods and Limitations | 153 projects; 5 criteria were ultimately narrowed down for the project selection (See Annex X for detailed description of round 2 criteria and rationale). These criteria were then combined to create projects that could hypothetically provide some comparison against each other. Six projects were determined to meet the requirements through a review of implementing partner award contracts and initial interviews with partner staff. The final criteria for site selection were: Closed project: • The project had been closed for more than 2 years (yes) Level of sustainability: • Had previous experience working in the award country (yes/no) • Designed project based on country goals (yes/no) • Had local partnerships established prior to the award(yes/no) Experienced a shock: • Project beneficiaries experienced a negative shock post-project close (yes/no) Two sites were selected for each of the following hypothesized categories: • Sustainable with a shock: A site that may theoretically support sustainability but has since experienced a negative shock post-project close. (Criteria 1-5 were “yes”) ● • Sustainable without a shock: A site that may theoretically support sustainability and has not experienced a significant (political, social, physical or economic) shock post-project close. (Criteria 1-4 were “yes,” criteria 5 was “no”) ● • Less likely to be sustainable: A site where project impacts are not expected to be sustained post-project close. (Criteria 1-4 were “no”, 5 was yes or no) After the final criteria were established, in June 2018, three closed projects – one for each category – were selected by USDA to be a part of the evaluation: ● • Guatemala 2012 (Sustainable with a shock) ● • Sri Lanka 2005 (Sustainable without a shock) ● • Benin 2011 (Less likely to be sustainable)1 ● • One backup project was also identified in case a mid-course change was needed2 After the final selection of the 3 project sites evaluation plan were created in a collaborative effort between stakeholders. Steps included: ● • Worked with USDA/FAS and implementing partners to determine data sources and methods, criteria and factors that were then incorporated into project specific evaluation plans. ● 1 It should be noted that no awards fully met the “less (not) likely to be sustainable” category criteria. Benin received a “yes” for criteria 2. Nevertheless, FAS felt that it still had the potential to provide a counterfactual case for exploring less sustained practices and outcomes. 2 After the 2019 Easter Day attacks in Sri Lanka it was determined that the evaluation team would not be able to conduct the field visit themselves, given that 80% of the project had been completed at this point it was not advised to use the back-up project to replace Sri Lanka and so data collection proceeded with a locally based team. 154 | Annexes  | Annex V — Valuation Approach, Methods and Limitations • Prepared evaluation plans (see Annex XX) for each case study, received and incorporated feedback from USDA/FAS and implementing partners. These plans included timelines, activities, methods and tools, data security procedures, and other needed content. ● • Based on these evaluation plans draft data collection tools were created including an evaluative rubric, interview protocols, and multi-step site visit protocols. ● • Interviews were conducted with stakeholders in country via phone and Skype to further refine the evaluation questions, methods and sources and to begin to narrow down culturally appropriate evaluation methods. Alongside evaluation question development, USDA-FAS staff informed the proposed methodology by articulating desired goals of the final results. ● The first design phase of data collection further informed the evaluation plans, built relationships with partners in country and facilitated the logistics of data collection. The input received during the design phase helped guide how local enumerators would be used, interpreters and experts; collaborating with local staff and consultants helps ensure that there is an understanding of a fuller picture of results; they bring expertise about local context, factors that affected the level of success experienced at each site, and how to interpret data we are seeing. Data Collection Methods and Rationale Methods were chosen based on multiple factors, including: cultural sensitivity, efficacy in answering evaluation questions, availability of data and feedback from implementing partners. The same methods were used in all sites to maintain data consistency for comparison across countries. These methods were: • Image grouping • In-depth interviews • Observations • Document review Local coordinators and the implementing partner in each country identified and recruited prospective participants. The evaluation team—of UMN faculty, graduate students, and Improve Group consultants—conducted in-person interviews, field observations and facilitated Image Grouping© sessions in Guatemala and Benin. A local Sri Lankan evaluation firm conducted these activities in Sri Lanka following a terrorist attack in May 2019, the time period that was scheduled for data collection. Image Grouping© Image Grouping© is a participatory mixed-methods data collection tool that helps increase engagement in research, evaluation, needs assessments, and other kinds of data collection. Image Grouping© is used to overcome literacy barriers, particularly in locations where local languages are not commonly written or participants likely have a low level of formal education. As a mixed-methods approach, Image Grouping© can provide both qualitative and quantitative data; an initial activity uses an image-based tool whereby participants use stickers to respond to survey-like questions. Annexes | Annex V — Valuation Approach, Methods and Limitations | 155 In-depth interviews As the name implies, in-depth interviews provide rich and deep knowledge on particular topics. Through anecdotes and detailed stories interviews with key informants have been shown to provide the depth of information one might be able to capture through multiple survey respondents. Interviewers can also ask for more detail than they would otherwise on a survey, providing opportunities for clarification of data. Interviewees helped to describe the design and implementation strategies used during the award, providing more context than program documents held. Interviewees further helped answer questions about the extent to which larger systems change occurred. Observations Observations allow for researchers to collect additional evidence of agricultural practices and technology being used, and to have informal conversations about project activities and farmers experiences with the project. Observation protocols (Annex XX) were created by the evaluation team to be used on all three sites. Another benefit to observations is the lower burden on respondents’ time. Literature review A literature review was conducted at two stages in the evaluation; one, prior to primary data collection to identify factors that increase or decrease the likelihood of sustainability. These factors guided data collection question design and served as a framework for initial analysis. A second stage of literature review occurred that further explored preexisting conditions relating to sustainability, elements of projects design, and implementation practices. The goal of the review was to determine study areas for exploration, orient data collection and analysis, and provide supporting or refuting secondary evidence of sustainability alongside primary data. Secondary data analysis Project reports; including proposal, midterm and final reports as well as success stories and project summaries were provided to the evaluation team to review and were used to inform research methods, data collection, and analysis. Survey The 2011 Guatemala project is the only one where any type of survey was carried out, the evaluation team chose to implement an online survey for one population of beneficiaries who would not participate in other data collection processes. Data Sources A wide range of sources were included in data collection. The evaluation team worked with implementing partners at the three sites to help conduct the evaluation. A range of stakeholders from numerous regions in all three countries were included in the evaluation to ensure a wide range of perspectives were gathered and to triangulate evidence. Stakeholder types include: • Extension agents ● • Farmer Association/Cooperative leaders ● 156 | Annexes  | Annex V — Valuation Approach, Methods and Limitations • Financial institutions ● • Financial institution loan recipients ● • Governments ● • Implementing partners ● • Lead farmers ● • Mobilizers ● • Partner institutions Many languages were also spoken to conduct data collection to gather a diverse range of perspectives. Throughout the three sites, data collection was administered in 13 languages, as shown by the table below. Number of participant languages used in data collection Language Benin Guatemala Sri Lanka Adia 21 Aguacateo 2 Cancobal 5 English 1 6 Fon 20 French 9 Inan 5 Mam 1 Mina, Ewe 17 Popti 1 Sinhala 93 Spanish 113 Wémè / Goun 20 Annexes | Annex V — Valuation Approach, Methods and Limitations | 157 Sample Statistics A wide range of stakeholders from the three sites participated in the evaluation. Implementing partners identified participants that fit various types of stakeholders identified by the evaluation team. The evaluation team conducted outreach with the intention of conducting 25 interviews that would cover all of the needed stakeholder types per site. A few participants participated in multiple forms of data collection within their respective site; as a result, the grand total listed in the Overall Sample Table below does not represent unique individuals that participated in data collection at a site. Image Grouping© participants were selected using a stratified convenience sample. This allowed the evaluation team to select groups based on additional hypotheses relevant to sustainability, such as the relationship between physical distance to markets and sustained outcomes. All Image Grouping© participants are farmers. Each site utilized slightly different criteria for the selection but included attention to: diversity of locations, variety of activity exposure during the project, and a balance of genders. Interviewees were selected based on key informant expertise. Individuals who had direct knowledge of the site’s project participated. Overall Sample Table Site Site Image Grouping Participants (4 workshops per site, 12 total) Interviews Survey Total Total beneficiaries served during project Guatemala 2012-2016 Counterpart International 97 34 107 238* 40,000 Benin 2011-2014 Partners for Development 76 23 99 6,000 Sri Lanka 2005-2013 World Vision 59 26 85 3,043 Grand Total 232 83 107 422 *The Guatemala sample is much larger due to the emailed survey that was sent out to extension agents, these survey data are primarily incorporated into the general conclusions about the extension systems interventions in that country. These additional data points were not part of assessing continued activities or impact. 158 | Annexes  | Annex V — Valuation Approach, Methods and Limitations Analysis Methods The evaluation team conducted qualitative analysis in Dedoose software to analyze the in-depth interviews and Image Grouping© discussion notes. Initially, the evaluation team developed codes using an inductive approach. These codes, as outlined by the code table below, were developed by identifying themes from literature on conditions needed for sustainability, as well as the evaluation questions. Following that, a deductive approach was used to find new themes based on the responses. The evaluation team lastly looked at relationships between themes and compared differences between different types of respondents, such as farmers and other key stakeholders. Code Table Project context and assumptions Was it sustained? What helped or hindered sustainability? What factors contributed to sustainability? • Value of practices ● • Project context • Threats • Equity What is sustained • Practices • Impacts What is not sustained • Practices • Impacts ● • Helped sustainability ● • Hindered sustainability • Design ● • Implementation ● • Environment ● • Pre-existing conditions ● • Motivation and cultural values ● • Shocks ● • Resources ● • Capacity ● • Communication ● • Relationships ● • Technical assistance Findings are based on codes that meet the following criteria: Substantive — There is a high number of mentions associated with the code, relative to other sub￾codes under the same parent code. Inter-ocular — One to only a few mentions are associated with the code, but the perspective is highly relevant or notable. For example if only a few women were interviewed and they mentioned a reduction in family violence, while no men mentioned this project impact, these few mentions would be considered highly relevant and thusly highlighted. Quantitative Image Grouping was analyzed using Excel and SPSS. Frequencies and cross tabs were conducted to understand differences between locations, or the activities that resulted in sustained outcomes. Themes across these responses were compared to those found in the qualitative results and integrated to achieve a mixed method understanding of the data. Annexes | Annex V — Valuation Approach, Methods and Limitations | 159 Survey responses analysis was done using Qualtrics cross-tab feature. Of particular interest beyond descriptive statistics was comparison across employment status, gender, and cohort groups when determining the final results of the project several years after the official project end date. The literature review was conducted to both inform our data collection and compare our findings to existing academic literature as well as program evaluations from other development project reports. Relevant academic literature was identified based upon relevance to agricultural extension, community agricultural development, community capacities, and relevant research in Guatemala, Benin and Sri Lanka. Project reports/evaluations were also identified based upon their relevance to the region and their focus on related project objectives and activities. This review was used to triangulate results, identify possible theme and explain some adherence or lack of for specific activities. Triangulation in mixed methods qualitative research refers to the application and combination of several research methods in the study of the same phenomenon. The analysis was completed with a keen eye toward using the data to inform decisions and future work for the program. We will used qualitative and quantitative analysis methods to identify and categorize themes to identify findings that are: • Prevalent: Common across multiple programs and respondents; • Insightful: unique to a few respondents, but share a perspective that is important to your work; • Illustrative: in-depth comments that do a particularly good job of describing a concept or idea. Limitations With all evaluations, there are limitations to the data that are gathered and the conclusions that can be drawn. Each of the three sites had limitations based on the unique natures, populations and circumstances surrounding each site, which are discussed in their respective sections of this report. Recall bias may have occurred due to the time that has passed following the closure of projects. This can lead to participants remembering events at a different scale or forgetting what they did at that time. As a result, some farmers may have reported participation in trainings and services mostly in areas that they still remember and use today, positively biasing findings towards a higher percentage of individuals sustaining practices. Image Grouping© methodology helps to reduce this likelihood by having farmers from similar areas participate in the same session and encouraging them to talk among themselves to remind each other whether they participated in certain activities over others. Positive response bias may have influenced farmers’ responses in two ways. First, in asking how participant lives are today as compared to prior to the project start, farmers may have been more likely to exaggerate the benefits of the project. In many data collection environments, participants aim to “please” data enumerators, responding in a way that they believe the enumerator is hoping for or contributes to their benefit. Farmers may have believed that by extolling the hardship they experienced prior to the project and highlighting how much they continue using today, that this would support future work in their area. Second, implementing partners were used to determine which sites to visit and who to recruit. Partners may have desired to show more successful 160 | Annexes  | Annex V — Valuation Approach, Methods and Limitations outcomes for their own audiences as well as potential influence on future funding through USDA. The evaluation team worked to compensate for this potential bias by requesting IPs to suggest sites that had varying degrees of success after being involved with the USDA project. The high rates of sustained practices suggest that this bias may be more prevalent in Guatemala and Benin data. These high rates of continued practices could also result from the short amount of time since the project has ended, 2 years in Guatemala and 4 in Benin. Therefore, information on what has hindered sustained practices, or challenged or reduces farmers face sustainability may be even more beneficial in reviewing to understand what to consider for creating an optimal environment for sustainability. Responses exceeding 100% of beneficiaries continuing practices occurred in all three cases, this happens when more respondents say they are continuing a practice than said they learned about it from the project. For example, in Benin, 54 participants said they were taught about safe pesticide handling from the GREEN project and 58 said they are continuing to use the safe pesticide handling practice they learned from the GREEN project. This results in over 100% “continued practice” this could be because the question was loosely interpreted by participants or translated to mean “do you use safe pesticide handling practices?” regardless of who they learned it from, participants may not have specifically remembered the training activity but they feel they are currently doing it, they may have learned from another project participant, or another source. Or participants may have simply been responding in a positive way to “please” data enumerators (see positive response bias above). However, in most cases responses exceeding 100% indicate in general that the practice or activity has been a success and is being applied by most or all people who learned about it. This project does not include control or comparison communities and thus cannot make claims about magnitude of impact or causality. As a result, although we were able to describe if income changed for participants and whether they attribute this to their respective projects, we do not know the extent to which income changes were solely the result of the USDA project. Likewise, secondary data did not exist at a geographic level similar to the implementation areas to demonstrate need or wealth status prior to project start. Therefore, we are unable to make claims about economic changes to regions where USDA projects were implemented. Annexes | Annex VI — Data Collection Instruments | 161 Annex VI — Data Collection Instruments 1 USDA Workshop Protocol - Benin PP 1 USDA Workshop Protocol Benin Welcome & introductions (20 minutes) DO: 10 minutes prior to start time, announce to the group that the workshop will begin shortly, and everyone should make sure to check in at the welcome desk and begin to take their seats. Let everyone know that they should have a packet of five sheets of paper which all look the same. If they do not have that packet, let them know where they can find it. DO: At workshop start, begin with the workshop introduction – SAY: Thank you for joining us today. As you may know, my name is (Odin) and these are my colleagues (Sara and Marie). We are working with Partners for Development to learn more about the project you experienced with them from 2011 to 2016. We want to better understand how your involvement in the project has changed your practices and livelihoods now that the project has been closed for some time. Some of us work for an independent research and evaluation consulting firm named The Improve Group, while others work for the University of Minnesota in the US. We are contracted with the United States Department of Agriculture to help them understand how the programs they fund impact lives in the long term. We are not obligated by the USDA to inform the U.S. Government of anything outside learnings about the project impacts. The information you share with us today will not be tied to you in any way. Your name and contact information will be deleted for analysis purposes. Your individual responses will be combined with that of other data collection participants to create larger themes. Findings will be reported in terms of group themes by region, such as “some participants from (REGION) stated…” or “many participants in (OTHER REGION) explained that…”. There are no right or wrong answers for this workshop. Your honest responses are important for us to understand the impact that the project has had on your practices and life. If you do not want to answer a question, you do not have to. There is no negative consequence for not participating at any time. The information we learn today will not increase your chances of USDA working with your community in the future. It may help other communities around the world who do get funding. We will continue the workshop in (LANGUAGE). It may be beneficial to sit next to someone who can help translate some words or activities if you need it. If anyone has questions, please feel free to speak with me at the end of our group time together. SAY: Our workshop is going to focus on things that you continue to do and experience today because of the project. In other words, what has been sustained from the project with USDA even though it has closed. To do this, we will discuss the following: • The things you learned about or participated in during the project • How those experiences and resources changed or adapted your practices after the project closed 162 | Annexes  | Annex VI — Data Collection Instruments 1 USDA Workshop Protocol - Benin PP 2 • What adaptations or changes in your work in agriculture you continue to do today as a result of what you learned in the project • In what ways your life has changed because of participating in the project • What things help you or hinder you from being able to continue to practice the things you learned as a result of the project • What unexpected or sudden events or experiences you may have had that influenced your ability to continue the practices or adaptations you learned about. We are going to walk through activities using the packet of paper and stickers you have. We will ask you to respond to questions using the paper and stickers. Then, we will ask you to raise your hands, so we can see which people have had similar experiences, and maybe have each person tell us their story! Reflections on project start (10 minutes) SAY: We are going to get you thinking about what things were like before the project—so you can better remember what you learned from the project. Let’s think back to 2011. It may be easier to think about this time in terms of other major events happening that year, such as President Yayi's party regaining control of parliament in elections. With the person sitting next to you, share a quick story about the following: What do you remember about what your life was like at that time? What do you remember about how you farmed, how well you and your family were doing, and the resources you had access to? How was everyone in your family’s health, finances, well-being? DO: Pause to let pairs discuss. After 5 minutes, bring the group back together to share back as a large group. SAY: 1. Let’s hear from about 5 people; I just want a few brief examples. Who can share an agricultural practice that you were doing back in 2016, before the project? For example, growing vegetables, selling your produce to markets, processing produce, and learning how to combat pests. 2. Let’s hear from another couple people. I just want a few very short stories. Who wants to share about what they remember about their lives in 2016? SAY: Thank you for sharing. Let’s discuss the project we’re focusing on today. Partners for Development worked with farmers on the Green Project. They trained farmers on how to grow vegetables like tomatoes, peppers, okra and leafy greens, appropriate use of pesticides, dry season cultivation, and how to handle vegetables immediately after harvesting. Green Project staff also worked with growers’ associations on marketing and business strategies to help them maximize their earnings potential as a group. Do you recall some of these trainings and workshops that you participated in during this time? If continued confusion, then SAY: Who else can explain the project so that the rest of the group remembers? Annexes | Annex VI — Data Collection Instruments | 163 3 Practices at project end DO (facilitator 2): Pass out green dot sticker sheets to participants. SAY: Let’s move on to an activity that we’ve put together. Please wait to use the green stickers until I tell you. We’re going to start with the sheet with “1” at the top, it looks like this. On this sheet, you see different pictures. These pictures represent the types of trainings, practices, support, and resources you accessed during the project. I am going to go through each one and explain what the picture represents. Before I describe the pictures, think about what you remember learning about or accessing during the project. What were the trainings you participated in? 3. As I go through each of the pictures on your sheet, place a green sticker on those images that represent things that you learned about in the project. DO: Go through each image on the sheet and explain its meaning, asking if the group has any questions. After the first image, ask the group, SAY: “who participated in trainings or services related to growing tomatoes?” Did you put a green sticker on this image? Continue with each image, with a slight pause between each to assess understanding of what image represents. No need to ask group the same questions. DO (facilitator 2): Observe the group to make sure everyone understands how to place stickers on their sheet. Assist as needed. Practices today (30 minutes) SAY: Great, thank you all for participating. Now we’ll flip the page to work on this sheet with “2” written on the top of it. You’ll notice that this looks just like the last sheet we were working on. Now we’d like to talk about your practices this agricultural cycle. These images represent the same things as the images on the last sheet. The difference is that this sheet represents continued use of what you learned, now. So, if you learned about how to grow tomatoes from the project, and now do something differently, we want to know! To do this, I’m going to pass out blue dot stickers. Please hold on to these stickers. DO: Pass out blue dot stickers. SAY: Look over the pictures on your sheet which represent the types of practices that you may do differently in your business or farming this agricultural cycle because of what you learned or accessed during the project. 4. Place a blue sticker on the images of practices that represent what you do differently in this agricultural cycle. [potentially, ask if the group wants you to go through the images again for reminders] DO: through each image on the sheet and explain its meaning. 1 USDA Workshop Protocol - Benin PP 164 | Annexes  | Annex VI — Data Collection Instruments 4 SAY: Once you have completed, raise your hands, so we can see which people have had similar experiences, and maybe have each person should share their response. 5. As you look at the practices on your sheet of paper, why are they important or valuable in your life today? SAY: Please raise your hands, and maybe have each person quickly tell us a short story of what their experience was. 6. Looking at the practices you don’t continue (those without a sticker), do those have value to you? Why or why not? SAY: Please raise your hands, and maybe have people tell us 2-3 reasons about the practices they don’t continue, and what value those have to their lives? a) Why don’t you continue those practices? b) What might keep you from using these practices in the future? SAY: Thank you. Now I’d like to think about other members of your community. I’d like to hear from 2-3 people about - 7. Who in your community may have not be using the skills or knowledge from the trainings/services? And why? SAY: Thank you. Lastly, I’d like to hear from a different 2-3 people for the following question - 8. Who can tell me a story about something bad or unpleasant that has happened because of this project? I would like to hear from a few people. a) BACKUP QUESTION: What about a story of something that happened that was not expected, because of this project? Outcomes SAY: Now we’ll switch to the next paper, which looks the same as the last two. We are going to pass out a sheet of stickers with pictures on them. We will explain what each sticker means. DO: Pass out stickers. SAY: These stickers represent the changes that you may experience because of the practices you began using after participating in the project. I’ll go through each one now and explain what they mean. We want to know what change you experience and what in the project helped you to achieve this. 9. Similar to how you put dot stickers on the activities you do, put these “changes”, or “outcomes” stickers on the practices which lead to these changes. For example, you may feel that X activity has helped you experience Y change, so you’d put this Y sticker on the X activity. So, if you have experienced the change I explain, place that sticker image on the training/service that helped you achieve it. Complete this on your own. 1 USDA Workshop Protocol - Benin PP Annexes | Annex VI — Data Collection Instruments | 165 5 DO: Facilitator explains each image and what it represents, pausing for questions. DO (facilitator 2): Observe the group to verify all understand the activity correctly. SAY: Please raise your hands, and maybe have each person quickly tell us a short story of what their experience was. Let us discuss￾10. What did you have in common or different about the changes you experience because of the project? SAY: Thank you for sharing. In the upcoming days, our team will be travelling to some of your communities to observe how your practices and lives have changed because of your involvement in the project. 11. I’d like to hear from 2-3 people about the types of things we should look for in your community that would demonstrate these changes you’ve experienced. Who would like to share? Supports (30 minutes) Maintenance SAY: Now I’d like to understand what helps you maintain these practices. Switch to your next sheet. This activity will be similar in that we will pass out a sheet of stickers with images on them, which represent different types of things. DO (facilitator 2): Pass out stickers. SAY: These are types of things that may help you in continuing the practices that you learned about during the project. I am going to discuss images that reflect the practices. I’ll ask you to provide examples of the types of things you experience that could be included within that image. Let’s begin with resources. What are some resources that have helped you maintain the agricultural practice(s) you learned from this project? SAY: Please raise your hands to give examples of the resources that have helped you. SAY: As you reflect on these stickers, think about what helps you in continuing the practices today that you learned about from the project. Think about which sticker best represents those helpful things or factors in your life. 12. Place those stickers on the practices which they help. You may select multiple of the same type of sticker to place on different practices, and you may place multiple stickers on the same practice. DO: Observe the group to verify all understand the activity correctly. SAY: Please raise your hands so we can discuss how these supports help you in continuing to use or practice the things you learned about during the project. Let us discuss the types of stickers you placed on your sheet, and why you selected those. Take note of those you had in common and those which were unique. 1 USDA Workshop Protocol - Benin PP 166 | Annexes  | Annex VI — Data Collection Instruments 6 13. SAY: Raise your hands so we know how everyone feels are the supports that help them in continuing their practices? 14. Looking at the supports that people have mentioned as being helpful in sustaining the practices they’ve used, are there any additional ways you’ve seen these supports help you and your family? I’d like to hear back from 2-3 individuals. Hindrances (30 minutes) SAY: Great, now we’ll work on the next sheet (5). Turn your page over. Again, you’ll notice that this is the same sheet you had worked on when we discussed practices you’ve learned about in the project. Now, we’d like to talk about things you may have experienced that have hindered or become a barrier to your ability to continue the practices you learned about in the project. Think back to your discussion about things that you learned about in the project, but do not use or do today. What things like not having enough money, not having enough workers or other people to help you get things done, or not having enough time to do the work keep you from continuing to use what you learned? This activity will be similar in that I will pass out a sheet of stickers with images on them, which represent different types of things that hinder you continuing to use what you learned. DO: Pass out stickers. SAY: These are types of things that may deter or inhibit you in maintaining the practices that you learned about during the project. Let’s begin with resources. What are some resources, or lack of resources, that inhibit you from maintaining the practices you learned from this project? DO: Give an example. Jot down additional examples listed by the group. Repeat for remaining, pausing for questions. SAY: Look over these same stickers and identify which have kept you from continuing to use the possible practices on your sheet. 15. Use the stickers to place them on the practices that do not continue, to explain what keeps you from using what you learned. DO: Observe the group to verify all understand the activity correctly. SAY: Let us discuss how these factors hinder or inhibit you in continuing the things you learned about during the project. Let us discuss the types of stickers you placed on your sheet, and why you selected those. Take note of those you had in common and those which were unique. 16. SAY: Again, I’d like to hear back from each of you, please tell me – a) What types of barriers did you put on your sheet? b) In what ways did those things serve as a barrier to your ability to experience or maintain these practices in your life? DO: Lead the group discussion and note the barriers and hindrances mentioned, and in what way. 1 USDA Workshop Protocol - Benin PP Annexes | Annex VI — Data Collection Instruments | 167 7 17. Looking at the barriers that you cited as standing in your way of implementing these practices, are there any additional barriers you’ve experienced that we should add? I’d like to hear from 2-3 people. SAY: Now I’d like to think about your current process and future plans. I’d like to hear from 2-3 people for this next question. 18. (In scenario where you’re running out of time, can skip) What are current threats/future barriers to your continued implementation of practices? Shocks (20 minutes) SAY: Since the project has closed, you may have experienced something that was sudden, upsetting, or a surprising event or experience which may have helped or hindered your ability to continue the practices you learned about. We’re calling this a shock. You may have experienced a shock that helped or hindered your ability to continue the practices you learned. We asked the [list type of local partner] to think about a few shocks, both positive and negative, that may have had an impact on your likelihood since the project has ended. Let’s go through a list of what those are, and how they may have supported or hindered you at that time. DO: Go through each shock, providing an example and eliciting additional examples from the group. Add any additional shocks as needed. SAY: Now I’d like to go through each shock and identify which you may have experienced. 19. How many of you have experienced X shock? Please raise your hand if you have. DO: Go through each shock, tallying the number of people who raise their hand for each shock. SAY: I’d like to hear from 1-2 people for each shock to better understand how that shock has impacted you. a) In what ways did that shock have an impact on you, e.g. on your agricultural business or families’ wellbeing? b) In what ways did you feel prepared or equipped to respond due to what you learned in this project? Closing (10 minutes) SAY: Great, thank you all for your thoughts. To wrap it up, we’d like to reflect on your experience overall. 20. What good did this project bring to your community? SAY: Thank you for your time. Those are all the questions we have for you today! Please feel free to stay and let me know if you have any questions or concerns at this time. 1 USDA Workshop Protocol - Benin PP 168 | Annexes  | Annex VI — Data Collection Instruments USDA Interview Protocol for Benin Farmers Welcome and introductions Hello, thank you for meeting with me today, my name is ______________ and I work for the University of Minnesota in St. Paul, MN. The University of Minnesota and The Improve Group a St. Paul-based evaluation firm have been contracted by the United States Department of Agriculture to complete a study aimed to understand what activities, practices, and impacts have continued among recipient communities after funding for the USDA/FFPr project ceased. We hope to learn for example, if people have continued to do the things they learned how to do as part of the project. I will take notes during our conversation and combine them with responses from other people we interview. Your individual responses will be kept confidential. We won’t tie your name to your comments in our reports. There are no right or wrong answers! We are hoping to hear your perspective and get your opinion. Finally, I would like to emphasize that you won’t receive any benefits from participating in this activity. Your participation is voluntary, so please feel free to skip any questions at any time. Context The Partners for Development (PfD) Green project was implemented in the southern areas of Benin between November 2011 and July 2016 and aimed to help smallholder vegetable farmers in Benin improve and increase crop production and expand the sale of their produce. PfD also sought to increase farmer association capacity to process crops post￾harvest, add to the value chain, and improve business and financial management, while increasing access to loans from local microfinance institutions. 1. Please share with us your name, and your role or connection to the project. Pre-existing conditions 2. Thinking back on a time before the project, before 2011, please describe the conditions [in Southern Benin] before the project started. What was it like for agricultural producers at that time? (around May 2011, one historical pointer is that it is when President Yayi's party regained control of parliament in elections.) 3. What conditions made your work [agricultural or agricultural enterprise] especially challenging at that time? Prompt for possible conditions; business, economic, policy, infrastructure, environment. a. You mentioned some conditions that made your work at before the start of the project challenging. In what ways have any of these challenges prevented -or been beneficial- for farmers to continue with the practices they learned during the project? 2 N_AUSDA Benin Interview Protocol - Farmers Annexes | Annex VI — Data Collection Instruments | 169 4. What conditions supported your work at that time? [agricultural or agricultural enterprise] Prompt for possible conditions; business, economic, policy, infrastructure, environment. b. You mentioned some conditions that made your work with before the project start easier. In what ways have any of these things (conditions) helped you continue with the practices you learned about during the project? 5. What do you remember about the environment at the time? Prompt for: biophysical factors weather, soils, biodiversity, crops, etc. a. How did these environmental conditions influence your agricultural work or enterprises? 6. What aspect of the project do you think should have been different so that it would be more effective (having a positive outcome) for agriculture in your community? 7. What positive outcomes from the project did you experience? To what extent do you believe other farmers in the project had these same experiences? 8. What would you say are negative outcomes of the project? To what extent do you believe other farmers in the project had these same experiences? Equity 9. Before the project started, what [groups] in particular faced additional disadvantages? Who were they and what barriers did they face? (prompt for women, children, the landless, religious minorities, tribes, etc.) 10. In what ways has their situation changed, or not? 11. What would their situation look like now if the project had not existed? 12. What about the project has improved things for these groups? 13. What about the project perhaps made things worse? 14. How would your situation today be different if the project had not occurred? Shocks 15. From previous interviews we have learned we have learned that project trainers who provided technical assistance were only available for the duration of the project, how has this impacted you? i. How has this impacted communities and the mentoring system started during this project? 16. How have the following shocks impacted people? (list "shocks" previous interviewees mentioned) Or -if you don’t have the list yet- What other shocks have occurred since the completion of the project? 17. How did these shocks influence the continuation of project practices and impacts? Sustainability 18. What lessons that you learned (and services accessed) through the project are still used and applied today? 2 N_AUSDA Benin Interview Protocol - Farmers 170 | Annexes  | Annex VI — Data Collection Instruments 19. What has helped you continue applying the practices today? i. Prompt for how valuable they did/not find the things they learned and accessed in the project. 20. What has kept you from continuing to apply or use the practices you learned? 21. What changes have you seen in your life and community as a result of your continued application of the things you learned and accessed through GREEN? You may prompt for the following outcomes if they only talk about agricultural practices and note (yes or not) if the interviewee has observed these. Be sure to have them describe the situation: • More money • Better health • More farming seasons • Improved nutrition • Food Security • Use of improved technologies for agricultural production • Increased Agricultural Productivity • Crop Diversification • More time • Community trust • Personal and community pride • Youth employment • Access to new markets for agricultural products • Improved relationships • Financial Services • Access to credit (for producers) • Access to grants (for organizations) • Transparent and Improved organization (organizational capacity/improved OCP scores) • Better access to information – (Associations that can provide support Improved Access to information about agricultural markets) Closing These are all of the questions I have for you. Do you have any questions for me? Is there anything that you would like to add that I didn't ask about? Thank you so much for your time and thoughtful responses! We are excited to explore this project and hope to be able to share with you what we learn. USDA Debrief Form Country: Village/Town: Date: 2 N_AUSDA Benin Interview Protocol - Farmers Annexes | Annex VI — Data Collection Instruments | 171 Your name: Method: ___________________________________________________________________________ If you had to give the interaction a headline and summary sentences, what would it be? (Headline should be one short phrase, followed with 2-3 summary statements) What information came up that was surprising? What information came up that was not surprising? What are some themes that are beginning to emerge for you? What codes would you suggest for the code tree, keywords or key terms? What concerns, or issues were raised that should be communicated to Sara and/or Britta? What are some things that need to be followed up on (community questions, etc.)? 2 N_AUSDA Benin Interview Protocol - Farmers 172 | Annexes  | Annex VI — Data Collection Instruments Anything else you can think of? 2 N_AUSDA Benin Interview Protocol - Farmers Annexes | Annex VI — Data Collection Instruments | 173 1 Protocolo de taller de USDA Guatemala Bienvenida y presentaciones (20 minutos) HAGA 10 minutos antes de la hora de inicio, anuncie al grupo que el taller comenzará en breve, y todos deben asegurarse de registrarse en la mesa de bienvenida y comenzar a tomar asiento. Hágales saber a todos que deben tener un paquete de cinco hojas de papel que tengan el mismo aspecto. Si no tienen ese paquete, hágales saber dónde pueden encontrarlo. HAGA: Al inicio del taller, comience con la introducción del taller - DIGA: Gracias por acompañarnos hoy. Como sabrán, mi nombre es Isabel y estos son mis colegas Kayla, Megan y Ronnie. Estamos trabajando con Counterpart International para obtener más información sobre el proyecto del que ustedes fueron parte en 2012-2015. Queremos entender mejor cómo su participación en el proyecto influenció la forma en la que hacen su trabajo, las maneras en las cuales sostienen a sus familias, y su vida diaria, aún después que el proyecto terminó. Algunos de nosotros trabajamos para una firma independiente de consultoría en investigación y evaluación llamada The Improve Group, mientras que otros trabajamos para la Universidad de Minnesota en los Estados Unidos. Tenemos un contrato con el Departamento de Agricultura de los Estados Unidos (conocido como USDA), y estamos de ayudarle a esta agencia a comprender cómo los programas que ellos financian impactan la vida de los beneficiarios de estos proyectos a largo plazo. Sin embargo, queremos aclarales que el USDA no nos obliga a informar al gobierno de los EE. UU. de otras cosas que no estén relacionadas con los los impactos del proyecto. Adicionalmente, la información que compartan con nosotros hoy no estará vinculada a ustedes de ninguna manera. Su nombre e información de contacto se eliminarán de nuestros listados durante nuestros procesos de análisis de la información. Sus respuestas individuales se combinarán con las de otros participantes en la recopilación de datos para crear temas más amplios. Esto quiere decir que los resultados de estas actividades los transmitiremos a nuestro cliente en términos de temas grupales por región, por ejemplo "algunos participantes de San Marcos mencionaron ..." o "muchos participantes en Huehuetenango explicaron que ...". También quiero recalcar que no hay respuestas correctas o incorrectas en este taller y que sus respuestas honestas son muy importantes para que entendamos el impacto que el proyecto ha tenido en sus prácticas y en su vida. Si no desea responder alguna pregunta, no tiene que hacerlo. No hay ninguna consecuencia negativa por no participar en algún momento del taller. La información que obtengamos hoy no aumentará las probabilidad que el USDA trabaje con su comunidad en el futuro. Sin embargo esta información podría ayudar a otras comunidades alrededor del mundo que tal vez en el futuro obtengan financiamiento a través de este tipo de proyectos. Este taller será completamente en español, por lo cual podría ser beneficioso sentarse junto a alguien que hable español como su lengua materna y que pueda ayudarlo a traducir algunas 3 USDA Workshop Protocol - Guatemala_ Español 174 | Annexes  | Annex VI — Data Collection Instruments 2 palabras o actividades si lo necesita. No duden también en preguntarme el significado de alguna palabra que no entiendan y yo haré mi mejor esfuerzo para proveer un sinónimo o una definición. Si alguien tiene preguntas, venga y hable conmigo al final de nuestro tiempo en grupo Diga: Nuestro taller se centrará en las cosas que continúan haciendo y experimentando hoy debido a la existencia del proyecto. En otras palabras, lo que se ha sostenido del proyecto de USDA a pesar de terminó en el 2017. Para ello, discutiremos lo siguiente: • Las cosas que aprendieron o actividades en las que participaron durante el proyecto. • Cómo esas experiencias y recursos han cambiado o se han adaptaron en sus prácticas después del cierre del proyecto • Qué adaptaciones o cambios en su trabajo agropecuario continúa haciendo hoy como resultado de lo que aprendió en el proyecto • De qué manera ha cambiado su vida por participar en el proyecto • ¿Qué cosas le han ayudado o impedido seguir practicando las cosas que aprendió como resultado del proyecto? • Los eventos o experiencias inesperados o repentinos que haya tenido que influyeron en su capacidad para continuar con las prácticas o adaptaciones de las cosas que aprendió. Vamos realizar las actividades de este taller usando el paquete de papeles pegatinas que les dimos en la mesa de bienvenida. Primero les pediremos que responda a alguna de las preguntas utilizando el papel y las pegatinas. Luego, le pediremos que discuta sus respuestas con un grupo pequeño. Y luego, haremos que un represéntate de su grupo informe al grupo más grande. Este proceso se repetirá varias veces. ¡Yo los guiaré paso a paso, así que no se preocupen! Antes de comenzar, por favor creen un grupo de tres con personas cercanas a usted. Seleccione una persona de su grupo que resumirá su conversación al grupo más grande. HAGA: Espere la conversación, confirme que todos tienen un "líder" antes de continuar. DIGA: Levante la mano cada líder de cada grupo, por favor Reflections on project start (10 minutes) SAY: Primero quiero que empiecen a pensar en cómo eran las cosas antes del proyecto, para que pueda recordar mejor lo que aprendió del proyecto. Pensemos en el 2012. Puede que sea más fácil pensar en este momento en términos de otros eventos importantes que suceden ese año, como la renuncia del Vicepresidente y el año en que los agentes de extensión fueron despedidos. Con la persona sentada que está sentada a su lado, comparta una breve historia respondiendo las siguientes preguntas: ¿Cómo recuerdan sus vida en ese momento? ¿Qué recuerdan sobre cómo 3 USDA Workshop Protocol - Guatemala_ Español Annexes | Annex VI — Data Collection Instruments | 175 3 estaba trabajando en su finca, qué tan bien te estaban usted y su familia y los recursos a los que tenía acceso? ¿Cómo estaban todos en cuestiones de salud, las finanzas, el bienestar de su familia? HAGA: Pausa para dejar discutir a las parejas. Después de 5 minutos, vuelva a reunir al grupo para compartirlo como un grupo grande. DIGA: 1. Por favor, cinco personas compartan breves ejemplos de sus conversaciones ¿Quién nos quiere contar de una práctica agrícola que estaba haciendo en 2012, antes del proyecto? Por ejemplo, cultivar cebollas, vender sus productos a los mercados o la apicultura (manejo de abejas mieleras). 2. Escuchemos a otras dos personas. Solo quiero unos cuentos muy cortitos. ¿Quién quiere compartir sobre lo que recuerdan de sus vidas en 2012? DIGA: Gracias por compartir. Vamos a discutir el proyecto en el que nos estamos enfocando hoy. El proyecto Counterpart International capacitó a agentes de extensión que trabajaron con promotores para compartir recursos y conocimientos con sus comunidades. Además, este proyecto apoyó el crecimiento y la organización de CADERES que ayudó a crear planes de negocios. Por último, el proyecto apoyó a cooperativas financieras, como MICOOPE, a proporcionar mejores préstamos comerciales a los agricultores locales. Tal vez recuerden que agentes de extensión o promotores que realizaron capacitaciones relacionadas con el riego de cultivos, el uso de fertilizantes, la implementación de diferentes cultivos, o talvez diferentes formas de cuidar a los animales de su finca. También es posible que algunos de ustedes hayan participado en la creación de planes de negocios o en la obtención de préstamos. ¿Recuerdan algunos de estos entrenamientos y talleres en los que participó durante este tiempo? Si la confusión continúa, entonces DIGA: ¿Quién más puede explicar el proyecto para que el resto del grupo lo recuerde? Prácticas al final del proyecto. HACER (facilitador 2): Distribuya hojas de pegatinas de puntos verdes a los participantes. DIGA: Pasemos a una actividad que hemos para ustedes. Por favor esperen para usar las pegatinas verdes hasta que les diga. Vamos a comenzar con la hoja con "1" en la parte superior, se ve así. En esta hoja, se ven diferentes imágenes. Estas imágenes representan los tipos de capacitación, prácticas, asistencia y recursos a los que accedió durante el proyecto. Voy a repasar cada uno y explicar lo que representa la imagen. Antes de describir las imágenes, por favor piensen en lo que recuerda haber aprendido o accedieron durante y mediante el proyecto. ¿En qué entrenamientos participaste? 3. A medida que reviso cada una de las imágenes en su hoja, coloque una etiqueta verde en las imágenes que representan cosas que aprendió en el proyecto. 3 USDA Workshop Protocol - Guatemala_ Español 176 | Annexes  | Annex VI — Data Collection Instruments 4 HAGA: Revise cada imagen en la hoja y explique su significado, indagando si el grupo tiene alguna pregunta. Después de la primera imagen, pregunte al grupo, DIGA: "¿quiénes participaron en los entrenamientos o servicios relacionados con la apicultura (manejo de abejas mieleras)?" ¿Pusieron una etiqueta verde en esta imagen? Continúe con cada imagen, con una ligera pausa entre cada una para evaluar la comprensión de lo que representa la imagen. No es necesario hacer las mismas preguntas al grupo. HAGA (facilitador 2): Observe al grupo para asegurarse de que todos entiendan cómo colocar pegatinas en su hoja. Ayudar según sea necesario. Prácticas hoy (30 minutos) DIGA: Genial, gracias a todos por participar. Ahora pasaremos a la siguiente página para trabajar en esta hoja con el número "2" escrito en la parte superior. Notarás que se parece a la última hoja en la que estábamos trabajando. Ahora nos gustaría hablar sobre sus prácticas durante este ciclo agrícola. Estas imágenes representan las mismas cosas que las imágenes en la última hoja. La diferencia es que esta hoja representa el uso continuo ahora de lo que aprendió con el proyecto. Entonces, si aprendió cómo hacer apicultura con el proyecto y lo continúa haciendo, ¡queremos saberlo! Para hacer esto, voy a repartir pegatinas de puntos azules. Por favor, conserve estas pegatinas. HAGA: Pasar las pegatinas de puntos azules. DIGA: Mire las imágenes en su hoja que representan los tipos de prácticas que puede realizar de manera diferente en su negocio o en la agricultura de este ciclo agrícola debido a lo que aprendió o accedió durante el proyecto. 4. Coloque una etiqueta azul en las imágenes de las prácticas que representan lo que hace de manera diferente en este ciclo agrícola. [potencialmente, pregunte si el grupo quiere que vuelva a revisar las imágenes para recordatorios] HAGA: revise cada imagen en la hoja y explique su significado. DIGA: Una vez que hayan completado esta actividad, reúnanse con su grupo. Por favor en su grupo, cada persona discuta la siguiente pregunta brevemente. Recuerde, su vocero informará al grupo más grande. Recuerden que todos deben compartir su respuesta con su grupo. 5. Al observar las prácticas en su hoja de papel, ¿por qué son importantes o valiosas en su vida actual? HACER: Después de dos minutos, pida al grupo que cambie a la segunda persona. Pida al grupo que cambie a la tercera persona después de que hayan transcurrido 5 minutos. Después de 7 minutos, regrese como al grupo grande. DIGA: ¿Una persona de cada grupo pequeño por favor dígame de 2 a 3 razones por las que su grupo discutió acerca de cómo estas prácticas son valiosas para sus vidas hoy? 3 USDA Workshop Protocol - Guatemala_ Español Annexes | Annex VI — Data Collection Instruments | 177 5 HAGA: escuche a una persona de cada grupo, administrando la conversación para que cada persona solo comparta 2-3 elementos por grupo. DIGA: Bueno, ahora discutiremos en grupos pequeños de nuevo. Y luego nos reuniremos nuevamente a informar al grupo más amplio sobre algunas de las cosas que discute. La pregunta que quiero que discutan es- 6. En cuanto a las prácticas que no continúan (las que no tienen una etiqueta), ¿son o fueron valiosas para ustedes? ¿Por qué o por qué no? HACER: Después de dos minutos, pida al grupo que cambie a la segunda persona. Pida al grupo que cambie a la tercera persona después de que hayan transcurrido 5 minutos. Después de 7 minutos, regresa como un grupo más grande. DIGA: ¿Una persona de cada grupo pequeño por favor comparta con todos 2 o 3 razones por las que su grupo valora o no las prácticas que no continúan? HAGA: escuche a una persona de cada grupo, administrando la conversación para que cada persona solo comparta 2-3 elementos por grupo. Una vez completado, DIGA: Por favor 2 o 3 personas compartan conmigo a. ¿Por qué no continúan con esas prácticas? b. ¿Qué podría impedirle utilizar estas prácticas en el futuro? DIGA: Gracias. Ahora me gustaría que pensáramos acerca de otros miembros de su comunidad. Me gustaría saber de 2-3 personas sobre 8. ¿Quién en su comunidad puede no haber estado utilizando las habilidades o el conocimiento de las capacitaciones/servicios del proyecto? ¿Y por qué? DIGA: Gracias. Por último, me gustaría que dos o tres personas diferentes a las que ya han participado me respondan la siguiente pregunta: 9. ¿Quién puede contarme una historia sobre algo malo o desagradable que haya sucedido a causa de este proyecto? Me gustaría escuchar a varias personas. 8. PREGUNTA DE RESPALDO: ¿Alguien me podría contar historia de algo que sucedió que no se esperaba debido a este proyecto? Resultados DIGA: Ahora cambiaremos a la siguiente hoja, se ve igual que los dos pasadas. Vamos a repartir una hoja de pegatinas con figuras en ellas. Explicaremos lo que significa cada pegatina. HACER: Pasar las pegatinas. DIGA: Estas pegatinas representan los cambios que pueden haber experimentado debido a las prácticas que comenzaron a usar después de participar en el proyecto. Voy a repasar cada uno 3 USDA Workshop Protocol - Guatemala_ Español 178 | Annexes  | Annex VI — Data Collection Instruments 6 ahora y explicaré lo que significan. Queremos saber qué cambio experimentó y qué en el proyecto lo ayudó a lograrlo. 9. De manera similar a cómo coloca las pegatinas en las actividades que realiza, coloque estas pegatinas de "cambios" o "resultados" en las prácticas que han contribuido a estos cambios. Por ejemplo, puede ser que la actividad X le ha ayudado a experimentar el cambio en Y, por entonces debe colocar la pegatina Y en la actividad X. Por lo tanto, si ha experimentado uno de estos cambio, coloque esa imagen de pegantina en el entrenamiento/servicio que lo ayudó a lograrlo ese cambio. Completen esta actividad cada uno por su cuenta. HAGA: El facilitador explica cada imagen y lo que representa, deteniéndose para hacer preguntas. HACER (facilitador 2): Observe al grupo para verificar que todos comprendan la actividad correctamente. DIGA: Una vez más, reúnanse en sus grupos pequeños y elijan a alguien para que informe al grupo más grande sobre esta próxima pregunta. En sus grupos pequeños, discutan – 10. ¿Qué tenías en común o diferente sobre los cambios que experimentaste debido al proyecto? HAGA: Después de dos minutos, pida al grupo que cambie a la segunda persona. Pida al grupo que cambie a la tercera persona después de que hayan transcurrido 5 minutos. Después de 7 minutos, regrese al grupo más grande. DIGA: ¿Por favor el vocero de cada grupo dígame de 2 a 3 ejemplos de cambios que tuvo en común o diferentes en su grupo? HAGA: escuche a una persona de cada grupo, administrando la conversación para que cada persona solo comparta 2-3 elementos por grupo. DIGA: Gracias por compartir. En los próximos días, nuestro equipo viajará a algunas de sus comunidades para observar cómo sus prácticas y vidas han cambiado debido a su participación en el proyecto. 11. Me gustaría escuchar 2-3 personas que compartan conmigo los tipos de cosas que debemos buscar en su comunidad que demuestren estos cambios que ha experimentado. ¿A quién le gustaría compartir? Soportes (30 minutos) de las actividades que continuan. DIGA: Ahora me gustaría entender qué les ha ayudado a mantener estas prácticas. Por favor cambién a la siguiente hoja. Esta actividad será similar en cuanto a que repartiremos una hoja de pegatinas con imágenes en ellas, que representará diferentes tipos de cosas que los han ayudado. 3 USDA Workshop Protocol - Guatemala_ Español Annexes | Annex VI — Data Collection Instruments | 179 7 HAGA (facilitador 2): Pasa las pegatinas. Colgar el rotafolio de las fotos de soporte. DIGA: Estos son tipos de cosas que pueden ayudarlo a continuar con las prácticas que aprendió durante el proyecto. Tengo las mismas imágenes que se muestran aquí en este rotafolio. Como grupo, repasemos cada imagen y hablemos sobre lo que representa. Les pediré que me den ejemplos de los tipos de cosas que experimentaron que podrían incluirse dentro de esa imagen. Comencemos con los recursos. ¿Cuáles son algunos recursos que le han ayudado a mantener las prácticas agrícolas que aprendió de este proyecto? HAGA: De un ejemplo. Anote los ejemplos adicionales enumerados por el grupo. Repita para el resto, haciendo una pausa para las preguntas. DIGA: Mientras reflexiona sobre estas pegatinas, piense en lo que le ha ayudado a continuar las prácticas que aprendió con el proyecto hoy en día. Piensen en qué pegatina representa mejor esas cosas o factores útiles en tu vida. 12. Coloca esas pegatinas en las prácticas que ayuden. Puede seleccionar múltiples del mismo tipo de etiqueta para colocar en diferentes prácticas, y puede colocar varias etiquetas en la misma práctica. HACER: observar al grupo para verificar que todos comprendan la actividad correctamente. DIGA: Ahora, en sus grupos pequeños, discuta cómo estos apoyos lo han ayudado a continuar usando o practicando las cosas que aprendió durante el proyecto. Comente en grupo los tipos de pegatinas que colocó en su hoja y por qué las seleccionó. Noten los que tienen en común y de los que eran únicos a cada uno de ustedes. HAGA: Después de dos minutos, pida al grupo que cambie a la segunda persona. Pida al grupo que cambie a la tercera persona después de que hayan transcurrido 5 minutos. Después de 7 minutos, regresa como un grupo más grande. 13. DIGA: ¿Una persona de cada grupo pequeño me dirá de 2 a 3 razones por las que su grupo discutió sobre los apoyos que experimentan y que les ayudan a continuar con sus prácticas? HAGA: Tome notas en el rotafolio para calcular la cantidad de veces que se enumera cada soporte y de qué manera lo hizo. Escuche a una persona de cada grupo, administrando la conversación para que cada persona solo comparta 2-3 elementos por grupo. DIGA: Gracias por compartir conmigo. Repasemos nuestra lista aquí. 14. Viendo los apoyos que sus grupos han dicho que han sido útiles para mantener las prácticas que utilizaron, ¿hay alguna otra manera de ver estos apoyos que lo ayudaron a usted y a su familia? Me gustaría escuchar de 2-3 personas. HACER: tomar notas en los rotafolios. Obstáculos (30 minutos) DIGA: Genial, ahora trabajaremos en la siguiente hoja (5). Una vez más, notará que esta es la misma hoja en la que ha venido trabajando durante este taller cuando discutimos las prácticas que ha aprendido en el proyecto. Ahora, nos gustaría hablar sobre las cosas que puede haber experimentado que han obstaculizado o se han impedido su capacidad de continuar las prácticas que aprendió en el proyecto. Piense de nuevo en su discusión sobre las cosas que aprendió en el proyecto, pero no las use o haga hoy. ¿Qué cosas como por ejemplo no tener suficiente dinero, no tener suficientes trabajadores u otras personas para ayudarle a hacer las cosas, o no tener el tiempo suficiente para hacer el trabajo le impiden continuar usando lo que aprendió? Esta actividad será 3 USDA Workshop Protocol - Guatemala_ Español 180 | Annexes  | Annex VI — Data Collection Instruments 8 similar en cuanto a que repartiré una hoja de calcomanías con imágenes, que representan diferentes tipos de cosas que le impiden seguir usando lo que aprendió. HAGA: Pasar las pegatinas. Colgar el rotafolio de las fotos de soporte. DIGA: Estos son tipos de cosas que pueden disuadirlo o inhibirlo para mantener las prácticas que aprendió durante el proyecto. Tengo las mismas imágenes que se muestran aquí en este rotafolio. Como grupo, repasemos cada imagen y hablemos sobre lo que representa. Le pediré que proporcione ejemplos de los tipos de cosas que experimenta que podrían incluirse dentro de esa imagen. Comencemos con los recursos. ¿Cuáles son algunos recursos, o la falta de recursos, que le impiden mantener las prácticas que aprendió de este proyecto? HAGA: Dar un ejemplo. Anote los ejemplos adicionales enumerados por el grupo. Repita para el resto, haciendo una pausa para las preguntas. DIGA: Mire estos mismos adhesivos/pegatinas e identifique cuáles le han impedido continuar usando las prácticas en su hoja. 15. Use los adhesivos para colocarlos en las prácticas que no continúan, para explicar qué le impide usar lo que aprendió. HAGA: observar al grupo para verificar que todos comprendan la actividad correctamente. DIGA: En sus grupos pequeños, analicen cómo estos factores le han impedido continuar con lo que aprendió durante el proyecto. Comente en su grupo los tipos de pegatinas que colocó en su hoja y por qué las seleccionó. Noten de los que tenías en común y de los que eran únicos. HACER: Después de dos minutos, pida al grupo que cambie a la segunda persona. Pida al grupo que cambie a la tercera persona después de que hayan transcurrido 5 minutos. Después de 7 minutos, regresa como un grupo más grande. 16. DIGA: Una vez más, me gustaría escuchar las respuestas del vocero de cada grupo sobre lo que su grupo discutió. ¿Quién quiere empezar? a. ¿Qué tipo de barreras discutió tu grupo? b. ¿De qué manera esas cosas impidieron la capacidad de experimentar o mantener estas prácticas en su vida? HAGA: Tome notas en el rotafolio para calcular la cantidad de veces que se incluye cada factor y de qué manera fue un obstáculo. Escuche a una persona de cada grupo, administrando la conversación para que cada persona solo comparta 2-3 elementos por grupo. DIGA: Revisemos nuestra lista aquí. 17. Observando las barreras que sus grupos han mencionado como impedimentos para implementar estas prácticas, ¿existen barreras adicionales que haya experimentado y que debamos agregar? Me gustaría escuchar de 2-3 personas. HAGA: Tome notas en el rotafolio. 3 USDA Workshop Protocol - Guatemala_ Español Annexes | Annex VI — Data Collection Instruments | 181 9 DIGA: Ahora me gustaría que piensen en su situación actual y sus planes futuros. Me gustaría saber de 2-3 personas para la siguiente pregunta. 18. (En el escenario en el que se está quedando sin tiempo, puede omitir) ¿Cuáles son las amenazas actuales o barreras futuras que podrían impedir o dificultar la continuación de estas prácticas? Shocks (20 minutos) DIGA: Desde que el proyecto terminó, es posible que haya experimentado algo que fue repentino, perturbador o un evento o situación que lo tomo de sorpresa y que pudo haberle ayudado o disminuido su capacidad de continuar con las prácticas que aprendió con el proyecto. Estamos llamando a esto un shock. Es posible que haya experimentado un shock que le ayudó o dificultó su capacidad para continuar con las prácticas que aprendió. Le pedimos a Counterpart International que pensarán en algunos choques, tanto positivos como negativos, que pueden haber tenido un impacto en sus prácticas desde que el proyecto terminó. Revisemos una lista de lo que son, y cómo pueden haberte apoyado u obstaculizado en ese momento. HAGA: Fije el rotafolio de “shocks” Repase cada choque, proporcione un ejemplo y obtenga ejemplos adicionales del grupo. Agregue adicional shock según sea necesario. DIGA: Ahora me gustaría pasar por cada shock e identificar quien de ustedes lo han experimentado. Empecemos por la parte superior y pasemos por cada uno. 18. ¿Cuántos de ustedes han experimentado el shock X? Por favor levante la mano si lo ha experimentado HAGA: Revise cada shock, calcule la cantidad de personas que levantan la mano para cada shock. DIGA: Me gustaría escuchar 1 o 2 personas que han experimentado cada shock para comprender mejor cómo cada shock lo impacto. 19. ¿De qué manera te impactó ese shock? Lo afecto ¿En su trabajo agropecuario o en el bienestar de su familia? 20. ¿Se sintió preparado en alguna manera para responder a esos shocks gracias a lo que aprendió en este proyecto? ¿De qué manera? Cierre (10 minutos) DIGA: Genial, gracias a todos por compartir con nosotros sobre sus experiencias. Para terminar, nos gustaría reflexionar sobre su experiencia en general. 21. ¿Qué cosas buenas trajo este proyecto a su comunidad? DIGA: Gracias por tu tiempo. ¡Esas son todas las preguntas que tenemos para ti hoy! Por favor, siéntase libre de quedarse y avísenme si tienen alguna pregunta o inquietud en este momento. 3 USDA Workshop Protocol - Guatemala_ Español 182 | Annexes  | Annex VI — Data Collection Instruments USDA Interview Protocol for Associations Welcome and introductions Hello, thank you for meeting with me today, my name is ______________ and I work for the University of Minnesota in St. Paul, MN. The University of Minnesota and The Improve Group a St. Paul-based evaluation firm have been contracted by the United States Department of Agriculture to complete a study aimed to understand what activities, practices, and impacts have continued among recipient communities after funding for the USDA/FFPr project ceased. We hope to learn for example, if people have continued to do the things they learned how to do as part of the project. I will take notes during our conversation and combine them with responses from other people we interview. Your individual responses will be kept confidential. We won’t tie your name to your comments in our reports. There are no right or wrong answers! We are hoping to hear your perspective and get your opinion. Finally, I would like to emphasize that you won’t receive any benefits from participating in this activity. Your participation is voluntary, so please feel free to skip any questions at any time. Context The Counterpart Food for Progress project implemented in the western areas of Guatemala between September 2012 and January 2017, aimed to increase agricultural production and trade through the reconstruction and strengthening of the Extension system, the expansion of financial services, and the improvement of the organizational capacity of local cooperatives dedicated to agricultural activities. 1. Please share with us your name and the name of your agency, and how you were involved in the USDA Counterpart project. Can you describe your Association? Pre-existing conditions 2. Thinking back on a time before the project, can you describe the situation in the communities where you work. What was it like for agricultural producers at that time? (around the year 2012, some historical pointers were that the Guatemala soccer team was suspended from FIFA, the Vice-president of the country went to jail, and there were floods during the dry season.) 3. Can you think of 5 threats or challenges (retos) for (rural communities, producers) at the time of the project. What were things that made your work [trainings, etc.] especially challenging at that time? Prompt for possible conditions; business, economic, policy, infrastructure, environment a. Do you still consider those things to be a challenge, and do they affect you in the same way now, as they did then? Si las situatcions y condiciones mantienen todavia, y les siguen apoyando? 4 Interview protocol Associations Annexes | Annex VI — Data Collection Instruments | 183 5. What were some things that facilitated your activities (can you think of around 5?) (agricultural) at that time? Prompt for possible conditions; business, economic, policy, infrastructure, environment b. Do you still consider those things to be beneficial or facilitate your activities, and do they affect you in the same way now, as they did then? Si las situatcions y condiciones mantienen todavia, y les siguen apoyando? Equity 6. Have there been changes in your membership? OR in leadership positions? (Since going through this process are new groups more involved, like women, youth participation? Design Implementation and expectations 7. Do you recall going through the OD assessment, and what level you scored at? (High Med Low) 4. Do you recall the areas that you were trained in? What modules were the most impactful/useful for your association? How are they being used today? Shocks 8. Have you experienced any shocks and how has that impacted your association? 9. Has the absence of extension agents effected your organization your projects? 10. What changes have you made has a result of these shocks? Sustainability 11. What aspects of the project are continuing today? 12. How would this organization today be different if the project had not occurred? 13. What changes have you seen in your organization and your membership? : 14. Based on what you know today, what would you have done differently if you could go back in time and change pieces of the project to increase its impact? -Community trust -health 4 Interview protocol Associations 184 | Annexes  | Annex VI — Data Collection Instruments Food Closing These are all of the questions I have for you. Do you have any questions for me? Is there anything that you would like to add that I didn't ask about? Thank you so much for your time and thoughtful responses! We are excited to explore this project and hope to be able to share with you what we learn. Solo para clarificar nosotros no somos la agencia que ejecuto el Proyecto, solo somos aca como equipo de evaluacion, entices, to tienen que agradesernos, nosotros deberiamos agradercerles a usdes por su participacion en el Proyecto y su prpesencia hoy dia. 4 Interview protocol Associations Annexes | Annex VI — Data Collection Instruments | 185 Esta encuesta es para actualizar la información sobre su experiencia laboral después de graduarse de la Universidad de San Carlos con un diplomado en Extensión Rural. La información que proporcione se utilizará para informar al Departamento de Agricultura de los EE. UU. del impacto continuo de su trabajo. Esta información no lo está evaluando a usted ni a Counterpart International que administró el proyecto, se usará para describir la iniciativa que está en curso en Guatemala relacionada con el proyecto inicial. Información Demográfica: Masculino Femenino Para la formación del diplomado, estuve en cohorte. (seleccione 1 de 6): Mi estado de empleo actual es: Empleado por MAGA Desempleado Empleado, haciendo trabajo de extensión / asesoramiento en otro lugar. Sondeo: Nombre del empleo actual Empleado, ya no hace extensión / asesoría. Sondeo: Mi ocupación actual es Indique si está de acuerdo o en desacuerdo con las siguientes afirmaciones. De acuerdo Neutro / Sin opinión En desacuerdo Mi trabajo contribuye a fortalecer la política de MAGA de responder a las necesidades específicas de las mujeres rurales. Mi trabajo promueve el acceso equitativo a los alimentos nativos y tradicionales para abordar la pobreza rural. Mi trabajo proporciona asistencia alimentaria y nutricional a través de la 5 DRAFT Guatemala extension agent survey - Spanish 186 | Annexes  | Annex VI — Data Collection Instruments producción comunitaria de alimentos. Mi trabajo fortalece la capacidad de liderazgo de las mujeres. Mi trabajo promueve la producción, la comercialización y las capacidades empresariales en las comunidades rurales. Mi trabajo empodera a las mujeres usando criterios de género de NSEA para responder adecuadamente a las necesidades de la población rural. Mi formación en el diplomado me preparó bien para trabajar en las comunidades rurales. ¿Qué tan efectivos se considera para hacer un cambio en su comunidad como parte de un equipo de agentes de extensión que trabajan para MAGA? Muy exitoso, exitoso, algo exitoso, no exitoso ¿Quiénes son sus clientes actuales? Promotores Agricultores Asociaciones de productores Otro (por favor listar) Visito a mis clientes ... Semanalmente Mensualmente Al comienzo de la temporada de crecimiento de cultivos 5 DRAFT Guatemala extension agent survey - Spanish Annexes | Annex VI — Data Collection Instruments | 187 En la temporada de cosecha Una vez al año Mis clientes solicitan asistencia para los siguientes temas (Marque todas las que correspondan): Demostraciones Conocimientos técnicos Información de mercadeo Información financiera Manejo postcosecha Técnicas de almacenamiento para mis productos agrícolas. Manejo integrado de plagas La siguiente lista incluye temas o módulos que formaron parte de la capacitación del diplomado. Indique si utiliza las siguientes habilidades / conocimientos en su trabajo hoy. La uso No la uso Nunca la ha utilizado Proceso histórico de extensión. Incluir el sondeo. Si nunca usó este tema, ¿por qué nunca se usó? Pedagogía versus Andragogía Entendimiento sobre el rol de la extensión en el desarrollo rural. Teorías de la comunicación a) Comunicación enfocada al cambio de comportamiento. b) Aspecto introductorio a la comunicación. Motivar y animar a los buenos participantes. Métodos de extensión de Campesino a campesino Análisis situacional Diagnósticos rurales 5 DRAFT Guatemala extension agent survey - Spanish 188 | Annexes  | Annex VI — Data Collection Instruments Evaluación Rural Rápida Planificación de programas Evaluación y Monitoreo Técnicas de comunicación para la extensión (por ejemplo, medios de comunicación, medios sociales, podcast, videos, etc.) ¿Cuáles son los factores que limitan que haga su mejor trabajo? (Marque todo lo que corresponda.) Hora Recursos Transporte Indique qué tan importante cada una de estas áreas de contenido tiene para su trabajo ahora. Muy importante Algo importante No es importante Huertos biointensivos Seguridad alimentaria y nutrición Extensión en áreas protegidas. Extensión en áreas forestales Extensión en café Extensión en sectores privados Extensión en agronomía Indique qué tan importante es cada una de estas áreas de contenido para su trabajo ahora. Muy importante Importante Algo importante No es importante Análisis de agrotecnología. Agricultura en el Corredor Seco Adaptación al cambio climático 5 DRAFT Guatemala extension agent survey - Spanish Annexes | Annex VI — Data Collection Instruments | 189 Aspectos básicos para el análisis de los créditos en la agricultura Análisis de rentabilidad. Costos (¿Precios?) Cosecha de agua de lluvia Producción artesanal y semi￾comercial de compost de lombrices (vermicultura). Circulo de banana Camara fria Microorganismos eficientes Construcción y uso del nivel A Sistemas agroforestales Manejo integrado de plagas ¿Cómo te comunicas con tu clientela? (Marque todo lo que corresponda.) Redes sociales Presentaciones Visitas de campo / a la parcela Demostraciones Teléfono móvil Email ¿Trabaja con los CADER? Sí. Si la respuesta es sí, ¿Cuál tipo de CADER? (Marcar todos los que apliquen): No Seleccione el producto para el que proporciona asistencia técnica para los miembros de un CADER. Café Cacao Vegetales Frutas Ganadería Abejas/ miel 5 DRAFT Guatemala extension agent survey - Spanish 190 | Annexes  | Annex VI — Data Collection Instruments Pescado Flores Maíz Frijol Aves de corral Arroz Sigo trabajando con CADER en mi comunidad (Marque todo lo que corresponda). Como empleado/a pagado Como voluntario/a Como un miembro ¿Continúa tomando cursos o capacitación de desarrollo profesional? Sí, en caso afirmativo, ¿qué tipo de curso? No ¿Cuáles temas de capacitación adicional le gustarían para poder hacer su trabajo mejor o proporcionar más valor a sus clientes? 5 DRAFT Guatemala extension agent survey - Spanish Annexes | Annex VI — Data Collection Instruments | 191 1 USDA Workshop Protocol Welcome & introductions (20 minutes) SAY: Thank you for joining us today. As you may know, my name is (NAME) and I work with EcoLanka. We are working with World Vision to learn more about the project you experienced with them starting in 2005 to early 2009. We want to better understand how your involvement in the project has changed your practices and livelihoods now that the project has been closed for some time. The information we talk about today will be shared with an independent research and evaluation consulting firm named The Improve Group, as well as the University of Minnesota in the US. They are contracted with the United States Department of Agriculture to help understand how the programs USDA funds impact lives in the long term. • The information you share will not be tied to you in any way. • Your name and contact information will be deleted for analysis purposes. • Your responses will be combined with that of other participants to create larger themes. • There are no right or wrong answers. • Your honest responses are important for us to understand the impact the project has had on your practices and life. • If you don’t want to answer a question, you do not have to. • There are no negative consequences to participating. • The information we learn today may help other communities around the world who get funding. If anyone has questions, please feel free to speak with me at the end of our group time together. Our workshop is going to focus on things that you continue to do and experience today because of the project. In other words, what has been sustained from the project with USDA even though it has closed. The workshop will take approximately four hours. Review tool (10 minutes) We are going to walk through activities using the packet of paper you have in front of you. Does everyone have a packet? We will ask you to respond to questions using the paper and stickers. You’ll notice that the first three pages are the same, and the last two pages are the same. We will ask you to respond to questions using stickers, and then we may ask you to raise your hands, so we can see which people have had similar experiences and have some of you tell us your story. These sections describe a grouping of activities that you likely participated in learning about during [project name]. Reflections on project start (10 minutes) We are going to get you thinking about what things were like before the project—so you can better remember what you learned from the project. Let’s think back to 2004. It may be easier to think about this time in terms of what your life and livelihood looked like. 6 USDA Workshop Protocol - Sri Lanka 192 | Annexes  | Annex VI — Data Collection Instruments 2 1. Let’s think about what was happening before the project. What were you doing differently then, from now? How was your life different then? What about the agricultural practices you were using? a. Prompts: wealth, land, wellbeing, health, etc prior to the project Practices at project end (10 minutes) Let’s move on to an activity that we’ve put together. Please wait to use the green stickers until I tell you. We’re going to start with the sheet with “1” at the top. These pictures represent the types of trainings, practices, support, and resources you accessed during the project. You will have green dot stickers to use. You will use these stickers by placing a sticker on the activity that you learned about during the project. Don’t worry about covering up the picture, it is best that you put the dot sticker right on the image, so we can be certain what you’re referencing. You can put stickers on as many or as few as applies for you – do not feel that you need to use all of the stickers. 2. Place a green sticker on those images that represent things that you learned about during the project. Practices today (30 minutes) Now we’ll flip the page to work on this sheet with “2” written on the top of it. You’ll notice that this looks just like the last sheet we were working on. Now we’d like to talk about what you’re still doing today. These images represent the same things as the images on the last sheet. The difference is that this sheet represents continued use of what you learned, now. So, if you learned about how to grow collect and store rain water from the project, and now do something differently, we want to know! Look over the pictures on your sheet which represent the types of practices that you may do differently in your business or farming this agricultural cycle because of what you learned or accessed during the project. 3. Place a blue sticker on the images of practices that represent what you do differently today, in this agricultural cycle. 4. As you look at the practices on your sheet of paper, why are they important or valuable in your life today? 5. Looking at the practices without a sticker, why don’t you continue certain practices? What are some reasons why you have not done them, or stopped doing them? Please raise your hands, and maybe have people tell us 2-3 reasons about the practices they don’t continue, and what value those have to their lives? a) Why don’t you continue those practices? b) What might keep you from using these practices in the future? 6. Who in your community may have not be using the skills or knowledge from the trainings/services? And why? 7. Who can tell me a story about something bad or unpleasant that has happened because of this project? I would like to hear from a few people. a) BACKUP QUESTION: What about a story of something that happened that was not expected, because of this project? 6 USDA Workshop Protocol - Sri Lanka Annexes | Annex VI — Data Collection Instruments | 193 3 Outcomes (30 minutes) Now we’ll switch to the next paper, which looks the same as the last two. We are going to pass out a sheet of stickers with pictures on them. We will explain what each sticker means. These stickers represent the changes that you may experience because of the practices you began using after participating in the project. We’ll go one by one to add them to our sheet. We want to know what change you experience and what in the project helped you to achieve this. Again, please only use the stickers that apply to you. This time, we’ll be thinking about the broader categories of activities. You’ll notice that the activities on your sheet are grouped by category, with each category having a quadrant. When you go to put the stickers that represent changes on your worksheet, you will only put one sticker for that change in the box. You may have multiple stickers per box, but those will be stickers that represent different changes. For example, if I felt that training on water irrigation and bee keeping both lead to increased production, I would only use one increased production sticker to represent that change. There’s no need to have a sticker on each activity, since they’re in the same box. Do not feel as though you need to use all of the stickers. Place them directly on the image that they correspond to, without concern about covering up that image. 8. Similar to how you put dot stickers on the activities you do, put these “changes”, or “outcomes” stickers on the boxes which lead to these changes. For example, you may feel that bee keeping has led to you experiencing an increase in income, since you can now sell honey, so you’d put this money sticker on the bee keeping activity. So, if you have experienced the change I explain, place that sticker image on the box of activities that helped you achieve it. Complete this on your own. Okay, now let’s take a look at the sheets we’ve put together. I’d like to hear from a few people. 9. What changes have you experienced because of the project? Thank you for sharing. [PAUSE FOR A 10 MINUTE BREAK FOR THE GROUP TO USE THE BATHROOM, GET COFFEE AND BREAD/REFRESHMENTS] Supports/Maintenance (30 minutes) Now I’d like to understand what helps you maintain these practices. Switch to your next sheet. This activity will be like our first activity, where you used colored dot stickers to note your responses to our questions. These are types of things that may help you in continuing the practices that you learned about during the project. I am going to discuss images that reflect the practices. I’ll ask you to provide examples of the types of things you experience that could be included within that image. Let’s begin with resources. What are some resources that have helped you maintain the agricultural practice(s) you learned from this project? Please raise your hands to give examples of the resources that have helped you. 6 USDA Workshop Protocol - Sri Lanka 194 | Annexes  | Annex VI — Data Collection Instruments 4 As you reflect on these images, think about what helps you in continuing the practices today that you learned about from the project. Think about which image best represents those helpful things or factors in your life. 10. Place green dot stickers on the types of things that help you in continuing the practices that you learned about during the project. You may select multiple things that support you, but please only put stickers on those that are present in your life. I’d like to go through each of these supports seeing a show of hands of who marked it down on their sheet. We’ll have a conversation about how each one has been supportive to you. 11. Let’s begin with this first image. Raise your hand if you placed a green dot sticker on this image. 12. Great, for those who raised their hands, can I hear one to two examples of how this has supported you in continuing the practices you learned about in the project? 13. Looking at the supports that people have mentioned as being helpful in sustaining the practices they’ve used, are there any additional ways you’ve seen these supports help you and your family? I’d like to hear back from 2-3 individuals. Hindrances (45 minutes) Great, now we’ll work on the next sheet (5). Turn your page over. For this activity, we’re going to do the same thing but this time think about how the absence or presence of these things may deter or hinder you in maintaining the practices you learned about in the project. Let’s go through a few of these images and hear examples about how the absence or presence of these things may be a hindrance in your life. Thank you for sharing. 14. Now we’ll pass out blue stickers. Similarly to how we did with green stickers, we want you to place a blue sticker on the absence or presence of things that hinder you from continuing the practices you learned about in the project. 15. Again, I’d like to hear back from each of you. Let’s go through each one and please raise your hand if you had identified this on your sheet. b) What types of barriers did you put on your sheet? c) In what ways did those things serve as a barrier to your ability to experience or maintain these practices in your life? 16. Looking at the barriers that you cited as standing in your way of implementing these practices, are there any additional barriers you’ve experienced that we should add? I’d like to hear from 2-3 people. Now I’d like to think about your current process and future plans. I’d like to hear from 2-3 people for this next question. 6 USDA Workshop Protocol - Sri Lanka Annexes | Annex VI — Data Collection Instruments | 195 5 17. What are current threats/future barriers to your continued implementation of practices? Closing (10 minutes) Great, thank you all for your thoughts. To wrap it up, we’d like to reflect on your experience overall. 18. How can this conversation today inform what you carry forward/things you might do in the future? Thank you for your time. Those are all the questions we have for you today! Please feel free to stay and let me know if you have any questions or concerns at this time. 6 USDA Workshop Protocol - Sri Lanka 196 | Annexes  | Annex VI — Data Collection Instruments USDA Interview Protocol for Farmers Welcome and introductions Hello, thank you for meeting with me today, my name is ______________ and I work for Eco Lanka. I am working with the University of Minnesota and The Improve Group based out of Minnesota, USA. The University of Minnesota and The Improve Group a St. Paul-based evaluation firm have been contracted by the United States Department of Agriculture to complete a study aimed to understand what activities, practices, and impacts have continued among recipient communities after funding for the USDA/FFPr project ceased. We hope to learn for example, if people have continued to do the things they learned how to do as part of the project. I will take notes during our conversation and combine them with responses from other people we interview. Your individual responses will be kept confidential. We won’t tie your name to your comments in our reports. There are no right or wrong answers! We are hoping to hear your perspective and get your opinion. Finally, I would like to emphasize that you won’t receive any benefits from participating in this activity. Your participation is voluntary, so please feel free to skip any questions at any time. Context (If need a reminder of the project): The World Vision (WV) project took place in four regions of Sri Lanka. The majority of project beneficiaries were farmers who faced high levels of poverty and had (to some extent) established agricultural associations. Activities intended to build farmer capacity to use Integrated Farming and Sustainable Agriculture (IFSA) and provided inputs that would improve their productivity over time. The project also included activities aimed to create or enhance market access. Do you have any questions for me? 1. Please share with us your name, and your position or role on the USDA/FFPr project. Pre-existing conditions 2. We are interested in understanding how things are the same or different for farmers and organizations as a result of the project. Think back to before the project, around 2005, what were conditions like? (historical pointers include the 2004 tsunami, split of the LTTE, and heightening of the civil war.) 3. Think of some threats or challenges that made your work especially challenging at that time. What were they? Prompt for possible conditions; business, economic, policy, infrastructure, environment. a. To what extent are these still present in your life today? do they affect you in the same way now, as they did then? 7 USDA Sri Lanka Interview Protocol - Lead Farmers Annexes | Annex VI — Data Collection Instruments | 197 4. What did you feel was going well for farmers’ around 2005? What do you think helped them to be successful agricultural producers? Prompt for possible conditions; business, economic, policy, infrastructure, a. In what ways are the things that were going well still present today? b. How do they affect you now? 5. (If environment has not been mentioned) How has the environment changed since 2005? Prompt for: biophysical factors weather, soils, biodiversity, crops, etc. a. How have these environmental conditions influenced your agricultural production? Sustainability 6. What did you learn from the project that you still use today? (Prompt for specific practices, methods, or tools) 7. What has helped you continue applying the practices (tools or methods) today? a. Prompt for how valuable they did/not find the things they learned and accessed in the project. 8. What didn’t you continue with, what has kept you from doing so? 9. What changes have you seen in your life and community as a result of your continued application of the things you learned and accessed through World Vision? 10. What is different in your life now, as a result of the project? a. To what extent do you believe other farmers in the project had these same experiences? 11. What do you think your life would be like today if the project had not been implemented? 12. What is worse for yourself because of this project? What about for other farmers? 13. If you could go back to the beginning of the project and tell World Vision what to do, what would you recommend? a. What would this address? Equity We want to understand who was and was not included in this project, and the extent to which those who may have been excluded were further disadvantaged. 14. When you think of the farmers in your region, who do you believe are the worst off or face the greatest challenges? (prompt for women, children, the landless, religious minorities, tribes, etc.) 15. What barriers did they face that were different from other farmers? a. "how does (your) religion affect a farmers (your) ability to continue practices or access resources?" 16. To what extent were they impacted by this project? Were they beneficiaries? 17. In what ways has their situation changed, or not? 18. What would their situation look like now if the project had not existed? 7 USDA Sri Lanka Interview Protocol - Lead Farmers 198 | Annexes  | Annex VI — Data Collection Instruments Shocks 19. We suspect that shocks have influenced peoples’ ability to continue using their knowledge and practices gained from the project. Have you experienced any shocks? If so, how has that impacted your organization? 20. How have these shocks impacted communities? Prompt for: continuation of project practices and impacts 21. How have you responded to these shocks? 22. In what ways did the project help you respond to these shocks? Closing These are all of the questions I have for you. Do you have any questions for me? Is there anything that you would like to add that I didn't ask about? Thank you so much for your time and thoughtful responses! We are excited to explore this project and hope to be able to share with you what we learn. Please fill the following debrief form shortly after the interview USDA Debrief Form Country: Village/Town: Date: Your name: Method: ___________________________________________________________________________ If you had to give the interaction a headline and summary sentences, what would it be? (Headline should be one short phrase, followed with 2-3 summary statements) What information came up that was surprising? 7 USDA Sri Lanka Interview Protocol - Lead Farmers Annexes | Annex VI — Data Collection Instruments | 199 What information came up that was not surprising? What are some themes that are beginning to emerge for you? What codes would you suggest for the code tree, keywords or key terms? What concerns, or issues were raised that should be communicated to Sara and/or Britta? What are some things that need to be followed up on (community questions, etc.)? Anything else you can think of? 7 USDA Sri Lanka Interview Protocol - Lead Farmers 200 | Annexes  | Annex VI — Data Collection Instruments FIELD OBSERVATION NOTES 1 Date Time Name Gender Role In Community Role In USDA Project Location Data collected by: Welcome and Introduction I will take notes during our conversation and combine them with responses from other people we visit. Would you allow us to take photos of some of the things we talk about today? Including farmland, irrigation systems, technologies and tools, plants and seeds, your home and or you or your family members. Is there anything that we should NOT take a photo of? (get verbal consent, and confirmation that they understand. Assure them that at any time they do not want a photo taken we will not do so.) Field Observation Which activities and trainings does the beneficiary recall participating in? *See full details in the interview notes. What items were they were provided with by the project? Examples: seedlings, fertilizer, irrigation system, pond, animals, hand tools What changes have you seen in your life and in your community as a result doing the practices that you mentioned and showed us? Did you take photos? If yes, insert 2-5 with your typed notes in this word document. Photos 8 observation template Annexes | Annex VI — Data Collection Instruments | 201 1 Sources of Icons and images: Freepik.com, el agricultor.com, and the Sri Lanka Mirror. Examples of workshop participant Image Grouping© data collection sheets . Image Grouping© is a participatory mixed-methods data collection tool that helps increase engagement in research, evaluation, needs assessments, and other kinds of data collection. Image Grouping© is used to overcome literacy barriers, particularly in locations where local languages are not commonly written or participants likely have a low level of formal education. As a mixed-methods approach, Image Grouping© can provide both qualitative and quantitative data; an initial activity uses an image-based tool whereby participants use stickers to respond to survey-like questions. See the workshop protocols for exactly how each sheet was used. Image Grouping© participants were selected using a stratified convenience sample. This allowed the evaluation team to select groups based on additional hypotheses relevant to sustainability, such as the relationship between physical distance to markets and sustained outcomes. All Image Grouping© participants are farmers. Each site utilized slightly different criteria for the selection but included attention to: diversity of locations, variety of activity exposure during the project, and a balance of genders. 9 example Participants sheets 202 | Annexes  | Annex VI — Data Collection Instruments 2 Sources of Icons and images: Freepik.com, el agricultor.com, and the Sri Lanka Mirror. Training in short-term production and animal husbandry Training in long and medium-term production Providing goods and products Training in production Training in farm business Short-term crop production Animal husbandry Soil fertilization & irrigation Post-harvest technologies Establishing farmer marketing groups Processing Water tanks and storage Livestock and fish Bee keeping and maintenance Bees and bee hives Soil care and conservation Long and medium￾term tree production Training in soil management and irrigation Long and medium￾term tree production Water tanks and storage Business planning Marketing 9 example Participants sheets Annexes | Annex VI — Data Collection Instruments | 203 3 Sources of Icons and images: Freepik.com, el agricultor.com, and the Sri Lanka Mirror. Training in short-term production and animal husbandry Training in long and medium-term production Providing goods and products Training in production Training in farm business Short-term crop production Animal husbandry Soil fertilization & irrigation Post-harvest technologies Establishing farmer marketing groups Processing Water tanks and storage Livestock and fish Bee keeping and maintenance Bees and bee hives Soil care and conservation Long and medium￾term tree production Training in soil management and irrigation Long and medium￾term tree production Water tanks and storage Business planning Marketing 9 example Participants sheets 204 | Annexes  | Annex VI — Data Collection Instruments 4 Sources of Icons and images: Freepik.com, el agricultor.com, and the Sri Lanka Mirror. Training in short-term production and animal husbandry Training in long and medium-term production Providing goods and products Training in production Training in farm business Short-term crop production Animal husbandry Soil fertilization & irrigation Post-harvest technologies Establishing farmer marketing groups Processing Water tanks and storage Livestock and fish Bee keeping and maintenance Bees and bee hives Soil care and conservation Long and medium￾term tree production Training in soil management and irrigation Long and medium￾term tree production Water tanks and storage Business planning Marketing 9 example Participants sheets Annexes | Annex VI — Data Collection Instruments | 205 5 Sources of Icons and images: Freepik.com, el agricultor.com, and the Sri Lanka Mirror. Infrastructure (transportation, roads) Knowledge Relationships Motivation Communication Existing conditions Money Training and technical support 9 example Participants sheets 206 | Annexes  | Annex VI — Data Collection Instruments 6 Sources of Icons and images: Freepik.com, el agricultor.com, and the Sri Lanka Mirror. Infrastructure (transportation, roads) Knowledge Relationships Motivation Communication Existing conditions Money Training and technical support 9 example Participants sheets