COVER PHOTO CREDIT: ACACIO AMARAL, SI TIMOR-LESTE TEAM USAID/TIMOR-LESTE AVANSA AGRIKULTURA PROJECT FINAL PERFORMANCE EVALUATION FINAL REPORT December 2020 This publication was prepared independently by Social Impact, Inc. at the request of the United States Agency for International Development. USAID/TIMOR-LESTE AVANSA AGRIKULTURA PROJECT FINAL PERFORMANCE EVALUATION FINAL REPORT December 2020 Submitted to: Cândido da Conceição, Contracting Officer’s Representative, USAID/Timor-Leste Contract No.: AID-486-I-14-00001 / Task Order No.: AID-472-TO-15-00003 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ACKNOWLEDGMENTS The authors of this report would like to thank the many individuals who provided technical input and advice during the development of the USAID/Timor-Leste Avansa Agrikultura Project Final Performance Evaluation. The study was a significant undertaking, and its completion would not have been possible without all those who graciously shared their time, insights, and recommendations. We especially appreciated the support and guidance of the USAID/Timor-Leste staff including Cândido da Conceição, David Howlett, and Milca Baptista. Additional thanks go to the Avansa Agrikultura Project senior management team including Paul Mason, Osorio C. de Rogo, Renato Zrnic, Anne Devine, Trevor Carbo, and Mary Beggs. We extend a final note of sincere gratitude to the many farmers and collectors, government officials and businesspeople, students, teachers, and other Timorese engaged in the horticulture sector who generously offered their time to speak candidly with our team. The Evaluation Team Andrew Carmona, Washington, DC Acacio Amaral, Timor-Leste Gabriela Leite Soares, Timor-Leste Zenny Correia, Timor-Leste Nazario Dos Santos, Timor-Leste Recommended Citation: Andrew Carmona, Acacio Amaral, Zenny Correia, Gabriela Leite Soares, Nazario Dos Santos. USAID/Timor-Leste Avansa Agrikultura Project Final Performance Evaluation. Prepared by Social Impact. 2020. CONTENTS 1. EXECUTIVE SUMMARY I 2. INTRODUCTION AND BACKGROUND 1 3. EVALUATION PURPOSE AND QUESTIONS 4 4. EVALUATION DESIGN AND METHODOLOGY 6 5. FINDINGS AND CONCLUSIONS 12 EQ1: HOUSEHOLD-LEVEL IMPACTS 12 EQ2: MARKET-LEVEL IMPACTS 22 EQ3: INDIVIDUAL-LEVEL IMPACTS 33 EQ4: INTEGRATED APPROACH 42 EQ5: SUSTAINABILITY OUTLOOK 45 6. RECOMMENDATIONS 59 7. ANNEXES 63 ANNEX I: PERFORMANCE EVALUATION SCOPE OF WORK 63 ANNEX II: EVALUATION DESIGN AND METHODS 68 ANNEX III: INFORMATION SOURCES 70 ANNEX IV: DATA COLLECTION INSTRUMENTS 72 ANNEX V: STATEMENTS OF DIFFERENCE 112 ANNEX VI: EVALUATION TEAM MEMBERS 113 ANNEX VII: DISCLOSURES OF CONFLICT OF INTEREST 114 TABLE OF TABLES Table 1: Endline EQs......................................................................................................................................................... 4 Table 2: Final Sample Achieved ...................................................................................................................................... 8 Table 3: Limitations and Mitigation Strategies.......................................................................................................... 10 Table 4: Daily per capita expenditure – 2015 prices.............................................................................................. 16 Table 5: Value of savings and productive goods by asset type ............................................................................. 17 TABLE OF FIGURES Figure 1: Percent of respondents by stakeholder group indicating a specific usage of income.................... 15 Figure 2: The Horticulture Value Chain in Timor-Leste ....................................................................................... 22 Figure 3: Percent of farmers or farmer groups indicating a market link to another value chain actor...... 24 Figure 4: Percent of beneficiaries mentioning the use of new technologies in horticulture activities........ 37 ACRONYMS & KEY TERMS AAP Avansa Agrikultura Project CDCS Country Development Cooperation Strategy COVID-19 Coronavirus Disease 2019 DAC Developing Agricultural Communities DEC Development Experience Clearinghouse DO Development Objective EQ Evaluation Question ET Evaluation Team FCR Findings, Conclusions, and Recommendations FGD Focus Group Discussion GDP Gross Domestic Product GESI Gender and Social Inclusion GI Group Interview GIZ Deutsche Gesellschaft für Internationale Zusammenarbeit GNI Gross National Income GOTL Government of Timor-Leste HIAM Health Hamutuk Ita Ajuda Malu Health HDI Human Development Index IGA Income-generating Activities INDIMO National Training Institute of Timor-Leste IP Implementing Partner KII Key Informant Interview MAD Minimum Acceptable Diet MAFF Ministry of Agriculture, Forestry, and Fisheries MCI Ministry of Commerce and Industry MFI Microfinance Institution MOE Ministry of Education MOU Memorandum of Understanding NGO Non-Governmental Organization PE Performance Evaluation SALT Sloped Agricultural Land Technology SI Social Impact, Inc. S&L Savings and Loans SME Small and Medium Enterprise TL Team Leader TOMAK To’os ba Moris Di’ak / Farming for Prosperity TVET Technical and Vocational Education and Training UN United Nations UNDP United Nations Development Program US United States USA United States of America USAID United States Agency for International Development WASH Water, Sanitation, and Hygiene WEIA Women’s Empowerment in Agriculture WFP World Food Program WHO World Health Organization i | AVANSA ENDLINE REPORT USAID.GOV 1. EXECUTIVE SUMMARY INTRODUCTION AND BACKGROUND The United States Agency for International Development (USAID)/Timor-Leste supported the five and a half year, $21.2 million Avansa Agrikultura Project, heretofore referred to as “Avansa Agrikultura.” The project sought to support increased economic activity and growth for Timorese citizens in targeted rural communities and municipalities. It was implemented by Cardno Emerging Markets USA, Ltd., commencing on April 1, 2015, and completed on December 15, 2020. In mid-2020, USAID/Timor-Leste contracted Social Impact, Inc. (SI) to conduct a Final Performance Evaluation (PE) of the Avansa Agrikultura Project. The objectives of the evaluation were to assess the change in status of the beneficiaries from the beginning of the Avansa Agrikultura Project to the end of the project, acknowledging that causality cannot be established given the evaluation methodology, and to generate lessons learned on the Avansa Agrikultura Project interventions. Intended users of the evaluation are USAID, the Government of Timor-Leste (GOTL), and other donors. METHODOLOGY The PE was guided by five key evaluation questions (EQs): EQ1. Household: To what extent have project activities contributed to improved outcomes of target beneficiary households? EQ2. Market: To what extent has the project strengthened private sector market linkages across the value chain for target crops? EQ3. Individual: To what extent have project activities contributed to changes in reported self￾efficacy among beneficiaries, particularly females and youth? EQ4. Integrated Approach: What interventions, or combination of interventions, did participants view as most relevant or valuable? Why? EQ5. Sustainability: What is the current outlook for sustainability of key Avansa Agrikultura Project interventions? SI used a mixed-methods approach to answer the EQs, which included a desk review of existing project monitoring documents, data, and reports; supplementary secondary quantitative data analysis; and remote and in-person qualitative interviews and focus groups with project and IP staff, farmer groups, national and municipal level government, input suppliers, collectors, USAID staff, agribusiness processors, supermarkets, vocational training centers, and microfinance institutions. From July to September 2020, the evaluation team conducted a total of 74 unique data collection events with 135 individuals in five municipalities – Dili, Aileu, Ainaro, Ermera, and Bobonaro – and remotely from the United States. Analysis of data was conducted using Dedoose software and basic pattern and content analysis. Findings were supplemented and triangulated with a quantitative survey of participating households happening in parallel to but separate from the PE. The evaluation faced limitations presented by the Coronavirus Disease 2019 (COVID-19) pandemic which resulted in some remote data collection, oversight, and evaluation management. For in-person data collection, Timor-Leste based team members implemented strict COVID-19 mitigation protocols. USAID.GOV AVANSA ENDLINE REPORT | ii FINDINGS AND CONCLUSIONS EQ1: Household-level impacts. Avansa Agrikultura interventions contributed to improved farm productivity, increased, and diversified income, and increased savings among participating farmer households. Avansa Agrikultura interventions with farmer groups likely contributed substantially to improved food security and nutritional indicators of participating farmers and their households. The impact of Avansa Agrikultura interventions on nutritional knowledge and attitudes was unclear. EQ2: Market-level impacts. Avansa Agrikultura-facilitated introductions between new farmers, collectors, and supermarkets which catalyzed strong linkages and led to positive market disruption. Collector networks strengthened by Avansa Agrikultura can fill critical service gaps where markets, input suppliers, and microfinance institutions’ (MFI) reach is limited. Access to finance through Kaebauk loans and Avansa Agrikultura cost-share is slowly transforming farmer mentality and ability in agriculture. Imperfect information and lack of coordination among and between farmers and collectors causes market inefficiencies and farmer-supermarket discord. Persistent issues with seed supply and price limit the full potential of horticulture interventions. Benefits from strengthened market linkages diminish as distance from Dili increases. EQ3: Individual-level impacts. Comprehensive training and market access facilitation by Avansa Agrikultura resulted in more skilled, financially aware, and confident farmers. Avansa Agrikultura support was pivotal in enabling vocational schools to operate at a high capacity, though it is unclear to what extent schools will have the capacity to operate post-Avansa Agrikultura. Aside from a few key examples among early adopters, the project’s impact on youth was muted in the short-term. Though women benefited from Avansa Agrikultura interventions, activities did not appear to be based on a gender-specific need or gap. EQ4: Integrated approach. Avansa Agrikultura’s shift to an integrated package of interventions to farmers and whole value chain focus after the midline evaluation ensured a holistic approach to improving outcomes along the horticulture value chain. Avansa Agrikultura staff and beneficiaries diverge somewhat in their perceptions of most valuable interventions. EQ5: Sustainability outlook. Horticulture farming is rooted among Avansa Agrikultura farmers given real changes in earning power and established linkages with markets. Supply and demand mismatches between farmers and supermarkets will continue in the absence of concerted inter-farmer and collector-market coordination and information flows. Farmer groups may eventually dissolve as farmers increase their independence and bargaining power, but groups could remain elastic to respond to follow-on donor interventions. Loss of Avansa Agrikultura technical assistance may slowly lead to reduced production, though there is opportunity for local actors to follow in Avansa Agrikultura’s wake. The triangle model (of agricultural loans) is active, impactful, and attractive to outside donors, yet the reliance on a sole provider of credit is a precarious risk. Collectors will continue to fill critical gaps in the value chain particularly farther out from Dili. Small scale processing is possible in Timor-Leste but will need to be aligned to a viable market demand to succeed. Inconsistent seed supply combined with low road quality and water scarcity in some areas continues to limit the full potential of horticulture farming in Timor-Leste. Attracting youth to agriculture will require a perception and mindset change that agriculture is a viable employment/livelihood. Avansa Agrikultura programming is increasingly expanding the role of women at the household and community level. Aside from a few potential partnerships, the public sector cannot be counted on to support or take up Avansa Agrikultura interventions across the value chain for horticulture, but there is ample opportunity for interested iii | AVANSA ENDLINE REPORT USAID.GOV donors to do so. There are additional opportunities to link horticulture farmers to buyers outside of the supermarket sector and explore targeting products other than horticulture among farmers. RECOMMENDATIONS For USAID: • Partner with a second local financial institution. Continue discussions with donors, local banks, and/or MFIs (other than Kaebauk) to lend to the agricultural sector and participate in the triangle model. Showcase recent successes from Avansa Agrikultura to help persuade reticent financial institutions to participate and help shape their expectations around working with the agricultural sector. • Conduct a gender and social inclusion (GESI) analysis and/or tailor future activities to findings of a GESI analysis. Following on results from a GESI analysis, use participatory approaches to ensure that women and youth are involved in planning of future programming. • Strategically partner with already existing and interested donors to sustain critical pieces of the value chain. Consider public sector engagement with institutions that show promise for meaningful collaboration. Consider targeting donor agencies and organizations to support easy-win areas such as cost-share grants, support to vocational training schools, and youth mentoring in agribusiness. Partner with institutions such as INDIMO, import regulatory bodies, and others that have capacity and willingness for policy reform. For IPs: • Improve seed supply through import regulation reform and increased use of collector networks. Work with the national regulatory bodies and other projects and donors (such as USAID) working on customs and border work to ease seed import restrictions. Engage collector networks to formally transport seed to municipal shops as a means to strengthen the consignment model. Work with input suppliers to play a part (e.g. logistics, transport, packaging, sales, etc.) in creation of a viable seed market. Consider engaging universities for research into production of local seed varieties. • Strengthen municipal horticulture associations to provide information, coordination, and technical assistance to farmers. Give technical assistance to help associations with recruitment, staffing, planning, accreditation, and initial resources to get off the ground. Connect farmer groups and collectors with horticulture associations to disseminate real-time information, coordination, and assistance via messaging and social media. Ensure regular cross-municipality association meetings to disseminate best practices and promote synergies. • Link collectors and input suppliers to ensure a consistent input supply chain to farther municipalities. Formalize linkages between collectors and Dili-based and municipality￾based suppliers via memorandums of understanding (MOUs) and contracts. Incentivize innovative collector-based approaches to servicing hard to reach localities via grants in areas such as cold chain. Continue financial literacy and business training to collectors. • Consider delivering nutrition and water, sanitation, and hygiene (WASH)-specific programming to women within existing savings and loans (S&L) groups. Continue to work with local organizations to deliver contextually relevant behavior change interventions that integrate nutrition and WASH and place an emphasis on the importance of nutrition-related decisions and purchases in the household. Start with all-women S&L groups and expand to USAID.GOV AVANSA ENDLINE REPORT | iv mixed gender. Consider expansion and incentivization of all-women S&L groups to ensure adequate geographic coverage. • Utilize youth champions and create a youth fund to encourage youth to enter agribusiness. Identify and highlight successful youth agribusinesses at vocational schools as case studies and in field trainings. Set aside grant money for youth-only funding competitions that award innovative agribusiness ideas. Continue partnering with the United Nations Development Program (UNDP) youth hub to offer targeted support to youth who express interest in the sector. • Conduct a targeted communication campaign to support the “local” food movement. Riding the coattails of the newly emerging movement in Timor-Leste, target expatriate and other Dili-based communities through print and digital media and events to support locally produced foods. For USAID and IPs: • Continue to refine geographic specific and appropriate interventions. Comprehensively assess what works where for horticulture, considering geography, climate and weather, water access, road accessibility, and farmer disposition. Target specific crops and inputs where they will have the largest cost-benefit. Strengthen collector networks in further municipalities. Where supermarkets are too far, link farmers to other nearby sectors (tourism, extractive, public schools) that demand horticulture products. • Diversify horticulture buyers by expanding to different sectors (tourism, extractive, etc.) and diversify agricultural product by including trainings on fruit and livestock. Assess other industries for willingness and viability of partnerships with horticulture producers. Integrate trainings that focus on high-value alternatives to horticulture such as fruit and livestock rearing among farmers that have the capacity and willingness to do so. 1 | AVANSA ENDLINE REPORT USAID.GOV 2. INTRODUCTION AND BACKGROUND INTRODUCTION The United States Agency for International Development (USAID)/Timor-Leste supported the five and a half year, $21.2 million Avansa Agrikultura Project, hereafter known as “Avansa Agrikultura.” Avansa Agrikultura sought to support increased economic activity and growth for Timorese citizens in targeted rural communities and municipalities. Avansa Agrikultura was implemented by Cardno Emerging Markets USA, Ltd., and sub-contractors Hamutuk Ita Ajuda Malu Health (HIAM Health), the Norman Borlaug Institute for International Agriculture/Texas, and Resonance Global (formerly SSG Advisors, LLC.). The contract with Cardno was signed on April 1, 2015, and the project was completed on December 15, 2020. USAID/Timor-Leste contracted Social Impact, Inc. (SI) through the Avansa Monitoring and Evaluation contract to conduct a Final Performance Evaluation (PE) of the Avansa Agrikultura Project. The objectives of the evaluation are: 1. To assess the change in status of the beneficiaries from the beginning of the Avansa Agrikultura Project to the end of the project, acknowledging that causality cannot be established given the evaluation methodology. 2. To generate lessons learned on the Avansa Agrikultura Project interventions, and the pros/cons of an integrated approach, for use by USAID, the Government of Timor-Leste, and other donors. This report starts with Section 2, which lays out context and background on agriculture in Timor-Leste, followed by a description of USAID’s Avansa Agrikultura Project. Section 3 briefly touches upon the evaluation purpose and evaluation questions, while Section 4 discusses the evaluation methodology, including data sources and methods, sampling, data analysis approach, and limitations. In Sections 5, 6, and 7, the report analyzes findings, conclusions, and recommendations, respectively, of the study. THE DEVELOPMENT PROBLEM The agriculture sector’s performance since Timor-Leste’s Independence Restoration in 2002 has been disappointing, with national accounts data showing a decline in the mean value of agricultural production per capita from 2000 to 2013.1 At the same time, approximately 70 percent of the country’s citizens are involved in the agriculture sector, and most farmers have plot sizes of less than one hectare. Agriculture production systems in Timor-Leste are among the least developed in the world. Subsistence systems dominate all subsectors, and most farming households are highly risk averse. Child nutritional status is still very low. According to World Bank data, in 2015 Timor-Leste’s Gross National Income (GNI) per capita was $2,180 and Timor-Leste was considered a middle-income country.2 However, since then, GNI per capita has fallen to $1,810 (2018) and Timor-Leste is now a lower middle-income country. Poverty is high in Timor-Leste. The World Bank also estimates that in 2014, 41.8 percent of the population lived below the national poverty line. The 2016 United Nations Development Program (UNDP) Human 1 Government of Timor-Leste, 2015. 2 Current prices, World Bank Atlas method. http://data.worldbank.org/country/timor-leste. USAID.GOV AVANSA ENDLINE REPORT | 2 Development Report ranked Timor-Leste 133 out of 188 countries on the Human Development Index (HDI), which takes into account GNI per capita, education, and life expectancy. By 2019, Timor-Leste had only improved by two rankings, to 131 out of 189. Approximately 70 percent of the country’s citizens are involved in the agriculture sector, with agriculture contributing 16 percent of the country’s non-oil output in 2017.3 However, the agriculture sector’s performance since Timor-Leste’s independence in 2002 has been disappointing, with national accounts data showing a decline in the mean value of agricultural production per capita from 2000 to 2013.4 Poor agricultural production practices and a lack of income to supplement home-grown products has contributed to high levels of food insecurity and malnutrition throughout Timor-Leste. Low agricultural productivity is a key cause of food insecurity among rural households and contributes to high rates of childhood malnutrition. Timor-Leste has a very young population. Households in Timor-Leste have an average of 5.3 members and 41 percent of the household population is under age 15.5 The nutritional status of Timorese children is very low. While the prevalence of wasting in children under five years was 11 percent in 2016, the percentage of stunting among children of the same age group was 50.2 percent.6 There is no significant difference in malnutrition between girls and boys, but malnutrition is much higher among children in rural than urban areas.7 In the Timor-Leste agriculture sector, women play a vital role as small-scale commercial farmers and agribusiness entrepreneurs, with 66 percent of employed women in the country being self-employed farmers. However, a 2018 analysis of the gender gap in the Timor-Leste agricultural sector found women produce on average 31 percent less than men on similar land. Differences in access to factors of production, such as women’s lack of access to hired labor, agricultural implements, farming networks and the ability to produce cash crops, are the driving factors behind this divide.8 PROJECT BACKGROUND Between 2010 and 2015, USAID funded the Developing Agricultural Communities (DAC) project, an $8.3 million, 4.5 year activity implemented by Developing Alternatives Inc. DAC supported 37 farmer groups through provision of technical support, horticulture inputs, and linkages with markets to sell new, higher quality vegetables that were intended to replace imports. As the DAC project neared its final year, in 2013, USAID/Timor-Leste undertook a comprehensive assessment of opportunities for accelerating inclusive economic growth in Timor-Leste.9 The assessment found a high unmet local demand for vegetables, supporting the possibility of significant results in the near term. The report indicated that investments in upgrading the horticulture value chain, 3 World Bank (2019). Timor-Leste Economic Report 2019. 4 Government of Timor-Leste, 2015. 5 General Directorate of Statistics, Ministry of Planning and Finance and Ministry of Health. 2016 Timor-Leste Demographic and Health Survey. 6 International Food Policy Research Institute (IFPRI), 2016. Wasting reflects low weight for height and indicates acute undernutrition. Stunting reflects low height for age and indicates chronic undernutrition. 7 Asian Development Bank (2014), Timor-Leste Country Gender Assessment. 8 World Bank (2018). Women Farmers in Timor-Leste: Bridging the Gender Gap in Agricultural Productivity. Policy Brief Issue 4: November 2018. 9 USAID (2019). Timor-Leste Economic Growth and Development. https://www.usaid.gov/timor-leste/economic￾growth-and-trade. 3 | AVANSA ENDLINE REPORT USAID.GOV specifically vegetables and fruits, offered the Mission’s best opportunity to achieve impact and results in the five-year time frame of its current strategic plan. As a follow-on to the DAC project, the USAID’s Avansa Agrikultura Project was predicated on the development hypothesis that if USAID/Timor-Leste increases productivity within the horticulture value chain and increases the ability of Timor-Leste’s citizens to engage in economic activities, then sustainable and inclusive economic growth would be accelerated in the agriculture sector. The overall purpose of the USAID Avansa Agrikultura Project was to achieve sustainable and inclusive economic growth accelerated in the horticulture sector. At the end of the project, 7,700 households (reduced from 33,000 at project inception) were expected to be earning higher incomes from improved sustainable horticulture production and market linkages. The project supported the Development Objective (DO) of the USAID/Timor-Leste Mission, institutional and human capacity for development strengthened to improve the lives of Timor­Leste's citizens. The project reported on one indicator that contributed to measuring the achievement of this high-level objective. This purpose was to be achieved through two project sub-purposes and two corresponding outputs. The first sub-purpose worked on strengthening market linkages and stimulating private sector investment in the targeted horticultural value chains. The second sub-purpose sought to improve productivity and natural resource management. Sub-Purpose 1: Improved Ability of Timorese Citizens to Engage in Economic Activities.  Output 1: Market Linkages Improved and Expanded Across the Horticultural Value Chain Sub-Purpose 2: Increased Productivity of Selected Horticultural Value Chains.  Output 2: Sustainable Horticulture Productivity Improved at the Farm Level To improve the health status of beneficiaries and align with the Global Food Security Initiative, the project implemented a nutrition-sensitive agricultural development approach. Activities to improve nutrition were integrated across the two project Outputs. Avansa Agrikultura sought to increase the availability of nutritious food to farmers and provide training on the importance of nutrition. The project sought to work with women and youth to increase their ability to participate in the horticulture sector. Activities included supporting women and youth as agribusiness entrepreneurs and promoting women as leaders of farmer groups. Through the promotion of sustainable production practices, increased functionality of farmer groups and associations, improved market linkages, and increased access to quality agricultural inputs, services and finance, the project sought to stimulate increased economic activity and growth in targeted rural communities and municipalities. Sustainability of these interventions would be achieved as the project worked to maintain sound policies and a supportive enabling environment. Efforts to increase resilience to climate change and improve farm-level natural resource management further built the foundation for sustainability. The project worked in six municipalities: Aileu, Ainaro, Bobonaro, Dili, Ermera, and Liquica and covered 68 villages (sucos). At the end, it was expected that about 33,000 people in 250 communities would be earning higher incomes, showing improved nutrition practices, and managing their economic assets more effectively. USAID.GOV AVANSA ENDLINE REPORT | 4 3. EVALUATION PURPOSE AND QUESTIONS EVALUATION OBJECTIVES & EVALUATION QUESTIONS The endline evaluation has two objectives: 1. To assess the change in status of the beneficiaries from the beginning of the Avansa Agrikultura Project to the end of the project, acknowledging that causality cannot be established given the evaluation methodology. 2. To generate lessons learned on the Avansa Agrikultura Project interventions, and the pros/cons of an integrated approach, for use by USAID, the Government of Timor-Leste, and other donors. The purpose of the evaluation is to document impacts and lessons learned of the Avansa Agrikultura Project to understand best practices for future, follow-on work. The primary users of this evaluation are intended to be USAID, the Government of Timor-Leste (GOTL), current and future implementing partners (IPs) of agriculture-related activities, and other interested donors. Beneficiary stakeholders of the Avansa Agrikultura Project were participating farmers and farmer groups, collectors, input suppliers, vocational school students, teachers, and staff, microfinance institutions (MFIs), small and medium enterprises (SMEs) in agribusiness, supermarkets, and a number of Timor-Leste government institutions. SI collaborated with the Avansa Agrikultura project and USAID to develop evaluation questions (EQs) for this evaluation, as part of the evaluation methodology. These questions allowed SI to understand stakeholder perceptions, challenges faced, and lessons learned. The EQs are listed in Table 1 below. Table 1: Endline EQs OBJECTIVE 1: CHANGE IN STATUS OF BENEFICIARIES EQ1 Household: To what extent have project activities contributed to improved outcomes of target beneficiary households? Which interventions were the most effective in doing so and why? EQ2 Market: To what extent has the project strengthened private sector market linkages across the value chain for target crops? Which interventions were the most effective in doing so and why? EQ3 Individual: To what extent have project activities contributed to changes in reported self-efficacy10 among beneficiaries, particularly females and youth? Which interventions were the most effective in doing so and why? OBJECTIVE 2: LESSONS LEARNED EQ4 Integrated Approach: What interventions, or combination of interventions, did participants view as most relevant or valuable? Why? EQ5 Sustainability: What is the current outlook for sustainability of key Avansa Agrikultura Project interventions? 10 Self-efficacy refers to an individual's belief in his or her capacity to execute behaviors necessary to produce specific performance attainments. Self-efficacy reflects confidence in the ability to exert control over one's own motivation, behavior, and social environment. American Psychological Association. https://www.apa.org/pi/aids/resources/education/self-efficacy 5 | AVANSA ENDLINE REPORT USAID.GOV For EQ1, specific focus areas at the household level were farm productivity, financial indicators (including assets, income, savings, and debt), and food security (including consumption and nutrition). For EQ2, the evaluation team (ET) looked at linkages among actors across all levels of the value chain, such as farmer groups, collectors, retail markets, input suppliers, processors, and access to finance. For EQ3, the ET looked at the effect of Avansa Agrikultura-supported trainings on farmer’s knowledge and skills as well as confidence to succeed in their profession. EQ4 and EQ5 aimed to generate lessons learned related to integration, perceived comparative value of the project’s interventions, and the outlook for sustainability of these interventions. USAID.GOV AVANSA ENDLINE REPORT | 6 4. EVALUATION DESIGN AND METHODOLOGY 4.1 DATA SOURCES AND METHODS SI used a mixed-methods approach to answer the EQs which included a desk review of existing project monitoring documents, data, and reports; supplementary secondary quantitative data analysis; and remote and in-person qualitative interviews and focus groups with USAID, Avansa Agrikultura staff, implementing partner staff, and project beneficiaries.11 DESK REVIEW AND SECONDARY QUANTITATIVE ANALYSIS Prior to data collection, in May and June 2020, the ET reviewed approximately one dozen documents including Avansa Agrikultura annual reports, survey data, monitoring reports, and the baseline and midline evaluation reports (See Annex III: Information Sources for a full list). Results from the desk review informed the overall evaluation design, approach, and development of instruments. Further, the ET was in a unique position to use supplementary data from Avansa Agrikultura’s annual household survey, carried out in parallel to the PE’s data collection work in July 2020. The household survey, carried out among 331 households in 37 sucos, provided pertinent quantitative data related to household-level outcomes on income, savings, consumption, nutrition, and women’s empowerment. The ET compared data from the household survey in Year 5 (2020) to the baseline (2015) and used analysis results to triangulate findings of the PE. Relevant results from the quantitative household survey are presented in this report under EQ1 and EQ3. A report detailing the findings over five years from the annual household survey can be provided upon request. KEY INFORMANT INTERVIEWS AND FOCUS GROUP DISCUSSIONS Between July and September 2020, Timor-Leste based team members of the ET conducted face-to-face data collection events, including key informant interviews (KIIs), group interviews (GIs), and focus group discussions (FGDs), and the Team Leader (TL) conducted remote KIIs from the United States. The team conducted KIIs with Avansa Agrikultura and IP staff, USAID, national and municipal level government officials, heads of farmer groups, input suppliers, collectors, agribusiness processors, supermarkets, and one microfinance institution. KIIs lasted 60–90 minutes and included a facilitator and notetaker. Remote KIIs were conducted via WhatsApp or Microsoft Teams. The in-country team conducted FGDs with farmer beneficiaries, agribusiness processors, and vocational training schools. FGDs lasted 90–120 minutes. The four team members of the ET located in Timor-Leste split into two mixed-gender teams and traded off facilitation and notetaking responsibilities during data collection events. All KIIs involving women, including two all-female FGDs, were facilitated by female members of the evaluation team. Teams utilized semi-structured data collection protocols to guide the interviews and discussions. The ET received approval for this study and the study instruments from the Social Impact Institutional Review Board, a process that included a comprehensive review of Coronavirus Disease 2019 (COVID-19) mitigation measures. 11 In-person interviews and focus groups were completed utilizing strict COVID-19 mitigation procedures. More information on these measures can be found in the Limitations section of this report. 7 | AVANSA ENDLINE REPORT USAID.GOV 4.2 SAMPLING RESPONDENT SELECTION PROCEDURES The ET targeted ten groups of stakeholders: Avansa Agrikultura and IP staff, farmer groups, national and municipal level government, input suppliers, collectors, USAID staff, agribusiness processors, supermarkets, vocational training centers, and microfinance institutions. To the extent possible, these stakeholders were distributed evenly across municipalities. The ET used purposive12 and convenience sampling approaches for all stakeholders outside of farmer groups due to the limited number of potential respondents in the sample pool and COVID-19 related restrictions that limited movement and participation. Using these techniques, the ET identified stakeholders to sample through recommendations and sampling frames provided by the Avansa Agrikultura project team and USAID as well as snowball sampling.13 For FGDs, the ET used lists of farmer groups provided by Avansa Agrikultura and randomly selected approximately 3–4 groups (with backup options) per municipality to interview, including a target of two all-female and two all-youth groups. The ET aimed to achieve as balanced a sample as possible, stratified across the five municipalities, with adequate gender and youth representation. SAMPLE ACHIEVED The ET conducted a total of 74 unique data collection events with 135 individuals: 56 KIIs with 59 individuals, and 17 group interviews and focus groups with 76 individuals. Due to its location as the seat of government and principal office for Avansa Agrikultura, most data collection events took place in Dili municipality (27 of 74, or 36 percent), with a near equal distribution across the four other municipalities (Aileu – 8 events, Ainaro – 9, Ermera – 10, Bobonaro – 12) and eight interviews conducted remotely with Avansa Agrikultura stakeholders located outside of Timor-Leste. The final distribution across gender amounted to 65 percent male and 35 percent female due, in part, to a high concentration of males among Avansa Agrikultura and IP staff (14 of 19 KIIs) and traditional gender roles that place males more often than females in positions of leadership in Timorese society. Forty-two percent of all data collection events (31 of 74) involved at least one female participant. Just under one in five respondents (25 of 138) was a youth, defined as aged 18–35, of which 40 percent (10 respondents) were female. Of the 17 GIs and FGDs conducted, two were female-only, five male-only, and ten mixed-gender. Of the 18 farmer groups (FGDs) or farmer group heads (KIIs) interviewed, two were all-youth (though a total of three included at least one youth), four all-female, six all-male, and eight mixed gender. Five of 18 farmer groups were close to Dili (defined as being in Dili or Aileu municipalities) and 13 were “far from Dili,” located in Ainaro, Ermera, or Bobonaro. The average size of a group interview or FGD was 4.5 participants, with a median of four, a minimum of three, and a maximum of eight participants. See Table 2 for the sampling breakdown. 12 Purposive sampling is a non-probability sampling technique that relies on the judgment of the researcher to select units of analysis. It is widely used in qualitative research for the identification and selection of information￾rich cases. This involves identifying and selecting individuals or groups of individuals that are especially knowledgeable about or experienced with a phenomenon of interest. 13 Snowball sampling is a research technique in which respondents are asked by the research team to identify other potential respondents for interviews that might not have been known to the researchers ahead of time. USAID.GOV AVANSA ENDLINE REPORT | 8 Table 2: Final Sample Achieved KEY RESPONDENT CATEGORIES DILI AILEU AINARO ERMERA BOBONARO REMOTE TOTAL EVENTS TOTAL PAX Avansa Agrikultura staff 7 1 1 1 1 4 15 15 Implementing Partners 1 3 4 4 USAID Staff 2 1 3 3 National government 3 3 4 Municipal-level government 2 1 2 5 7 Input suppliers 5 1 1 7 7 Supermarkets 4 4 4 Microfinance Institutions/Banks 1 1 1 Agro-input Business Association14 1 1 1 Vocational training schools 1 1 2 1 5 19 Farmer groups15 1 4 2 6 5 18 52 Collectors 1 1 1 2 5 8 Agribusiness processors 2 1 3 11 Total events 27 8 9 10 12 8 74 Total persons 34 19 29 25 20 8 135 Total female 9 12 10 6 7 3 47 Total youth 6 5 5 6 3 0 25 4.3 DATA ANALYSIS ONGOING ANALYSIS AND QUALITY ASSURANCE Over the course of data collection, the in-country team and the TL conducted a total of six weekly debriefs via WhatsApp voice-over-IP phone calls to discuss initial findings. The team subsequently adapted and revised the data collection instruments to account for emerging trends and new areas of interest for probing. Members of the ET took comprehensive notes in Tetum during KIIs and FGDs, translated them to English, and uploaded the notes to a shared Google Drive on a weekly basis. The TL then commented on the notes, clarifying discrepancies or confusion with the local team members via email and WhatsApp. The TL synthesized the finalized notes using a Findings, Conclusions, and Recommendations (FCR) matrix to track emerging themes. The matrix ensured that the ET prepared a systematic and thorough response to each evaluation question, verified that preliminary analysis accounted for gender and social dimensions, identified any gaps where additional clarification or analysis would be necessary, and served as the basis for developing the out-brief and evaluation report. The TL conducted separate meetings with in-country team members to help understand and nuance the findings. Initial results were triangulated with findings from the desk review, secondary quantitative analysis, and across stakeholder interview notes. 14 While the Project did not work directly support this association, insight from staff members of the association contributed important perspective and input to the EQs. The stakeholder interviewed in this event was also included in an input supplier interview; two separate events are counted in the table though only one respondent is tallied which accounts for the slight discrepancy in the total persons interviewed line. 15 Ten farmer group interviews were GIs or FGDs; eight were KIIs with heads of the farmer group only. Three farmer groups were also savings and loans groups and one was also a nutrition group. 9 | AVANSA ENDLINE REPORT USAID.GOV POST-FIELDWORK ANALYSIS Prior to completing the fieldwork, the ET developed a qualitative coding framework based on the evaluation questions, instruments, and emerging themes. After completing the fieldwork, the team revised the coding framework based on themes from the interviews, the FCR matrix, and interview notes. The team subsequently conducted qualitative coding of interview notes using Dedoose software. The ET compared results across stakeholder groups, municipalities, and demographic groups (gender, youth) where appropriate to assess convergence and divergence in perspectives. In addition to coding, the ET conducted basic content and pattern analysis on the finalized notes. GENDER AND SOCIAL INCLUSION ANALYSIS Consistent with USAID evaluation policy and the objectives of the project, SI applied a gender and social inclusion (GESI) lens to the evaluation design and ensured that research activities included female participation, both as team members and respondents. SI employed a dedicated Gender Specialist to work with the team to ensure compliance and gender sensitivity on tools and protocols, data collection, and data analysis, including eliciting GESI-segmented data. EQ3 focuses on the impact of Avansa Agrikultura’s programming on women, youth and other traditionally socially excluded groups, and the ET also disaggregated findings by municipality to better understand geographic inequities of Avansa Agrikultura impact. Potential unintended consequences of the program for these groups were discussed during KIIs and FGDs, and then analyzed to understand any positive or negative consequences. Members of the ET completed Protecting Human Research Participants training as well as USAID’s Gender 101 Training to ensure a thorough understanding of gender policies and gender considerations in evaluation. During data analysis, the ET looked not only looked at stated responses about gender and social inclusion, but divergence and convergence in responses among the different genders. TRIANGULATION AND VALIDATION The ET’s employed analytical triangulation approaches as part of the development of findings and conclusions. Triangulation enabled the ET to cross-verify and cross-validate the findings that emerged from using the above methods and data sources to identify correlations between findings and determine overall program effectiveness. In particular, the ET utilized methodological triangulation to develop parallel protocols with the same or similar questions across its KII and FGDs. This enabled greater data triangulation because each method addressed sub-sets of the same evaluation questions, and their findings could be validated or refuted by the other techniques. Methodological triangulation also enabled the ET to strengthen the potential linkages and accuracy of its data if the results obtained through one method were less conclusive than another method. Finally, where relevant, the ET used secondary quantitative data sources, such as Avansa Agrikultura’s annual household and firm-level surveys, to supplement and triangulate qualitative findings. USAID.GOV AVANSA ENDLINE REPORT | 10 4.4 LIMITATIONS AND BIASES Table 3: Limitations and Mitigation Strategies LIMITATION/ RISK DETAILS MITIGATION STRATEGY Recall Bias Beneficiaries may respond to questions posed by the ET with answers that blend their experiences into a composite memory. Respondents who may have participated in more than one project activity or in similar activities conducted by other implementers may not be able to distinguish between them. Additionally, depending on when beneficiaries participated in the project’s activities, perceptions of events may change over time, and the ability to remember specific details may fade. The ET conducted multiple KIIs and FGDs to triangulate responses and to increase the validity of the evaluation findings. The ET also adapted instruments to probe questions about time periods prior to and during COVID-19. Response Bias Key informants may be motivated to provide responses that would be considered socially desirable or influential in obtaining donor support. A training participant may remark positively about an activity because they would like to attend similar trainings in the future. Respondents may believe that negative evaluation findings could result in reduced assistance. Prior to each KII and FGD, the ET delivered a clear informed consent script to participants which emphasized confidentiality, honesty, and the voluntary nature of the interview. It is expected that reassurances by the qualitative researchers, communicated through the informed consent, encouraged participants to give more accurate and honest answers thereby reducing response bias. Selection Bias Selection bias is an inherent risk when implementers help to facilitate contact with project beneficiaries. The ET worked closely with staff based in the Project Field Office to organize FGDs and KIIs with project beneficiaries. There was a risk that project staff selected the most active, responsive, or engaged beneficiaries, creating the risk that the ET only heard from key informants who reported positive experiences. The ET solicited sampling frames from Avansa Agrikultura and then randomly sampled individuals and groups to be included in interviews (for farmer groups). For KIIs, the ET employed best practice in qualitative interviewing that emphasized non-leading, probing questions to get at the most accurate and clear information. Gender Bias Traditional gender norms and customs regarding the role of men and women in the household, in agriculture, and other related fields may influence the way that men and women answer questions related to gender. Women may be afraid to speak or give their candid opinion in the presence of men. The ET had a mixed-gender local team and a female facilitator led interviews where women were involved, including facilitating all-female FGDs. Further, the ET received a comprehensive Gender 101 training. Attribution Timor-Leste’s agriculture sector has received assistance from USAID and other donors for multiple years. Donors other than Cardno overlapped with similar interventions in the same zone of influence as Cardno. This makes judgments about attribution to the project difficult. The methodology used precludes the ability to attribute Avansa Agrikultura activities to observed outcomes, however, triangulation through qualitative data collection, quantitative comparisons, and secondary sources helped to establish strong association between the two. COVID-19 COVID-19 related mitigation measures such as restrictions on assembling of groups, social distancing, and other measures affected the ET’s ability to conduct work as expected. For in-person data collection, the ET employed social distancing, wore personal protective equipment such as masks, and frequently utilized good hygiene practices such as handwashing and sanitation. For focus groups, the ET convened smaller groups than normal (an average of 4.5 participants) and reduced some FGDs to GIs. 11 | AVANSA ENDLINE REPORT USAID.GOV LIMITATION/ RISK DETAILS MITIGATION STRATEGY Remote data oversight Limitations due to the COVID-19 pandemic prevented non-local ET members from traveling to Timor-Leste for fieldwork. The absence of the TL in Timor-Leste limited the ability to do high quality, frequent data quality oversight. SI conducted weekly check-ins with team members in Timor-Leste (via WhatsApp) to understand the progress of fieldwork, assist with logistical challenges, and examine initial findings. The TL conducted 17 remote interviews with local stakeholders with and without translation support to contribute to local data collection. COVID-19 effects on data COVID-19 has further implications for this evaluation as it represents an external factor with high potential to confound the evaluation’s results. A May 2020 Rapid Food Security Assessment16 commissioned by the Timor-Leste Ministry of Agriculture and Fisheries with the goal of assessing the impact of COVID-19 on food security found, among many things, that most households’ food security had been negatively affected by COVID-19, there was a marked migratory shift of people from urban to rural areas, and the inability to travel to markets and/or closures of markets had resulted in severe economic shocks for many households. Given the evaluation’s focus on outcomes of food security and financial/economic stability, these COVID-19 related shocks have the potential to muddle or bias negatively the overall impact of the project’s interventions. The ET introduced questions into data collection instruments which sought to understand the impact of COVID-19 on beneficiary groups. This helped to tease out the effect of the pandemic on outcomes of interest. Further, the ET included retrospective questions to get at the state of beneficiaries’ outcomes prior to the pandemic. Questions on COVID-19 were posed to different types of respondents and asked about the level of affectedness by COVID-19 for the outcomes of interest. Using this methodology, the ET was better able to understand the associations between levels of affectedness, social groups, and outcomes of interest. 16 Ministry of Agriculture and Fisheries Timor-Leste. Rapid Food Security Assessment 2020: Full Report. Timor￾Leste. 9 June 2020. USAID.GOV AVANSA ENDLINE REPORT | 12 5. FINDINGS AND CONCLUSIONS EQ1: HOUSEHOLD-LEVEL IMPACTS To what extent have project activities contributed to improved outcomes of target beneficiary households? Which interventions were the most effective in doing so and why? To answer EQ1, the ET looked at changes across three principal household-level outcomes: farm productivity, income (including spending and savings habits), and food security (including consumption patterns and nutrition). Throughout EQ1, the ET used results from Avansa Agrikultura’s annual household survey on income and food security related outcomes to triangulate qualitative findings with quantitative data. FARM PRODUCTIVITY Most farmer groups (12 of 18) interviewed for the evaluation reported positive impacts of Avansa Agrikultura interventions on farm productivity. The groups often cited increased frequency of planting and harvesting—in most cases a switch from seasonal to year-round harvests—enabled by new crop management techniques (e.g. crop rotation and distance farming) and inputs (e.g. plastic tunnels and mulch) as well as the generally short planting cycle of vegetables relative to the more commonly planted rice, maize, and coffee. The groups also mentioned faster and more efficient farming due to modern, mechanized inputs such as the mini tiller and hand tractor, which reduced time and burden in managing and harvesting crops. Finally, groups mentioned increased output of produce based on shorter planting cycles of vegetables, better and expanded use of farmland, and improved seeds, fertilizers, and irrigation practices (e.g. drip irrigation). “A lot of farmer groups received partial support to purchase agriculture technology such as hand-tractor[s] through the project. This seems to be really successful in helping the farmers to increase their productivity.” -Input Supplier Non-farmer group stakeholders interviewed for the evaluation confirmed many of the findings on improved farm productivity. About two-thirds (14 of 22) of Avansa Agrikultura, IP, and USAID staff noted Avansa Agrikultura-related impacts on farm productivity, with 11 of the 14 mentioning either farm expansion, increased production, or increased productivity. Similarly, 20 of 29 government and private sector stakeholders interviewed mentioned positive farm-related outcomes of Avansa Agrikultura programming, with 13 of the 20 specifically stating an increase in farm productivity, notably more frequent harvests, and year-round planting. For example, when asked about the largest household-level impact of Avansa Agrikultura, a collector in Bobonaro commented on the change from seasonal to year￾round continuous planting that had helped to increase demand for their services, while an input supplier in Dili clearly linked the provision of new inputs after Avansa Agrikultura support and higher yields on farms. A collector in Bobonaro, who was also a farmer, stated that the most beneficial impact of Avansa 13 | AVANSA ENDLINE REPORT USAID.GOV Agrikultura was “planting vegetables continuously and grow vegetables in different time to secure continuous production.”17 Not all references to impact on farm-level outcomes were wholly positive. Five of 18 farmer groups noted positive impacts of Avansa Agrikultura but caveated these impacts with limitations. They cited a lack of existing demand for the increased volume of produce, a heavy dependence on the weather (also mentioned by seven additional non-farmer stakeholders as a major impediment to year-round production), and the benefits of increased volume being canceled out by lower prices. Four stakeholders in Bobonaro mentioned an inability to grow crops during the rainy season, low soil quality, or Avansa Agrikultura inputs such as drip irrigation and tunnels as contextually inappropriate given geographic considerations. Limitations to productivity were supported by findings from the comparison of household data over time, which showed that 88 percent of households experienced a shock in the previous 12 months, with the most prevalent shocks to farmers being too little rain or too much rain, followed by crop pests and crop disease. Finally, the four farmer groups interviewed for the evaluation that had been supported by the Developing Agricultural Communities (DAC) project, the predecessor to Avansa Agrikultura, reported no new changes in farm-level outcomes around productivity.18 Farmers in Ermera PHOTO CREDIT: GABRIELA LEITE-SOARES, SI TIMOR-LESTE TEAM INCOME, EXPENDITURES, AND SAVINGS Across the board, farmer groups mentioned substantial increases in income when asked about household-level financial impacts of Avansa Agrikultura. Sixteen of 18 farmer groups indicated an increase in income due directly to new vegetables and related technical assistance from Avansa Agrikultura, while one group reported a decrease, and one reported no change. During FGDs, farmer groups often enumerated the changes in income garnered from sales of vegetables at typically two to three times above vegetable incomes pre-Avansa Agrikultura. Farmer groups noted that increased income came from more frequent sales—as often as two to three times per week compared to seasonally before—given larger volume of vegetables available to sell and more buyers (e.g. collectors 17 In this report, quotes from interviewed stakeholders are presented as they appear in data collection notes and transcripts after translation from Tetum to English. Translations reflect interpretation in the moment by evaluation team members. 18 While the four DAC-supported groups reported no change in farm-level productivity, one group noted new access to inputs via Avansa’s cost-share model and another group mentioned a change from chemical to organic pesticides. USAID.GOV AVANSA ENDLINE REPORT | 14 and supermarkets) demanding produce at more regular intervals. Six stakeholders, including two farmer groups, mentioned the introduction by Avansa Agrikultura of pricing based on weight and the changing of farmer mindset from subsistence to commercial farming as pivotal for increasing income. Despite generally positive findings around income, six of 18 farmer groups mentioned recent substantial decreases in income after reduced demand in the wake of COVID-19, while three groups noted muted or no changes in income due to large loan payments, low demand for products, and loss of business from a bankrupt supermarket (Kmanek). Avansa Agrikultura, IP, and USAID stakeholders confirmed most of the positive findings on income, with 16 of 22 interviewees mentioning increases in income for farmer groups as a result of Avansa Agrikultura programming. Likewise, 21 of 29 governmental and private sector stakeholders said Avansa Agrikultura-related assistance led to increased incomes of farmers, while three of 29 indicated a decrease in income among farmer groups due to COVID-19, the bankruptcy of Kmanek supermarket, or overproduction. Farmer groups and other stakeholders reported increased income as a result of Avansa Agrikultura activities through means other than increased vegetable production and sales. Four stakeholders, including a farmer group head and two collectors, noted that some farmers had become collectors due to observable substantial increases in income for collectors following new Avansa Agrikultura-facilitated market connections. Four of five collectors interviewed mentioned increases in income after Avansa Agrikultura support, with one noting that the difference between selling at a local market and a newly accessible Dili market was two-fold. According to five farmer groups, two collectors, two supermarkets, and a processor, these market connections also helped to reduce overall transport costs—and thereby increase income—since markets were willing to go directly to farmers, collectors eased the transport burden on markets, or Avansa Agrikultura co-financed transport for some agribusinesses. This also meant that farmer groups closer to Dili had reduced transport costs which contributed to increased income. “Our income definitely increase because before when we sell in the local market in Dili we only earn $150-$200 USD. However, now we can earn $250-$400 for every drop in Dili.” -Bobonaro Collector The ET examined the indirect impacts of Avansa Agrikultura by asking farmers how they spent the increased income from Avansa Agrikultura-supported activities. The number one use of new income among the 16 farmer groups that indicated increased income from Avansa Agrikultura was school fees for children (11 of 16 groups), followed by improvements to a dwelling (8 of 16), and vehicle purchases (5 of 16). Groups purchasing vehicles were mixed gender or all-female, suggesting that women may have invested in transport more frequently than others. Four farmer groups stated they used their increased “The household income for the group member indeed increased. The evidence can be seen that some of our group member able to take loan from Kaebauk to build their house. They pay back Kaebauk with the money that they get from selling the horticulture products.” -Ermera Farmer group 15 | AVANSA ENDLINE REPORT USAID.GOV income on agricultural inputs and two mentioned savings. Those groups furthest from Dili were much less likely to have spent increased income on agricultural inputs (2 of 13 groups) compared to those near Dili (2 of 5), and of these four groups purchasing inputs with increased income, all included women or were made up entirely of women. Three farmer groups commented that increased income had enabled them to start other income-generating activities (IGA) such as a fishpond, chicken farm, and purchase of livestock. An FGD with collectors in Ainaro confirmed that farmer groups in the municipality had expanded into floriculture, kiosk, and agritourism related IGAs with the additional income they received after Avansa Agrikultura-related horticulture interventions. Figure 1: Percent of respondents by stakeholder group indicating a specific usage of income Note: Respondents could list multiple responses or no responses; therefore, the total percent of responses for each stakeholder group may not add to 100%. Avansa Agrikultura, IP, and USAID stakeholder responses concurred in some areas and diverged in others with respect to farmer use of income. Sixteen of 22 respondents mentioned beneficiaries had used new income from Avansa Agrikultura-related activities, with nine indicating that it was used to improve a dwelling or purchase agricultural inputs, eight spent on children’s education, seven to purchase a vehicle, and three on starting a new IGA. Fifteen government and private sector stakeholders commented on income usage, with most (eight) mentioning children’s education, six mentioning a vehicle, and three each mentioning improving a dwelling or input purchase. To triangulate results related to income, the ET examined changes in household income among Avansa Agrikultura-supported farmer households through a comparison of household-level data collected in 2015 and in 2020. Using expenditures as a proxy for income,19 data showed a slight decrease in per 19 Deaton, A. (2008). The analysis of household surveys: A microeconomic approach to development policy. Baltimore, MD: The Johns Hopkins University Press. Household consumption and expenditures are often preferred to income when measuring poverty due to the difficulty in accurately measuring income, as expenditure data are less prone to error, easier to recall, and more stable over time than income data. For the Avansa households surveys, a per capita daily consumption aggregate was constructed by summing daily values of a household’s food expenditures over 7 days, non-food expenditures over the past month, and non-food USAID.GOV AVANSA ENDLINE REPORT | 16 capita daily expenditures over time from $1.63 in 2015 to $1.54 in 2020, as displayed in Table 4.20 These results do not necessarily support the notion of increased income among Avansa Agrikultura-supported farmer households but may be complicated by several factors. First, the result follows closely the overall trend in Timor-Leste in agricultural GDP over time, which saw a 7.7 percent decrease between 2013 and 2018. Second, changes in per capita daily expenditure differed greatly across expenditure types: average food expenditure decreased by $0.32, while non-food items purchased over the past month decreased by $0.07, and non-food items purchased over the past year increased by $0.32. This relatively large decrease in food expenditures may be due in part to the fact that Avansa Agrikultura heavily encouraged farmers to set aside portions of food for consumption instead of sale (see the next section on Food Security for more information). With increased production at farms and more consumption of their own food, it is likely that farmers spent less on food purchases. Indeed, the largest decreases in per capita daily expenditure among food items were vegetables and cereals, decreasing by 6 cents each. Third, data from the household survey showed an average increase in per capita expenditures over the past year (+$0.32) but a decrease in the past week (-$0.32) or month (-$0.07), a finding that may further substantiate the effects of COVID-19 on expenditure calculations. Finally, the average household size grew from 6.71 members at baseline to 7.47 members at endline, which would lower expenditures per capita if household expenditures remained constant. Table 4: Daily per capita expenditure – 2015 prices Disaggregate Baseline (2015) Endline (2020) Difference Estimate n Standard Error Estimate n Standard Error USD All Households $1.63 1,200 $0.07 $1.54 306 $0.10 -$0.09 Gendered Household Type MF $1.63 1,133 $0.08 $1.48 292 $0.09 -$0.15 FNM $1.32 40 $0.30 $1.76 10 $0.27 +$0.44 MNF $2.63 27 $0.48 $5.84 4 $1.95 +$3.21 Household Size 1-5 Members $2.16 434 $0.15 $2.32 84 $0.28 +$0.16 6-10 Members $1.41 666 $0.08 $1.35 175 $0.05 -$0.06 11+ Members $1.00 100 $0.09 $0.88 47 $0.06 -$0.12 Municipality Aileu $1.44 244 $0.06 $1.63 82 $0.28 +$0.19 Ainaro $1.53 332 $0.10 $1.34 63 $0.11 -$0.19 Bobonaro $1.29 211 $0.09 $1.84 79 $0.05 +$0.55 Dili $1.68 111 $0.22 $1.34 33 $0.12 -$0.34 Ermera $2.11 302 $0.19 $1.34 49 $0.11 -$0.77 Education of Primary Respondent No Education $1.57 592 $0.10 $1.66 130 $0.21 +$0.09 Primary School $1.57 341 $0.10 $1.35 92 $0.09 -$0.22 Secondary and higher $1.86 267 $0.16 $1.57 84 $0.10 -$0.29 Any Shock expenditures over the past year, then dividing by the number of household members. Note that in this approach, every household member is assumed to have an equal share of the total consumption, regardless of age and other household member characteristics. 20 Social Impact (2020). Endline Survey Report for the Avansa Agrikultura Project. 17 | AVANSA ENDLINE REPORT USAID.GOV Disaggregate Baseline (2015) Endline (2020) Difference Estimate n Standard Error Estimate n Standard Error USD No shocks - - - $1.67 37 $0.15 - At least 1 shock - - - $1.52 269 $0.10 - Note: Baseline value was previously reported as $1.76. This value included monthly loan interest payments. The loan interest section of the endline survey was revised to better fit the local context, and does not ask for loan interest expenditures on a monthly basis; thus the figures presented in this table for both baseline and endline do not include loan interest expenditures. Another potential explanation for the contradicting qualitative and quantitative results on income relates to savings. According to the quantitative data displayed in Table 5, between 2015 and 2020 the average household savings increased from $133.62 to $700.68, indicating that beneficiary households may be spending less and saving more than the general population at baseline. This is further supported by evidence from the household data comparison which showed an increase in savings as a proportion of household assets over time from about 7 percent in 2015 to 34 percent in 2020. On the other hand, non-savings productive goods (e.g. livestock, vehicles, other goods) as a share of household assets decreased over time from 93 percent to 66 percent, with the largest relative decrease in livestock (63 percent decrease over time). This could indicate an increased focus on horticulture and a move away from livestock as a principal source of agriculture-related income among Avansa Agrikultura beneficiaries. Finally, quantitative data supports increases in vehicle-related asset ownership, which more than doubled in both absolute and relative value over time among households. Table 5: Value of savings and productive goods by asset type Disaggregate Baseline (2015) Endline (2020) Estimate Percent of Assets Estimate Percent of Assets Savings $133.62 6.93% $700.68 33.83% Cash $50.16 3.60% $301.22 21.11% BNCTL Government Bank $26.59 0.31% $64.65 1.81% UBSP (Savings and loan group) $1.27 0.11% $96.47 7.03% Moris Rasik $8.78 0.28% $14.64 0.81% Other bank $4.67 0.30% $20.10 0.50% Gold, silver, or other precious metals $4.39 0.26% $140.03 6.17% Jewelry $8.73 0.36% $6.09 0.84% Other $1.46 0.09% $0.00 0.00% Loans $27.56 2.34% $57.48 3.05% Productive Goods $1,793.39 93.07% $1,370.64 66.17% Fishing $64.04 3.47% $4.55 0.36% Livestock $1,152.14 62.90% $513.02 25.86% Household durables $109.72 12.94% $138.22 12.23% Transport $305.05 7.55% $620.83 16.68% Farm Equipment $162.42 5.49% $94.02 3.54% Transfers $84.69 - $39.52 - Cash $58.38 - $18.74 - In-kind $26.31 - $20.78 - USAID.GOV AVANSA ENDLINE REPORT | 18 Qualitative data supported an increase in savings among Avansa Agrikultura-supported households with four farmer groups of 18 noting the critical role that contributing to and participating in savings and loans (S&L) groups played in enabling access to funds to purchase agricultural inputs. Four non-farmer stakeholders cited S&L groups, particularly those run by women, as a highly successful and sustainable Avansa Agrikultura intervention. Two Avansa Agrikultura senior staff members with intimate knowledge of S&L groups noted that while some S&L groups had dissolved in the recent past, at least ten currently active women-run S&L groups had accumulated over $50,000 in savings—much of which has been used as seed money to start several agribusinesses. “Saving and loans has helped some farmers to purchase motorbikes and hand tractors, without the saving and loans they may not be able to purchase them.” -Bobonaro Farmer group FOOD SECURITY According to Avansa Agrikultura’s Year 5 Annual Report, the project conducted a range of nutrition￾related activities, including encouraging nutrition gardens and retention of food for consumption, delivering an eight-week nutrition training course to 391 beneficiaries, training trainers of three Timorese non-governmental organizations (NGOs) to continue nutrition-specific activities, and reaching over 500 households with nutrition posters. To evaluate these activities, the ET looked at changes in consumption and nutrition practices for household-level outcomes related to food security. Fifteen of 18 farmer groups interviewed noted positive changes in food security as a result of Avansa Agrikultura￾related interventions, with seven reporting increased nutrition and seven reporting positive changes in consumption habits. Four groups noted increases in the use of their own crops for household consumption, two groups of which had done so for the first time, while four other groups noted increases in diversity of consumption following Avansa Agrikultura-related support. Specifically, farmers discussed eating more and different vegetables because of the presence of new vegetables within their own plots and due to nutrition education interventions of Avansa Agrikultura. As a farmer in Ermera stated, “the nutritional program is great because Avansa taught us on how to consume better diets,” and, from a collector in Ainaro, “I used to buy only rice [for consumption], but now I buy veggies.” In an FGD with participants of a vocational training school in Aileu, students indicated taking home important lessons learned at school on nutrition to their families, encouraging them to consume more vegetables. “Before we didn’t consume lots of vegetable since we have to buy from the local market but now since we grow [vegetables] ourselves we can consume and don’t need to buy from the local market anymore.” -Ainaro Farmer group Avansa Agrikultura, IP, and USAID staff in large part mentioned substantial impacts of Avansa Agrikultura programming on food security. Specifically, seven of 19 respondents in this category indicated that farmers were now setting aside some of their own food for consumption when they typically did not before. Ten of 19 respondents mentioned that farmer consumption increased over 19 | AVANSA ENDLINE REPORT USAID.GOV time, with eight of 19 stating that diversity of consumption among farmers also increased. According to one collector in Bobonaro, “prior to Avansa [farmers] only grow few vegetables and fruits just to sell in the local market and [did] not think about their consumption.” With respect to the idea of nutrition, six in 19 respondents in this category mentioned that farmers experienced an increase in nutrition. Only 11 of 29 government and private sector stakeholders mentioned food security as an impact of Avansa Agrikultura programming at the household-level, five of which noted increased consumption, three mentioned increased use of farmers’ own products in consumption, and three indicated an increase in nutrition among farmers. When looking a bit closer at nutrition-related outcomes, the majority of farmers who mentioned nutrition were limited in their ability to specify exactly what improved nutrition meant, rarely going beyond statements that said, “we get improved nutrition.” Of the seven farmer groups that indicated improved nutrition, four mentioned the importance of a varied diet for nutrition. One particular farmer in an FGD in Ainaro said “we began to understand the importance of nutritional values and how it can support our daily activities,” though this group was singular in its understanding of the link between eating nutritious vegetables, and reduction of malnourishment in the community. Five stakeholders, including four Avansa Agrikultura senior staff and a municipal-level government official, concurred that as of yet there was generally “no recognition of nutrition fundamentals” among farmers and that farmers prioritize selling their product before consuming it. Referring to consumption of the newer, more nutritious vegetables, according to one Avansa Agrikultura municipal coordinator, farmers consider “money [as] more important than to consume,” and, per a senior Avansa Agrikultura staff member, “[farmers] are getting more food, but it doesn’t mean they’re getting more good food.” Observations from a member of the ET added that as farmers gain more income and purchase more food, they are oftentimes likely to purchase less nutritious food such as packaged noodles or fast food. A comparison of household-level data among Avansa Agrikultura-supported households from 2015 to 2020 and Avansa Agrikultura’s own Year 5 annual report support findings on changes in consumption patterns. According to both sources, household moderate and severe hunger decreased from 15 percent to less than one percent over the time period.21 With respect to nutrition, over the same time 21 Seasonal variation could have played a role in influencing these results. The baseline in 2015 was carried out in November during the hungry season compared to endline data collection in July 2020, which typically coincides with the harvest season. USAID.GOV AVANSA ENDLINE REPORT | 20 period household survey results showed an increase in minimum acceptable diet (MAD)22 for children aged 6–23 months from 41 to 46 percent. While encouraging and supportive of increased nutrition, the results were highly imbalanced across municipality and gender. Dili experienced the largest relative change over time (a 41 percent increase) with smaller and smaller changes as distance from Dili increased: Aileu saw a 25 percent increase in MAD of children aged 6–23 months over time, Ainaro seven percent, with decreases of 12 and 30 percent in Bobonaro and Ermera, respectively. Though the sample size is small, qualitative data reflects a similar geographic inequity, with 60 percent (3 of 5) of farmer groups in or near Dili mentioning positive impacts of Avansa Agrikultura on nutrition compared to 31 percent (4 of 13 groups) in farther municipalities. With respect to gender imbalance, boys increased MAD by four percentage points over time from 39 to 43 percent while girls increased nine points from 42 percent at baseline to 51 percent at endline.23 Additional evidence supporting increased nutrition came from household-level data which showed that women of reproductive age experienced an increase in the mean number of food groups consumed from 6.79 at baseline to 7.51 food groups at endline. Unlike the findings on MAD of children aged 6–23 months, the increase in mean food groups for women of reproductive age was consistent across municipalities. Multi-crop field in Ermera PHOTO CREDIT: ZENNY CORREIA, SI TIMOR-LESTE TEAM HIAM Health, a local Timorese organization, was contracted by Avansa Agrikultura to provide training on nutrition and sanitation-related topics to farmers. Given Timor-Leste’s high stunting rates,24 coupling sanitation with nutrition interventions is critical for future health outcomes, particularly as research shows that reducing incidence of diarrhea through hygiene and sanitation infrastructure-related 22 Minimum dietary diversity for breastfed children 6–23 months is defined as four or more food groups out of the following seven food groups: Grains, roots and tubers, Legumes and nuts, Dairy products (milk, yogurt, cheese), Flesh foods (meat, fish, poultry and liver/organ meats), Eggs, Vitamin-A rich fruits and vegetables, Other fruits and vegetables. World Health Organization. (2010). Indicators for assessing infant and young child feeding practices: Part II Measurement. http://apps.who.int/iris/bitstream/10665/44306/1/9789241599290_eng.pdf. 23 According to a 2018 report commissioned by University Research Co. (URC), stunting may be higher in boys than girls since boys are likelier to “have slightly higher nutritional requirements than girls,” are generally more mobile than girls, and, in Timor-Leste, were breastfed for one month less than girls on average. The URC report also found similar disparities in nutrition indicators across municipalities. URC report accessed at: https://www.urc￾chs.com/sites/default/files/urc-Addressing-Stunting-Timor-Leste-Assessment-Report-1809.pdf. 24 The 2016 Demographic and Health Survey measured that the prevalence of stunting was 45.6 percent of children under five. Prevalence of wasting of children under five was 45.6 percent. General Directorate of Statistics - GDS/Timor-Leste. Ministry of Finance/Timor-Leste and ICF. 2018. Timor-Leste Demographic and Health Survey 2016. Dili, Timor-Leste and Rockville, Maryland, USA: GDS and ICF. https://dhsprogram.com/pubs/pdf/FR329/FR329.pdf. 21 | AVANSA ENDLINE REPORT USAID.GOV interventions are integral to reduce stunting.25 Findings from this evaluation on sanitation were generally limited. One farmer group in Ainaro mentioned that Avansa Agrikultura had trained them on the importance of handwashing; this was also repeated by a municipal-level government stakeholder in Bobonaro. Four of 22 Avansa Agrikultura, IP, and USAID stakeholders indicated improved sanitation of farmer groups. Handwashing, handwashing with soap, and a move away from open defecation are important sanitation-related indicators that require strong behavior change efforts to ensure success. The ET was unable to assess the extent to which sanitation was strengthened, though it will remain a critical, complementary component to nutrition interventions moving forward. CONCLUSIONS FOR EQ1 Avansa Agrikultura interventions likely contributed to improved farm productivity and increased and diversified income. The introduction of new and diverse vegetables with short planting cycles coupled with modern inputs and technical assistance on horticulture farming contributed to higher incomes, as farmers are able to sell crops more frequently than prior to Avansa Agrikultura’s support. A majority of farmers are spending most of their extra income on schooling for their children, shelter upgrades, and savings. Avansa Agrikultura interventions with farmer groups likely contributed substantially to improved food security and nutritional indicators of participating farmers and their households. Rather than selling all their product, more farmers are setting aside portions of food for consumption. Levels of hunger dropped dramatically between baseline and endline; at the same time, prevalence of children receiving a MAD among children aged 6–23 months and mean number of food groups among women of reproductive age increased between 2015 and 2020. The impact of Avansa Agrikultura interventions on nutritional knowledge and attitudes is unclear. The opportunity for income trumps nutrition priorities as farmers are still likely to sell off new, high-earning nutritious food before consuming it. Farmers understand that they have improved nutrition, but struggle to specify what that means. There is no indication that improved sanitation, an important complementary outcome for nutrition, has changed as a result of Avansa Agrikultura. 25 University Research Co. LLC, 2018. Addressing Stunting in Timor-Leste: An Assessment Report. Accessed at: https://www.urc-chs.com/sites/default/files/urc-Addressing-Stunting-Timor-Leste-Assessment-Report-1809.pdf. USAID.GOV AVANSA ENDLINE REPORT | 22 EQ2: MARKET-LEVEL IMPACTS To what extent has the project strengthened private sector market linkages across the value chain for target crops? Which interventions were the most effective in doing so and why? For EQ2, findings are organized by value chain actor or service. While the value chain for horticulture in Timor-Leste is relatively horizontal, the section begins with those actors traditionally at the bottom (producers, or farmer groups) and related services (input supply, access to finance), followed by middle￾chain services (collectors and processors) and finally end-market consumers (local and supermarkets). Technical assistance and its related actors (vocational schools, associations, and extension services) are included in EQ3. Figure 2 depicts the current value chain for horticulture in Timor-Leste. Figure 2: The Horticulture Value Chain in Timor-Leste PRODUCERS AND PRODUCER GROUPS Beginning with its predecessor project, DAC, Avansa Agrikultura organized farmers into farmer groups in the five targeted municipalities. Groups were initially siloed in nature receiving one-off interventions, for example S&L or nutrition, however a shift after the midline evaluation in 2017 sought to reframe farmer groups as entities that would receive integrated packages of interventions. The move ensured a holistic approach to improving outcomes. Five of 18 farmer groups interviewed remarked on the importance of being in a farmer group. According to one farmer in Ermera, “Yes, each farmer has their own plot to grow their crops [but Avansa-supported groups] inspire other groups to grow their own product and form their own group,” and from a different Ermera farmer group FGD, “when other farmer[s] see our group, it inspires them to establish their own group and engage in farming as well.” As one national-level government stakeholder simply put: “we would not have seen farmer groups [if it weren’t for Avansa].” 23 | AVANSA ENDLINE REPORT USAID.GOV An Avansa-supported farm in Ainaro PHOTO CREDIT: ACACIO AMARAL, SI TIMOR-LESTE TEAM Avansa Agrikultura staff indicated that farmer groups were primarily a vehicle for mass delivery of training and a means to attract supermarkets to guaranteed volume of sales. According to one municipal coordinator and one senior Avansa Agrikultura staff member, the groups also allowed farmers to come together during times of need and to demonstrate different technology and knowhow among farmers. Despite these advantages, farmers in farmer groups overall did not farm collective plots or share profits. In qualitative probing during interviews, three municipal coordinators admitted that some farmer groups had disbanded as of late and cast doubt on the overall sustainability of farmer groups after the Avansa Agrikultura project concludes. With respect to Avansa Agrikultura-related impact on strengthened market linkages, 17 of 18 farmer groups interviewed mentioned an Avansa Agrikultura-facilitated link between the farmer group and a supermarket, 14 groups linked to a local market, 11 groups linked to a collector, 10 groups linked to an input supplier, and 8 groups each mentioned a link to finance or a link to transportation. Among interviews with Avansa Agrikultura, IP, and USAID stakeholders, 9 of 22 mentioned linking farmer groups with other private sector value chain actors as a major impact of Avansa Agrikultura with 5 of 22 mentioning creation of farmer groups. For government and private sector interviewees, 16 of 17 collectors, supermarkets, and input suppliers noted strengthened linkages to farmer groups as an impact of Avansa Agrikultura. USAID.GOV AVANSA ENDLINE REPORT | 24 Figure 3: Percent of farmers or farmer groups indicating a market link to another value chain actor Note: Respondents could list multiple responses or no responses. ACCESS TO FINANCE Avansa Agrikultura facilitated access to credit for farmers through various interventions including nurturing the creation of savings and loans groups, linking farmers and farmer groups to a microfinance institution (MFI) called Kaebauk, training farmers on good financial management (see EQ3), training Kaebauk loan officers in catering loans to farmers, and liaising with input suppliers to participate in this model. Avansa Agrikultura helped to create a “triangle model,” connecting farmers, input suppliers, and microfinance with the goal of enabling farmers to obtain high quality, modern agricultural inputs for their farms. Seventeen of 18 farmer groups mentioned access to finance for farmers as a major impact of Avansa Agrikultura in interviews, as did 16 of 22 of Avansa Agrikultura, IP, and USAID stakeholders, and 18 of 29 of government and private sector interviewees. One IP stakeholder noted that simply giving Kaebauk a list of farmers was transformational in catalyzing a connection between the two entities: “that list, that simple act, made a huge difference.” The work of Avansa Agrikultura was equally impactful on the MFI as well. While two other large banks were too risk averse to enter the market, Kaebauk’s willingness and eagerness facilitated work between the MFI, Avansa Agrikultura, and farmers. Kaebauk expanded the number of loan officer positions, and loan officers benefitted from Avansa Agrikultura training in structuring loans for farmers. Beyond linking farmers and Kaebauk, Avansa Agrikultura went a step further by initially offering grants to farmers and then switching to a cost-share model. As part of this model, Avansa Agrikultura paid 50 percent (and later 40 percent) of the cost to purchase inputs, with the farmer responsible for the other half (eventually 60 percent). Farmers typically financed their portion through Kaebauk, a S&L group loan, or paid cash. The cost-share was not only limited to farmers; three of five collectors interviewed for the evaluation noted using it to purchase scales and vehicles, including a cold-chain vehicle, and one fertilizer agribusiness purchased processing machinery. The cost-share model was cited as the most effective Avansa intervention by the largest array of stakeholders. It was particularly popular among farmers and collectors, with 13 of 18 farmer groups and all five collectors mentioning it in the context of a strengthened market linkage, as well as 14 of 22 Avansa Agrikultura, IP, and USAID stakeholders, five of eight input suppliers, and three of eight government stakeholders. 25 | AVANSA ENDLINE REPORT USAID.GOV “The Avansa shared cost program is successful because it help[s] our group to take credit from Kaebauk to purchase a hand tractor in November 2018 (60-40), and we were able to pay it for 18 months.” -Ermera Farmer group For farmers that were unable to access Kaebauk loans, such as two farmer groups in Ermera and Bobonaro, another method for accessing credit was through S&L groups set up by Avansa Agrikultura. As one farmer in an S&L group in Bobonaro noted, “Saving and loans [groups] has helped some farmers to purchase motorbikes and hand tractors, without the saving and loans they may not be able to purchase them.” Seven of 18 farmer groups noted the ability to access loans through S&L groups— typically as high as $500—to help purchase inputs that were otherwise out of reach financially. According to the Avansa Agrikultura Year 5 Annual Report, of all loans disbursed by Kaebauk in 2019, 37 percent were done so through S&L groups. However, S&L groups depended heavily on repayment by members and, in that respect, were highly vulnerable. The ET noted several instances of S&L groups disbanding due to non-repayment. An Avansa Agrikultura staff member indicated that many S&L groups had dissolved—including seven in Ermera alone—due to a majority of the female members “being married off and not repaying.” According to a farmer in Bobonaro, “We used to have saving and loans group, but no more because some members borrow but did not return the money.” An Avansa Agrikultura municipal coordinator generalized that in the municipality they oversaw, women tended to repay at much higher rates than men and youth. Despite these findings, instances of non-repayment in S&L groups appear to have been few and far between, with many more positive anecdotes, including one group that tripled its savings over the course of the project. When it came to repayment of Kaebauk loans, several respondents familiar with those data, including Avansa Agrikultura’s Year 5 reporting, noted that repayment rates were well over 95 percent. INPUT SUPPLY Avansa Agrikultura worked with several major Dili-based input supply companies to link them to farmer groups, expand input supply to municipalities through small input shops, and strengthen input supply models and networks through study tours. Qualitative findings suggested that more inputs, in particular seeds, fertilizer, and equipment such as plastic tunnels, mulch, mini tillers, hand tractors, and drip irrigation supplies were reaching more farmers as a result of Avansa Agrikultura interventions. According to two large Dili-based input suppliers, this was due, in part, to input suppliers being more proactive in asking what farmers needed after being introduced through Avansa Agrikultura, and the use of collector networks to disperse inputs as well. As one input supplier said: “the increase of technology… has increased the production of the farmer, in result it increases the demand for inputs as well.” Eighteen of 22 Avansa Agrikultura, IP, and USAID stakeholders mentioned connecting farmer groups with input suppliers as an example of a strengthened market linkage. The ET spoke with seven input suppliers and the agro-input business association for the evaluation. Of these eight stakeholders, seven mentioned that Avansa Agrikultura had strengthened a linkage between the supplier and a farmer group and seven also stated that they had seen an increase in sales. Five of eight also noted the cost-share model as being instrumental for allowing farmers to purchase inputs from the supplier on credit. Two input suppliers noted that they had been introduced to farmers and made meaningful connections at Farmers Field Day expos organized by Avansa Agrikultura. A USAID.GOV AVANSA ENDLINE REPORT | 26 particularly successful anecdote came from the input supplier Vinod Patel. Primarily a home goods store prior to Avansa Agrikultura, the company expanded considerably in the area of agricultural inputs during the lifetime of Avansa Agrikultura and now, in addition to input supply, provides technical assistance to farmers on how to use inputs. Vinod Patel noted an increase in income as a result of Avansa Agrikultura support as well as expansion to new shops. “We only sell around 20 types of vegetables seeds and fertilizers. Our size and the input supplies we sell is less compared to bigger shops. We have to purchase our supplies from Dili and Atambua (West Timor, Indonesia). It’s cheaper to purchase from Atambua than from Dili. Due to the close distance to Atambua, it is just cheaper to purchase our supplies from Atambua.” -Input supplier in Bobonaro Not all stakeholders commented positively about input supply: an Avansa Agrikultura staff member and two farmer groups in Ainaro, Aileu, and Bobonaro commented that input suppliers were too far away to help farmer groups or that seeds in particular were oftentimes hard to come by. Groups in Bobonaro spoke about purchasing seeds illegally from Indonesia to address the dearth of seed inputs in the municipality. Two input suppliers also noted that their businesses were only viable due to the importation of cheap inputs such as seeds and fertilizer from Indonesia—local inputs, they said, were too expensive. The theme of inadequate seed supply was repeated throughout data collection by ten different stakeholders with the problem highly exacerbated during the COVID-19 pandemic. Avansa Agrikultura sought to address issues of the input supply chain through a consignment model that would connect the larger Dili-based supply companies with smaller municipal stores. The ET spoke with three municipal stores, all of which noted the importance of their presence in supplying inputs to distant farmer groups. Effectiveness of the linkages in the consignment model was mixed: one shop indicated the impact of Avansa Agrikultura in helping them to choose high quality and relevant seeds and fertilizer for farmers; another said that while Avansa Agrikultura initially connected them with Vinod Patel for seed supply, the two companies interacted once and then never again. Senior Avansa Agrikultura staff members admitted that the consignment model “never really took off, probably because it was begun too late.” A member of the ET noted that smaller input suppliers did not appear to benefit from Avansa Agrikultura support: “Those who are already big, they are doing well. Those who are small, this has little effect.” A final Avansa Agrikultura intervention on strengthening linkages with respect to input supply funded international study tours for several input suppliers. The ET spoke with three companies that participated in study tours to Indonesia and Singapore. All three were grateful for the opportunity and indicated that they had established useful networks and, in one case, negotiated better prices with an Indonesian supplier. One company noted that the price of local Timorese inputs remained too high for a reciprocal relationship with another country, such as Indonesia; further, two input suppliers noted major challenges to benefitting from the new relationships made with suppliers through study tours since the Timorese import quarantine law acted as a bottleneck to importing seeds from abroad. COLLECTION AND TRANSPORT As the middle-market connectors between the farm gate and markets, collectors, whether an individual entrepreneur or in groups, play a pivotal role in the horticulture value chain in Timor-Leste, especially in 27 | AVANSA ENDLINE REPORT USAID.GOV municipalities furthest from Dili. Avansa Agrikultura supported collectors by connecting them with farmers, and with markets, by offering them assistance in financing business-related purchases through the 50 percent cost-share model, and with business and financial management training. According to multiple stakeholders, Avansa Agrikultura helped to create collector networks in some municipalities where none had existed prior. One senior Avansa Agrikultura staff remarked that collectors “help as intermediary [and] trusted representative and they bridged the divide” between farmers and markets, oftentimes as informal arbiters of disputes between the two entities. “Previously we just sold to local market but after got support from Avansa we could access to supermarkets as mentioned. Moreover, some collector also came here to buy our products…” -Dili Farmer group New linkages with collectors and the corresponding impact was noted often by stakeholders in qualitative interviews. Of the 18 farmer groups interviewed, 11 discussed the impact of Avansa Agrikultura in connecting them with a collector or collector group, as did 3 of 4 supermarkets interviewed. Among Avansa Agrikultura, USAID, and IP staff, 20 of 22 stakeholders noted a strengthened market linkage with respect to collectors. Of the five collectors interviewed, all reported increased connections with supermarkets and farmer groups as well as an increase in sales as a result of Avansa Agrikultura support. All five also indicated that the cost-share model had been impactful for them, allowing three of the five to purchase a vehicle for their business. The specific impacts of expanded collector connections were myriad. Ten stakeholders including farmer groups and markets emphasized a reduction in transport costs with increased presence of collectors since producers and buyers no longer needed to pay for or arrange transport themselves. While supermarkets sometimes went directly to farms to pick up produce—especially to those closest to Dili—the option of a collector enabled access to distant farmers and provided for more efficient allocation of transport if consistent volume of a product couldn’t be guaranteed by farmers, a common finding across interviews. On the supply side, connecting farmers and collectors allowed farmers to get their product to market and enabled collectors to access more farmers thus increasing their transport volume and sales. Collector actions were not limited to simply moving produce; four stakeholders including two input suppliers stated that collectors helped deliver seeds and other inputs to farmers when the input suppliers could not reach farmer groups. Further, three collectors and a senior Avansa Agrikultura staff member noted that collectors, after having gone through Avansa Agrikultura financial literacy training, began disbursing microloans to farmers who could not easily access loans from Kaebauk or otherwise. Qualitative evidence suggested an innovative and flexible approach to collection: one collector noted that farmers will oftentimes call them directly to pick up produce and an Avansa Agrikultura staff indicated that collectors were recently using WhatsApp and Facebook to communicate and coordinate with farmers and amongst themselves to arrange for home delivery and pick up. According to farmer groups and supermarkets, collectors were sometimes the source of misunderstandings with respect to supply, demand, and price. One senior Avansa Agrikultura staff member noted that “[Avansa] provided training for collectors… [to] increase the knowledge of collector about the product and the price,” but later admitted that “communication between producer, collector and supermarkets is least successful which lead to some mistrust and misunderstanding.” A supermarket confirmed the above, blaming collectors for not accurately conveying supply and demand USAID.GOV AVANSA ENDLINE REPORT | 28 changes—and the resulting effects on price—to farmers. In general, communication and/or coordination among farmers and between farmers and collectors around planting choices and price appeared lacking. Five farmer groups mentioned the problem of overproduction since many farmers in groups were growing the same product, the most common cause of price drops. To address concerns around price, early on in the project collectors, supermarkets, and farmers, with the help of Avansa Agrikultura, worked together to standardize prices, however this did not take into account market dynamics associated with swings in consumer demand or overproduction. Findings suggest that collectors and farmers were unaware of the effects of market volatility on supply needs and price at supermarkets. Considering that collectors were often the only contact between markets and farmers, they received a disproportionate share of the blame from both parties when price or volume did not meet expectations. PROCESSING Small and medium enterprises (SMEs) acting mostly as processors and food preservers play an active role in the horticulture value chain in Timor-Leste. According to Avansa Agrikultura’s Year 5 Firm-Level survey, 96 SMEs participating in the horticulture value chain generated $2 million worth of sales in 2018. Over the lifetime of the project, Avansa Agrikultura supported 28 agribusinesses, six of which were owned by women, with access to finance, access to inputs, business training, and linkages to markets to sell their product. The ET interviewed three processors (two of which included women stakeholders) for the evaluation and triangulated their responses during interviews with Avansa Agrikultura staff and other stakeholders knowledgeable about agribusinesses in Timor-Leste. All three processors noted the positive impact of Avansa Agrikultura support, especially with respect to the project kickstarting the business via grants or other finance mechanisms (such as the cost-share model). Two of the three indicated increased sales and income as a result of Avansa Agrikultura support, but all three also indicated recent decreases in sales due to a combination of COVID-19 and challenges related to the viability of their products. Slightly more than half (12 of 22) of Avansa Agrikultura, IP, and USAID respondents felt that support to small agribusinesses such as processors was an example of effective market linkage strengthening. Strawberries from a farm in Ermera PHOTO CREDIT: ZENNY CORREIA, SI TIMOR-LESTE TEAM Stakeholders mentioned several positive anecdotes of agribusiness success after receiving Avansa Agrikultura support. Avansa Agrikultura senior staff and municipal coordinators pointed to processing 29 | AVANSA ENDLINE REPORT USAID.GOV or preserving businesses started with Avansa Agrikultura grants such as those related to pickling, strawberries, papaya, tomato sauce, mushrooms, tea bags, banana crisps, flowers, and fertilizer. A fertilizer processing business which received support from Avansa Agrikultura connecting them to local markets and input suppliers noted that “Our activity … does inspire other group[s] to [be] involved as well since they see the benefit of the activities” while another processor noted that their new business “encourages our neighbors to plant [the same product].” Senior Avansa Agrikultura staff noted that processing businesses are important as a means to address overproduction, particularly for a product such as strawberries which could easily benefit from jam or juice processing given its new popularity in Timor-Leste. The creation of women-run agribusinesses was highlighted as a sign of success by Avansa Agrikultura staff, as were two youth-run businesses (banana crisps, mushrooms) started after trainings delivered by Avansa Agrikultura in vocational schools. Women and youth were specifically targeted by Avansa Agrikultura, and the success of women-run agribusiness in particular fits somewhat with women’s relatively larger role in handling finances in Timor-Leste. Agribusinesses interviewed by the ET and findings related to agribusiness in general synthesized from interviews with other stakeholders indicated a challenging environment for success. According to two vocational schools, processors face high prices for inputs, especially jars and bottles, and transport costs for heavy inputs such as fertilizer. Meanwhile, one municipal coordinator indicated that processed or preserved goods were not necessarily big sellers in local or supermarkets either due to unfamiliarity with the product among the local populace or cheaper versions imported from abroad. An IP stakeholder agreed, commenting that processing will have a hard time succeeding “not because farmers don’t know how to do it but because demand is low.” Finally, a senior Avansa Agrikultura staff noted that simply receiving money and training will not suffice for processors: those “who have networks and connections succeed, those who don’t have them won’t.” END MARKETS Avansa Agrikultura connected supermarkets with farmer groups and collectors as explained above via lists and facilitated introductions. Though four of 18 farmer groups and three of five collectors interviewed had already-established links with markets as a result of DAC or their own initial business interactions, supermarkets indicated that new connections were impactful. Qualitative analysis showed that three of four markets interviewed mentioned new connections with collectors or farmer groups as an impactful component of Avansa Agrikultura, while all four mentioned transport-related benefits and improved quality of products. Connections with farmer groups reduced the burden on markets to look for buyers on their own, and, as mentioned above, linkages with collectors helped to reduce transport costs. Supermarkets played a proactive role with farmer groups they were linked to, for example selling farmers seeds for products that the markets wanted planted and, according to one supermarket, traveling to farms to give technical assistance on horticulture farming. In one case, a supermarket linked a collector to Avansa Agrikultura for financial support. Twenty-one of 22 Avansa Agrikultura, IP, and USAID stakeholders mentioned linking farmer groups to supermarkets as a way that Avansa Agrikultura had strengthened market linkages. USAID.GOV AVANSA ENDLINE REPORT | 30 Cabbage from a farm in Ermera PHOTO CREDIT: GABRIELA LEITE-SOARES, SI TIMOR-LESTE TEAM Throughout the life of the project, Avansa Agrikultura facilitated the signing of memorandums of understanding between supermarkets and farmer groups in order to formalize relationships (one supermarket chose to use verbal agreements only). MOUs typically guaranteed a set volume of produce at a set frequency and at a set price from farmers to supermarkets. As part of the deal, supermarkets often picked up produce directly from farms. While MOUs formalized a connection between the two groups, location of farmer groups heavily influenced actual regular contact with supermarkets; those closer to Dili benefitted from frequent, direct pick up from supermarkets while those further away in Ainaro, Emera, and Bobonaro had less frequent pick up or interacted with markets through collectors only. In practice, the terms of MOUs appeared to be more optimistic than what the reality suggested was possible. Two supermarkets interviewed mentioned that farmers grew what was determined vis a vis the MOU, but volume of produce was not consistent, and the markets stopped conducting regular pick up; this was confirmed by three farmer groups. According to another supermarket, farmers did not abide by the terms of the MOU and produced according to their own whims, leading to overproduction. One senior Avansa Agrikultura staff member and several farmer groups confirmed these findings, in some cases blaming it on collectors who provided imperfect information on price changes and quantity demands while others noted that farmers did not coordinate well amongst each other in terms of planting. According to one government stakeholder, “[farmers] need to grow different commodity to avoid over supply, which will affect the price,” and per a supermarket, “collector[s] did not control what type of produce need to be planted, hence a lot groups end up growing the same commodity resulting in over supply and the price decrease.” “This group is growing with the rotation system, so that those items that required from supermarkets will always be ready and never stop. Through this system also could reduce the over production.” -Ermera Farmer group Additional positive impacts of strengthened linkages between supermarkets and farmers included more targeted, relevant planting by farmers, and increased quality of produce. Prior to Avansa Agrikultura, farmers planted products for sale in the local market and/or for household consumption, however, six farmer groups noted that their planting decisions had changed after Avansa Agrikultura support to be 31 | AVANSA ENDLINE REPORT USAID.GOV more demand-driven according to what the supermarkets need. By catering to demands of the supermarkets, farmers were also forced to adapt and provide produce at the level of quality required by supermarkets. Ten different stakeholders discussed changes in quality due to new linkages between farmers and supermarkets. According to two supermarkets: “in the beginning farmers do not know, but after [Avansa] training, farmers understand the standard of quality” and “farmers now understand it is essential to meet the quality standards set by us and why this standard is there in the first place.” A third supermarket noted “the products we receive from farmers are 90% with good quality… [they] sold out quickly.” A few stakeholders did note low quality produce as a persistent issue though not representative of the majority of delivered products. One collector in Dili estimated that 30 of every 100 kilograms they picked up was low quality and returned to producers while another in Ermera recounted that one supermarket decided to stop buying from a set of farmer groups in that municipality due to persistent quality issues. Most stakeholders commenting on quality agreed that while quality had increased substantially, farmers still need to work hard to meet supermarket standards. CONCLUSIONS FOR EQ2 Avansa Agrikultura-facilitated introductions between new farmers, collectors, and supermarkets catalyzed strong linkages, improved quality, and led to positive market disruption. Many stakeholders spoke of the simple act of Avansa Agrikultura providing lists and raising awareness of different groups as transformational to establishing linkages. MOUs and verbal agreements legitimized initial linkages. Diversification of selling and buying choices is leading to more competitive behavior along the value chain. Quality of horticulture products in supermarkets has largely increased. Collector networks strengthened by Avansa Agrikultura can fill critical service gaps where markets, input suppliers, and MFI reach is limited. Collectors are playing diverse roles from traditional transport and input supplier to credit provider and mediator. Anecdotal evidence suggests some participants in Avansa Agrikultura programs are switching full time to collecting due to large observable income opportunities. The cost-share model and financial training has been especially impactful in growing collector businesses. Access to finance through Kaebauk loans and Avansa Agrikultura cost-share is slowly transforming farmer mentality and ability in agriculture. Beneficiary stakeholder groups mentioned the cost-share model most frequently as an impactful piece of Avansa Agrikultura programming. Combined with saving and loans and financial training, farmers are now able to access inputs they never could before and better manage their debt. Cost-sharing promotes independence and ownership over assets, contrasting strongly with historical handouts to farmers. Imperfect information and lack of coordination among and between farmers and collectors causes market inefficiencies and farmer-supermarket discord. Collectors sometimes do not adequately understand or are not accurately communicating to farmers market volatility conditions. Farmers tend to focus on planting the fastest growing vegetables, are limited to planting vegetables whose seeds are available, do not on the whole coordinate planting decisions among each other, and are oftentimes unaware of changes in supermarket demand, all of which lead to overproduction that drives down price and increases wastage. MOUs between farmers and supermarkets were initially inflexible to changes in supply, demand, and price in the market, though the project has recently taken steps to move away from direct single-buyer MOUs. Persistent issues with seed supply and high price of inputs for processing limit the full potential of horticulture interventions. A wide array of stakeholders noted problems in obtaining USAID.GOV AVANSA ENDLINE REPORT | 32 adequate volume and quality local seeds. Input suppliers and farmers in border regions rely heavily on Indonesian seed imports. Processors note high price of inputs and limited demand for their products in addition to strong competition from cheap imports. Benefits from strengthened market linkages diminish as distance from Dili increases. Farmers in Ermera, Ainaro, and Bobonaro report lower access to inputs and less frequent sales than farmers in Aileu and Dili. Road quality and water issues are persistent barriers to success for farmers farther from Dili. 33 | AVANSA ENDLINE REPORT USAID.GOV EQ3: INDIVIDUAL-LEVEL IMPACTS To what extent have project activities contributed to changes in reported self-efficacy among beneficiaries, particularly females and youth? Which interventions were the most effective in doing so and why? To answer EQ3, the ET sought to determine if farmers exposed to Avansa Agrikultura programming had increased self-efficacy, or the ability to exert control over one’s environment due to a belief in one’s ability to do so—to have the confidence to know that one has the capabilities to succeed. To measure changes in self-efficacy, the ET examined the extent to which Avansa Agrikultura interventions, especially training, resulted first in knowledge and skills gain, and secondly in attitudinal change toward agriculture-related abilities among farmers, with a particular eye to differential effects on women and youth. Effects of Avansa Agrikultura programming on vocational training schools is included in the section on youth impacts. SKILLS AND KNOWLEDGE CHANGE The ET categorized Avansa Agrikultura beneficiaries as 44 distinct interviews, whether one-off KIIs or focus groups with many people, that included farmers, collectors, input suppliers, vocational schools, and supermarkets. Of these 44 interviews, 42 mentioned an increase in knowledge or skills after receiving Avansa Agrikultura-related support. Twenty-four of 44 interviews mentioned receiving an impactful technical assistance or capacity building training. Of 40 interviews mentioning increased knowledge or skills with a new technology, respondents were most likely to mention learning about the use of improved seeds (23 of 44 respondents), mulch and plastic tunnels (20 of 44 respondents), pesticides (19 of 44 respondents), organic fertilizer (18 of 44 respondents), hand tractors (17 of 44 respondents), inorganic fertilizer (13 of 44 respondents), and drip irrigation (12 of 44 respondents). According to one input supply shop, “Avansa brings farmers to our shop… [and trained them] on how to select good seeds. So now farmers know better about what to purchase from our shop whenever they come here.” And with respect to organic fertilizer, a farmer group in Aileu said “before [Avansa] all organic leaves just wasted, but now we use all for compost,” while another group in Bobonaro used rotted produce as a result of overproduction in the manufacture of compost. Safe pesticide use was cited as a main focus of training at vocational schools, with diverse stakeholders including three farmer groups, three schools, and three input suppliers noting its increased role and importance in agriculture as a result of Avansa Agrikultura training. Manuals distributed in vocational schools PHOTO CREDIT: GABRIELA LEITE-SOARES, SI TIMOR-LESTE TEAM USAID.GOV AVANSA ENDLINE REPORT | 34 Frequently mentioned trainings included those revolving around financial literacy and business with 8 of 44 interviews indicating a positive impact on a pre-existing business and 9 of 44 noting financial literacy training as impactful. Avansa Agrikultura municipal staff noted a marked change in farmers’ ability to conduct financial planning and an openness toward business. Two collectors in Bobonaro and Dili benefitted from training on how to manage their business, with one saying, “for us the collector, the training that we get challenge us to take risk in doing business.” An IP stakeholder present throughout the project noted that “basic numeracy skills were low in the beginning,” however, a government stakeholder mentioned that for one organic composting business, he had seen farmers incorporate Avansa Agrikultura-led financial management training subject matter into their operational budgets. Not all of those interviewed had positive responses with respect to financial training. Two farmer groups in Aileu and Bobonaro stated that they had received financial literacy training once and then never again and were in need of refreshers. Avansa Agrikultura staff with knowledge of business trainings felt that the true impact of the trainings might be muted: “Book keeping record is least likely to continue … because for [farmers] they see it as a report to Avansa rather than for their own benefit.” “For us the collector, the training that we get challenge us to take risk in doing business.” -Collector Despite abundant statements of learnings by beneficiaries of Avansa Agrikultura programming, a small subset indicated insufficient technical assistance given or knowhow gained. Technical assistance from Avansa Agrikultura was cited often by beneficiaries as unsustainable intervention simply because it would disappear after the project concluded. At the forefront of trainings and technical knowledge for which stakeholders expressed apprehension was pest management. Seven respondents including a farmer group, a collector, two municipal coordinators, two supermarkets, and a vocational school concurred in this area, mentioning that Avansa Agrikultura provided important on-demand in-field assistance to manage pests and that farmer groups generally needed continued mentoring. In this regard, Avansa Agrikultura had provided critical extension services. Stakeholders were not optimistic that the Ministry of Agriculture, Forestry, and Fisheries (MAFF) extension workers could fill the gap left in Avansa Agrikultura’s wake, due to lack of motivation and financial incentives. A senior Avansa Agrikultura staff member felt that Avansa Agrikultura’s work with training and input supply “led to a dependence [by farmers]” and that the private sector, namely input suppliers, needed to get more involved in extension services to ensure transfer and retention of knowledge in the future. ATTITUDE CHANGE Logically flowing from enhanced skills and knowledge gain is attitudinal and opinion change by farmers about their ability to succeed in agriculture. This change was most highly reflected in the theme of beneficiaries “becoming independent,” a phrase used often by a wide swath of respondents. Just over one in four of the 44 beneficiary interviews mentioned becoming independent or more independent as a result of Avansa Agrikultura interventions. The transformation was more pronounced among groups with women (10 of 29) and highest among FGDs with youth where six of six groups reported that the project’s activities had enabled them to become more independent. The concept of becoming independent was not limited to beneficiaries: 9 of 22 Avansa Agrikultura, IP, and USAID stakeholders also mentioned the transformation. 35 | AVANSA ENDLINE REPORT USAID.GOV “Also in community or farmers level, even [though] Avansa ended…they will continue those activities that [were] already implemented, because they have got several trainings so they could [work] independently to sustain.” -Youth from Vocational Technical School The idea of becoming independent was reflected in several different examples. Stakeholders mentioned a marked shift from traditional to modern agriculture that enabled substantial rises in income and financial flexibility. Some stakeholders also focused on a shift from subsistence to commercial farming that, according to one female national government stakeholder, “helped [to change] the mentality of the farmer” to being self-employed. Becoming independent was also reflected in an increased ability to make one’s own decisions. In the case of a collector in Bobonaro, an Avansa Agrikultura-facilitated purchase helped increase confidence: “When work as collector and prior to [Avansa] when I do not have the vehicle I do not have confidence in this activity, however now it has changes,” and with a farmer in Ainaro whose planting decisions were altered after Avansa Agrikultura support: “farmers no longer had to depend on the seasons.” Increased ownership over assets due to a shift away from handouts to credit-based input acquisition further contributed to increased confidence and self-efficacy. A municipal￾level government stakeholder reflected this in a comment on the cost-share model: “Unlike other project Avansa … help farmers with 50 or 60 percent. This method teach[es] our farmer to become independent and not guarantee on the project. In addition, Avansa also makes the farmer feel a sense of ownership for what they have.” A female IP stakeholder concurred on the idea of independence by teaching financial responsibility versus giving handouts, “… If you just give, you never teach them to be independent… [with Avansa] they have something that granted them to be independent.” Aside from shifts in mentality, farmers and collectors demonstrated innovation, evidenced by creation of several small niche agribusinesses kickstarted with Avansa Agrikultura grants and training, cold truck businesses from two collectors, and input rental initiatives from two farmer groups. The ET inferred a greater sense of independence and self-employment among farmer groups via substantially increased negotiating power with supermarkets. According to a senior-level Avansa Agrikultura staff and a municipal coordinator, strengthened market linkages with different collector groups and different supermarkets (one farmer group sold to as many as seven different markets) allowed farmers access to more buyers, in turn increasing their ability to negotiate better deals and to disrupt monopolies by supermarkets in certain geographic areas. Confidence in the ability to provide for their families also increased among beneficiaries, expressed explicitly by a processor, “we can definitely purchase some necessity for our family without any trouble compare to before we engaged with the project.” And from a farmer group in Bobonaro, “Confidence in engaging in the agriculture increased … and confidence in providing income for family increase because now we can send our kids to good school.” According to a farmer in Aileu, “The training that I participated [in] through Avansa enabled me to gain new practical skills for my business in growing vegetables. As a result, I am able to support my family even after the passing of my husband. Prior to joining this activity, I struggled to support my family financially… I am [now] able to send my children to higher education in Dili and build this house.” IMPACTS ON WOMEN Avansa Agrikultura characterized the inclusion and improvement of women-related outcomes as an embedded, or cross-cutting theme in its activities. Interventions focused on promoting participation of women in farmer groups, provision of grants to women to start businesses, provision of financial management training, and women-only savings and loans groups. The ET spoke with 47 women (35 USAID.GOV AVANSA ENDLINE REPORT | 36 percent of all respondents interviewed for the evaluation) in 29 separate events, including 12 of 18 farmer groups, and analyzed their responses as one method to understand which parts of Avansa Agrikultura female stakeholders felt were effective. Women and groups that included women mentioned increased skills or knowledge through the learning and application of new technologies in 24 of 29 events. A majority (14 of 24 groups) mentioned new learnings related to the use of improved seeds and pesticides, followed by hand tractors (12 of 24 groups), organic fertilizer, plastic tunnels, and mulch (10 of 24 groups each), mini tiller and pesticides (8 groups each), and inorganic fertilizer and drip irrigation (7 groups each). Compared to all interviews conducted with beneficiaries, interviews with female stakeholders and stakeholder groups that included women saw mentions of technologies about as often, except for the mini tiller (mentioned about half as often as all beneficiaries), the hand tractor (mentioned about 80 percent as often), and pesticides (mentioned about a third more often) (Figure 4). In four of the 29 interviews that included women, a positive impact on a pre-existing business was mentioned; an impact on financial literacy was mentioned in six of 29 interviews. When women and groups that included women were asked specifically about differential impacts of Avansa Agrikultura on women relative to men, 17 of 29 stakeholder events mentioned a specific impact. Of these, 15 indicated women’s participation or increased participation, three mentioned women’s empowerment, four noted financial literacy and two on gender norms. Of those mentioning improved women’s empowerment, one noted “women [do not] need to be dependent to men” and another mentioned increased spending power due to more money. One of the two stakeholders that mentioned impacts on gender norms noted that Avansa Agrikultura had taught them that women were just as able as men to lead farmer groups while the othersaid that women’s inclusion in entrepreneurship by Avansa Agrikultura was beneficial given traditional roles that place women at the forefront of handling finances. With respect to impactful interventions, general horticulture management interventions were mentioned most often by women stakeholders and stakeholder groups that included women (22 of 29 groups), followed by financial literacy (12 groups), savings and loans (11 groups), and nutrition (7 groups). Female farmer in Ainaro PHOTO CREDIT: ACACIO AMARAL, SI TIMOR-LESTE TEAM 37 | AVANSA ENDLINE REPORT USAID.GOV Figure 4: Percent of beneficiaries mentioning the use of new technologies in horticulture activities Note: Respondents could list multiple responses or no responses; therefore, the total percent of responses for each stakeholder group may not add to 100%. The ET asked project beneficiaries, regardless of gender, about the impact of Avansa Agrikultura on women. Of 44 project beneficiary interviews (farmer groups, collectors, input suppliers, supermarkets, and vocational schools), slightly less than half (20 groups) mentioned a positive impact of Avansa Agrikultura on women, with 17 stating increased participation of women in agriculture, six citing some impact on gender norms, four on women’s economic empowerment, and three on financial literacy. With respect to gender norms, the six stakeholders noted themes relating to hearing the voices of women in groups more often, women selling produce in the market and helping their family income, women are now helping with farming in addition to taking care of children, and banks giving loans mostly to women. On the topic of women’s empowerment, the four stakeholders noting a positive impact here mentioned that women are more likely to be running S&L groups, women are now selling in the market and contributing to family income, and women have more spending power since they now have more money. For non-beneficiaries such as government stakeholders and Avansa Agrikultura, IP, and USAID staff, when asked about impact on women, they tended to focus on enumerating women participants rather than transformational impacts on women. This was particularly true of local Avansa Agrikultura staff, the vast majority of which noted a large or increased presence of women in trainings or in business, implying that the presence of women or an increase in the presence of women was an impact. The proportion of women beneficiaries of Avansa Agrikultura overall increased from 32 percent during DAC to 44 percent in Avansa Agrikultura; however the focus on enumeration, or simply stating that more women attended trainings or received benefits, indicated that Avansa Agrikultura did not necessarily target an identified need or gap with respect to women nor did it comment on effectiveness of Avansa Agrikultura interventions on women. In fact, out of a total of 50 pages, Avansa Agrikultura’s Year 5 annual report dedicates only two paragraphs to results of women-related programming. Though activities were not designed for a specific set of women’s needs, a few stakeholders did mention specific impacts of Avansa Agrikultura programs on women. As detailed earlier in the report, women in particular were highlighted by Avansa Agrikultura staff and farmer groups as the main purveyors of processing agribusinesses, as were women in running S&L groups. An Avansa Agrikultura municipal USAID.GOV AVANSA ENDLINE REPORT | 38 coordinator stated that “[women’s] savings and loans groups are in general independent.” Further, seven stakeholders including four Avansa Agrikultura staff, two farmer groups, and a government stakeholder noted a perceived impact of financial training on women; according to a farmer group head in Bobonaro, “Finance literacy benefit women the most, because women are the one who keep the money.” Multiple stakeholders and members of the ET commented on the interaction of traditional gender roles that women play in Timor-Leste and Avansa Agrikultura interventions. They noted that women’s time taking care of children and the household needed to be accounted for and that because of this, women did not necessarily play a large role in physical farming activities. When it came to financial decisions, women play a relatively large role in managing household finances, sharing joint power with their husbands. Findings from household survey data confirmed the above, indicating women had a relatively high degree of financial empowerment in the household,26 slightly more so in decisions regarding loans (borrowing and using the funds) than in decisions related to income and production activities. Qualitative findings supported these data, suggesting that women tended to participate—and find success in—S&L groups and processing, since both complemented the role women already play in helping to manage household finances. Kaebauk noted that in the beginning of the project, most loans were solicited by women, though recently it had become more of a mix of men and women as men saw the benefits that S&L groups afforded. Two government stakeholders commented that in their observation “farmer groups were dominated by women” who were “mostly involved in the financial aspects.” Finally, one senior Avansa Agrikultura staff noted that “most of the farmers that have become collectors [after Avansa support] are women and youth.” VOCATIONAL TRAINING SCHOOLS AND YOUTH IMPACT To assess Avansa Agrikultura’s impact on youth, we looked at youth-specific findings from stakeholder responses as well as the impact of Avansa Agrikultura on vocational training schools. Avansa Agrikultura supported vocational training schools in all five municipalities to strengthen schools’ ability to provide quality technical agriculture information and education to students who are primarily youth. The project accomplished this through training of teachers on agricultural techniques, material support in the form of in-kind equipment and inputs for training demonstration, and revisions to agricultural-related curriculum coupled with advocacy to institutionalize the revised curriculum at the national level. Vocational School FGD in Ainaro PHOTO CREDIT: ZENNY CORREIA, SI TIMOR-LESTE TEAM 26 Findings were from endline data collection conducted in July 2020. The women’s empowerment modules were not included during baseline data collection in 2015, therefore the ET cannot comment on change over time. 39 | AVANSA ENDLINE REPORT USAID.GOV A variety of stakeholders from Avansa Agrikultura staff to government officials and school administrators noted the importance and impact of Avansa Agrikultura support on vocational schools. A senior Avansa Agrikultura staff noted that Avansa Agrikultura’s support had directly turned around three schools with low capabilities, allowing them to be fully operational and reflecting Avansa Agrikultura trainings as recently as August 2020. According to stakeholder comments, Avansa Agrikultura support to schools enabled direct practice of agricultural principles—in particular safe pesticide use—given the project’s provision of inputs for training, facilitated transfer of new business ideas to youth, and that teaching trickled down from students to parents. Support from Avansa Agrikultura to vocational schools was pivotal, particularly since, according to a senior Avansa Agrikultura staff member, in many schools “there is no water. There is no laboratory. There is no access to infrastructure…” On the other hand, a municipal-level government stakeholder noted that despite support from Avansa Agrikultura, teachers at the school needed continuous training and follow up. Teachers and students from one vocational school confirmed the need for continuous training for teachers and mentioned the school would need support for maintenance of inputs provided by Avansa Agrikultura once the project left and the inputs inevitably broke down. Seven stakeholders noted the key role of Avansa Agrikultura in advancing a revised agricultural training curriculum in vocational schools. They highlighted the successful contextualization of curriculum to Timor-Leste, inclusion of teachers and the Ministry of Education (MOE) in curriculum development, the holistic approach of including financial literacy, processing, and preservation in training, and a perceived sustainability of Avansa Agrikultura’s work due to training manuals and videos produced for future students. However, at the time of writing of this report, final approval of the revised curriculum still rested with INDIMO, the national training institute, and prospects for its formalization were unclear. Despite vigorous support for the curriculum from a cadre of teachers across five vocational schools, interviews with INDIMO revealed that the institute was cautious: “the government needs to invest more money to the implementation of the curriculum that INDIMO and Avansa is developing, otherwise we cannot measure or argue that this project is successful…” On a positive note, in the period directly following data collection for this report, two vocational schools were successfully accredited with horticulture certificates by INDIMO. Four stakeholders including three Avansa Agrikultura staff and a vocational training school called attention to the important role Avansa Agrikultura played in graduating four batches of youth interns (a total of 70 graduates), 65 percent of which had been placed with professional jobs. These were mentioned as an example of impact on youth, although not all internships were necessarily in the agricultural sector. At the same time, two additional Avansa Agrikultura staff noted that “connecting interns and placements was haphazard, and not necessarily systematic” and that the impact was unclear. They mentioned that many interns left their placements before the end date of the internships to take higher paying jobs in Dili. According to two municipal coordinators and a senior Avansa Agrikultura staff member, youth might continue in agriculture through internships but that they would require additional mentoring in grant-related activities to be successful in agriculture-related business. “The activities that Avansa provided do benefits to students and also the teachers, because [they did] not merely focus on the trainings but directly implemented it in the field therefore the students could learn the techniques in practice.” -Youth in Vocational Technical School USAID.GOV AVANSA ENDLINE REPORT | 40 The ET carried out 17 FGDs and group interviews for the evaluation, six of which included youth respondents (three were all-youth and three others included at least two youth). The ET interviewed a total of 25 youth who accounted for 19 percent of all evaluation respondents. In all six interviews that included youth, respondents indicated increased skills or knowledge through the learning and application of new technologies. A majority mentioned learnings about use of improved seeds and pesticides (5 of 6 groups each), followed by organic fertilizer and hand tractors (4 groups each), and mini tiller, mulch, and plastic tunnels (3 groups each). Two of the six groups indicated that vocational training provided by Avansa Agrikultura-supported schools had an impact on their work in agriculture and two groups mentioned that women’s participation, regardless of being a female youth or not, had increased and that women had benefitted from financial literacy. When asked about the most effective interventions, all six groups mentioned general horticulture farming techniques, while half the groups mentioned financial literacy and S&L. Of 44 project beneficiary interviews (farmer groups, collectors, input suppliers, supermarkets, and vocational schools), slightly less than half (20 groups) mentioned a positive impact of Avansa Agrikultura on youth, with 14 stating increased participation of youth in agriculture and six citing impacts on vocational training. Stakeholders, however, tended to generalize that youth did not work hard and therefore were not necessarily suited for agricultural work. Four stakeholders, including a government respondent, a collector, and two Avansa Agrikultura staff, felt that youth were driven by money and therefore more attracted to jobs in Dili that paid higher than agriculture. One senior Avansa Agrikultura staff member noted: “Youth are less likely to continue [to be] engaged in farming once the project is terminated. This because most of the want to work in other jobs,” while an Avansa Agrikultura municipal coordinator and a collector generalized youth as lazy and unmotivated. A vocational school noted that “youth are driven by money and we need to change their mindset,” while a collector in Bobonaro observed an increase in youth in farming activities only after Avansa Agrikultura demonstrated that farming could bring in a respectable income. Anecdotal evidence from interviews identified a few successful examples of youth-led agribusiness ventures begun after Avansa Agrikultura support, mostly in the form of grants. The most mentioned was that of the youth group FOSTAL27 which transitioned from a soccer club to an inorganic food processor, and a youth in Ainaro who started a family-run banana crisp processing operation after receiving technical assistance from Avansa Agrikultura at the municipal Vocational Training School. Two Avansa Agrikultura staff and a farmer group referred to other youth who had either become collectors or had started an organic fertilizer processing business. Aside from these few anecdotes, larger scale impact on youth was unclear due to a low youth sample size and for reasons mentioned earlier in this section. 27 The article “FOSTAL: A football club with an agriculture habit” was featured on UNDP Timor-Leste website’s success stories page in September 2020. 41 | AVANSA ENDLINE REPORT USAID.GOV CONCLUSIONS FOR EQ3 Comprehensive training and market access facilitation by Avansa Agrikultura resulted in more skilled, financially aware, and confident farmers. Farmers on the whole mention improved skills and knowledge in a range of technical areas. Beneficiaries are more confident in their ability to engage in commercial activities as a result of shifting from subsistence to commercial farming, improved technical and financial management skills, and increased negotiating power in the market. Farmers are putting more money toward longer-term investments such as savings and children’s education. Avansa Agrikultura support was pivotal in enabling vocational schools to operate at a high capacity though it is unclear to what extent schools can operate post-Avansa Agrikultura. Provisions of in-kind inputs for training demonstration, training to teachers, and revision of curriculum strengthened vocational school capacity. Continued accreditation of schools and formalization of curriculum will be completely dependent on the public sector after Avansa Agrikultura. Schools note a lack of resources to maintain or purchase inputs and request additional training. Aside from a few examples, the project’s impact on youth was muted. Avansa Agrikultura support helped graduate 70 youth interns and grants catalyzed a handful of successful youth-run agribusinesses. Attracting youth to and retaining them in agriculture faces substantial hurdles such as attractive high paying jobs in Dili and perception of little money to be made in agriculture. Though women benefited from Avansa Agrikultura interventions, activities did not appear to be based on a gender-specific need or gap. Women benefitted most from access to grants for businesses and S&L group activities. Stakeholders focused on a simple enumeration of women in Avansa Agrikultura interventions with little to no comment on changes to gender norms or power structures. The ET found little evidence that Avansa Agrikultura interventions were predicated on pre-identified needs or gaps of women. USAID.GOV AVANSA ENDLINE REPORT | 42 EQ4: INTEGRATED APPROACH What interventions, or combination of interventions, did participants view as most relevant or valuable? Why? To answer EQ4, the ET asked Avansa Agrikultura staff about the integrated approach of Avansa Agrikultura, probing on the use of the term “integrated,” historical changes in adopting an integrated approach, and how those staff defined the approach in a practical sense. For beneficiary stakeholders of Avansa Agrikultura, the ET asked to what extent they had received combinations of interventions and why or why not such an approach was impactful, if at all. Many of the findings for EQ4 echo those of EQ1 and EQ2. A PACKAGE OF TRAININGS Among farmer groups interviewed for this evaluation, all mentioned receiving multiple trainings as part of Avansa Agrikultura support. Trainings mentioned tended to be grouped into four areas: (1) technical horticulture and land management, (2) financial literacy, (3) use of inputs, and (4) nutrition. When asked about receiving combinations of trainings, farmers most often mentioned cost-share and input provision, followed by pesticide and fertilizer use. Many also mentioned generally the idea of commercializing production, or the combination of horticulture farming and land management coupled with business training to sell extra produce. Nutrition and other cross-cutting trainings, such as leadership and gender were mentioned sparsely. “For me, all the training complements with our agriculture activities and the training has been very helpful and useful.” -Aileu Farmer While the demonstration plots (“demo plots”) used by Avansa Agrikultura were discontinued in 2018 according to a senior Avansa Agrikultura staff, six stakeholders referred to the demo plots when asked about combined interventions. The respondents noted the utility of demo plots in showing the full gamut of interventions, with two input suppliers mentioning them as the most impactful interventions that Avansa Agrikultura carried out. Demo plots were noted by two stakeholders as impactful for students at vocational schools to be able to see what was possible in horticulture farming. At the same time, one senior Avansa Agrikultura staff noted that demo plots were contentious due to perceptions of privileged treatment of farmers whose land was chosen to receive a demo plot. Stakeholders, from farmer groups to government officials, recognized the importance of a package of activities. According to one Ermera farmer, “The combinations of activities are really positive such [as] growing vegetables, trainings, and link to supermarkets.” Kaebauk recognized that each piece complemented or built on another, noting that Avansa Agrikultura was successful “because all are linked [and] work together [to] contribute to the success, I think nothing was not successful.” Another farmer in an Aileu FGD noted, “We gained new skills that are practical in our profession as a farmer: we are able to know technical tips to succeed in growing veggies of various types and management of our income as well.” While farmers were able to mention the complementarity of some trainings, beneficiary stakeholders tended to simply mention their preferred training or trainings when prompted about a “combination” of activities. Interventions mentioned most often or those mentioned as the most impactful by stakeholders are summarized earlier in this report in findings under EQs 1, 2, and 3. 43 | AVANSA ENDLINE REPORT USAID.GOV Anecdotal evidence from stakeholders suggested that Avansa Agrikultura had a positive reputation and was considered as having accomplished a lot relative to its budget. When asked what Avansa Agrikultura could have done better or differently, four stakeholders asked for trainings on seed production, tropical fruits, and livestock interventions around manure use and protein consumption. SHIFTING MIDCOURSE A major finding of the Avansa Agrikultura midline evaluation in 2017 centered on Avansa Agrikultura’s disparate indicators as being both too many and highly siloed in nature. The report mentioned that trainings were given once and not repeated, and themes such as horticulture farming, business basics, and nutrition were seen as separate interventions and not linked. Over the course of several interviews for this evaluation, senior Avansa Agrikultura staff respondents mentioned a marked shift in programming by Avansa Agrikultura after the midline evaluation in 2017. According to two Avansa Agrikultura staff, cross-cutting themes such as nutrition, gender, and youth were integrated across activities better and Avansa Agrikultura reduced the number of indicators it reported on. Avansa Agrikultura staff in general were highly aware of the integrated nature of the project, and while IP and USAID staff did not necessarily use the term “integrated,” their definitions of the work aligned with the integrated approach. One IP staff noted “DAC, TOMAK, Seeds of Life, etc. All the projects I see, it’s not like what Avansa does for the last five years. They were bringing the agriculture value chains. Every item in agriculture was linked to nutrition, economics, business – everything. Our farmers were not only receiving one training, they received all the trainings.” Municipal coordinators, local Timorese Avansa Agrikultura staff who are closest to the activities on the ground, were particularly aware of the integrated nature of various Avansa Agrikultura programming and could speak clearly about its benefits. They painted a picture that linked new farming techniques with business and financial literacy training to change farmer mindsets from subsistence to commercial, all while linking it with household-level outcomes such as nutrition and savings. “Avansa implement[s] complete value chains. Starting from establishing farmers’ group, then train them on how to increase their production, link them to market, encouraging women and youth in doing business. In addition, the nutritional product also helps farmer not only to increase their production to sell in the market but also to consume a nutritional diet. All these interventions contribute to improve income.” -Avansa Municipal Coordinator Avansa Agrikultura senior staff admit that some activities were begun relatively late and may have been less impactful because they did not have adequate time to take hold. Examples of projects cited as starting or ramping up late were natural resource management, including work with nurseries, and nutrition interventions around cooking. In its sixth year, Avansa Agrikultura worked to introduce a cookbook and nutrient-rich recipes for foods such as nutritious porridge and cakes for young children. The ET found extremely sparse information on these interventions, with one farmer group in Ainaro noting that the nutritious porridge and cakes would not be continued since the foods were not something farmers ate and children did not enjoy the taste. With respect to natural resource management, the midline evaluation found that tree planting was executed haphazardly and sporadically to little effect; at endline one stakeholder, an Ainaro government representative, mentioned that tree planting would not continue because other (non-Avansa Agrikultura) projects had taken it up. Four stakeholders, including two collectors, a farmer group, and a processor, mentioned that Avansa Agrikultura trained them on nurseries though these stakeholders did not expand upon the effectiveness USAID.GOV AVANSA ENDLINE REPORT | 44 of the intervention. Avansa Agrikultura’s Year 5 Annual Report mentioned reaching 14 nurseries with seedling support, and 129 household with Sloped Agricultural Land Technology (SALT) techniques but the ET was unable to interview any of these nurseries or farmers as part of the evaluation. Farmer/nutrition group FGD in Ainaro PHOTO CREDIT: ZENNY CORREIA, SI TIMOR-LESTE TEAM CONCLUSIONS FOR EQ4 Avansa Agrikultura’s shift to an integrated package of interventions to farmers and whole value chain focus after midline ensured a holistic approach to improving outcomes along the horticulture value chain. Avansa Agrikultura adapted well after the midline evaluation to consolidate indicators and adopt an integrated approach. Avansa Agrikultura staff are aware of and support the shift to the integrated approach and many beneficiary stakeholders can point to multiple, reinforcing activities that positively affected outcomes. Avansa Agrikultura staff and beneficiaries diverge somewhat in their perceptions of most valuable interventions. Beneficiaries focused on cost-share/access to credit, fertilizer and pesticide, and horticulture trainings while Avansa Agrikultura staff indicated market linkages as the most valuable interventions. 45 | AVANSA ENDLINE REPORT USAID.GOV EQ5: SUSTAINABILITY OUTLOOK What is the current outlook for sustainability of key Avansa Agrikultura Project interventions? In EQ5, the ET synthesized findings from the first four EQs and additional stakeholder responses to comment on the outlook for sustainability of Avansa Agrikultura’s interventions and impacts. The EQ is broken out by major themes relating to sustainability that cut across household, market, and individual￾level outcomes presented in the previous sections. THE PUBLIC SECTOR AND POLITICAL IMPASSE A cornerstone of international development work, particularly in the age of the Journey to Self-Reliance, stipulates that local capacity is built alongside programming, and local actors, such as government, assume the role of project implementers after initial donor-funded work is completed. The possibility of the public sector assuming any of the Avansa Agrikultura interventions after the project concludes was met with pessimism by stakeholders. Two input suppliers and one supermarket doubted the government’s ability, with one stating “the GOTL commitment is questionable because they [haven’t] demonstrate their commitment to the agricultural sector yet.” Two vocational schools similarly questioned the financial and human resources capacity of the public sector, with one noting that “we still need international agencies to support in agriculture sector development.” A municipal-level government representative admitted, “as government just coordinate, we do not do intervention,” as did a MAFF representative, who conceded that “MAFF does not have the competence to provide market to the farmer.” Most Avansa Agrikultura and IP staff were reserved about the prospects, though one staff who was knowledgeable on nutrition interventions was optimistic that the government would work with international donors such as the World Health Organization (WHO), World Food Programme (WFP), and other United Nations (UN) agencies to continue nutrition programming. Seven stakeholders, including three Avansa Agrikultura staff, an IP, and two government officers mentioned the political impasse taking place in Timor-Leste as a major impediment for any sort of public sector involvement in agriculture. They pointed out the extremely low investment of GDP in agriculture (1-2 percent) and frequent political turnover, with the resulting changing out of civil servants with new elections and political party changes. According to one national-level government respondent, “the government hasn’t demonstrated interest in agriculture sector adequately.” An Avansa Agrikultura senior staff member and two government officials concurred that the government relied on Avansa Agrikultura to help push pieces of the National Strategic Plan of MAFF forward, including increasing the productivity of agricultural products, such as horticulture. HORTICULTURE FARMING Stakeholders generally believe that horticulture farming and the increased production associated with it will continue after the project concludes. Fifteen of 22 Avansa Agrikultura, IP, and USAID stakeholders agreed on this point, as did three of eight input suppliers, three of four markets, three of five collectors, four of eight government stakeholders, and 11 of 18 farmer groups. According to one senior Avansa Agrikultura staff, 60 percent of farmer groups are independent and “they can do their business without project support,” and, per a farmer group in Aileu, “yes, [horticulture will continue] because the project already teach[es] us what we see as useful and relevant for our field.” Two farmer groups remarked that horticulture sales had become their primary source of income, even replacing coffee in one case. According to one farmer in Bobonaro, “We still continue the [production], because we need money, and we can get money through this activity.” USAID.GOV AVANSA ENDLINE REPORT | 46 FARMER GROUPS The sustainability prospects for farmer groups were unclear. A senior Avansa Agrikultura staff member noted that after Avansa Agrikultura leaves, individual farmers will be focused on their own plots, and therefore less likely to work as a group; however, the groups could remain elastic, coming back together to provide a united front for future donors. Three Avansa Agrikultura staff admitted that farmer groups had disbanded in the recent past due to a lack of tangible material benefits from Avansa Agrikultura. A farmer group in Ermera confirmed that many of its groups had reduced in number because members had found other jobs or youth felt that the work was too hard. An Avansa Agrikultura municipal coordinator and a municipal-level government official in Bobonaro noted that farmer groups often had internal conflicts with Avansa Agrikultura acting as an arbiter. Aside from the facilitation of mass training, the utility of farmer groups to farmers themselves was unclear to the ET. At the same time, the ET found no evidence to suggest that disbanding of farmer groups would necessarily be a threat to sustainability of the value chain. “Even if Avansa terminated we will continue our activities and this group will continue because we have the market and already established the connection with them.” -Farmer Group in Ermera On the positive side, the ET noted several instances of snowball interest, where non-Avansa Agrikultura farmers wanted to join in Avansa Agrikultura interventions after observing the impacts of the project on participating farmers. Nine stakeholders, including four farmer groups, all three processors, a collector, and an IP stakeholder mentioned interest from non-Avansa Agrikultura groups in joining. In particular, respondents mentioned that interested parties wanted to start their own farmer groups, farm strawberries, plant vegetables, and begin processing operations after observing Avansa Agrikultura farmer groups. Two processors noted that youth who saw successful operations were particularly keen on starting up their own businesses. According to one farmer group in Ainaro, “[Avansa] also inspires other groups to grow their own product and form their own group.” NUTRITION Non-beneficiary stakeholders did not discuss the topic of nutrition and the sustainability of nutrition interventions and impacts. Qualitative and quantitative data suggest increases in MAD and food group consumption, yet retention of fundamentals of nutrition is uncertain. One Avansa Agrikultura senior staff member thought the gains would be sustainable, as did a USAID stakeholder who said, “I suspect the nutrition ones, those are behavior change, so those are likely to sustain as well.” However, a different senior Avansa Agrikultura staff noted that “the nutrition side will need more work to sustain.” Furthermore, a senior Avansa Agrikultura staff familiar with the nutrition interventions felt that the government would continue to rely heavily on health sector donors such as WHO, UNICEF, and WFP to carry out nutrition-related programming. 47 | AVANSA ENDLINE REPORT USAID.GOV Strawberry plants in Ermera PHOTO CREDIT: ZENNY CORREIA, SI TIMOR-LESTE TEAM PRIVATE SECTOR LINKAGES A wide range of respondents who commented on market linkages established through Avansa Agrikultura were optimistic about sustainability of those linkages, including 12 of 22 Avansa Agrikultura, IP, and USAID stakeholders, three farmer groups, and three government stakeholders. According to one senior Avansa Agrikultura staff member, “Definitely the market linkages are self-sustaining. Participants are motivated, markets for same reason,” and “I think the collectors’ network and connection of farmers to end market is going to be ongoing. I think that several of the linkages that have been made with input suppliers particularly from Indonesia will have long term sustainability potential.” A processor thought that the connections made would enable sustainability: “Our production will continue even without Avansa because we already have the connection with the market.” Three farmer groups, one municipal government official, one supermarket, and one collector made similar statements. Four Avansa Agrikultura staff, one farmer group, and three of five collectors concurred that linkages with collectors would continue; according to a senior Avansa Agrikultura staff member, one reason is “because it’s their own investment.” Two government stakeholders felt that the existence of official MOUs linking farmer groups to markets help solidify their sustainability; however, a member of the ET recounted that farmer groups, after time, tended to find MOUs highly inflexible and anti-competitive in nature. The ET saw no advantage for farmers to continue the use of MOUs. Seven of eight input suppliers thought linkages between their businesses and farmers would continue. According to an input supplier in Bobonaro, “as long as farmers continue to grow vegetables, they will continue coming to our shop and I know that it will not stop even after the project ends,” and, per another input supplier in Dili, “The market linkages especially with the supermarkets and inputs suppliers will likely…continue because the connection has been established.” According to a senior Avansa Agrikultura staff, “I think that several of the linkages that have been made with input suppliers USAID.GOV AVANSA ENDLINE REPORT | 48 particularly from Indonesia will have long term sustainability potential,” while another said “Collectors, retailers, input suppliers selling, those aspects are catching and I have confidence they will continue.” An input supplier indicated that their business model was sustainable because they were able to import most of their inputs from Indonesia which was cheaper than purchasing locally. Finally, an additional input supplier noted that increases in income garnered from horticulture farming meant that farmers would continue to be able to purchase inputs. While most statements were positive in nature with respect to linkages, a farmer group in Aileu and Kaebauk, noted some uncertainty: “it will be better if [an]other project can take over,” each commented. When a major supermarket, Kmanek, went bankrupt in 2019, farmer groups that were linked to the market suffered immediate and experienced drastic losses in income. According to a stakeholder in Bobonaro, “Supermarket like Kmanek no longer comes to Maliana to purchase fresh vegetables. This has affected farmers’ income significantly in a negative way.” The Kmanek example demonstrates the potential vulnerability and inherent risk with linkages between farmers and markets, particularly those which involve multiple farming groups and only one market. 49 | AVANSA ENDLINE REPORT USAID.GOV TECHNICAL ASSISTANCE Most stakeholders believed that learnings from technical assistance would be sustained among farmers for basic horticulture planting techniques and crop management. Seven of 18 farmer groups mentioned they would continue horticulture farming because they knew how and the large increases in income had been transformational in their lives. According to a farmer group in Ainaro, “We will continue practice on what we have learned from Avansa…For us Avansa might be gone but their knowledge and lessons will remain with us.” And another in Dili said, “Avansa could end, but the skills and knowledge that we have as a big value for us to continue to grow vegetables.” A third group in Ermera commented, “If Avansa terminate it program we will continue to engage in our farming activities and practice the farming technique and other knowledge that we have learned from Avansa.” These views were shared by a municipal-level government official who noted, “…although Avansa finished…the technical skills and knowledge that farmers have or got from the capacity building, the farmers will continue [to use].” As mentioned in the section on EQ3 earlier in this report, one area that stakeholders commented on as needing additional technical assistance was in pest management (seven stakeholders). Two municipal coordinators, three farmer groups, and two collectors felt that technical assistance would disappear altogether once Avansa Agrikultura ends. Water pump in Ermera PHOTO CREDIT: NAZARIO DOS SANTOS, SI TIMOR-LESTE TEAM Capacity building and knowledge transfer on how to maintain inputs that Avansa Agrikultura introduced is a major component of sustainability examined by the ET. One farmer group in Ermera noted that Avansa Agrikultura trained them on how to maintain inputs, and a major Dili-based input supplier said that the supplier goes to farms to install and fix inputs as needed. Six stakeholders, including two farmer groups, felt that maintenance of inputs could be problematic, particularly more complex systems such as drip irrigation and plastic tunnels. These stakeholders, in addition to a vocational school, indicated that technical assistance from Avansa Agrikultura was still needed since none of the groups would have the money to repair inputs should they break. The ET found that extension services were virtually non-existent in Timor-Leste. MAFF admitted that extension worker numbers are limited and extensionists sometimes cover multiple sucos in one area. Three municipal coordinators noted that while they had included some MAFF extension workers in Avansa Agrikultura trainings, they didn’t expect MAFF extension services to be available in the future due to insufficient numbers and a dependence of extensionists on per diems. One municipal coordinator remarked critically that “during five years [working]… the MAFF extension workers are not cooperative. We work with them to work together … at the moment, they cannot come to good cooperation with us.” A municipal coordinator noted that “local NGOs” could take up training but USAID.GOV AVANSA ENDLINE REPORT | 50 declined to mention which ones. As mentioned in EQ2, at the time of the writing of this report, the ET found evidence to suggest that several input suppliers and markets were providing informal technical assistance to farmers at point of sale and on farms, as was the agro-input association. Plastic Tunnel in Bobonaro PHOTO CREDIT: ACACIO AMARAL, SI TIMOR-LESTE TEAM VOCATIONAL TRAINING A senior Avansa Agrikultura staff member with knowledge of vocational school training interventions felt that “trainings will no doubt continue by INDIMO” since the curriculum was expected to be approved. One administrator of a vocational school felt that teachers had been sufficiently trained and, in conjunction with an approved curriculum, could continue their work after Avansa Agrikultura concluded, a thought also shared by a municipal-level government official in Aileu. Of the five FGDs with vocational school stakeholders, four groups felt Avansa Agrikultura’s work would be sustained, with one group citing the existence of soft-copy manuals and videos as evidence of sustainability. On the other hand, one of five groups was cautious: “However this school still wanted the project as Avansa to continue to accompany the teachers and students in agricultural techniques.” Further, as mentioned in EQ3, INDIMO felt that more investment was needed by the government in supporting the implementation of the new curriculum for it to be truly successful. One training school administrator was more direct: “It would be almost impossible for our students to have hands-on learning [after Avansa],” referring directly to the fact that Avansa Agrikultura had supplied all training inputs in-kinds, and the schools did not have the money to maintain or purchase new inputs. With respect to the youth internship program, one Avansa Agrikultura staff familiar with the work indicated that it “involved too many actors (university, employers, UNDP, students, Avansa)… planning need to be flexible, very difficult to have them [all] together.” However, a municipal coordinator stated that UNDP has already taken up the program and would continue it after Avansa Agrikultura ends. FINANCE AND FINANCIAL LITERACY The ET found that inability of farmers to pay for inputs and a general lack of resources among farmers will continue to be a barrier to sustainability. Input prices “are still too high” according to one municipal coordinator, with an FGD of Dili-based farmers citing plastic tunnels, mulch, hand tractors, and mini tillers specifically as still being out of reach for them financially. Financial literacy and business training were linked directly to new business ventures among a select few youths and collectors, the results of 51 | AVANSA ENDLINE REPORT USAID.GOV which suggested that continued financial literacy training could be beneficial in the future. A senior Avansa Agrikultura official with intimate knowledge of S&L groups felt that “S&L groups in general are independent [of Avansa],” and a municipal coordinator, an S&L group, and one vocational training school felt that S&L groups would continue. On the other hand, as mentioned earlier in EQ3, the ET found that some farmers filled out bookkeeping notes as a report for Avansa Agrikultura and not necessarily for their own benefit. Calendar from Avansa demonstrating recommended expenses, financial literacy tips PHOTO CREDIT: NAZARIO DOS SANTOS, SI TIMOR-LESTE TEAM With respect to loans, repayment rates to Kaebauk were exceptionally high and were not expected to be a barrier to future loan disbursement from Kaebauk. However, anecdotal evidence from farmer groups indicated that repayment among S&L groups was variable and the cause of dissolution of some S&L groups. A senior Avansa Agrikultura official felt that the triangle model was well established and would continue unless threatened by government subsidies: “The system is in place, and whoever follows this system setup will be fine—donor or otherwise. But if government keeps giving subsidies like seeds and technologies, it will destroy this system.” Five stakeholders mentioned government subsidies and handouts from donors as threats to sustainability; as mentioned earlier in EQ2, some farmers refused to participate in Avansa Agrikultura or dropped out since they did not receive direct material benefits. With respect to loans, Kaebauk and four of eight input suppliers were cautiously optimistic about loans to farmers commenting that many farmers would continue to take out loans, though not all farmers might be able to do so after the Avansa Agrikultura cost-share model disappears. Kaebauk was confident that the MFI would continue providing credit for farmers since it was the only provider in the country of such loans, however, Kaebauk’s sole status meant that the entire triangle system was highly vulnerable should any issues arise with the MFI. USAID.GOV AVANSA ENDLINE REPORT | 52 “It is important to note that Avansa never provided any tangible support to farmers.” -Avansa senior adviser MIDDLE-MARKET COLLECTION Transportation costs remain a major limiting factor for farmers and markets, especially those farmer groups in rural, hard to reach areas and those furthest from Dili. Stakeholders associated transport costs most frequently with fertilizer (three farmer groups noted it), which is relatively larger and heavier to transport than other horticulture products. Collectors will continue to play a pivotal role in the middle of the value chain, filling gaps when key services, such as access to finance and inputs cannot reach farmers. A collector in Ermera said that their business was oftentimes providing seeds to farmers for purchase or on credit because farmers either lacked sufficient cash or access to input suppliers; an Avansa Agrikultura senior staff noted that collectors could help to strengthen the consignment model in the future. Collectors in Dili, Ermera, and Bobonaro began to disburse microloans to farmers after business training from Avansa Agrikultura, an easy and quick way for farmers to get access to credit. Collectors have generally been highly prosperous and critical in the entire value chain, and many farmers have chosen to switch to collecting as their primary employment due to real gains in income. Collectors maintain important relationships with value chain actors and are highly trusted by farmers and markets. Increased numbers of collectors provide diverse options for farmers, enabling them to increase their competitiveness and bargaining power. Collectors ensure a consistent flow of financial liquidity to farmers since they pay up front and visit often. And in times of need, such as during the COVID-19 pandemic, collectors were agile and adaptable, becoming urgent buyers for farmers when supermarkets drastically lowered demand and transport options. SEED SUPPLY CHAIN Seed supply chain challenges limit the ability to scale horticulture farming and the overall value chain in Timor-Leste. According to several stakeholders including two input suppliers, two farmer groups in Bobonaro, and an Avansa Agrikultura senior staff member, local seed supply in Timor-Leste was variable, costly, or low quality. To account for this, two input suppliers noted that they import cheaper seeds from Indonesia, an action which was essential to ensure that their businesses were viable. Farmer groups which the ET spoke to in Bobonaro also imported seeds from Indonesia because those in Dili were too expensive. Per one Bobonaro-based input supplier, purchasing inputs (seeds) from Dili impacted their business model: “When we purchase input supplies from Dili that cost more we are unable to get any profit because farmers cannot afford to purchase input supplies when it costs too much for them.” A senior Avansa Agrikultura staff with knowledge of the seed supply chain remarked that, “seed capacity in Timor is not at commercially viable levels” and that Avansa Agrikultura often had to provide large quantities of seeds to farmer groups. One supermarket noted that they were currently using an innovative model of providing seeds on credit to farmers with farmers repaying the market in produce. A farmer group in Dili noted a desire to learn how to produce seeds for themselves and an input supplier noted that a lack of local seed production in Timor-Leste contributed to seed supply challenges. 53 | AVANSA ENDLINE REPORT USAID.GOV INFORMATION AND COORDINATION A common theme heard by the ET, and one which poses a major risk to sustainability of horticulture farming, was overproduction by farmers. Overproduction was mentioned by nine of 18 farmer groups, two of four markets, three of eight input suppliers, three of five collectors, three of five municipal coordinators, and two government stakeholders. An Avansa Agrikultura municipal coordinator noted that the short planting life cycle of certain vegetables, like mustard, were attractive to farmers since they can be planted, harvested, and sold more frequently, yet these factors made the plant highly susceptible to overproduction. The volume of demand from supermarkets was inconsistent, according to another municipal coordinator, which led to farmers under or overproducing. Given the time frame of data collection, it is unclear if the inconsistencies were due to COVID-19-related effects on demand. Two markets indicated that collectors did not accurately reflect supply and demand shifts, and corresponding price changes to farmers. In sum, a lack of understanding on market dynamics around supply, demand, and price by farmers and collectors resulted in poor information flow between farmers and markets and unfulfilled expectations around quantity and price. The situation was exacerbated by the fact that farmer groups do not coordinate amongst each other regarding what products to plant. “They [farmers] want the assistance of Avansa because their communication with retailers and supermarket wasn’t successful.” -Avansa Municipal Coordinator Seven different stakeholders mentioned the potential role that horticulture associations could play in the value chain to resolve issues around supply and demand mismatch. These associations could contribute to coordinating and convening farmers and their activities and disseminating accurate, up-to-date market information to all parties. The approach is promising, however the associations were relatively new and still needed capacitation (support to develop and strengthen business and operational functions) and formalization (by the relevant national government agencies) at the time of data collection, according to an Avansa Agrikultura municipal coordinator. The current agro-input business association, which was supported by Avansa Agrikultura, believed that it would still have members after Avansa Agrikultura concludes. A municipal-level government stakeholder in Bobonaro believed that once the horticulture association had been established there at a future date, the market for horticulture products would increase. EXCESS SUPPLY AND PROCESSING There are several steps that could be taken to alleviate excess supply of horticulture products in the market, including exportation and processing. The idea of exporting was typically met with excitement and optimism by the few stakeholders who spoke about it. USAID respondents familiar with the explosive growth of strawberries after Avansa Agrikultura support felt that the fruit or its byproducts (e.g., jam, juice) was a good candidate for exporting. The same stakeholders were also cautious about the possibility, given the nascent state of Timor-Leste’s quality and sanitary control regulations. One training school and one IP stakeholder concurred that the government had no capacity currently to export. Processing continues to face challenges. As mentioned in the processing section of EQ2, the cost of inputs and transport are high, prices of some processed products are noncompetitive, and demand in markets for processing goods is low. According to one municipal coordinator, “food preservation will not continue because they require continuous Avansa support.” Two Avansa Agrikultura staff and an IP USAID.GOV AVANSA ENDLINE REPORT | 54 stakeholder confirmed this finding, specifically calling out processors’ heavy reliance on the project and Avansa Agrikultura’s outsized role in communication and marketing of processed goods. Two stakeholders noted that if any product could succeed in this area, it would be strawberries, due to their explosive growth in popularity after Avansa Agrikultura-facilitated introduction to the market. WOMEN AND YOUTH INVOLVEMENT With respect to women, impact was hard to gauge and therefore sustainability of interventions and/or impacts may require a different approach to working with women. With respect to Avansa Agrikultura interventions, success was seen most often in agribusiness ventures and in S&L groups. One senior Avansa Agrikultura staff stated that S&L groups were “the crux” of the impact on women, since it was the intervention where groups of women had been most likely to succeed and continue operating. Increases in income garnered through these initiatives helped put more purchasing power in the hands of women, who already have relatively high power in financial decision making in the household. At the same time, cultural expectations and traditional norms put a heavy time burden on women, who are traditional caretakers of the household in Timor-Leste. The ET found it difficult to define impact of Avansa Agrikultura interventions on youth and noted myriad challenges to sustaining interventions that sought to bring youth into agriculture livelihoods. Retaining youth faced barriers such as the strong allure of attractive, high paying jobs in Dili over agriculture jobs and the perception that farming does not constitute a lucrative employment opportunity. According to two municipal coordinators and a senior Avansa Agrikultura staff, the project had some successes in connecting youth to viable opportunities through social media, and grant opportunities gave youth an initial leg up and a chance to overcome barriers. As mentioned earlier, two processors noted that youth who saw successful Avansa Agrikultura-supported businesses were motivated to start their own due to perceived income-related opportunities among other youth-started businesses. The youth group FOSTAL, an Avansa Agrikultura-supported new agribusiness, received substantial support from the UNDP youth center in Timor-Leste and Avansa Agrikultura to get off the ground. WATER Water, either too little or too much, continues to play an important limiting role in success for producers in the horticulture value chain. Two municipal coordinators noted that “during dry season most farmer cannot produce," and, for strawberry farming in particular, farmers “do not have access to water or their terrain is not favorable.” One senior staff noted the agile response of one farmer group that switched to livestock rearing in an area that lacked sufficient water access for horticulture farming. An input supplier in Bobonaro remarked on the lack of competitiveness of Bobonoaro farmers compared to those in Aileu and Ermera due to limitations in accessing water. Two Avansa Agrikultura advisors confirmed that in order to scale Avansa Agrikultura interventions, future projects will need to consider access to water and land types. With respect to too much water, according to a senior Avansa Agrikultura official and one farmer group in Bobonaro, erosion control and water management in the form of holistic natural resource management programming will be essential in the future for certain municipalities, such as Ainaro and Bobonaro. Quantitative data from the household survey comparison confirmed that too little or too much water were the most common shocks encountered by households in the previous 12 months. 55 | AVANSA ENDLINE REPORT USAID.GOV “We also have difficulty with access to water! So, our farmers here are having a tougher time to grow diverse types of vegetables and that makes us less competitive compared to Aileu and Ermera.” -Bobonaro Input Supplier DISTANCE AND ROAD QUALITY Distance and road quality are additional external factors that affect the sustainability of Avansa Agrikultura interventions and their impacts. Twelve stakeholders, eleven of which were in Bobonaro, Ainaro, and Ermera, noted that distance and road quality constituted obstacles to selling produce and accessing inputs. Five additional Dili-based Avansa Agrikultura, IP, and USAID stakeholders confirmed the finding. A senior Avansa Agrikultura staff noted that roads are so bad in some places that even collectors cannot access certain communities. An input supplier in Bobonaro indicated that while markets initially came as far as Bobonaro to purchase produce, they were not doing so as much in the recent past; though this could have been a result of COVID-19. Findings from the Avansa Agrikultura midline evaluation and from interviews with two input suppliers during this evaluation indicated that Aileu was seen as the “priority” municipality for Avansa Agrikultura and had received the bulk of support, probably due to its close proximity to Dili. Interestingly, one Avansa Agrikultura municipal coordinator suggested that it was more difficult to get farmers close to Dili to take up horticulture due to the presence of more attractive options in the city. In addition to road quality, several stakeholders in Bobonaro discussed the relatively low soil quality in the municipality as a limiting factor for planting and use of certain inputs. MARKET SATURATION AND EXPANSION TO OTHER SECTORS Anecdotal evidence suggested that non-Avansa Agrikultura groups and individuals had started to replicate the Avansa Agrikultura model, such as non-Avansa Agrikultura supported middle-market collectors buying and selling new horticulture products and organic fertilizers, and even one account of MAFF implementing tunnels. Despite these encouraging findings, a high-ranking national government official interviewed for this evaluation, along with members of the ET and several collectors, noted that the market for vegetables in Timor-Leste is too small and is slowly becoming saturated with the increase in horticulture farmers. To account for this saturation, stakeholders suggested sectors outside of supermarkets and products other than horticulture for future projects to promote among farmers and the private sector. Raspberries grown in Ermera PHOTO CREDIT: ZENNY CORREIA, SI TIMOR-LESTE TEAM USAID.GOV AVANSA ENDLINE REPORT | 56 Seven stakeholders, including five Avansa Agrikultura staff, an input supplier, and one vocational training school discussed realistic options for scaling, including expansion to high value crops, expansion to additional municipalities, processing, and exporting. For high value crops, three of four markets recommended moving into tropical fruits such as pineapples, dragon fruit, bananas, and mangos. Two stakeholders recommended introducing livestock-related initiatives, such as manure-related interventions. Livestock-related expansion fits with comments from an Avansa Agrikultura staff who noted that input supply shops are expanding their selection of inputs beyond horticulture to fisheries and livestock products. Stakeholders suggested that horticulture farmers expand beyond supermarkets. A municipal-level government official in Aileu suggested farmers could partner with the government to provide school lunches. A senior Avansa Agrikultura staff member suggested that farmers link with the tourism industry, such as hotels and restaurants, or extractive industry such as oil derricks. Collectors indicated that they were already selling to restaurants and hotels on a micro level. Finally, two stakeholders noted that the “local” food movement is budding in Timor-Leste, with a new regular farmer’s market in Timor Plaza and several small initiatives to support local farmers. “Despite my limited knowledge of Avansa- I have heard so much about how Avansa has transformed farmers’ lives. I hope that this initiative can be scaled-up in other parts of Timor-Leste.” -National-level GoTL official DONOR PROSPECTS Throughout data collection, the ET noted several instances of donors interested in taking up or replicating parts of the Avansa Agrikultura model. While the Avansa Agrikultura project’s successes come arguably from its integrated model, donors interested in continuing critical parts of the model could contribute to overall sustainability. Stakeholders mentioned several donors already active in the horticulture value chain or donors that were keen to join. A municipal coordinator noted that input suppliers are already supported by international NGOs and mentioned Mercy Corps as a major interested player in the horticulture sector. Farming for Prosperity (TOMAK) was mentioned by two stakeholders as another potential donor interested in continuing financial literacy training, as well as work with collectors and horticulture farming. At the time of data collection, Plan International was conducting horticulture programming targeting youth in Aileu. World Vision was also working in Aileu with nutrition and savings and loans groups. GIZ, the German bilateral aid agency, was mentioned as interested in continuing the youth internship work with the UNDP-funded youth center. PERMATIL, an Australian NGO focused on permaculture, works with vegetable planting and grants to farmers. An IP stakeholder mentioned that Kaebauk intended to continue financial literacy training as well through its parent company. CONCLUSIONS FOR EQ5 Horticulture farming is rooted among Avansa Agrikultura farmers given real changes in earning power and established linkages with markets. Farmers overwhelmingly note that they will continue farming vegetables given tangible benefits reaped, particularly in the form of increased income and formalized linkages with markets. Observable successes are attracting new customers to horticulture. Linkages have helped to realize growth not only for farmers but also other actors along the value chain. 57 | AVANSA ENDLINE REPORT USAID.GOV Supply and demand mismatches between farmers and supermarkets will continue in the absence of concerted inter-farmer and collector-market coordination and information flows. Consistent demand, supply, and prices cannot necessarily be guaranteed in the market and farmers, collectors, and supermarkets must be flexible to market volatility. Budding horticulture associations could play a role in providing timely and accurate information to farmers and collectors and coordinating efficient and appropriate planting schemes. Farmer groups may eventually dissolve as farmers increase their independence and bargaining power, but groups could remain elastic to respond to follow-on donor interventions. It may be difficult to motivate farmers to continue meeting in groups without a coordinating outside entity, especially when monetary benefits are few. There is evidence, however, that the group model is inspiring non-Avansa Agrikultura farmers to replicate it. Future donors could tap back into the already-established group structure. Given horticulture market saturation, supermarkets may slowly winnow linkages to larger, more established individual farmer operations. Loss of Avansa Agrikultura technical assistance may slowly lead to reduced production, though there is opportunity for local actors to follow in Avansa Agrikultura’s wake. Beneficiary groups expressed continued weakness in the areas of pest management, input maintenance, and training support for teachers at vocational schools. Public sector extension services are weak and incapable of filling the gap left by Avansa Agrikultura. Input suppliers and some markets are already providing technical assistance to farmers and could continue to do so. Horticulture associations can play a role as well. The triangle model is active, impactful, and attractive to outside donors, yet its foundation on a sole provider of credit (Kaebauk) is a precarious risk. Other banks are risk averse to entering the agriculture credit market. Cost-sharing has been transformative for farmers but ensuring future access to inputs for farmers may require a concerted donor effort to cover parts of the cost. Collectors will continue to fill critical gaps in the value chain. Collectors provide crucial services when input suppliers, markets, or credit providers cannot reach farmers. Collectors prospered under Avansa Agrikultura and demonstrated their ability to provide urgent support as buyers and transporters of farmer product during the COVID-19 quarantine. Small scale processing is possible in Timor-Leste but will need to be aligned to a viable market demand to succeed. With and without Avansa Agrikultura support, SMEs in the horticulture sector fill important niche areas with substantial sales. Inputs for many processors, especially food preservers, are prohibitively expensive, and products do not sell especially well in local markets due to low demand or noncompetitive pricing with imports. Nevertheless, occasional prospects, such as strawberries, stand out among contenders. Inconsistent seed supply combined with low road quality and water scarcity in some areas continues to limit the full potential of horticulture farming in Timor-Leste. Seed supply is variable and local seeds are seen as low quality. Input suppliers and farmers in border regions import seeds from abroad, often illegally. Despite new water conservation techniques and inputs from the Avansa Agrikultura program, geographic-specific factors around road access, water scarcity, and extreme weather continue to limit some farmers’ ability to grow year-round or access markets. Attracting youth to agriculture will require a perception and mindset change that agriculture is a viable employment/livelihood. Retaining youth in agriculture faces several barriers, USAID.GOV AVANSA ENDLINE REPORT | 58 such as high-paying jobs in Dili and negative perceptions around viability as a livelihood. Youth are encouraged when they observe other youth succeeding in agricultural ventures. Youth appear to benefit in business-related skills when receiving close mentoring from entities such as advisers or the youth entrepreneurship hub. Avansa Agrikultura programming is increasingly expanding the role of women in the household and community level. Findings suggest Avansa Agrikultura programming was attended by a balance of men and women. Women are being exposed to horticulture trainings more so than before, with farmer groups headed by more and more women. Findings show that all-women S&L groups have particularly excelled under Avansa Agrikultura, with substantially increased savings and relative independence at the end of the project. Women have relatively high decision-making power around household financial decisions which fits well with their participation in S&L groups. The public sector cannot be counted on to support or take up Avansa Agrikultura interventions across the value chain for horticulture, but there is ample opportunity for donors to do so. Most stakeholders, including government itself, doubted the ability of the private sector to take up any pieces of Avansa Agrikultura programming after Avansa Agrikultura concludes. Many donors exist who are either already working in the horticulture value chain or are interested in pieces of Avansa Agrikultura programming. There are opportunities to link horticulture farmers to new sectors and explore targeting products other than horticulture among farmers. Oversupply of horticulture is already happening in the country; processing and exporting do not appear to be viable options to ease the burden. Some farmers are already expanding into livestock and fisheries. Markets demand tropical fruits. Tourism, extractive, and public education (schools) are potential end consumers in the horticulture value chain. 59 | AVANSA ENDLINE REPORT USAID.GOV 6. RECOMMENDATIONS Recommendations synthesized from the conclusions are presented below. Each recommendation includes associated evidence and rationale for its proposal. Recommendations are grouped by stakeholders to which they are targeted: USAID, IP of a follow-on project, or both. TARGETED STAKE￾HOLDER RECOMMENDATION EVIDENCE/ RATIONALE USAID & IP Continue to refine geographic specific and appropriate interventions. Comprehensively assess what works where for horticulture, considering geography, climate and weather, water access, road accessibility, and farmer disposition. Target specific crops and inputs where they will have the largest cost-benefit. Strengthen collector networks in further municipalities. Where supermarkets are too far, link farmers to other nearby sectors (tourism, extractive, public schools) that demand horticulture products. The Avansa Agrikultura interventions are not one size fits all. Municipality-specific differences in water access, climate, access to markets, and road quality affect the ability of farmers to be profitable. Overproduction is commonplace and resulting in high wastage. Horticulture markets are becoming saturated and other sector opportunity exists. IP Improve seed supply through import regulation reform and increased use of collector networks. Work with the national regulatory bodies and other projects and donors (such as USAID) working on customs and border work to ease seed import restrictions. Engage collector networks to formally transport seed to municipal shops as a means to strengthen the consignment model. Work with input suppliers to play a part (e.g. logistics, transport, packaging, sales, etc.) in creation of a viable seed market. Consider engaging universities for research into production of local seed varieties. Stakeholders consistently report seed supply chain problems with volume and quality. Input suppliers are enthusiastic to remain engaged after experiencing growth in their business. Cheap imported seeds dominate the market, but also must pass through cumbersome import restrictions that oftentimes lead to wastage. Successful policy reform was reported as being more realistic with import/export bodies compared to other institutions. The viability of local seed production is low, but more research could be done to understand its future potential. USAID.GOV AVANSA ENDLINE REPORT | 60 TARGETED STAKE￾HOLDER RECOMMENDATION EVIDENCE/ RATIONALE IP Strengthen municipal horticulture associations to provide information, coordination, and technical assistance to farmers. Give technical assistance to help associations with recruitment, staffing, planning, accreditation, and initial resources to get off the ground. Connect farmer groups and collectors with horticulture associations to disseminate real-time information, coordination, and assistance via messaging and social media. Ensure regular cross￾municipality association meetings to disseminate best practices and promote synergies. Farmers lack authoritative institutions to advocate on their behalf and provide support. Technical assistance from Avansa Agrikultura will disappear and extension services are wholly unavailable to farmers. Accurate information on price and supply and demand in the horticulture market is not always reaching farmers. Farmers do not coordinate among themselves and plant what is easiest, has the shortest life span, or what markets dictate, oftentimes leading to oversupply. Farmers may be unaware of new technologies or opportunity sectors. IP Link collectors and input suppliers to ensure a consistent input supply chain to farther municipalities. Formalize linkages between collectors and Dili-based and municipality￾based suppliers via MOUs and contracts. Incentivize innovative collector-based approaches to servicing hard to reach localities via grants in areas such as cold chain. Continue financial literacy and business training to collectors. The consignment model has not taken off. Collectors are informally disbursing inputs and loans to farmers further from Dili who cannot access either. Collectors are prospering under Avansa Agrikultura, growing their businesses as a result of cost￾share and financial literacy. They have proven to be agile, flexible, and innovative in their approach to business and resolving problems. USAID Partner with a second local financial institution. Continue discussions with donors, local banks, and/or MFIs (other than Kaebauk) to lend to the agricultural sector and participate in the triangle model. Showcase recent successes from Avansa Agrikultura to help persuade reticent financial institutions to participate and help shape their expectations around working with the agricultural sector. Kaebauk is the only provider of credit for agribusiness in Timor-Leste and is dependent on an Australian government guarantee. Avansa Agrikultura farmers have increased incomes, increased savings, increased financial literacy and management skills, and increased collateral to make them more attractive to financial institutions. MFIs may need work in adapting their expectations to work with farmers, such as adapting repayment schemes to be more realistic to harvest schedules. USAID Conduct a gender and social inclusion (GESI) analysis and/or tailor future activities to findings of a GESI analysis. Following on results from a GESI analysis, use participatory approaches to ensure that women and youth are involved in planning of future programming. Avansa Agrikultura interventions did not appear to target a pre-identified need or gap with respect to women. Avansa Agrikultura staff was dominated by men and the extent to which women’s/youth’s voices were heard in the design of gender programming is unclear. 61 | AVANSA ENDLINE REPORT USAID.GOV TARGETED STAKE￾HOLDER RECOMMENDATION EVIDENCE/ RATIONALE IP Consider delivering nutrition and WASH￾specific programming to women within existing S&L groups. Continue to work with local organizations to deliver contextually relevant behavior change interventions that integrate nutrition and WASH and place an emphasis on the importance of nutrition￾related decisions and purchases in the household. Start with all-women S&L groups and expand to mixed gender. Consider expansion and incentivization of all-women S&L groups to ensure adequate geographic coverage. Three local NGOs have been trained to deliver comprehensive nutrition and WASH training. All-women S&L groups have seen success under Avansa Agrikultura, many of which are independent and growing their savings. Women have increased access to finance. Women are traditional caretakers of the household and have relatively high levels of decision-making power in household spending. Improved outcomes in WASH under Avansa Agrikultura are unclear but are complementary to nutrition. IP Utilize youth champions and create a youth fund to encourage youth to enter agribusiness. Identify and highlight successful youth agribusinesses at vocational schools as case studies and in field trainings. Set aside grant money for youth-only funding competitions that award innovative agribusiness ideas. Continue partnering with the UNDP youth hub to offer targeted support to youth who express interest in the sector. Youth are encouraged to engage in agriculture when they see other youth succeeding. Avansa Agrikultura has found success with youth when offering them grants and follow-up mentorship in business. Youth￾focused entrepreneurship and business support opportunities are increasing in Timor-Leste and the UNDP’s youth hub is already engaged and familiar with the project. USAID & IP Diversify horticulture buyers by expanding to different sectors (tourism, extractive, etc.) and diversify agricultural product by including trainings on fruit and livestock. Assess other industries for willingness and viability of partnerships with horticulture producers. Integrate trainings that focus on high-value alternatives to horticulture such as fruit and livestock rearing among farmers that have the capacity and willingness to do so. Anecdotal evidence suggests that vegetable markets are becoming saturated. Collectors are already making initial connections with tourism-sector entities such as hotels and restaurants for horticulture supply. Markets are demanding high-value tropical fruits. Several farmers noted opportunities to utilize livestock for manure-related purposes and to promote animal-sourced fat consumption. USAID.GOV AVANSA ENDLINE REPORT | 62 TARGETED STAKE￾HOLDER RECOMMENDATION EVIDENCE/ RATIONALE USAID Strategically partner with already existing and interested donors to sustain critical pieces of the value chain. Consider public sector engagement with institutions that show promise for meaningful collaboration. Consider targeting donor agencies and organizations to support easy-win areas such as cost-share grants, support to vocational training schools, and youth mentoring in agribusiness. Partner with institutions such as INDIMO, import regulatory bodies, and others that have capacity and willingness for policy reform. Individual donors are already carrying out similar activities along the horticulture value chain and are interested to pick up parts of Avansa Agrikultura. Parts that are the most unsustainable could be the easiest to engage and come with relatively high impact: providing funds for cost-share and vocational school operations and mentoring to youth. INDIMO and import regulatory bodies, as opposed to MAFF, have demonstrated interest in working with Avansa Agrikultura on policy reform. IP Conduct a targeted communication campaign to support the “local” food movement. Riding the coattails of the newly emerging movement in Timor-Leste, target expatriate and other Dili-based communities through print and digital media and events to support locally produced foods. Some stakeholders noted the presence of a small but growing local food movement in Dili. Expatriate communities and other wealthier socioeconomic groups could be specifically targeted. 63 | AVANSA ENDLINE REPORT USAID.GOV 7. ANNEXES ANNEX I: PERFORMANCE EVALUATION SCOPE OF WORK SCOPE OF WORK: AVANSA ENDLINE PERFORMANCE EVALUATION BACKGROUND The United States Agency for International Development (USAID)/Timor-Leste is supporting the Avansa Agrikultura Project, a five-year, $19.2 million project. The project seeks to support increased economic activity and growth for Timorese citizens in targeted rural communities and municipalities. The project is implemented by Cardno Emerging Markets USA, Ltd., and sub-contractors Hamutuk Ita Ajuda Malu Health, the Norman Borlaug Institute for International Agriculture/Texas, and SSG Advisors. The contract with Cardno was signed on April 1, 2015, and the project will be completed in September 2020. The agriculture sector’s performance since Timor-Leste’s independence in 2002 has been disappointing, with national accounts data showing a decline in the mean value of agricultural production per capita from 2000 to 2013.28 Agriculture production systems in Timor-Leste are among the least developed in the world. Subsistence systems dominate all subsectors, and most farming households are highly risk averse. Child nutritional status is still very low. In 2013, USAID/Timor-Leste undertook a comprehensive assessment of opportunities for accelerating inclusive economic growth in Timor-Leste. The assessment found that high un-met local demand for vegetables, supporting the possibility of significant results in the near term. The Avansa Ag project is predicated on the development hypothesis that if USAID/Timor-Leste increases productivity within the horticulture value chain and increases the ability of Timor-Leste’s citizens to engage in economic activities, then sustainable and inclusive economic growth will be accelerated in the agriculture sector. The project strategy has two sub-purposes. The first sub-purpose is defined as “Increased Productivity of the Horticulture Value Chain” and the second sub-purpose is defined as “Increased Ability of Timorese Citizens to Engage in Economic Activities.” At the end of this project, it is expected that about 33,000 people in 250 communities will be earning higher incomes, showing improved nutrition practices, and managing their economic assets more effectively. STUDY OBJECTIVES As part of Social Impact’s (SI’s) M&E support to USAID in Timor-Leste, SI will conduct a final performance evaluation of the Avansa Agrikultura Project. The objectives of the Final Performance Evaluation are: 1. To assess the change in status of the beneficiaries from the beginning of the Avansa Agrikultura Project to the end of the project, acknowledging that causality cannot be established given the evaluation methodology. 2. To generate lessons learned on the Avansa Agrikultura interventions, and the pros/cons of an integrated approach, for use by USAID, the Government of Timor-Leste, and other donors. 28 Government of Timor-Leste, 2015. USAID.GOV AVANSA ENDLINE REPORT | 64 EVALUATION QUESTIONS In order to meet these objectives, the following evaluation questions are proposed: Objective 1: Change in status of beneficiaries This set of evaluation questions seeks to understand changes in priority outcome areas at the individual, household, and market level. While the focus for individual level outcomes is specified in the evaluation question, the specific focus areas at the household level will be financial stability, including assets, income, savings, and credit, and food security, including dietary diversity, food group quantity, etc. The market level will focus on access to markets among farmers via private sector-led market linkages across levels of the value chain, such as inputs, retail, collection, etc. In answering these questions, the evaluation team will also explore the effectiveness of Avansa interventions in contributing to these changes as well as remaining barriers to improvement in each area. 1. Household: To what extent have project activities contributed to improved outcomes of target beneficiary households? a. Which interventions were most effective in doing so and why? 2. Market: To what extent has the project strengthened private sector market linkages across the value chain for target crops? a. Which interventions were most effective in doing so and why? 3. Individual: To what extent have project activities contributed to changes in reported self￾efficacy among beneficiaries, particularly females? a. Which interventions were most effective in doing so and why? Objective 2: Lessons learned This set of evaluation questions aims to generate lessons learned related to integration, perceived comparative value of the Avansa interventions, and the outlook for sustainability of these interventions. In exploring sustainability, the evaluation team will consider key stakeholders that will drive sustained activities, as well as the assets or resources willing to be committed by beneficiaries and other stakeholder groups29 in the future. 4. Integrated Approach: What interventions, or combination of interventions, did participants view as most relevant or valuable? Why? 5. Sustainability: What is the current outlook for sustainability of key Avansa Agrikultura interventions?30 METHODOLOGY SI will use a mixed-methods approach to address these objectives. SI will use the following methods: • Desk review of existing project monitoring documents, data and reports. • Remote and in-person31 qualitative interviews and focus groups with USAID, Avansa, implementing partner staff, and beneficiaries. 29 Key beneficiaries are defined as: farmers, input suppliers, retailers, and collectors. Stakeholders include these groups as well as USAID, Avansa, Government of Timor-Leste, and other implementing partners. 30 Key interventions to be defined in collaboration with Avansa. 31 In-person interviews and focus groups to be completed pending the status of COVID-19 related regulations. 65 | AVANSA ENDLINE REPORT USAID.GOV • Supplementary, secondary quantitative data. The ET will use data gleaned from Avansa’s annual household survey, where appropriate, to nuance and supplement qualitative findings. DESK REVIEW SI will review existing project monitoring documents, data, and reports provided by Avansa, and SI’s own historical documents from the baseline and midline evaluations. This will include PIRS, existing monitoring data, annual/quarterly reports, Avansa’s survey reports, and evaluation reports. The ET will also review secondary data in the form of Avansa’s annual household survey. This data will be used to supplement our findings where possible. The evaluation team (ET) will work with Cardno to identify and obtain relevant documents as needed throughout the evaluation and SI will provide a comprehensive list of reviewed documents in the report’s annex. The ET will begin to fill in a findings, conclusions and recommendations (FCR) matrix based on desk review documents prior to fieldwork and will update the FCR matrix throughout the fieldwork process. TOOLS The ET will develop qualitative interview protocols for key informant interviews (KIIs) and focus group discussions (FGDs) based on the evaluation questions and existing gaps in information gathered from the desk review. SI will recruit at least two ET members who speak Tetum. These team members will advise on cultural context and translation, and they will pre-test, pilot, and adapt tools prior to implementation. The ET will design protocols specific to respondent categories: Avansa and sub-implementing partners, female beneficiaries, heads of household, collectors, input suppliers, retailers, and farmer group heads.32 Interviews will be semi-structured to allow for probing and nuance in responses to the evaluation questions. KEY INFORMANT INTERVIEWS The in-country members of the ET will conduct in-person33 interviews with the following beneficiary groups: famer beneficiaries separated by gender, heads of beneficiary households, collectors, input suppliers, retailers, and farmer group heads. The US-based members of the ET will conduct remote phone-based interviews with Avansa and sub-implementing partners, as well as any other KIIs that may be conducted in English. SI expects to conduct FGDs with beneficiaries such as farmers but understands that there is a high possibility that focus groups may not be possible due to COVID-19. The ET will consider small group interviews, with appropriate social distancing, to supplement data typically collected during FGDs. As mentioned above, the ET will pilot and pre-test KII guides among a small set of beneficiaries and adapt/modify instruments as necessary. SAMPLING SI will use a purposive sampling approach34 to identify respondents, working through recommendations from Cardno and USAID as well as snowball sampling as necessary. The ET will strive to balance the 32 Respondent groups to be finalized during inception report stage with Avansa and ET. 33 Pending Covid-19 regulations. All face-to-face interviews will employ appropriate social distancing techniques. 34 Purposive sampling is a non-probability sampling technique that relies on judgment of the researcher to select units of analysis. It is widely used in qualitative research for the identification and selection of information-rich cases. This involves identifying and selecting individuals or groups of individuals that are especially knowledgeable about or experienced with a phenomenon of interest. USAID.GOV AVANSA ENDLINE REPORT | 66 sample of beneficiaries on factors such as gender, municipality and age. For stakeholders such as USAID, Avansa, and other partners, the ET will strive to speak to all relevant involved respondents. ANALYSIS ET members will take comprehensive notes during KIIs, share their reflections with team members in nightly debriefs, and update the FCR matrix throughout fieldwork. SI HQ team members will hold at minimum weekly check-ins with the team in Timor-Leste to discuss data collection progress and initial findings and reflections. Team members will review and summarize interview notes at the end of each day and submit electronic copies of notes, both in raw Tetum and translated to English, to the SI HQ team. The specific method for analysis will be determined by the Team Leader in collaboration with the ET, and may involve any combination of qualitative coding of responses using software such as Dedoose, Atlas.ti, Excel tally sheets, or other analysis methods. Data will be disaggregated by gender, municipality, and respondent type. DELIVERABLES The table below details the specific tasks and timeframes for all endline performance evaluation activities, as well as associated deliverables. TASK APPROXIMATE TIMEFRAME DELIVERABLE SOW Development. SI will work closely with USAID and Cardno to develop the scope of work and finalize the evaluation questions. SI: May 1 – May 6 USAID/Avansa: May 7-11 SI: May 12-13 -- Design Report. SI will further develop the proposed methodology including sampling strategy, fieldwork procedures and draft instruments, and summarize in a design report. SI: May 6 – May 18 USAID/Avansa: May 19 – May 25 SI: May 26 – May 29 Design Report Desk Review. The evaluation team will review relevant program documents including monitoring documents, data and reports, and begin to populate a Findings, Conclusions & Recommendations (FCR) matrix which will inform final edits to the Instruments, as well as the Draft Report. SI: May 11 – June 5 -- Instrument Design. Drawing on the evaluation questions and contents of the FCR from the Desk Review, the evaluation team will design instruments for qualitative data collection. The team will submit instruments to USAID and potentially Cardno for review and comment. SI: May 11 – June 12 Data Collection Instruments Primary Data Collection. The evaluation team will conduct remote primary data collection via phone. During fieldwork the team will continue to populate the FCR with added findings. June 22 – July 10 Weekly Fieldwork Updates 67 | AVANSA ENDLINE REPORT USAID.GOV TASK APPROXIMATE TIMEFRAME DELIVERABLE Draft Report. Drawing on the FCR, the evaluation team will draft a report answering the evaluation questions. Draft report will be submitted in MS Word. June 22 – August 31 Draft Report Final Report. The Team will address all comments provided by USAID in the Final Report. Final report will be submitted in MS Word and PDF formats. Aug. 31 – Sept. 30 Final Report USAID.GOV AVANSA ENDLINE REPORT | 68 ANNEX II: EVALUATION DESIGN AND METHODS EVALUATION QUESTION DATA SOURCE DATA COLLECTION METHOD DATA ANALYSIS METHOD Objective 1: Change in status of beneficiaries 1. Household: To what extent have project activities contributed to improved outcomes of target beneficiary households? 1a. Which interventions were the most effective in doing so and why? • Review and analysis of Avansa Agrikultura Project progress reports, annual household surveys, baseline survey, and sector documents. • KIIs with: i.) AAP Project Staff; Sub￾Implementing Partners; USAID; ii.) Government of Timor-Leste (GoTL) (municipal level – heads of municipalities); iii. farmer group heads (particularly female and youth led farmer groups). • FGDs with: i.) farmer beneficiaries; ii.) heads of households (particularly female headed households); iii.) traders, collectors and aggregators (disaggregated by gender / youth). • Desk review • KIIs • GIs • FGDs • Secondary household survey • Outcome analysis of household data by intervention area. • Qualitative coding to track KII and FGD data and do thematic analyses. • Disaggregated analysis of gender and youth by intervention area. • Basic quantitative tabulations and summaries on the household survey 2. Market: To what extent has the project strengthened private sector market linkages across the value chain for target crops? 2a. Which interventions were the most effective in doing so and why? • Desk review of AAP Timor-Leste Agriculture Market Systems Analysis Consultancy Report • KIIs with: i.) AAP Project Staff; Sub￾Implementing Partners; USAID; ii.) GoTL (national – MAFF and MCI, and municipal levels (heads of municipalities)); iii.) traders, collectors, and aggregators; iv.) input suppliers; v.) retailers, including supermarkets; vi.) banks and micro-finance institutions; vii.) Agro-input Business Association); viii.) farmer group heads; ix.) technical and vocational training institute staff and/or students. • FGDs with i.) farmer beneficiaries (particularly women and youth); ii heads of households (particularly female headed households; ); iii.) traders, collectors and aggregators (disaggregated by gender / youth). • Desk review • KIIs • GIs • FGDs • Secondary firm-level survey • Comparative thematic review of findings of Market Systems Analysis Report and KII and FGD assessment data. • Qualitative coding to track KII and FGD data and do thematic analyses. • Basic quantitative tabulations and summaries on the firm-level survey 3. Individual: To what extent have project activities contributed to changes in reported self-efficacy among beneficiaries, particularly females and youth? 3a. Which interventions were the most effective in doing so and why? • KIIs with AAP Project Staff; Sub￾Implementing Partners; USAID; • FGDs with i.) farmer beneficiaries (particularly women and youth) ; ii heads of households (particularly female headed households; ); iii.) traders, collectors and aggregators (disaggregated by gender / youth). • KIIs • GIs • FGDs • Secondary household survey • Qualitative coding to track KII & FGD data and do thematic analyses. • Basic quantitative tabulations and summaries on the household survey Objective 2: Lessons Learned 4. Integrated Approach: What interventions, or • KIIs with KIIs with: i.) AAP Project Staff; Sub-Implementing Partners; USAID; ii.) GoTL (national – MAFF and MCI, and • KIIs • GIs • FGDs • Qualitative coding to track KII data and do thematic analyses 69 | AVANSA ENDLINE REPORT USAID.GOV EVALUATION QUESTION DATA SOURCE DATA COLLECTION METHOD DATA ANALYSIS METHOD combination of interventions, did participants view as most relevant or valuable? Why? municipal levels (heads of municipalities)); iii.) traders, collectors, and aggregators; iv.) input suppliers; v.) retailers, including supermarkets; vi.) banks and micro-finance institutions; vii.) Agro-input Business Association); viii.) farmer group heads; ix.) technical and vocational training institute staff and/or students. • FGDs with i.) farmer beneficiaries (particularly women and youth); ii heads of households (particularly female headed households; ); iii.) traders, collectors and aggregators (disaggregated by gender / youth). 5. Sustainability: What is the current outlook for sustainability of key Avansa Agrikultura Project interventions? • KIIs with KIIs with: i.) AAP Project Staff; Sub-Implementing Partners; USAID; ii.) GoTL (national – MAFF and MCI, and municipal levels (heads of municipalities)); iii. farmer group heads (particularly female and youth led farmer groups). • FGDs with FGDs with i.) farmer beneficiaries (particularly women and youth) ; ii heads of households (particularly female headed households; ); iii.) traders, collectors and aggregators (disaggregated by gender / youth). • KIIs • GIs • FGDs • Qualitative coding to track KII & FGD data and do thematic analyses USAID.GOV AVANSA ENDLINE REPORT | 70 ANNEX III: INFORMATION SOURCES DOCUMENTS REVIEWED The ET reviewed the following documents as part of the initial desk review. Findings from this review contributed to and informed the evaluation design, approach, and methodology. • Developing Agriculture Communities (DAC) Project Final Performance Evaluation, 2015. • Avansa Agrikultura Project Baseline Report, 2015 • Avansa Agrikultura Project Midterm Performance Evaluation, 2017 • Avansa Agrikultura Project Year 5 (2019) summary highlights: o Avansa Insights HH Survey o Technology Adoption, Climate Risk Reduction o Technology Adoption, General o Production Insights • Avansa Agrikultura Project Agribusiness Insights Survey Report, 2018 • Avansa Agrikultura Project Year 5 Annual Progress Report • Avansa Agrikultura Project Firm-Level Survey Report, 2019 • Rapid Food Security Assessment, led by MAFF, Mercy Corps, and Oxfam, June 2020 SITES VISITED The Evaluation Team visited five municipalities in Timor-Leste for data collection: Dili, Alieu, Ermera, Ainaro, and Bobonaro. KEY INFORMANT GROUPS The Evaluation Team interviewed 13 different types of respondents. For data confidentiality reasons, the ET cannot name the individuals interviewed, but instead a list of types of respondents is detailed below. In total, 135 interviews were conducted across stakeholders, project staff, and USAID staff. For more detail on respondent breakdown, see Table 2. − Avansa Agrikultura staff − Implementing Partner staff − USAID staff − National government 71 | AVANSA ENDLINE REPORT USAID.GOV − Municipal-level government − Input suppliers − Supermarkets − Microfinance Institutions and Banks − Agro-Input Business Association − Vocational training schools − Farmer groups − Collectors − Agribusiness processors USAID.GOV AVANSA ENDLINE REPORT | 72 ANNEX IV: DATA COLLECTION INSTRUMENTS RESPONDENTS BY DATA COLLECTION TOOL KII Tools 1. Project staff: leadership, output leaders, district managers, municipal coordinators, training providers, and sub-implementing partners – Resonance, HIAM Health, Norman Borlaug Institute for International Agriculture/Texas 2. USAID Timor-Leste staff 3. GoTL: i.) National and ii.) Sub-National (Municipal) levels 4. Farmers’ group heads, and particularly female Farmer’s group heads where available 5. Input suppliers 6. Retailers, including supermarkets 7. Banks and micro-finance institutions 8. Agro-input Business Association Group Interview Tools 1. Technical and vocational training institute staff and/or students FGD Tools 1. Farmers’ group heads, and particularly female Farmer’s group heads where available 2. Farmer beneficiaries 3. Traders, collectors, and aggregators 73 | AVANSA ENDLINE REPORT USAID.GOV INFORMED CONSENT STATEMENT- KII Introductions: Thank you for joining us today. First, let me introduce the team. [INTRODUCTIONS] Purpose of the Interview: We work for Social Impact, a US-based research firm, and are working with USAID to evaluate the Avansa Agrikultura Project (referred to as ‘Avansa’). We are here now to discuss with you the effects the Avansa project has had on various beneficiary groups, including [BENEFICIARY GROUP]. Our objectives are to understand the effects of the Avansa project on beneficiaries, and gather lessons learned that can be used in designing similar projects in the future. You are one of about 50 people who have been selected to participate in these one-on-one or group interviews because of your knowledge and experience with the Avansa project. Though these interviews we are talking to Avansa project staff, implementing partners, USAID staff, Government of Timor-Leste, farmers’ groups, input suppliers, retailers, banks, and agro-input business associations. We will also be conducting Focus Group Discussions with project beneficiaries. In our interview today, we will discuss the nature of your interaction with Avansa, any changes you’ve seen as a result of the project (both good and bad), which combinations of interventions were the most effective, and your ideas about how the Avansa project may be sustained in the future. Time: This interview will take between 1-1.5 hours. Voluntary Participation: Your participation in this discussion is voluntary. If there are any questions you don’t feel comfortable answering, you do not have to do so, and you are welcome to excuse yourself at any time during our conversation without any consequences. Risks and Benefits: We do not foresee any significant risks or direct benefits from your participation in this study. Neither this research nor your answers to our questions will in any way determine if a project will be implemented here in the future or affect your ability to access or receive any services, now or in the future. While any close contact with others poses a risk of COVID-19 infection, we have taken the following precautionary steps to reduce the risk for COVID-19 transmission to a minimal level: we will wear face masks throughout the interview, remain 6 feet (about 1.5 meters) apart, provide face masks for all participants if needed, we have hand sanitizer available, and are conducting the interviews in an outdoor space. As there have been no new cases of COVID-19 in Timor-Leste since late April, we anticipate the risk of COVID-19 in our research to be minimal. Confidentiality: Everything you tell us will be kept confidential by the evaluation team to the fullest extent permitted by the law in Timor-Leste, the United States, and the policies of USAID. Only the SI evaluation team will have access to your name in combination with the information you share. We will combine the information you share with others and deliver it in a report to USAID. This report will also be made publicly available, but nothing you share with us will have your name attributed to it in our report. Additionally, the information you share with us will not be shared to your peers and colleagues, and will not be used for any additional research in the future. For Group Interviews: Due to the private nature of this research, we ask that all participants agree not to share anything that is discussed with anyone outside of this group once this conversation ends. Nonetheless, there is a risk that other discussion participants will repeat what is shared here today. Remember that you are free to refuse to answer any question. USAID.GOV AVANSA ENDLINE REPORT | 74 Recording: With your permission, we would like to audio record our conversation and take notes. The recordings and the notes will not be shared with anyone outside the evaluation team – we are conducting many interviews, so this is mainly to help us make sure we do not misunderstand anything you tell us or misrepresent anything in our notes. If you choose to opt out of an audio recorded interview, we will proceed with the interview and will not turn on the audio recording device, but we will still take notes. Informed Consent Question: • Do you have any questions for us before we begin? [Yes / No] • Are you willing to participate in this interview? [Yes / No] • Is it okay if we record our conversation? [Okay to record / Not okay to record] If you have any questions or concerns, you may contact the Social Impact Team Leader, Mr. Andrew Carmona, at acarmona@socialimpact.com, or the Social Impact Institutional Review Board at irb@socialimpact.com. Locally, you may also contact our Senior Qualitative Researcher, Gabriela Leite-Soares, at +670 7749 1780. Additionally, if you are interested in receiving a copy of the final report, you may email the Social Impact Project Manager, Alison Smith, at asmith@socialimpact.com any time after September 1, 2020. You will also be able to find this report online by looking on the USAID Development Experience Clearinghouse (DEC) website after September 30, 2020. Would you like a moment to write down this contact information so you have if for reference should you want to follow up after this interview? Alternatively, if you do not have a way to record this contact information, one of our team members can text it to you after the interview. Would you like to provide us with your phone number so we can send you this contact information? 75 | AVANSA ENDLINE REPORT USAID.GOV KEY INFORMANT INTERVIEW SEMI-STRUCTURED QUESTIONNAIRES PROJECT STAFF (INCLUDING PROJECT LEADERSHIP, OUTPUT LEADERS, DISTRICT MANAGERS, MUNICIPAL COORDINATORS, TRAINING PROVIDERS, AND SUB-IMPLEMENTING PARTNERS) Questionnaire for Project Staff Introductory Questions What is your role in the Avansa Agrikultura project, and when did you start working with the project? Please describe your main responsibilities/tasks within the project. Do you participate in any coordination activities with USAID or the GoTL in relation to the project. How frequently do you interact with project beneficiaries?  Daily  Weekly  Monthly  Quarterly  Infrequently – Ad hoc Questions related to Objective 1: Change in status of beneficiaries EQ 1. To what extent have project activities contributed to improved outcomes of target beneficiary households? a. Which interventions were the most effective in doing so and why? Which project interventions achieved their expected results (target indicators)? Of these interventions, which have been the most successful and what were the supporting factors for their success? Were certain interventions more successful for different groups and what types of interventions were more successful women, for youth, for adults? On a scale of 1 – 5, please indicate the level of technical assistance support required to make this intervention successful.  1 – low  2 – somewhat low  3 – medium  4 – somewhat high  5 -- high What type of improved outcomes were realized by the beneficiaries for these successful interventions? (Probe: food security outcomes, financial/economic outcomes) How do you know that beneficiaries realized these improved outcomes? What kind of evidence do you have that tells you so? Which project interventions did not achieve their expected results (target indicators)? USAID.GOV AVANSA ENDLINE REPORT | 76 Of these interventions, which were the least successful and what were the constraining factors for their limited achievement? On a scale of 1 – 5, please indicate the level of technical assistance support provided to these interventions.  1 – low  2 – somewhat low  3 – medium  4 – somewhat high  5 -- high Do you have any additional observations on the effectiveness of the Avansa interventions in increasing household-level outcomes? EQ 2. To what extent has the project strengthened private sector market linkages across the value chain for target crops? a. Which interventions were the most effective in doing so and why? How would you rate the project contribution to strengthening private sector market linkages?  Negative strengthening or weakening of private sector market linkages  No strengthening of private sector market linkages  Limited strengthening of private sector market linkages  Moderate strengthening of private sector market linkages  High strengthening of private sector market linkages Can you tell me what led you to choose that answer? Which project interventions were the most successful in strengthening private sector market linkages overall? For women? For youth? How do you define a “successful” intervention in this sense? (Probe: number of people reached, largest impact, etc.) How do you know that the interventions were successful? What kinds of evidence is available that tells you so? What were the supporting factors for these interventions strengthening private sector market linkages? On a scale of 1 – 5, please indicate the level of technical assistance support provided to these interventions.  1 – low  2 – somewhat low  3 – medium  4 – somewhat high  5 -- high Which project interventions were the least successful in strengthening private sector market linkages? (Probe: at the producer, collector/aggregator, and retail levels) What were the constraining factors for these interventions having a limited role in strengthening private sector market linkages? 77 | AVANSA ENDLINE REPORT USAID.GOV Do you have any additional observations on the effectiveness of the Avansa interventions in strengthening private sector market linkages? EQ 3. To what extent have project activities contributed to changes in reported self￾efficacy among beneficiaries, particularly females and youth? a. Which interventions were the most effective in doing so and why? To what extent have project activities affected outcomes among traditionally disadvantaged populations such as women and youth? What evidence is there to support this? Can you provide some examples? Do you view any of the interventions as contributing to self-efficacy among these disadvantaged groups, or others? Which interventions were most effective in improving self-efficacy of these disadvantaged groups? Do you have any additional observations on how the project has been able to – or can better -- contribute to improvements in self-efficacy of the beneficiaries? Questions related to Objective 2: Lessons Learned EQ 4. Integrated Approach: What interventions, or combination of interventions, did participants view as most relevant or valuable? Why? Thinking of the successful project interventions you described above, are there any certain combinations of interventions that provided the most value to beneficiaries? Are there any interventions you described as successful that rely on another Avansa interventions to succeed? Which ones? Why? Could other interventions be added to these combinations to enhance their success? What type of interventions? Were there missing elements or other interventions that could have been added to these combinations to make them successful? Do you have any additional observations on how the project could have better combined interventions to serve the needs of beneficiaries? EQ 5. Sustainability: What is the current outlook for sustainability of key Avansa Agrikultura Project interventions? Which of Avansa’s interventions would you say are the most likely to continue? What about in the absence of the Avansa project? Does this ability to sustain project actions/results differ between women/men/youth or location? In what ways? What specific things are needed among each of these groups to ensure sustainability? Which of the project’s interventions would you say are the least sustainable or not likely to be continued? USAID.GOV AVANSA ENDLINE REPORT | 78 Why do you say that? What evidence do you have to demonstrate this? To what extent are the project’s successful interventions scalable and / or able to be replicated in other areas of Timor-Leste? What are some factors you might consider in whether to scale any of these interventions? Do you have any additional observations on how the project could further promote sustainability among its interventions? Thank you for your participation in this interview. Your responses will be kept confidential and will be used solely for the purposes of this evaluation. 79 | AVANSA ENDLINE REPORT USAID.GOV USAID TIMOR-LESTE STAFF Questionnaire for USAID Timor-Leste Staff Introductory Questions What is your role within USAID Timor-Leste and what is your relation to the Avansa Agrikultura Project? When did you start working with the project? Please describe your main responsibilities/tasks in regard to the project, and if you participate in any coordination activities with project leadership and GoTL in relation to Avansa. How frequently do you interact with the project?  Daily  Weekly  Monthly  Quarterly  Infrequently – Ad hoc Questions related to Objective 1: Change in status of beneficiaries EQ 1. To what extent have project activities contributed to improved outcomes of target beneficiary households? a. Which interventions were the most effective in doing so and why? In your opinion, do you feel that the project has contributed to improved outcomes of target beneficiary households? Has there been any difference in outcomes for women? For youth? Or in different locations? What evidence exists to support this? In your opinion, what have been the most successful of the project interventions? Success in terms of what? What have been the most successful of the project interventions for women? For youth? In different locations? Do you feel that some of the project interventions were not successful? Which interventions and why? Do you have any additional observations on the effectiveness of the overall Avansa project and its specific interventions on household-level outcomes? EQ 2. To what extent has the project strengthened private sector market linkages across the value chain for target crops? a. Which interventions were the most effective in doing so and why? In your opinion, how do you feel that the project has contributed to the objective of strengthened private sector market linkages across the value chain for horticultural crops? (Probe: at the producer, collector/aggregator, and retail levels) Can you tell me what led you to that answer? USAID.GOV AVANSA ENDLINE REPORT | 80 Which project interventions do you feel were the most successful in strengthening private sector market linkages? How do you define a “successful” intervention in this context? (Probe: number of people reached, size of impact, etc.) Which project interventions do you feel were the least successful in strengthening private sector market linkages? (Probe: at the producer, collector/aggregator, and retail levels) Do you have any additional observations on the effectiveness of the Avansa interventions in strengthening private sector market linkages? EQ 3. To what extent have project activities contributed to changes in reported self￾efficacy among beneficiaries, particularly females and youth? a. Which interventions were the most effective in doing so and why? What is the impact, if any, of Avansa interventions on outcomes of traditionally disadvantaged groups such as women and youth? (Probe: outcomes of self-efficacy) Why do you say this? What evidence is there to support it? Which interventions do you feel were most effective in improving outcomes of these disadvantaged groups? Do you have any additional observations on how the project has been able to – or might better -- contribute to improvements in key outcomes, such as self-efficacy, of disadvantaged groups, such as women, youth, disabled or other disadvantaged individuals? Questions related to Objective 2: Lessons Learned EQ 4. Integrated Approach: What interventions, or combination of interventions, did participants view as most relevant or valuable? Why? Avansa employed an integrated model for its interventions. Do you think this was important for meeting the needs of beneficiaries of the project? Why or why not? Are there any certain combinations of interventions that provided the most value to beneficiaries? Which ones? Why do you say that? What evidence can you point to? Do you have any additional observations on how the project could have better combined interventions as part of an integrated model? EQ 5. Sustainability: What is the current outlook for sustainability of key Avansa Agrikultura Project interventions? In your opinion, will GoTL, project stakeholders, and project beneficiaries be able to sustain the actions of the project following the conclusion of external support by USAID? 81 | AVANSA ENDLINE REPORT USAID.GOV Which of the project’s interventions do you believe are the most sustainable and likely to be continued? Why those? Which of the project’s interventions do you feel are least sustainable and not likely to be continued? Why those? In your opinion, are the project’s successful interventions scalable and / or able to be replicated in other areas of Timor-Leste? In your opinion, what would need to happen/what resources would be needed to sustain the progress of Avansa? Are there different types of support that are required for different groups? For women? For youth? In different locations? To your knowledge, is USAID Timor-Leste planning any similar or complementary action to the Avansa Agrikultura Project with the next Country Development Cooperation Strategy (CDCS)? Do you have any additional observations on how the project’s results can be further sustained? Thank you for your participation in this interview. Your responses will be kept confidential and will be used solely for the purposes of this evaluation. USAID.GOV AVANSA ENDLINE REPORT | 82 GOTL NATIONAL LEVEL Questionnaire for GoTL National Staff Introductory Questions Which GoTL ministry do you represent and in what capacity? Does your ministry have any policy or strategy related to the outcomes of the Avansa Agrikultura project? (Market linkages improved and expanded across the horticultural value chain; Sustainable horticulture productivity improved at the farm level.) Please describe your relationship with the Avansa Agrikultura Project and how long you have had an association with the project? Please describe any coordination activities you participate in with the Avansa project leadership, with USAID Timor-Leste and other GoTL ministries involved in the project. How frequently do you interact with the project?  Daily  Weekly  Monthly  Quarterly  Infrequently – Ad hoc Questions related to Objective 1: Change in status of beneficiaries EQ 1. To what extent have project activities contributed to improved outcomes of target beneficiary households? a. Which interventions were the most effective in doing so and why? In your opinion, do you feel that the project has contributed to improved outcomes of target beneficiary households? How has the project done this? How do you know that outcomes have been improved? What evidence do you have that demonstrates this? In your opinion, what have been the most successful of the project interventions? What factors helped facilitate the success? How do you define a “successful” intervention? (Probe: number of people reached, magnitude of impact) Do you feel that some of the project interventions were not successful? Which interventions and why? Do you have any additional observations on the effectiveness of the overall Avansa project and its specific interventions at the household-level? EQ 2. To what extent has the project strengthened private sector market linkages across the value chain for target crops? a. Which interventions were the most effective in doing so and why? 83 | AVANSA ENDLINE REPORT USAID.GOV In your opinion, do you feel that the project has led to strengthened private sector market linkages across the value chain for horticultural crops? (Probe: at the producer, collector/aggregator, and retail levels) Which project interventions do you feel were the most successful in strengthening private sector market linkages? Why would you categorize them as “successful?” Which project interventions do you feel were the least successful in strengthening private sector market linkages? Why would you categorize them as “least successful?” Do you have any additional observations on the effectiveness of the Avansa interventions in strengthening private sector market linkages? EQ 3. To what extent have project activities contributed to changes in reported self￾efficacy among beneficiaries, particularly females and youth? a. Which interventions were the most effective in doing so and why? Which groups do you consider the most marginalized/vulnerable in the project locations and why? (Probes: Women, youth, poor, etc.) What is your understanding of the project’s impact on women and youth? Why or why hasn’t it made an impact and in what ways? (Probe: skills gained, improved self-efficacy) Questions related to Objective 2: Lessons Learned EQ 4. Integrated Approach: What interventions, or combination of interventions, did participants view as most relevant or valuable? Why? Avansa employed an integrated model for its interventions. How do you think the integrated model [specify interventions] supported beneficiaries compared to using just one of these interventions? What are the specific needs that were met by the integrated model? Which populations benefitted from this the most? The least? In your opinion, which combination of interventions were particularly successful? Why do you say that? What evidence can you point to? Do you have any additional observations on how the project could have better combined interventions? EQ 5. Sustainability: What is the current outlook for sustainability of key Avansa Agrikultura Project interventions? In your opinion, will GoTL, project stakeholders, and project beneficiaries be able to sustain the actions of the project following the conclusion of external support by USAID? Which of the project’s interventions do you feel are most sustainable and likely to be continued? Why those? USAID.GOV AVANSA ENDLINE REPORT | 84 Which of the project’s interventions do you feel are least sustainable and not likely to be continued? Why those? In your opinion, are the project’s successful interventions scalable and / or able to be replicated in other areas of Timor-Leste? In your opinion, what would need to happen/what resources would be needed to sustain the progress of Avansa? To your knowledge, is the GoTL planning any similar or complementary action to the Avansa Agrikultura Project in the next planning and budgeting cycle? Thank you for your participation in this interview. Your responses will be kept confidential and will be used solely for the purposes of this evaluation. 85 | AVANSA ENDLINE REPORT USAID.GOV GOTL SUB-NATIONAL LEVEL Questionnaire for GoTL Sub-National Staff Introductory Questions Which municipality or other government sub-division do you represent and in what capacity? Please describe your relationship with the Avansa Agrikultura Project and how long you have been working with the project? Please describe any coordination activities you participate in with the Avansa project leadership, with USAID Timor-Leste and national GoTL ministries involved in the project. How frequently do you interact with the project?  Daily  Weekly  Monthly  Quarterly  Infrequently – Ad hoc Questions related to Objective 1: Change in status of beneficiaries EQ 1. To what extent have project activities contributed to improved outcomes of target beneficiary households? a. Which interventions were the most effective in doing so and why? In your opinion, do you feel that the project has contributed to improved outcomes of target beneficiary households? How has the project done this? How do you know that outcomes have been improved? What evidence do you have that demonstrates this? In your opinion, what have been the most successful of the project interventions? What factors helped facilitate the success? How do you define a “successful” intervention? (Probe: number of people reached, magnitude of impact) Do you feel that some of the project interventions were not successful? Which interventions and why? Do you have any additional observations on the effectiveness of the overall Avansa project and its specific interventions at the household-level? EQ 2. To what extent has the project strengthened private sector market linkages across the value chain for target crops? a. Which interventions were the most effective in doing so and why? In your opinion, do you feel that the project has led to strengthened private sector market linkages across the value chain for horticultural crops? (Probe: at the producer, collector/aggregator, and retail levels) USAID.GOV AVANSA ENDLINE REPORT | 86 Which project interventions do you feel were the most successful in strengthening private sector market linkages? Why would you categorize them as “successful?” What factors enabled them to be successful? Which project interventions do you feel were the least successful in strengthening private sector market linkages? Why would you categorize them as “least successful?” What factors acted as barriers to success for these interventions? Do you have any additional observations on the effectiveness of the Avansa interventions in strengthening private sector market linkages? EQ 3. To what extent have project activities contributed to changes in reported self￾efficacy among beneficiaries, particularly females and youth? a. Which interventions were the most effective in doing so and why? Have you heard anything about effects of Avansa programming on disadvantaged populations such as women and youth? If so, what have you heard? What is your understanding of the project’s impact on women and youth? Why or why hasn’t it made an impact and in what ways? (Probe: skills gained, improved self-efficacy) Questions related to Objective 2: Lessons Learned EQ 4. Integrated Approach: What interventions, or combination of interventions, did participants view as most relevant or valuable? Why? Avansa employed an integrated model for its interventions. How do you think the integrated model [specify interventions] supported beneficiaries compared to using just one of these interventions? What are the specific needs that were met by the integrated model? Which populations benefitted from this the most? The least? In your opinion, which combination of interventions were particularly successful? Why do you say that? What evidence can you point to? Do you have any additional observations on how the project could have better combined interventions? EQ 5. Sustainability: What is the current outlook for sustainability of key Avansa Agrikultura Project interventions? In your opinion, will GoTL, project stakeholders, and project beneficiaries be able to sustain the actions of the project following the conclusion of external support by USAID? Which of the project’s interventions do you feel are most sustainable and likely to be continued? Why those? 87 | AVANSA ENDLINE REPORT USAID.GOV Which of the project’s interventions do you feel are least sustainable and not likely to be continued? Why those? In your opinion, are the project’s successful interventions scalable and / or able to be replicated in other areas of Timor-Leste? In your opinion, what would need to happen/what resources would be needed to sustain the progress of Avansa? Do you have any additional observations on how the project’s results can be further sustained? Thank you for your participation in this interview. Your responses will be kept confidential and will be used solely for the purposes of this evaluation. USAID.GOV AVANSA ENDLINE REPORT | 88 FARMERS’ GROUP HEADS Questionnaire for Farmers’ Group Heads Introductory Questions Does your group have a name? Where is your group located? What type of activities is the group involved in? How many members are there in your group __? How many female __, youth __, members with disabilities __? How long have you been the leader of this group? Please describe how project activities are coordinated between Avansa, yourself and the group. How frequently do you interact with Avansa staff?  Daily  Weekly  Monthly  Quarterly  Infrequently – Ad hoc How frequently do you interact with the group members?  Daily  Weekly  Monthly  Quarterly  Infrequently – Ad hoc Questions related to Objective 1: Change in status of beneficiaries EQ 1. To what extent have project activities contributed to improved outcomes of target beneficiary households? a. Which interventions were the most effective in doing so and why? How did the group decide /select which Avansa-supported activities it wanted be involved in? To what extent were all types of members in the group (e.g. women, men, youth) aligned in their choice of activities? How were women, men, youth involved in this decision-making process? In what ways did participation in Avansa activities, or receiving support from Avansa, affect your farmer group or members of your farmer group?  Affected it very negatively  Affected it negatively  No effect either way  Affected it positively  Affected it very positively Please specify why you chose this answer. How did participation in Avansa related activities affect your farmer group or members of your farmer group? Of the Avansa-related activities or support to your group, which have been the most beneficial and why have they been beneficial? What activities have been most beneficial to the women, youth, and other disadvantaged members of your group? 89 | AVANSA ENDLINE REPORT USAID.GOV On a scale of 1 – 5, please indicate the level of technical assistance support the group has received to make these activities successful.  1 – low  2 – somewhat low  3 – medium  4 – somewhat high  5 -- high What type of benefits were realized by the group members by participating in Avansa-related activities or receiving Avansa support? (Probe: income, improved diets and nutrition, etc.) Were there any project activities that were not beneficial? What were they? Why were they not successful for your group? How, if at all, has COVID-19 affected your group’s production activities? Has your group developed a strategy to be able to continue their activities despite the COVID-19 situation? Please explain. Do you have any additional observations on the effectiveness of the Avansa interventions on outcomes of households that are in your farmer group? EQ 2. To what extent has the project strengthened private sector market linkages across the value chain for target crops? a. Which interventions were the most effective in doing so and why? To what extent has the project assisted your group to strengthen private sector market linkages for the crops you produce? (Probe: connections to input suppliers, markets, traders/collectors, etc.)  Negative strengthening or weakening of private sector market linkages  No strengthening of private sector market linkages  Limited strengthening of private sector market linkages  Moderate strengthening of private sector market linkages  High strengthening of private sector market linkages In what ways, if at all, did Avansa support strengthen private sector market linkages for your farmer group? What factors facilitated or hindered the strengthening of private sector market linkages for your farmer group? Which project activities were the least successful in strengthening private sector market linkages? What were the constraining factors for these activities having limited private sector market linkages? How, if at all, has COVID-19 affected your group’s private sector market linkages? Has your group developed a strategy to be able to continue to maintain their private sector market linkages despite the COVID-19 situation? Please explain. Do you have any additional observations on the effectiveness of the Avansa project in strengthening private sector market linkages? EQ 3. To what extent have project activities contributed to changes in reported self￾efficacy among beneficiaries, particularly females and youth? a. Which interventions were the most effective in doing so and why? Thinking about women and youth in your farmer group, how, if at all, have Avansa-related support or activities affected these groups? Compared to men and non-youth? USAID.GOV AVANSA ENDLINE REPORT | 90 Have you observed changes in self-efficacy among your group members? If yes, can you provide some examples? Have women and youth improved their self-efficacy from their involvement in the project? If yes, can you provide some examples? Which interventions were most effective in improving self-efficacy: a.) for all beneficiaries? b.) for women? c.) for youth? d.) for any other disadvantaged groups (such as people with disabilities)? Do you have any additional observations on how the project has been able to contribute to improvements in self-efficacy of the beneficiaries? Questions related to Objective 2: Lessons Learned EQ 4. Integrated Approach: What interventions, or combination of interventions, did participants view as most relevant or valuable? Why? Did you receive any training from Avansa on group leadership, group dynamics or gender? If yes, what types of training? Did you have any interaction with other group leaders? If yes, what types of interaction? To what extent did Avansa promote a combination of activities among your farmer group, as opposed to just one single activity? If there were a combination of activities, which combination of activities were particularly successful, and what were the supporting factors? Why do you consider these a success? Could other activities be added to these combinations of activities to enhance their success? What type of activities? Which combination of activities were least successful, and what were the constraining factors? Do you have any additional observations on how the project could have better combined activities? EQ 5. Sustainability: What is the current outlook for sustainability of key Avansa Agrikultura Project interventions? Will your group be able to sustain the actions, or continue the benefits, of the project following the conclusion of external support from Avansa? Which of the group’s activities are the most sustainable and likely to be continued? Why those? Which of the group’s activities are the least sustainable and not likely to be continued? Why those? Potential question: Will your group be able to sustain the actions, or continue the benefits, of the project despite the COVID-19 situation? Do you have any additional observations on how the project could further promote sustainability? Thank you for your participation in this interview. Your responses will be kept confidential and will be used solely for the purposes of this evaluation. 91 | AVANSA ENDLINE REPORT USAID.GOV INPUT SUPPLIERS Questionnaire for Input suppliers Introductory Questions Which input supplier do you represent and in what capacity? Is your company a small, mid-size, or large scale provider of inputs? How many locations does your company serve? Please describe your relationship with the Avansa Agrikultura Project and how long you have been involved with the project? How frequently do you interact with the project?  Daily  Weekly  Monthly  Quarterly  Infrequently – Ad hoc Questions related to Objective 1: Change in status of beneficiaries EQ 1. To what extent have project activities contributed to improved outcomes of target beneficiary households? a. Which interventions were the most effective in doing so and why? In your opinion, do you feel that the project has contributed to improved outcomes of target beneficiary households? (Probe: improved food security, incomes) How has the project done this? In your opinion, what have been the most successful project interventions? Why do you say they’re “successful?” Which have been the least successful project interventions? Why? How has the work that you do contributed to outcomes (food security, incomes) at the household level? What role has Avansa support played in this in the past five years? EQ 2. To what extent has the project strengthened private sector market linkages across the value chain for target crops? a. Which interventions were the most effective in doing so and why? How has the support of Avansa affected you and your business, if at all? How has your business changed, if at all, since receiving this support? (Probe: decreased business, stayed the same, increase in business) Have you been able to supply inputs and supplies across all levels of the horticultural value chain? Why or why not? In your opinion, do you feel that the project has led to strengthened private sector market linkages across the value chain for horticultural crops? (Probe: at the producer, input, aggregator/collector, retail levels) Which project interventions do you feel were the most successful in strengthening private sector market linkages, particularly at the input level? What has made them successful? USAID.GOV AVANSA ENDLINE REPORT | 92 Which project interventions do you feel were the least successful in strengthening private sector market linkages, particularly at the input level? What has made them unsuccessful? How, if at all, has COVID-19 affected your activities with the Avansa production groups? Have you developed a strategy to be able to continue to provide inputs and services to the production groups? Please explain. Do you have any additional observations on the effectiveness of the Avansa interventions in strengthening private sector market linkages, particularly at the input level? At other levels (probe: producer, aggregator, retail)? EQ 3. To what extent have project activities contributed to changes in reported self￾efficacy among beneficiaries, particularly females and youth? a. Which interventions were the most effective in doing so and why? Not applicable to this group of respondents. Questions related to Objective 2: Lessons Learned EQ 4. Integrated Approach: What interventions, or combination of interventions, did participants view as most relevant or valuable? Why? What Avansa interventions or combination of Avansa interventions do you feel were the most relevant for meeting the needs of you and your business? What interventions or combination of interventions do you feel were the most relevant for meeting the needs of the beneficiaries and the sector? In your opinion, which combination of interventions were particularly successful? What made them successful? Has the project utilized an integrated approach in applying inputs and supplies for all levels of the horticultural production value chain and strengthening market linkages? Why or why not? What were the challenges and successes of doing so? Do you have any additional observations on how the project could have better combined interventions? EQ 5. Sustainability: What is the current outlook for sustainability of key Avansa Agrikultura Project interventions? In your opinion, will GoTL, project stakeholders, and project beneficiaries be able to sustain the actions of the project following the conclusion of external support by USAID? Will you be able to sustain your level of business without Avansa support? Why or why not? Can the project beneficiaries purchase necessary inputs and supplies without project funding support? Which of the project’s interventions do you feel are most sustainable and likely to be continued? Which of the project’s interventions do you feel are least sustainable and not likely to be continued? What would you and your business need, if anything, to sustain the progress made with Avansa’s support? 93 | AVANSA ENDLINE REPORT USAID.GOV Are you able to sustain your level of business activities despite the COVID-19 situation? Do you have any additional observations on how the project’s results can be further sustained? Thank you for your participation in this interview. Your responses will be kept confidential and will be used solely for the purposes of this evaluation. USAID.GOV AVANSA ENDLINE REPORT | 94 RETAILERS, INCLUDING SUPERMARKETS Questionnaire for Retailers, including supermarkets Introductory Questions Which retailer or supermarket do you represent and in what capacity? Please describe your relationship with the Avansa Agrikultura Project and how long you have been involved with the project? Please describe the services you have received from the project. How frequently do you interact with the project?  Daily  Weekly  Monthly  Quarterly  Infrequently – Ad hoc Questions related to Objective 1: Change in status of beneficiaries EQ 1. To what extent have project activities contributed to improved outcomes of target beneficiary households? a. Which interventions were the most effective in doing so and why? Not applicable for this group of respondents. EQ 2. To what extent has the project strengthened private sector market linkages across the value chain for target crops? a. Which interventions were the most effective in doing so and why? How has the support of Avansa affected you and your business, if at all? How has your business changed, if at all, since receiving this support? (Probe: decreased business, stayed the same, increase in business) In your opinion, to what extent do you feel that the project has led to strengthened private sector market linkages across the value chain for horticultural crops? (Probe: at the producer, input, aggregator/collector, retail levels) Have the products (project interventions) you have received from the project been up to market standards of quality? Which products (project interventions) do you feel were the most successful in strengthening private sector market linkages? What has made them successful? Which products (project interventions) do you feel were the least successful in strengthening private sector market linkages? What made them unsuccessful? How, if at all, has COVID-19 affected private sector market linkages across the value chain for horticultural crops? Please explain. 95 | AVANSA ENDLINE REPORT USAID.GOV What additional products (project interventions) would you like to have seen from the project, particularly at the retail level? How, if at all, has COVID-19 affected your ability to market and sell horticultural crops? Have you developed a strategy to be able to continue to market and sell horticultural crops? Please explain. Do you have any additional observations on the effectiveness of the Avansa interventions in strengthening private sector market linkages, particularly at the retail level? EQ 3. To what extent have project activities contributed to changes in reported self￾efficacy among beneficiaries, particularly females and youth? a. Which interventions were the most effective in doing so and why? Not applicable to this group of respondents. Questions related to Objective 2: Lessons Learned EQ 4. Integrated Approach: What interventions, or combination of interventions, did participants view as most relevant or valuable? Why? Not applicable to this group of respondents. EQ 5. Sustainability: What is the current outlook for sustainability of key Avansa Agrikultura Project interventions? In your opinion, will GoTL, project stakeholders, and project beneficiaries be able to sustain the actions of the project following the conclusion of external support by USAID? Will the demand remain high for the type of horticultural products supported by the project? Which of the products (project interventions) do you feel are most sustainable and likely to be continued? Why those? Which of the products (project interventions) do you feel are least sustainable and not likely to be continued? Why not those interventions? What would you and your business need, if anything, to sustain the progress made with Avansa’s support? Do you have any additional observations on how the project’s results can be further sustained? Thank you for your participation in this interview. Your responses will be kept confidential and will be used solely for the purposes of this evaluation. USAID.GOV AVANSA ENDLINE REPORT | 96 BANKS AND MICRO-FINANCE INSTITUTIONS Questionnaire for Banks and Micro-Finance Institutions Introductory Questions Which bank or micro-finance institution do you represent and in what capacity? Please describe your relationship with the Avansa Agrikultura Project and how long you have been involved with the project? Please describe any banking or microfinance services you have provided for the project and its beneficiaries. Questions related to Objective 1: Change in status of beneficiaries EQ 1. To what extent have project activities contributed to improved outcomes of target beneficiary households? a. Which interventions were the most effective in doing so and why? In your opinion, do you feel that the project has contributed to improved outcomes of target beneficiary households? How has the project done this generally and, in particular, with respect to you and your work? In your opinion, what have been the most successful of the project interventions? How, if at all, has the provision of banking or micro-finance services affected farmer households? What kind of banking or micro-finance services were you able to provide to farmer households before and after Avansa support? Do you feel that some of the project interventions were not successful? Which interventions and why? Did these interventions lack banking or micro-finance services? Do you have any additional observations on the effectiveness of the overall Avansa project and its specific interventions on household-level outcomes (e.g. food security, income, and nutrition)? EQ 2. To what extent has the project strengthened private sector market linkages across the value chain for target crops? a. Which interventions were the most effective in doing so and why? In your opinion, do you feel that the project has led to strengthened private sector market linkages? How, if at all, has the role of banking and / or micro-finance in strengthening private sector market linkages changed since receiving Avansa-related support? Which project interventions do you feel were the most successful in strengthening private sector market linkages, with respect to the banking and microfinance sectors? Which project interventions do you feel were the least successful in strengthening private sector market linkages, with respect to the banking and microfinance sectors? How, if at all, has COVID-19 affected your relationship with clients seeking business / micro-finance services? Has the level of services: ___ stayed the same ___ decreased ___ increased? 97 | AVANSA ENDLINE REPORT USAID.GOV Do you have any additional observations on the effectiveness of the Avansa interventions in strengthening private sector market linkages? EQ 3. To what extent have project activities contributed to changes in reported self￾efficacy among beneficiaries, particularly females and youth? a. Which interventions were the most effective in doing so and why? Approximately what proportion of your services (loans, microfinance, etc.) are accessed by men versus women? By youth? How has this changed, if at all, since receiving Avansa support? (Probe: more women accessing, less women accessing, etc.) To what extent are women and youth able to access banking or micro-finance services? Why do you say that? How might we be able to increase access to finance for women and youth? Questions related to Objective 2: Lessons Learned EQ 4. Integrated Approach: What interventions, or combination of interventions, did participants view as most relevant or valuable? Why? Have banking and microfinance services been combined with other interventions along the agricultural value chain? Which ones? In your opinion, which combination of interventions were particularly successful? What factors made them successful? Do you have any additional observations on how the project could have better combined interventions, particularly with respect to access to finance? EQ 5. Sustainability: What is the current outlook for sustainability of key Avansa Agrikultura Project interventions? In your opinion, will GoTL, project stakeholders, and project beneficiaries be able to sustain the actions of the project following the conclusion of external support by USAID? Do you think that the project beneficiaries use banking and microfinance services to purchase necessary inputs and supplies without project support? Why do you say that? Which of the project’s interventions do you feel are most sustainable and likely to be continued? Which of the project’s interventions do you feel are least sustainable and not likely to be continued? Do you have any additional observations on how the project’s results can be further sustained? Thank you for your participation in this interview. Your responses will be kept confidential and will be used solely for the purposes of this evaluation. USAID.GOV AVANSA ENDLINE REPORT | 98 AGRO-INPUT BUSINESS ASSOCIATION Questionnaire for the Agro-Input Business Association Introductory Questions What is the role / relationship of the Agro-Input Business Association with the project? How long has the association been involved with the project? Please describe any agro-input business services you have provided for the project and its beneficiaries. Questions related to Objective 1: Change in status of beneficiaries EQ 1. To what extent have project activities contributed to improved outcomes of target beneficiary households? a. Which interventions were the most effective in doing so and why? In your opinion, do you feel that the project has contributed to improved outcomes of target beneficiary households? How has the project done this? In your opinion, what have been the most successful of the project interventions? How if at all, has the provision of the Agro-Input Business Association services contributed to improved outcomes of farmer households (Probe: food security, income, nutritional)? What kind of services were provided for these interventions? Do you feel that some of the project interventions were not successful? Which interventions and why? Did these interventions lack banking or micro-finance services? Do you have any additional observations on the effectiveness of the overall Avansa project and its specific interventions on household level outcomes? EQ 2. To what extent has the project strengthened private sector market linkages across the value chain for target crops? a. Which interventions were the most effective in doing so and why? In your opinion, do you feel that the project has led to strengthened private sector market linkages for horticulture? (Probe: at the input, producer, aggregator, retail levels) What has been the role of the Agro-Input Business Association in strengthening private sector market linkages? How, if at all, did the work of your association change from before to after Avansa support? In what ways did it change? Which project interventions do you feel were the most successful in strengthening private sector market linkages? Which project interventions do you feel were the least successful in strengthening private sector market linkages? 99 | AVANSA ENDLINE REPORT USAID.GOV Do you have any additional observations on the effectiveness of the Avansa interventions in strengthening private sector market linkages generally, and also in particular with respect to you and your association’s work? EQ 3. To what extent have project activities contributed to changes in reported self￾efficacy among beneficiaries, particularly females and youth? a. Which interventions were the most effective in doing so and why? Not applicable to this respondent. Questions related to Objective 2: Lessons Learned EQ 4. Integrated Approach: What interventions, or combination of interventions, did participants view as most relevant or valuable? Why? Have the services of the Agro-Input Business Association been combined with other Avansa-supported activities or services to strengthen the agricultural value chain? Which ones? In your opinion, which combination of interventions were particularly successful? What factors made them successful? Do you have any additional observations on how the project could have better combined interventions, particularly with respect to your association and its services? EQ 5. Sustainability: What is the current outlook for sustainability of key Avansa Agrikultura Project interventions? In your opinion, will GoTL, project stakeholders, and project beneficiaries be able to sustain the actions of the project following the conclusion of external support by USAID? Will the Agro-Input Business Association continue to be able to provide support to the beneficiaries following the end of project support? Why or why not? Which of the project’s interventions do you feel are most sustainable and likely to be continued? Which of the project’s interventions do you feel are least sustainable and not likely to be continued? Do you have any additional observations on how the project’s results can be further sustained? Thank you for your participation in this interview. Your responses will be kept confidential and will be used solely for the purposes of this evaluation. 101 | AVANSA ENDLINE REPORT USAID.GOV GROUP INTERVIEW SEMI-STRUCTURED QUESTIONNAIRES TECHNICAL AND VOCATIONAL TRAINING INSTITUTE STAFF AND/OR STUDENTS Questionnaire for Technical and vocational training institute staff and/or students Introductory Questions Which technical and vocational training institute do you represent and in what capacity? Please describe your relationship with the Avansa Agrikultura Project and how long you have been involved with the project? Please describe any technical and vocational training activities you provide as part of the project. How frequently do you interact with the project?  Daily  Weekly  Monthly  Quarterly  Infrequently – Ad hoc Questions related to Objective 1: Change in status of beneficiaries EQ 1. To what extent have project activities contributed to improved outcomes of target beneficiary households? a. Which interventions were the most effective in doing so and why? In your opinion, do you feel that the project has contributed to improved outcomes of target beneficiary households? How has the project done this? What evidence have you seen to back this up? In your opinion, what have been the most successful project interventions? What factors have facilitated the success of the interventions? Have technical / vocational training activities contributed to the success of these interventions? What kind of technical / vocational training activities? Are there separate technical and vocational training activities for women, youth and other disadvantaged groups? Do you feel that some of the project interventions were not successful? Which interventions and why? Were there any missed opportunities for providing technical / vocational training that could have improved the effectiveness of interventions? Do you have any additional observations on the effectiveness of the overall Avansa project and its specific interventions at the household-level? EQ 2. To what extent has the project strengthened private sector market linkages across the value chain for target crops? a. Which interventions were the most effective in doing so and why? USAID.GOV AVANSA ENDLINE REPORT | 102 In your opinion, do you feel that the project has led to strengthened private sector market linkages across the value chain for horticultural crops? Has your technical / vocational training institution provided any training related to value chain addition and strengthening private sector market linkages? (Probe: at the producer, collector/aggregator, and retail levels) Which project interventions do you feel were the most successful in strengthening private sector market linkages? How do you define “successful?” What factors made them successful? Which project interventions do you feel were the least successful in strengthening private sector market linkages? Why were they unsuccessful? How, if at all, has COVID-19 affected your relationship with Avansa production groups? Have you developed a strategy to be able to continue to interact with the Avansa production groups? Please explain. Do you have any additional observations on the effectiveness of the Avansa interventions in strengthening private sector market linkages? EQ 3. To what extent have project activities contributed to changes in reported self￾efficacy among beneficiaries, particularly females and youth? a. Which interventions were the most effective in doing so and why? To what extent do women and youth have access to training and vocational opportunities compared to other groups in Timor-Leste? How has this access changed, if at all, in the past five years? (Probe: Decreased access, stayed the same, increased access) What do you think is the cause of these changes? What proportion of students are women in your courses? What proportion are youth (10-29)? Has enrollment of women and youth changed since your institute began its work with Avansa? If so, has it decreased, stayed the same, or increased? How, if at all, does access to TVET impact beneficiaries, such as women and youth? (Probe: skills gained, self-efficacy) What evidence do you have/have you seen to support the impact you discussed? Questions related to Objective 2: Lessons Learned EQ 4. Integrated Approach: What interventions, or combination of interventions, did participants view as most relevant or valuable? Why? Avansa employed an integrated model for its interventions, which essentially means they used a combination of activities, working together, to achieve results. How, if at all, has the project utilized an integrated approach in combining technical and vocational training activities with other different types of interventions? 103 | AVANSA ENDLINE REPORT USAID.GOV What have been the successes and challenges of applying this approach? Do you have any additional observations on how the project could have better combined interventions to achieve improved outcomes among farmers, including women and youth, and the private sector in agriculture? EQ 5. Sustainability: What is the current outlook for sustainability of key Avansa Agrikultura Project interventions? In your opinion, will GoTL, project stakeholders, and project beneficiaries be able to sustain the actions of the project following the conclusion of external support by USAID? What additional technical and vocational training activities or general resources are needed to improve the project’s sustainability? Which of the project’s interventions do you feel are most sustainable and likely to be continued? Which of the project’s interventions do you feel are least sustainable and not likely to be continued? Do you have any additional observations on how the project’s results can be further sustained? Thank you for your participation in this interview. Your responses will be kept confidential and will be used solely for the purposes of this evaluation. USAID.GOV AVANSA ENDLINE REPORT | 104 INFORMED CONSENT STATEMENT- FGD Introductions: Thank you for joining us today. First, let me introduce the team. [INTRODUCTIONS] Purpose of the Interview: We work for Social Impact, a US-based research firm, and are working with USAID to evaluate the Avansa Agrikultura Project (referred to as ‘Avansa’). We are here now to discuss with you the effects the Avansa project has had on various beneficiary groups, including [BENEFICIARY GROUP]. Our objectives are to understand the effects of the Avansa project on beneficiaries, and gather lessons learned that can be used in designing similar projects in the future. You are one of about 100 people who have been selected to participate in these discussions because of your knowledge and experience with the Avansa project. Through these discussions we are talking to farmer group heads, farmers, traders, collectors, and aggregators. In our discussion today, we will discuss the nature of your interaction with Avansa, any changes you’ve seen as a result of the project (both good and bad), which combinations of interventions were the most effective, and your ideas about how the Avansa project may be sustained in the future. Time: (Focus) These discussions will last between 1.5-2 hours. Voluntary Participation: Your participation in this discussion is voluntary. If there are any questions you don’t feel comfortable answering, you do not have to do so, and you are welcome to excuse yourself at any time during our conversation without any consequences. Risks and Benefits: We do not foresee any significant risks or direct benefits from your participation in this study. Neither this research nor your answers to our questions will in any way determine if a project will be implemented here in the future or affect your ability to access or receive any services, now or in the future. While any close contact with others poses a risk of COVID-19 infection, we have taken the following precautionary steps to reduce the risk for COVID-19 transmission to a minimal level: we will wear face masks throughout the interview, remain 6 feet (about 1.5 meters) apart, provide face masks for all participants if needed, we have hand sanitizer available, and are conducting the interviews in an outdoor space. As there have been no new cases of COVID-19 in Timor-Leste since late April, we anticipate the risk of COVID-19 in our research to be minimal. Confidentiality: The Evaluation Team will not share any information about you, your participation in this study, or what you said, however the nature of focus group discussions prevents us from guaranteeing confidentiality. We would like to remind participants to respect the privacy of your fellow participants and not repeat what is said in this discussion to others. Nonetheless, there is a risk that other discussion participants will repeat what is shared here today. A report will also be made publicly available, but nothing you share with us will have your name attributed to it in our report. This information will not be used for any additional research in the future. Remember that you are free to refuse to answer any question. Recording: With your permission, we would like to audio record this discussion and take notes. The recordings and the notes will not be shared with anyone outside the evaluation team – we are conducting many interviews and discussions, so this is mainly to help us make sure we do not misunderstand anything you tell us or misrepresent anything in our notes. If you choose to opt out of an audio recorded interview, we will proceed with the interview and will not turn on the audio recording device, but we will still take notes. 105 | AVANSA ENDLINE REPORT USAID.GOV Informed Consent Question: • Do you have any questions for us before we begin? [Yes / No] • Are you willing to participate in this discussion? [Yes / No] • Is it okay if we record this discussion? [Okay to record / Not okay to record] If you have any questions or concerns, you may contact the Social Impact Team Leader, Mr. Andrew Carmona, at acarmona@socialimpact.com, or the Social Impact Institutional Review Board at irb@socialimpact.com. Locally, you may also contact our Senior Qualitative Researcher, Gabriela Leite-Soares, at +670 7749 1780. Additionally, if you are interested in receiving a copy of the final report, you may email the Social Impact Project Manager, Alison Smith, at asmith@socialimpact.com any time after September 1, 2020. You will also be able to find this report online by looking on the USAID Development Experience Clearinghouse (DEC) website after September 30, 2020. Would you like a moment to write down this contact information so you have if for reference should you want to follow up after this interview? Alternatively, if you do not have a way to record this contact information, one of our team members can text it to you after the interview. Would you like to provide us with your phone number so we can send you this contact information? USAID.GOV AVANSA ENDLINE REPORT | 106 FOCUS GROUP DISCUSSION SEMI-STRUCTURED QUESTIONNAIRES FARMER BENEFICIARIES FGD/Group Interviews- Farmer beneficiaries Number of participants: _______ Sex: ___M ____F Location:_______________ Date:___________________ Questions for Farmers Beneficiaries / Households Question EQ addressed 1. Please tell us how you came to know about the Avansa Agrikultura Project and how long have you been been involved with the project? Introductory a. What type of project activities do you participate in? Introductory b. Are you a member of a farmer group? More than one group? Introductory c. How did you select which activities you wanted be involved in? Introductory d. What suco are you from? Introductory 2. What have been the results of participating in these activities? EQ1 a. Probe: Increased agricultural output to sell? EQ1 b. Probe: Increased agricultural output to consume? EQ1 c. Probe: Increased ability to access and sell at markets? EQ2 d. Probe: Increased access to collectors and traders. EQ2 e. Probe: Increased involvement in farmer groups? EQ2 f. Probe: Other non-agricultural income generating activities? EQ1; EQ2 g. Probe: Knowledge about different agricultural techniques? EQ1; EQ2 h. Probe: More access to finance and equipment, and inputs? EQ2 3. Of the activities you have been involved in, which activities have been the most successful for you and your family and why have they been successful? EQ1; EQ2 107 | AVANSA ENDLINE REPORT USAID.GOV Question EQ addressed 4. Were there any project activities that you were involved in that were not successful? What were they? Why were they not successful? EQ1; EQ2 5. Have you experienced any personal changes as a result of accessing the services of the project? EQ3 a. In what ways has your confidence in engaging in agriculture or other activities changed? EQ3 b. In what ways has your confidence in providing food and income for your family changed? EQ3 6. Do you think some types of people experienced greater personal change than others? EQ3 a. Probe: What about women compared to men? EQ3 b. Probe: What about youth compared to older people? EQ3 7. Which interventions were most effective in creating personal change for you? In allowing you to be more confident in providing for yourself and your family? EQ3 8. How, if at all, has COVID-19 affected the Avansa activities that you participate in? EQ 1 a. Have you been able to develop a strategy to continue these activities despite the COVID-19 situation? Please explain. EQ 1 9. Are their activities which have been significantly affected by COVID-19? Please explain. EQ 1 10. How, if at all, has COVID-19 affected the income you have been able to earn from the activities you are involved in? EQ 1 a. Has your project related income: ___ stayed the same ___ decreased ___ increased? EQ 1 11. Did you participate in just one activity or a combinations of activities? EQ4 a. Why do you think it matters to participate in a combination of activities versus just one activity? EQ4 b. What was the benefit and drawbacks of participating in a combination of activities? EQ4 c. In your opinion, what were the most successful combination of activities and why? EQ4 d. What missing elements or other activities could have been added to these combinations of activities to make them more successful? EQ4 12. What do you think will happen after Avansa project support ends? EQ5 a. Probe: Do you think you continue to use what you learned and continue to benefit from the work? Why or why not? EQ5 USAID.GOV AVANSA ENDLINE REPORT | 108 Question EQ addressed b. Which of the activities you have been involved in will continue and why? EQ5 c. Which activities will not continue and why? EQ5 13. Do you have any additional observations on the Avansa project you’d like to share with us? General Thank you for your participation in this interview. Your responses will be kept confidential and will be used solely for the purposes of this evaluation. 109 | AVANSA ENDLINE REPORT USAID.GOV TRADERS, COLLECTORS AND AGGREGATORS FGD/Group Interviews- Traders, collectors and aggregators Number of participants: _______ Sex: ___M ____F Location:_______________ Date:___________________ Questions for Traders, collectors and aggregators Question EQ addressed 1. Please tell us how you came to know about the Avansa Agrikultura Project and how long have you been been involved with the project? Introductory 2. What type of trading, collecting and aggregating activities do you participate in? Introductory 3. How frequently do you interact with Avansa staff? How frequently do you interact with Avansa beneficiary producers? Introductory 4. What suco are you from? Introductory 5. What have been some of the results for you and your business of participating in Avansa activities or receiving Avansa support? EQ2 a. Probe: Access to producers and product generally? EQ2 b. Probe: Access to retailers? EQ2 c. Probe: Volume of products to trade? EQ2 d. Probe: Quality of products EQ2 e. Probe: Access to finance? EQ2 f. Probe: Access to business services? EQ2 g. Probe: Collectivizing EQ2 h. Probe: Income? Cost? Price? EQ2 6. How, if at all, has COVID-19 affected your activities with the Avansa production groups? EQ1 a. Have you developed a strategy to be able to continue to provide trade, collection and aggregation services to the production groups? Please explain. EQ1 USAID.GOV AVANSA ENDLINE REPORT | 110 Question EQ addressed 7. What have been the results for your family and household, or other farmers and their households that you know of after participating in Avansa-supported activities? EQ1 a. Probe: Increased agricultural output to sell? EQ1 b. Probe: Increased agricultural output to consume? EQ1 c. Probe: Increased ability to access and sell at markets? EQ2 d. Probe: Increased access to collectors and traders. EQ2 e. Probe: Increased involvement in farmer groups? EQ2 f. Probe: Other non-agricultural income generating activities? EQ1; EQ2 g. Probe: Knowledge about different agricultural techniques? EQ1; EQ2 h. Probe: More access to finance and equipment, and inputs? EQ2 8. Which activities conducted or support given by the project have been the most beneficial from the perspective of trading, collecting and aggregating and why have they been beneficial? EQ2 9. Are you able to sustain your level of business activities despite the COVID-19 situation? EQ1 10. Were there any project activities that were not beneficial to you? What were they? Why were they not beneficial? EQ2 11. Have you experienced any personal changes as a result of accessing the services of the project? EQ3 a. In what ways has your confidence in engaging in agriculture or other activities changed? EQ3 b. In what ways has your confidence in providing food and income for your family changed? EQ3 12. Do you think some types of people experienced greater personal change than others? EQ3 a. Probe: What about women compared to men? EQ3 b. Probe: What about youth compared to older people? EQ3 13. Which interventions were most effective in creating personal change for you? In allowing you to be more confident in providing for yourself and your family? EQ3 14. Did you participate in just one activity or a combinations of activities? EQ4 111 | AVANSA ENDLINE REPORT USAID.GOV Question EQ addressed a. Why do you think it matters to participate in a combination of activities versus just one activity? EQ4 b. What was the benefit and drawbacks of participating in a combination of activities? EQ4 c. In your opinion, what were the most successful combination of activities and why? EQ4 d. What missing elements or other activities could have been added to these combinations of activities to make them more successful? EQ4 15. What do you think will happen after Avansa project support ends? EQ5 a. Probe: Do you think you continue to use what you learned and continue to benefit from the work of Avansa? Why or why not? EQ5 b. Which of the activities you have been involved in or the benefits you received will continue and why? EQ5 c. Which activities or benefits will not continue and why? EQ5 16. Do you have any additional observations on the Avansa project you'd like to share with us? General Thank you for your participation in this interview. Your responses will be kept confidential and will be used solely for the purposes of this evaluation. USAID.GOV AVANSA ENDLINE REPORT | 112 ANNEX V: STATEMENTS OF DIFFERENCE No statements of difference have been submitted. 113 | AVANSA ENDLINE REPORT USAID.GOV ANNEX VI: EVALUATION TEAM MEMBERS A team of highly skilled sectoral and evaluation specialists will carry out this evaluation – thoroughly reviewing and analyzing desk review documentation, designing data collection tools, collecting data, analyzing and present findings, and preparing the evaluation report and other deliverables. Members of the ET are: Team Leader, Andrew Carmona: Mr. Andrew Carmona is a Senior Program Manager at Social Impact, Inc. with over 12 years’ experience in international development and evaluation. In his current role, he oversees a diverse portfolio of performance and impact evaluations; he has served as a Team Leader or Deputy Team Leader on three evaluations and an Evaluation Specialist on six evaluations since 2016. Mr. Carmona oversaw an agriculture value chain evaluation in Haiti in 2019 and has been the M&E advisor on a climate-resilient agriculture project in Guatemala since 2017. He served as an M&E Specialist in Timor-Leste in 2015 on an assessment of ICT across the horticulture value chain and oversaw a health sector evaluation in Timor-Leste in 2020. Mr. Carmona holds a Bachelor of Arts in International Studies from the University of California, San Diego, and a Master’s in Public Administration - Development Practice from Columbia University. Qualitative Researcher, Acacio Amaral: Mr. Acacio Amaral is a local researcher with 9 years of experience in qualitative research and has worked on multiple USAID evaluation projects in Timor￾Leste, including his work as a team member for the Avansa midline evaluation. He is a lecturer at the National University of Timor Lorosa’e and previously served as the Dean of Agriculture. Mr. Amaral holds a PhD in Veterinary Epidemiology and a Master’s of Tropical Veterinary Sciences. Qualitative Researcher, Gabriela Leite-Soares: Ms. Gabriela Leite-Soares is a local researcher with 11 years of experience in qualitative research, mostly focused on women’s health. In her previous experiences, she worked for the World Bank in Timor-Leste and well as in Washington, D.C. She previously worked on evaluations in Timor-Leste focused on women, health, and development. She has extensive skills in qualitative research from her consultancies on evaluations in Timor-Leste for the World Health Organization, UNICEF, International Finance Corporation, and the World Bank among others. She holds a bachelor’s degree in Political Science from Gonzaga University, and a Master’s in Public Administration from Cornell University. Qualitative Researcher, Zenny Correia: Ms. Zenny Correia is a local researcher with 11 years of extensive experience in qualitative research and monitoring and evaluation projects. She has worked on evaluations in Timor-Leste as a specialist, researcher, and team leader. Her evaluation work has focused on agriculture and development, including the Avansa midline evaluation. She holds a bachelor’s degree in Teacher Training and Education Science of Mathematics and a Master’s in Peace in Conflict Studies from the National University of Timor-Leste. Local Logistician/Translator, Nazario dos Santos: The core three-person ET will be supported by a local logistician, Nazario dos Santos, who will manage the ET’s field agenda while in-country, arrange travel between data collection sites, and organize meetings. The team will also be supported by the logistician for translation, fluent in English and Tetum, with additional language skills of Portuguese and Indonesian. Mr. dos Santos also has qualitative research experience on evaluations and will assist the team when needed with qualitative data collection and analysis. He holds a bachelor’s degree in Political Science and International Studies from Luther College. Home Office Support. The ET will be supported by a home office team who will supervise the technical direction of the evaluation (including a gender-focused review of all deliverables), ensure contractual and budgetary compliance, and provide administrative and logistical support. The home office team will also provide quality assurance (QA) on all data collection instruments and deliverables and implement SI’s data QA approaches. USAID.GOV AVANSA ENDLINE REPORT | 114 ANNEX VII: DISCLOSURES OF CONFLICT OF INTEREST 115 | AVANSA ENDLINE REPORT USAID.GOV USAID.GOV AVANSA ENDLINE REPORT | 116 117 | AVANSA ENDLINE REPORT USAID.GOV