PHOTOGRAPH BY AMANDA STEK AVANSA M&E AVANSA AGRIKULTURA PROJECT MIDTERM PERFORMANCE EVALUATION USAID/Timor-Leste AID-486-I-14-00001/ AID-472-TO-15-00003 July 17, 2017 Dr. Richard Columbia, Chief of Party, Avansa M&E Social Impact Building no. G, Oceano-White Sands Apartment Bidau Lecidere, Dili, Timor-Leste Tel: (+670) 7702-6807 Email: rcolumbia@socialimpact.com This publication was produced at the request of the United States Agency for International Development (USAID). It was prepared independently by Matteo Borzoni (Team Leader), Alberto Correia, and Amanda Stek, for Social Impact, Inc.; with support from Dr. Acáçio Cardoso Amaral and Eugenia Correia. DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i TABLE OF CONTENTS Executive Summary ......................................................................................................................................................................... v I. Introduction ............................................................................................................................................................................ 1 II. The Development Problem and USAID’s Response..................................................................................................... 2 The Development Problem................................................................................................................................................. 2 USAID’s Response................................................................................................................................................................. 3 III. Evaluation Purpose, Users, and Questions...................................................................................................................... 6 Evaluation Purpose and Users............................................................................................................................................ 6 Evaluation Questions............................................................................................................................................................ 6 IV. Evaluation Methodology....................................................................................................................................................... 8 Data Collection Methods..................................................................................................................................................... 8 Data Analysis......................................................................................................................................................................... 10 Biases and Other Limitations............................................................................................................................................ 12 V. Findings and Conclusions................................................................................................................................................... 13 Evaluation Question 1 ........................................................................................................................................................ 13 Evaluation Question 2 ........................................................................................................................................................ 24 Evaluation Question 3 ........................................................................................................................................................ 31 Evaluation Question 4 ........................................................................................................................................................ 34 Evaluation Question 5 ........................................................................................................................................................ 37 VI. Recommendations............................................................................................................................................................... 40 Recommendations for USAID.......................................................................................................................................... 40 Recommendations for USAID and AVANSA AG........................................................................................................ 40 Recommendations for AVANSA AG.............................................................................................................................. 41 VII. Lessons Learned................................................................................................................................................................... 42 VIII. Annexes ................................................................................................................................................................................. 44 Annex A: Evaluation Statement of Work ...................................................................................................................... 44 Annex B: Team composition............................................................................................................................................. 46 Annex C: Data Collection Protocols.............................................................................................................................. 47 Annex D: Documents Reviewed...................................................................................................................................... 88 Annex E: Evaluation Respondents and Observation Sites......................................................................................... 92 Annex F: Data Collection Schedule............................................................................................................................... 104 Annex G: Evaluation Design Matrix .............................................................................................................................. 106 Annex H: Disclosure of Conflicts of Interest............................................................................................................. 110 FIGURES, TABLES, AND IMAGES Figure 1: Percent Achieved against 5-year Project Targets for Sub-purpose 1 Output Indicators...............................14 Figure 2: Percent Achieved against 5-year Project Targets for Sub-purpose 2 Output Indicators...............................20 Figure 3: Indicator 3.3 - Number of individuals who have received USG-supported short-term agricultural sector productivity...................................................................................................................................................................28 Figure 4: Indicator 5.1 - Number of farmer groups and associations with buyer agreements.......................................28 Table 1: Key Informants by Type and Gender............................................................................................................................ 9 ii Table 2: FGD Groups by Type and Municipality........................................................................................................................ 10 Table 3: Group Characteristics and Contribution to Avansa Ag........................................................................................... 17 Table 4: Groups by Municipality and Type .................................................................................................................................. 25 Table 5: Farmer Group Performance by Municipality .............................................................................................................. 26 Image 1: Portion of Logframe from the RFP............................................................................................................................ 5 Image 2: Avansa Ag Project (and Evaluation) Sites................................................................................................................. 5 Image 3: Women-only FGD in Aileu........................................................................................................................................ 22 Image 4: Woman Farmer from Women-only FGD in Aileu .............................................................................................. 25 Image 5: Seedlings at NRM Group in Bonbonaro................................................................................................................. 31 Image 6: A purchase from Farmer Group in Maliana by a supermarket......................................................................... 35 Image 7: Farmer Field Day in Maliana, Bonbonaro on May 4, 2017 ................................................................................. 38 iii ACRONYMS ADB Asian Development Bank AMEP Activity Monitoring and Evaluation Plan ASEAN Association of South-East Asian Nation Cardno Cardno Emerging Markets USA CBO Community Based Organization CDCS Country Development and Cooperation Strategy CSO Civil Society Organization DAC Dezenvolve Agricultura Comunitaria DAC Development Assistance Committee DRR Disaster Risk Reduction EG Economic Growth ET Evaluation Team EQ Evaluation Question EU European Union FAO Food and Agriculture Organization FtF Feed the Future FGD Focus Group Discussion FY Fiscal Year GIS Geographic Information System GOTL Government of Timor-Leste GNI Gross National Income HDI Human Development Index HIAM Hamutuk Ita Ajuda Malu Health HWG Horticultural Working Group IR Intermediate Result KII Key Informant Interview MAF Ministry of Agriculture, Forestry and Fishery MFI Micro Finance Institution MOU Memorandum of Understanding MSME Micro Small and Medium Enterprise MTE Midterm Evaluation NGO Non-Governmental Organization NRM Natural Resources Management PLUP Participatory Land Use Planning PSC Project Steering Committee RFA Request for Applications RFP Request for Proposals S&L Savings and Loan SALT Sloping Agriculture Land Technology SI Social Impact, Inc. SDP Strategic Development Plan SOP Standard Operating Procedure iv SoW Statement of Work SPS Sanitary and Phyto-Sanitary Standards TOC Theory of Change ToT Training of Trainers TWG Technical Working Group UNDP United Nations Development Programme USAID United States Agency for International Development USG United States Government WTO World Trade Organization v EXECUTIVE SUMMARY INTRODUCTION The United States Agency for International Development (USAID)/Timor-Leste is supporting the Avansa Agrikultura Project, (hereafter Avansa Ag), a five-year, $19.2 million project. The project seeks to support increased economic activity and growth for Timorese citizens in targeted rural communities and municipalities. The project is implemented by Cardno Emerging Markets USA, Ltd. (hereafter Cardno), and sub-contractors Hamutuk Ita Ajuda Malu (hereafter HIAM) Health, the Norman Borlaug Institute for International Agriculture/Texas, and SSG Advisors.1 The contract with Cardno was signed on April 1, 2015, and the project will be completed in 2020. This report details the results of a midterm performance evaluation (MTE) of Avansa Ag, conducted by Social Impact, Inc. (SI) for USAID/Timor-Leste. THE DEVELOPMENT PROBLEM AND USAID’S RESPONSE The agriculture sector’s performance since Timor-Leste’s independence in 2002 has been disappointing, with national accounts data showing a decline in the mean value of agricultural production per capita from 2000 to 2013.2 Agriculture production systems in Timor-Leste are among the least developed in the world. Subsistence systems dominate all subsectors, and most farming households are highly risk averse. Child nutritional status is still very low. In 2013, USAID/Timor-Leste undertook a comprehensive assessment of opportunities for accelerating inclusive economic growth in Timor-Leste. The assessment found that a high un-met local demand for vegetables, supporting the possibility of significant results in the near term. The Avansa Ag project is predicated on the development hypothesis that if USAID/Timor-Leste increases productivity within the horticulture value chain and increases the ability of Timor-Leste’s citizens to engage in economic activities, then sustainable and inclusive economic growth will be accelerated in the agriculture sector. The project strategy has two sub-purposes. The first sub-purpose is defined as “Increased Productivity of the Horticulture Value Chain” and the second sub-purpose is defined as “Increased Ability of Timorese Citizens to Engage in Economic Activities.” At the end of this project, it is expected that about 33,000 people in 250 communities will be earning higher incomes, showing improved nutrition practices, and managing their economic assets more effectively. PURPOSE OF THE EVALUATION AND EVALUATION QUESTIONS The primary purpose of this MTE will be to provide evidence to improve the design and implementation of the ongoing Avansa Ag intervention, adapt the interventions to potentially new contexts, and explore evolving approaches for addressing identified problems. The primary users of the MTE will be the Economic Growth (EG) team at USAID/Timor-Leste. It is also expected that Cardno will use the results of the evaluation to fine-tune their activities/approach and improve implementation for the final years of the project. Five specific evaluation questions (EQ) guided this evaluation (reported in the findings and conclusions section). 1 SSG Advisors is a US woman-owned 8(a), HUB Zone SDB and leading partner for forming public private partnerships and alliances on USAID programs to leverage private sector investment. Information on other sub-contractors can be found on their respective websites: http://www.hiamhealth.org/ and https://borlaug.tamu.edu/. 2 Government of Timor-Leste (GOTL), 2015. vi EVALUATION METHODOLOGY The evaluation team (ET) utilized a theory-based, utilization-focused approach to this MTE. More specifically, the ET employed a qualitative design consisting of the following data collection methods: document review, key informant interviews (KIIs), focus group discussions (FGDs), and observations. The ET also reviewed quantitative data (indicator and survey data) collected by the project since inception. The ET visited and collected data from each of the municipalities in which Avansa Ag works. The ET followed a purposive sampling approach for KIIs and FGDs. In total, the ET conducted 57 KIIs (with 76 individuals), 23 FGDs (with 254 individuals), and 18 observations. The ET’s data analysis approach utilized data triangulation to cross-check results and several analysis methods to provide evidence for the evaluation’s findings and conclusions, including content, contribution, comparative and gender analysis. FINDINGS AND CONCLUSIONS EQ 1: To what extent has Avansa Ag been successful in achieving project sub-purposes (and associated outputs)? The project was tasked to deliver inputs on twenty key policies that constrain the agricultural sector during the life of the project (output 1). At the time of the MTE, inputs have been provided on nine policies (indicator 1.1). Project staff and USAID reported the main achievements to be progress on the Sanitary and Phytosanitary Standards (SPS) Road Map and the Pesticide Law, both of which have yet to be finalized/formalized. The government of Timor-Leste (GOTL) key informants reported appreciation for Avansa Ag support on these areas of legislation, and expressed understanding regarding the slow policy change process in the Ag sector.. The project has experimented with a wide range of Natural Resource Management (NRM) approaches (output 2). The ET did not see evidence to suggest that the Climate Change Adaptation and Disaster Risk Reduction Study conducted by the project in 2015 was used to identify the location for the NRM approaches. The exact location of NRM initiatives, according to respondents, was left to community members and grantees. Furthermore, FGDs with NRM groups revealed that trees distributed by the project were mainly planted in individual gardens rather than large reforestation sites, thus limiting the potential soil erosion control and landslide prevention effects. Similarly, when considering the ultimate aim of increasing resilience, the planting of trees on one individual’s plot of land was not found to be an effective approach. Limiting factors for the expansion of reforestation and NRM activities are the absence of secure land titles (which discourage long-term investments that provide incomes many years after planting), and the fact that large portions of land belong to the church or GOTL. The project has delivered a wide range of activities aimed at developing the horticultural value chain (outputs 3 – 6). In general, the ET found that Avansa Ag has impacted change in a majority of the farmer groups (13 out of 16) sampled for the MTE (in one way or another), though not in consistent ways across farmer groups. For example, a majority of farmer groups (10 out of 16) have made progress in using new production techniques brought to them by Avansa Ag. Eleven of the groups (out of 16) are linked to buyers, who consist of supermarkets or wholesalers. Seven of these 11 groups started a new relationship with a buyer thanks to contacts established by the project. A selected number of groups experienced changes, but did not experience change in each of the areas Avansa Ag promotes (namely use of improved production techniques, links to output and input markets, and access to credit). vii Avansa Ag’s nutrition intervention (output 7) includes training on nutrition and sanitation, school nutrition garden competitions, and behavior change campaigns. FGDs with groups that received nutrition and sanitation trainings from Avansa Ag (three nutrition groups and three farmer groups) revealed that participants had not changed behavior as a result of information shared, though groups did report learning about the topics. Participants explained that there were various reasons why they did not change their behavior as a result of the completion of a three day training including their lack of consistent access to nutritious food. Lastly, Avansa Ag promotes the inclusion and integration of women throughout project activities (output 8). In year 2, the target of 20 percent women participation was over-achieved (44 percent of women engaged). By the midterm, the project had established five women-led Savings and Loan (S&L) groups (the ET visited three). One S&L group visited by the ET was not currently saving, but the other two groups were. The project did not have data on the use of loans by individuals in the groups, but respondents reported using loans for school fees and the purchase of agriculture materials/inputs. The project successfully coordinated two conferences targeting women; and conducted multiple Women in Business Trainings. The ET asked women to recall the Women in Business Training. Across all municipalities, women struggled to recall the five-day training (by a project grantee), and could only vaguely remember topics covered. The grantee implementing this training confirmed that a one-time training was insufficient to address the needs of women. In conclusion, the ET found more evidence of progress in sub-purpose 1 than in sub-purpose 2. FGDs with farmer groups and interviews with key informants indicate increases in profitability in the horticulture value chain at the farmer level through increased use of new production techniques, increased number of links to buyers/markets, and increased access to inputs (outputs 3-6). However, these changes were not experienced by all groups; access to markets (buyers), inputs, and credit remain constraints for many Avansa Ag groups. The project’s policy work (output 1) targets key constraints in the sector. The NRM approach (output 2) has resulted in trees planted in scattered areas, and it is questionable whether this approach can lead to increased community resilience. The ET identified steps taken toward the goal of sub-purpose 2. However, no substantial changes were detected by the ET at the midterm in terms of beneficiary nutrition and sanitation practices. The project has achieved impressive numbers for the involvement of women. At this stage of the project, a key challenge in both sub-purposes exists in terms of delivery of technical assistance and training at the farmer level (delivering effective training and appropriate follow-up support). Without a well-delivered follow-up support activity (or activities), it will be difficult for the project to ultimately achieve these two sub-purposes and the ultimate project goal. However, through the project’s actions in several outputs thus far (particularly through learning and resulting adjustment of the NRM activity, and through their steps taken to learn more about effectiveness in output 7 and 8), they have shown their commitment to learning and adapting to most effectively and inclusively achieve targeted results. EQ 2: Were achievements toward Avansa Ag purposes varied by geography, community type, or gender and vulnerable/marginalized groups? The project has supported more groups in Ainaro relative to other municipalities, and far fewer groups in Dili relative to other municipalities. The project also has invested more resources in supporting farmer groups (including strawberry and S&L groups) relative to NRM and Nutrition groups at the midterm. viii There were also geographic differences observed in farmer group characteristics related to vegetable sales. For example, in 2016, Aileu received 67 percent of the total money flow generated by sales from Avansa Ag farmer groups. Both the Aileu and Bobonaro groups were the most productive relative to other municipalities in 2016 and in the initial months of 2017. The lowest sales per group are in Dili, Ainaro and Ermera municipalities. Through document reviews, FGDs and KIIs, the ET noted that farmer groups, demonstration groups, and DAC groups were more consolidated and productive. In FGDs, almost all farmer groups (15 out of 16) reported some kind of change resulting from Avansa Ag trainings. In comparison, half of the nutrition and NRM groups (three out of six) reported utilizing a new production technique, and an additional group reported some ‘other’ change as a result of Avansa Ag support. The ET noted substantial differences between demonstration and non-demonstration groups in terms of materials and follow-up support provided by Avansa Ag, with groups managing demonstration plots receiving much more support in terms of visits and equipment received. The ET visited five groups (out of 22) that were formed under DAC. These groups received fewer trainings from Avansa Ag because they had already received similar trainings under DAC. However, the groups were well connected to markets and had regular vegetable production. The project has taken efforts to ensure equal access of men and women to project benefits (participation). The ET noted that improvements in irrigation and links to buyers (directly coming to plots to purchase produce) save women time. However, women also reported increased levels of work since Avansa Ag started considering their involvement in trainings and their increased engagement on their plots (though they were satisfied with this increased work considering it led to increased incomes). The ET also noted that men (and boys) were not considered primary targets of S&L and nutrition and sanitation trainings. In conclusion, Avansa Ag achievements were found to vary by geography. Results also varied by group type, with farmer groups, DAC groups, and demonstration groups (or plots) appearing more consolidated at the time of the MTE relative to other groups. At the midterm, women’s access to training (participation), credit, and materials provided by the project appeared equal to men. EQ 3: To what extent are the activity’s purposes (including outputs/indicators and targets) still valid for the current development context in Timor-Leste? The project is in line with the GOTL Action Plan for a Hunger and Malnutrition Free Timor-Leste (2014), the Medium-Term Operation Plan (2014-2018) for the Agricultural Sector and the Timor-Leste Strategic Development Plan 2011-2013. Additionally, farmer groups (15) reported that the project was conducting activities that were relevant to their current needs. The Statement of Work (SoW) includes a section titled “Theory of Change” (TOC), however it is short and does not cover all causal pathways and components addressed by the project. This made it difficult for the ET to assess relevancy through this lens. Interviews confirmed, however, that the Avansa Ag project has identified and is seeking to address the key constraints that exist in the agricultural sector (from the policy to the farmer level), however, the project is considered too ambitious (in terms of targets) and complex (in terms of sectors and outputs included) with respect to resources and time available. ix The project focus on import substitution remains valid, while exporting vegetables within the life frame of the project is considered unrealistic by a wide range of key informants. As a consequence, the project is promoting export of dry and processed products. Grants are used by the project to tailor interventions to the needs of small businesses and stimulate demand for agricultural products, thus increasing incomes for upstream producers. Some grants also contribute to export of dry and processed products. The grant component of the project is only 3.5 percent of the project, which limits the potential impact of this approach. In conclusion, the Avansa Ag approach is still valid for the current development context in Timor-Leste. The project is well aligned with GOTL policies, the current context, and beneficiary needs. However, the scope (sectors included) and scale (targets) of the project may not be taking into account the complexity and constraints that exist. While it is difficult to comment on the relevancy of indicators and targets without a finalized TOC, respondents felt strongly that resources were being diverted to counting and reporting away from effective (and sustainable) implementation of the project approach. EQ 4: To what extent has the overall management of Avansa Ag (including management from the steering committee, USAID, and Cardno) contributed to/inhibited project progress? What opportunities, if any, exist for improvement in management of the activity? The project was characterized by a slow start-up phase. The project also struggled with initial communication and coordination with the Ministry of Agriculture, Forestry and Fishery (MAF), which left MAF frustrated and skeptical about the approach. The recent change of project leadership, however, has positively impacted how the project is perceived by stakeholders like MAF, who now report consistent and helpful communication with the project. During interviews with Avansa Ag field staff, it was reported that the project is understaffed (for field positions) with respect to the number of groups supported and the typologies of activities implemented. The distribution of field staff is not proportional to the number of groups engaged by the project. Also, field staff commute from Dili to conduct project activities in one of the municipalities (Ermera). This reduces the time for field activities. The project has made use of interns, which was justified by the fact they may be retained after the internship period. However, field staff expressed frustration with the turnover of interns and added that it is challenging to re-train them every six months. Farmer groups also expressed a similar frustration with Avansa Ag staff (primarily intern) turnover. Field staff reported that regular monitoring and field activities are conducted only for farmer groups (from twice per week to once per month), while follow-up and monitoring activities are not conducted for non-farmer groups (NRM, S&L and nutrition groups), which is left to the trainers or Avansa Ag Dili￾based staff (for example, output leaders). Four sub-contractors and grantees the ET spoke with, however, reported no budget for follow-up and post-trainings related to health, S&L, women’s leadership and NRM. Internal communication and coordination between Avansa Ag field staff and Dili￾based staff was reported to work properly for farmer groups-related activities but not for the S&L and the nutrition component. The M&E system has improved in recent months and now has improved tracking of groups and beneficiaries. However, it was found to still include some inaccuracies. Additionally, the M&E system was x reported to be mainly used for accountability purposes by Avansa ag management rather than as an input for adaptive management. The ET also noted that, while sex-disaggregation is conducted, no further disaggregation of collected data is carried out to analyze how different beneficiaries experience project results/outcomes. Strategic decisions are more influenced by project targets than by monitored indicators and data reported from the municipalities. In conclusion, the management of the project has improved since the first year, as shown by the increase in the expenditure rates, improved coordination with stakeholders, and clarified interventions, which have positively contributed to the progress and achievements. However, issues related to the staffing and internal project coordination have resulted in uneven application of the Avansa Ag project approach and some farmers’ lack of satisfaction with Avansa Ag. It has also resulted in a lack of follow-up with groups (particularly on nutrition and S&L interventions). The high number of indicators measured by the project makes monitoring a complex task involving multiple staff, without serving many adaptive management purposes. Also, the high targets and number of indicators skew project resources toward tracking and away from informed implementation. EQ 5: To what extent has Avansa Ag made progress in ensuring sustainability of activities beyond USAID support? There are aspects of the project design that key informants reported as being conducive to sustainability, particularly for sub-purpose 1; these include the provision of technical assistance on production aspects; the development of business linkages between farmer groups, buyers and input suppliers; and access to credit. Key elements reported by farmers for sustainability are links to supermarkets (buyers). However, at the midterm the project has not developed formalized sustainability plans for the different categories of beneficiary groups. Other aspects respondents highlighted as contributing to sustainability are the behavioral change campaigns (only being launched in year 3), avoidance of free delivery of expensive equipment, and training of private extension workers (from supermarkets). FGDs with farmers and interviews with supermarkets at the midterm, however, revealed that supermarket staff has little time to provide technical assistance and training. The project is also integrating interns among field staff with the purpose of training a pool of young agronomists to support farmers now and in the future. There is evidence from previous projects of supermarkets employing previous project staff. In conclusion, while the Avansa Ag project currently does not have a formalized, defined sustainability approach or plan, the project design includes aspects that are related to ensuring sustainability. Without a defined sustainability plan for groups, the capacity of the project to implement sustainable solutions for beneficiary groups is limited. Aspects limiting the sustainability of the project are the use of one-time trainings (with its limited effect on behavioral change and consequently on sustainability) and the limited capacity of supermarkets to serve the role of farmer group trainers, which will limit access to technical assistance once the project is over. RECOMMENDATIONS 1. USAID/Timor-Leste should consider re-scaling the project by using one or more of the following approaches: 1) removing outputs or components, 2) reducing targets starting with the number of beneficiaries/communities, 3) lengthening the duration of the project. Any re-scaling should be based on a clarified TOC. 2. USAID/Timor-Leste should consider increasing the budget for grants to support small companies xi that stimulate the demand of agricultural products. 3. USAID/Timor-Leste and Avansa Ag should focus on a reduced number of key policies that are critical for the project goal. 4. Avansa Ag should deliver the efficiency of technical assistance to groups by conducting follow-up trainings (and training series) and support visits (through Avansa Ag staff or grantees). 5. Avansa Ag should re-consider geographic differences across municipalities and sucos in the horticulture value chain. If estimated production quantity and type of vegetables are not sufficient to attract supermarkets in a given area, the project should consider either recommending alternatives to farmer groups or reducing the number of planned farmer groups to be established. 6. Avansa Ag should consider utilizing a staffing structure aligned with the sustainability plan for farmer groups, which most likely would require a more proportional (group/staff) approach to the distribution of staff. The project should not pursue increases in staff through interns only. 7. Avansa Ag should improve coordination between district and Dili-based staff to improve the implementation of activities, data collection and quality of M&E data. 8. Avansa Ag should develop a sustainability plan particularly at the group level to ensure all receive technical assistance from the project necessary to achieve productivity and increase profitability. LESSONS LEARNED • M&E should not only serve accountability purposes, but also provide input for an adaptive management approach to project implementation. • Coordination among complementary support services/markets is crucial for the development of horticulture value chains. • Contract farming is a possible solution for a sustainable coordination of complementary transactions. 1 I. INTRODUCTION Since May 2002 (and the Restoration of Independence in Timor-Leste), Timor-Leste’s agriculture sector has been plagued with low productivity. Approximately 70 percent of the country’s citizens are involved in the agriculture sector, and most of these are smallholder farmers with plot sizes less than one hectare. Issues such as poor soil for agriculture, steep topography (in upland areas), poor water conservation practices, lack of access to markets, and limited availability of high-quality agricultural inputs (including seeds, plant protection chemicals, and agricultural production equipment) contribute to low productivity. Further complicating this situation are the effects of poor health in the country. As the Avansa Agrikultura (Avansa Ag) Statement of Work (SoW) reports, “Prevailing research demonstrates that an individual’s ability to engage and benefit from economic opportunity is limited by the effects of chronic malnutrition.”3Any comprehensive and inclusive economic development strategy should consequently utilize nutrition-based interventions, and include women and children. Women are responsible for food preparation within households. Additionally, World Bank estimates included in the project SoW calculate that an improved infant and young child feeding program (utilizing effective education and counselling services) would have a 1,400 percent return on investment. The United States Agency for International Development (USAID)/Timor-Leste supported Avansa Ag, a five-year, $19.2 million project, to utilize a value chain-based approach complemented by interventions focused on building farm families’ human and social capital to improve opportunities for engagement in economic activities. The project seeks to support increased economic activity and growth for Timorese citizens in targeted rural communities and municipalities. The project is implemented by Cardno Emerging Markets USA, Ltd. (hereafter Cardno), and partners Hamutuk Ita Ajuda Malu (hereafter HIAM) Health, the Norman Borlaug Institute for International Agriculture/Texas A&M AgriLife Research, and SSG Advisors. The contract with Cardno was signed on April 1, 2015, and the project will be completed in 2020. The project is predicated on the development hypothesis that if USAID/Timor-Leste increases productivity within the horticulture value chain and the ability of Timor-Leste’s citizens to engage in economic activities, then sustainable and inclusive economic growth will be accelerated in the agriculture sector. The project has two Sub-Purposes and a total of eight corresponding outputs. Sub-Purpose 1 is defined as “Increased Productivity of the Horticulture Value Chain”, and Sub-Purpose 2 is defined as “Increased Ability of Timorese Citizens to Engage in Economic Activities”. This report details the results of a midterm performance evaluation (MTE) of Avansa Ag, conducted by Social Impact, Inc. (SI) for USAID/Timor-Leste in response to the Avansa Monitoring and Evaluation (M&E) Task Order, whose purpose is to develop and implement a dynamic M&E program that works to contribute to the Collaboration, Learning and Adaptation (CLA) function of Avansa Ag. In what follows, the evaluation team (ET) lays out the development problem and USAID’s response; evaluation methodology and limitations; findings and conclusions, organized by evaluation question; recommendations; and lessons learned for USAID and implementing partners going forward. 3 Oot. L. Sethuraman, K. Ross, J. Sommerfelt, E. (2016) The Effect of Chronic Malnutrition (Stunting) on Learning Ability, a Measure of Human Capital: A Model in PROFILES for Country-Level Advocacy. Technical Brief. Food and Nutriton Technical Assistance III Project. https://www.fantaproject.org/sites/default/files/resources/PROFILES-brief-stunting-learning-Feb2016.pdf 2 II. THE DEVELOPMENT PROBLEM AND USAID’S RESPONSE THE DEVELOPMENT PROBLEM According to the most recent data from the World Bank (2015), the Gross National Income (GNI) per capita in Timor-Leste was $2,180.4 This ranks Timor-Leste as a middle-income country. However, poverty is high in Timor-Leste. The World Bank also estimates that in 2014, 41.8 percent of the population lived below the national poverty line, while the United Nations Development Programme (UNDP) calculates that in 2015, the percentage of the population in multi-dimensional poverty5 was 64.3. Moreover, the 2016 UNDP Human Development Report ranked Timor-Leste 133 out of 188 countries on the Human Development Index (HDI), which takes into account GNI per capita, education and life expectancy. Child nutritional status is still very low. While the prevalence of wasting6 in children under five years was 11 percent in 2016, the percentage of stunting7 among children of the same age group was 50.2 percent.8 There is no significant difference in malnutrition between girls and boys, but malnutrition is much higher among children in rural than urban areas.9 These high numbers combined with poor agricultural production practices and a lack of income to supplement home-grown products has contributed to high levels of food insecurity and malnutrition throughout Timor-Leste. Agricultural development is crucial for sustainable growth and poverty reduction. Low agricultural productivity is a key cause of food insecurity among rural households and contributes to high rates of childhood malnutrition. Sector performance since independence in 2002 has been disappointing, with national accounts data showing a decline in the mean value of agricultural production per capita from 2000 to 2013.10 Agriculture production systems in Timor-Leste are among the least developed in the world. Subsistence systems dominate all subsectors, and most farming households are highly risk averse (unwilling, for example, to take risks like changing crops or utilizing new techniques). Annual and perennial crop yields are extremely low when compared to neighboring countries.11 Crop productivity in Timor-Leste is very low by any standards, and the objectives of food self-sufficiency and poverty reduction will not be achieved without substantial increases in productivity. Rice, maize, cassava, and sweet potato are the main staple crops.12 Coffee, which has been grown in Timor-Leste for 200 years, accounted for 98 percent of non-oil merchandise exports in 2013. It is grown on small family plots and provides the principal source of cash income for approximately one-quarter of Timorese households.13 Timor-Leste is a patriarchal society in which social norms and cultural values influence gender roles. Men are expected to be responsible for decision-making in the household and are the major income earners in the formal economy. In most rural households, women also share the role of providing for 4 Current prices, World Bank Atlas method. http://data.worldbank.org/country/timor-leste. 5 Multi-dimensional poverty complements monetary measures of poverty by considering overlapping deprivations suffered by individuals at the same time. The index identifies deprivations across the same three dimensions as the Human Development Index (HDI) and shows the number of people who are multi-dimensionally poor (suffering deprivations in 33 percent or more of the weighted indicators) and the number of weighted deprivations with which poor households typically contend with. 6 Wasting reflects low weight for height and indicates acute undernutrition. 7 Stunting reflects low height for age and indicates chronic undernutrition. 8 International Food Policy Research Institute (IFPRI), 2016. 9 Asian Development Bank (2014), Timor-Leste Country Gender Assessment. 10 Government of Timor-Leste (GOTL), 2015. 11 Asian Development Bank (ADB), 2015. 12 Ibid. 13 Ibid. 3 their families through their contributions to agriculture, fisheries, and raising livestock.14 Religious and cultural values reinforce male authority and restrict the choices available to women and girls. Cultural practices that perpetuate gender inequality have included polygamy and bride price, as well as customary principles relating to property rights, and inheritance. Gender gaps in labor force participation are significant. According to the 2013 Labor Force Survey (LFS) 40 percent of men are classified as being in the labor force (or economically active) compared with only 21 percent of women.15 An Asian Development Bank study (ADB)13 reports that the proportion of wage-earning jobs has tripled over the last decade, however, this has benefited men more than women. Also, women are more likely to have a vulnerable employment.16 The LFS found that the majority of vulnerable workers are women (70 percent) when compared to men (48 percent). The distribution by sector indicates that the agricultural sector employs more women (56 percent) than men (38 percent), the industrial sector has a predominant presence of male workers (15.5 percent versus 7 percent), while in the service sector employment rates are similar (45.4 percent for men and 44.6 percent for women).14 Since the country’s independence in 2002, gender equality in Timor-Leste has been emphasized as an important goal for the nation’s development.17 Nonetheless, research highlights that women farmers are less productive than men, largely because women often lack access to the same agricultural inputs including land, finance, technology, knowledge, and trainings. While women do a significant portion of farm activities, their work in agricultural settings is often not visible. Further, women often lack access to existing networks such as farming associations, which can help increase their yield. Increasing women’s access to agricultural resources and services would greatly improve food security and positively impact societies.18 USAID’S RESPONSE In 2013, USAID/Timor-Leste undertook a comprehensive assessment of opportunities for accelerating inclusive economic growth, which found a high un-met local demand and future export potential for vegetables, supporting the possibility of significant results in the near term. The report indicated that investments in upgrading the horticulture value chain, specifically vegetables and fruits, offered the Mission’s best opportunity to achieve impact and results in the five-year time frame of its current strategic plan. The findings of a similar Growth Diagnostic Assessment conducted by USAID in 2012 to address the binding constraints to economic growth in the country also support continued work in the agriculture sector. Both reports cited human capacity as a limiting factor in achieving economic growth. The 2012 Growth Diagnostic Assessment noted that malnutrition and poor health status lead to decreased workforce productivity and constrain both short-term and long-term economic gains. However, the USAID Gender Analysis in 2012 for the private sector stressed that training for women on improved storage techniques, product preservation and processing, management, and entrepreneurship, among other interventions, would substantially contribute to their agricultural outputs. 14 Asian Development Bank (2014), Timor-Leste Country Gender Assessment. 15 GOTL, Timor-Leste Labor Force Survey 2013. 16 The combined proportion of own-account workers and and contributing facility workers 17 Ibid. 18 AECID (2012), Studies about Gender in Five Districts of Timor Leste Bollinger, K. (2012). Gender analysis: private sector in Timor-Leste. USAID 4 Research noted in the Avansa Ag SoW explains that while a value chain-based approach can improve opportunities for engagement in economic activities, it does not guarantee that other constraints and barriers to participation experienced at the household or individual level will be addressed. To enhance the prospects for success, these activities should be complemented by interventions focused on building the human and social capital needed to more effectively engage rural households to participate in and benefit from economic activities. This research also notes that interventions need to account for the specific social, cultural and human considerations required to maximize the positive impact of value￾chain activities. The project is predicated on the development hypothesis that if USAID/Timor-Leste increases productivity within horticulture value chains and increases the ability of Timor-Leste’s citizens to engage in economic activities, then sustainable and inclusive economic growth will be accelerated in the agriculture sector. Avansa Ag aims to address key challenges of rural poverty, natural resource degradation, food insecurity, and under-nutrition. The value chain approach aims to achieve increased productivity along key horticulture value chains that include vegetables, fruits and legumes. The project intends to stimulate and support increased economic activity and growth in targeted rural communities and municipalities through the promotion of sustainable production practices, increased functionality of farmer groups and associations, improved market linkages, access to finance, and increased availability of and access to quality agricultural inputs and services. The sustainability of these interventions is promoted by maintaining sound policies, an enabling environment for the sector, as well as efforts to improve natural resource management and resilience to climate change. Avansa Ag was tasked to include interventions focused on transitioning subsistence farming households to market-oriented growers to address barriers to participation in economic activities, access/demand for credit and investment in productive assets. The project identifies opportunities to improve women’s access to productive economic resources (e.g. assets, credit, income, employment) to engage in formal economic activities. The project builds on development gains made through the USAID Dezenvolve Agricultura Comunitaria (DAC) project, such as established farmer groups and relationships with supermarkets, and scale up of DAC’s successful interventions to reach the target number of communities over the five-year project period. The project strategy has two sub-purposes and a total of eight corresponding outputs.19 Sub-purpose 1 is defined as “Increased Productivity of the Horticulture Value Chain” and has six outputs: 1. Policy and Enabling Environment Maintained or Improved 2. Climate Change Resilience Increased through Natural Resource Management 3. Sustainable Production Practices Adopted at the Farm Level 4. Capacity of Farmers’ Groups and Associations Strengthened 5. Improved Market Linkages 6. Availability of Quality Agricultural Inputs and Services Increased 19 While project documents including the Performance Monitoring and Evaluation Plan (PMEP) refer to eight ‘outcome’ areas, project annual and quarterly reports (in addition to the titles for leaders within the project team) refer to these areas as ‘output’ areas. For this reason, the ET has decided to refer to these areas as ‘output’ areas for the duration of this report. 5 Sub-purpose 2 is defined as “Increased Ability of Timorese Citizens to Engage in Economic Activities” and has two related outputs: 7. Improved Nutrition and Sanitation Practices 8. Improved Skills to Utilize Productive Assets for Economic Activity Image 1 is a portion of the Avansa Ag Logframe as included in the Request for Proposal (RFP). This more fully explains how the sub-purposes lead to the project goal. Image 1: Portion of Logframe from the RFP Avansa Ag fully integrates the two primary Feed the Future (FtF) objectives of inclusive agricultural sector growth and improved nutritional status, particularly for women and children. The project works in five municipalities: Aileu, Ainaro, Bobonaro, Dili and Ermera. See Image 2 for a map of project sites. At the end of this project, it is expected that about 33,000 people in 250 communities will be earning higher incomes, showing improved nutrition practices, and managing their economic assets more effectively. Image 2: Avansa Ag Project (and Evaluation) Sites 6 III. EVALUATION PURPOSE, USERS, AND QUESTIONS EVALUATION PURPOSE AND USERS The primary purpose of this MTE is to improve the design and implementation of the Avansa Ag intervention, adapt the interventions to potentially new contexts, and explore evolving approaches for addressing identified problems. This will be done by assessing the effectiveness of project approaches to￾date, the relevance of Avansa Ag objectives/purposes, the efficiency of Avansa Ag’s progress in achieving results, and the sustainability of different aspects of the Activity. The primary users of this MTE will be the Economic Growth (EG) team at USAID/Timor-Leste. Cardno is also expected to use the evaluation to fine-tune their interventions/approach and improve implementation for the final years of the project. The Government of Timor-Leste (GOTL) and key donors in the sector may also use the evaluation recommendations to design new interventions in the sector. EVALUATION QUESTIONS Illustrative evaluation questions (EQ) were included in the Avansa M&E Project SoW. Based on experience over the past two years as well as preliminary consultations with Avansa Ag, the ET refined the original questions and proposed the questions below for consideration by all stakeholders. The questions build on Development Assistance Committee/Organization for Economic Cooperation and Development (DAC/OECD) evaluation criteria and reflect the evaluation purpose and focus on the most useful types of questions at this stage in implementation. As part of the evaluation methodology, the ET worked with USAID and other stakeholders to validate and improve the EQs. Each of these questions allows SI to understand stakeholder perceptions in each of these areas as well as challenges faced and lessons learned. 1. To what extent has Avansa Ag been successful in achieving project sub-purposes (and associated outputs)? a. What evidence exists that the value chain approach taken by the project has led/is 7 leading to increased productivity in the horticulture sector (sub-purpose 1, outputs 1 - 6)? b. What evidence exists that Avansa Ag activities have led/are leading to increases in the ability of Timorese citizens (men and women) to engage in economic activities (sub￾purpose 2, outputs 7 and 8)? 2. Were achievements toward Avansa Ag purposes varied by geography, community type, or gender and vulnerable/marginalized groups? 3. To what extent are the activity’s purposes (including outputs/indicators and targets) still valid for the current development context in Timor-Leste? 4. To what extent is the overall management of Avansa Ag (including management from the steering committee, USAID, and Cardno) contributed to or inhibited project progress? What opportunities, if any, exist for improvement in the management of the activity? 5. To what extent has Avansa Ag made progress in ensuring sustainability of activities beyond USAID support? 8 IV. EVALUATION METHODOLOGY Based on the type of questions and purpose noted above, the ET’s methodology required an evaluative approach with a keen eye towards making well substantiated recommendations about future directions. Furthermore, Avansa Ag is a wide-ranging program with an ambitious scope. It involves a multitude of actors in the horticultural value chain (farmers, input suppliers, supermarket wholesalers, banks and micro-finance institutions, quarantine authorities, etc.), in NRM, and in nutrition. To provide substantive, evidence-based conclusions and recommendations, the evaluation needed to be targeted in its approach. The ET utilized a theory-based, utilization-focused approach to this MTE. Theory-based evaluation focuses on providing an in-depth analysis of a program’s underlying logic and causal linkages. Importantly, it recognizes that a multitude of factors and interactions influences a program’s impact and looks to identify those causal factors judged to be most critical to a program’s overall success. A utilization￾focused approach is based on the principle that evaluations should be judged on their usefulness to their intended users. Therefore, evaluations should be planned and conducted in ways that enhance the likely utilization of both the findings and the process itself to inform decisions. A commitment to this approach guided the ET in their interaction with USAID, the implementers, and beneficiaries during data collection and analysis. The evaluation employed a qualitative design consisting of the following data collection methods: document review, KIIs, FGDs, and observations, described below. The ET also reviewed quantitative data (indicator and survey data) collected by the project since inception. These methods provided data to answer each of the evaluation questions. The team composition and qualifications can be found in Annex B DATA COLLECTION METHODS The ET visited and collected data from each of the municipalities in which Avansa Ag works, including Aileu, Ainaro, Bobonaro, Dili, and Ermera. In each of these municipalities, the ET conducted KIIs, FGDs, and observations. Annex F includes the specific evaluation schedule. Document Review The ET benefitted from a rich project document set. Prior to the initiation of fieldwork, the team reviewed the evaluation SoW, Avansa Ag Project Description, Avansa Ag Results Framework, Avansa Ag Activity Monitoring and Evaluation Plan (AMEP), initial studies (market system analysis, climate change adaptation and disaster risk reduction, policy and enabling environment action matrix, rapid partnership appraisal, and the annual agribusiness and household survey), annual reports, and work plans. This provided a useful overview of the diversity of activities undertaken by the project and allowed the team to develop targeted data collection tools. The team worked with Cardno and USAID/Timor-Leste to identify and obtain other relevant documents as needed throughout implementation of the evaluation, including non-project and secondary documents. A list of documents reviewed for this evaluation is included in Annex D. Key Informant Interviews (KIIs) The ET used a purposeful sampling approach20 to identify candidates for KIIs. While the selection of key 20 Purposeful sampling is a non-probability sampling technique that relies on judgment of the researcher to select units of analysis. It is widely used in qualitative research for the identification and selection of information-rich cases. This involves identifying and selecting individuals or groups of individuals that are especially knowledgeable about or experienced with a phenomenon of interest. 9 informants was informed by recommendations from Cardno and USAID, the ET also applied other criteria in the selection process, including the key informants’ relative positions of authority within their respective organizations/communities, the degree to which they were beneficiaries of Avansa Ag support, and the value of the responses they were likely to provide to the evaluation effort. KII protocols consisted of interview topics and questions that addressed and were derived from the EQs, as well as from the ET’s document review, their discussions with USAID and Cardno staff, and their evaluation design knowledge. These protocols guided KIIs with target respondent types, and are included in Annex C. The ET conducted KIIs at locations selected by (and therefore convenient for) the interviewees. A total of 57 KIIs (including one-on-one and group interviews21) were conducted with 76 individuals from all project municipalities. The key informants for the MTE are disaggregated below in Table 1 by respondent type and gender. Table 1: Key Informants by Type and Gender Key Informant Type Male Female TOTAL USAID 2 1 3 Avansa Ag (implementers including Avansa Ag, HIAM Health, SSG, and Texas A&M) 22 8 30 Avansa Ag Grantee/Sub-Contractor 2 2 4 National GOTL 3 - 3 Municipal GOTL 6 - 5 Bank/Micro-Finance Institution (MFI) 4 - 4 Horticultural Association 2 - 2 Input Supplier/Aggregator 5 - 5 Supermarket 4 - 4 Trader 4 1 5 Other (including other Ag projects, other donors, etc) 7 3 10 TOTAL 61 15 76 Focus Group Discussions (FGDs) The ET conducted FGDs with farmer groups located in the project sites in Aileu, Ainaro, Bobonaro, Dili and Ermera. There are three different categories of groups, namely Farmer (sub-divided into Farmer, Strawberry and Savings and Loan – hereafter S&L22), NRM, and Nutrition.23 The ET collected the complete list of groups established by Avansa Ag and purposively sampled groups. The ET coordinated with Avansa Ag Municipal Coordinators to communicate with farmer group members, identify an FGD location, and encourage participation. The ET took steps to ensure that women, men, and youth were represented in each FGD, and engaged equally in the discussion. Groups were made up of women, men, and youth. Based on discussions with Avansa Ag staff and previous experience in the country, the ET determined that participation of both men and women in the same FGD would not inhibit free and open participation of all; however, in order to properly account for gender dynamics two FGDs were intentionally conducted solely with women. The ET made every effort 21 Group interviews included two or more respondents but followed the KII protocols. 22 Farmer groups produce a wide range of vegetables, Strawberry groups only produce strawberries. Saving and Loan groups are farmer groups that also have an internal savings and loan scheme. 23 Avansa Ag recently started school garden competition sites in February 2017. These sites were not included in the FGD sampling frame, but the ET visited one site in Bobonaro, and the school garden competition plans and progress were discussed in KIIs. 10 to remove any obstacles that would prevent women from taking part in KIIs and FGDs (including avoiding conducting KIIs and FGDs after 5:00 pm and on Sundays, and holding meetings in locations where women could easily meet). All FGDs were conducted on group plots or group leader homes, ensuring participation was convenient for group members. FGD protocols consisted of interview questions that addressed and were derived from the EQs, as well as from the ET’s document review, their discussions with USAID and Cardno staff, and their evaluation design knowledge. These protocols guided FGDs with target groups, and are included in Annex C. All FGDs followed a common format and lasted between 1 and 1.5 hours. The ET conducted a total of 23 FGDs with groups across all municipalities, as detailed in Table 2 below, disaggregated by group type and location. Of note, most groups considered ‘farmer’ groups had also received trainings in nutrition and/or in S&L. While most groups are categorized in the table below as ‘Farmer’ (74 percent), these groups were also able to provide information about other Avansa Ag interventions as well. FGDs included a total of 254 respondents (126 M, 128 F) and had an average of 11 participants each (ranging from 5 to 17 individuals). Group participants were invited to the FGDs through group leaders, who invited all members with the hopes that between 8 to 10 would attend. Table 2: FGD Groups by Type and Municipality Municipality Group Type Farmer (including Strawberry, S&L, and Women’s Groups NRM Nutrition TOTAL Dili 1 - - 1 Aileu 4 1 1 6 Ainaro 3 - 1 4 Ermera 4 1 1 6 Bobonaro 5 1 - 6 TOTAL 17 3 3 23 Observations Since the FGDs were conducted with farmers’ groups across the five municipalities, the ET concluded a select number of FGDs with field observations in the group farming area. The ET requested group permission to observe plots and followed the observation protocol included in Annex C. The ET documented notes and photos for each observation, and used observations to ground-truth details shared both during FGDs and information from project documents. The ET conducted a total of 18 observations across the five municipalities. DATA ANALYSIS The ET’s data analysis approach utilized data triangulation to cross-check results and several analysis methods to provide evidence for the evaluation’s findings and conclusions. Data Analysis Processes Evaluation team members took notes during KIIs, FGDs, and observations and shared reflections on each data collection activity with other team members within two days of the data collection activity. These sessions, called internal debriefs, allowed for discussion of evidence collected, patterns, and discrepancies that helped answer the evaluation questions. Post-fieldwork, the team cleaned and shared electronic summaries of interview notes. During analysis, the team disaggregated data by sex, group 11 type, and geographic location, when appropriate, to capture any differences across these categories. The team captured preliminary findings and conclusions in an evaluation findings matrix that organized analysis and recommendations by evaluation question. The matrix served as a basis for the preliminary findings presentation conducted with USAID and Cardno on May 15, as well as for the evaluation report. Data Triangulation The ET used analytical triangulation approaches as part of the development of findings and conclusions. Triangulation enabled the ET to cross-verify and cross-validate the findings that emerged from using the above methods and data sources to identify correlations between findings and determine overall program effectiveness. In particular, the ET utilized methodological triangulation to develop parallel protocols with the same or similar questions across its KIIs and FGDs. This enabled greater data triangulation because each method addressed sub-sets of the same evaluation questions, and their findings were validated or refuted by the other techniques (findings that were refuted by other techniques were not retained). Methodological triangulation also enabled the ET to strengthen the potential linkages and accuracy of the data if the results obtained through one method were less conclusive than another method. Data Analysis Methods The ET employed several data analysis methods to identify key findings from the collected data, as well as to draw conclusions and make recommendations on sustaining the positive results of Avansa Ag. Analysis methods included the following: 1. Content Analysis – The ET conducted content analysis through intensive review of collected KII and FGD data to identify and highlight notable examples of Avansa Ag’s successes (or lack of successes) that contributed to (or inhibited) Avansa Ag’s contributions to their identified objectives. This was triangulated with project documents/data and observations results. 2. Contribution Analysis – Contribution analysis is an approach for assessing and inferring causality in program evaluations. Contribution analysis was conducted particularly using FGD data, focusing on questions that asked group members to share changes observed since beginning engagement with Avansa Ag. 3. Comparative Analysis – The ET also conducted comparisons of Avansa Ag results across stakeholder groups, geographies, and gender to assess either convergence or divergence in perspectives (triangulation). 4. Gender Analysis – Gender analysis included disaggregation of data by gender and analyzing the effects of the project on men, women, and children (more on this below). Consistent with USAID evaluation policy, and recognizing that effects of integration and the success of the project might vary across gender, SI applied a gender perspective and ensured that research activities included female study participants. The SI Gender Specialist worked with the team to ensure compliance and gender sensitivity on data collection protocols and tools as well as other contract deliverables. Additionally, each team member completed USAID’s Gender 101 Training and discussed the plan for interviewing women farmers with the Avansa Ag Gender and Youth Specialist. More specifically, the FGDs and KIIs included questions to assess the extent to which women beneficiaries have been involved in the development of the horticultural value chain, and have participated in NRM and nutrition activities, particularly related to EQ 2. To capture different perspectives, two FGDs were conducted with only women. The ET analyzed how gender issues have 12 been addressed both in the initial design of the project activities and in implementation (in output 8 and across all outputs). BIASES AND OTHER LIMITATIONS USAID/Timor-Leste and other donors have provided assistance in the agricultural sector in Timor-Leste for many years. The fact that multiple donors and implementers have been actively working in this sector presents challenges for making judgments about attribution. Given that this limitation is understood by USAID and the ET from the outset, the evaluation focused less on attribution and more on implementation, approach, and contributions of the Avansa Ag Activity. During fieldwork and data analysis, the ET was aware of several limitations and risks for bias. The ET took every effort to mitigate the risks below (listed in order of frequency with which the ET encountered them) over the course of the evaluation. • Recall Bias: Recall bias is a common evaluation problem. Avansa Ag beneficiaries, in certain instances, responded to questions posed by the ET with answers that blended their experiences into a composite memory. Respondents who participated in more than one Avansa Ag activity, in similar activities conducted by other implementers, or were participants in DAC were not always able to distinguish Avansa Ag activities (trainings, events, meetings) as separate activities. The ET mitigated this risk by a) asking the question to multiple group members, b) asking questions using various terms for “training”, and c) conducting as many KIIs and FGDs as necessary within the available evaluation period to triangulate responses and increase the validity of the evaluation findings. • Response Bias: Response bias is the risk that key informants may be motivated to provide responses that would be considered socially desirable or influential in obtaining donor support. Response bias is also connected to cultural and social norms and impacted by gender or social ranking. This was a risk identified during FGDs. If the first person who speaks in a FGD is the most senior in the group, other participants might take their cues from this person and only echo his/her responses. The ET effectively probed on programmatic challenges and limitations (asking specific questions about desired alternative courses of action), thus minimizing this bias. • Selection Bias: Selection bias is an inherent risk when implementers help to facilitate contact with project beneficiaries. The ET worked closely with Avansa Ag to organize FGDs and KIIs with project beneficiaries. As there was a risk that Avansa Ag would select the most active, responsive, or engaged beneficiaries, the ET also selected additional respondents from the complete sampling frame to ensure balanced responses. Beneficiaries were forthcoming about Avansa Ag challenges and limitations, however. This suggests that bias did not prevent the collection of necessary data. 13 V. FINDINGS AND CONCLUSIONS EVALUATION QUESTION 1 To what extent has Avansa Ag been successful in achieving project sub-purposes (and associated outputs)? A. What evidence exists that the value chain approach taken by the project has led to/is leading to increased productivity in the horticulture sector (sub-purpose 1, outputs 1 - 6)? B. What evidence exists that Avansa Ag activities have/are leading to increases in the ability of Timorese citizens (men and women) to engage in economic activities (sub-purpose 2, outputs 7 and 8)? Findings Sub-purpose 1: Increased Productivity of the Horticulture Value Chain At the time of the evaluation, the project had M&E data through March 2017. There are 23 project indicators in total in the AMEP under sub-purpose 1, and 19 of those indicators target output level results (for outputs 1 – 6). After 24 months of implementation, and with 40 percent of the project life time elapsed, the project has achieved 40 percent of the project five-year target24 for 10 indicators out of 19 (53 percent of the indicators). For the remaining nine indicators, the project has achieved less than 30 percent of the five-year target. Figure 1 shows that the greatest results have been achieved in Indicator 1.2 (Number of key stakeholders regularly supported), Indicator 5.2 (Number of food security private enterprises (for profit), producers’ organizations, water users’ associations, women’s groups, trade and business associations, and community-based organizations (CBOs) receiving USG assistance), Indicator 2.3 (Number of co-management/user groups formed, active and strengthened) and Indicator 5.4 (Number of MSMEs, including farmers, receiving business development services from USG-assisted sources). The figure also shows that output 3 (Sustainable Production Practices Adopted at Farm Level) and output 5 (Improved Market Linkages) have the most indicators with achievements below 30 percent.25 The lowest achievement is in indicator 4.1 (amount of credit extended to farmers, farmer groups and associations), with only $ 6,198 granted as credit (at the time of the evaluation) out of a project target of $ 1 million . 24 The project revised targets in March 2017. The new targets are referenced here and in Figure 1below. 25 The project has achieved the five-year target for two of these indicators, namely 1.2 and 5.2. 14 Figure 1: Percent Achieved against 5-year Project Targets for Sub-purpose 1 Output Indicators Source: AMEP data as of March 2017 Note: ‘N’ refers to ‘number’; ‘Ha’ refers to ‘Hectares’; ‘USG’ refers to ‘United States Government’ Output 1: Policy and Enabling Environment Maintained or Improved The project was tasked to deliver inputs and technical assistance on twenty key policies that constrain the agricultural sector in Timor-Leste. Target pieces of legislation were set either a) in the Avansa Ag contract by USAID or b) through the Market Systems Analysis (resulting in a policy reform matrix26), an analysis that identified areas of legislation related to the sector and then highlighted those that were the most critical (and feasible) for project focus. This analysis identified five policy areas for the project to work on starting in year 2. At the time of this evaluation, inputs have been provided on nine policies (indicator 1.1) according to the initial plan and policy reform matrix in the areas of pesticide legislation, fertilizer and pesticide import procedures, and organic produce. The project has engaged a total of 54 stakeholders on these policies (indicator 1.2). The project has coordinated with the MAF to build support in the areas of chemical usage, sanitary, and phytosanitary standards (SPS) and is planning to develop a National Horticultural Policy. The project has also collaborated with the United Nations Food and Agriculture Organization (FAO) to draft a pesticide law and manual. The FAO consultants, partially supported by the project, conducted wide consultations with government authorities, civil society organizations, and the private sector. The draft law has yet to be presented to the GOTL. The project has also developed a Pesticide Evaluation 26 Avansa Ag Policy Reform Matrix, September 2015. 15 Report and Safer Use Action Plan (PERSUAP). The PERSUAP brings the project into compliance with USAID’s environmental regulations on pesticide use and offers best practices on pesticide use and management, thus reducing the chances of environmental and health impacts due to pesticide training, promotion and use. In addition, the project has participated in meetings for the National Forestry Law, which is being led by the European Union (EU). Other inputs provided by the project have included the translation of the Expropriation and Real Estate Fund Laws from Portuguese to English, which was sent to the Parliament. On SPS specifically, the project is developing a Road Map to guide the GOTL in implementing necessary measures to meet SPS requirements for the World Trade Organization (WTO) and Association of Southeast Asian Nation (ASEAN) standards compliance. The project has also provided support to the national quarantine office. A training needs assessment was developed along with fertilizer biosecurity risk profiles and standard operating procedures (SOP) for fertilizers import permit issuances, checklists for fertilize import application assessments and SOP for fertilizes sampling and inspections. Trainings for quarantine staff were provided on the application of the SOPs and relevant checklists. KIIs with MAF officials revealed that the work done by the project on quarantine is important, however there is still much to be done toward the development of proper quarantine systems since human and laboratory capacities are minimal in this area. The project also organized and facilitated meetings of the Horticulture Working Group (HWG) and Technical Working Group (TWG) to discuss the policies noted above. During one meeting of the TWG, the project presented the results of an Avansa Ag study on economic viability of organic production in Timor-Leste.27 Project staff and USAID reported the main achievements in this output to be progress on the SPS Road Map and the Pesticide Law, both of which have yet to be finalized/formalized. The inputs on these policies were seen as ‘practical and pragmatic’ by project staff and USAID. While GOTL key informants reported appreciation for Avansa Ag support on these areas of legislation, they acknowledged the significant challenges remaining toward the ultimate goal of an efficient agriculture sector. Output 2: Climate Change Resilience Increased through Natural Resource Management For the second output, the project has experimented with a wide range of approaches including the establishment of nurseries for forest seedling and fruit tree seedling production, the distribution of seedlings to community members, the implementation of participatory land use planning (PLUP) in one community in Hera (Dili), trainings on sloping agriculture land technology (SALT), incorporation of climate change information into training modules, and the development of NRM plans. The project promoted suco-managed NRM activities and has thus far developed 13 NRM groups (of which the ET visited three). Project staff reported that most farmers were reluctant to undertake NRM activities at the beginning of the project as they did not see the economic benefit. Consequently, based on learning from experience in years 1 and 2, the project re-oriented the NRM approach to emphasize the economic benefit to planting trees, by promoting planting of perennial crops that generate income (for example, fruit trees including rambutan, mango, jackfruit, and passion fruit; trees including mahogany, bamboo, and teak; and crops like candlenut, clove, and pepper). 27 Avansa Ag (2017) Economic Viability of Organic Production in Timor-Leste. 16 During the first year of the project, Avansa Ag conducted a Climate Change Adaptation and Disaster Risk Reduction study.28 This study reviewed climate change sustainability aspects of the project, focused on disaster risk reduction (DRR) and management in Timor-Leste, and included suco-level information for hazards in geographic information system (GIS) format. However, the ET did not see evidence to suggest that the study was used to identify the location for the approaches detailed above including reforestation, SALT and NRM interventions. The exact location of NRM initiatives, according to interviews with Avansa Ag staff, was left to community members and grantees implementing these interventions. Furthermore, FGDs with NRM groups revealed that trees distributed by the project were mainly planted in individual gardens rather than large reforestation sites, thus limiting the potential soil erosion control and landslide prevention effects. Similarly, when considering the ultimate aim of increasing resilience29, tree planting on one individual’s plot of land is not an effective approach. In one case, all 2,200 bamboo seedlings produced in one community in 2016 were planted in the land of one person. Such an approach, although it can generate immediate incomes for people involved in seedling production and planting, increases the resilience of the plot owner only (who can sell bamboo trees in presence of shocks or stresses). Of the three NRM groups that the ET visited, all had received polybags and other nursery materials (e.g. mash) and two had received trainings on seedling production from Avansa Ag. Groups could articulate how the tree planting prevents erosion and landslides, and reported that they could use the trees for additional income in the future. According to a government MAF advisor, limiting factors for the expansion of reforestation and NRM activities are the absence of secure land titles (which discourage long-term investments that provide incomes many years after planting), and the fact that large portions of land belong to the church or GOTL. Output 3 – 6: Sustainable Production Practices Adopted; Farmer Groups and Associations Strengthened; Market Linkages Improved; Availability and Quality of Agricultural Inputs Improved The project has delivered a wide range of activities aimed at developing the horticultural value chain including the following: establishment of farmer groups; establishment of demonstration plots; providing technical assistance/trainings to farmer groups on topics ranging from production techniques to group/business management; and linking farmer groups to buyers, input suppliers, and potential sources of credit. In general, the ET found that Avansa Ag has impacted change in a majority of the farmer groups sampled (13 out of 16) for this evaluation (in one way or another), though not in consistent ways across farmer groups. The ET found that the technical assistance provided to farmer groups differs by group in terms of the trainings provided by Avansa Ag, the frequency of Avansa Ag staff visits, and the materials provided by Avansa Ag. For example, two groups (of those formed by Avansa Ag) reported participating in as many as six trainings while four other groups reported participating in none or one. Seven groups reported frequent visits (twice per week) by Avansa Ag staff, while seven other groups reported no visits or monthly visits. Additional information about the management of the project is presented under EQ 4. 28 USAID/Timor-Leste. Climate Change Adaptation and Disaster Risk Reduction for USAID’s Avansa Agrikultura Project in Timor-Leste Draft Report. 29 USAID defines resilience as ‘the ability of people, households, communities, countries, and systems to mitigate, adapt to, and recover from shocks and stresses in a manner that reduces chronic vulnerability and facilitates inclusive growth’. https://www.usaid.gov/resilience. 17 Table 3 summarizes the results of 16 FGDs with farmers producing vegetables and strawberries with Avansa Ag support. The table depicts the groups’ responses to two sets of questions, the first regarding existing group characteristics (see the first and second column) and the second regarding whether Avansa Ag contributed to those characteristics (third column, characteristics recently experienced as a result of Avansa Ag assistance). Table 3: Group Characteristics and Contribution to Avansa Ag30 Group Characteristics N of farmer groups (out of 16) N of farmer groups attributing changes to Avansa Ag (out of 16) Linkage31 to buyers 11 7 Linkage to input suppliers 11 6 Access to credit or internal saving and loan scheme 4 2 Use of new production techniques NA 10 Ten groups are now using agricultural production techniques that they were not using before the project. These included the following: better production of vegetable seedlings in plastic trays; better spacing between planted vegetables; use and production of liquid and organic fertilizers (e.g. compost); establishment and maintenance of raised beds; use of drip irrigation; and establishment and use of low tunnels. During observations, the ET noted the frequent use of raised beds. Except for the groups in Hato-Udo (Ainaro), farmers have not been as successful in maintaining low tunnels due to the quality of the plastic and high winds in certain areas. On production aspects, the main challenges reported by farmers were access to water during the dry season, availability of manure for compost production (which affects plant nutrition), and the quality of seeds supplied by local shops or supermarkets. These challenges were also confirmed during interviews with Avansa Ag staff and another key expert. The table shows that eleven of the groups are linked to buyers, who consist of supermarkets or wholesalers. Seven of the eleven groups started a new relationship with a buyer thanks to contacts established by the project, while four already were linked to buyers before Avansa Ag. This is the case for groups that were established by the previous USAID-funded agriculture project, DAC, or by projects implemented by Non-Governmental Organizations (NGO) like World Vision. A common complaint during the ETs FGDs with farmer groups is that supermarkets often refuse to buy a portion of the vegetable production either because of low quality or oversupply. Avansa Ag staff reported that one key impact of the project is the establishment of regular business relations (links) between farmer groups and supermarkets. M&E data report that there are 33 groups with a buyer agreement with a supermarket of a wholesaler at the time of the MTE.32 However, farmers made comments about not fully understanding the nature of these documents. For example, farmers 30 Of note, the ET also asked about changes observed in technical assistance (excluding technical assistance provided by Avansa Ag). No groups reported receiving assistance from MAF or from supermarkets at the time of the midterm. This is therefore not included in the table. 31 A link (or linkage) is a commercial relationship between a farmer group and a) a buyer (who purchases a group’s produce) and/or b) an input supplier (who sells agricultural inputs to farmer groups). A group that sells to a local market would not be considered “linked” to a buyer via a commercial relationship, for example. 32 Indictor 5.1: Number of farmers, farmer groups and associations with buyer agreements. 18 reported that they produced different produce (and different quantities of produce) than agreed to with the supermarkets. Also, the annex of the MoU specifying quantities and type of vegetables to be produced was not consistently filled. Farmers then expressed frustration regarding the supermarkets’ refusal to purchase produce that was not agreed to previously (exemplifying a misunderstanding by the farmer groups of the agreements made between them and the supermarket). Supermarkets also reported that they are not particularly interested in MoUs; they need farmers to produce what has been agreed to, through a verbal or written agreement. They can operate with MoUs, but can also operate without them through other forms of informal agreements (as they do with groups not in Avansa Ag). The project is addressing the lack of available inputs in rural areas in three main ways: 1) by linking farmers to buyers that also provide seeds, 2) by promoting consignment agreements between input suppliers based in Dili and small shops in rural areas, and 3) by introducing representatives of farmer groups to input shops. FGDs revealed that the main sources of seeds for farmers are supermarkets, which distribute seeds to farmers and then subtract the cost of seeds from the produce harvested. In this regard, Table 3 shows that the number of groups with buying agreements is the same as the number of groups with access to input markets. This is because the groups selling to supermarkets also buy seeds from supermarkets. In this way, the project is promoting interlinkage contracts, whereby two parties (i.e. the farmer and the supermarket) trade in at least two markets (i.e. vegetable markets and input markets) on conditions that the terms of such trade are jointly determined. In addition, during a study tour in Indonesia, five Timorese input supplies were introduced to input suppliers in Indonesia to start new business relationships. As a result, two companies based in Dili are now importing tillers from Indonesia (on a cost-share basis with the project). Consignment agreements had recently started when the MTE was conducted, so the ET could not find evidence on how the consignment agreements were working. Evidence of the effect of the introduction of farmers to shops is mixed. For instance, two groups in Hato-Udu (Ainaro) were introduced to an input shop in Same and are still buying seeds and other inputs from that shop. In Aileu, a shop owner who was interviewed reported that the farmers introduced to his shop by Avansa Ag did not come back to buy inputs. Additionally, farmers from one FGD reported that they have never returned to Dili to buy inputs after the visit organized by the project. During a farmer field day in Bobonaro that the ET attended, input suppliers reported engagement with Avansa Ag staff, but could not yet report impacts from their engagement (they had not yet experienced an increase in clientele), while an input shop in Dili reported an increase in sales of about 10% in 2017 in comparison to 2016, attributing this to Avansa Ag. Table 3 also shows that only four groups had some access to credit at the time of the MTE. Two of the groups the ET met with have members with a loan from a bank or micro-finance institution (MFI). However, they were granted the loans before Avansa Ag implementation. The two additional farmer groups established an internal S&L scheme as a result of trainings provided by Avansa Ag (see findings in output 8).33 As project staff note, however, the project started its strategic focus on access to credit in year 3. This is reflected in the low achievement of indicator 4.1 (Quantity of credit extended to farmers) for the five-year target in March 2017 (see Figure 1). The ET talked with two microfinance institutions (MFI) and one bank, and all reported recent engagement with the project and hope for eventual results from the engagement for the benefit of farmers. 33 One S&L group visited by the ET had received one S&L training but was not currently saving or loaning out from the group to its members, despite project records that reported this was occurring. 19 The project has invested resources to establish demonstration plots (called ‘demo plots’), which are managed by farmer groups with considerable assistance from Avansa Ag in terms of staff follow-up, equipment and inputs delivered. Demo plots were intended to have a demonstration function for other farmers, however the ET found little evidence of demo plots being used in this way. With the exclusion of “Farmer Field Day” in Maliana (Bobonaro), during which the demo plot of Maliana was visited by other farmers, all visited farmer groups that do not manage demo plots reported not visiting or learning from demo plots. When comparing changes (reported by farmers) reported by farmer groups managing demo plots vs regular farmer groups, the latter experienced more changes as a result of Avansa Ag support. Lastly, 15 farmer groups out of 16 mentioned “other” changes that Avansa Ag contributed to, including (but not limited to) the following: grow new tree/crop, higher quantity produced, and higher quality produced (not noted in Table 3). New crops varied by farmer group and largely depended on their specific buyer and their needs. At least three groups visited by the ET had no experience in vegetable or strawberry production before the project. They reported that vegetable production presents an important additional source of income. As an example, a farmer, while commenting about his experience with starting to produce vegetables with Avansa Ag support, reported that “Chefe xxx34 has changed my life. I had nothing to do before the project and no income.” However, FGDs and KIIs with Avansa Ag staff confirmed that farmers are reluctant to engage in the production of the most profitable vegetables because these vegetables require longer growth periods before the harvest (e.g. broccoli and cauliflower). Farmers tend to prefer producing vegetables with short production periods (e.g. bok choy). Overall, Avansa Ag staff reported that one of the main problems for a successful delivery of the farmer￾level intervention is the farmers’ low education levels (especially regarding finance basics) and the advanced age of the farmers that tend to participate in Avansa Ag activities. Actions have been taken by Avansa Ag to mitigate these aspects by organizing trainings on financial literacy and by promoting youth participation in groups and trainings. However, these aspects remain a concern for ultimate achievement of project results. In addition to changes observed at the farmer level, the project also conducted activities under these outputs promoting exports. To promote new market linkages with foreign countries, in February 2016 Avansa Ag organized a study tour to Singapore, which was attended by MAF, private sector actors and Avansa Ag staff. Participants in the study tour reported that buyers in Singapore were mainly interested in broccoli, at a minimum quantity of five containers per week. However, five key informants (a supermarket, STTA, input supplier, a MAF advisor, and wholesaler/exporter) reported that such a quantity of fresh produce is impossible to export from Timor-Leste within the lifetime of this project. In this regard, one key informant reported, “It is more probable that Timor-Leste wins the football world cup than the project export vegetables.” Export of dry products is promoted by Avansa Ag through grants for small equipment and by supporting the participation of two start-up companies in Café Asia 2017, which is a regional food and beverage expo in Singapore. 34A title the farmer used for the Avansa Ag Municipality Coordinator. This literally translates to boss or chief. For confidentiality reasons, the name of Municipality Coordinator is removed from the quote. 20 Sub-purpose 2: Increased Ability of Timorese Citizens to Engage in Economic Activities AMEP data show that Avansa Ag has achieved 40 percent of the five-year project target for three of the seven output indicators (43 percent) of sub-purpose 2, while achieving 30 percent or less for the other four indicators. Figure 2 shows more progress in output 7 relative to output 8. Data for indicator 7.1 (Percent of 6.23 month old children receiving acceptable diet) and indicator 7.2 (Number of food consumed by women of reproductive age) suggest that the project is strongly improving nutrition habits among target communities. However, the data for these indicators is based on a comparison between a baseline survey (collected in 2015) and a nutrition survey (collected in 2016). The surveys had different respondent groups (the baseline was conducted with a a representative, random group of citizens from the Avansa Ag municipalities, while the nutrition survey was a sampled group of project beneficiaries as of year 1), and the latter survey included a small of number of respondents (12 women). While the nutrition survey data, targeting Avansa Ag beneficiaries (as required by FtF methodology) will be useful to collect and use for comparison between years of Avansa Ag support (Year 1 vs Year 2, Year 2 vs Year 3, etc), the differing methods of sampling between the baseline and the nutrition surveys make comparison less reliable. Additionally, the ET found no evidence of changed/improved nutrition habits during qualitative data collection as discussed below.35 Figure 2: Percent Achieved against 5-year Project Targets for Sub-purpose 2 Output Indicators Source: AMEP data as of March 2017 Note: ‘N’ refers to ‘number’; ‘MDDS’ refers to ‘Minimum Diet Diversity Score’; ‘USG’ refers to ‘United States Government’ Output 7: Improved Nutrition and Sanitation Practices Avansa Ag’s nutrition intervention is implemented primarily by HIAM Health and includes training on nutrition and sanitation (including development of community gardens), school nutrition garden competitions36, and behavior change campaigns. The latter portions of this intervention have only recently been launched (in year 3), but nutrition and sanitation trainings have been conducted in target communities since year 2. Nutrition and sanitation trainings are conducted over the course of three 35 More specifically, the nutrition survey is beneficiary-based. When the sample was drawn, the sampling frame was limited due to the recent launching of output 7 activities. Moreover, there was a limited number of female respondents (12) out of total respondents of the nutrition survey (144). 36 This work is conducted by two sub-contractors, namely HIAM Health and the Borlaug Institute. 21 days with target Avansa Ag nutrition groups and/or farmer groups. During the first two days, the groups received information on vegetable production, and during the third day, topics discussed included the importance of consuming different food groups, complementary food for 6-23 month old children, diet for pregnant and lactating mothers and hand washing. Interviews with Avansa Ag staff and HIAM Health revealed that no follow-up training was budgeted or planned for after the initial three-day training. Implementers were concerned about the up-take (and sustainability) of nutrition and sanitation practices considering the short duration of the training. Furthermore, FGDs with groups that received nutrition trainings from Avansa Ag (three nutrition groups and three farmer groups) revealed that participants had not changed behavior as a result of information shared on nutrition and sanitation, though groups did report learning about the topics. For example, most women FGD respondents who were asked specific nutrition and sanitation questions by the ET37 did not report a) change in hand washing behavior before preparing meals, b) increase in vegetables used in porridge, and/or c) increase in consumption of vegetables, eggs and fruits for pregnant and lactating women/mothers. When asked whether there are foods that lactating mothers should not eat, a recurrent answer was “all fruits and eggs”, a common belief in Timor-Leste and one the trainings tried to dispel. Women with 6-23 month old children were shown pictures of porridge prepared with a) the appropriate amount of water and b) vegetables, and all mothers and fathers of 6-23 month old children selected the watery porridge photo (which implies a lower nutrient consumption per volume of food ingested) and the picture with no vegetables as the one that most resembled the porridge they prepared for their children on a daily basis. Participants noted that they did not change their behavior as a result of this training due to various things including the following: a) the training was conducted only once with no follow up support provided from the trainers, b) the participants do not always have access to nutritious food to cook with (as recommended in the training – for example for lactating mothers), and c) participants and their communities continue to follow cultural norms in lieu of recommendations included in the training. After signing a MOU with the Ministry of Education in year 2, Avansa Ag launched the School Nutrition Garden Competition in year 3 (January – March 2017 quarter). The intervention was developed by HIAM Health and the Borlaug Institute, and is comprised of a School Nutrition Gardening Competition Rubric, Session Plan (including eight modules) and Competition Guidelines. Avansa Ag starts engagement with identified schools through meetings and information sessions, followed by detailed training for the school principal and three teachers and socialization events with the community (including police, parents, and leaders). A total of 25 students, ranging in age from 12 – 15, are involved in each competition. At the time of the MTE, the competition had been socialized in five schools and had started implementation of modules in three schools. 37 Nutrition and sanitation questions were asked to women that had attended the nutrition trainings. The ET used a variety of methods to identify whether behavior change had occurred, including secret voting about recent sanitation practices followed (women could secretly identify whether they had/had not washed their hands, for example); and review of pictures of prepared food. 22 According to interviews with Avansa Ag staff the school gardens promoted by the project link to the government school feeding program by increasing the availability of vegetables for children and families in target communities (contributing to set indicators for this output). However, the ET found no evidence of specific plans to link the production of vegetables from school gardens to school feeding programs for students in interviews or in the document review. Avansa Ag staff confirmed that the School Garden Competition is meant to be “a laboratory for horticulture learning purposes, health/hygiene and nutrition practices that include cooking demonstration, food preparation, etc. Also, the students can learn the value of nutrition after consuming the different type of food groups.” Furthermore, a School Director from a school implementing the competition noted that the students could sell the produce to the school feeding program, but the quantity would most likely be too small to contribute. Project staff explained that they have a commitment to improving the nutrition intervention. They recognize that the approach of targeting nutrition groups (as opposed to already established Avansa Ag farmer groups) was pursued to achieve indicator targets, but is not probably resulting in the desired impact. The project has plans to train existing farmer groups in year 3 on nutrition and sanitation in the hopes that working with groups already established and already committed to the project will improve results. Avansa Ag management believes that this approach has a higher likelihood of achieving desired results since groups are already committed and have a production of vegetables that can be used to feed the household. However, there are concerns among Avansa Ag management that this approach runs the risk of leaving out the poorest communities originally targeted for nutrition training in the project’s initial years, because members of the farmer groups are not necessarily the poorest members of communities. Output 8: Improved Skills to Utilize Productive Assets for Economic Activity The output largely targets women and includes activities like the following: facilitate gender networking, business management, financial literacy, household financial management, and budgeting; promote female entrepreneurship; and expand household savings and financial literacy (activities from the year 2 work plan and annual report). Output 8 is meant to promote the inclusion and integration of women throughout the Avansa Ag outputs 1-8 activities. For example, the project has set targets for women’s involvement in farmer groups, trainings, and activities. In year 2, the target was 20 percent and the project achieved 44 percent women engaged (1,240 of 3,123). By the midterm, the project had established five S&L groups, which are mostly comprised of women (74 women out of 91 members, according to project data).38 Additionally, Avansa Ag has successfully coordinated two conferences 38 At the time of the midterm, S&L groups had only received one training from Avansa Ag. In the January – March quarter of 2017, however, the project has recruited nine field agents (in collaboration with Mercy Corps) to establish and monitor saving and loan clubs utilizing the Mercy Corps methodology. The ET met one of these field agents, but the agent was just beginning his work and could not yet discuss experiences with the project and the intervention. Image 3: Women-only FGD in Aileu 23 targeting women39; engaged with female-owned enterprises; and conducted multiple Women in Business Trainings throughout the targeted municipalities (through grantees). Exactly half of FGDs were comprised of women during MTE data collection, revealing high levels of engagement by women in the project and its activities (confirming project data). Women reported involvement in agriculture activities (watering, planting, harvesting, selling) and Avansa Ag trainings. Women reported being engaged in decision-making processes regarding agriculture production (and were primary ‘savers’ in the S&L groups), and were in some instances leaders of the farmer groups (as the ET found in five of the 16 groups visited). Women did not report challenges in attending Avansa Ag trainings, though they noted that they are more likely to attend if the trainings are on their plots (as they can easily take care of children or arrange childcare for a short period of time). Avansa Ag staff and USAID reported satisfaction with the conferences conducted and with the level of engagement that the project has been able to achieve with women thus far. The project realized they needed to make a more concerted effort to engage women across all outputs. The project has since hired a fulltime output manager (the Gender and Youth Specialist), conducted an internal Gender Assessment, and conducted gender sensitivity training for all staff. Staff did report, however, that additional guidance and ideas regarding integration of gender are needed and welcome. There is not a specific project strategy followed for gender integration; rather, the project has reviewed existing analysis and USAID documents on gender integration and female empowerment, taken advantage of opportunities for engaging with women in the agriculture sector, and attempted to increase staff capacity on gender. They have also added requirements in all grants for grantees to propose gender integration plans. Regarding the S&L groups, one visited by the ET was not currently saving, but the other two groups were. The project did not have data on the use of loans by individuals in the groups, but respondents reported using loans for school fees and the purchase of agricultural materials/inputs. Women in these groups requested additional training and support, as they had only received one training on S&L practices. The ET also asked women to recall the Women in Business Training that many reportedly participated in. Across all municipalities, women struggled to recall the five-day training (by a project grantee), and could only vaguely remember topics covered.40 FGD respondents frequently confused the Women in Business Training with the Business Formation training. The grantee implementing this training confirmed that a one-time training was insufficient to address the needs of women. The ET found the training materials, methodology, and topics to be relevant and useful for women41. However, a lack of follow-up training left most participants without the motivation to make changes in their lives. The project also targets inclusion of youth in Avansa Ag activities. The project has implemented the school nutrition garden competition (discussed in output 7) and encouraged youth involvement in farmer groups (discussed in outputs 3 – 6). The project also has a large intern program (with two interns in each municipal field office), during which interns support the municipal office in monitoring, training, technical assistance for farmers and data collection. Engaging youth through the intern program helps address the stereotypical view of agriculture in Timor-Leste (that it is not lucrative and not a 39 Unleashing Women’s Economic Potential: Women in Agribusiness Timor-Leste Conference and Women in Agribusiness Conference. 40 One group of women were able to recall a few topics included, and reported that they learned what to wear when going to the market to sell produce and how to keep records (basic book keeping). 41 For example, the training was conducted using a participatory method and was conducted only five hours per day to ensure women could return home for their household responsibilities. They also used experiential learning methods, and implemented pre- and post-tests. 24 profession for the next generation) and provides more local-level knowledge on agriculture practices (for example, as extension workers). The project sees this as a sustainable approach to staffing, considering it builds local capacity while also, importantly, staffing their municipal offices. Interns the ET spoke with had been trained on various agricultural practices and techniques and helped conduct monitoring and data collection with farmer groups. These youths have a six-month contract. Conclusions In conclusion, the ET found more evidence of progress in sub-purpose 1 than in sub-purpose 2. Despite the fact that the project launched 24 months before the MTE, FGDs with farmer groups and interviews with key informants indicate positive changes in the horticulture value chain at the farmer level through increased use of new production techniques, number of links to buyers/markets, and access to inputs (outputs 3-6). However, these changes were not experienced by all groups; access to markets (buyers), inputs, and credit remain constraints for many Avansa Ag groups. Additionally, while groups have shown use of improved production techniques, many have not yet utilized this new information and are requesting additional follow-up and support. The project’s policy work (output 1) is aligned with farmer￾level activities, in that it targets key constraints in the sector. The slow pace of progress in policy reform, however, suggests that a target of an additional 11 policies before project close is ambitious. The NRM approach (output 2) has been tested and adjusted and has resulted in trees planted in scattered areas. It is questionable whether this approach can lead to increased community resilience (which can only be measured through specific tools that go beyond the scope of this evaluation). The ET identified steps taken toward the goal of increasing the ability of Timorese citizens (men and women) to engage in economic activities, including engagement of women and youth in conferences, trainings, and activities; and training citizens on the basics of nutrition and sanitation. No substantial changes were detected by the ET at the time of the MTE, however, in terms of beneficiary nutrition and sanitation practices. For the school nutrition garden competition, at the midterm there was no clarity from the project about how the intervention would contribute to the sub-purpose outputs and indicators. The project has achieved impressive numbers for the involvement of women. However, women beneficiaries could not confirm the topics of the trainings they received (particularly related to the Women in Business training). The employment of a full-time gender and youth specialist, the development of the internal gender assessment and show that the project has demonstrated a capacity to learn and adapt based on results from activities in year 1. However, without clarity on the gender strategy/approach for the entire project (in the agriculture sector), it will be difficult to ensure recently added requirements for grantees to ‘incorporate gender’ contribute toward ultimate project goals, in addition to ensuring women and men experience outcomes of the project equally (discussed in EQ 2). At this stage of the project, a key challenge in both sub-purposes exists in terms of delivery of technical assistance and training at the farmer level (delivering effective training and appropriate follow-up support). Without well-delivered follow-up support, it will be difficult for the project to ultimately achieve these two sub-purposes and the ultimate project goal. However, through the project’s actions in several outputs thus far (particularly through learning and resulting adjustment of the NRM activity, and through their steps taken to learn more about effectiveness in output 7 and 8), they have shown their commitment to learning and adapting to most effectively and inclusively achieve these targeted results. This attribute bodes well for the likelihood of increasing progress in both of these sub-purposes in the concluding years of the project. EVALUATION QUESTION 2 25 Were achievements toward Avansa Ag purposes varied by geography, community type, or gender and vulnerable/marginalized groups? Findings Geography The ET identified various ways in which project results/achievements varied by geography at the MTE. First, the project has supported/developed a different number of groups in each municipality. Based on data shared with the ET by the project, Table 4 presents total groups by municipality and type. Table 4: Groups by Municipality and Type Municipality Farmer (including Strawberry and S&L) NRM Nutrition TOTAL Dili 4 1 - 5 Aileu 17 2 3 22 Ainaro 33 16 1 50 Bobonaro 14 1 - 19 Ermera 12 7 4 20 TOTAL 80 27 8 115 The project has supported more groups in Ainaro relative to other municipalities, and far fewer groups in Dili relative to other municipalities. As explained in EQ 1, the project also has invested more resources in supporting farmer groups (including strawberry and S&L groups) relative to NRM and Nutrition groups at the MTE. Despite these differences, the project utilizes a similar approach to staffing and training delivery in each municipality (further discussed in EQ 4). Second, Table 5 shows main geographic differences observed in farmer group characteristics related to sales. The table only includes the farmer groups that have reported sales to the project at least once. Farmer groups supported by the project that reported zero or no sales42 are not included, to avoid skewing the data. Data in the table also does not include dry products such as candlenut, copra or konjac (for which sales data per farmer group were not made available to the ET). The table shows that in 2016, Aileu received 67 percent of the total money flow generated by sales from Avansa Ag farmer groups. The table also shows changes in time; when the first three months of 2017 are added in the table, Aileu still receives a majority of the money flow (48%) relative to other municipalities; however Bobonaro increased in percentage sales by more than 15 percent (to 30 percent when the first quarter of 2017 is included). The final column presents data per farmer group, and both the Aileu and Bobonaro groups remain the most productive relative to other municipalities. In 2016, Aileu had the highest average sales per group 42 These groups may have been recently established, yet to harvest, or yet to report their production sales to the project. Image 4: Woman Farmer from Women-only FGD in Aileu 26 ($4,896), however Bobonaro had the highest sales per group as of March 2017 ($3,850). Table 5: Farmer Group Performance by Municipality January 2016 - December 2016 Municipality Total sales (linked & local market) - $US Perc. Sales from each municipality Sales to linked markets - $US Perc. linked vs local market N. of groups with registered sales Average sales per group - $US Aileu 39,164 67% 39,164 100% 8 4,896 Ainaro 7,338 13% 1,134 15% 9 815 Bobonaro 9,203 16% 9,203 100% 3 3,068 Ermera 2,708 5% 2,150 79% 4 677 Dili 202 0% 202 100% 2 101 Total 58,615 100% 51,854 88% 26 2,254 January 2016 - March 2017 Municipality Total sales (linked & local market) - $US Perc. Sales from each municipality Sales to linked markets - $US Perc. linked vs local market N. of groups with registered sales Average sales per group - $US Aileu 48,892 48% 48,892 100% 18 2,716 Ainaro 14,230 14% 6,255 44% 12 1,186 Bobonaro 30,833 30% 20,520 67% 8 3,854 Ermera 7,366 7% 5,456 74% 7 1,052 Dili 984 1% 202 21% 2 492 Total 102,305 100% 81,325 79% 47 2,177 Source: ET calculations on data collected by Avansa Ag The high percentage of total project sales generated in Aileu is due both to the higher number of groups established/engaged by Avansa Ag in that municipality and to the higher sales of crops sold by the groups in that area. Aileu as a municipality also has a longer history of horticulture relative to other municipalities, as reported by project staff and previous DAC documents. This has led to an overall more productive municipality in comparison to others. Another factor influencing these geographic differences in sales includes the willingness of supermarkets to purchase from a given municipality. This decision is often influenced by road conditions, crops produced, and distance from Dili, as reported by four supermarkets interviewed by the ET. Elevation and soil type also dictate suitable crops in each geographic location. The lowest sales per group are in Dili, Ainaro and Ermera municipalities. Project data for groups in these areas show that in Ermera, groups are on average smaller (9.3 farmers against an average of 12 for all municipalities). Dili presents the lowest sales per group. One explanation for this was given by Avansa Ag staff, explaining that group members in Dili have other aspirations besides farming, and they have a more urban style of life. Their commitment to vegetable production might be lower than in other project intervention areas because they have other means of earning an income. Another explanation is that in Dili group members conduct more sales individually. Therefore, recordkeeping is not conducted by the group with the result that group sales are underreported. The closeness of group members in the Dili municipality to the country capital makes access to retailers easier.43 43 MDF (2013) Sector Assessment Report and Sector Growth Strategy for Agribusiness, Processing and Rural Distribution. 27 Ermera is well known for its coffee production. Here, famers tend to focus more on coffee than on vegetable production. In this regard, the Avansa Ag baseline study reports that 50 percent of the coffee farmers of the five municipalities where the project is implemented are in Ermera. In Ainaro, the percentage of sales to linked markets is only 44 percent. FGDs and interviews with Avansa Ag staff revealed that in Hato-Udo (which is part of Ainaro), there are no supermarkets coming to buy vegetables. This was confirmed during KIIs with supermarkets who explained that in Hato-Udo, they cannot find the vegetables they need in sufficient quantity to cover transport costs, and for which they have an unsatisfied demand. Going to Hato-Udo from Dili involves substantial transport costs both because of distance and because of poor road conditions. In Hato-Udo one FGD reported that vegetable sale prices reduced since Avansa Ag started and stated that this was due to the increase in the number of farmer groups in the area coupled with the lack of market outlets. As an example, the group reported that the current sale price of a Chinese cabbage is $0.5, while before Avansa the sale price was $1.5. One of the member of the group stated “Avansa Ag is coming to destroy our life because we have a very low income from vegetables sales compared to before Avansa came”. The ET cannot confirm whether this price reduction was caused by the project (such an analysis would require the use of economic price models that go beyond the scope of this MTE); however an alternative explanation was provided by Avansa Ag field staff, who explained that new local traders started to sell vegetables at a low price in Hato-Udo, which are bought in a nearby area (Maubisse). The project collects sex-disaggregated data, but it does not disaggregate indicator achievements routinely by group type or geography. The project, reasonably, focuses on reporting according to indicator and Performance Indicator Reference Sheet (PIRS) requirements. However, they are missing information that disaggregation of indicator data can provide regarding how different beneficiaries experience project results/outcomes. For example, the ET calculated disaggregation by geography for indicators 3.3 (Figure 3), 4.1, 5.1 (Figure 4), and 6.2 and for the average production and sales per farmer group (see Table 5 above). Although not required in the PIRS, this disaggregation helped the ET understand how achievements varied across group types and locations. Market Development Facility – Timor-Leste. 28 Figure 3: Indicator 3.3 - Number of individuals who have received USG-supported short-term agricultural sector productivity Source: AMEP data as of March 2017 Figure 4: Indicator 5.1 - Number of farmer groups and associations with buyer agreements Source: AMEP data as of March 2017 Group Type There are various types of groups in Avansa Ag, namely farmer, strawberry, S&L, nutrition, and NRM. Among those types, there are demonstration and non-demonstration groups, and groups established by Avansa Ag or previously established by other projects (e.g. DAC). Through document review and FGDs and KIIs, the ET noted that farmer groups, demonstration groups, and DAC groups were more consolidated and productive. In FGDs, farmer groups relative to NRM and nutrition groups received 0 100 200 300 400 500 600 700 800 900 1000 Aileu Ainaro Bobonaro Ermera Dili 0 5 10 15 20 25 Aileu Ainaro Bobonaro Ermera Dili 29 more trainings and more Avansa Ag support at the time of the MTE (via visits by municipal staff and materials). As noted in EQ 1, almost all of these groups (15 out of 16) reported some kind of change resulting from Avansa Ag trainings and support (though the changes varied by farmer group). In comparison, half the nutrition and NRM groups (three out of six) reported utilizing a new production technique, and an additional group reported some ‘other’ change as a result of Avansa Ag support (production of seedlings and use of a new crop – for example, mahogany). All visited nutrition groups (3) reported one-time trainings and infrequent or no follow-up visits by Avansa. A key informant also shared that some nutrition groups are currently inactive. The ET visited both demonstration and non-demonstration plots and observed substantial differences between the groups in terms of materials and follow-up support provided by Avansa Ag, in addition to productivity of the farms. In the demonstration plot in Maliana, Bobonaro, for example, the group reported receiving a visit from Avansa Ag staff once a day since January 2017, and receiving most of the materials for the plot free of charge. A farmer group established at roughly the same time, and not more than five kilometers away, reported no participation in trainings and receipt of only plastic to cover a nursery they had developed. The ET notes that Avansa Ag implements with a different approach in demonstration vs non-demonstration plots. However, in the instance of Bobonaro, one group that was formed at a similar time and with similar capacities was largely ignored. The ET visited five groups (out of 22) that were formed under DAC. These groups received fewer trainings from Avansa Ag because they had already received similar trainings under DAC. However, the groups were well connected to markets and had regular vegetable production. Three of the five groups reported an existing link with a supermarket established before Avansa Ag, while the other two have already established a link with a supermarket since Avansa Ag started. Four of the five groups had established input suppliers, two had members who borrowed from a bank or an MFI, and two other groups had an internal saving and loan scheme. All DAC established groups were using improved production techniques (either attributable to DAC or Avansa Ag). Gender In EQ 1, the ET described the project’s main achievements as related to inclusion of women, women’s empowerment, and gender integration into project activities. Based on project data, the project has met their target for the percentage of women involved in project activities as of year 2.44 The project has also worked to involve as many women as possible in trainings, farmer groups, and conferences/activities, reconsidering and adjusting their strategy for seeking women’s participation in year 2 and 3. The project has supported women in leadership roles. For example, the project has promoted women as leaders of groups (five of the groups the ET visited were led by women) and has engaged with women-owned enterprises. In these ways, the project has taken efforts to ensure equal access of men and women to project benefits (participation). Considering changes in groups at the farmer level (such as use of new production techniques) have only recently been identified at the farmer level (and only in a portion of sampled groups), the ET could not determine if benefits from the project (beyond access) were evenly experienced by men and women 44 According to project tracked indicators: 40 percent of the individuals that received training by the time of the midterm were women (1240 of 3123, indicator 3.3). For indicator 6.4 (safe chemical use), 34 percent of training beneficiaries at the time of the MTE were women (68 of 202). For indicator 8.3, 41 percent of beneficiaries at the time of the evaluation were women (279 of 681). 30 (and boys and girls).45 The ET noted that improvements in drip irrigation and links to buyers (through MOUs or formal agreements that bring supermarkets directly to pre-established aggregation points at the village level) save women time. However, women also reported increased levels of work since Avansa Ag started considering their involvement in trainings and their increased engagement on their plots/in their gardens (though they were satisfied with this increased work considering it led to increased incomes). The ET also noted that men (and boys) were not considered primary targets of S&L and nutrition and sanitation trainings. More specifically, in interviews with Avansa Ag staff, respondents did not identify men (or boys) as critical recipients of the nutrition intervention. They continually emphasized that the most important beneficiaries of the trainings were women because in their perspective, women and children suffer from poorer nutrition relative to men. The project is now planning to deliver nutrition trainings to farmer groups instead of establishing individual nutrition groups. Given that the majority of farmer group members are men (62 percent)46, the number of male beneficiaries of the nutrition training will increase. Conclusions Avansa Ag achievements were found to vary by geography. The project developed and/or supported more groups in certain municipalities (like Ainaro) relative to others. Groups in Aileu and Bobonaro were more productive relative to other municipalities. Hato-Udo (in Ainaro) appears to be characterized by a low-level equilibrium trap, meaning that supermarkets do not come to buy the vegetables they need (because they are not produced in a sufficient quantity to cover transport costs) and farmers do not produce more profitable vegetables because supermarkets are not coming to buy vegetables. These differences can be explained in part by considering distance from Dili and markets, willingness of supermarkets to buy from certain municipalities, road conditions, and soil type and elevation. Results also varied by group type, with farmer groups, DAC groups, and demonstration groups (or plots) appearing more consolidated at the time of the MTE relative to other groups. The variation was due, in part, to the level of Avansa Ag staff support experienced by these groups (with farmer and demonstration groups receiving more in-person support from Avansa Ag field staff). The variation was also due to the length of support they had received from USAID (with DAC groups appearing more consolidated due to longer engagement over multiple projects). Some level of variation across project results is reasonable and inevitable, and this analysis does not imply that the project should seek equal results across all geographies or group types. Some of the variation noted in the findings above, however, should be noted for consideration in future project planning (with adequate amount of Avansa Ag staff support) in each municipality based on their unique contexts and needs. While at the midterm, women’s had equal access to inputs and activities from the project, the ET cannot say at this stage whether this has resulted (or will result) in similar outcomes for women and men (and 45 The ET explored this through FGDs with men and women (combined groups and women-only groups). Participants were asked about changes experienced since Avansa Ag started. Facilitators then encouraged respondents to consider how these changes impacted their day-to-day lives. Men and women had some unique answers, noted in this section, but most responses across genders were the same. This is most likely because the changes have only recently occurred/been experienced. It may be too early to assess if equal access has resulted in equal outcomes across men and women. 46 641 male members out of 1,028 31 girls and boys). It is a critical time for the project to begin collecting information on this potential variation and analyzing how achievements varied by gender at the project midpoint to ensure equal outcomes for men and women (and for groups) at project close. The project will increase the number of male beneficiaries of nutrition and sanitation trainings since these trainings will be provided to farmer groups in the future. This is positive, as the fact that men have not been considered a primary target of nutrition and sanitation trainings potentially threatens the ultimate goal of behavioral change. True behavior change in women and increased nutrition indicators for women (let alone children and the family in general) might be compromised if men are not equally engaged in nutrition and sanitation initiatives. This is because they continue to hold positions of authority in the household and over household decisions (for example, over food consumption and distribution of food).. EVALUATION QUESTION 3 To what extent are the activity’s purposes (including outputs/indicators and targets) still valid for the current development context in Timor-Leste? Findings To address this question, the ET explored the alignment of the project (its design) with GOTL Plans, the USAID Timor-Leste Country Development Cooperation Strategy (CDCS), and with the current development context (including needs of beneficiaries). The ET explored these areas through document review (for the two former areas) and interviews (for the latter). GOTL Plans The project design aligns with the GOTL Action Plan (NAP) for a Hunger and Malnutrition Free Timor-Leste (2014)47. More specifically, with its focus on income generation and food production, the project intervention logic is coherent with outcome 1.1 (Poor and marginalized people have more diverse and more secure entitlements to food) of the First Pillar of the Action Plan and with outcome 4.1 (Productivity of various agricultural products of smallholder farmers including youth and women farmers increased), which also specifies fruit and vegetable production as key elements. The credit development and market linkage component of Avansa Ag is consistent with the outputs under outcome 4.2 of the Action Plan, which includes access to financial services such as saving and credit and enhancement of access to markets. The nutrition component of the project is consistent with outcome 2.1 (Improved nutrient intake - macro and micronutrients - for mothers, children under five and adolescent girls), outcome 2.3 (Improved nutritious food utilization at household, community and national levels), outcome 2.4 (Improved access to water and sanitation facilities and hygiene intervention, which also specifies hand washing among outputs), and outcome 2.5 (Families’ nutrition behavior and practices 47 GOTL (2014) Zero Hunger Challenge. National Action Plan for a Hunger and Malnutrition Free Timor-Leste. Image 5: Seedlings at NRM Group in Bonbonaro 32 improved) of the Second Pillar. The DRR component of the project is also reflected under the outputs of the outcome 3.2 (Sustainable and climate resilient agriculture practiced) of the Action Plan. Furthermore, the development of the pesticide law and manual, the attempts of the project to promote quality inputs, and the actions aimed at linking farmer groups to supermarkets are consistent with two sub-programs of the GOTL Medium-Term Operation Plan (2014-2018) for the Agricultural Sector of MAF.48 Lastly, one of the goals of the Timor-Leste Strategic Development Plan 2011-2013 (SDP) is import substitution of at least 50 percent of fruits and vegetables by 2020. The stated Avansa Ag project strategy on import substitution makes the intervention particularly relevant for the country SDP. USAID CDCS Given the project focus on smallholders and poor farmers, the project is well aligned with the Inclusive Economic Growth intermediate result (IR) of the USAID/Timor-Leste CDCS and its two sub-IRs. In fact, the first sub-IR addresses nutrition-sensitive agriculture, while the second sub-IR is related to the increased productivity of selected agricultural value chains. Timor-Leste Development Context In the Avansa Ag project AMEP, there are listed assumptions49, risks50 and supporting pre-conditions necessary to achieve the project’s goal. As of May 2015, the project had reviewed the assumptions included in the RFP and updated them to reflect potential risk impact at the launch of the project.51 According to the assumptions noted in the May 2015 version, those related to the political context and the market (the policy enabling environment and market dynamics) appear to have remained at a low risk probability. This was confirmed by respondents both from within the project and outside the project (including the following: security and political situation remains stable; policy and enabling environmental remain the same or improve; and no major crop disease outbreaks)52. The project SoW includes a section titled “Theory of Change” (TOC) (which the project later referred to as a “Logframe”). This section is short (less than two pages) and does not cover all causal pathways and components addressed by the project. As noted in the AMEP, the project proposed to develop “a big-picture view of the project, encompassing all of the assumptions, risks and supporting pre-conditions necessary to achieve the project’s vision” in year 2. That TOC was to “complement the current Logframe model developed from the RFP” (see Image 1).53 Unlike a logframe, a TOC is not linear but includes feedback loops and iterations; a TOC describes “how” and “why” changes occur (if they do, indeed, occur). Areas that are missing from the logframe include NRM and DRR, for example. There is also not a clear explanation of how a farmer group achieves sustainability (further discussed in EQ 5). Without an updated TOC, it is difficult to assess the relevance of indicators and targets for either sub￾purpose to the development context (beyond commenting on assumptions, as done above, and documenting progress thus far – see Evaluation Question 1). 48 Sub-program 2.2 (Promotion of Quality Inputs), Sub-program 2.5 (marketing and support for farmer groups) 49 Assumptions can be defined as the necessary positive conditions that allow for a successful cause-and-effect relationship between the different levels of results. 50 Risk corresponds to a potential future event, fully or partially beyond our control that may (negatively) affect the achievement of objectives. 51 In the 2015 AMEP, the project noted that a Theory of Change including updated risks and assumptions would be developed in year 2. This has not yet occurred at the time of the midterm. See the following paragraph for more details. 52 Other risk/assumption areas that the ET did not have time to explore in-depth include: project-level; nutrition and sanitation; climate change; agriculture; NRM; and socio-economic. In general, assumptions were not found by the ET to have increased levels of risk than those identified in May 2015. 53 Avansa Ag AMEP, Section 4.1. 33 Interviews with other donors working in the agriculture sector and MAF confirmed that the Avansa Ag project has identified and is seeking to address the key constraints that exist in the sector (from the policy to the farmer level). These constraints include inadequate national legislation regulating the sector, inadequate input markets, lack of farmer knowledge on production techniques, etc. These same respondents, however, together with GOTL and project respondents, were concerned about the targets imposed on the project considering the multi-faceted (and multi-sectoral) constraints that exist in the agriculture context. The project has targeted 33,000 people in 250 communities, and has 33 indicators (including federal and custom indicators). Based on perspectives of stakeholders from within and outside the project, there was consensus that the project addressed all key horticulture value chain steps and constraints, but is too ambitious (in terms of targets) and complex (in terms of sectors and outputs included). Multiple project and non-project respondents noted that when a project is asked to achieve high beneficiary targets, project resources begin to shift to counting and reporting as opposed to implementing a quality intervention. The ET observed this was the case with multiple project staff at the Dili office and in the field offices.54 This is complicated further when additional sectors are added to the intervention; in this case, the project also implements a nutrition and NRM component alongside the horticulture value chain activities. While these were seen as potentially complimentary by respondents (and by the FtF Initiative), it seemed unreasonable to these same respondents to include additional sectors given the project duration, the significant number of constraints facing agriculture development in the country and available resources. For example, based on the budget, there are 26-times less financial resources per beneficiary in comparison to DAC.55 FGDs with project beneficiaries also confirmed that the project is relevant to the needs of these groups. More specifically, farmers discussed the main obstacles they face in the agriculture sector, and then were asked how the project is working with them to address these issues. With the exclusion of one farmer group, all other farmer groups (15) reported that the project was conducting activities that were relevant to their current needs. The farmer group that, when prompted, complained about the relevance of the project was based in Hato-Udo and reported that if Avansa Ag does not link the group to new buyers, no improvement of incomes are possible. The group was, in fact, already producing vegetables before Avansa Ag but experienced a clear problem of market outlet access. Furthermore, KIIs with supermarkets, experts working for MAF, and other donors confirmed that the focus of the project on fresh produce import substitution remains valid, considering exporting vegetables is believed unrealistic within the short project timeframe (by these respondents). The project is also promoting processing of agricultural products and export of dry products (e.g. candlenut) by supporting small enterprises through grants (funding the delivery of equipment). According to interviews with Avansa Ag staff, supporting small processing businesses in rural areas allows the project to increase incomes for upstream producers (that is, by creating the “backward linkages56”). Avansa Ag staff also reported that grants are used to tailor interventions to the needs of small businesses, thus contributing to the development of processed products and/or horticultural value chains. Examples include grants for passion fruit juice production, coconut oil production, candlenut de￾shelling, and the establishment of a small cold room. The grant component of the project is only 3.5 54 Multiple staff reported anxiety over achieving targets, and noted that they were not clear how the target will be achieved (the “how” typically detailed in a TOC). 55 $15,000 per beneficiary in DAC vs $580 per beneficiary in Avansa Ag, including fixed fees. 56 Backward linkages are demand-side connections a company has with upstream producers 34 percent of the total budget. According Avansa Ag staff, this limits the potential impact of grants to develop processing activities with backward linkages by tailoring interventions to the needs of small enterprises. Conclusions The Avansa Ag approach is still valid for the current development context in Timor-Leste. The project is found to be well aligned with GOTL policies, USAID strategy in Timor-Leste, the current context, and beneficiary needs. However, the scope (sectors included) and scale (targets) of the project may not be taking into account the complexity and constraints that exist in the Timor-Leste development context. While it is difficult to comment on the relevancy of indicators and targets without a finalized TOC, respondents felt strongly that resources were being diverted to counting and reporting away from effective (and sustainable) implementation of the project approach. The project has appropriately focused on import substitution and export of dry and processed products, and has been promoting income generating opportunities in rural areas through the support of small processing businesses. The funding available to support small businesses, however, is limited. EVALUATION QUESTION 4 To what extent has the overall management of Avansa Ag (including management from the steering committee, USAID, and Cardno) contributed to/inhibited project progress? What opportunities, if any, exist for improvement in management of the activity?57 Findings Leadership The project was characterized by a slow start-up phase as evidenced by the fact that during the first year of the project, only 61 percent of planned financial resources were spent by Cardno. The project staff reported challenges with the slow start-up, and reported that the initial implementation of outputs was conducted in a “siloed” manner (each output existing in their own activities and objectives). USAID and project respondents attributed these issues mostly to leadership in the project. This slow start-up initially frustrated MAF partners (who reported requesting more information on the project strategy intervention, and achievements, in year 1), but at the MTE, MAF reported positive and productive communication with the project. They did not report any skepticism about the project approach or achievements. Also, at the midterm coordination was reported as positive and sufficient by USAID and MAF. In addition, since the change to the current Chief of Party, the project has increased expenditure rates (reaching 91 percent of planned resources during the first 11 months of the second year) and clarified project interventions. The project steering committee (PSC) meetings are reported by participants to be mainly used as information sharing events about the project. Overall the PSC was not found to play a role in guiding the project and its implementation (therefore, not contributing to or inhibiting project progress). The PSC has an advisory role and is not tasked to approve project plans or annual reports. Staffing During interviews with Avansa Ag staff, they reported that the project is understaffed, especially at the municipal level, with respect to the number of groups supported and the variety of activities 57 The ET will address the first question in this section and the second question in the recommendations section below. 35 implemented. The project currently has 25 field58 staff (including interns), of which two are women. If the total number of groups is divided by the number of field staff, the resulting ratio is 4.6 (compared to 1.1 in DAC). Moreover, the distribution of field staff per municipality is not proportional to the number of groups in each municipality. For instance, Ainaro has 50 groups (of which 33 are farmer groups) and Aileu has 22 groups (including 17 farmer groups), while the number of field staff is the same in both municipalities (six). In Ermera, the field staff do not live in the municipality and commute from Dili. This reduces the time available for field activities and support visits to farmers (as reported by the field staff). Groups that only received infrequent data collection and monitoring visits from Avansa Ag staff (as opposed to training and follow-up support) reported dissatisfaction with the project. Field staff noted that they only conduct support visits regularly for farmer groups (varying from twice a week to once a month, depending on the municipality and other competing priorities). They reported that they did not conduct follow-up and monitoring for topics unrelated to farmer group interventions. While they visit other groups to collect group profile data for the M&E system (NRM, nutrition, S&L groups), they are not trained or tasked specifically on conducting monitoring or follow-up on project-implemented trainings. This is supposedly tasked to the trainers (sub-contractors or grantees); however four sub-contractors and grantees interviewed by the ET reported no budget for follow-up and post-training for trainings related to health, women’s leadership, NRM, savings and loan. The project has made use of interns to increase the number of field staff and reported plans at the midterm to use more interns in the coming months. This was justified by Avansa Ag management by the fact that interns will be probably retained as staff after the internship period, along with the necessity to integrate more youth and train a wider set of Timor￾Leste citizens in agriculture practices (see EQ 1). However, respondents noted frustration with the turnover of interns and their lack of training and knowledge. Avansa Ag staff also expressed that it is challenging to train and re-train interns every six months when they needed to also keep up with demands of Avansa Ag groups. Farmer group participants expressed similar frustration with Avansa Ag staff (primarily intern) turnover. Staffing concerns at the midterm must also be considered in light of the substantial changes happening in the project in year 3 – the project has just recently launched their grants program which will result in more monitoring, spot check, and administration requirements for staff at all levels. Internal Project Coordination Internal project coordination involves communication and information sharing regarding project planning and data. At the MTE, communication between Avansa Ag Dili-based staff and field staff was reported to work properly for famer group-related activities. However, field staff reported limited to no 58 Referring to staff that work in municipal offices and support direct beneficiaries (groups), in Dili (3 staff), Ermera (5 staff), Bobonaro (5 staff), Ainaro (6 staff), and Aileu (6 staff). Image 6: A purchase from Farmer Group in Maliana by a supermarket 36 involvement in coordination, implementation, and monitoring of activities for the nutrition and S&L components. For example, the field staff knew activity plans for outputs 3-6 (farmer groups), but were not aware of plans for NRM groups, nutrition groups, women’s trainings, and savings and loan groups. As noted above, the field staff reported confidence in reporting updates to Dili regarding farmer groups (including M&E data), but had limited time and resources to provide follow-up and monitoring to other groups. In Maliana (Bobonaro), for example, various non-farmer groups formed over six months ago had not been visited by field staff. Overall, field staff reported no involvement in project planning and instead reported back to Dili about activities completed and data collected. The M&E system utilizes several forms and sheets to track a total of 33 indicators that the field staff were familiar with, though these were largely related to farmer groups (with the exception of the Group Profile). The M&E system has improved in recent months and now tracks groups and beneficiaries more accurately. However, the system was found to include some inaccuracies at the midterm. The ET sought to verify trainings received and basic group profile data during each FGD; while some differences in reporting by groups is to be expected due to recalling difficulties, substantial variations in several instances existed (with groups reporting limited to no assistance received when M&E system data reported multiple trainings received). In relation to AMEP indicators, indicator 2.1 (the number of people implementing risk reduction practices as a result of USG assistance) and indicator 3.1 (number of farmers who have applied improved technologies or management practices as a result of USG assistance) are calculated by aggregating data collection forms recently improved by Avansa Ag. The form now better reflects the technologies and practices introduced by the project in comparison with the previous form (that listed techniques not addressed by the project, such as hybrid varieties of rice and maize or coffee pruning). The form still, however, does not clearly distinguish whether a particular management practice or technology has been utilized as a result of USG assistance, as the definition of the indicator requires. While the introduction of some technologies can be safely attributed to the project (e.g. low tunnels which are not promoted by other organizations), others cannot (e.g. compost).59 Overall, the M&E system is considered by Avansa Ag a very complicated and with too many indicators. The ET found that the project uses indicator and M&E data for accountability purposes rather than as inputs to adaptive management (as noted in EQ 3). As noted by project respondents, strategic decisions are more influenced by work plans (based on initial project targets) than by monitored indicators and data reported from the municipalities. As an example, the project identified new groups to receive nutrition and sanitation training as opposed to working with existing farmer groups in order to contribute to the project beneficiary target; however, project staff noted that at the time and presently they believed this was a less than ideal approach considering working with existing farmer groups may have increased the likelihood of information uptake and learning. Currently, Avansa Ag management is revising this approach with the purpose of integrating nutrition groups with farmer groups. Conclusions The management of the project has improved since the first year as shown by the increase in the expenditure rates, improved coordination with stakeholders, and clarified interventions, which have positively contributed to the progress and achievements. However, issues related to the staffing and 59 Other M&E data that the ET identified as requiring further review were indicator 3.2 and indicator 5.5. The ET also previously commented in Evaluation Question 1 about indicators related to output 7 (7.1 – 7.4). Considering the mandate of Avansa M&E to provide oversight and quality assurance over these indicators, the ET will leave further comments on indicator data quality to Avansa M&E reporting (for example, see recent Performance Data Review). 37 internal project coordination have resulted in uneven application of the Avansa Ag project approach and some farmers’ lack of satisfaction with Avansa Ag. It has also resulted in a lack of follow-up with groups (particularly on nutrition and savings and loan interventions). The high number of indicators measured by the project makes monitoring a complex task involving multiple staff, without serving many adaptive management purposes. The project would like to utilize a more adaptive approach (and has shown the capacity to do so in the past, in several key instances noted in EQ 1); however, the high targets and number of indicators skew project resources toward tracking and away from informed implementation. EVALUATION QUESTION 5 To what extent has Avansa Ag made progress in ensuring sustainability of activities beyond USAID support? Findings The Avansa Ag project currently does not have a formalized, defined sustainability plan. The project design, however, has several aspects that are related to ensuring sustainability. Key informants (including GOTL, Avansa Ag, and other donors) reported that the project design is conducive to sustainability (particularly in sub-purpose 1) since it addresses technical horticulture production aspects, links to markets, links to input suppliers, and access to credit, all of which are reportedly required to achieve a productive farmer group. By addressing these areas, the project is relevant (as noted in EQ 3) and is sustainable if implemented as planned with farmer groups. Farmer groups reported in FGDs that they saw the link to supermarkets as the key to their sustainability in the future; if they had a secure buyer for their produce and knew what was in demand by that buyer, they could remain productive in the future (with or without Avansa Ag support). However, FGD revealed that not all groups are linked to a buyers (5 groups out of 16). Progress toward achieving these aspects critical for sustainability and productive farmer groups, however, is mixed (as reported in EQ 1). Due in part to management of the project (as reported in EQ 4), the project has unevenly implemented the approach with the current 81 farmer groups. Some groups have received multiple trainings and are currently linked with supermarkets, while others have been working with the project for a similar length of time but have participated in only one or two trainings (and have made no progress in accessing formal input or output markets). 38 Other design aspects Avansa Ag staff (senior management and output leaders) highlighted as likely to contribute to sustainability in the project are behavior change campaigns (only being launched in year 3),60 training of private sector extension workers, and avoidance of free provision of expensive materials to farmer groups. Regarding training of private sector extension workers, the project has invited staff from supermarkets (as an example) to Avansa Ag trainings (under output 6). Indicator 6.1 tracks the project’s progress in this area, and in year 2, the project had achieved 62 percent of the five year project target (33 trained; target of 50). When the ET spoke with farmers, however, only one farmer group (out of 16) reported receiving assistance from a supermarket extension worker. The supermarkets, similarly, reported that their staff had little time to provide this type of training and support to farmer groups they worked with. The project has reportedly focused on the training of private sector extension workers in lieu of coordinating and working directly with MAF extension workers at the suco level, as the latter are seen as largely ineffective and lacking the funds necessary to do their jobs with farmer groups. However, MAF extension workers are at times invited to trainings organized by Avansa Ag. Another donor in the agriculture sector, however, has recently conducted an assessment and found these stereotypes to be false. In addition to working with private sector extension workers, in interviews, Avansa Ag staff also saw the use of interns as contributing to this goal. They see the intern program as a way to train “a pool of young agronomists” to support farmers now and in the future. As noted in EQ 1 and 4, the project has trained multiple interns and is using them to support field offices and groups. These interns do, however, complete work with Avansa Ag after 6 months unless they are hired as fulltime Avansa Ag staff. There is a history from the DAC project of supermarkets hiring trained USAID project staff, and Avansa Ag could continue to promote this with supermarkets, particularly in relation to interns. 60 The aim of these campaigns, according to Avansa Ag staff, will be to more holistically address constraints/issues the project is targeting (including issues of under nutrition, sanitation, and agricultural productivity), and will hopefully contribute to sustainability of project results. Image 7: Farmer Field Day in Maliana, Bonbonaro on May 4, 2017 39 The project requests cost-share with farmers to distribute expensive equipment (farmers cover 50 percent of the cost) and limits free distribution of materials to five demo plots and relatively inexpensive inputs (plastic for tunnels, plastic mulch, seeds, manure, yeasts for organic fertilizers). This ensures that the project is not offering free handouts but is rather equipping groups to access and utilize all the tools (inputs, access to credit, etc) they have at their disposal to achieve high productivity. Notably, project staff (both in Dili and in the field offices) were concerned about the sustainability of all groups (the main beneficiaries of the project). Some staff mentioned farmer groups while others mentioned nutrition and NRM groups. This concern is corroborated by the facts presented in Evaluation Question 1 regarding the uptake (albeit limited) of new production techniques and positive changes in groups since starting to work with Avansa Ag. The inhibiting factors to sustainability noted by project respondents included staffing levels and use of one-time trainings. Conclusions While the Avansa Ag project currently does not have a formalized, defined sustainability plan, the project design includes aspects that are related to ensuring sustainability; these include addressing key constraints in the horticulture value chain, behavior change campaigns, training of extension workers, and avoidance of provision of free materials. The project staff showed evidence of thinking through these project aspects and how to utilize them to achieve sustainability. If the design is implemented effectively, sustainability is possible. However, without a defined sustainability plan for groups, the capacity of the project to design and implement sustainable solutions for beneficiary groups is limited. Supermarket extension workers have not served the role of farmer group trainers. The capacity and willingness of supermarkets to have staff play this role was found to be inconsistent at the time of the MTE, which represents a limit for future access to technical assistance for farmers once the project is over. Moreover, access to markets remains a constraint for some of the farmer groups. Finally, the use of one-time trainings has not resulted in gained knowledge or behavior change consistently across groups at the midterm (as noted in EQ 1). Based on project progress at the midterm, the final years of the project require focus on consolidating work with targeted groups to ensure sustainability at the farmer (key beneficiary) level. 40 VI. RECOMMENDATIONS RECOMMENDATIONS FOR USAID 1. USAID/Timor-Leste should consider re-scaling the project There is evidence at the time of the MTE to suggest that there is a risk that project impacts will be diluted and unevenly dispersed because of ambitious project targets and a multi-sectoral focus. USAID should consider re-scoping the project by adopting one or more of the following approaches: 1) removing outputs or components, 2) reducing targets, starting with the number of beneficiaries/communities, and/or 3) lengthening the duration of the project. Any re-scoping should be informed by a clarified TOC and a sustainability plan (mentioned below), which explains how each activity and component contribute to the project goal. More specifically, the ET recommends one (or both) of the following: • Reduce target beneficiaries and communities (particularly the number of targeted groups) to better take into account the realities/constraints of the agriculture sector. This revised target should be based on a developed sustainability plan and TOC that details a) characteristics of ‘sustainable’, ‘productive’ groups, b) project support required to achieve these characteristics, and c) time required for achieving these characteristics (based on point a and b). Following a reduction in project beneficiaries, adjust all indicators based on this beneficiary count in the project AMEP. • Re-consider the NRM approach in light of the ultimate goal of increasing resilience for communities. More specifically, consider removing the NRM output if interventions cannot be implemented coherently with identified climate related risks, resilience-building opportunities (e.g. providing additional source of incomes for communities) and nutrition aspects (e.g. planting moringa, which is used in porridge preparation). 2. USAID/Timor-Leste should consider increasing the budget for grants to allow for a greater number of partnerships with small companies that stimulate demand of agricultural products Grants have been used by the project in different ways, including to allow for partnership with local NGOs and companies/businesses. New businesses have been established by tailoring support to install processing equipment and storage with the ultimate purpose of developing backward linkages through which rural households can increase their incomes. The limited project budget for grants limits the potential impact of this approach. Any increase in grants should be matched by an appropriate increase in human resources to provide necessary administrative support including monitoring and follow-up. RECOMMENDATIONS FOR USAID AND AVANSA AG 3. USAID/Timor-Leste and Avansa Ag should focus on a reduced number of key policies that are critical for the agriculture sector and project goals The project in total targets 20 policies over the life of the project, meaning it still has 11 policies to achieve before project close. At the midterm, while the project has worked on 9 policies, none have been formalized or passed. Given the slow process of policy reform in Timor-Leste, USAID/Timor￾Leste and Avansa Ag should consider focusing on a small number of policies/pieces of legislation for the duration of the project that the project has the potential to impact for the long term. In order to maximize impacts and build ownership among relevant government ministries, USAID/Timor￾Leste and Avansa Ag should identify policies on which they can work together with other donors, that are more likely to be passed into law by the end of the activity, and that have the largest potential effects on the sectors covered by the activity. 41 RECOMMENDATIONS FOR AVANSA AG 4. Avansa Ag should improve the delivery of technical assistance to all groups (regarding farming, nutrition and sanitation, NRM and Savings and Loan) by conducting follow-up trainings (or training series) and support visits (through grantees and/or Avansa Ag field staff). Changes were found for some of the farmer groups in terms of access to markets, inputs and use of new production techniques under sub-purpose 1. However, not all groups experienced changes. There were no changes found in nutrition groups under sub-purpose 2. All groups noted that one￾time trainings were not sufficient to lead to behavior change, whether that change be related to the use of new farming techniques or improved hand washing practices. The project could deliver improved technical assistance at the farmer level by considering the use of a series of trainings (as opposed to one-time trainings), follow-up/support visits post-trainings, and knowledge tests (ex￾posttests) to identify if knowledge is retained and used toward improving practices and changing behavior. Support visits can be conducted by grantees (if they are provided the resources to do so) or Avansa Ag staff (if they are provided the resources and guidance to do so), with the support of Dili-based output leaders. Based on evidence from the MTE, follow-up visits need to be consistent (for example, once a month for six months after the training took place) particularly when they are regarding the implementation of new techniques (farmer and NRM groups) or new practices (nutrition and S&L groups). 5. Avansa Ag should re-consider geographic differences across municipalities (and sucos) in the horticulture value chain at this project midpoint. This should include a consideration of supermarket demand and interest (vegetable quantity and type for each area). If estimated quantity and type of vegetables are not sufficient to attract supermarkets, the project should consider either recommending alternatives to farmer groups or reducing the number of planned farmer groups to be established. The project is aware of geographic differences across the municipalities after 24 months of implementation. The ET, however, identified several instances in which it would be useful for the project to re-visit the exact constraints and opportunities in each area, based on farmer feedback and input. As an example (presented in EQ 2), no supermarkets are currently buying vegetables in Hato-Udo. Transport costs from Hato-Udo to Dili are high because of distance and road conditions; furthermore, supermarkets cannot find sufficient quantities of the vegetables from this area that their customers are demanding. Farmers are currently producing vegetables (under the guidance of Avansa Ag) that they can only sell in the local markets. Additionally, farmers do not produce vegetables that supermarkets are interested in because there are no supermarkets coming to buy. This results in a coordination failure, and presents an opportunity for a project like Avansa Ag. By consulting supermarkets, Avansa Ag should carefully estimate the minimum quantities and types of vegetables to be produced in the area to attract supermarkets, and should coordinate production between farmer groups and supermarkets (such an analysis should be repeated twice per year). If the estimated quantities to be produced to attract supermarkets are not realistic, Avansa Ag should consider a reduction in the planned number of farmer groups in this area and instead pursue other interventions that meet community needs. Establishing more farmer groups producing vegetables without an increase in market outlets can easily result in a price collapse. 6. Avansa Ag should consider utilizing a staffing structure aligned with the sustainability plan for farmer groups (see recommendation 8), which most likely would require a more proportional (group/staff) approach to the distribution of staff. The project should not pursue increases in staff through interns only. 42 In order to achieve consolidated and sustainable farmer (and other) groups, groups need to be supported on a consistent, regular basis. For example, farmers need regular encouragement and support in order to shift production to more profitable crops for which they have no production experience. Nutrition groups need follow-up visits from trained staff (or grantees) who know how to encourage the adoption of practices discussed in trainings. Lack of sufficient Avansa Ag staff at the municipal level has been widely acknowledged as a challenge facing project implementation. Issues of staffing have resulted in an unequal application of technical assistance at the field level among groups. Moreover, in year 3 the volume of activities will substantially increase. Avansa Ag is planning to recruit more interns; however, interns require training and re-training and are less skilled (relative to fulltime municipal staff). With staff in municipal offices more proportional to the number of groups requiring support in that municipality, the project could increase the likelihood of achieving sustainability by project close. These staff may include interns but should not exclusively be made up of interns. The use of interns should be limited to cases in which the project has clear plans (and resources) to integrate them into municipal activities after the internship period (whether as fulltime staff, STTA, etc). 7. Avansa Ag should improve coordination between district and Dili-based staff to improve the implementation of activities, data collection and quality of M&E data Field staff did not appear to be involved in the planning, implementation and monitoring of activities unrelated to farmer groups. This meant that field staff were not visiting or providing support to nutrition and S&L groups beyond collecting basic M&E data. Avansa Ag should conduct municipal coordination meetings that include briefing for field staff on all intervention planned activities and data needs (outputs 2 – 8), particularly to ensure more information sharing on nutrition, NRM and S&L activities. The project should also clarify what role field staff are required to play in support of non-farmer groups (this depends on how the project allocates responsibility for follow-up with groups – see recommendation 4). 8. Develop a sustainability plan particularly at the group level to ensure all receive technical assistance from the project necessary to achieve productivity The ET recommends that, at a minimum, farmer groups should a) receive training on a set of production techniques, b) be informed about input market links available to them (with possible partnership facilitation conducted by the project), c) be informed about output market links available to them (with possible partnership facilitation conducted by the project); and d) be introduced to avenues for credit that they can access. If these items are delivered by the project, the group can be considered as receiving the complete intervention. This plan should include an easy-to-use tracker that the project can integrate into their M&E system; this will be able to quickly inform the team where each group is in the development and consolidation process, toward the ultimate goal of a sustainable group. Based on this plan, the team can determine the length of time it takes to help a group achieve sustainability, and can adjust project targets accordingly (see recommendation 1). VII. LESSONS LEARNED M&E should not only serve accountability purposes, but also provide input for an adaptive management approach to project implementation. The project has 33 indicators that it has valiantly attempted to collect sufficient data on, tracking from a baseline through the project midpoint. Revisions and adjustments to the system have been made by project staff since inception to improve this system and ensure it results in accurate and reliable data that coincides with PIRS requirements. Staff, however, have not utilized the data or this system as inputs into their project planning and implementation approach. As presented in this report, this is due in some part to the number and 43 complexity of indicators tracked by the project. This is also due to the project’s M&E system that does not include an interactive database to more efficiently track progress. At the project midpoint, it is not reasonable to develop a new database to support the M&E system; however, the project (and projects in the future) can adopt a more adaptive management approach by utilizing indicator data (and disaggregating data by geography, group type, and gender) as inputs for programmatic decisions. Coordination among complementary support services/markets is crucial for the development of horticulture value chains. Essential complementary aspects for the development of the horticultural value chain are a) access to output market, b) access to input market, c) access to credit, and d) information regarding production techniques (i.e. technical assistance). However, in some of the project intervention areas, there are no input suppliers operating, farmers do not have access to the credit market, and there are no buyers coming to purchase vegetables. Farmers’ access to one of these key elements (and demand for related services) depends on their expectations on access to the supply of complementary markets/services. For instance, farmers’ demand for agricultural inputs may depend on access to finance, and credit provision depends upon access to output markets. Also, demand for agricultural inputs depends on expectations on access to output markets. Agricultural development projects like Avansa Ag can play a crucial coordinating role among actors operating in the above mentioned complementary sectors. Contract farming is a possible solution for a sustainable coordination of complementary transactions. The coordination function that development projects like Avansa Ag can play (see above) is limited to the implementation period of the project. For a sustainable development of the horticultural value chain, such a coordination function needs to be developed through other mechanisms. Avansa Ag has promoted the use of contract farming, which involves the advancement of seeds from supermarkets (or from other well-established wholesalers) and an obligation to deliver a specific quantity of vegetables at harvest time from farmers at a pre-established price. Contract farming is a kind of interlinked contract, where two parties (e.g. a farmer and a supermarket) establish agreements in at least two markets (e.g. credit, input, output, technical assistance). The use of interlinked contracts represents an interesting alternative for a sustainable coordination of service provision and access to complementary markets and can be considered for replication in other projects. 44 VIII. ANNEXES ANNEX A: EVALUATION STATEMENT OF WORK Below is a streamlined scope of work (SOW) that SI developed based on language about the MTE in the task order as well as from discussions with both USAID and Cardno over the last two years. This SOW was presented to USAID for review and validation in March 2017. EVALUATION PURPOSE Considerations for Review: Social Impact tailors its evaluations to the needs of the evaluation user. Understanding exactly how the evaluation will be used, who will use it and what decisions it will inform, will help the team get evaluation users the information they need to move forward. The MTE purpose is to assess the effectiveness of project approaches to-date; the relevance/validity of Avansa Ag objectives/purposes; the efficiency of Avansa Ag’s progress in achieving results; and the sustainability of different aspects of the activity. The primary use of this MTE is to provide evidence to the Economic Growth team at USAID to help them improve the design and implementation of the ongoing intervention, adapt the interventions to potentially new contexts, and explore evolving approaches for addressing identified problems and facilitate dialogue in the sector with the GOTL and key stakeholders. The GOTL and key donors in the sector might use recommendations from the evaluation to design new interventions in the sector. It is also expected that the Avansa Agrikultura implementer, Cardno, use the results of the evaluation to fine tune their activities/approach and improve implementation. EVALUATION QUESTIONS Considerations for Review: The evaluation design is ultimately structured to answer each of the evaluation questions. USAID recommends that evaluations have no more than 5 questions. Illustrative evaluation questions (EQ) were included in the Avansa M&E Project Statement of Work. Based on its experience over the past two years as well as consultations with Avansa Ag and an evolving context, the Evaluation Team (ET) has refined the original questions and proposes the questions below for consideration by all stakeholders. The questions build on DAC/OCED evaluation criteria and reflect the evaluation purpose and focus on the most useful types of questions at this stage in implementation. Questions one and two cover effectiveness—if the project is meeting its expected results and if those results are different amongst different categories of beneficiaries. Question 3 focuses on relevance— understanding if the assumptions upon which the project was built are still valid, if the intervention meets beneficiary needs and if they apply the national action plan in Timor Leste. Question 4 (efficiency) focuses on management and the extent to which implementation maximizes the use of human and financial resources to achieve these results. Finally, question 5 looks at sustainability, looking into any indications that the results of this project will be sustained beyond the life of the project. As part of the evaluation methodology, the ET will work with USAID and other stakeholders to validate and improve the questions. Each of these questions will allow SI to understand stakeholder perceptions in each of these areas as well as challenges faced and lessons learned through these contexts. 1. To what extent has Avansa been successful in achieving project sub-purposes (and associated outputs)? a. What evidence exists that the value chain approach taken by the project has led to/is 45 leading to increased productivity in the horticulture sector (sub-purpose 1, outputs 1 - 6)? b. What evidence exists that Avansa Ag activities have/are leading to increases in the ability of Timorese citizens (men and women) to engage in economic activities (sub-purpose 2, outputs 7 and 8)? 2. Were achievements toward Avansa Ag purposes varied by geography, community type, or gender and vulnerable/marginalized groups? 3. To what extent are the activity’s purposes (including outputs/indicators and targets) still valid for current development context in Timor-Leste and lessons from the past months of implementation? 4. To what extent is the overall management of Avansa Ag contributed to/inhibited project progress? (This includes management from the steering committee, USAID and Cardno, the Avansa Ag implementer). What opportunities, if any, exist for improvement in management of the activity? 5. To what extent has Avansa Ag made progress in ensuring sustainability of activities beyond USAID support? PROPOSED METHODOLOGY Considerations for Review: The methodology determines the type of data that you will receive from the evaluation. Provided the type of questions above, the evaluation team’s methodology will require an evaluative approach with a keen eye towards making well substantiated recommendations about future directions. Avansa Ag is a wide-ranging program with an ambitious scope. It involves a multitude of actors at in the horticultural value chain, in the area of natural resources management (NRM), and nutrition. In order to provide substantive, evidence-based conclusions and recommendations the evaluation will need to be targeted in its approach. The evaluation will employ a mixed-methods design consisting of three data collection methods: 1) document review, 2) key informant interviews and focus groups discussions, 3) observations. These methods will provide data to answer each of the evaluation questions. The inception report will provide further detail of the evaluation methodology which will be built around each evaluation question. 46 ANNEX B: TEAM COMPOSITION The Team Leader (TL) for this evaluation was Dr. Matteo Borzoni, an international development professional with a strong inter-disciplinary background and 15 years’ experience as a consultant, researcher, and project director for a wide range of donors, including USAID. He has substantial experience in implementing, designing and evaluating projects aimed at diversifying livelihoods, accessing markets, enhancing the capacities of farmers’ cooperatives, developing agricultural value chains, increasing agricultural production, and improving food security and nutrition. His credentials includes a PhD in Management and Economics, an MSc with Distinction in Rural Development and Agricultural Economics and a BSc with honors in Tropical and Sub-Tropical Agriculture. Ms. Amanda Stek served as the Evaluation Specialist, specifically supporting evaluation design and tool development, fieldwork management, data analysis, and report writing. Ms. Amanda Stek is an international development professional with eight years of experience in performance monitoring, project management, performance and impact evaluation, and research in multiple sectors including Democracy and Governance, Human Security, Health, Education and Environment. At Social Impact, Ms. Stek works on multiple projects where she has served as both the Senior Program Manager, Team Leader, and Evaluation Specialist. Mr. Alberto Correia served as Senior Local M&E Specialist. Mr. Correia brings seven years of experience working in Timor-Leste in monitoring and evaluation and international development. He has expertise in indicator tracking, monitoring, data collection tool development, and qualitative and quantitative evaluation methods. At Social Impact, Mr. Corriea spent two years supporting the Avansa M&E project full time as the Senior Local M&E Specialist. Dr. Acáçio Cardoso Amaral served as the Senior National Sector Expert and was responsible for identifying key informants, conducting focus group discussions (FGDs) and key informant interviews (KIIs), and providing insights into findings and recommendations. Mr. Amaral has 14 years of experience in international development and agriculture analysis. He has a PhD in Veterinary Epidemiology from the University of Murdoch, Western Australia and specializes in the sectors of agriculture, animal health and science, and agro socio economy. Ms. Eugenia Correia served as Senior Local M&E Specialist She was responsible for facilitating FGDs (particularly with women), collecting administrative data from participants, and taking FGD/KII notes. She has eight years of experience in international development in Timor-Leste and holds a Master’s Degree in Peace and Conflict Studies from the National University of Timor-Leste. Dr. Jennifer Mandel and Dr. Richard Columbia served as Gender Specialists and ensured integration of gender perspectives into the evaluation’s deliverables using SI’s Gender Scorecard tool. They reviewed and provided feedback on the Draft Evaluation Inception Report, corresponding data collection tools, and the Draft Evaluation Report. Dr. Jennifer Mandel has a PhD in Geography, with an emphasis on women’s empowerment. She brings seven years of experience as a professor of Geography and Women’s Studies and more than four years of experience designing evaluations, impact assessments, and research studies related to gender analysis for USAID projects. Dr. Richard Columbia is an international development professional with more than 30 years of experience in program management including strategic planning, evidence based programming, results based management, project start up and administration, personnel management, financial management, local capacity building, coalition building, and client liaison. Dr. Columbia has supported activities in health, education, youth and gender in more than 50 countries and holds a PhD in Applied Anthropology. 47 ANNEX C: DATA COLLECTION PROTOCOLS This annex contains all information related to the data collection tools for the Avansa Ag MTE. The annex is divided into data collection methods: KIIs, FGDs, and Observations. For KIIs and FGDs, coversheets and introductory statements (including consent script) are included. For FGDs, further guidance is provided for the moderator for how to manage the FGD. A participant list is also provided for FGDs, the completion of which will be monitored by the note-taker. Coversheets and introductory statements will be used before each type of KII and FGD. For each question included in the stakeholder data collection tools, the evaluation team has noted the related/relevant evaluation question (1, 2, 3, 4, or 5) in parenthesis following the question. This is to assist the team with data analysis post-fieldwork. If a question does not have an evaluation question noted, this means the question is related to background information/details and is unrelated to a specific evaluation question/topic. Notes to the facilitator/interviewer are provided in brackets [ ] throughout the protocols. Protocols will be collated into separate Word documents for printing and use during fieldwork/data collection. The team will take written notes, using recordings in FGDs to assist in later note completion and finalization. Key Informant Interviews Coversheet and introductory statement *Names and position titles will be documented in a separate document to protect anonymity Interview Code: Date of Interview: Time Start: Team (Sub-Team A, B): Time End: Interviewer(s): Location: Note taker(s): Category of Interviewee: Age (if group interview, list all): No. of Interviewees _________ M _____ F______ Provided Consent: Y or N Notes about Consent Provided (if any): 48 Good morning/afternoon and thank you for meeting with us today. As mentioned during our interview request, we are working with USAID to conduct a midterm evaluation of the Avansa Agrikultura Project (Avansa Ag). We work for a company called Social Impact. The evaluation is intended to provide an informed assessment of Avansa Ag progress to date and recommend potential modifications to the project, if needed. It is very important for you to know that this study will not determine if you will receive any services/support/funding now or in the future. The way you answer our questions will not determine if you will receive any kind of government or NGO assistance now or in the future. There is no direct benefit to you or your community for your participation in this study, besides toward the goal of improving the Avansa Ag project. Our team has had the opportunity to review some background documents to get a better sense of the design and implementation of the project. However, these documents can only tell us so much. We would like to speak with you today to hear about your experience, in your own words, in order to help us better understand how these projects look and function “on the ground.” It is important to understand that while we would like your help in this study, you do not have to take part if you do not want to, and you do not have to answer any questions if you do not feel comfortable doing so. If you decide to take part, your responses will be kept strictly confidential. This means that your name will not be mentioned anywhere in the report, and will not be provided to anyone, including USAID/Carndo/Avansa Ag or anyone in your community. Specifically: • We will collect information on individuals’ names, organizations, and positions. A list of key informants will be made available as an annex to the final evaluation report, but those names and positions will not be associated to any particular findings or statements in the report. • We may include quotes from respondents in the evaluation report, but will not link individual names, organizations, or personally identifiable information to those quotes, unless express written consent is granted by the respondent. Should the team desire to use a particular quote, photograph, or identifiable information in the report, the evaluators will contact the respondent(s) for permission to do so. • All data gathered will be used for the sole purposes of this evaluation, and will not be shared with other audiences or used for any other purpose. Your participation in this interview is voluntary and if you do not feel comfortable answering a particular question, please let us know and we will simply go on to the next question. Do you consent to participate in this interview? By saying “yes,” and participating in this study, you are indicating that you have heard this consent statement, had an opportunity to ask any questions about your participation and voluntarily consent to participate. Do you have any questions for us before we get started? If you have questions or concerns about the study after we leave today, you can contact [name of national or international evaluation team member]; the contact information is on the card I gave you. 49 KII questions by respondent category Notes to Facilitator/Interviewer The interview should be led by the interviewee, and only guided by the interviewer. The interview should be a relaxed conversation. The tables reported below present a list of topics to be covered during interviews. They also include concrete questions to guide the conversation and to cover each topic. The reason for focusing on topics rather than on questions is that the way in which a question would need to be formulated varies considerably depending on the interviewee’s specific knowledge, attitude and circumstance. The listed topics will allow the interviewer to achieve clear predetermined interview goals, but the actual questions will be adapted –as one does in a normal conversation- to the conversation tone, interviewee’s attitude and knowledge. It is to be expected that the same key issue may naturally be transformed into several questions with one interviewee and only into one question with another. 1 USAID TOPIC QUESTIONS Your work with Avansa Ag ● What is your role in the Mission, and when did you start working with the Avansa Ag project? ● Please describe your main responsibilities/tasks that you complete in support of the project Communication with the project (EQ4) ● Who do you communicate most frequently with at Avansa Ag and within the GOTL? ● To what extent do you receive all the information you need from Anvansa Ag and the GOTL to do your job? What additional information would you like to receive that you do not? Project coordination (EQ4) ● Please explain your and the project’s coordination with the Government of Timor-Leste, PSC, and other donors/projects (if any). ● What could be improved to better support you in the achievement of project objectives (goal, outcome, output, etc.)? Achievement of project objectives (EQ1) ● What are the main achievements of the project? ● Where do you feel the project is falling short? ● Please describe the progress of the project towards the following outputs (to be listed individually) and tell us what you think should be improved ● Improving the policy and enabling environment (has the regulatory environment changed)? ● Increasing climate change resilience (through NRM initiatives) ● Promoting the adoption of sustainable production practices at the farm level ● Strengthening the capacity of farmers groups and associations ● Improving market linkages ● Improving the functioning of the private sector base input supply system ● Improving nutrition and sanitation practices ● Improving skills to utilize productive assets for economic activity 50 ● Why is the project supporting exports while the SoW clearly state that no activities are envisioned to generate exports of agricultural commodities? Challenges (EQ1, EQ3, EQ4) ● What are the main challenges faced by this project thus far in achieving project objectives? ● How has the Timor-Leste context changed, affecting markets or project participants, if at all? ● Please explain your opinion of the prime contractor’s performance? Information needs and communication (EQ4) ● Please explain the amount of information (monitoring data) you receive from the project. ● To what extent do you have enough monitoring data to make good decisions about how to improve the achievement of project outputs? Why or why not? Implementation modality (EQ4) ● Please describe how the project uses consultants (international and local experts), grantees, sub-contractors, etc. Gender and other cross-cutting themes (EQ2, ● What, if any, challenges has the project faced with gender integration? To what extent, if any have you faced challenges with achieving women’s participation, and why? ● What , if any, challenges have you faced with M&E, and communication/knowledge sharing across project activities/outputs/districts? Geographic integration (EQ1, EQ2) ● Which geographic areas show greater integration in the project activities, and why? Sustainability (EQ 5) ● Please comment on the suitability of the Avansa Ag approach in the Timor-Leste context, based on what you know now from the first months of implementation. ● Which parts of the project (partnerships, farmer groups, linkages, etc) are most likely to be sustained after the project ends, at this stage (based on where the project is now)? Why? ● What sustainability plans has the project developed thus far, if any? ● If no plan has been developed, what plans are in place to develop this in the future? What do you anticipate being the main factors/points in that plan? ● What needs to be improved now in order to improve the sustainability of the project? Recommendations ● If a project like this were to come again in the future, what would you like to see done differently? ● Do you have any recommendations for the program in its final years of operation? 2 Avansa Ag Implementer (Cardno staff, sub-contractors – SSG Advisors and HIAM Health) 51 COP and DCOP TOPIC QUESTION Your work with Avansa Ag ● What is your role in the Avansa Ag project, and when did you start working with the project? ● Please describe your main responsibilities/tasks that you complete for the project. Communication and coordination with USAID and GOTL (EQ1, EQ4) ● Who do you communicate most frequently with at Avansa Ag, USAID, and MAF? ● Please explain your coordination with USAID, MAF, PSC, and other donors/projects (if any). Does this help you do your work/achieve project goals? What could be improved to better support you in the achievement of project objectives? Staffing and budget (EQ4) ● Is staffing/budget/USAID support in the project sufficient to meet project objectives? If no, please explain. Project design (Q3) ● Do you feel that the original project design is still valid? ● Why is the project supporting exports while the SoW clearly state that no activities are envisioned to generate exports from agricultural commodities? ● Progress towards objectives (EQ1) ● Are the activities that are conducted in this project sufficient to meet the goals of the project? Why or why not? ● Can you please tell us main achievements and problems with regards to the following expected outputs? [to be listed individually] ● Improving the policy and enabling environment (has the regulatory environment changed?) ● Increasing climate change resilience (through NRM initiatives) ● Promoting the adoption of sustainable production practices at the farm level ● Strengthening the capacity of farmers groups and associations ● Improving market linkages ● Improving the functioning of the private sector base input supply system ● Improving nutrition and sanitation practices ● Improving skills to utilize productive assets for economic activity Challenges (EQ1, EQ3, EQ4) ● What are the main challenges faced by you/this project thus far in achieving project objectives? ● How many of these challenges have been/are being addressed by the project/you (if any)? ● Have the project sub-contractors performed as expected? Information needs and communication (EQ4) ● Do you have enough data/information to make good decisions about how to improve the achievement of project outputs? ● Does the internal M&E system/data/Avansa Ag leadership provide you with the credible information needed to guide the project to a successful conclusion? Implementation modality (EQ4) ● Please describe how you selected beneficiaries for this project (from districts and sucos to individuals). What activities did you conduct to facilitate this selection process? 52 ● Please describe how the project uses grantees ● Do you find these that mechanisms involving grantees and sub-contractors are used to be sufficient/flexible enough? Gender and other cross-cutting themes (EQ1, EQ2, EQ4) ● What, if any, specific challenges has the project faced with gender issues? Have you faced challenges with achieving women’s participation, and if yes, why? ● Have you faced challenges with M&E, and communication/knowledge sharing across project activities/outputs/districts? ● What data challenges do you face, if any? Is it difficult to communicate project information to various stakeholders and staff throughout the country, and if yes, why? Geographic integration (EQ1, EQ2) ● Which geographic areas show greater integration in the project activities, and why? Sustainability (EQ 5) ● Please comment on the suitability of the Avansa Ag approach in the Timor￾Leste context, based on what you know now from the first months of implementation. ● Which parts of the project (partnerships, farmer groups, linkages, etc) are most likely/least likely to be sustained after the project ends, at this stage (based on where the project is now)? Why? ● What sustainability plans has the project developed thus far, if any? ● If no plan has been developed, what plans are in place to develop this in the future? What do you anticipate being the main factors/points in that plan? ● Recommendation s ● Are there new opportunities in this market (for example with new crops, or local vs export markets) that you would like to take advantage of now or in the future? ● If a project like this were to come again in the future, what would you like to see done differently? ● Do you have any recommendations for the program in its final years of operation? For Output Leaders/Managers (including HIAM Health) Questions are included for all output leaders – not all leaders will be asked all questions listed. TOPIC QUESTIONS Your work with Avansa Ag (EQ4) ● What is your role in the Avansa Ag project, and when did you start working with the project? ● Please describe your main responsibilities/tasks that you complete for the project. ● Do you work with farmer groups, and which groups do you work with most frequently? ● Were you part of selecting groups? ● Have you trained farmers/conducted trainings for Avansa Ag personally? ● Do you collect data from farmers/district managers and report data to the M&E team? ● Do you participate in project planning and review? 53 Communication and coordination with USAID and GOTL (EQ1, EQ4) ● Who do you communicate most frequently with at Avansa Ag, USAID, and MAF? ● Please explain your coordination with USAID, MAF, PSC, and other donors/projects (if any). Does this help you do your work/achieve project goals? ● What could be improved to better support you in the achievement of project objectives in terms of communication and coordination? Implementation modality (EQ4) ● Please describe how you selected beneficiaries for this project (from districts and sucos to individuals). What activities did you conduct to facilitate this selection process? ● To what extent was gender balance a consideration and how did you go about trying to achieve it? Information needs and communication (EQ4) ● To what extent you have enough data/information to make good decisions about how to improve the achievement of project outputs? Why or why not? Staffing and budget (EQ4) ● Is staffing/budget/USAID support in the project sufficient to meet project goals? If no, please explain. Improving the policy and enabling environment (EQ1) ● What policies has this project worked on? ● How were these policies selected as a focus of the project? ● Which kind of consultations were conducted for the design of policies? ● At which stage of formal government/parliamentarian approval are they at this moment? ● How was the policy Reform Matrix Developed? Increasing climate change resilience (through NRM initiatives) (EQ1, EQ2) ● What are the main activities you have conducted in NRM? ● What are the NRM plans? ● How was the vulnerability assessment used? Was it used to select intervention sites? How were reforestation sites selected? ● Which geographic areas (i.e. district, sucos)showed greater responsiveness? Promoting the adoption of sustainable production practices at the farm level (EQ1, EQ2) ● What are the main activities you have conducted in this output? ● Which agricultural practices have been adopted more by farmers, and which less? Why? ● Which groups have been most receptive and which least (in terms of gender, age, etc)? ● In which geographic areas have production practices been adopted more and in which less? Why? Strengthening the capacity of farmers groups and associations (EQ1, EQ2) ● What are the main activities you have conducted in this output? ● Do you have any evidence that shows that Avansa Ag has improved farmer groups’ business capacities? ● How many farmers groups are receiving credits from banks and/or MFIs? ● Which are the main challenges in access to finance for farmer groups? ● Have you noted any differences in gender or age? ● In which geographic areas have farmer groups been more successful, and in which less? ● Were inputs delivery on credit supported by the project? ● What is the role of the horticultural working group? 54 Improving market linkages (EQ1, EQ2) ● What are the main activities you have conducted in this output? ● Which supermarket involvement approach was used? ● How many supermarkets are buying from farmer groups supported by the project? Were these the same supermarkets from the previous DAC project? ● Which linkages have been established with traders? ● Do traders provide credit to farmer? ● To what extent are traders more/less inclined to give to some groups rather than others (differences by gender/age/other)? ● How has the market system analysis been used? How so? ● How is the development of export oriented vegetables working? ● How do the grants support the development of market linkages? Can you provide some examples? ● In which geographic area has the establishment of new market linkages been more successful? In which less? Improving the functioning of the private sector base input supply system (EQ1, EQ2) ● What are the main activities you have conducted in this output? ● How was the input supplies monitoring used? ● In general terms, do input suppliers provide inputs on credit to farmers? ● How many farmers groups have been linked to input suppliers? ● How do the linkages developed by work? ● Is there any geographic difference regarding the responsiveness of input suppliers in the five project areas? Improving nutrition and sanitation practices (EQ1, EQ2) ● What are the main activities you have conducted in this output? ● Was porridge preparation part of the training? ● How are vegetables produced in school gardens distributed/used? Did you establish school garden management committees? ● In which municipalities have school and home gardens been more successful, and in which less? ● During which months are the availability of vegetables higher? ● During which months are the availability of vegetables lower? ● Do you have any evidence of actual changes of behaviors regarding distribution of nutrients within households and culinary practices? Improving skills to utilize productive assets for economic activity (EQ1, EQ2) ● What are the main activities you have conducted in this output? ● How many saving and loan groups have been created? ● Did you note any spontaneous replications of saving and loan groups? ● In which geographic areas have activities with female entrepreneurs been most successful, and why? Trainings (EQ1, EQ2, EQ3, EQ4) ● What trainings/activities have you supported during Avansa Ag? Please describe the following for each training/activity. ● Participants [Probe: To what extent were men, women, and youth engaged equally? What role did you play in selecting participants for trainings/activities?] ● Trainer [Probe: To what extent was the trainer effective and knowledgeable? Did the farmers respond positively to the training?] ● Materials [Probe: Were farmers provided materials for the training that they could keep? Were the materials in their local dialect?] 55 ● Location/facilities/timing/schedule [Probe: Where were the trainings held, and was this appropriate for the topic? Was the training conducted at a time convenient for men and women? How long were the trainings, and was all the material covered?] ● Participation [Probe: Were there any issues with participants’ attendance for the entire training/series of trainings?] ● Method of instruction [Probe: Which method of instruction was used (frontal lesson, practical exercise, etc.)? To what extent was the method of instruction effective at transferring knowledge? Did the farmers respond positively to the methods/tools/structure used for the training?] ● Follow up [Probe: What follow up was conducted after the trainings, if any? How was feedback collected from the farmers/groups, if at all?] ● Knowledge retention [Probe: In your opinion, have farmer groups/participants maintained the knowledge that they learned through the trainings? How do you know this?] ● Relevance/applicability [Probe: Were the trainings focused on topics that are relevant/important for the participants? How did Avansa Ag select the topics for the trainings?] Sustainability (EQ5) ● Which aspects are more likely to be sustained after the project ends, at this stage? ● Which aspects are less likely to be sustained after the project ends, at this stage? ● What sustainability plans has the project developed thus far, if any? ● Do you think technical assistance will be required after the project ends? If yes, who do you think can provide technical assistance once the project is over? ● What are the main enablers/inhibiters to project sustainability? Recommendation s ● If a project like this were to come again in the future, what would you like to see done differently? ● Do you have any recommendations for the program in its final years of operation? District Managers TOPIC QUESTIONS Your work with Avansa Ag (EQ4) ● What is your role in the Avansa Ag project, and when did you start working with the project? ● Please describe your main responsibilities/tasks that you complete for the project. ● Did you receive any trainings to prepare you to complete these responsibilities? If yes, please explain. Did you find these trainings effective? Why or why not? ● Do you work with farmer groups, and which groups do you work with most frequently? ● Were you part of selecting groups? ● Have you trained farmers/conducted trainings for Avansa Ag personally? ● Do you collect data from farmers/district managers and report data to 56 the M&E team? Do you plan/manage other activities/ events/ meetings? ● Do you participate in project planning and review? Communication and coordination with (EQ4) ● Who do you communicate most frequently with at the Avansa Ag Dili Head Office? Is this communication frequent enough for you to feel updated and equipped to do your job for the project? (EQ4) ● Please explain your coordination with MAF, and other donors/projects in the area (if any). Staffing and budget (EQ4) ● Is staffing in the Dili office and in your district sufficient to meet your objectives? If no, please explain. Farmer groups (EQ1, EQ5) ● Please describe the farmer groups you support. ● What crops are most lucrative for them? How profitable are they? ● How do they access finance, if they require it? How do they operate? ● What inputs do they use? ● Who do they sell to? ● What supermarkets/finance institutions/traders/aggregators do they partner with? Has this changed since Avansa Ag? Have you seen any changes in these groups since you started working with them? If yes, please explain, and explain what you attribute this change to. ● Who is in charge of sales within households and who retain revenues? ● What are the main challenges faced by the farmer groups that you work with? How many of these challenges are addressed by Avansa Ag (if any)? [Probes: Pests, credit, inputs, etc] ● Are the farmer groups likely to continue after Avansa Ag support ends? Why or why not? Increasing climate change resilience (through NRM initiatives) (EQ1, EQ5) ● Which activities have been conducted? ● How were reforestation sites selected? ● How are the forestry nurseries working? Who is managing them? ● Are the forestry nurseries likely to continue after after Avansa Ag support ends? Why or why not? Who will buy seedlings after the project? Improving nutrition and sanitation practices (EQ1, EQ5) ● What was your role in this area? ● How were people selected for participation? ● How are vegetables produced in school gardens distributed/used? ● Are the school gardens likely to continue after the project ends? Why? Improving skills to utilize productive assets for economic activity (EQ1, EQ5) ● How many saving and loan groups have been created in your district? ● Which challenges do the groups face? ● Did you note any spontaneous replications of saving and loan groups? ● Are the saving and loan groups likely to continue after the project ends? Why? Trainings (EQ1, EQ2, EQ3, EQ4) ● What trainings/activities have you supported during Avansa Ag? Please describe them according to the following: ● Participants [Probe: To what extent were men, women, and youth engaged equally? What role did you play in selecting participants for trainings/activities?] ● Trainer [Probe: To what extent was the trainer effective and knowledgeable? Did the farmers respond positively to the training?] ● Materials [Probe: Were farmers provided materials for the training that they could keep? Were the materials in their local dialect?] 57 ● Location/facilities/timing/schedule [Probe: Where were the trainings held, and was this appropriate for the topic? Was the training conducted at a time convenient for men and women? How long were the trainings, and was all the material covered?] ● Participation [Probe: Were there any issues with participants’ attendance for the entire training/series of trainings?] ● Method of instruction [Probe: Which method of instruction was used (frontal lesson, practical exercise, etc.)? To what extent was the method of instruction effective at transferring knowledge? Did the farmers respond positively to the methods/tools/structure used for the training?] ● Follow up [Probe: What follow up was conducted after the trainings, if any? How was feedback collected from the farmers/groups, if at all?] ● Knowledge retention [Probe: In your opinion, have farmer groups/participants maintained the knowledge that they learned through the trainings? How do you know this?] ● Relevance/applicability [Probe: Were the trainings focused on topics that are relevant/important for the participants? How did Avansa Ag select the topics for the trainings?] Implementation (EQ1, EQ4) ● What are the main challenges faced by you and your staff in your work in this district? Recommendations ● If a project like this were to come again in the future, what would you like to see done differently? ● Do you have any recommendations for the program in its final years of operation? 3 National Government For Ministry of Ag and Fisheries - MAF TOPIC QUESTIONS Involvement in the project (EQ4) ● Are you aware of the USAID’s Avansa Agrikultura project? If yes, what do you know about the project? ● Who do you interact with most frequently from the project? Describe this interaction (is it normally regarding update about the project, consultation about project activities, etc?) Do you feel you have all the information you need/require from the project? Capacity of the project to improve policies (EQ1, EQ4) ● Are you aware of any policy support provided by the project? What policies has the project focused on? ● What is your personal opinion of this work? Is it useful? Is it done effectively? ● Please describe the consultations the project conducted regarding the development of polices drafted by the project At which stage of government/parliamentarian approval are these policies? ● Have there been any changes in these policies since the project started? What led to these changes? Who/what entity was the primary 58 driver/owner of these changes? (EQ1, EQ3) Project implementation (EQ4) ● Have you (or your staff/representatives) participated in any Avansa Ag activities, meetings, or conferences? What did you think of these events (were they managed well, useful, interesting, etc)? Involvement of districts ● How are district MAF involved with Avansa Ag? Horticultural Working Group (EQ1) ● Are you a part of the Horticultural Working Group/Association (nationally, district level)? Please explain how the group works, and what benefits (if any) MAF receives through your participation. Progress towards achievement of objectives (EQ1) In your opinion, has the project made sufficient progress toward its objectives [list below]? Please explain. [Encourage reference to specific data/examples. Probe for variation across groups by age/gender] ● Improving the policy and enabling environment ● Increasing climate change resilience (through NRM initiatives) ● Promoting the adoption of sustainable production practices at the farm level. ● Strengthening the capacity of farmers groups and associations ● Improving market linkages ● Improving the functioning of the private sector base input supply system Sustainability (EQ5) ● Please comment on the suitability of the Avansa Ag value chain model in the Timor-Leste context. Recommendation s ● If a project like this were to come again in the future, what would you like to be different? ● Do you have any recommendations for the program in its final years of operation? 4 District Government For Ag and Fisheries Office - MAF TOPIC QUESTIONS Involvement in the project ● Are you aware of the USAID’s Avansa Agrikultura project? If yes, what do you know about the project? ● Who do you interact with most frequently from the project? Describe this interaction (is it normally regarding update about the project, consultation about project activities, etc?) Do you feel you have all the information you need/require from the project (EQ4) Trainings ● Have you (or your staff/representatives) participated in any Avansa Ag activities, trainings, meetings, or conferences? What did you think of these events (were they managed well, useful, interesting, etc)? (EQ4) ● Do you feel you and your staff have the necessary skills to provide technical assistance to farmers on vegetable production once the project is over? (EQ5) Progress towards achievement of objectives (EQ1) ● In your opinion, has the project made sufficient progress toward its objectives [list below]? Please explain. [Encourage reference to specific data/examples] 59 ● Increasing climate change resilience (through NRM initiatives) ● Promoting the adoption of sustainable production practices at the farm level ● Strengthening the capacity of farmers groups and associations ● Improving market linkages ● Improving the functioning of the private sector base input supply system Sustainability (EQ5) ● Please comment on the suitability of the Avansa Ag value chain model in the Timor-Leste context [Prob: do you think that the farmer groups will be active after the project ends? What about the nutrition groups, the school gardens, the saving and loan groups? Recommendations ● If a project like this were to come again in the future, what would you like to be different? ● Do you have any recommendations for the program in its final years of operation? 5 Bank and Micro-Finance Institutions TOPIC QUESTIONS Involvement in the project ● How do you work with Avansa Ag, and when did you start working with Avansa Ag? [via MOU, grant, sub-contract, etc?] (EQ4) ● Who do you interact with most frequently from the project? Do you feel you have all the information you need/require from the project in order to implement/complete your agreement? (EQ4) Training and assistance from Avansa Ag ● Have you (or your staff/representatives) participated in any Avansa Ag activities or trainings? If yes, please explain. Did you find the activity/training useful? If yes, why? If no, why? ● Did the project helped you to develop loan products? If so, how? Loans characteristics ● Do you provide loans to farmer groups/individuals through Avansa Ag? If yes, to how many groups and individuals? ● Please describe the demographics and businesses of your loan recipients (farmers) from the Avansa Ag project. [Note if they loan to women, men, youth, groups, etc] (EQ1, EQ2) ● Do you provide short-term loans? If so, can you please describe main loan characteristics: min. and max loan value, loan duration, interest charged, type of collateral requested? ● Do you provide loans for investments? If so, can you please describe main loan characteristics: min. and max loan value, loan duration, grace period, interest charged, type collateral requested? ● As of today, what is the total amount you have loaned to farmer groups/individuals through Avansa Ag? (EQ1) [Check against project documents/data] Sustainability (EQ5) ● What are the benefits/advantages and cost/disadvantages in working with these groups through Avansa Ag for your institution? ● What is the average default rate? ● Is it profitable for you to work/partner with these groups? Please explain/provide details. 60 ● To what extent do you find that the linkage you now have to farmer groups is a better/more profitable/more sustainable linkage than you had before? ● Do you envision continuing this work that you started with Avansa Ag support (after the project ends)? If not, why not? (EQ1, EQ5) 6 Supermarket buyers) TOPIC QUESTIONS Involvement in the project ● How do you work with Avansa Ag, and when did you start working with Avansa Ag? ● Do you have a formal agreement with Avansa Ag or any kind of MOU, grant, sub-contract? ● (EQ4) ● Who do you interact with most frequently from the project? Do you feel you have all the information you need/require from the project in order to implement/complete your agreement? (EQ4) Assistance from Avansa Ag ● Have you (or your staff/representatives) participated in any Avansa Ag activities, trainings or seminars? If yes, please explain. Did you find the activity/training useful? If yes, why? If no, why? Sourcing from local farmers ● From how many individual farmers and farmer groups supported by the Avansa Ag do you buy from? What districts are they from? [Probe for variation across groups] ● How did you begin working with these groups [describe Avansa Ag’s role in this]. Before your work with Avansa Ag, did your supermarket/company work with farmer groups? [Confirm against project data] (EQ1) ● Please describe your work/partnership with these groups. Does this experience vary by district/suco? [If detailed records are available, take note.] (EQ1, EQ2) [Probes, as follows: ● Do you provide extension services to farmers? ● Do you provide inputs to farmers? Are these inputs local or imported? ● Do you pick up produce (what/which crop(s)?), and how often do you pick up this produce? ● How much (volume) do you pick up at these times? ● Do you provide credit to farmers? Describe repayment. ● How many staff do you have working with farmers [per district, if applicable]? What do these staff do to support farmers, and how are they trained to provide this support? ● What are the most challenging aspects of working with farmers/farmer groups? Sustainability (EQ5) ● What are the benefits/advantages and cost/disadvantages in working with these groups through Avansa Ag for your supermarket/company? [Probes: crop quality, compliance to agreements, timing of crop supply, farmer repayment] (EQ1, EQ5) ● Is it profitable for you to work/partner with these groups? Please explain/provide details. 61 ● Do you find that the linkage you now have to farmer groups is a better/more profitable/more sustainable linkage than you had before? ● Do you envision continuing this work that you started with Avansa Ag support (after the project ends)? If not, why not? (EQ1, EQ5) 7 Traders (including importers and exporters) TOPIC QUESTIONS Involvement in the project ● Have you worked with the Avansa Ag? If so, how and when did you start working with Avansa Ag? [via MOU, grant, sub-contract, etc?] (EQ4) ● Who do you interact with most frequently from the project? Do you feel you have all the information you need/require from the project in order to implement/complete your agreement? (EQ4) Assistance from Avansa Ag ● Have you (or your staff/representatives) participated in any Avansa Ag activities, trainings or seminars? If yes, please explain. Did you find the activity/training useful? If yes, why? If no, why? Imports ● Which vegetables do you import and which vegetables do you source locally? ● For imported vegetables, why do you prefer to buy from abroad rather than sourcing locally? Sourcing locally ● What districts do your products come from? ● If engaged with Avansa Ag, have the districts you buy from changed since your engagement with Avansa Ag? ● Do you maintain consolidation centers in the districts? ● Do you provide agricultural inputs to some of the farmers you buy from? To how many? Are they supported by Avansa Ag? [Probe for variations across groups] ● Do you provide agricultural finance to some of the farmers you buy from? To how many? Are they supported by Avansa Ag? ● Do you provide technical assistance to some of the farmers you buy from? To how many? Are they supported by Avansa Ag? ● Do you mainly buy in the field, or do farmers bring their produce to you? ● From how many individual farmers and farmer groups supported by the Avansa Ag do you buy from? ● Before your work with Avansa Ag, did your company work with farmer groups? [Confirm against project data] (EQ1) ● How many staff do you have working with farmers [per district, if applicable]? What do these staff do to support farmers, and how are they trained to provide this support? ● Do you have storage facilities in Timor-Leste? Do you have sufficient space for your supply? ● What are the most challenging aspects of working with farmers/farmer groups? Exports ● Do you exports vegetables? ● What agricultural products o you regularly export, and where do you export? ● What amount (volume) of the products you just mentioned did you 62 export during the last month? And in 2016? ● Has this changed since your engagement with Avansa Ag? ● Did Avansa played any role in supporting your export activity? If so, how? ● Did Avansa Ag help you to find buyers abroad? ● Did you recently start exporting? ● Do you export different goods or quantities then you did Avansa Ag actions If yes, please explain. ● Do you export to different locations before Avansa Ag actions? ● Is there any government regulation that you feel is hindering your export efforts? Sustainability (EQ5) ● Do you export the produce supplied/sourced from farmers that are supported by Avansa Ag? ● What are the benefits/advantages and cost/disadvantages in working with these groups through Avansa Ag for your company? [Probes: crop quality, compliance to agreements, timing of crop supply, farmer repayment] (EQ1, EQ5) ● Is it profitable for you to work/partner with these groups? Please explain/provide details. ● Are there new opportunities that you would like to take advantage of now or in the future? ● Do you envision continuing this work that you started with Avansa Ag support (after the project ends)? If not, why not? Recommendations ● If a project like this were to come again in the future, what would you like to be different? ● Do you have any recommendations for the program in its final years of operation? 8 Input Suppliers (including Aggregators) TOPIC QUESTIONS Involvement in the project ● Have you worked with the Avansa Ag? ● If so, how and when did you start working with Avansa Ag [via MOU, grant, sub-contract, etc?]? How long will you continue to work with Avansa Ag? (EQ4) ● Who do you interact with most frequently from the project? Do you feel you have all the information you need/require from the project in order to implement/complete your agreement? (EQ4) Assistance from Avansa Ag ● Have you (or your staff/representatives) participated in any Avansa Ag activities, trainings or seminars? If yes, please explain. Did you find the activity/training useful? If yes, why? If no, why? Imports and sourcing (EQ1) ● Have you started importing new agricultural inputs during the last two years? Was this somehow related to the work of Avansa Ag? ● Did Avansa Ag helped you to find new input suppliers? Selling ● How many Avansa Ag groups do you work with, and in which districts? How did you begin working with these groups [describe Avansa Ag’s role in this]. Before your work with Avansa Ag, did your company work with 63 farmer groups? [Confirm against project data] (EQ1) ● How many staff do you have working with farmers [per district, if applicable]? What do these staff do to support farmers, and how are they trained to provide this support? ● Please describe your work/partnership with these groups. Does this experience vary by district/suco? [If detailed records are available, take note.] [Probe for variation across groups by age/gender] (EQ1, EQ2) ● Do you provide technical assistance to farmers on vegetable production? Do your staff go to farmers’ fields for this? ● How many staff do you have providing technical assistance to farmers? What do these staff do to support farmers, and how are they trained to provide this support? ● Do you sell inputs on credit to farmers (to be paid after the harvest)? To how many farmers do you provide inputs on credit? Are they supported by Avansa Ag? Agricultural inputs (EQ1, EQ3, EQ5) ● What procedures (timing, cost, etc) are you required to follow to import inputs? ● Is there any government regulation that you feel is hindering your business? ● What are the main input demands from Avansa Ag farmer groups? Is this different or similar to the input demand you have from other farmers groups or customers? ● Do you face challenges with the quality of the inputs you sell to customers? ● Where do you source the inputs that you sell locally from? Are certain inputs more difficult to get than others, and why? Sustainability (EQ5) ● What are the benefits/advantages and cost/disadvantages in working with the farmer groups supported by Avansa Ag for your company? [Probes: crop quality, compliance to agreements, timing of crop supply, farmer repayment] (EQ1, EQ5) ● Is it profitable for you to work/partner with these groups? Please explain/provide details. ● Do you find that the linkage you now have to farmer groups is a better/more profitable/more sustainable linkage than you had before Avansa Ag? ● Are there new opportunities that you would like to take advantage of now or in the future? ● Do you envision continuing this work that you started with Avansa Ag support (after the project ends)? If not, why not? Recommendations ● Do you have any recommendations for the program in its final years of operation? ● If a project like this were to come again in the future, what would you like to be different? 9 Training Providers TOPIC QUESTIONS 64 Involvement in the project ● Have you worked with the Avansa Ag? ● If so, how and when did you start working with Avansa Ag [via MOU, grant, sub-contract, etc?]? How long will you continue to work with Avansa Ag? (EQ4) ● Who do you interact with most frequently from the project? Do you feel you have all the information you need/require from the project in order to implement/complete your agreement? (EQ4) Trainings ● What training services did you provide for Avansa Ag thus far? [List all trainings provided, including locations and topics] What services will you provide in the future? (EQ1) ● How were participants selected for your trainings? To what extent wee men, women and youth equally represented in your trainings? When was the training held (at what time, what day of the week, location)? What districts/sucos did you train in? (EQ1, EQ2) ● Did you or Avansa Ag prepare materials for the training? Did you provide materials to all participants? (EQ1, EQ4) ● Please explain how you managed/ran the training (depending on the size of the training). Were Avansa Ag staff/representatives present? How many facilitators were there? ● Did you conduct pre- and post- tests to determine if knowledge had been gained through the training? If yes, what did you find? If not, how did you determine if your training had been successful or not? [Probe: How did you collect feedback from participants?] (EQ1, EQ4) ● What lessons have you learned from the trainings you conducted thus far? What has worked, and what has not? [What has been most effective, and what has been least effective at transferring knowledge and skills to participants? Were there needs of the participants that were not met by the training?] (EQ1, EQ4) ● What are the main challenges that the beneficiaries face in learning the material (during the training), maintaining knowledge (after the training), and implementing new practices? ● In your opinion, can training participants use the skills you have provided them without further training and support? Recommendations ● What additional trainings, if any, would you suggest for Avansa Ag participants? ● Do you have any recommendations for the program in its final years of operation? (EQ5) 10 Horticultural Association TOPIC QUESTIONS Association Details ● What is the main goal/vision of this association? What do you do, and why? ● Please describe the formation process for this association. How did Avansa Ag facilitate/help this process, if at all? (EQ1) ● Please explain your current operations (bylaws, board/committee members, election of members, legal registration, funding for activities, 65 member fees, employment of expert staff). ● How many Special Horticultural Area (SHA) groups are there in this association? How many farmers do you support through this association? ● When was your last association meeting? What topics were discussed, and who attended? How often during the year do you typically meet? Association Work ● Please explain how Avansa Ag has worked with you/supported you since this association was formed. To what extent has this support benefited the association, and if yes, how? To what extent has this support made anything more difficult for the association, and if yes, how? ● What services do you provide to group members? [Probe: Do you provide trainings/inputs/information of any kind?] Have group members expressed appreciation for these services? If yes, why? If not, what are the challenges? ● Please explain this association’s relationship with supermarket buyers. Do they assist you/work with you in any way? If yes, please explain. ● Please explain if/how this association is working on national policy advocacy. Which laws are you targeting, and why? What would you like to see change in the future? ● Has the national association made progress toward supporting a cell phone based agri-market information platform (as promoted through Avansa Ag last year)? Recommendations and sustainability ● What are the main challenges you face in managing this association and its members? ● Do you have any recommendations for the program in its final years of operation? (EQ5) 66 Focus Group Discussions Coversheet, participant sign in, and introductory statement *Names (of individuals and groups) will be documented in a separate document to protect anonymity FGD Code: Date of FGD: Time Start: Team (Sub-Team A, B): Time End: Facilitators(s): Location: Note taker(s): Recording Number: Category of Group: Age of group: No. of Interviewees ___________ M _____ F______ Notes about Consent Provided (if any): Good morning/afternoon and thank you for meeting with us today. As mentioned during our focus group discussion request, we are working with USAID to conduct a midterm evaluation of the Avansa Agrikultura Project (Avansa Ag). We work for a company called Social Impact. The evaluation is intended to provide an informed assessment of Avansa Ag progress to date and recommend potential modifications to the project, if needed. It is very important for you to know that this study will not determine if you will receive any services/support/funding now or in the future. The way you answer our questions will not determine if you will receive any kind of government or NGO assistance now or in the future. There is no direct benefit to you or your community for your participation in this study, besides toward the goal of improving the Avansa Ag project. Our team has had the opportunity to review some background documents to get a better sense of the design and implementation of the project. However, these documents can only tell us so much. We would like to speak with you today to hear about your experience, in your own words, in order to help us better understand how these projects look and function “on the ground.” It is important to understand that while we would like your help in this study, you do not have to take part if you do not want to, and you do not have to answer any questions if you do not feel comfortable doing so. If you decide to take part, your responses will be kept strictly confidential. This means that 67 your name will not be mentioned anywhere in the report, and will not be provided to anyone, including USAID/Carndo/Avansa Ag or anyone in your community. Specifically: • We will collect individuals’ names and the farmer group they belong to. A list of focus group participants will be made available as an annex to the final evaluation report, but those names and positions will not be associated to any particular findings or statements in the report. • We may include quotes from respondents in the evaluation report, but will not link individual names, organizations, or personally identifiable information to those quotes, unless express written consent is granted by the respondent. Should the team desire to use a particular quote, photograph, or identifiable information in the report, the evaluators will contact the respondent(s) for permission to do so. • All data gathered will be used for the sole purposes of this evaluation, and will not be shared with other audiences or used for any other purpose. Your participation in this focus group discussion is voluntary and if you do not feel comfortable answering a particular question, please let us know and we will simply go on to the next question. We will request your consent to record the interview. We will record the interview for the sole purpose of revising and improving our note-taking during our data analysis process. If you would like at any time for us to stop using the recording, please let us know. Do you consent to participate in this interview and have it recorded for our note-taking purposes (confirm everyone’s consent individually)? By saying “yes,” and participating in this study, you are indicating that you have heard this consent statement, had an opportunity to ask any questions about your participation and voluntarily consent to participate. Do you have any questions for us before we get started? If you have questions or concerns about the study after we leave today, you can contact [name of national or international evaluation team member]; the contact information is on the card I gave you. Participant List: # Name Verbal or Written Consent (signature or mark) Sex (M or F) Age 1 2 3 4 5 68 6 7 8 9 1 0 1 1 1 2 Instructions for FGD Moderator/Facilitator: I. Introduction: The moderator should first introduce herself or himself, instruct participants to sign in (note-taker to monitor), follow the script above, and then welcome and thank all of the focus group discussion (FGD) participants for being there. The moderator should then explain the purpose of the FGD, how participants were selected, and how long it will take. For this study, focus groups will take between 1 – 1.5 hours. II. Ground Rules: While the ground rules will vary depending on the FGD, they will generally include the following: ● Everyone is encouraged to share their ideas, and the FGD is stronger if everyone participates. ● There are no wrong answers, and everyone’s perspective is equally valued. ● The ideas shared during the FGD should not be shared outside the FGD with non-participants in order to respect participants’ privacy. ● Disagreements about ideas can be valuable and productive, but we ask that participants be respectful and courteous to one another. After establishing these ground rules, the moderator should ask if there are any questions or concerns participants have, and these issues should be addressed and consensus reached as a group before moving on. 69 FGD questions by respondent category 1. Farmer groups (including Strawberry and S&L groups) Part I: Group Details [Answers to questions 1.1-1.6 should be filled in according to project documentation before the FGD. Corrections to project information can then be made via the FGD.] Questio n No. Question Response Notes 1.1 When did this group start working with the Avansa Agrikultura Project? __ __ __ __ [Year] __ __ [Month] 1.2 How many members are in this group? __ __ __ [Total] __ __ __ [Men] __ __ __ [Women] __ __ __ [Youth] 1.3 Do you pay membership fees? If yes, how much and how often? [Document in “notes” column”.] Y N 1.4 Are there elected leaders? Y N 1.5 Does your group work with/participate in the regional horticultural association? Y N 1.6 Do you keep records of crop production and sales? Y N 1.7 Please describe how this group functions, and the role that each of you play. (EQ1, EQ2) [Facilitator to probe for the role that women, men and youth play in the following: ● Group operations (record keeping, decision-making) ● Farming activities (planting, maintaining/growing, harvesting, transport, selling – as applicable to the particular group) ● Avansa Ag activities (training, site visits, information sessions)] Part II: Experience with Avansa Ag II.1 Please discuss/describe how your group first began working with Avansa Ag. Who did you first interact with from Avansa Ag? Were there any challenges to group formation/start-up/initial work with Avansa Ag [as opposed to other donors the group may have worked with in the past]? If yes, please explain. [Probe: Do you understand the project and its goals?] (EQ1, EQ4) 70 II.2 Please discuss and list the activities/trainings you participated in under Avansa Ag since your group was formed. [Ask participants to list topics of the trainings/activities – noting that individuals may have participated in different trainings/activities than others. Facilitators should have a list of these trainings/activities from project documents, and should use the FGD to confirm data. Make sure to check whether both men, women, and youth FGD participants participated in activities.] (EQ1, EQ2) We want to ask you a few more questions about these activities/trainings. [For each of the identified activities/trainings, probe about the following:] Training/Activity 1 2 3 II.3 Please discuss if this activity met your needs as a group? If yes, why and how? (EQ1, EQ3) [Probe: Did you believe the facilitator/trainer was an expert on the given topic? Did the topic relate to a need you had as a group at the time? Did the training teach you something new? Did you find the method being discussed effective/useful to your group?] II.4 Were you provided materials/guides/handouts for this activity? If yes, were they provided for each participant, or the group overall? Do you still have these materials? Do you still use/reference these materials? (EQ1, EQ4) II.5 Please discuss the extent to which you feel confident in using the knowledge you gained from this training/activity? Why or why not? Can you provide examples of how you have used this knowledge as a group since the training? (EQ1, EQ4) II.6 For farmers only: this activity/training, did you visit/learn from a demonstration farm/plot/site? If yes, was this helpful? If yes, why? If not, why not? (EQ1, EQ4) II.7 For saving and loan groups only: for this activity did you talk with other groups? If yes, was this helpful? If yes, why? If not, why not? (EQ1, EQ4) II.8 How does Avansa Ag communicate with you as a group, to inform you of activities/trainings/program news? How often do you communicate with someone from Avansa Ag? Do you have the information/receive the communication you need as a group to actively participate in/benefit from the project (for example, do you know about upcoming trainings or new focus areas for year three of the project)? (EQ4) II.9 Have you ever received data/feedback from the project about your group’s progress/achievements? If yes, what have you learned through the presentations? Did this information help you, and if yes, how? [This could be data/feedback from the baseline study, from the annual survey, assessments, from ongoing monitoring, etc] (EQ4) II.10 Please list the benefits/advantages, if any, of being a part of this Special Horticulture Area (SHA). Please list the costs/disadvantages, if any, of being a part of this SHA. (EQ1, EQ2, EQ3) [Probe as to whether these benefits/costs are experienced differently by different individuals in 71 the group.] [For Savings and Loans Groups only] II.11 Does this group keep cash (dollars) in your homes and/or deposit money? Please describe the reasons behind these decisions. II.12 Does this group provide loans? If yes, for what amounts (on average)? What percentage interest do you charge (per year/month)? II.13 How are the loans being used, and why whom? II.14 Do borrowers repay on time and according to loan specifications? If not, why not? Part III: Group Engagement/Activities in the Ag Sector (EQ1, EQ3) For each of the items in the table below [to be displayed on a flipchart or poster board during the FGD], we would like to discuss before Avansa Ag and now (during Avansa Ag). [Note that for groups started under Avansa Ag, only fill out the ‘Avansa Ag’ details. The moderator should document any differences/changes between pre-Avansa Ag and Avansa Ag, and probe for explanations as to the cause of these differences (see final column).] Pre-Avansa Ag Avansa Ag Explanation of different/change (if any is reported) 3.1 Group Operations [see details collected on group in Part 1, but probe about any changes that have happened related to group operations] [Probe: Does this group interact with MAF regularly? How does the group make decisions? Who records crop data/production results?] 3.2 Crops [Probe: Please describe crops grown and quality of crops] 3.3 Inputs [Probe: Has the group engaged with input providers with Avansa Ag support? Has the quality of the inputs changed?] 3.4 Farming techniques/technologies 3.5 Financing (loans, credit) 3.6 Buyers/Selling process (from harvesting and collection to transport and sale) [Probe: Does the group have formal agreements with buyers/traders?] 72 3.7 Price of crops/produce 3.8 Profit from group sales [Probe: Group and individual profit/monthly earning] 3.9 Number and role of women Probe: number of women belonging to the group Role of women (production, sales) 3.10 Which crops are the most profitable, and why? 3.11 If differences are noted in use of inputs above: What safety precautions, if any, do you use when using these inputs? Did you follow these precautions before your engagement with Avansa Ag? 3.12 Have you had crops/produce rejected by buyers/traders? If yes, please explain. Part IV: The Future [Sustainability] (EQ5) 4.1 Will this group continue to operate in the future? Why or why not? (EQ5) 4.2 Will your current group crops/input use/selling arrangement continue in the future [current practices as described previously under ‘Avansa Ag’]? Why or why not? (EQ5) 4.3 What are the biggest obstacles you face (if any) to increasing your productivity as a group in the future? [Probe: Are there key inputs/knowledge that you do not have access to that you believe you need in order to increase productivity? Are there activities that you need/requested that you have not received?] (EQ1, EQ3, EQ5) 4.4 Have you shared what you have learned from Avansa Ag with others? Please explain. (EQ5) 4.5 Is there anything else you would like to tell us about your involvement with Avansa Ag that we did not ask you about? 73 2. Natural Resource Management Groups Part I: Group Details [Answers to questions 1.1-1.6 should be filled in according to project documentation before the FGD. Corrections to project information can then be made via the FGD.] Questio n No. Question Response Notes 1.1 When did this group start working with the Avansa Agrikultura Project? __ __ __ __ [Year] __ __ [Month] 1.2 How many members are in this group? __ __ __ [Total] __ __ __ [Men] __ __ __ [Women] __ __ __ [Youth] 1.3 Do you pay membership fees? If yes, how much and how often? [Document in “notes” column”.] Y N 1.4 Are there elected leaders? Y N 1.5 Does your group work with/participate in the regional horticultural association? Y N 1.6 Do you keep records of crop production and sales? Y N 1.7 Please describe how this group functions, and the role that each of you play. (EQ1, EQ2) [Facilitator to probe for the role that women, men and youth play in the following: ● Group operations (record keeping, decision-making) ● Farming activities if any (planting, maintaining/growing, harvesting, transport, selling – as applicable to the particular group) ● Avansa Ag activities (training, site visits, information sessions)] Part II: Experience with Avansa Ag 2.1 Please discuss/describe how your group first began working with Avansa Ag. Who did you first interact with from Avansa Ag? Were there any challenges to group formation/start-up/initial work with Avansa Ag [as opposed to other donors the group may have worked with in the past]? If yes, please explain. [Probe: Do you understand the project and its goals?] (EQ1, EQ4) 74 2.2 Please discuss and list the activities/trainings you participated in under Avansa Ag since your group was formed. [Ask participants to list topics of the trainings/activities – noting that individuals may have participated in different trainings/activities than others. Facilitators should have a list of these trainings/activities from project documents, and should use the FGD to confirm data. Make sure to check whether both men, women, and youth FGD participants participated in activities.] (EQ1, EQ2) We want to ask you a few more questions about these activities/trainings. [For each of the identified activities/trainings, probe about the following:] Training/Activity 1 2 3 2.3 Please discuss to the extent this activity met your needs as a group? If yes, why and how? (EQ1, EQ3) [Probe: Did you believe the facilitator/trainer was an expert on the given topic? Did the topic relate to a need you had as a group at the time? Did the training teach you something new? Did you find the method being discussed effective/useful to your group?] 2.4 Were you provided materials/guides/handouts for this activity? If yes, were they provided for each participant, or the group overall? Do you still have these materials? Do you still use/reference these materials? (EQ1, EQ4) 2.5 Please discuss the extent to which you feel confident using the knowledge you gained from this training/activity? Why or why not? Can you provide examples of how you have used this knowledge as a group since the training? (EQ1, EQ4) 2.6 For this activity/training, did you visit/learn from a another nursery/demonstration farm/plot/site? If yes, was this helpful? If yes, why? If not, why not? (EQ1, EQ4) 2.7 How does Avansa Ag communicate with you as a group, to inform you of activities/trainings/program news? How often do you communicate with someone from Avansa Ag? Do you have the information/receive the communication you need as a group to actively participate in/benefit from the project (for example, do you know about upcoming trainings or new focus areas for year three of the project)? (EQ4) 2.8 Have you ever received data/feedback from the project about your group’s progress/achievements? If yes, what have you learned through the presentations? Did this information help you, and if yes, how? [This could be data/feedback from the baseline study, from the annual survey, assessments, from ongoing monitoring, etc] (EQ4) Part III: Group Engagement/Activities in the Ag Sector (EQ1, EQ3) For each of the items in the table below [to be displayed on a flipchart or poster board during the FGD], we would like to discuss before Avansa Ag and now (during Avansa Ag). [Note that for groups started under Avansa Ag, only fill out the ‘Avansa Ag’ details. The moderator should document any differences/changes between pre-Avansa Ag and Avansa Ag, and probe for explanations as to the cause of these differences (see final column).] 75 Pre-Avansa Ag Avansa Ag Explanation of different/change (if any is reported) 3.1 Group Operations [see details collected on group in Part 1, but probe about any changes that have happened related to group operations] [Probe: Does this group interact with MAF regularly? How does the group make decisions? Who records crop data/production results?] 3.2 Plant species / crops [Probe: Please describe crops grown and quality of plants/trees planted] 3.3 Techniques/technologies 3.4 Buyers/Selling process (only if soe product is sold) (from harvesting and collection to transport and sale) [Probe: Does the group have formal agreements with buyers/traders?] 3.5 Processing/ transformation/post-harvest handling 3.6 Price of products (to be discussed only if products are sold) 3.7 Profit from group sales (to be discussed only if products are sold) [Probe: Group and individual profit/monthly earning] 3.8 Which plant species /crops are the most profitable, and why? 3.9 If differences are noted in use of inputs above: What safety precautions, if any, do you use when using these inputs? Did you follow these precautions before your engagement with Avansa Ag? 3.10 Have you had products rejected by buyers/traders [to be asked only if products are sold]? If yes, please explain. Part IV: The Future [Sustainability] (EQ5) 4.1 Will this group continue to operate in the future? Why or why not? (EQ5) 76 4.2 Will your current group crops/input use/selling arrangement continue in the future [current practices as described previously under ‘Avansa Ag’]? Why or why not? (EQ5) 4.3 What are the biggest obstacles you face (if any) to you activities as a group in the future? [Probe: Are there key inputs/knowledge that you do not have access to that you believe you need in order to increase productivity? Are there activities that you need/requested that you have not received?] (EQ1, EQ3, EQ5) 4.4 Have you shared what you have learned from Avansa Ag with others? Please explain. (EQ5) 4.5 Is there anything else you would like to tell us about your involvement with Avansa Ag that we did not ask you about? 77 3. Nutrition Groups Part I: Group Details [Answers to questions 1.1-1.6 should be filled in according to project documentation before the FGD. Corrections to project information can then be made via the FGD.] Questio n No. Question Response Notes 1.1 When did this group start working with the Avansa Agrikultura Project? __ __ __ __ [Year] __ __ [Month] 1.2 How many members are in this group? __ __ __ [Total] __ __ __ [Men] __ __ __ [Women] __ __ __ [Youth] 1.3 Do you pay membership fees? If yes, how much and how often? [Document in “notes” column”.] Y N 1.4 Are there elected leaders? Y N 1.5 Does your group work with/participate in the regional horticultural association? Y N 1.6 Do you keep records of crop production and sales? Y N 1.7 Please describe how this group functions, and the role that each of you play. (EQ1, EQ2) [Facilitator to probe for the role that women, men and youth play in the following: ● Group operations (record keeping, decision-making) ● Farming activities if any (planting, maintaining/growing, harvesting, transport, selling – as applicable to the particular group) ● Avansa Ag activities (training, site visits, information sessions)] Part II: Experience with Avansa Ag 2.1 Please discuss/describe how your group first began working with Avansa Ag. Who did you first interact with from Avansa Ag? Were there any challenges to group formation/start-up/initial work with Avansa Ag [as opposed to other donors the group may have worked with in the past]? If yes, please explain. [Probe: Do you understand the project and its goals?] (EQ1, EQ4) 78 2.2 Please discuss and list the activities/trainings you participated in under Avansa Ag since your group was formed. [Ask participants to list topics of the trainings/activities – noting that individuals may have participated in different trainings/activities than others. Facilitators should have a list of these trainings/activities from project documents, and should use the FGD to confirm data. Make sure to check whether both men, women, and youth FGD participants participated in activities.] (EQ1, EQ2) We want to ask you a few more questions about these activities/trainings. [For each of the identified activities/trainings, probe about the following:] Training/Activity 1 2 3 2.3 Please discuss if this activity met your needs as a group/community? If yes, why and how? (EQ1, EQ3) [Probe: Did you believe the facilitator/trainer was an expert on the given topic? Did the topic relate to a need you had as a group at the time? Did the training teach you something new? Did you find the method being discussed effective/useful to your group?] 2.4 Were you provided materials/guides/handouts for this activity? If yes, were they provided for each participant, or the group overall? Do you still have these materials? Do you still use/reference these materials? (EQ1, EQ4) 2.5 Please discuss the extent to which you feel confident using the knowledge you gained from this training/activity? Why or why not? Can you provide examples of how you have used this knowledge as a group since the training? (EQ1, EQ4) 2.6 For this activity/training, did you visit/learn from a another nutrition group? If yes, was this helpful? If yes, why? If not, why not? (EQ1, EQ4) 2.7 How does Avansa Ag communicate with you as a group, to inform you of activities/trainings/program news? How often do you communicate with someone from Avansa Ag? Do you have the information/receive the communication you need as a group to actively participate in/benefit from the project (for example, do you know about upcoming trainings or new focus areas for year three of the project)? (EQ4) 2.8 Have you ever received data/feedback from the project about your group’s progress/achievements? If yes, what have you learned through the presentations? Did this information help you, and if yes, how? [This could be data/feedback from the baseline study, from the annual survey, assessments, from ongoing monitoring, etc] (EQ4) 2.9 Please list the benefits/advantages, if any, of being a part of this Special Horticulture Area (SHA). Please list the costs/disadvantages, if any, of being a part of this SHA. (EQ1, EQ2, EQ3) [Probe as to whether these benefits/costs are experienced differently by different individuals in the group.] Part III: Group Engagement/Activities in Nutrition (EQ1, EQ3) For each of the items in the table below [to be displayed on a flipchart or poster board during the FGD], we would like to discuss before Avansa Ag and now (during Avansa Ag). [Note that for groups 79 started under Avansa Ag, only fill out the ‘Avansa Ag’ details. The moderator should document any differences/changes between pre-Avansa Ag and Avansa Ag, and probe for explanations as to the cause of these differences (see final column).] Pre-Avansa Ag Avansa Ag Explanation of different/change (if any is reported) 3.1 Group Operations [see details collected on group in Part 1, but probe about any changes that have happened related to group operations] [Probe: How does the group make decisions?] 3.2 Crops [Probe: Please describe crops grown and quality of crops] 3.3 Availability of vegetable during the year [Probe: Please describe the months during which you have less vegetables to eat] 3.4 Farming techniques/technologies 3.5 Financing (loans, credit, loan and saving groups) 3.6 Buyers/Selling process (from harvesting and collection to transport and sale) [Probe: Does the group have formal agreements with buyers/traders?] 3.7 Price of crops/produce 3.8 Profit from group sales [Probe: Group and individual profit/monthly earning] 3.9 Diet: consumption [Probe: ● Please describe how you have changed your diet ● Please describe how you have changed your diet throughout the year ● Which are the months when the availability of food is lower? ● Please describe how you prepare meals ● Please describe how you conserve meals before and after cooking (aihan seguro) ● 3.10a Diet: for children, pregnant and 80 lactating women [Probe: ● Has everyone in your family changed their diet, or only certain members? ● Please describe how 6-23 month old children are fed ● Please describe how porridge (sassoro) is prepared (show picture) ● Please describe how pregnant and lactating mother are fed (prompting question: is there any food that lactating mothers should not eat?) 3.10b Porridge (sassoro): mothers and fathers of 6-23 month old children are presented two couple of pictures and are asked to identify the picture that better resemble the porridge prepared on a daily basis. 81 First couple: 82 Second couple 83 84 3.11a Pocket voting: Women only are asked to put a bean in the relevant box (the box is hidden behind facilitator/note takers). Facilitator then count the number of beans in each box YES NO Only for women: did you wash your hands yesterday before preparing dinner? 3.11b Hygiene/Sanitation ● Please describe when you wash your hands Part III: The Future [Sustainability] (EQ5) 4.1 Will this group continue to operate in the future? Why or why not? (EQ5) 4.2 What are the biggest obstacles you face (if any) to increasing your productivity as a group in the future? [Probe: Are there key inputs/knowledge that you do not have access to that you believe you need in order to increase productivity? Are there activities that you need/requested that you have not received?] (EQ1, EQ3, EQ5) 4.3 Have you shared what you have learned from Avansa Ag with others? Please explain. (EQ5) 4.4 Is there anything else you would like to tell us about your involvement with Avansa Ag that we did not ask you about? 85 Observations Coversheet and introductory statement Observation Code: Date of the visit: Time Start: Team (Sub-Team A, B): Time End: Observer/interviewer: (s): Location: Category of site: No. of accompanying persons ___________ M _____ F______ Provided Consent: Y or N Notes about Consent Provided (if any): Good morning/afternoon and thank you for meeting with us today. As mentioned during our request, we are working with USAID to conduct a midterm evaluation of the Avansa Agrikultura Project (Avansa Ag). We work for a company called Social Impact. The evaluation is intended to provide an informed assessment of Avansa Ag progress to date and recommend potential modifications to the project, if needed. We may have to ask you some clarifications regarding what we observe. It is very important for you to know that this study will not determine if you will receive any services/support/funding now or in the future. The way you answer our questions will not determine if you will receive any kind of government or NGO assistance now or in the future. There is no direct benefit to you or your community for your participation in this study, besides toward the goal of improving the Avansa Ag project. Our team has had the opportunity to review some background documents to get a better sense of the design and implementation of the project. However, these documents can only tell us so much. We would like to observe the project results in the field, in order to help us better understand how these projects look and function “on the ground.” It is important to understand that while we would like your help in this study, you do not have to take part if you do not want to, and you do not have to answer any request for clarifications questions if you do not feel comfortable doing so. If you decide to take part, your responses will be kept strictly confidential. This means that your name will not be mentioned anywhere in the report, and will not be 86 provided to anyone, including USAID/Carndo/Avansa Ag or anyone in your community. Do you agree if we visit your field/reforestation site? Do you consent to participate? Do you agree by saying “yes,” and participating in this study, you are indicating that you have heard this consent statement, had an opportunity to ask any questions about your participation and voluntarily consent to participate. Observation form/guide This guide will provide structure for observations conducted of NRM groups and reforestation sites, and any other sites identified for selection based on FGD results. Reforestation sites observations Responses Number of planted seedlings Extension of the reforestation site Month and year when seedlings were planed ________ [Year] _____[Month] Planted spices Irrigation __________[________________________[Year] __________________________________[How many times] __________________________________[Irrigation technique] Observations General impressions Average distance among planted seedlings Number of living seedlings (counted in a radium of 3 meters at interval of 10 meters) Number of dead seedlings (counted in a radium of 3 meters at interval of 10 meters) 87 Vegetable production Responses Extension of the planted area Planted vegetables Substractum used for seedling production (soil, cocopeat) Observations General impressions Availability of water (tank) Consistency in the height of plants Presence of pests Deficiencies in nutrition Status of seedling production (and material used) Status of tunnels (if any) 88 ANNEX D: DOCUMENTS REVIEWED Documents referenced/cited in the evaluation report: 1. Asian Development Bank. Sector Assessment (Summary): Agriculture, Natural Resources, and Rural Development. ADB Country Partnership Strategy: Timor-Leste 2016-2010. 2015. https://www.adb.org/sites/default/files/linked-documents/cps-tim-2016-2020-ssa-07.pdf 2. Government of Timor Leste. Timor-Leste National Accounts 2000-2013. Ministry of Finance, General Directorate of Statistics. 2015. 3. Indian Institute of Spices Research, Kozhikode- 673 012. Good Agricultural Practice (GAP) for propagation of black pepper (Piper nigrum L.) in nurseries. Kerala, India. 4. International Food Policy Research Institute. Global Hunger Index. 2016. http://ghi.ifpri.org/. 5. Oot. L. Sethuraman, K. Ross, J. Sommerfelt, E. (2016) The Effect of Chronic Malnutrition (Stunting) on Learning Ability, a Measure of Human Capital: A Model in PROFILES for Country-Level Advocacy. Technical Brief. Food and Nutriton Technical Assistance III Project. https://www.fantaproject.org/sites/default/files/resources/PROFILES-brief-stunting-learning￾Feb2016.pdf 6. AECID. Studies about Gender in Five Districts of Timor Leste. Research Developed by SEPI and RDPL II with SEPI's Methodology. 2012. 7. Bollinger, K. Gender Analysis: Private Sector in Timor Leste, USAID/Timor-Leste. 2012 8. GOTL. Labor Force Survey 2013. http://www.statistics.gov.tl/wp- content/uploads/2015/04/LFS_2013_ENGLISH_VERSION.pdf 9. Patton, M.Q. Qualitative Evaluation and Research Methods. Sage Publications, Newbury Park CA. 1990. 10. UNDP. Human Development Report. 2016. http://hdr.undp.org/en/2016-report. 11. World Bank. Data Bank. 2016. http://databank.worldbank.org/data/reports.aspx?source=2&country=TLS# 12. Asian Development Bank (2014), Timor-Leste country gender assessment. https://www.adb.org/documents/timor-leste-country-gender-assessment 10. MDF. Sector Assessment Report and Sector Growth Strategy for Agribusiness, Processing and Rural Distribution, Market Development Facility - Timor-Leste. Working Document. http://marketdevelopmentfacility.org/wp￾content/uploads/2012/12/140916_Agribusiness-Processing-and-Rural-Distribution.pdf. 2013 11. GOTL. Zero Hunger Challenge. National Action Plan for a Hunger and Malnutrition Free Timor-Leste. National Council for Food Security, Sovereignty and Nutrition in Timor-Leste, Dili http://www.fao.org/fileadmin/templates/rap/files/ESP/Action_plan_for_a_hunger___malnutrit ion_free_Timor-Leste__update_.pdf. 2014. 12. GOTL. Agriculture Sector Development. Medium Term Operational Plan (2014- 2018). Ministry of Agriculture and Fishery. National Directorate of Policy and Planning http://www.gafspfund.org/sites/gafspfund.org/files/Documents/6c.%20Timor￾Leste_Ministry%20of%20Agriculture%20and%20Fisheries%20Medium%20Term%20Operation%2 0Plan%20%28full%29.pdf 13. GOTL. Timor-Leste Strategic Development Plan 2011-2030 http://timor-leste.gov.tl/wp-content/uploads/2011/07/Timor-Leste-Strategic-Plan-2011- 20301.pdf AVANSA Project Documents: 89 1. Antara, N. S. Integrated Pest Management (IPM) Concept. USAID’s Avansa Agrikultura Project: Dili, Timor Leste. PowerPoint. 2. McDermott, A. Gender Evaluation STTA Assignment. USAID’s Avansa Agrikultura Project. 2016. 3. Harlow, D. E. Pesticide Evaluation Report and Safer Use Action Plan (PERSUAP). USAID’s Avansa Agrikultura Project. 2015 4. Lopes, Cesaltino and Smith, Katherine. Climate Smart Agriculture Protects Your Land, Protects Your Resources, Protects Your Crops, Increases Your Yields. USAID’s Avansa Agrikultura Project. 5. USAID’s Avansa Agrikultura Project. 2.3 Selection of Sucos for activities in year two. 6. USAID’s Avansa Agrikultura Project. Approved Grant Applications Record. 7. USAID’s Avansa Agrikultura Project. Annual Agribusiness Questionnaire Modules for Farmers Final Survey. 8. USAID’s Avansa Agrikultura Project. Annual Agribusiness Survey Questionnaire Modules for USAID’s Avansa Agrikultura Project Annual Household Survey: Nutrition. 9. USAID’s Avansa Agrikultura Project. Avansa Agrikultura Baseline Survey. PowerPoint. 10. USAID’s Avansa Agrikultura Project. Avansa Agrikultura PMEP Indicators Table. 2016. 11. USAID’s Avansa Agrikultura Project. Avansa Agrikultura SOW. Section C – Statement of Work. 12. USAID’s Avansa Agrikultura Project. Avansa Mid-Term Performance Evaluation: Survey Report. 13. USAID’s Avansa Agrikultura Project. Ten Keys to Success in Annual Fruit and Vegetable Crop Production. 14. USAID’s Avansa Agrikultura Project. Performance Data Review Findings Notes. 15. USAID’s Avansa Agrikultura Project. PMEP Results Quarter 1 and 2 2017. 16. USAID’s Avansa Agrikultura Project. Aileu Monthly Report: December 2016. 17. USAID’s Avansa Agrikultura Project. Aileu Monthly Report: February 2017. 18. USAID’s Avansa Agrikultura Project. Aileu Monthly Report: February 2017. 19. USAID’s Avansa Agrikultura Project. Aileu Monthly Report. January 2017. 20. USAID’s Avansa Agrikultura Project. Aileu Monthly Report. November 2016. 21. USAID’s Avansa Agrikultura Project. Aileu Monthly Report. October 2016. 22. USAID’s Avansa Agrikultura Project. Aileu Monthly Report. September 2016. 23. USAID’s Avansa Agrikultura Project. Ainaro Monthly Report: February 2017. 24. USAID’s Avansa Agrikultura Project. Ainaro Monthly Report: January 2017. 25. USAID’s Avansa Agrikultura Project. Ainaro Monthly Report: November 2016. 26. USAID’s Avansa Agrikultura Project. Fertilizer Import Permit Issuance Standard Operating Procedure Fertilizer Permit Import Application Checklist. 27. USAID’s Avansa Agrikultura Project. Fertilizer Biosecurity Risk Profile. 28. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- 1.4 29. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- 2.1 30. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- 2.2 31. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- 2.4 32. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- 3.1 33. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- 3.2 34. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- 3.3 90 35. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- 4.2 36. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- 4.3 37. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- 5.1 38. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- 5.2 39. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- 5.3 40. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- 5.4 41. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- 6.2 42. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- 6.5 43. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- 6.6a 44. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- 6.6b 45. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- 6.7 46. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- 7.1 47. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- 7.2 48. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- 7.3 49. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- 7.6 50. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- 8.4 51. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- 8.5 52. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- DO 1 53. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- P 2 54. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- P 4 55. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- SB 2.1 56. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- SB 2.2 57. USAID’s Avansa Agrikultura Project. Performance Indicator Reference Sheet- SB 1.4 58. USAID’s Avansa Agrikultura Project. Climate Smart Agriculture Calendar 2017. 2017. 59. USAID’s Avansa Agrikultura Project. Layered Garden Brochure. 60. USAID’s Avansa Agrikultura Project. Minutes from First Project Steering Committee (PSC) Meeting. 61. USAID’s Avansa Agrikultura Project. Newsletter. January-March 2016, Volume III. 2016. 62. USAID’s Avansa Agrikultura Project. Newsletter July-September 2016. 2016. 63. USAID’s Avansa Agrikultura Project. Newsletter October-December 2016. 2016. 64. USAID’s Avansa Agrikultura Project. Standard Operating Procedures – Sampling and Inspecting Fertilizers. 65. USAID’s Avansa Agrikultura Project. USAID/Avansa Agrikultura Performance Indicator Reference Sheet- 1.1 66. USAID’s Avansa Agrikultura Project. USAID/Avansa Agrikultura Performance Indicator Reference Sheet- 1.0.1 67. USAID/Timor-Leste. Project Results in Year 2 Implementation. Washington, D.C.: USAID. Print. 68. USAID/Timor Leste. Rapid Partnerships Appraisal (RPA) Report. Washington, D.C.: USAID. 2015. Print. 69. USAID/Timor-Leste. Avansa M&E Task Order: CLA Agenda and Action Plan. Washington, D.C.: USAID. 2015. Print. 70. USAID/Timor-Leste. Avansa M&E Task Order: CLA Needs Assessment Report. Washington, D.C.: USAID. 2015. Print. 91 71. USAID’s Avansa Agrikultura Project. Minutes from First Project Steering Committee (PSC) Meeting. 72. USAID’s Avansa Agrikultura Project. Second Project Steering Committee Meeting (PSC) Minutes. 2016. 73. USAID/Timor-Leste. Baseline Survey Report for the Avansa Agrikultura Project. Washington, D.C.: USAID. 2016. Print. 74. USAID/Timor-Leste. Country Development Cooperation Strategy (CDCS) 2013-2018. Washington, D.C.: USAID. Print. 75. USAID/Timor-Leste. Climate Change Adaptation and Disaster Risk Reduction for USAID’s Avansa Agrikultura Project in Timor-Leste Draft Report. Washington, D.C.: USAID. 2015. Print. 76. USAID/Timor-Leste. Draft Performance Management and Evaluation Plan (PMEP). USAID’s Avansa Agrikultura Project. Washington, D.C.: USAID. 2015. Print. 77. USAID/Timor-Leste. Economic Viability of Organic Production in Timor-Leste. USAID’s Avansa Agrikultura Project. Washington D.C.: USAID. 2017. Print. 78. USAID/Timor-Leste. Final Performance Evaluation of USAID’s Developing Agricultural Communities. Washington, D.C.: USAID. Print. 79. USAID/Timor-Leste. First Year Work Plan: April 1, 2015-September 30, 2015. USAID’s Avansa Agrikultura Project. Washington, D.C.: USAID. 2015. Print. 80. USAID/Timor-Leste. Life of Project Work Plan: April 1, 2015-March 30, 2020. USAID’s Avansa Agrikultura Project. Washington, D.C.: USAID. 2015. Print. 81. USAID/Timor-Leste. Quarterly Report: April – June 2016. USAID’s Avansa Agrikultura Project. Washington, D.C.: USAID. 2016. Print. 82. USAID/Timor-Leste. Success Story: Accessing Markets and Increasing Good Nutrition. Washington, D.C.: USAID. Print. 83. USAID/Timor-Leste. Success Story: Improving Health and Nutrition through Family Garden Production. Washington, D.C.: USAID. Print. 84. USAID/Timor-Leste. Second Year Work Plan: October 1, 2015-September 30, 2016. USAID’s Avansa Agrikultura Project. Washington, D.C.: USAID. 2015. Print. 85. USAID/Timor-Leste. Third Year Work Plan: October 1, 2016-September 30, 2017. USAID’s Avansa Agrikultura Project. Washington, D.C.: USAID. 2016. Print. 86. USAID/Timor-Leste. Timor-Leste Agriculture Market Systems Analysis Consultancy Report. Washington, D.C.: USAID. 2015. Print. 87. USAID/Timor-Leste. Working Together for Women’s Economic Empowerment. Washington, D.C.: USAID. Print. 88. USAID/Timor-Leste. Year 1 Annual Progress Report: April 1, 2015-September 30, 2015. USAID’s Avansa Agrikultura Project. Washington, D.C: USAID. 2015. Print. 89. USAID/Timor-Leste. Year 2 Annual Progress Report: October 1, 2015-September 30, 2015. USAID’s Avansa Agrikultura Project. Washington, D.C.: USAID. 2015. Print. 90. USAID/Timor-Leste. Avansa Ag Policy Reform Matrix, September 2015 91. USAID/Timor-Leste Avansa Ag. (2017) Economic Viability of Organic Production in Timor-Leste. USAID’s Avansa Agrikultura Project, January 10, 2017 92 ANNEX E: EVALUATION RESPONDENTS AND OBSERVATION SITES Key Informants Name Sex Organization Position Dina Karic Female Cardno International COP W. Jeff Gucker Male Cardno International DCOP Kristopher Rowell Male USAID COR Avansa Ag; Agriculture Officer Flavia A Da Silva Female USAID Former COR Avansa Ag Mark Henderson Male USAID Director of Office of Economic Growth, Former COR of Avansa Ag Zulmira Alvez dos Reis Female Cardno International Dili Senior Municipal Coordinator Custodio Malidao Male Cardno International Dili Municipal Coordinator Adelino Fredy Fximenes Male Cardno International Dili Intern Daniel Charles Fyffe Male Cardno International STTA Irrigation, Hort. Agro. And Farmer Md Ahsan Ullah Bhuiyan Male Moris Rasik Managing Director Angelo B.C. Soares Male Kaebauk Investimentu No Finansas, SA (KIF) Chief Executive Officer Kamran Saeed Male Kaebauk Investimentu No Finansas, SA (KIF) Finance Manager Americo Brito Male Ministry of Agriculture, National Agriculture, Horticulture Extension and Protection Plan Xisto Martins Male National Horticultural Association; Raebia Timor Leste Director; Executive Director Nilton A.L. Aniceto Male Loja Agi Agricultura; Agro￾Input Business Association Technical Adviser/Trainer; President Juergen W.E. Glembotzki Male Timor Organic Director Clarence Lim H.M. & Eligito Barros Male Kmanek Trading Director and Manager Shalendra Rattan Male Vinod Patel Branch Manager Jagath Prasanna Male Vinod Patel Accountant Adi Setiyadi Male W4 Supermarket Manager Boby Lay Male Timor Global Director Hergui Luina Fernandes Albes Male Nova Casa Fresca Director Valente Quintao Male Ministry of Agriculture, National Directorate of Quarantine and Biosecurity National Director Amaro Ximenes Male Ministry of Agriculture, Horticulture National Director Luis do Costa Fernandes Male Cardno International Community Development Coordinator, Output Leader 3 Cesaltino Lopes Male Cardno International CAA/NRM Specialist, Output Leader 2 93 Name Sex Organization Position Heather Beeston Female Cardno International M&E Specialist Katherine Smith Female Cardno International Communications Specialist Jose M.S Costa Male Cardno International Output Leader 1, Governance Advisor Brigido de Sousa Male BNCTL President Sebastiana da C Pereira Female Cardno International Output Leader 8, Gender and Youth Specialist Celestina Lopes da Fonseca Female Cardno International Grants and Subcontract Manager Zoffimo Hanjan Corbafo Male SSG Advisors Partnership and Marketing Adviser, Output Leader 5 Jacob Calar Male Cardno International Business Advisor, Output Leader 6 Osorio C do Rego Male Cardno International Agribusiness Specialist – Output Leader 4 Rosaria Martins Da Cruz Female Cardno International, HIAM Health Output Leader 7, Director of HIAM Health Peter Dougan Male Farm Pro Director Domingos Soares Male Cardno International Municipal Coordinator Jose de Deus Male Ermera MAF Director MAF Ermera Antonio Maia Lay Male Bobonaro MAF Chief of Department Agriculture, Horticulture, Extension, Mechanization, Irrigation, and Food Security Agostinho Bere Male Bobonaro MAF Suco extension worker (Malilait Village) Evangelino Belo Male Cardno International Senior Municipal Coordinator Laurentino Amaral Male Cardno International Municipal Coordinator (Atabae) Roni P da Costa Lopes Male Cardno International DAA (admin and data collection) Domingos Vicente Male Cardno International FSI Aquino P Vicente Male Cardno International FSI Maria Leonelia Female HIAM Health (Cardno International) HIAM Health trainer Richard Holloway Male TOMAK Team Leader Jenny Ikelberg Female TOMAK Value Chain Specialist Joaninha Oliveira Guterres Female TOMAK Value Chain Specialist Beatriz Antonia dos Santos Female IADE Alberthus Benu Male Tuba Rai Metin Program Manager Andy Trisyono Male FAO Consultant Bruno Benavente Male Australian Aid (DFAT) Strategic Agriculture Policy Advisor Carlos de Deus Male Loja Moris Mesa Kios owner José Ranjeli Male Cardno International Senior MC Andre Dos Reis Male Cardno International Intern Teresa Suarez Male Cardno International MC Blacius Maia Male Cardno International MC Galieni Sebastiao Da Costa Male MAF MAF Director Aileu 94 Name Sex Organization Position Silveiro Amaral Barretu Felix Male Ahmusal President Otavio Mendoza Oliveria Male Grupo Recaf President Justino Talan Male Loja Ainaro Owner Salustiano Freitas Pinto Male Cardno International, HIAM Health Senior MC Joao Bosco Cortereal Male Cardno International, HIAM Health Admin and data collection Remigio Pereira Male Nunu Boco President Rodolfo De Araujo Male Self-employed Owner Natalino Marquez Male Tokenela President Abel Antonio Male Grupo Haupi President Lucio Romeo Ruveuri Male MAF MAF Director Ainaro Manule Lopez Da Cruz Male Fundaçao Hader Director Manuel Pereria Male Dilimart Horticultura manager Robert Williams Male AI-Com Team leder Antonio Marcal Male Primary School of Hatulia (Ermera) School Director Berta da Conceicao Lemos Female PUTLEO, Lda. Director Teresinha de Fatima Soares Female DFAT Senior Coordinator-Nutrition 95 FGD Respondents Note: FGD Respondent names have been omitted. Municipality Category Group Name Sex Dili Farmer Inawai Male Dili Farmer Inawai Male Dili Farmer Inawai Male Dili Farmer Inawai Male Dili Farmer Inawai Male Dili Farmer Inawai Male Dili Farmer Inawai Male Dili Farmer Inawai Male Dili Farmer Inawai Male Ermera Farmer Maudio I Male Ermera Farmer Maudio I Male Ermera Farmer Maudio I Male Ermera Farmer Maudio I Male Ermera Farmer Maudio I Male Ermera Farmer Maudio II Female Ermera Farmer Maudio II Male Ermera Farmer Maudio II Male Ermera Farmer Maudio I Male Ermera Farmer Maudio I Male Ermera Farmer Maudio I Male Ermera Strawberry Mertutu Male Ermera Strawberry Mertutu Male Ermera Strawberry Mertutu Male Ermera Strawberry Mertutu Male Ermera Strawberry Mertutu Male Ermera Strawberry Mertutu Female Ermera Strawberry Mertutu Female Ermera Strawberry Mertutu Female Ermera Strawberry Mertutu Male Ermera Strawberry Mertutu Female Ermera Strawberry Mertutu Female Ermera Strawberry Mertutu Female Ermera Strawberry Mertutu Female Ermera Strawberry Mertutu Male Ermera Strawberry Mertutu Female Ermera Strawberry Mertutu Male Ermera Strawberry Mertutu Male 96 Municipality Category Group Name Sex Ermera NRM Fuhak Male Ermera NRM Fuhak Male Ermera NRM Fuhak Female Ermera NRM Fuhak Female Ermera NRM Fuhak Male Ermera NRM Fuhak Female Ermera NRM Fuhak Male Ermera NRM Fuhak Female Ermera NRM Fukak Male Ermera NRM Fuhak Female Ermera NRM Fuhak Male Ermera Nutrition Haburas Derema Male Ermera Nutrition Haburas Derema Male Ermera Nutrition Haburas Derema Male Ermera Nutrition Haburas Derema Male Ermera Nutrition Haburas Derema Male Ermera Nutrition Haburas Derema Male Ermera Nutrition Haburas Derema Female Ermera Nutrition Haburas Derema Male Ermera Nutrition Haburas Derema Female Ermera Nutrition Haburas Derema Male Ermera Nutrition Haburas Derema Male Ermera Nutrition Haburas Derema Female Ermera Nutrition Haburas Derema Male Ermera Nutrition Haburas Derema Female Ermera Nutrition Haburas Derema Female Ermera Nutrition Haburas Derema Male Ermera Nutrition Haburas Derema Female Ermera Savings and Loan Haburas Feto Female Ermera Savings and Loan Haburas Feto Female Ermera Savings and Loan Haburas Feto Female Ermera Savings and Loan Haburas Feto Male Ermera Savings and Loan Haburas Feto Male Ermera Savings and Loan Haburas Feto Male Ermera Savings and Loan Haburas Feto Female Ermera Savings and Loan Haburas Feto Female Ermera Savings and Loan Haburas Feto Female Ermera Savings and Loan Haburas Feto Female Ermera Savings and Loan Haburas Feto Female Ermera Savings and Loan Haburas Feto Female 97 Municipality Category Group Name Sex Bobonaro Farmer Loeleku/Maliana (Demplot) Male Bobonaro Farmer Loeleku/Maliana (Demplot) Male Bobonaro Farmer Loeleku/Maliana (Demplot) Female Bobonaro Farmer Loeleku/Maliana (Demplot) Female Bobonaro Farmer Loeleku/Maliana (Demplot) Female Bobonaro Farmer Loeleku/Maliana (Demplot) Male Bobonaro Farmer Loeleku/Maliana (Demplot) Female Bobonaro Farmer Loeleku/Maliana (Demplot) Female Bobonaro Farmer Loeleku/Maliana (Demplot) Female Bobonaro Farmer Loeleku/Maliana (Demplot) Female Bobonaro NRM Malilait Group (Funan Buras) Male Bobonaro NRM Malilait Group (Funan Buras) Male Bobonaro NRM Malilait Group (Funan Buras) Female Bobonaro NRM Malilait Group (Funan Buras) Female Bobonaro NRM Malilait Group (Funan Buras) Female Bobonaro Farmer Diruaben (Adusanan Ditu Aben) Male Bobonaro Farmer Diruaben (Adusanan Ditu Aben) Male Bobonaro Farmer Diruaben (Adusanan Ditu Aben) Female Bobonaro Farmer Diruaben (Adusanan Ditu Aben) Male Bobonaro Farmer Diruaben (Adusanan Ditu Aben) Male Bobonaro Farmer Diruaben (Adusanan Ditu Aben) Male Bobonaro Farmer Diruaben (Adusanan Ditu Aben) Male Bobonaro Farmer Diruaben (Adusanan Ditu Aben) Female Bobonaro Farmer Habat Female Bobonaro Farmer Habat Male Bobonaro Farmer Habat Female Bobonaro Farmer Habat Female Bobonaro Farmer Habat Female Bobonaro Farmer Habat Female Bobonaro Farmer Habat Female 98 Municipality Category Group Name Sex Bobonaro Farmer Habat Male Bobonaro Farmer Habat Female Bobonaro Farmer Habat Female Bobonaro Farmer Habat Male Bobonaro Farmer Habat Male Bobonaro Farmer Habat Male Bobonaro Farmer Habat Female Bobonaro Farmer Habat Male Bobonaro Farmer Habat Male Bobonaro Farmer Habat Female Bobonaro Farmer Aiasa Male Bobonaro Farmer Aiasa Male Bobonaro Farmer Aiasa Male Bobonaro Farmer Aiasa Male Bobonaro Farmer Aiasa Male Bobonaro Farmer Aiasa Male Bobonaro Farmer Aiasa Male Bobonaro Farmer Aiasa Male Bobonaro Farmer Aiasa Male Bobonaro Farmer Aiasa Male Bobonaro Farmer Matak Female Bobonaro Farmer Matak Male Bobonaro Farmer Matak Female Bobonaro Farmer Matak Female Bobonaro Farmer Matak Female Bobonaro Farmer Matak Male Bobonaro Farmer Matak Female Bobonaro Farmer Matak Female Bobonaro Farmer Matak Female Bobonaro Farmer Matak Female Aileu Farmer Hakiak Moris Female Aileu Farmer Hakiak Moris Female Aileu Farmer Hakiak Moris Female Aileu Farmer Hakiak Moris Female Aileu Farmer Hakiak Moris Female Aileu Farmer Hakiak Moris Male Aileu Farmer Hakiak Moris Male Aileu Farmer Hakiak Moris Male Aileu Farmer Hakiak Moris Male Aileu Farmer Hakiak Moris Male 99 Municipality Category Group Name Sex Aileu Farmer Hakiak Moris Male Aileu Farmer Hakiak Moris Male Aileu Farmer Tasonih Female Aileu Farmer Tasonih Female Aileu Farmer Tasonih Female Aileu Farmer Tasonih Female Aileu Farmer Tasonih Female Aileu Farmer Tasonih Female Aileu Farmer Tasonih Female Aileu Farmer Tasonih Female Aileu Farmer Habelar Fini Male Aileu Farmer Habelar Fini Male Aileu Farmer Habelar Fini Male Aileu Farmer Habelar Fini Female Aileu Farmer Habelar Fini Female Aileu Farmer Habelar Fini Male Aileu Farmer Habelar Fini Male Aileu Farmer Habelar Fini Male Aileu Farmer Habelar Fini Female Aileu Farmer Habelar Fini Female Aileu Farmer Habelar Fini Female Aileu Farmer Habelar Fini Female Aileu Nutrition Be'e Moris Female Aileu Nutrition Be'e Moris Female Aileu Nutrition Be'e Moris Male Aileu Nutrition Be'e Moris Male Aileu Nutrition Be'e Moris Female Aileu Nutrition Be'e Moris Female Aileu Nutrition Be'e Moris Male Aileu Nutrition Be'e Moris Female Aileu Nutrition Be'e Moris Female Aileu Nutrition Be'e Moris Female Aileu Nutrition Be'e Moris Female Aileu NRM Aileu Bamboo Male Aileu NRM Aileu Bamboo Male Aileu NRM Aileu Bamboo Male Aileu NRM Aileu Bamboo Male Aileu NRM Aileu Bamboo Male Aileu NRM Aileu Bamboo Male Aileu NRM Aileu Bamboo Female 100 Municipality Category Group Name Sex Aileu NRM Aileu Bamboo Female Ainaro Farmer Debo Leuk Female Ainaro Farmer Debo Leuk Female Ainaro Farmer Debo Leuk Male Ainaro Farmer Debo Leuk Male Ainaro Farmer Debo Leuk Female Ainaro Farmer Debo Leuk Female Ainaro Farmer Debo Leuk Female Ainaro Farmer Debo Leuk Female Ainaro Farmer Debo Leuk Female Ainaro Farmer Debo Leuk Male Ainaro Farmer Debo Leuk Male Ainaro Farmer Debo Leuk Male Ainaro Farmer Debo Leuk Female Ainaro Farmer Debo Leuk Male Ainaro Nutrition Ailele Laran Female Ainaro Nutrition Ailele Laran Female Ainaro Nutrition Ailele Laran Male Ainaro Nutrition Ailele Laran Male Ainaro Nutrition Ailele Laran Female Ainaro Nutrition Ailele Laran Female Ainaro Nutrition Ailele Laran Male Ainaro Nutrition Ailele Laran Female Ainaro Nutrition Ailele Laran Male Ainaro Nutrition Ailele Laran Male Ainaro Nutrition Ailele Laran Male Ainaro Nutrition Ailele Laran Male Ainaro Nutrition Ailele Laran Female Ainaro Farmer Haburas Hatoudu Demo Male Ainaro Farmer Haburas Hatoudu Demo Male Ainaro Farmer Haburas Hatoudu Demo Male Ainaro Farmer Haburas Hatoudu Demo Male Ainaro Farmer Haburas Hatoudu Demo Male Ainaro Farmer Haburas Hatoudu Demo Female Ainaro Farmer Haburas Hatoudu Demo Female Ainaro Strawberry Ululeva 1+2 Male Ainaro Farmer Ululeva 1+2 Male Ainaro Farmer Ululeva 1+2 Male Ainaro Farmer Ululeva 1+2 Male Ainaro Farmer Ululeva 1+2 Male 101 Municipality Category Group Name Sex Ainaro Farmer Ululeva 1+2 Male Ainaro Farmer Ululeva 1+2 Male Ainaro Farmer Ululeva 1+2 Male Ainaro Farmer Ululeva 1+2 Female Ainaro Farmer Ululeva 1+2 Female Ainaro Farmer Ululeva 1+2 Female Ainaro Farmer Ululeva 1+2 Female Ermera Nutrition Haburas Bera+Matak Malirin+Haburas Guhei+Tabaco Laran Female Ermera Nutrition Haburas Bera+Matak Malirin+Haburas Guhei+Tabaco Laran Female Ermera Nutrition Haburas Bera+Matak Malirin+Haburas Guhei+Tabaco Laran Female Ermera Nutrition Haburas Bera+Matak Malirin+Haburas Guhei+Tabaco Laran Female Ermera Farmer Haburas Bera+Matak Malirin+Haburas Guhei+Tabaco Laran Female Ermera Farmer Haburas Bera+Matak Malirin+Haburas Guhei+Tabaco Laran Female Ermera Farmer Haburas Bera+Matak Malirin+Haburas Guhei+Tabaco Laran Female Ermera Farmer Haburas Bera+Matak Malirin+Haburas Guhei+Tabaco Laran Female Ermera Farmer Haburas Bera+Matak Malirin+Haburas Guhei+Tabaco Laran Female Ermera Farmer Haburas Bera+Matak Malirin+Haburas Guhei+Tabaco Laran Female Ermera Farmer Haburas Bera+Matak Malirin+Haburas Guhei+Tabaco Laran Female Ermera Farmer Haburas Bera+Matak Malirin+Haburas Guhei+Tabaco Laran Female Ermera Farmer Haburas Bera+Matak Malirin+Haburas Female 102 Municipality Category Group Name Sex Guhei+Tabaco Laran Aileu Farmer Foin Kman+Fo Liman+Hisik Kosar Female Aileu Farmer Foin Kman+Fo Liman+Hisik Kosar Female Aileu Farmer Foin Kman+Fo Liman+Hisik Kosar Female Aileu Farmer Foin Kman+Fo Liman+Hisik Kosar Female Aileu Farmer Foin Kman+Fo Liman+Hisik Kosar Female Aileu Farmer Foin Kman+Fo Liman+Hisik Kosar Female Aileu Farmer Foin Kman+Fo Liman+Hisik Kosar Female 103 Observation Sites Municipality Site Description Dili Demo Plot, Inawai Group Ermera Demo Plot, Maudio I and II Group Ermera Strawberry Farm Bobonaro Farmer Field Day Bobonaro NRM plants (mahogany, seedlings from Avansa) Bobonaro School Garden Competition school Aileu Kios - Loja Moris Mesa Aileu Vegetable production (FGD Hakiak Moris) Aileu Reforestation site Aileu Vegetable production (FGD Tassonish) Aileu Demo plot Aileu Reforestation site (FGD Aileu Bamboo) Ainaro Loja Ainaro Ainaro Vegetable production (FGD Debo Leuk) Ainaro Vegetable production (FGD Ailele Laram - nutrition group) Ainaro Demo plot Hatu-udu Ainaro Vegetable production (group Haupi) Ainaro Strawberry production 104 ANNEX F: DATA COLLECTION SCHEDULE Date KII Activities FGD Activities Municipality 25-Apr-17 KII - Cardno International and USAID N/A Dili 26-Apr-17 KII - Moris Rasik, Kaebauk Investimentu No Finansas, SA (KIF), Ministry of Agriculture, National Agriculture, Horticulture Extension and Protection Plan, National Horticultural Association; Raebia Timor Leste, Loja Agi Agricultura; Agro-Input Business Association, Timor Organic, Kmanek Trading, Vinod Patel N/A Dili 27-Apr-17 KII - W4 Supermarket, Timor Global, Nova Casa Fresca, Ministry of Agriculture, National Directorate of Quarantine and Biosecurity Farmer-Inawai Group in Metinaro Dili 28-Apr-17 KII - Ministry of Agriculture, Horticulture, Cardno International, BNCTL, SSG Advisors N/A Dili 29-Apr-17 KII - Cardno International, HIAM Health N/A Dili 1-May-17 KII - Farm Pro and Loja Moris Mesak Farmer - Maudio I Group in Gleno Ermera Aileu 2-May-17 KII - Cardno International, Aileu MAF Director, AHMUSAL - Strawberry – Mertutu Group in Ermera - NRM – Fuhak Group in Ermera - Farmer-Hakiak Moris Group in Aileu - Farmer/Saving and Loan-Tasonih Group in Aileu Ermera Aileu 3-May-17 KII - Cardno International and Ermera MAF Director - Nutrition- Haburas Derema in Ermera - Savings and Loans￾Haburas Feto in Ermera - Farmer-Habelar Fini Group in Aileu - Nutrition-Bee Moris Group in Aileu Ermera Aileu 4-May-17 KII - Grupo Recaf and Loja Ainaro - Farmer-Loeleku/Maliana (Demplot) - NRM-Aileu Bamboo Group Aileu Maubessi Ainaro 5-May-17 KII - Bobonaro MAF Director, Cardno International and Nunu Boco - NRM- Malilait Group (Funan Buras) - Farmer-Debu Leuk Group in Hatoudo - Nutrition-Ai Lele Laran Bobonaro Ainaro 105 Date KII Activities FGD Activities Municipality Group in Hatoudo 6-May-17 KII - HIAM Health (Cardno International), Trader - Self-employed, Tokenela, Grupo Haupi - Farmer- Diruaben (Adusanan Ditu Aben) in Bobonaro - Farmer - Haburas Hatoudu Demoplot in Hatoudo Bobonaro Ainaro 8-May-17 KII – MAF Director - Farmer-Habat Group in Atabae - Farmer-Aiasa Group in Atabae - Strawberry- Ululeva 1+2 Groups in Maubessi Bobonaro Ainaro 9-May-17 KII – FAO, Fundaçao Hader, Dilimart and AI-Com - Farmer-Matak Group in Atabae Bobonaro Dili 10-May-17 KII - TOMAK , Australian Aid (DFAT), IADE and Tuba Rai Metin N/A Dili 11-12 May￾17 Preliminary data analysis on findings, conclusion and recommendations Dili 15-May-17 Presentation of the preliminary findings (out-brief) to Avansa Agrikultura staff and USAID Dili 24-May-17 N/A Women FGD- Haburas Bera+Matak Malirin+Haburas Guhei+Tabaco Laran Ermera 18-May-17 KII – Director of Primary School of Hatulia (Ermera) N/A Ermera 19-May-17 KII - Follow-up interview with HIAM Health N/A Dili 25-May-17 KII – Women Entrepreneur of PUTLEO, Lda. N/A Ermera 26-May-17 N/A Women FGD-Foin Kman+Foo Liman+Hisik Kosar Aileu 30-May-17 KII – Follow up interview with Director of Primary School of Hatulia (Ermera) N/A Ermera 31-May-17 KII – DFAT Senior Nutrition Coordinator N/A Dili 106 ANNEX G: EVALUATION DESIGN MATRIX Evaluation criteria Evaluation questions related to the criteria Data sources Data collection methods Data analysis method Effectiveness To what extent has Avansa been successful in achieving project sub￾purposes (and associated outputs)? What evidence exists that the value chain approach taken by the project has led to/is leading to increased productivity in the horticulture sector (sub-purpose 1, outputs 1 - 6)? - Policy and enabling environment action matrix - Project progress reports - Project monitoring tools and data - Text of the policies regulations, and administrative procedures - Desk review - KII with government officials and sector experts - Content analysis of KIIs on consultations for the policies regulations, and administrative procedures developed by the project. - Comparison of legal instruments passed by the government with those developed by the project - Project progress reports - Project monitoring tools - vulnerability assessment and climate change adaptation - other project documents -field observations - desk review - KII with government officials and sector experts - Analysis of the actual use of risk mapping tools - Analysis of the coherence of adaptation interventions with vulnerability assessment and climate change adaptation studies - Extension of reforestation initiatives and calculation of survival rates of planted seedlings 1. - Project monitoring tools and data -field observations - desk review - FGDs with farmer groups - Trend analysis of monitored data for the use of sustainable production practices -Analysis of reliability of project monitored data on relevant indicators. - Project monitoring tools and -desk review - Trend and gap analysis of 107 Evaluation criteria Evaluation questions related to the criteria Data sources Data collection methods Data analysis method 2. data - FGDs with farmer groups - KII with banks and MFI - KII with supermarkets, traders, and aggregators monitored data on the development of market linkages and strengthening of farmers groups - Comparative analysis of farmer groups - Content analysis of FGDs and KII for market linkages and strengthening of farmers groups 3. - Project monitoring tools and data - Project progress reports - Desk review - FGDs with famers - KII with input suppliers - KII with representatives of agro￾input business association - Trend and gap analysis of monitored data on input supply - Content analysis of FGDs and KII on input supply To what extent has Avansa been successful in achieving project sub￾purposes (and associated outputs)? What evidence exists that the value chain approach taken by the project has led to/is leading to increased productivity in the horticulture sector (sub-purpose 1, outputs 1 - 6)? - Project monitoring tools and data - Project progress reports - FGD with nutrition groups - KII with individual beneficiary women - Field observations of home gardens and school gardens - Trend and gap analysis of monitored data on nutrition and sanitation practices - Content analysis of FGDs and KII for: - availability of nutrients over the year - changes in behaviors - distribution of nutrient foods within households - Project monitoring tools and data - Project progress reports - FGD with saving and loan groups - FGD with farmer groups - Content analysis of FGD and KII on improvements in the use of assets for productive purposes 108 Evaluation criteria Evaluation questions related to the criteria Data sources Data collection methods Data analysis method Were achievements toward Avansa Ag purposes varied by geography, community type, or gender and vulnerable/marginalized groups? - Project monitoring tools and data - Project progress reports - Desk review - FGD with saving and loan groups - FGD with nutrition groups - FGD with farmer groups -Gap analysis on male versus female beneficiaries -Content analysis of FGD on gender aspects - Project progress reports - Project monitoring data - Desk review - KII with aggregators, supermarket, input suppliers banks, and MFIs - KII with implementer Comparative analysis of geographic areas Relevance To what extent are the activity’s purposes (including outputs/indicators and targets) still valid for current development context in Timor-Leste and lessons from the past months of implementation? - Project progress reports - Project monitoring data - Project budget - DAC final evaluation - Agribusiness and household survey - Project workplans - Project SoW - Desk review - KII with implementer staff and sector experts - FGDs with beneficiaries - Contribution analysis - Analysis of project internal coherence 109 Evaluation criteria Evaluation questions related to the criteria Data sources Data collection methods Data analysis method Efficiency To what extent has the management of Avansa Ag contributed to or inhibited project progress? What opportunities, if any, exist for improvement in management of the activity? -PSC meeting minutes - Project progress reports - PIRs, PMEP - KII with implementer staff - KII with USAID - KII with member of PSC - Desk review - Analysis of the M&E system - Analysis of the grant selection system - Analysis of the adequacy staff arrangements - Analysis of coordination mechanisms with USAID and PSC Sustainability To what extent has Avansa Ag made progress in ensuring sustainability of activities beyond USAID support? Project progress reports -KII with all actors of the horticultural value chain - KII with ministries and other donors - FGD with nutrition groups, farmer groups, saving and loans groups - Desk review -Content analysis of KII and FGDs 110 ANNEX H: DISCLOSURE OF CONFLICTS OF INTEREST A. Acacio Amaral 111 B. Matteo Borzoni 112 C. Eugenia Correia 113 D. Amanda Stek 114 E. Alberto Correia 115 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523