PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT FINAL REPORT September 2021 COVER PHOTO CREDIT: USAID Needs Assessment of the National Visitor Center in Dili, 2018 This publication was prepared independently by Social Impact, Inc. at the request of the United States Agency for International Development. PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT FINAL REPORT September 2021 Submitted to: Charlee Doom, Contracting Officer’s Representative, USAID/Timor-Leste Contract No.: AID-486-I-14-00001/ Task Order No.: 72048619F00001 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ACKNOWLEDGMENTS The evaluation team (ET) members would like to thank the many individuals who provided technical input and advice during the development of the USAID/Timor-Leste Tourism for All Performance Evaluation. The study was a significant undertaking, and its completion would not have been possible without all those who graciously shared their time, insights, and recommendations. The ET especially appreciated the support and guidance of the USAID/Timor-Leste staff including Cândido da Conceição, Flavia da Silva, and Milca Baptista. Additional thanks go to Chief of Party Peter Semone and the Tourism for All team. The ET would also like to acknowledge the support of Social Impact staff Lauren Higgs, Scott Miller, Justin Archangel, Amy Porter, Carly Farver Mphasa, and Holly Dentz, who contributed to the evaluation. The ET extends a final note of sincere gratitude to the many tourism operators, Government of Timor-Leste officials, partner organizations, grantees, donors, and tourism sector association and community members who generously offered their time to speak candidly with our team. The Evaluation Team Sara Currie, Australia Zofimo Corbafo, Timor-Leste Zenny Correia, Timor-Leste Nazario Dos Santos, Timor-Leste CONTENTS EXECUTIVE SUMMARY I INTRODUCTION AND BACKGROUND I METHODOLOGY II FINDINGS AND CONCLUSIONS II RECOMMENDATIONS IV INTRODUCTION AND BACKGROUND 1 EVALUATION PURPOSE AND QUESTIONS 3 EVALUATION PURPOSE & OBJECTIVES 3 EVALUATION QUESTIONS 4 EVALUATION DESIGN AND METHODOLOGY 5 OVERVIEW 5 DATA SOURCES AND METHODS 5 SAMPLING 6 DATA ANALYSIS 8 LIMITATIONS AND BIASES 9 FINDINGS AND CONCLUSIONS 12 EVALUATION QUESTION 1A 12 EVALUATION QUESTION 1B 14 EVALUATION QUESTION 1C 16 EVALUATION QUESTION 1D 20 EVALUATION QUESTION 1E 23 EVALUATION QUESTION 1F 27 EVALUATION QUESTION 2 28 EVALUATION QUESTION 3 31 EVALUATION QUESTION 4 35 EVALUATION QUESTION 5 36 RECOMMENDATIONS 40 ANNEX I: PERFORMANCE EVALUATION SCOPE OF WORK 42 ANNEX II: INFORMATION SOURCES 51 ANNEX III: UN WOMEN RECOMMENDATIONS FOR ENGENDERING TIMOR￾LESTE TOURISM SECTOR 53 ANNEX IV: DATA COLLECTION INSTRUMENTS 55 ANNEX V: EVALUATION TEAM MEMBERS 85 ANNEX VI: DISCLOSURES OF CONFLICT OF INTEREST 87 TABLE OF TABLES Table 1: Project Objectives, Policy Themes, and Results 2 Table 2: Evaluation Questions 4 Table 3: List of Respondents 7 Table 4: Limitations and Risks 9 TABLE OF FIGURES Figure 1: Sample Achieved 5 Figure 2: Desk Review 5 Figure 3: Key Intervention Areas EQ1a 12 Figure 4: Notable TFA Support to Atauro Community 16 Figure 5: Notable TFA Support to Hatobuilico Community 18 Figure 6: Domestic Marketing Campaign Feedback 21 Figure 7: Private Sector Operators’ Views of the Tourism Awards & Activities 23 Figure 8: Stakeholder Suggestions for Improvements to the Grant Program 25 ACRONYMS & KEY TERMS ASTRABEKA Asosiasaun Turizmu Ramelau, Blehitu, and Kablake (Hatobuilico Tourism Association) ATKOMA Asosiasaun Turizmu Koleku Mahanak Ataúro (Atauro Tourism Association) ATM-TL Asosiasaun Turizmu Maritima Iha Timor-Leste (Marine Tourism Association) COVID-19 Coronavirus Disease 2019 DFAT Department of Foreign Affairs and Trade EQ Evaluation Question ET Evaluation Team FCR Findings, Conclusions, and Recommendations FGD Focus Group Discussion FOHATA A cultural and traditional dance group GESI Gender and Social Inclusion GoTL Government of Timor-Leste IADE Institute of Business Support ILO International Labor Organization IP Implementing Partner IRB Internal Review Board KII Key Informant Interview LGBTI Lesbian, gay, bisexual, transgender, and intersex MAF Ministry of Agriculture and Fisheries MEL Monitoring, Evaluation, and Learning MFAT MOU Ministry of Foreign Affairs and Trade Memorandum of Understanding MS Microsoft MTCI Ministry of Tourism, Commerce, and Industry NTP National Tourism Policy PE Performance Evaluation POC PPP Point of Contact Public Private Partnerships ROI Return on investment SDP Strategic Development Plan SI Social Impact, Inc. SMP Sustainable Management Plan SOW Scope of Work TBEH Turizmu Ba Ema Hotu TEDP Tourism Enterprise Development Program TFA Tourism for All Project TL Team Leader UN United Nations UNESCO United Nations Educational, Scientific and Cultural Organization USAID United States Agency for International Development i | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV EXECUTIVE SUMMARY INTRODUCTION AND BACKGROUND The United States Agency for International Development’s (USAID) contracted Chemonics International to implement the Timor-Leste Tourism for All Project (TFA) from 2018 through present. Overall, the TFA aims to support the competitiveness of tourism in Timor-Leste while ensuring that tourism development is inclusive, sustainable, and preserves natural resources and cultural heritage. TFA’s implementation approach is aligned with the Government of Timor-Leste’s (GoTL) Tourism Policy by organizing its activities into five thematic areas: 1-Priority (engaging the government); 2-Prosperity (private sector and business); 3-Protection (environmental and conservation management); 4- Partnership (investment and public-private collaboration); and 5-People (civil society, skills, and communities). The initial geographical focus for the project includes Dili, Atauro Island, and Ramelau, Hatobulico. As the project moved into its second option year,1 USAID commissioned Social Impact, Inc. (SI) to conduct an independent performance evaluation (PE) to assess TFA’s progress towards its stated goals and objectives, as well as to validate assumptions and results against the reality on the ground. The evaluation team (ET) developed five key evaluation questions (EQs) to conduct the evaluation. Table A: Evaluation Questions # EVALUATION QUESTION 1 Has Tourism for All been successful/unsuccessful in the following key intervention areas: a) Working effectively with stakeholders to develop tourism-related policies or regulations? b) Supporting implementation of the National Tourism Policy? c) Establishing effective and sustainable tourism sector associations and advocacy groups? d) Supporting destination marketing domestically and internationally? e) Building partnerships with a reasonable likelihood of stimulating private sector investment in the tourism sector? f) Establishing Sustainable Management Plans within protected areas? 2 Is there evidence that the six key intervention areas outlined in Evaluation Question #1 are likely to grow, scaling up and out? What specific actions will be required to ensure future sustainability after USAID Tourism for All ceased? 3 What specific stakeholder groups2 within the tourism sector have benefited the most from USAID’s Tourism for All interventions? Are there additional stakeholders with whom USAID’s Tourism for All should engage in order to support effective implementation of the National Tourism Policy that could stimulate private sector investment in the tourism sector? 1 This evaluation was intended to take place prior to the execution of the first option year but was postponed due to the COVID-19 pandemic. 2 Stakeholder groups as referenced in evaluation question 3 include the following groups: Timorese women; Timorese youth (ages 18 to 35); Marginalized groups in Timor-Leste such as LGBTI, religious minorities, & persons with disabilities; Tourism associations that Tourism for All established such as ASTRABEKA, ATKOMA, ATM-TL; Cafe and Restaurant working group; Dili Municipal Tourism Authority; Hotel Owners Timor-Leste Association (HOT-L); Government counterparts: Ministry of Tourism, Secretary State of Arts & Culture, Tourism Development Authority; and Dili Institute of Technology. The role of these groups and their integration into the evaluation design as relevant to answer the evaluation questions are elaborated in the discussion of evaluation methods. USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | ii 4 Given efforts by the Government of Timor-Leste and various development partners to more effectively implement the National Tourism Policy or to stimulate private sector investments in the tourism sector, what is USAID’s comparative advantage to achieve or advance these same goals? 5 How has the Tourism for All Project adapted in response to the impact of COVID-19 and how effective has this been, and what additional adjustments should the Project consider in order to further mitigate COVID-19’s impact on the industry? METHODOLOGY The ET used a mixed-methods approach including remote key informant interviews (KII), group interviews, focus group discussions (FGD), a data validation workshop, and a desk review of over 25 TFA progress reports, academic literature, partner and government tourism strategies and economic reports. The ET used purposive sampling in its respondent selection and primary data collection took place in June and July 2021 with stakeholders from USAID, TFA, government, donor partners, community and tourism associations, and private sector and partner organizations (Figure A). Figure A: Data Collection Summary The ET used a Findings, Conclusions, and Recommendations (FCR) matrix to analyze themes by evaluation question. The ET triangulated findings, identified gaps where additional clarification or analysis was necessary, and compared results across stakeholder groups and by sex, where appropriate, to assess convergence and divergence in perspectives. FINDINGS AND CONCLUSIONS EQ1 Tourism for All’s progress in six key intervention areas: a) Tourism policy and regulation. Key TFA initiatives including support towards the establishment of a Tourism Marketing & Promotion Board, improved tourism regulation in Hatobuilico and regulation for the whale tourism industry, are regarded as important interventions and positive for tourism. Private sector stakeholders have welcomed TFA’s efforts in advocating to the government for the establishment of a Tourism Marketing & Promotion Board. Development and implementation of regulation in both Hatobuilico and for whale tourism, have been impeded by limited clarity around the distinct responsibilities of TFA, MTCI and associations. b) Implementation of the National Tourism Policy. TFA has faced ongoing challenges in its support of the implementation of the National Tourism Policy. Progress has been stalled by iii | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV challenges in building a strong working relationship with MTCI, ongoing bureaucratic processes and high-level MTCI personnel changes. c) Sustainable tourism sector associations. TFAs establishment of three main tourism associations – ATKOMA in Atauro, ASTRABEKA in Hatobuilico, and ATM-TL for marine tourism – benefited private sector operators through the formalization of an industry body to enable a collective voice. While ATKOMA is running effectively and sustainably, a lack of clarity around project expectations – including the role of members, community, MTCI and the project – has been problematic for both ASTRABEKA and ATM-TL. Both ASTRABEKA and ATM-TL require additional support, primarily financial, to bridge the gap between the inception of the associations and long-term sustainability. d) International and domestic marketing. The Hau-Nia Timor-Leste domestic tourism marketing campaign is largely well regarded for providing a necessary pivot during the COVID￾19 pandemic, stimulating demand, and increasing revenue for tourism businesses. However, some stakeholders have questioned the need for a separate slogan solely for domestic tourism, while others supported the use of two distinct brands for different audiences. As there is limited measurement of the domestic campaign’s return on investment (ROI), it is not possible to quantify its economic value. TFA’s digital marketing interventions have been well received, particularly by the private sector, while initiatives with MTCI have been impacted by miscommunication. e) Partnerships to stimulate private sector investment. The TFA project has built strong partnerships with private sector operators and has played an important role in tourism advocacy and uniting the sector under a common goal. The grant program has stimulated private sector investment in the industry; however, its full economic benefit is difficult to quantify due to limited Monitoring & Evaluation (M&E). The bureaucratic USAID application process for the grant program may have limited participation, potentially excluding smaller operators. TFAs work towards the establishment of PPPs is regarded as a valuable but also an ambitious initiative given the challenging operating environment in Timor-Leste. f) Sustainable management plans. Data was limited due to low awareness of Sustainable Management Plans among communities and as such the benefit of this effort remains unclear at this stage. EQ2: Sustainability of the six key intervention areas. Sustainability of TFA initiatives varies across the project’s activities. TFA’s work with MTCI, primarily supporting implementation of the NTP, has been impacted by a lack of institutionalization. Conversely, the majority of private sector grantees are confident that work will be sustained following the conclusion of the TFA project. The ATKOMA tourism association in Atauro is more likely to be sustained than ASTRABEKA in Hatobuilico, due to the community’s higher base-level knowledge and understanding of sustainable tourism. Improved coordination with MTCI, donors and other partners could support future sustainability of the TFA project, alongside continued collaboration with the World Bank to enable sustainability of longer-term initiatives such as establishment of the Tourism Marketing & Promotion Board. EQ3: Stakeholder benefit. The private sector is regarded as the main benefactor of TFA through a range of initiatives including tourism events, awards, grants and the domestic tourism campaign. Community members, particularly in Atauro, have also benefited and noted positive improvements in community awareness of tourism and customer service as a result of the project. It was noted that community benefit could be increased through improved support with grant applications and the provision of paid local positions with the project. There is scope for TFA to also improve relationships USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | iv with existing stakeholders – primarily MTCI and Dili-based donors – through increased consultation and a shared agenda. Both women and youth have benefited from the project (their perspectives are detailed in the full report), although it is not possible to assess the overall results and/or benefits for these groups as longitudinal data is not available. EQ4: USAIDs comparative advantage. USAID's TFA investment in Timor-Leste is larger than other donor funded tourism programs in both scope and budget. USAID’s TFA has focused on what the government needs as opposed to always waiting for direction on what it wants. The advantage of the approach is USAID’s ability to tackle larger, bolder initiatives and ‘pull the trigger.’ Stakeholders regard the disadvantages as a perceived lack of collaboration with other Dili-based donors and limited integration with MTCI. EQ5: Adaptation to COVID-19. TFA is regarded as an important project to support tourism recovery from COVID-19. TFA’s ability to adapt and shift in strategy due to COVID-19 was particularly well received among diverse stakeholders, including the pivot to online formats, creating a highly visible domestic marketing campaign for tourism and provision of grants programs to tourism related businesses and associations. Stakeholders noted that TFA’s response could be strengthened via greater coordination with MTCI and other donors, ongoing training opportunities to facilitate ‘knowledge sharing,’ a greater reliance on local expertise to reduce reliance on internet accessibility in remote communities, and a continued focus on digital marketing to prepare for border re-opening. RECOMMENDATIONS FOR USAID: 1. Lead the reinstatement of a donor working group. The group should include the Australian Department of Foreign Affairs and Trade (DFAT), the New Zealand Ministry of Foreign Affairs and Trade (MFAT), the World Bank, and implementing partner agencies to lead the identification of opportunities in tourism and tourism-related industries for a shared agenda and collaborative work with specific responsibilities of each donor and/or implementing partner outlined through a memorandum of understanding (MOU). FOR USAID and IP: 2. Facilitate improved collaboration with the Timorese government including MTCI and other ministries. Consider embedding a staff member within MTCI to further support communication processes and NTP implementation. There is the potential to develop an MOU between TFA and MTCI and other relevant government ministries to clearly delineate roles and responsibilities. USAID/IP could continue to engage the government in private sector-led initiatives, such as establishment of a Tourism Board, and engage the government in coordination with the World Bank to support aligned activities and further work towards inter-ministerial collaboration and regulatory reform. 3. Continue private sector engagement activities and grant program but with key modifications. Consider a more inclusive grant application process, potentially multi-stage, to enable an initial pitch/idea discussion before full application and greater support to potential grant applicants through increased on the ground training/information with TFA staff. Improved M&E could chart grant implementation across all recipients, measuring program objectives against output and (where possible) providing an indicator of ROI. v | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV 4. Continue and scale up training programs for the private sector. USAID should consider extending project remit to offer increased training opportunities for communities outside the main centers of Dili, Beloi, and Ramelau. The IP should develop a strategy to increase and/or improve awareness of current training for Timorese youth including opportunities to collaborate with universities. As well as, continued training and engagement with TFA grantees to enhance sustainability of programs. 5. Provide additional support for key tourism associations (ATKOMA ASTRABEKA and ATM-TL). Improve implementation processes such as assistance with establishing organizational structure and leadership, paid local consultants and additional information around funding and/or grant processes that consider the needs of women, youth, and marginalized groups. A communication strategy could improve collaboration and clearly delineate TFA, MTCI, and community roles and responsibilities. 6. Develop a strategy on how the TFA project has, and can continue to, engage female tourism stakeholders and entrepreneurs. TFA could map and collate its successes working with women to date, identify new opportunities, and refine its strategy for future female engagement that leverages what is working well. This review would consider how the project has supported marginalized women in tourism, how it can increase its support, and which opportunities are available to meaningfully engage women as leaders in the tourism industry. 7. Continue support for domestic marketing campaigns with key modifications to concurrently build brand awareness and increase ROI. Continue work with MTCI to support marketing training to ensure transfer of ownership once the project concludes. Consider increasing focus on the expatriate market to stimulate demand and increase revenue for hotels, the use of domestic campaign imagery to build brand awareness for the international market under the international slogan and incorporation of positive environmental messaging in the campaign. Increased M&E could support measurement of ROI or local awareness building. 8. Engage local expertise. Consider increased partnerships with local organizations, tourism grantees and/or advocates to manage programs during the COVID-era both to ensure understanding of context and reduce reliance on online methods. USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | vi 1 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV INTRODUCTION AND BACKGROUND Often touted as an “unspoiled” or “unsung” tourist destination, Timor-Leste boasts pristine coastal and mountainous scenery and has tremendous potential for a successful tourism industry. The Government of Timor-Leste (GoTL) has recognized tourism as an important potential source of economic growth and diversification and has developed a National Tourism Policy (NTP) which sets the goal of 200 thousand annual tourist arrivals by 2030. This framework aims to focus and align the efforts of government, industry, civil society, academia, and development partners in supporting the implementation of this policy. Through the Timor-Leste Tourism for All Project (TFA), the United States Agency for International Development’s (USAID) investment in the tourism sector in Timor-Leste aims to support the competitiveness of tourism in the country while ensuring that tourism development is inclusive, sustainable, and preserves natural resources and cultural heritage. The stated project goal is to “Expand and improve Timor-Leste’s tourism offerings using an inclusive and sustainable approach.” USAID’s Tourism for All Project is implemented by Chemonics International, began in 2018, and is currently in its first option year of implementation. The project’s implementation approach is meant to align with the GoTL’s Tourism Policy by organizing its activities into the five thematic areas: 1-Priority (engaging the government); 2-Prosperity (private sector and business); 3-Protection (environmental and conservation management); 4-Partnership (investment and public-private collaboration); and 5-People (civil society, skills, and communities). The initial geographical focus for the project includes Dili, Atauro Island, and Ramelau, Hatobulico, which fall under Zone 2 (the Central Tourist Zone) as identified in GoTL’s Strategic Development Plan (SDP) for 2011-2030.3 The project also aims to emphasize five niche market segments representing promising areas of tourism development for Timor-Leste, as identified in the SDP 2011–2030, including (1) eco and marine, (2) historical and cultural, (3) adventure and sport, (4) religious and pilgrimage, and (5) conference and convention. Most of the project activities are located in the priority geographic areas of Atauro Island and the Mt. Ramelau area, Hatobuilico. The theory of change for the Tourism for All project is as follows: ● If institutions, policies, and laws are in place to implement the National Tourism Policy, then Timor-Leste will bolster the confidence of the private sector and improve Timor-Leste’s ability to attract the investment necessary to develop its tourism industry. ● If local communities are engaged to develop their tourism offerings, replicable and sustainable tourism development models may be introduced that promote Timor-Leste’s cultural and natural heritage. By creating socially inclusive partnerships that offer a “win-win” for public, private, and community stakeholders, then participation in Timor-Leste’s tourism sector will expand and increase income for a broad range of Timorese. The project’s activities that align with each of the five thematic policy areas identified by the National Tourism Policy are presented in Table 1. 3 http://timor-leste.gov.tl/wp-content/uploads/2011/07/Timor-Leste-Strategic-Plan-2011-20301.pdf USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 2 Table 1: Project Objectives, Policy Themes, and Results OBJECTIVE 1: FACILITATE IMPLEMENTATION OF THE NATIONAL TOURISM POLICY OBJECTIVE 2: PROMOTE SUSTAINABLE TOURISM INVESTMENTS Priority ● Result 2: Facilitate Approval and Implementation of Key Tourism Framework Documents ● Result 3: Selected Government Agencies Better Able to Perform Their Duties Protection ● Result 4: Support Key Maritime, Terrestrial, and Cultural Sites to Achieve Protected Status ● Result 13: Establish Sustainable Management Plans Within Protected Sites Partnership ● Result 1: Promote Public and Private Participation in Policy Dialogue and Development ● Result 11: Increase Public Investment to Support Private Sector to Develop Tourism ● Result 10: Increase Private Investment in Tourism Development Prosperity ● Result 6: Improved Conditions for Private Sector Investment in Number of Tourism Types ● Result 7: Improved Conditions for Private Sector Investment in Number of Tourism Products ● Result 9: Improve Tourist Satisfaction with Niches/Sites/Products ● Result 12: Strengthen Associations Involved in the Tourism Sector People ● Result 5: Introduce and Establish Models of Sustainable Tourism Development ● Result 8: Improve Business Skills of Selected Beneficiaries A selection of the project’s major achievements to date include the development of strategies and action plans for tourism development at the national and municipal levels; establishment of several professional associations and working groups; signed agreements for tourism development activities; capacity-building and technical assistance to a range of government agencies; organization of the Turizmu Ba Ema Hotu (TBEH) tourism awareness campaign; market research; small grants for tourism enterprises; and multiple well-attended events including an international faith-based tourism convention. To date, the project has collaborated with a wide range of government agencies, private enterprises, development partners, and other non-governmental organizations to further its objectives. 3 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV EVALUATION PURPOSE AND QUESTIONS EVALUATION PURPOSE & OBJECTIVES USAID commissioned Social Impact, Inc. (SI) to conduct an independent performance evaluation (PE) as the project moved into its second option year,0F 4 to assess USAID Tourism for All Project’s (TFA) progress towards its stated goals and objectives, as well as to validate assumptions and results against the reality on the ground. The primary stakeholders to benefit from the findings of the performance evaluation include: ● USAID for the purpose of potentially exercising future option years on the project, ● The USAID TFA team, including Chemonics International and its partners, for adaptive management for the remaining Life of Project if the second option year is granted or for future programming, ● USAID’s external bilateral, regional, and international partners and key stakeholders addressing tourism issues across the Asia-Pacific region to share lessons learned, including participating host country government agencies and other international partners, and ● The U.S. Embassy, Timor-Leste, which is tracking tourism as a critical economic growth sector. The objectives of the TFA PE were to: ● To determine the relevance, efficacy, potential impact, and sustainability of the results achieved. The contractor must undertake a formative assessment of the project to gauge progress made in the implementation of planned activities toward reaching stated goals and objectives. ● To assess the wider project context to validate project assumptions and results against the reality on the ground, based on actual USAID TFA implementation experience to date. The contractor must develop recommendations for adjustments of assumptions and results indicators, as warranted, to enhance project implementation if further option years are exercised. ● To assess the impact of the Coronavirus Disease 2019 (COVID-19) on the tourism industry and the TFA activity and whether there should be any shift in focus should a second option year be awarded. ● To provide an evidentiary basis for determining whether or not to exercise the second option year. 4 This evaluation was intended to take place prior to the execution of the first option year but was postponed due to the COVID-19 pandemic. USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 4 EVALUATION QUESTIONS The ET answered the following evaluation questions: Table 2: Evaluation Questions EQ # EVALUATION QUESTION 1 Has Tourism for All been successful/unsuccessful in the following key intervention areas: a) Working effectively with stakeholders to develop tourism-related policies or regulations? b) Supporting implementation of the National Tourism Policy? c) Establishing effective and sustainable tourism sector associations and advocacy groups? d) Supporting destination marketing domestically and internationally? e) Building partnerships with a reasonable likelihood of stimulating private sector investment in the tourism sector? f) Establishing Sustainable Management Plans within protected areas? 2 Is there evidence that the six key intervention areas outlined in Evaluation Question #1 are likely to grow, scaling up and out? What specific actions will be required to ensure future sustainability after USAID Tourism for All ceased? 3 What specific stakeholder groups5 within the tourism sector have benefited the most from USAID’s Tourism for All interventions? Are there additional stakeholders with whom USAID’s Tourism for All should engage in order to support effective implementation of the National Tourism Policy that could stimulate private sector investment in the tourism sector? 4 Given efforts by the Government of Timor-Leste (GoTL) and various development partners to more effectively implement the National Tourism Policy or to stimulate private sector investments in the tourism sector, what is USAID’s comparative advantage to achieve or advance these same goals? 5 How has the Tourism for All Project adapted in response to the impact of COVID-19 and how effective has this been, and what additional adjustments should the Project consider in order to further mitigate COVID-19’s impact on the industry? 5 Stakeholder groups as referenced in evaluation question 3 include the following groups: Timorese women; Timorese youth (ages 10 to 29); Marginalized groups in Timor-Leste such as LGBTI, religious minorities, & persons with disabilities; Tourism associations that Tourism for All established such as ASTRABEKA, ATKOMA, ATM-TL; Cafe and Restaurant working group; Dili Municipal Tourism Authority; Hotel Owners Timor-Leste Association (HOT-L); Government counterparts: Ministry of Tourism, Secretary State of Arts & Culture & Tourism Development Authority; and Dili Institute of Technology. Though information on the role and integration of all stakeholder groups was not available during data collection, the findings and conclusions sections for EQ3 are disaggregated by stakeholder groups the ET did capture data about. 5 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV EVALUATION DESIGN AND METHODOLOGY OVERVIEW To answer the evaluation questions, the ET used a mixed-methods approach including remote key informant interviews (KII), group interviews, focus group discussions (FGD), a data validation workshop, and a desk review of over 25 TFA progress reports, academic literature, partner government tourism strategies, and economic reports. Primary data collection took place across June and July 2021 with stakeholders from USAID, TFA, government, donor partners, community and tourism associations, and private sector and partner organizations. A qualitative approach was selected to allow for an in-depth and nuanced understanding of the TFA project. Figure 1: Sample Achieved DATA SOURCES AND METHODS DESK REVIEW AND SECONDARY ANALYSIS Figure 2: Desk Review Prior to and during data collection, the ET reviewed over 25 documents including TFA Annual Reports, Quarterly Progress Reports, impact and assessment studies, Government of Timor-Leste tourism USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 6 development strategies and World Bank economic reports to provide additional context. The ET additionally reviewed reports on tourism and gender in Timor-Leste, notably a comprehensive 2020 report by United Nations (UN) Women that provided recommendations of global best practices for gender inclusion (Annex III: UN Women Recommendations for Engendering Timor-Leste Tourism Sector), contextualized to the Timor-Leste tourism industry (See Annex II: Information Sources for a full list). Results from the desk review informed the overall evaluation design, approach, and development of instruments. Analysis of secondary research was also vital in enabling triangulation with the primary data collection findings. KEY INFORMANT INTERVIEWS. GROUP INTERVIEWS, & FOCUS GROUP DISCUSSIONS In June and July 2021, the ET conducted a series of data collection events, including KIIs, groups interviews, and FGDs. All data collection events were conducted online, through Microsoft Teams, Zoom or WhatsApp. The team conducted KIIs with TFA project staff, USAID, GoTL, other donor partners, private sector, and partner stakeholders to gain a first-hand, in-depth understanding of the TFA project implementation. KIIs lasted 60–90 minutes and included a facilitator and notetaker. The Dili-based team conducted four virtual FGDs or group interviews with community and tourism association members in Atauro and Hatobuilico to enable group discussion and a diversity of community perspectives. FGDs or group interviews lasted 90–120 minutes and included a facilitator and notetaker.6 The ET received approval for this study and the study instruments from the Social Impact Institutional Review Board, a process that included a comprehensive review of Coronavirus Disease 2019 (COVID￾19) mitigation measures. DATA VALIDATION WORKSHOP In July 2021, members of the ET hosted an online data validation workshop with USAID, donors, private sector, and community stakeholders to present initial compiled findings, confirm initial assumptions and enable additional opportunity for stakeholders to provide feedback. Participants were presented key findings under each evaluation question, then divided into two groups to discuss perspectives through guided questions provided by the ET. The two groups were each facilitated by a male and female member of the ET and supported by notetakers. All previous informants across all stakeholder groups were invited, with the session lasting for 120 minutes. The workshop was conducted remotely via Zoom. Results from the data validation workshop were then incorporated into the evaluation findings and recommendations. SAMPLING Respondent Selection. The ET targeted seven groups of stakeholders: TFA project staff, USAID, GoTL, donor partners, private sector and partners, tourism associations and community in Atauro, and Hatobuilico. The ET used a purposive sampling approach for all stakeholders7 due to the limited number 6 The ET had initially considered using a short online survey to supplement the KIIs, group interviews, and FGDs if a balanced sample was difficult to reach. However, given the balanced qualitative sample achieved, data collection via survey was not required. 7 Purposive sampling is a non-probability sampling technique that relies on the judgment of the researcher to select units of analysis. It is widely used in qualitative research for the identification and selection of information- rich cases. This involves identifying and selecting individuals or groups of individuals that are especially knowledgeable about or experienced with a phenomenon of interest. 7 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV of potential respondents in the sample pool and COVID-19 related restrictions that limited movement and participation. Using this technique, the ET identified stakeholders to sample through recommendations provided by TFA and USAID and knowledge of the structure of the Timor-Leste tourism sector provided by ET members. The ET aimed to achieve as balanced a sample as possible, stratified across the major stakeholder groups, and ensured adequate representation from the project’s focus areas: Dili, Atauro, and Hatobuilico. Stakeholders deemed vital for the evaluation were approached for either a KII, group interview, or FGD. Sample Achieved. The ET conducted a total of 35 unique data collection events with 65 individuals. This comprised of 30 individual KIIs, 2 group interviews (between two and four participants per interview and carried out in alignment with FGD methods), 2 FGDs (between five and seven participants per FGD) and a data validation workshop with 14 participants. Diversity of stakeholder groups was achieved with KIIs, group interviews, and FGDs including USAID (1), TFA (5), government (4), donor partners (5), community and tourism associations (4) and private sector and partner organizations (15). A larger sample of private sector and partner stakeholders was deemed necessary to enable diversity of stakeholder sub-groups including successful and unsuccessful grant recipients, association members and operator groups (such as hotel, tourism guide). 8 Table 3: List of Respondents STAKEHOLDER GROUPS METHOD9 SEX AGE TFA 4 KIIs 4 Male, 1 Female 0 Youth USAID 1 KIIs 1 Male, 1 Female 0 Youth Timor-Leste Government 4 KIIs 4 Male, 0 Female 0 Youth Donors 5 KIIs 4 Male, 1 Female 0 Youth Private Sector/Partners 15 KIIs 13 Male, 5 Female 2 Youth Tourism Associations 2 Group Interviews 4 Males, 1 Female 3 Youth Community 2 FGDs 10 Males, 2 Females 10 Youth Data Validation Workshop Data Validation Workshop 9 Males, 5 Females 1 Youth Total 30 KIIs, 2 Group Interviews, 2 FGDs, 1 Workshop 49 Males, 16 Females (75 percent, 25 percent)10 16 Youth 8 Fluidity within stakeholder groups was also noted; for example, a former government respondent was also a project partner, and a private sector grantee was also President of ATKOMA tourism association. Stakeholders have therefore been placed within the group in which they expressed having the most experience with the project. 9 Please note that a KII could have more than one respondent present. 10 Though overall sex representation should be noted in readers’ interpretation of the report findings, a breakdown by sex will not be presented per finding to protect the anonymity of respondents. USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 8 As Dili is the capital of Timor-Leste and principal office for TFA, the majority of KIIs, group interviews, and FGDs were conducted with stakeholders based there (26 out of 51 or 50 percent), compared to Atauro (10 out of 51 or 20 percent), Hatobuilico (10 out of 51 or 20 percent) or overseas (5 out of 51 or 10 percent). The final distribution across sex for the KIIs amounted to 78 percent male (40 out of 51) and 22 percent female (11 out of 51), in part due to traditional gender roles that place males more often than females in positions of leadership in Timorese society. In total 16 youth respondents11 were reached, which reflects 25 percent of the total sample. The low percentage is due to youth being absent from certain categories, e.g., Timor-Leste Government and donors. Youth representation among interviewees from the private sector, tourism associations, and community stakeholders was significantly higher, at 43 percent of respondents (15 of 35). The data validation workshop comprised participants from USAID (4), TFA (2), other donors (3), private sector and partners (5). Workshop participants were all either located in either Dili or overseas, with 64 percent male and 36 percent female representation. DATA ANALYSIS ONGOING ANALYSIS AND QUALITY ASSURANCE Over the course of data collection, the ET conducted a weekly call with SI representatives and thrice￾weekly team debriefs via WhatsApp to discuss initial findings. The team subsequently adapted and revised the data collection instruments to account for emerging trends and new areas of interest for probing. Members of the ET took comprehensive notes in either Tetum or English which were translated into English (if required) and uploaded to SharePoint which is password protected. The team leader (TL) provided feedback on the notes, clarifying discrepancies or confusion with the local team members via email and WhatsApp, the most commonly and easily used encrypted internet messaging system in Timor-Leste for international communication, and conducted separate meetings with Dili￾based ET members to help understand and nuance the findings. Initial results were triangulated with findings from the desk review and secondary analysis. POST-FIELDWORK ANALYSIS The TL synthesized the finalized notes using a Findings, Conclusions, and Recommendations (FCR) matrix to analyze themes by evaluation question. The matrix ensured that the ET prepared a systematic and thorough response to each evaluation question, identified any gaps where additional clarification or analysis would be necessary, and served as the basis for developing the out-brief, data validation workshop and evaluation report. The ET compared results across stakeholder groups, where appropriate, to assess convergence and divergence in perspectives. Further analysis was then undertaken on the main findings under each EQ at the data validation workshop, with any new stakeholder data then inputted into FCR prior to development of the final evaluation report. GENDER AND SOCIAL INCLUSION ANALYSIS 11 Per the USAID definition, youth are defined as those between 10-29 years old. 9 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV Consistent with USAID evaluation policy and the objectives of the project, SI applied a gender and social inclusion (GESI) lens to the evaluation design and ensured that research activities included female participation, both as team members and respondents. SI employed a dedicated GESI Specialist to work with the team to ensure compliance and gender sensitivity on tools and protocols, data collection, and data analysis, including eliciting GESI-segmented data. The ET ensured EQ3, in particular, provides insight into the impact of TFA on women and other traditionally socially excluded groups. During data analysis, the ET not only looked at stated responses about gender and social inclusion, but divergence and convergence in responses among the different genders. Additionally, secondary literature on women in the Timor-Leste tourism industry was reviewed and integrated with primary data analysis to further engender the evaluation. TRIANGULATION AND VALIDATION The ET employed analytical triangulation approaches as part of the development of findings and conclusions. Triangulation enabled the ET to cross-verify and cross-validate the findings that emerged from using the above methods and data sources to identify correlations between findings and determine overall project effectiveness. In particular, the ET utilized methodological triangulation to strengthen the potential linkages and accuracy of its data if the results obtained through one method were less conclusive than another method. Where relevant, the ET used secondary qualitative data sources, including TFA Annual Reports, Quarterly Progress Reports, impact, and assessment studies, GoTL tourism development strategies and World Bank economic reports, to supplement and triangulate qualitative findings. LIMITATIONS AND BIASES Table 4: Limitations and Risks LIMITATION/ RISK DETAILS MITIGATION STRATEGY Recall Bias Beneficiaries may respond to questions posed by the ET with answers that blend their experiences into a composite memory. Respondents who may have participated in more than one project activity or in similar activities conducted by other implementers may not be able to distinguish between them. Additionally, depending on when beneficiaries participated in the project’s activities, perceptions of events may change over time, and the ability to remember specific details may fade. The ET conducted multiple KIIs, group interviews, and FGDs to triangulate responses and to increase the validity of the evaluation findings. The ET also conducted a data validation workshop to enable participants to review and discuss data presented and, where relevant, provide new and contextual insights. USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 10 Response Bias Key informants may be motivated to provide responses that would be considered socially desirable or influential in obtaining donor support. A training participant may remark positively about an activity because they would like to attend similar training in the future. Respondents may believe that negative evaluation findings could result in reduced assistance. Prior to each KII, group interview, and FGD, the ET delivered a clear informed script to participants which emphasized confidentiality, honesty, and the voluntary nature of the interview. It is expected that reassurances by the qualitative researchers, communicated through the informed consent, encouraged participants to give more accurate and honest answers thereby reducing response bias. Selection Bias Selection bias is an inherent risk when implementers help to facilitate contact with project beneficiaries. The ET initially worked from a stakeholder list provided by USAID. There was a risk that project staff selected the most active, responsive, or engaged beneficiaries, creating the risk that the ET only heard from key informants who reported positive experiences. Prior to conducting KIIs, group interviews, and FGDs, the ET reviewed and edited the initial stakeholder list to ensure it reflected the contingent of relevant tourism stakeholders in Timor-Leste. The ET also employed best practice in qualitative interviewing that emphasized non-leading, probing questions to get at the most accurate and clear information. Gender Bias Traditional gender norms and customs regarding the role of men and women in the household, in agriculture, and other related fields may influence the way that men and women answer questions related to gender. Women may be afraid to speak or give their candid opinion in the presence of men. The ET had a mixed-gender ET (2 female, 2 male) and a female facilitator was always available as required to lead any data collection activities that involved only women. COVID-19 and travel COVID-19 related mitigation measures such as travel restrictions, social distancing, and other measures affected the ET’s ability to conduct work as expected. The ET made an early decision to conduct the evaluation in a fully remote format to ensure COVID-19 restrictions did not impact plans mid-way through data collection. A variety of online communication tools were employed including Microsoft Teams, zoom and WhatsApp and contact with stakeholders was made well in advance of proposed interview times to provide remote stakeholders ample opportunity to select a time when they could access internet connectivity. COVID-19 and remote data oversight Limitations due to the COVID-19 pandemic prevented non-local ET members from traveling to Timor-Leste for fieldwork. The absence of the TL in Timor￾Leste limited the ability to do high quality, frequent data quality oversight. SI conducted weekly check-ins with team members in Timor-Leste (via MS Teams) to understand the progress of fieldwork, assist with logistical challenges, and examine initial findings. The TL conducted (at a minimum) thrice weekly calls with locally based ET to answer any questions and discuss data findings. The TL was able to conduct 19 remote interviews with local stakeholders to contribute to local data collection. 11 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV Project Scope The TFA project is both extensive and diverse in its scope and interacts with a wide range of stakeholders. As such, few stakeholders held strong knowledge of all its key features with awareness often limited to their area of interaction (for example, private sector knowledge of grants programs). This made it difficult for ET to gather extensive data from all stakeholders. The ET used a range of methods to gather and triangulate data including secondary sources and the data validation workshop. The ET was also mindful of the sampling approach to ensure an adequate number of stakeholders were able to respond to each EQ. Project Communication Many stakeholders informed the ET that TFA communication around project aims and successes was limited and they did not possess as much information about the project as they would have liked. This further impeded the ETs ability to gather informed data from the KIIs, group interviews, and FGDs. The ET used a range of methods to gather and triangulate data including secondary sources and the data validation workshop. The ET has also noted limited project communication in EQ conclusions and provided suggestions for improvement in the recommendations. Project Remit The TFA project collaborates closely with MTCI and donors on a range of initiatives. As such, the ET noted during the evaluation that some stakeholders were confused about the role of TFA in various initiatives and incorrectly attributed MTCI initiated projects to TFA. The ET used a range of methods to gather and triangulate data including secondary sources and the data validation workshop. The ET revisited stakeholder consultations and additional secondary data sources to further validate data and assumptions. USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 12 FINDINGS AND CONCLUSIONS EVALUATION QUESTION 1A To answer EQ1a, the ET sought to understand the diverse range of tourism-related policies and regulations instigated by the project, and to determine if these were developed through effective engagement with tourism stakeholders. The ET was aware the project’s efforts under this EQ included a diverse range of initiatives12 and that stakeholder response may not cover all interventions. Stakeholder response focused on three key areas: Figure 3: Key Intervention Areas EQ1a 1A FINDINGS FOR EQ1A National Tourism Marketing & Promotion Board. The project’s work towards the establishment of a Tourism Board13 was praised by respondents (5 out of 13)14, particularly from the private sector. A private sector stakeholder explained: “(A) Tourism Board is one of the very important things we have to see through. That will be in the history books for a long time. You see, every country has always started with that Tourism Board to push tourism and Timor-Leste needs that badly.” A leading Timorese tourism expert agreed: “The future is a Tourism Board.” Respondents explained that a Tourism Board will not only enable private sector tourism experts to support Timor-Leste’s marketing efforts, but also allow for consistency in approach, as Tourism Board representatives would not have to change with every change in government. However, private sector and TFA stakeholders acknowledged that effective collaboration to develop a Tourism Board has not been easy. A TFA representative explained: “There’s been a great deal of talk over the years and decades in terms of what form this should take. Never been easy to create any type of representative body – tends to fall into factions. [It is] better when you have associations that are registered, fully representative and can really support the private sector and communities in which they operate.” This view was supported in the data validation workshop by a donor who suggested that the project’s work towards the establishment of tourism associations could aid in a collective private sector approach. 12 Based on desk review and USAID Quarterly Progress Report Q2FY2021 13 A Tourism Board is a standalone, private-sector-led body to lead tourism marketing and promotions. 14 Not all 13 respondents who discussed regulation would have had the knowledge to speak on all key interventions. These key interventions were the most commonly discussed by respondents who spoke to regulation topics. Has Tourism for All been successful/unsuccessful in the following key intervention areas: Working effectively with stakeholders to develop tourism-related policies or regulations? 13 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV According to the project reporting, progress is being made. The USAID Quarterly Progress Report Q2 FY2021 outlined that the project has been asked to provide technical assistance toward the development of a concept paper for the establishment of a Timor-Leste Tourism Marketing and Promotion Board to lead destination marketing in domestic and international markets. A private sector stakeholder supported the project’s efforts in ‘selling the idea’ to the government and generating support, arguing this has been well received by the Timor-Leste tourism sector. Tourism Regulation in Hatobuilico. All respondents (5 out of 13) outlined that the project’s support for regulating tourism – including the establishment of a parking area15 and a fee for entry to increase tourism revenue – had been valuable. However, ineffective consideration of wider policy implications, primarily consideration by the project on how the community might collect, distribute, and spend the revenue, has caused issues. Members of the Hatobuilico community expressed concern that there was a lack of clarity around expectations, responsibilities, and project delivery: “We need to be clear about what we want to bring into community-based tourism activities – what TFA has to offer and what community responsibilities… everything should be clear so don’t create any confusion or conflict in the future – mostly if anything related to money, it is very sensitive, so it has to be clear!” A member of ASTRABEKA – the tourism association in Hatobuilico established by the project – explained that while the tourism-related regulation was positive for the community, it had not been effectively socialized with community members and the association to ensure smooth implementation. Representatives from TFA outlined that the entry fee in Hatobulico was initiated by MTCI and that while the project had provided technical support, responsibility for implementation had always resided with MTCI and its staff in Hatobuilico. TFA also confirmed the project is now advocating for ASTRABEKA to gain access to the entrance fees collected by MTCI local staff and hopes to strengthen the capacity of ASTRABEKA to take the lead on managing revenue. Regulations for whale tourism. Respondents (3 out of 13) concurred that development of a regulatory framework for whale tourism was both positive and essential. TFA representatives explained the project’s role included assistance to ATM-TL – the marine tourism association established by the project – to develop a strategy for the association, which was shared among members, and advocated the need for Whale Watching Guidelines. However, stakeholder respondents (3 out of 3) expressed a lack of clarity around the roles and responsibilities of TFA, government, the associations, and industry in the establishment of regulations for whale tourism. An ATM-TL member explained that while regulating the industry was a priority: “We simply do not know how, by whom, and importantly, when, this will be done. So, the ATM-TL, on behalf of the industry, took the lead and decided, we’ll just do it ourselves – until government regulations are in place.” The stakeholder advised that support from other stakeholders and donors in the sector was urgently needed to ensure sustainability. Another respondent highlighted issues of collaboration between the project, other associations, and marine tourism operators, saying: “I think the lack of collaboration has delayed progress.” The stakeholder explained that a ‘fight for ownership’ of initiatives had put some marine operators on the back foot, with several choosing to step away. 15 The ET notes that the establishment of a parking area in Hatobulico is not an initiative of TFA and might have been inadvertently attributed to the project by respondents. USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 14 CONCLUSIONS FOR EQ1A The project’s work towards a National Tourism Marketing & Promotion Board has been well received by stakeholders. While the process has not been easy, stakeholders have welcomed the project’s advocacy and ongoing efforts and highlighted the importance of this initiative to the future of tourism in Timor-Leste. Clarity around the role and responsibilities of TFA, stakeholders and MTCI in relation to tourism regulation could be improved. The development of regulations for Hatobuilico and the whale tourism industry are viewed as important and positive for tourism. However, stakeholders are unsure about what can be reasonably expected of the project while issues of coordination, both between stakeholder groups and with MTCI, appear to have impacted interventions. In the case of Hatobuilico, insufficient consideration and planning for the wider policy implications has (unintentionally) created additional challenges for the community and interventions led by MTCI have been attributed to the TFA project. EVALUATION QUESTION 1B 1B Has Tourism for All been successful/unsuccessful in the following key intervention areas: Supporting implementation of the National Tourism Policy? To answer EQ1b, the ET sought to understand progress towards implementation of the National Tourism Policy (NTP), as well as the factors which could be perceived as impeding implementation. The majority of respondents (21 out of 23) provided views on the relationship between government (primarily MTCI) and the project, given collaboration is integral to the successful implementation of the NTP. FINDINGS FOR EQ1B TFA respondents (3 out of 3) articulated that the project has helped guide the implementation of the NTP under its 5Ps – priority, prosperity, protection, partnerships, and people – by developing a Timor￾Leste Tourism Development Strategy and Implementation Plan. However, despite ongoing reminders and offers of assistance to the Ministry of Tourism, Commerce, and Industry (MTCI),16 approval of the strategy and implementation plan has not been forthcoming. MTCI representatives (2 out of 2) suggested that the plans will take time to finalize and approve. A government stakeholder also noted that there has been an unwelcome push from TFA to accelerate the approval process. Representatives from TFA outlined that the draft strategy, requested by the government in writing, had been awaiting MTCI approval since June 2020. The majority of private sector and donor respondents (15 out of 16) commented that interaction with MTCI can be challenging, with many citing ongoing changes in the MTCI leadership structure across the project’s duration, which has included a period without a Minister of Tourism, two Acting Ministers of Tourism, and a new Vice Minister for Tourism. A donor stakeholder explained: “At one point [the project] 16 USAID Quarterly Progress Report Q2FY2021, page 9 15 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV tried to bring partners together to reach an agreed position of what needed to happen. This was good; but then the government changed and it all stalled.” It was also noted that interaction with the government was made more difficult due to the COVID-19 pandemic and limited opportunity to meet face to face. Stakeholders from MTCI (2 out of 2) outlined that while the project is running well from their perspective, communications between the Ministry and project need to be improved. A government stakeholder noted that from their perspective, the project should be country-owned and country-led, however explained: “But the project thinks the government follows what they want; therefore, we need better communication.” The issue of a disconnect between MTCI and the TFA project was supported by a former senior tourism expert, who said: “You have to institutionalize tourism. I believe you have to connect with a government program. It has to be more flexible.... TFA takes a top-down approach and the government says, ‘this is TFA’s program, not ours.’ How to incentivize so the government feels it belongs to them?” The importance of collaboration with MTCI was echoed by communities in Hatobuilico and Atauro who shared that government support is essential for successful implementation of the overall project objectives. The community in Hatobuilico suggested that good management practice for religious and cultural tourism in Timor-Leste will require MTCI support, and that rules and regulations for community-based tourism activities must be government-led. Similarly, in Atauro, community members called for a more integrated approach with MTCI to promote industry growth. A community stakeholder in Atauro explained: “We need to encourage cooperation close to government officers." All stakeholder groups concurred that a close working relationship with MTCI will be essential to successful implementation of the NTP and project overall. A TFA respondent explained: “Business will act alone if need be – but will be 10 times stronger if there was ministerial support for all aspects of tourism.” A workshop participant also noted that the TFA project is predicated on the NTP, and its implementation must consider the NTP a ring fence within which the project operates. This was supported by a donor stakeholder who contended that the project will continue to face difficulties if it is not fully institutionalized with MTCI. While there was stakeholder agreement on the importance of working closely with MTCI on implementation of the NTP, few stakeholders spoke to how this could best occur. A TFA stakeholder said: “I see that the owner of the project is the MTCI – but it is not actively working alongside the TFA project. It is difficult to have good communication with them. The [TFA] project approaches them, they keep postponing.” Two stakeholders noted a potential challenge for the project’s relationship with MTCI residing in its structure, suggesting that TFA, as a project led by a private contractor working for a foreign state, may never be as integrated as projects which are fully embedded with the government. Challenges regarding government collaboration and integration were also discussed in the data validation workshop. Two private sector operators expressed frustration that MTCI has not been supporting operators in promoting tourism or encouraging investment, with another suggesting that industry is often more acutely aware of tourism priorities/connections/challenges, more so than some government players. A donor stakeholder questioned the level at which the project was engaging with the government and whether there could be an opportunity to further collaborate with government officials in addition to MTCI, while a TFA representative confirmed the project is already engaging across the government, including the Dili Municipality, Ministry of Finance, Ministry of Transport, Ministry of Agriculture and Fisheries, and the District Administrator in Atauro. A partner stakeholder advised that USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 16 Timor-Leste’s complex development landscape requires more official facilitation of collaboration – such as through formal agreements and memorandum of understandings (MOU). CONCLUSIONS FOR EQ1B TFA has faced ongoing challenges in its support of the implementation of the National Tourism Policy. While some progress has been made in developing strategic materials to aid implementation of the NTP, progress has been stalled by challenges to building a strong working relationship with MTCI, ongoing bureaucratic processes, and high-level MTCI personnel changes. All stakeholder groups noted that the project needs to improve its engagement and integration with MTCI to ensure effective implementation of the NTP. EVALUATION QUESTION 1C Has Tourism for All been successful/unsuccessful in the following key intervention areas: Establishing effective and sustainable tourism sector associations and advocacy groups? To answer EQ1c, the ET focused on the three main tourism associations17 the TFA project has supported to establish and strengthen. These include Asosiasaun Turizmu Koleku Mahanak Ataúro (ATKOMA), Asosiasaun Turizmu Ramelau, Blehitu, and Kablake (ASTRABEKA), and Asosiasaun Turizmu Maritima Iha Timor-Leste (ATM-TL), which support collective advocacy and development efforts for tourism in Timor-Leste. Findings are presented by geographic area because of different approaches and stages of implementation. 1C 17 Note that as tourism associations are a main output of the TFA program (result 12, objective 2) (Source: Annual Report FY 2020) and are more developed than tourism working groups, the research team determined to focus only on the three main tourism associations in KIIs, group interviews, and FGDs given limited time and resources. FINDINGS FOR EQ1C Figure 4: Notable TFA Support to Atauro Community 17 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV ATKOMA. Both community stakeholders in Atauro and ATKOMA members relayed positive feedback on the association. Many operators were part of informal private sector groups since 2000 (including handicraft businesses, guest houses, tour guides groups, restaurants, local products such as boneka atauro) and had received significant investment from other donors over the past 20 years. According to community and ATKOMA members, the most useful support delivered by the TFA project included: 1) support in organizing tours and tour packages, 2) price setting, 3) data collection to map tourist sites and activities, 4) legalization and formalization of the association, 5) socializing rules and regulations and 6) conducting training including on financial literacy, tour guiding, and management. Respondents also noted three women’s groups – Boneka Atauro, Wawata Ropu and HOMAN – had received valuable training from TFA. An ATKOMA member explained: “We are lucky because ATKOMA has registration in Timor-Leste and now it’s going well. They are helping a lot with USAID TFA and give a lot of information. Now [we] have been working with them for three years and we have a lot of knowledge. Never in Timor-Leste has the government done that… never support simple tourism. Simple tourism is important, like in Atauro, a lot of guests come for a simple place.” Community members expressed a desire to see the project extend beyond Beloi, alongside increased support for women and youth, particularly in remote areas, and additional training opportunities. The community also expressed a desire for support in building a Tourism Information and Conservation Center, the conceptualization of which was supported by TFA. The ET also noted from secondary research that the TFA worked with the community in Atauro to establish a Sustainable Tourism Development Strategy and Sustainable Management Plan. In furthering the institutionalization of these interventions, a partner stakeholder advocated for continued partnerships with key local tourism and conservation organizations such as Blue Ventures and Roman Luan to support sustainable tourism planning and training for association members. ASTRABEKA and community members in Hatobuilico related a different experience. During group interviews, members of the association expressed they lost their hope and trust in the TFA project. They concurred that the project was well coordinated in its initial stages and was successful in gathering people under the establishment of the association. However, concerns had grown as few concrete activities had taken place, leading the community to question whether the project had given “false promises.” Stakeholders identified the project’s lack of coordination with the community as being the most problematic, explaining: “We don’t have knowledge about what is the concept of TFA itself and its expectation to work with the community for tourism activities.” Confusion from the Hatobuilico community on the remit of TFA compared to MTCI was also noted by the ET in EQ1a. Respondents reported concerns about the financial sustainability of ASTRABEKA, as a TFA grant proposal had been rejected, leaving many to feel the grant process lacked transparency and expectations were not fully communicated. A stakeholder asked: “We don’t understand why we didn’t have a chance to revise our proposal, but the project just dismissed it? We can’t expect the association to work on tourism activities in supporting communities if there are no funds available or without any support from TFA.” However, the ET’s secondary research found that the association had received a TFA grant for the ASTRABEKA office renovation.18 Association members had praised the support of the project in establishing the office, while not directly linking it to the grant mechanism. A senior figure in the Hatobuilico community, also acknowledged that the community had an important role to play in to cultivating and supporting the 18 Source: USAID Annual Report FY2020 USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 18 association, suggesting: “We need more youth with spirit of voluntarism, to encourage members of the association to commit to their work and continue providing support to the association.” Respondents also explained that the community in Hatobuilico has a limited understanding of community-based tourism, citing a strong need for support to socialize ideas and information not only around Ramelau but also in the three surrounding municipalities. This was echoed by TFA stakeholders who confirmed that the project had faced some challenges in Hatobuilico, acknowledging the limited knowledge of tourism in the community, issues with the MTCI- led initiative to instigate an entrance fee (as outlined in EQ1a), and a lack of opportunity to visit the community during the COVID-19 lockdown. A member of ASTRABEKA advocated for a public figure to lead and empower the community and that, given its early stages of development, capacity-building and leadership training could help the community understand the process for working with donor organizations. Figure 5: Notable TFA Support to Hatobuilico Community Despite concerns, community members cited three main positive aspects from their interaction with the project, including: 1) formal registration of the association, 2) office materials, and 3) training opportunities including tour guiding, financial literacy, and customer service. A small business owner in Hatobuilico attributed to TFA her improved understanding of financial processes, which she had been glad to pass on to her staff. An ASTRABEKA member also spoke positively about two women’s groups – Feto Uniaun Hatobuilico (for skills and creative pursuits such as handicrafts) and FOHATA (a cultural and traditional dance group) which were established under ASTRABEKA and received training through TFA. “Many of members have very limited time and financial resources to devote to establishing and building the association. If TFA is to establish industry associations to help the development of the tourism sector, it must do more than just ‘launch’ them – it must also support them to survive and grow.” -ATM-TL member 19 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV ATM-TL. ATM-TL members expressed support and gratitude for the project’s vision in establishing the association as an important national industry body and ‘voice’ for the developing marine tourism sector in Timor-Leste, including ongoing encouragement and support from project staff. However, concerns were reiterated around a lack of clarity as well as type and amount of support ATM-TL stakeholders could reasonably expect from the project – especially financial support. Support for ATM-TL was identified as particularly critical in the short-term to ensure its sustainability. An ATM-TL member shared: “The concept is a success and I think the concept is great; us operators we are still working with each other, but it’s always difficult with COVID-19 and things interrupting the flow of things. I do feel we lacked understanding from all parties at the start. There was always some confusion about who was doing what.” The stakeholder acknowledged that many operators and association members were also busy with their own businesses and thus needed to look at how to make the association more self-sufficient. Challenges with collaboration between ATM-TL and the broader marine tourism industry in Timor￾Leste were also noted. Four stakeholders explained that existing organizations had at times felt sidelined by the newly established marine association, citing concerns that the project had positioned ATM-TL as the only organization allowed to run marine tourism activities, despite existing programs having operated in the space for many years prior. A stakeholder said: “TFA had an opinion that they were the only people allowed to work in marine tourism in Timor-Leste” and that operators who were not part of the organization were shut out. Similar issues identified with ASTRABEKA stakeholders around funding and the grants program were noted by ATM-TL, with a member explaining that many individual owner-operators and small businesses are time poor and do not have the necessary administrative/financial resources and skills to complete the USAID grant application process – particularly Timorese members. CONCLUSIONS FOR EQ1C The effectiveness and sustainability of the three main tourism sector associations varies significantly. ATKOMA appears to be running the most effectively with stakeholders praising the project’s support across a range of areas including formalization of the association and provision of training. Conversely, in Hatobuilico, stakeholders reported that the level of support they had initially expected has not been forthcoming. The COVID-19 pandemic and limited awareness of both sustainable tourism development and donor programs appears to have impacted progress and support for the association in Hatobuilico. Ineffective stakeholder collaboration and coordination is impacting the sustainability of tourism sector associations. While all three sector associations have the ability to enhance industry coordination, a lack of clarity around project expectations – including the role of members, community, MTCI, and the project is evident for both ASTRABEKA and ATM-TL. Both associations require additional support, primarily financial, to bridge the gap between inception of the association and long￾term sustainability. USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 20 EVALUATION QUESTION 1D 1D Has Tourism for All been successful/unsuccessful in the following key intervention areas: Supporting destination marketing domestically and internationally? To answer EQ1d, the ET asked all respondents their views on the project’s role in supporting both domestic and international marketing efforts. The ET was aware that the project had undertaken significant work in the area of marketing, including but not limited to an online branding and marketing analysis, a Singapore outbound market study, market sounding in Indonesia, marketing training at national government and community level, work towards a Tourism Marketing Board (discussed under EQ1a), management of the Timor-Leste destination website and e-library, dissemination of the Timor￾Leste Tourism Update e-newsletter, and the Hau-Nia Timor-Leste (My Timor-Leste) domestic marketing campaign.19 Due to the currency and high visibility of the domestic marketing campaign, the majority of stakeholders who responded to this question (18 out of 20), excluding TFA project members, focused their feedback on this intervention. A further seven stakeholders commented on TFA’s digital marketing initiatives. FINDINGS FOR EQ1D Digital Marketing. Two government stakeholders provided feedback on the project’s marketing support more broadly and acknowledged TFA’s role in developing a niche marketing paper, managing the official destination website, and providing digital marketing training. While both stakeholders relayed appreciation for the project’s support, they also raised concerns around misunderstandings, between MTCI and project, on content approval and handover of the Timor-Leste tourism website. A TFA representative refuted this concern, confirming that content approval is always requested from MTCI. Private sector stakeholders spoke positively of TFAs digital marketing efforts, primarily the promotion of events and the e-newsletter, noting that the work has been important at a local level and for bringing people together. A donor stakeholder commended the e-newsletter increasing the visibility of Timor￾Leste globally, while a grantee expressed gratitude for the project’s support enabling digital marketing training and exchanges at the local level. Stakeholder views on the importance of international marketing during the current COVID-crisis were divided. A private sector stakeholder shared the project could consider marketing initiatives for when tourism returns, such as billboards at airports, while another argued it is unrealistic to be considering international marketing when the country was in a COVID lockdown, and that the project’s domestic focus was sensible and effective. 19 Based on desk review and USAID Quarterly Progress Report Q2FY2021 21 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV Figure 6: Domestic Marketing Campaign Feedback Domestic Marketing Campaign. The ET found high recognition among all stakeholder groups of TFA’s domestic marketing campaign under the tagline Hau-Nia Timor-Leste. Of the private sector stakeholders who had interacted with the campaign, the majority (8 out of 9) reported positive feedback, and more than half (5 out of 9) indicated that the campaign had helped support their business and generate additional revenue. Positive feedback was also noted from all community and ATKOMA tourism association members in Atauro. A tourism operator and ATKOMA member explained that the campaign had positively supported small, grassroots businesses: “Hau-Nia Timor-Leste is working well, helping small businesses in Atauro for everyone to know about it. Some places like Arlo, Coconut Place, they’re posting in Hau-Nia Timor-Leste, and someone sees that and they go there. I ask: why do you know Arlo has coconut? They say they saw Hau-Nia Timor-Leste and want to support the local community. I feel like ‘oh thank you’ for their support to the local people.” In Hatobuilico, the campaign was similarly praised for bringing domestic tourists to the region. Community stakeholders recommended an educational campaign around environmental protection could accompany the marketing to ensure respect for sacred sites: “Since the campaign we observed [the] number of visitors came to Ramelau was increasing drastically and people were interested to explore more areas – however we need to educate also attitude of tourist or visitors towards importance of value and respect towards religious tourism area like Ramelau including community itself on how to cultivate the norm of its culture and respect to Sanctuary of Mama Mary Ramelau [Sanctuary of St. Virgin Mary Ramelau].” Conversely, some stakeholders questioned the extent to which the domestic campaign had supported increased tourism or revenue. A private sector stakeholder, who reported no net impact from the campaign, suggested that its target audience – primarily Timorese youth – rarely spend money on a hotel and prefer to stay with relatives. The stakeholder recommended the campaign pivot to support an increased focus on the higher-spending expatriate market based in Dili. “I’d be getting out to the malae where the demand is and the money is.” The stakeholder advocated packaged domestic trips, similar to those promoted across the Pacific, could be valuable for Timor-Leste. Donor stakeholders (4 out of 5) reported similar concerns with the campaign’s target audience and ROI, suggesting that the campaign USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 22 was launched without a clear articulation of its economic return, particularly as there is a limited evidence base for the value of domestic tourism in Timor-Leste. A donor stakeholder explained: “There were lots of beautiful photos taken; but I wasn’t convinced that was actually generating income for communities. It doesn’t matter how many likes and shares you get on Facebook, if you’re not generating income.” In clarifying the purpose of the domestic campaign, TFA representatives explained it was primarily launched to keep spirits high during the COVID-19 crisis and enhance the awareness of tourism amongst Timorese as proposed in the NTP under one of its five Ps: ‘people’. This view was supported by two private sector stakeholders who argued that while the campaign may not have directly brought income to all local businesses, it had played a vital role in changing the perspective of tourism in Timor￾Leste – both by engendering pride in Timor-Leste as tourism destination and encouraging middle class Timorese to experience being a tourist in their own country. They also suggested that the campaign has supported local Timorese businesses to better understand the customer service mentality, which will be valuable in building the tourism product for when international borders reopen. Alignment with international marketing. The majority of donor stakeholders (4 out of 5) argued the domestic campaign is poorly aligned to the existing, government approved, Explore the Undiscovered destination brand and logo.20 They raised that by not utilizing the approved international brand, the Hau￾Nia Timor-Leste campaign was dividing the brand messaging in Timor-Leste and missing an opportunity to build promotional content for the international campaign for when borders eventually reopen. A donor stakeholder said: “Timor-Leste is too small of an industry for competing branding.” This view was refuted in the data validation workshop by two private sector stakeholders who contended that separate international and domestic brands was in fact appropriate given the distinct audiences of the two brands. TFA representatives explained that a separate domestic brand was predicated on the successful use of distinct international and domestic brands in neighboring nations. Vanuatu, for example, markets internationally with Answer the Call of the Vanuatu slogan, but domestically launched the Sapotem Lokol Turisim (support local tourism) campaign in response to the COVID-19 pandemic. CONCLUSIONS FOR EQ1D The domestic marketing campaign, Hau-Nia Timor-Leste, has been well received by private sector and community stakeholders. The campaign has provided a necessary pivot during the COVID-19 pandemic, keeping tourism on the agenda, and reportedly stimulating demand and increased revenue for tourism businesses. The campaign has engendered pride in Timor-Leste as a tourism destination and supported local businesses to prepare for the resumption of international tourism. As there is limited measurement of the campaign’s ROI, it is not possible to quantify its economic value. The choice of domestic marketing slogan – Hau-Nia Timor-Leste – has been contentious. In developing a distinct slogan from the international slogan which was previously approved by the Timor￾20 As outlined in USAID’s Online Marketing and Branding Analysis, 2018, the Explore the Undiscovered brand – which involved consultations with stakeholders from the private sector and NGOs, surveys undertaken to test the target market, and strategic advice and work by Timorese designers – was officially adopted by the Timor-Leste Council of Ministers in July 2017. However, as outlined in the Asia Foundation’s Timor-Leste Tourism Barometer 2018, the Explore the Undiscovered brand had not been effectively marketed by government nor was there a spirit of universal ownership of the destination brand. 23 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV Leste Council of Ministers, the project has been perceived as being uncollaborative with donor and government stakeholders. Some stakeholders are concerned that two tourism brands in a small country have divided the message and will not raise international brand awareness for Timor-Leste during the global shutdown. Conversely, other stakeholders noted that two brands are in fact appropriate for the two distinct audiences and in-line with the destination marketing approach of neighboring countries. TFA’s digital marketing interventions have been better received by the private sector than MTCI. Private sector operators acknowledged the project’s support towards the e-newsletter, event promotion, and digital marketing training. While MTCI expressed appreciation for initiatives including market studies, management of the official destination website and digital marketing training, progress appears to have been impacted by miscommunication between MTCI and TFA, making it difficult to determine the impact of these interventions. EVALUATION QUESTION 1E Has Tourism for All been successful/unsuccessful in the following key intervention areas: Building partnerships with a reasonable likelihood of stimulating private sector investment in the tourism sector? 1E To answer EQ1e, the ET sought to first understand the scope and success of partnerships instigated by the project, and second to determine if these partnerships had or were likely to stimulate increased private sector investment in tourism. As the project’s efforts under this EQ include a diverse range of initiatives21 respondents’ awareness across the full range of work was limited. Stakeholder response to this EQ focused primarily on private sector programs (16 stakeholders) including the grant program, tourism events and the Tourism Champions Awards, and 5 stakeholders discussed the project’s work towards Public Private Partnerships (PPPs). Figure 7: Private Sector Operators’ Views of the Tourism Awards & Activities 21 Based on desk review and USAID Quarterly Progress Report Q2FY2021 USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 24 FINDINGS FOR EQ1E Tourism Awards and Activities. The TFA-supported Tourism Champions Awards, Dada Lia, and tourism events were particularly well received by private sector operators, with most (6 out of 7) suggesting they had enabled greater unity among operators and helped to build partnerships within the sector. A private sector stakeholder commented: “Not many people understand tourism, so [it is] sometimes very hard for the private sector to invest in tourism. The TFA program just highlights how big the tourism industry is and how it can benefit tourism growing in the country.” Operators noted that Timor-Leste is a challenging place to unite sectors and that bringing people together helped the industry to work towards the same goals. Tourism Awards and Sector Advocacy. A private sector operator commented that the project has played an important role in tourism advocacy overall, explaining: “[It has] helped people identify they’re a sector or an industry. That sounds simple – but it’s very useful.” In the data validation workshop, this perspective was supported by an additional two private sector operators who explained that people working in tourism in Timor-Leste often don’t think or know that they are – and thus the project’s domestic focus has enabled a deeper understanding of the value of tourism, and, in doing so, will encourage greater investment in the sector through employment and skills development. Only one private sector respondent reported negative feedback on the project’s private sector events, citing a heavy focus in Dili, as opposed to operators and partners based in the districts. The stakeholder said: “You go to the conferences, business and tourism, and everyone says positive things – about government, about tourism. You can’t say negative things – it’s too small … The people who aren’t happy are those who don’t have a voice.” Grant Program and Sustained Investment. Respondents reported mixed feedback on the grant program’s ability to generate sustained investment. The TFA grants program received extremely positive feedback from private sector respondents (8 out of 9), with a consensus amongst successful grantees on the importance of the training that the grant had enabled. A partner stakeholder commented: “One of the best aspects of the program is the way it has supported local dive companies to train Timorese divers/instructors. This activity has been an outstanding success. And very much demonstrates how the private sector can really assist in growing the marine tourism sector, if they are given the opportunity, and supported.” A successful grantee in Dili also emphasized the grant program’s support for ‘knowledge transfer’ activities which have helped grow and strengthen the tourism industry in Timor-Leste. The stakeholder said: “What they funded was the extraction of traditional knowledge that existed and the conversion of that into a product. I’m not sure if there’s ever been a more effective grant in Timor for converting a dollar amount into something scalable.” This point was echoed by a grantee in Atauro who emphasized the importance of the grant program in building tourism investment not only for his business, but within the wider community: “When I got a grant, the community said ‘oh thank you’ as it means guests come and there’s more work for everyone and then they get money for themselves. I am working hard to help them. It’s not income for just me... A lot of tourists come, and the community has income from that.” Donor partners, however, (4 out of 5) questioned the sustainability of the grant program in terms of 1) financial expectations, 2) monitoring and evaluation (M&E) and 3) transparency. A donor stakeholder argued the approach appeared to be ‘scattergun’, as there was limited co-investment required from partner organizations. A second donor stakeholder proposed that although the grant program facilitates 25 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV important opportunities, given Timor-Leste’s nascent tourism industry, there is a risk that it also creates a false sense of hope, particularly during the COVID-19 pandemic: “I think the grant mechanisms can create false economies especially when businesses try to strengthen their supply when there is no demand.” Donor stakeholders also questioned the level of M&E conducted on the program and therefore the ability to measure its return on investment at an individual operator level and overall. Donors further shared that transparency on the number and type of grant recipients was not always forthcoming from the project, which had resulted in missed opportunities for collaboration with other donor partners. A TFA representative shared a contrasting view and suggested that while grantees were not obliged to provide a co-contribution, the project had aimed to achieve a 25 percent co-contribution (by dollar amount) to enable ownership, commitment, and support sustainability of grantee activities. Secondary data analysis suggests that of grants awarded by end of Q3 FY2021, only 2 percent of grantees had not co-contributed.22 TFA representatives also conveyed a willingness to share grantee information with donors and other tourism stakeholders. Grant Program Accessibility. One of the biggest concerns reported on the grant program, even by successful grantees, was the bureaucratic nature of the program, mandated by USAID regulation, which precluded applications from some potential businesses – particularly Timorese operators in the districts. A successful grantee explained: “There could be more explanation of that at the start. How many hours investment – and you’ve got x chance of getting in. You don’t want people putting in a huge amount and getting rejected. It’s a tricky balance.” A second grantee suggested timelines were difficult to predict, due to the lengthy application and approval process. Another stakeholder outlined that the financial and administrative requirements of the application are outside the capacity of smaller organizations labeling the USAID process as ‘far too complex’ and ‘unrealistic,’ particularly the high level of auditing and financial reporting requirements.23 Figure 8: Stakeholder Suggestions for Improvements to the Grant Program Three respondents provided suggestions for improvements to the grant program, which included a simplified, less bureaucratic process, greater support with grant applications, and improved information and collaboration (with the project) prior to application. A private sector stakeholder compared the TFA grant program with another donor-funded program and said: “It’s not ‘you put in something, and I’ll evaluate it.’ [With the other program] you have more of an idea – we chatted, they said ‘here’s what I’d like you 22 Summary of TFA grantees at Q3 2021, provided to ET by TFA 23 Secondary research conducted by the ET found TFA efforts to increase accessibility of the program including multi-lingual application process, information sessions and online videos. USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 26 to flesh out… we went back and forth and ended up with a more creative, interesting product.” A representative from TFA suggested that the project had been conservative in reaching out to potential grantee organizations, in attempt to remain impartial and avoid creating a sense of favoritism or conflict of interest. Public-Private Partnerships. Stakeholder views on PPPs were limited. A donor stakeholder cited PPPs as an example of where TFA had ‘gone the extra mile’ and could be a valuable engine for growth in Timor-Leste. This view was emphasized by a stakeholder from the same organization during the data validation workshop, who agreed that the project’s work towards large-scale private investments was key. From a private sector stakeholder perspective, two respondents raised concerns that the project was ambitious in its work towards PPPs and that many of its pilots, despite the investment, may be unlikely to get off the ground given the difficult operating environment in Timor-Leste. This is supported by secondary research which suggests that Timor-Leste’s weak enabling environment and lack of financing may explain some of the paralysis in development of the tourism sector. 24 A partner stakeholder also emphasized the importance of the relationship with government to ensure legal frameworks were in place and that there would be sufficient buy-in to ensure success. “The other thing is that we need to have a clear intention from the Government that we want to do this, and if people who are in position does not think to develop tourism through private investment it will difficult everything because if there is no intention there will be no coordination and this, we have seen in many projects that’s on hold because there is no goodwill from the government.” –Partner stakeholder CONCLUSIONS FOR EQ1E The TFA project has built strong partnerships with private sector operators. This has been achieved primarily through the grant program, tourism events and the Tourism Champions Awards, all of which have been well received by tourism operators. The project has played an important role in tourism advocacy and uniting the sector under a common goal. The grant program has stimulated private sector investment in the Timor-Leste tourism industry. Grantees have improved the quality of their tourism offering, primarily through increased training opportunities, and access to resources, such as basic infrastructure. Grantee feedback is positive, not only in regard to individual benefits but increased tourism activity and revenue for the community more broadly. However, the full economic benefit of the grant program, at both operator and country-level, is difficult to quantify due to limited M&E and as such its impact on growing private sector investment in the tourism industry is not fully understood.25 The bureaucratic applications process for the grant program may have reduced inclusivity. Respondents noted issues with the application process, including the detailed financial information and reporting requirements stipulated by USAID, alongside limited support from TFA in grant preparation. This might have limited participation, potentially excluding Timorese applicants from smaller 24 Source: USAID Quarterly Progress Report Q2FY2021, page 29. 25 The ET acknowledges Tourism for All’s Grantee Highlights update, USAID Quarterly Progress Report Q2FY2021, page 19- 22. 27 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV organizations based outside Dili. Respondents welcomed a more streamlined process and increased support from TFA to enable greater participation. The impact of TFA’s work towards PPPs could not be fully assessed. Stakeholders regard this intervention both as a valuable engine for growth and an ambitious initiative given the challenging operating environment in Timor-Leste. However, the ET was not able to gather sufficient data to fully assess the likelihood of PPPs stimulating private sector investment in the tourism sector. EVALUATION QUESTION 1F 1F Has Tourism for All been successful/unsuccessful in the following key intervention areas: Establishing Sustainable Management Plans within protected areas? To answer EQ1f, the ET sought to first understand stakeholders’ knowledge of the Sustainable Management Plans (SMP) and then ascertain their views on its perceived level of success. The ET was aware that the Atauro Island SMP had been drafted and that a grant had been awarded to the Ministry of Agriculture and Fisheries (MAF) to conduct a demarcation of the Mount Ramelau Protected Area.26 The ET is also aware the Atauro SMP will be fully socialized with the community after it is presented to the Council of Ministers.27 As a result, stakeholder response to this EQ was very low due to limited awareness, with only two stakeholders able to directly discuss the SMPs. FINDINGS FOR EQ1F The MAF reported positive feedback on planning meetings held with TFA on the development of Sustainable Management Plans. However, there was a perception that work was in the communication phase and that no plans for development had yet been realized, in addition to confusion around when the TFA project had commenced. A second stakeholder, a private sector partner, questioned the purpose of the SMP in Atauro, asking the ET: “What is the relationship between ATKOMA Sustainable tourism plan and the Sustainable Management Plan for Atauro?” Although the community in Atauro did not comment on the SMP, positive feedback was received for sustainability initiatives, most particularly the concept of an eco-lodge which would further sustainable tourism initiatives on the island and socialize the concept of community-based tourism as opposed to mass tourism. An ATKOMA member also commented: “We want to work more with the TFA, because they give information, they have a meeting with the community in Atauro and we feel this is the best point to help in Atauro. We want to grow sustainably in Timor-Leste.” Speaking more broadly about the development of TFA policies and plans, a partner stakeholder raised concerns about a reliance on international consultants for some of the larger tourism initiatives – particularly those who are not based in Timor-Leste and have limited understanding of context. At the data validation workshop, a second stakeholder further highlighted the importance of engaging local or 26 Based on desk review and USAID Quarterly Progress Report Q2FY2021 27 Based on desk review and USAID Tourism for All Annual Report FY2020 USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 28 Timorese experts to support initialization of sustainable tourism plans. A donor stakeholder explained that protected areas and management plans would be more effective if institutionalized with government macro-level plans to support a greater level of ownership. This view was supported by a TFA representative who explained that the Atauro SMP has received input from a range of stakeholders across government, including the MAF.28 CONCLUSIONS FOR EQ1F Awareness and implementation of Sustainable Management Plans is limited. With only two stakeholders able to speak on this topic, the ET was unable to fully assess this intervention. While MAF representatives confirmed consultation has occurred, there appears to be limited stakeholder understanding of the Plans’ purpose or progress towards implementation. The benefit of this initiative for communities remains unclear at this stage. EVALUATION QUESTION 2 2 Is there evidence that the six key intervention areas outlined in Evaluation Question 1 are likely to grow, scaling up and out? b) What specific actions will be required to ensure future sustainability after USAID Tourism for All ceased? To answer EQ2, the ET sought to firstly understand stakeholder perspectives on the specific programs within TFA (based on their awareness from EQ1) which they felt were likely to grow, scaling up and out,29 then secondly to gather stakeholder feedback on specific actions required to ensure future sustainability of the initiatives and the project overall. While respondents were able to comment on factors affecting the durability of their projects post-intervention, information was scarce regarding mainstreaming of initiatives into law, policies, or programs; replicating initiatives under similar scale and conditions; or scaling-up over larger geographic areas and/or stakeholder groups. FINDINGS FOR EQ2 Durability. The majority of grantee respondents (6 out of 7) felt confident they could continue the work supported by their grant following project cessation. Three of the grantees noted the scale of activities would need to be reduced once grant funding had ended. Several grantees also noted that continuing the program, especially its private sector support, will be vital in ensuring the sustainability or realization of many initiatives. A grantee commented: “A lot of young Timorese have started business in this time thanks to what they’ve learned. There’s a momentum that was slowed down by COVID-19, so being able to continue things, and to motivate people to hang in there and keep pushing forward.” Another stakeholder noted that their grant had been made possible through mentorship from another grantee, suggesting 28 ET notes list of government stakeholders who provided input on page 77 of the Atauro SMP. 29 Given low stakeholder awareness and diversity of response per sub question (for example EQ1f), it was not practical for the ET to individually assess stakeholder response to each sub question under EQ1, 29 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV communities of practice amongst private sector operators were already established thanks to the project.30 Different experiences were reported between the two regional tourism associations of ATKOMA and ASTRABEKA in Atauro and Hatobuilico, respectively. A TFA stakeholder, speaking of the project’s work in Atauro, explained that members are now undertaking a range of activities independently, including bookkeeping and tour guiding, while the project continued to support with initiatives including the website and business plan. This view was supported by an ATKOMA member who expressed confidence in the sustainability of the association, suggesting training on some of the newer initiatives, such as the website, was already taking place and noting (as per EQ1c) that operators in Atauro had been part of informal private sector tourism groups since 2000. The stakeholder explained: “When the project is finished, we will continue alone, there is no problem. But we need more help when there’s not tourists; we want to practice a lot. In one year, I hope TFA is still going.” In Hatobuilico, a TFA stakeholder suggested that sustainability of the association will be more challenging; a view which is supported by TFA research.31 The stakeholder explained that the project is focusing on building the association’s capacity, the development of marketing materials, and linking ASTRABEKA to other organizations within Timor-Leste, which would bring in a diversity of stakeholders and is vital to its long-term sustainability. As outlined in EQ1c, the community is concerned that the association is still in its infancy and struggling to find suitable leadership. Community stakeholders in Hatobuilico also noted their limited understanding of community-based tourism, and as such, the additional support which will be required to develop and then sustain association operations. The project has acknowledged that the association models are working well but there is still uncertainty as to whether they will remain sustainable after the project concludes.32 The importance of sustainability was emphasized across all stakeholder groups. Six stakeholders commented on the importance of the longevity of donor-funded programs and urged the TFA project to continue. A private sector stakeholder commented: “TFA is very good for Timor tourism. I’d like that to be continued. A lot of time you see a project going on, half done, and then the funding finishes. And then why did we do those four years or three years and then it stops?” A partner stakeholder also shared that sustainability would be increased not only through the longevity of the project, but also the key staff within it: “If you recruit new people, you have to invest first, maybe for six months up to one year, then you lose the opportunity when they leave. No continuity and therefore no sustainability.” Capacity-building. Three private sector stakeholders emphasized the importance of training, acknowledging that customer service, financial, and leadership training was already a highly effective attribute of the project, which, if extended and replicated, could further support the sustainability of initiatives. One stakeholder suggested that training could be extended to facilitate funding of coordination between tourism businesses, providing opportunities to move people between businesses and enabling communities of practice. Another stakeholder noted that the project’s current and continued investment in youth was vital and that involvement from young people could help progress key initiatives. In speaking to the importance of training, a youth respondent from the private sector 30 The project has also noted that grantees are an important avenue to support sustainability – see for example USAID Tourism for All Quarterly Progress Report Q2 FY2021, page 20: “As a result, market opportunities and linkages will be facilitated, which will assist individuals and groups of beneficiaries to increase revenue and market share in the tourism industry.” 31 See USAID Assessment of Mt Ramelau 2018, particularly page 37. 32 Source: USAID Tourism for All Annual Report FY2020, page 51. USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 30 said: “I agree to support young entrepreneurs and women entrepreneurs. Every woman has their own story as I meet them, but they didn’t know how to do it. They want to teach other young women as well.” This respondent and those suggesting peer training and facilitating communities of practice denoted best practice strategies echoed in the UN Women report (2020) that advocates for these types of engagements as ways that women (and youth) can strengthen their work and social lives to improve the tourism industry overall and improve their ability to take on leadership roles within it. Ownership and Collaboration. Stakeholders across all groups explained that while the project’s work towards implementation of the NTP could generate significant benefits for tourism in Timor-Leste, the project’s lack of institutionalization with MTCI may impact its sustainability. A former senior government official said: “TFA has already presented the mechanism to implement the 5Ps. But the government doesn’t understand very well. The TFA has a good program, but not a good connection with the government.” This concern was acknowledged by TFA stakeholders given, as outlined in EQ1b, the Timor-Leste Tourism Development Strategy and Implementation Plan are still pending MTCI approval.33 This challenge was highlighted by stakeholders in the data validation workshop, several of whom suggested that high-level MTCI ownership of tourism policies and plans will be essential to ensuring sustainability. Four donor respondents emphasized that a collaborative approach between TFA and other donor projects could support the implementation of initiatives: “I think if we can all work together, to ensure that we’re transparent in terms of what the destination is, there’s a greater chance that activities will be sustainable.” It was agreed that the project could consult more broadly with stakeholders, including MTCI, about their desired direction and how the project could best support. This perspective was supported at the data validation workshop, with stakeholders agreeing that USAID could lead collaboration on tourism activities, including policies, and to increase both the effectiveness of the project initiatives as well as opportunities to scale up and out. Across the KIIs and data validation workshop, four separate stakeholders advocated for an increased use of MOUs between USAID and partners to ensure a collaborative approach. A donor stakeholder suggested ongoing collaboration with the World Bank could also ensure the continuation of TFA goals and achievements, given the World Bank’s upcoming tourism support project in Timor-Leste. As the TFA project may conclude around the same time as the World Bank project commences, important TFA initiatives, such as support to establish a Tourism Board, will unlikely just ‘die on the day of closing’ but could be sustained by the government, particularly if the close collaboration between TFA and the World Bank continues. CONCLUSIONS FOR EQ2 The likely sustainability of TFA initiatives varies significantly across the project. The project’s work with MTCI, primarily support to the implementation of the NTP, has been impacted by a lack of institutionalization and as such, stakeholders suggest that MTCI ownership of TFA initiatives, such as the Timor-Leste Tourism Development Strategy and Implementation Plan, will be essential to ensuring sustainability. Conversely, positive feedback was reported from the private sector with grantees confident that work will sustain following TFA secession. Clear differences were noted between tourism 33 However, recent research suggests a greater willingness on the part of the Directorate General of Tourism (DGoT) of MTCI to collaborate with the project. Source: USAID Tourism for All Quarterly Progress Report Q2FY2021, page 26. 31 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV associations in Atauro, suggesting that the Atauro community’s more advanced level of development (with tourism associations having operated informally since 2000) will increase the likelihood of sustainability compared to Hatobuilico where community-based tourism remains a new concept. Improved collaboration with MTCI, donors and other partners will support future sustainability of the TFA project. Additionally, ongoing training opportunities – for youth, women and to support greater coordination between tourism businesses – will enable stakeholder self￾sufficiency and less reliance on the project as it scales out. Continued collaboration with the World Bank on its upcoming tourism project could support sustainability of longer-term initiatives such as establishment of the Tourism Marketing & Promotion Board. EVALUATION QUESTION 3 3 a) What specific stakeholder groups within the tourism sector have benefited the most from USAID Tourism for All interventions? b) Are there additional stakeholders with whom USAID Tourism for All should engage in order to support effective implementation of the National Tourism Policy that could stimulate private sector investment in the tourism sector? To answer EQ3, the ET sought to understand respondent awareness of the stakeholder groups with which the project has interacted in order to ascertain stakeholder perspectives on which groups have derived the greatest benefit from the interventions. Of the stakeholders who responded to this question, excluding TFA project members, the majority were either private sector and partner stakeholders (9 out of 14) or donor partners (3 out of 14). FINDINGS FOR EQ3A Private Sector. Stakeholder feedback suggests that the private sector has been the greatest benefactor of the TFA program, particularly operators based in Dili and Atauro. The majority of private sector and partner stakeholders (8 out of 9) praised the project’s support and welcomed the focus on private sector initiatives. The project’s support towards the establishment of a Tourism Board, tourism events, the Tourism Champions Awards, the grant program and the Hau-Nia Timor-Leste domestic tourism campaign were the initiatives most commonly cited by respondents as instigated by the project in support of the private sector. A private sector stakeholder explained, “The TFA model is more business friendly. The smart thing they do is not get sidetracked by the government…” A private sector stakeholder from the data validation workshop noted that the project has been positive for the industry, particularly for its role in encouraging young people to be involved in tourism and see the present opportunities. At the same event, a donor stakeholder congratulated the program on the private sector energy and advocacy that it had created, while another private sector stakeholder explained that greater connections and collaborations had been formed between private sector operators because of the project. Community. Community stakeholders, particularly in Atauro, also appear to have benefitted significantly from the project. A TFA stakeholder explained the benefits of collaboration through associations, local government, and community activities which creates awareness and support for USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 32 communal, tourism-related goals, which in turn has a positive impact on the communities. This view was supported by community members in Atauro who reported gains from the project including improved customer service at restaurants and guest houses, the establishment of working groups including for women and youth, as well as regional communities outside Beloi (for example in the hills of Makadadi, Berau, and Akrema) now showing interest to tourism activities.34 An ATKOMA member said: “For me, USAID is the best [organization] to come to Atauro; it is helping a lot.” “When the community are sitting together, they make sense of the knowledge of the business, talk about how to generate the income, attract more tourists to visit in their place... And they start feeling the impact of tourism activities. Beforehand, there were no activities. Only a private company running a business, but no socialization with the community. No one showing them that tourism is a good impact to the community. But through the association, through the local government and through the socialization with the community, they are starting to see it as a good impact for the community. Starting to create tourism groups and taking tourists around the village. So, they feel that gives good impact for their lives.” –TFA stakeholder FINDINGS FOR EQ3B To answer the second part of EQ3, the ET probed stakeholder responses to the project’s engagement with marginalized groups such as women, youth, and those living with a disability. Several stakeholders (5 out of 14) noted that the program did not necessarily need to engage new stakeholders, but rather improve relationships with existing stakeholders through increased consultation and a shared agenda. Government Engagement. As outlined in EQ1b, the project has struggled to build a strong working relationship with MTCI. A donor stakeholder proposed that MTCI might not have benefited from the TFA project as much as it might have initially hoped. This view was reinforced by a partner stakeholder who said: “The TFA has a good program, but not good connection with the government. They have many projects, but the MTCI has a different perception.” This issue was also noted by TFA project stakeholders during consultations and through secondary data analysis conducted by the ET.35 As outlined in EQ1b, MTCI has advocated for improved communication with the project and a greater sense of ownership – “for the project to be country owned and country led.” Donor Engagement. The issue of donor coordination has been noted as a priority by TFA.36 However, in speaking broadly about the project, many donors (4 out of 5) contended that collaboration with the project has been fraught. In some cases, this has resulted in the duplication of initiatives, for example work around aviation policy, and mixed messages. A donor stakeholder explained: “We have tried hard to improve coordination, but it remains the greatest challenge. Not entirely what it should be. A sense of competition, and we’re missing opportunities to work better together. We would like to see better 34 This is further supported by secondary data analysis which suggests that in FY2020 Tourism for All reached 330 rural households (more than the target of 196) who benefited from project interventions including training, technical assistance, financial support and access to job opportunities. Source: USAID Tourism for All Annual Report FY2020 page 44. 35 For example, see USAID Tourism for All Quarterly Program Report Q2 FY2021, page 31: “There remains significant opportunity to support the GoTL to further prioritize tourism as an economic diversification opportunity for the country. However, leadership from MTCI is essential to guide this process.” 36 For example, see USAID Tourism for All Quarterly Program Report Q2 FY2021, page 31: “It is imperative that any intervention initiated by USAID’s Tourism For All is considered in coordination with the activities of other donor-funded programs to prevent misunderstanding among donors, duplicate efforts, or cause confusion among beneficiaries.” 33 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV communication and coordination.” The point was echoed by a second donor stakeholder who said: “It’s much more valuable if we work together. Governments will change; the TFA project will end at some point – as will (other donor programs) but Timor-Leste will remain as will the need to grow tourism.” The view was also supported by a partner stakeholder who reinforced the importance of partner collaboration, suggesting: “It’s important to communicate with other donor partners to see what their program is and also the personal style of leadership.” Geographic Accessibility. Despite positive feedback from the project’s engagement with communities, concerns were raised around TFA’s accessibility to regional and remote communities. As outlined in EQ1e, stakeholders suggested that smaller organizations were unlikely to have been able to take advantage of grant applications without support. A partner stakeholder intimated that international consultants have often been hired for work that might have best been delivered by local NGOs, community members, or local tourism experts. While the project is named ‘Tourism for All’, stakeholders noted that the USAID intervention has only focused on three geographic regions within Timor-Leste: Dilli, Atauro, and Hatobuilico.37 While many stakeholders agreed the focus is necessary to generate a best practice ‘model of tourism development’ for replication in other regions, four stakeholders raised concerns that stakeholders outside these regions have been overlooked. A private sector stakeholder commented that the decision to only support three regions has caused market distortion: “By only supporting people in a geographic area they are crowding out operators in other districts. The tourism dollar only goes so far, distorting the marketing, putting people out of business, and losing their jobs.” Another private sector respondent questioned if there was an opportunity to provide marketing and simple support for operators beyond the three chosen regions. The stakeholder proposed USAID and TFA stakeholders consult with operators from other regions and support their businesses, for example at a minimum, by staying at local hotels. In terms of Gender and Social Inclusion, the ET also found many grantees had been encouraged by TFA to consider inclusion of women and youth in their grants. A private stakeholder and successful grantee explained: “We’ve been able to make it gender-focused with our grant. Women are on the front lines of tourism and are often people with kids managing houses. For me it’s less about using the word gender – but more doing it as a practical thing. Women to women exchanges so they can speak up about tourism issues. All our women to women activities were more successful than other activities.” This perspective was enforced by a private sector stakeholder in the data validation workshop, who emphasized the importance of women’s digital marketing and coffee training experiences, as well as opportunities for young Timorese, which had been made possible through the TFA grant program. Stakeholder responses about the involvement of women in tourism are reinforced by secondary literature. From a UN Women Report (2020), women make up the majority of the workforce in the Timor tourism industry, often doing work that reflects tasks socially considered “women’s work,” such as operating guest houses, preparing and selling food and beverages, and producing craft work. However, tasks taken up by women tend towards to reinforce the acceptability of low wages, maintain their employment at lower levels – rather than in setting tourism policy and management – keeping women out of decision-making roles. Social norms that enable women to be active in the tourism industry are reported to also be potentially problematic in that tasks deemed to be “women’s work” excludes men from supporting women to perform common tourism industry tasks, further enforcing 37 The ET notes the geographic parameters for the TFA program were set by USAID to ensure a focussed intervention. USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 34 stigma that keeps women in lower-paid labor-intensive roles, but away from leadership within the industry. Supporting women to take up a majority of leadership roles, to match their majority representation within the tourism industry, would require cultural shifts beyond the scope of TFA. Within the TFA mandate, TFA grants have primarily been awarded to women and there is a woman in a top leadership post for the last three years. However, some stakeholders have questioned the extent to which women have been fully incorporated into the TFA project. For example, one stakeholder commented that most of the TFA staff they had engaged with at meetings had been all men. TFA has acknowledged the importance of supporting women and marginalized groups, with notable achievements such as 80 percent of grant beneficiaries being women,38 and training provided to 850 people – over 550 of which were women.39 The same UN 2020 study on women in Timor’s tourism industry strongly advocates for providing education and leadership opportunities for women, as a means to improve women’s businesses and to foster greater female participation in leadership positions within the industry. Specifically, the report recommends strategies such as, supporting gender-responsive procurement measures to empower marginalized small and medium-sized business owners who might otherwise not have access to procurement opportunities. A TFA representative also explained that the project, through its coaching and mentoring initiatives, is supporting women to grow in leadership roles in tourism, breaking them away from a traditional patriarchal society. However, this work is being done in a slow and gradual manner so as not to harm local cultural norms. A table detailing the reports’ timely recommendations are mapped onto USAID’s gender domains of analysis in Annex III. CONCLUSIONS FOR EQ3 The private sector is regarded as the main benefactor of the TFA project. Private sector stakeholders, most particularly grantees and those in the focal geographic areas, welcomed the project’s support and initiatives including the establishment of a Tourism Board, tourism events, the Tourism Champions Awards, the grant program, and the Hau-Nia Timor-Leste domestic tourism campaign. Community members are regarded as the second largest benefactors of the project, with the Atauro community highlighting improvements in community awareness of tourism and customer service. As outlined in EQ1c, there also appears significant scope to improve collaboration and integration with community members in Hatobuilico. There is scope for the project to improve relationships with existing stakeholders – primarily MTCI and Dili-based donors. While the project has expressed a strong desire to work more closely with both MTCI and other donor partners, collaboration appears to be fraught. This has resulted in poor communication and, in some instances, a duplication of activities. However, both MTCI and donors welcome improved engagement and there is an opportunity for USAID to consider engagement with stakeholders outside its main geographic focal areas as well to support greater inclusivity of all operators and partners and ensure the tourism support is available ‘for all.’40 38 See USAID Tourism for All Annual Report 2020, page 37. 39 From consultation with USAID Tourism for All Chief of Party. 40 The ET notes the geographic parameters for the TFA program were set by USAID to ensure a focussed intervention. 35 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV Marginalized stakeholders – including women and youth – have benefited from the project. Inclusion of their needs is a requirement to be addressed by all grant applications and the ET notes that 80 percent of grant beneficiaries are women. However, it is not possible to assess the overall results and/or benefits for these groups as longitudinal data is not available. EVALUATION QUESTION 4 Given efforts by the Government of Timor-Leste (GoTL) and various development partners to more effectively implement the National Tourism Policy or to stimulate private sector investments in the tourism sector, what is USAID’s comparative advantage to achieve or advance these same goals? 4 To answer EQ4, the ET sought to better understand stakeholders’ feedback on the successes and challenges of the USAID TFA approach and, where possible, to compare these to other donor-funded programs in the sector. As stakeholder awareness across the breadth of all donor programs was limited, the majority of respondents (11 out of 18) commented solely on the USAID approach with only four donor and three private sector stakeholders able to compare the project to other donor initiatives. FINDINGS FOR EQ4 Scale and Concentration. In comparing USAID’s investment to other donor work in tourism, the majority of respondents (6 out of 7) cited its larger size and impressive reach. A private sector stakeholder argued that its advantage lies in its focus, given it is fully concentrated on tourism (unlike other donor programs which are often multi-sectorial): “To me they (USAID’s TFA) are a different scale altogether – what they’re doing compared to other programs in the tourism sector. They are more complete.” Another private sector stakeholder praised the project’s collaborations with local organizations, suggesting that while some programs struggle to achieve community impact, USAID through TFA, has focused on local and grassroots tourism – such as supporting training and local businesses such as coffee shops – which has been highly effective. “What I know about who was on the receiving end – it’s always been with a bigger picture in mind. There’s a human factor within the team of people we’ve met, you can see they were good at sensing the groups to plug into.” Respondents also responded positively about its selection of qualified tourism staff and its well-respected Chief of Party.41 That USAID branding is prominently used to signify program achievements was cited as both a positive and negative attribute. “I think it was admirable that they put in a huge effort to do something at a time when not much was happening in tourism.” –Private sector stakeholder Initiative. Respondents also noted that USAID, throughout the project, has taken an approach to development that sets it apart from other donor partners in the sector. A private sector stakeholder explained the point as differentiating between waiting for what the government wants and elevating what a project suggests it needs. This point was echoed by donor partners, with one explaining: “As 41 Several stakeholders noted that the project’s change in Chief of Party in 2018 had been exceptionally well received and that the project’s achievements and stakeholder collaboration had improved significantly under the new leadership. USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 36 development partners we’re good at identifying the things that need to be done – and wait for government or tourism businesses to pull the trigger on them. At times that’s good for inclusive sustainable development. Sometimes it’s good to pull the trigger and just do it. For better or worse, TFA and USAID have been successful in just pulling the trigger on things.” A second donor stakeholder suggested that while the two approaches were different it was not to say one was better than the other and explained: “there is space in the sector for both.” Respondents also noted that the advantage of this approach was that it could jumpstart initiatives on which other donor programs could build. It was also noted that USAID’s project had played an important role during the COVID-19 shutdown when several donor programs in Timor-Leste had pivoted away from the tourism industry. Disadvantages. Conversely, stakeholders noted disadvantages with the USAID approach as not consistently developing a strong evidence base for all initiatives – such as the economic value of domestic tourism, – being as collaborative with other donor partners, or ensuring projects are government-led. A former senior tourism expert argued the greatest challenge with the USAID approach had been its lack of integration with MTCI, explaining: “Sometimes the communication, the coordination, is not there... How to integrate the TFA program with the government program. If you integrate it well, everything will go ahead... People are rational. How can you make (the government) empathetic with the project?” This point was reiterated in EQ1b by government stakeholders, who explained that USAID, through the project, had not been fully institutionalized and was thus lacking a sense of ownership from MTCI. CONCLUSIONS FOR EQ4 USAID's TFA project has taken a different approach to other, smaller donor programs in the Timor-Leste tourism industry. Stakeholders describe the approach as essentially undertaking what the government needs as opposed to always waiting for government direction on what it wants. The advantage of the approach is the ability to tackle larger, bolder initiatives and ‘pull the trigger.’ USAID’s investment is also larger than other donor tourism projects in Timor-Leste in both scope and budget. The disadvantage of USAID’s TFA approach is that it has in some instances resulted in a lack of collaboration with other Dili-based donors and limited integration with MTCI, the latter of which may impact sustainability of initiatives (as outlined in EQ2). EVALUATION QUESTION 5 To answer EQ5, the ET found it necessary to divide the question into three parts: 1) stakeholders’ views on how the TFA project had adapted in response to COVID-19, 2) stakeholders’ perceived effectiveness with the project’s adaptation, and 3) stakeholders’ feedback on future project adjustments to mitigate the impact of COVID-19 on the industry. The ET was aware that the TFA had undertaken significant work to support industry following COVID-19, most notably the Tourism Reboot Initiative and the domestic marketing campaign. A majority of stakeholders (22) were able to provide feedback on How has the TFA Project adapted in response to the impact of COVID￾19 and how effective has this been? What additional adjustments should the project consider in order to future mitigate COVID-19's impact on the industry? 5 37 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV the first part of the question, with 18 responding to part two, and only six stakeholders able to respond to part three. FINDINGS FOR EQ5 Domestic Tourism Campaign Pivot. Respondents primarily recognized the project’s pivot to the Hau-Nia Timor-Leste domestic marketing campaign, move towards online communications, and support of COVID-safe training programs through the grant program initiative as leading the process of recovery for tourism in Timor-Leste. As outlined in EQ1d, the Hau-Nia Timor-Leste campaign had high recognition and was held in high regard among many stakeholders particularly private sector and partners (15 out of 18 mentioned and praised the campaign). When Timor-Leste went into further lockdown due to an outbreak of COVID-19 cases (early- to mid-2021), the program further pivoted by running a campaign on the Facebook page to stimulate takeaway. A TFA stakeholder explained: “We were getting messages of thanks for the support. Giving them a sense of hope during the difficult time. Practical support, advice, and financing when called for. Just listening to them. No one else has really been there for them.” Three stakeholders also commented on the importance of the program during COVID-19. While other programs could pivot away from tourism, TFA remained in the industry and the support, particularly provided to the private sector, was important. Grants program. The grants program was also highlighted by stakeholders (10 out of 18) for the important role it has played in keeping businesses alive during COVID-19 and providing hope during a difficult time for industry. A private sector stakeholder explained: “The grant process has helped people to survive through the COVID-era. The better businesses will survive, they will find a way. The other tourism programs haven’t had that impact of flexibility. They don’t have that understanding of businesses.” As outlined in EQ1e, the grant program was also noted, specifically by donor stakeholders, as potentially providing false hope for businesses and helping stimulate supply during COVID-19 in the absence of demand. A private sector stakeholder also noted the program had primarily supported businesses in the project’s three main regions: Dili, Atauro, and Hatobuilico which could place regional businesses at an even further disadvantage during a challenging time. Grant recipients praised the project for its flexibility in grant process, with a grantee commenting: “It was all going to be face to face, then online, then we had flexibility to do digital marketing but making them COVID-safe. At every milestone we had to meet, they were able to pivot. I am confident to say there is not really another project in Timor-Leste that has been like this one in terms of understanding the industry and adapting to reality.” It was also noted that the program supported businesses to develop COVID-safe training videos for the community, which not only provided useful information, but brought local business together under the common goal of improving hygiene standards.42 At the data validation workshop, a respondent described the project’s response to COVID-19 as: “excellent, timely and well thought out. What was needed.” The stakeholder also praised the project’s support of local tourism experiences – for example through the domestic campaign and grant program. Online Coordination. The project’s pivot to online communication was also discussed by a number of stakeholders (9 out of 18). Feedback from ATKOMA in Atauro suggests the move to online communication has been largely positive, with a member suggesting meetings have taken place online and via WhatsApp: “We are still working with TFA. Preparing tour guides, information. COVID-19 was coming 42 As also outlined in USAID Tourism for All Quarterly Progress Report Q2 FY2021, page 20. USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 38 and we kept going. (There are) good systems in place.” Greater challenges were noted for ASTRABEKA and the community in Hatobuilico who raised concerns that the program had stalled due to COVID-19 and that communication had been limited. A partner stakeholder explained that TFA’s necessary pivot to online training programs, while valuable, had meant training was not always accessible for Timorese in more remote communities, due to the poor telecommunication infrastructure across Timor-Leste. Reflection and Learning. Stakeholder feedback on additional project adjustments to further mitigate the impact of COVID-19 on the industry was limited (6 respondents) and focused largely on the need for USAID to improve relationships with other donors and MTCI to ensure a coordinated approach to industry recovery. A representative from TFA outlined that the project had worked to ensure a collaborative industry-wide response to the pandemic, with TFA’s Timor-Leste Tourism Reboot Initiative strategy shared widely with stakeholders across government, industry, and donor partners. Stakeholders reinforced the importance of readying the industry now to better prepare the nation once borders reopen , with a private sector respondent suggesting: “COVID-19 has given the time for people to work on their products from safety to capacity-building and training and so on. It makes you better prepared for tourists… But with the program, they can still use the time of the crisis to ready themselves for international tourists. It’s a unique opportunity that challenge of COVID-19 that gives you time to prepare a few things.” At the data validation workshop, a partner stakeholder emphasized the importance of developing a digital market platform for Timor-Leste, which, once the country re-opens for tourism, could help reduce the reliance on foreign assistance.43 “I think the (TFA) project is probably even more valuable now. We know Timor-Leste needs to develop tourism as a priority industry. It’s just been hit by COVID-19; when things start to clear up and there are opportunities for travel to return, if MTCI is not prepared, we’ll miss that opportunity.” –Private sector stakeholder A private sector stakeholder also shared the important role the project had played in providing training opportunities throughout the pandemic and equipping young Timorese for a future in tourism. The stakeholder discussed not only the importance of training programs, but how the project had enabled the transfer of traditional knowledge through associations and coordination. This was echoed by a stakeholder in the data validation workshop who emphasized the importance of training young entrepreneurs about hospitality and administration, business, surveying, taxation, and customer service. A partner stakeholder suggested the project could engage more local expertise to manage programs and training during the COVID-era to ensure understanding of context and reduce reliance on online methods. CONCLUSIONS FOR EQ5 While other donors pivoted away from tourism activities, TFA’s ability to adapt and shift in strategy due to COVID-19 was particularly well received among diverse stakeholders. TFA adapted by shifting to online formats, creating a highly visible domestic marketing campaign for tourism (and then local support during lockdown), and providing grants programs to tourism-related businesses and associations. However, challenges were noted in the accessibility of some programs 43 ET notes that Tourism for All has already been supporting the Timor-Leste destination website, digital domestic marketing, and digital marketing training for ATKOMA members in Atauro. 39 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV (grants and online training) for more remote communities due to challenges with internet connectivity across Timor-Leste. TFA is regarded by stakeholders as an important project to support tourism recovery from COVID-19. Stakeholders suggest the project’s response could be strengthened via greater coordination with MTCI and other donors, ongoing training opportunities to facilitate ‘knowledge sharing,’ a greater reliance on local expertise to reduce reliance on internet accessibility in remote communities, and a continued focus on digital marketing to prepare for border re-opening. USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 40 RECOMMENDATIONS Recommendations synthesized from the conclusions are presented below and grouped by stakeholders to which they are targeted: USAID, implementing partners (TFA and future), or both. # RECOMMENDATIONS #1 USAID should lead the reinstatement process for a donor working group that includes the Australian Department of Foreign Affairs and Trade (DFAT), the New Zealand Ministry of Foreign Affairs and Trade (MFAT), the World Bank, and implementing partner agencies. The working group should lead the identification of opportunities for shared agendas and collaborative work with specific responsibilities for each donor and/or implementing partner outlined through an MOU. #2 USAID IPs should facilitate improved collaboration with the Timorese government including MTCI and other ministries. Consider embedding a staff member within MTCI to further support communication and personnel transition processes, as well as NTP implementation. If deemed appropriate, develop MOUs between TFA and MTCI/other relevant government ministries to clearly delineate roles and responsibilities and contribute to a common understanding of government and IP objectives. USAID/IPs could continue to engage the government in private sector-led initiatives, such as establishment of a Tourism Board, and engage government, in coordination with the World Bank, to support aligned activities and further work towards inter-ministerial collaboration and regulatory reform. #3 USAID IPs should continue private sector engagement activities (dependent on COVID-19 guidelines) including the Tourism Champions awards, tourism fairs, and conferences, given their strong support and positive feedback. While the Tourism for All grant program model should be continued, especially given its successes including high levels of women and youth grant applications, improvements can be made to increase its effectiveness as a tourism driver. USAID should consider a more inclusive application process that supports those with disabilities and ensures that financial reporting requirements are understood and not unduly burdensome on youth and female-applicants (who may have difficulty accessing credit based on their age or sex). IP should consider a multi-stage process, enable an initial pitch/idea discussion before full application submission as well as greater support to potential grant applicants through increased on the ground training/information sharing facilitated by TFA staff. In addition, IPs could improve M&E grant implementation tracking across all recipients, measure program objectives against output, and (where possible) provide an indicator of ROI. #4 USAID IPs should continue and scale up training programs for the private sector. Consider increased training opportunities for communities outside the main centers of Dili, Beloi, and Ramelau. Develop a strategy to increase and/or improve awareness of current trainings for Timorese youth including opportunities to collaborate with universities. In addition, engage local expertise to support TFA grantees to enhance sustainability of programs. 41 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV #5 USAID IPs should continue to support the key tourism associations of ATKOMA, ASTRABEKA, and ATM-TL. Specifically, support should be provided to improve implementation processes such as assistance to establish organizational and leadership structures, member training, and provide a paid local consultant/staff to support association mandates during startup (1-2 years) and ensure that information around funding and/or grant processes consider the needs of women, youth, and marginalized groups. In addition, develop a communication strategy to improve collaboration and clearly delineate IP, MTCI, and community roles and responsibilities. #6 USAID IPs should develop a comprehensive strategy based on how the TFA project has, and can continue to, engage female tourism stakeholders and entrepreneurs. TFA could map and collate its successes working with women to date, identify new opportunities, and refine its strategy for future female engagement that leverages what is working well. The strategy could consider: a. How the project can/is supporting marginalized women in tourism, including those in informal sectors such as handicrafts. b. Opportunities to meaningfully engage women as leaders in the tourism industry, ensuring women’s groups are not only grantees but also program partners. #7 USAID IPs should continue support for a domestic marketing campaign with key modifications to concurrently build brand awareness and increase ROI. Continue to work with MTCI to support marketing training, ensuring transfer of ownership once TFA concludes. Consider increasing focus on the expatriate market to stimulate demand and increase revenue for hotels, use of domestic campaign imagery to build brand awareness for the international market under the international slogan, and incorporation of positive environmental messaging in the campaign. Increased M&E could support measurement of ROI or local awareness building. #8 USAID IPs should further engage local expertise including local partner organizations, tourism grantees, and/or advocates, as outlined above, to manage programs during the COVID-era both to ensure understanding of context and expand beyond reliance on online methods. USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 42 ANNEX I: PERFORMANCE EVALUATION SCOPE OF WORK USAID/Timor-Leste USAID Tourism for All Project Performance Evaluation STATEMENT OF WORK I. TITLE OF EVALUATION Performance Evaluation for USAID Tourism for All Project II. BACKGROUND Timor-Leste is a land of rich heritage and beautiful scenery with ocean waters labeled some of the most biodiverse in the world. A hidden gem in South-East Asia, it gets a fraction of the visitors’ regional neighbors, despite having some of the best diving and trekking available anywhere. The government’s national tourism policy calls for developing a major tourism industry by 2030 and USAID is at the forefront of this effort, which will make the country more self-reliant. Timor-Leste recognizes its oil￾dependent economy must be diversified and sees tourism as a logical means to that end. USAID Tourism for All promotes heritage-based, eco-friendly tourism as the market niche most suited to growing tourism in Timor-Leste. In 2018, USAID awarded $9 million for a three-year (January 2018-January 2021) contract to Chemonics International for USAID’s Tourism for All Project with two potential options years (January 2021- January 2023). USAID’s investment in the tourism sector through the Tourism for All Project promotes Timor-Leste’s competitiveness as an international tourism destination while preserving its unique environmental and cultural heritage. The purpose of this activity is to expand and improve Timor-Leste’s tourism offering using an inclusive and sustainable approach; to facilitate private investment and co￾creation by working with government to establish incentives and obligations that are more private sector-friendly; to promote steady and sustainable tourism growth; and to safeguard and conserve the environment and social characteristics of Timor-Leste’s local communities. The USAID - Chemonics International contract was signed in January 18, 2018 for $14,997,010.00 with the following breakdowns for based and options years: Base Year of $9,035,036 and Two Options Years of $5,961,974. The project start-up operations were delayed briefly due to the changes in the Chief of Party after the award. The new Chief of Party was only on board in September 2018. Since then, the project has made considerable progress in the implementation of planned activities toward meeting project objectives. The project staff was re-structured to align with the Government of Timor-Leste’s National Tourism Policy by having Leads in the five government priority areas in Tourism which include priority, prosperity, protection, partnership, and people. In July 2020, the Mission approved the exercise of Option Year 1 which extended the performance duration of the project for 12 months from March 19, 2021 to March 18, 2022 inclusive of two months no-cost extensions due to COVID-19 pandemic. Thus, as the project completed the base year and starting its option year one, it is important that an objective outside review be undertaken to assess 43 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV progress made towards reaching stated goals and objectives, as well to validate assumption and results indicators against the reality on the ground, based on USAID’s Tourism for All implementation experience to date. When COVID-19 global pandemic hit Timor-Leste, USAID/Timor-Leste mission postponed the midterm evaluation as initially planned due to the inability to field an external ET. While the COVID-19 situation remains the same, a performance evaluation of the TFA activity is still critical to the success of its implementation. Therefore, Mission will continue to carry out this evaluation to inform the Mission about the activity’s performance and achievements and make programmatic adjustments. III. PURPOSE The objective of this assignment is to carry out a performance evaluation of USAID’s Tourism for All project to: ● Determine the relevance, efficacy, impact, and sustainability of the impact achieved. The evaluators will undertake a normative assessment of the project to gauge progress made in the implementation of planned activities toward reaching stated goals and objectives. ● Assess the wider project context to validate project assumptions and results indicators against the reality on the ground, based on actual USAID’s Tourism for All implementation experience to date. This should be done with a view toward making recommendations for adjustments of assumptions and results indicators, as warranted, to enhance project implementation if further option years are exercised. ● Assess the impact of COVID on the tourism industry and the Tourism for All activity and whether there should be any shift in focus should a second option year be awarded. ● Assess the effectiveness of the Tourism for All activity to collaborate, learn, and adapt in light of the COVID-19 crisis. ● Provide an evidentiary basis for determining whether or not to exercise the second option year. IV. AUDIENCE AND INTENDED USES The primary stakeholders to benefit from the findings of the performance evaluation include: A. USAID for the purpose of potentially exercising future option years on the project; B. The USAID Tourism for All team, including Chemonics and its partners, for adaptive management for the remaining Life of Project if the option years are granted or for future programming; C. USAID’s external bilateral, regional, and international partners and key stakeholders addressing tourism issues across the Asia-Pacific region to share lessons learned, including participating host country government agencies and other international partners; D. The U.S. Embassy, Timor-Leste, which is tracking tourism as a critical economic growth sector. V. EVALUATION QUESTIONS The Evaluation Team must provide professional services sufficient to meet the purposes set forth above. The Evaluation Team will address and build upon the following Evaluation Questions: USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 44 1) Has Tourism for All been successful/unsuccessful in the following key intervention areas: a. working effectively with stakeholders to develop tourism-related policies or regulations; b. supporting implementation of the National Tourism Policy; c. establishing effective and sustainable tourism sector associations and advocacy groups; d. supporting destination marketing domestically and internationally; e. building partnerships with a reasonable likelihood of stimulating private sector investment in the tourism sector; and f. Establishing Sustainable Management Plans within protected areas. 2) Is there evidence that the six key intervention areas outlined in Evaluation Question #1 are likely to grow, scaling up and out? What specific actions will be required to ensure future sustainability after USAID Tourism for All ceased? 3) What specific stakeholder groups44 within the tourism sector have benefited the most from USAID Tourism for All interventions? Are there additional stakeholders with whom USAID Tourism for All should engage in order to support effective implementation of the National Tourism Policy that could stimulate private sector investment in the tourism sector? 4) Given efforts by the Government of Timor-Leste (GoTL) and various development partners to more effectively implement the National Tourism Policy or to stimulate private sector investments in the tourism sector, what is USAID’s comparative advantage to achieve or advance these same goals? 5) How has the Tourism for All Activity adapted in response to the impact of COVID-19 and how effective they have been? What additional adjustments the Activity should consider in order to further mitigate the COVID-19 impact on the industry? It is envisioned that once the USAID Tourism for All evaluation team is on-board, further “sub￾questions” will be developed for each of these. The evaluation team must present findings to substantiate answers to these questions; findings must be based on facts, evidence, and data, and be specific, concise, and supported by quantitative (where appropriate/available) and qualitative information that is reliable, valid, and triangulated. Recommendations must be action-oriented, practical, prioritized and specific. VI. GENDER, FEMALE EMPOWERMENT AND SOCIAL INCLUSION As best practice in evaluation design and implementation, it is essential that evaluations be engendered and properly explore and adequately capture the situations and experiences of men and women, boys and 44 Stakeholders refer to the following: ● Timorese women ● Timorese youth (ages 18 to 35) ● Marginalized groups in Timor-Leste such as LGBTI, religious minorities, & persons with disabilities ● Tourism associations Tourism for All established such as ASTRABEKA, ATKOMA, ATM-TL ● Dili Municipal Tourism Authority ● Hotel Owners Timor-Leste Association ● Government counterparts: Ministry of Tourism, Secretary State of Arts & Culture & Ministry of Agriculture and Fisheries ● Dili Institute of Technology 45 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV girls, where relevant. Using the primary research question set noted above, the researchers should apply a gender lens to ascertain the following, 1. The extent to which gender equality was achieved in regard to the program’s opportunities (i.e., beneficiaries’ participation), results, and female empowerment, as defined in USAID’s Gender Equality and Female Empowerment Policy; 2. The extent to which any other categories of social inclusion (age, disability status, lesbian, gay, bisexual, transgender, and intersex (LGBTI)-identification, and race/ethnicity) were important in shaping the program’s results. In order to understand the gender differences and gender-specific effects, the researchers should utilize a mixed method approach; however, at a minimum, the sex-disaggregated individual-level data should be analyzed. If the sex-disaggregated data does not exist, those gaps should be identified. In the draft and final report, where possible, the data should always be presented in a sex and age-disaggregated format when possible. Additional information can be found in USAID’s How-To-Note: Engendering Evaluation at USAID (December 2016). VII. METHODOLOGY USAID/Timor-Leste does not define a full methodology for the evaluation within this document, but rather outlines several required components of a methodology that the Contractor should build upon in the technical proposal. The Evaluation Team will examine both quantitative and qualitative approaches and a combination of secondary (existing) and primary (new) data during the course of the evaluation. The Contractor is encouraged to propose additional or different evaluation methods as deemed appropriate, but evaluation methodology must include at a minimum the following: 1) Desk Review. At the initial stages of the evaluation, the Evaluation Team will conduct a desk review of all relevant USAID Tourism for All reports and findings to date. This will include the original contract, the annual work plans and annual reports, progress reports, Monitoring, Evaluation, and Learning (MEL) plans, as well as key technical deliverables. In addition, the Contractor is required to review other relevant literature/documents from non-USAID interventions and research. Through this exercise – as well as discussions with USAID staff – it is expected that the aforementioned Evaluation Questions will be further refined, and the Evaluation Team will be in a strong position to develop the evaluation framework and instruments. 2) Stakeholder Engagement. Throughout its duration, USAID Tourism for All has engaged a wide-range and large number of stakeholders. These include but are not limited to: National/Sub-National GoTL officials, tourism-related businesses, and local and international nongovernmental organizations. Refer to footnote on page 3 for more details. The Evaluation Team will develop a series of detailed research questions that are tailored to these individual stakeholder groups in order to sufficiently collect data/information to effectively answer each of the final evaluation questions as agreed by the Mission. The Contractor is required to propose specific approaches/combinations of approaches that are the most effective for conducting this USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 46 evaluation. This will involve a mix of a range of data collection and analysis methods such as virtual and/or face-to-face survey and/or other methods, as appropriate. 3) Key Informant Interviews/Focus Groups. The Evaluation Team will identify those individuals who are classified as ‘Key Stakeholders’ – individuals/ institutional representatives that will be most impacted by USAID’s continued engagement in Timor-Leste’s tourism sector. The team will conduct in-depth interviews and/or focus groups to gather more detailed information associated with the proposed Evaluation Questions. Geographical Coverage Institutions, businesses, and communities located in Dili, Atauro island, and Mount Ramelau areas. VIII. DELIVERABLES AND WORK PLAN The contracted Evaluation Team must produce the following work and materials: 1) Performance Evaluation Methodology and Workplan: The Evaluation Team must submit a detailed evaluation methodology that illustrates what kinds of evidence the team will use to answer each evaluation question and clearly define methods and tools for collecting this evidence. The Work plan must provide a timeline for the different stages of the evaluation work and delineate responsibilities among the Evaluation Team members. The Contractor will identify local holidays, religious events, or other factors that may affect the evaluation’s work schedule and incorporate the findings into the workplan. 2) In-Brief with USAID/Timor-Leste. The Contractor must make arrangements through the Evaluation Manager to provide relevant staff at USAID/Timor-Leste with an introductory oral briefing upon completion of Deliverable 1. At this meeting, the Contractor will be expected to review the final design and implementation plan, as well as final data collection instruments. The Contractor should be prepared to brief the U.S. Ambassador to Timor-Leste, if required. 3) Field Data Collection The Contractor must conduct field work in Dili, Atauro Island, & Mt. Ramelau within appropriate timelines. The Contractor will report to the Evaluation Manager immediately if any factors/occurs that may affect the data collection to collaboratively find solutions. 4) Out brief with USAID/Timor-Leste. The Contractor must present a summary evaluation purpose and methodology, as well as preliminary findings to all interested staff at USAID/Timor￾Leste and then incorporate comments received during the briefing into the evaluation report. The out-brief purposes are for the Contractor to obtain additional inputs/insight on the contexts and data interpretation and to solicit ideas for actionable recommendations, as well as for Mission staff learning. 5) Validation Consultation with USAID, Implementing Partner, and Other Stakeholders. The team will independently present the major findings of the evaluation to the 47 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV relevant USAID staff, partners and other stakeholders including Chemonics, Inc. in Dili. The debriefing will include a discussion of findings, conclusions, and recommendations. The evaluation team will consider partner comments and draft reports accordingly, as appropriate. This event can be held virtually or in-person if conditions permit45. 6) Draft Performance Evaluation Report: The Evaluation Team will produce the evaluation report using a standard template provided by USAID/Timor-Leste. The report must meet the criteria outlined in USAID’s Evaluation Policy. 46 The Contractor must propose concrete recommendations that can be offered for consideration in the design of future USAID Tourism for All interventions. The USAID/Timor-Leste will provide written feedback on the draft report according to the agreed timeline. 7) Final Performance Evaluation Report: See Annex 1: Requirements for Submission of Reports for more details. 8) Data Submission: All data collected by the evaluation team must be provided in machine￾readable, non-proprietary formats and submitted to the Development Data Library at https://www.usaid.gov/data as required by USAID’s Open Data policy (see ADS 579). The data should be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of all datasets developed for this evaluation. The Contractor must not use any data or information obtained through this evaluation, in full or in part, for purposes other than to develop the Report for USAID. The following time frame for this evaluation is presented for illustrative purposes only, as it is recognized that the contractor will develop the actual time frame in consultation with USAID/Timor￾Leste as part of the implementation of the evaluation. It is estimated that the evaluation and the production of the requested reports will require approximately within 23 work weeks. TIME FRAME ACTIVITY Week 1 The Evaluation Team mobilizes. Weeks 2-4 ET reviews key project documents forwarded by USAID and beyond. Weeks 2-17 ET conducts an ongoing desk review. Week 3-4 The ET meets (virtually and could be multiple meetings as needed) with USAID staff, USAID Tourism for All staff and reviews relevant data and results. The ET finalizes its research instruments and approach. The draft Performance Evaluation Work Plan and Methodology are submitted. SI submits evaluation design for Internal Review Board (IRB) review. 45 Continue to monitor the Government of Timor-Leste’s State of Emergency measures. 46 https://www.usaid.gov/sites/default/files/documents/1870/USAIDEvaluationPolicy.pdf USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 48 Week 5-6 USAID reviews and returns comments on Work Plan and Methodology Week 7 ET submits final Performance Evaluation Work Plan and Methodology. Week 8-9 USAID approves Final Work Plan; SI finalizes evaluation team mobilization and logistics Week 10-14 The ET conducts in-brief and undertakes data collection in the field. Week 14 Out brief with the Mission Weeks 15-16 The ET analyzes data, develops findings and initial conclusions and recommendations. Week 17 The ET virtually presents a summary of initial findings to USAID/Timor-Leste, IP & other stakeholders to receive feedback, jointly formulate actionable recommendations, and continues drafting the evaluation report. Weeks 18-19 ET submits the first draft of the evaluation report. Week 19-20 USAID reviews and returns comments on the draft evaluation report. Weeks 21-22 The ET begins work on the final evaluation report. ET submits the final draft of the evaluation report. Weeks 23 USAID verifies comments have been addressed in the report; ET prepares a clean final copy and submits all data and the final Evaluation Report as described herein. IX. COMPOSITION OF THE EVALUATION TEAM The Contractor is encouraged to propose any kind of personnel structure deemed appropriate to conduct the work outlined in this scope of work (SOW). An illustrative team might include one Evaluation Team Leader, a subject matter expert with experience in the tourism sector, and an Evaluation Specialist. They must have expertise and experience on tourism issues in Southeast Asia and/or Oceania, with a particular emphasis on Timor-Leste is preferred. The Contractor should consider language barriers and make sure English-Tetum interpreters are on-site at all times. The Contractor must provide CVs for the suggested positions as well as the roles of the Evaluation Team Leader and other team members. USAID staff members may join the ET as appropriate. X. PERIOD OF PERFORMANCE It is estimated that the evaluation must not take more than 26 work weeks from the agreed commencement date. XI. INSPECTION AND ACCEPTANCE OF WORK PERFORMED TO CONDUCT THE AGREED UPON PROCEDURES AND THE REPORT 49 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV The work plan (including detailed steps) and the draft evaluation report will be subject to acceptance by USAID/Timor-Leste Program Office, with concurrence by the Economic Growth Office. XII. RELATIONSHIPS AND RESPONSIBILITIES The Contractor must work in coordination with USAID/Timor-Leste. The Contractor will liaise with the Evaluation Manager at the Mission for technical issues and day-to-day implementation of the evaluation. All contractual issues must be liaised with the CO through the TO’s COR at USAID/RDMA. XIII. LOGISTICS The Contractor must be responsible for providing all logistical support, including transportation for personnel and equipment required for the completion of the assignment, workspace, technological (IT) support, professional editing, etc. USAID/Timor-Leste will facilitate introductions to GoTL counterparts and implementing partners as agreed upon in the initial work plan. XIV. ANNEXES Annex 1 – Requirements for Submission of Reports Evaluation Report: Copies of the report must be prepared in English. The report must follow the guidelines for formatting outlined in the “How to Note: Preparing an Evaluation Report” (which also refers to the USAID Graphic Standards Manual and Sample Evaluation Report Template). The report should include the following sections: 1) Acronyms, 2) Executive Summary, 3) Background, 4) Evaluation Purpose and Evaluation Questions, 5) Evaluation Methodology and Limitations, 6) Findings, Conclusions, and Lessons Learned, 7) Recommendations, and 8) Annexes. Annexes should include: 1) Evaluation Statement of Work, 2) Data Collection Instruments, 3) Sources of Information (List of Persons Interviewed, Bibliography of Documents Reviewed, Databases, etc…), 4) Disclosure of Any Conflicts of Interest, 5) Statement of Differences/Dissenting Views (if applicable). The English version of the report must be professionally edited. To the extent possible the Contractor must align the evaluation and reports with the “USAID Evaluation Policy.” In particular, the Contactor should carefully review Section 5 entitled, “Evaluation Requirements.” One example of the many points USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 50 highlighted in this section is that, where available, the evaluation should use sex and age-disaggregated data and incorporate attention to gender relations in all relevant areas. XV. OTHER KEY DOCUMENTS 1) Annual Work Plans including Year-3 Priority Action Plans 2) Progress Reports 3) Activity Monitoring Evaluation Plans (Year 1 - Year 3) 4) National Tourism Policy 5) Other Relevant Documents 51 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV ANNEX II: INFORMATION SOURCES DOCUMENTS REVIEWED ● USAID Tourism for All Annual Report FY2020 ● USAID Tourism for All Annual Report FY2019 ● USAID Tourism for All Annual Report FY2018 ● USAID Tourism for All FY2020 Workplan ● USAID Tourism for All FY2019 Workplan ● USAID Tourism for All FY2018 Workplan ● USAID Tourism for All MEL Plan ● USAID Tourism for All Quarterly Progress Report Q2 FY21 ● USAID Tourism for All Quarterly Progress Report Q1 FY21 ● USAID Tourism for All Quarterly Progress Report Q3 FY19 ● USAID Tourism for All Quarterly Progress Report Q2 FY19 ● USAID Tourism for All Quarterly Progress Report Q1 FY19 ● USAID Tourism for All Quarterly Progress Report Q3 FY18 ● USAID Tourism for All Quarterly Progress Report Q2 FY18 ● USAID Tourism for All Public Private Partnership Investment Report ● USAID Tourism for All Project: A Position Paper on Barriers to Tourism Development in Timor-Leste ● USAID Hospitality and Tourism Training in Timor-Leste Report ● USAID Tourism Sector Opportunities and Constraints Report ● USAID Timor-Leste Tourism Laws Report ● USAID Indonesia Market Assessment of Potential for Faith-Based Tourism in Timor-Leste ● USAID Assessment of Mount Ramelau and Hatobuilico Tourism with Recommendations for Medium to Long term Planning ● Tourism Development Strategies Timor-Leste Presentation ● List of Stakeholders - Tourism - Project document ● Timor-Leste. National Tourism Policy - National Policy https://www.timorleste.tl/documents/timor-leste-national-tourism-policy/ ● World Bank. Timor-Leste Tourism website. https://data.worldbank.org/indicator/ST.INT.ARVL?locations=TL ● World Bank Timor-Leste Economic Report 2020 ● UN Women. (2020). For Equality and the Economy: Exploring the Gender Dimensions of Timor-Leste’s Tourism Sector. https://asiapacific.unwomen.org/- /media/field%20office%20eseasia/docs/publications/2021/03/tl-tourism-sector-s.pdf?la=en&vs=431 ● Jebson, R., Ikelberg, J. (2014). Timor-Leste Tourism Research and Development. Institute of Business Support (IADE)/ International Labor Organization (ILO). SITES REACHED ● Dili ● Atauro Island ● Mt. Ramelau area (Hatobuilico). RESPONDENT GROUPS ● Community Members (12) USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 52 ● Government Agencies and Organizations with Partnerships with the Project (4) ● Other Donor Parties (5) ● Tourism Enterprise Development Programs Grantees (5) ● Tourism for All Project Staff (5) ● Tourism Operators/Hotels/Other Local Enterprises (18) ● USAID Staff (2) 53 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV ANNEX III: UN WOMEN RECOMMENDATIONS FOR ENGENDERING TIMOR-LESTE TOURISM SECTOR KEY RECOMMENDATIONS FROM UN WOMEN STUDY: EXPLORING THE GENDER DIMENSIONS OF TIMOR￾LESTE’S TOURISM SECTOR (2021) Representation and Leadership / USAID gender domain: 47 Access to and control over assets and resources • Provide capacity development and resources to women’s tourism networks, NGOs and tourism cooperatives to actively work towards women’s social and economic empowerment in the sector; monitoring and addressing any potential of intimate partner violence by providing information on violence against women and girls and pathways for accessing support; • Facilitate and encourage women’s representation and leadership in trade unions; • Raise the visibility and profiles of women champions in industry; • Promote equitable sharing of unpaid care work in communities and companies engaged in tourism, and support initiatives that engage female entrepreneurs and their spouses to promote more equal household decision making and respectful relationships; • Create platforms for women in tourism to network & access peer-support, utilizing ICT to promote women owned businesses, connecting female entrepreneurs with opportunities. Education and Training / USAID Gender domain: Gender roles, responsibilities and time use • Create temporary special measures for women to access training and education opportunities, including scholarships, and remote training options; This should include expanding access of training opportunities to rural areas and covering digital literacy and foundational skills-building that could be accessed through different formats (online, peer-to-peer, etc.) • Develop programmes tailored for rural women entrepreneurs to gain exposure to tourism experiences and also develop the quality of their services and products; • Develop professional skills-building opportunities that recognize and facilitate women’s participation given their burden of unpaid care work and promote equitable care-giving roles among both women and men; • Incorporate gender into tourism and hospitality curriculums, ensuring a more sustainable tourism industry; • Provide training on gender equality for tourism educators and address gender-related barriers in tourism education institutions; • Provide targeted training for diverse groups of women (including those with disabilities) to ensure that they can use digital technologies and have access to resources to innovate through digital technologies in tourism. Institutional Gender Mainstreaming / USAID gender domain: patterns of power and decision-making • Mainstream gender in tourism policy, planning and operations and allocate sufficient budget into promoting gender equality and addressing barriers to women’s participation in tourism; • Ensure the Ministry of Tourism has a functional and staffed Gender Working Group financed by the State Budget and complementary funds; • Integrate gender analysis in the development and monitoring of legal and macroeconomic policies, including promotion of a conducive legal environment for women’s entrepreneurship and monitoring the Equal Pay Resolution and Sexual Harassment Guidelines within the tourism industry; • Support gender-responsive procurement measures to empower artisans and micro, small and medium￾enterprises that might not have information or access to procurement opportunities; • Encourage private sector companies to sign the WEPs to contribute to a more resilient and equitable tourism industry by systematically promoting equality and better business practices, including related to zero tolerance of sexual harassment. Gender Responsive Policies and Environment / USAID gender domain: Laws, policies regulations and institutional practices • Collect and utilize sex, age and disability-disaggregated data specific related to tourism leadership, employment, entrepreneurship, and economic empowerment to inform policies & programmes; • Support policies and regulatory frameworks that facilitate women’s entrepreneurship and access to markets; • Promote industry policies that address discrimination and stigma based on gender identity and sexual orientation, in line with Timor-Leste’s human rights commitments and obligations; 47 USAID considers five Gender Domains of analysis, which are mapped onto UN Women key areas. The fifth domain: cultural norms and beliefs, is cross-cutting. USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 54 • Ensure formal and informal tourism workers access and benefit from social protection measures; • Conduct gender analysis, consult civil society actors, and integrate a gender perspective into all phases of tourism policy development and programming. 55 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV ANNEX IV: DATA COLLECTION INSTRUMENTS INFORMED CONSENT STATEMENT- KII Hello. My name is [name] and I am contacting you on behalf of Social Impact. We are independent evaluators, hired by USAID to conduct an evaluation of the Tourism for All project, funded by USAID and implemented by Chemonics. We are not affiliated with and do not represent either USAID or Chemonics (the Tourism for All project). The information we collect will inform USAID and Chemonics about the project’s progress to date, including any specific successes or challenges faced. We have invited you to participate in an interview, lasting no more than [one hour], to discuss the Tourism for All project and tourism in general in Timor-Leste and/or in your community. You were selected given your involvement with the project. If you choose to participate you would be one of approximately [#] stakeholders participating in interviews as part of this evaluation. Timor-Leste and/or in your community. You were selected given your involvement with the project. If you choose to participate you would be one of approximately [#] stakeholders participating in interviews as part of this evaluation. Your participation in this study is completely voluntary, and you are under no obligation to participate. You may stop the interview at any time for any reason, or can decline to answer any of our questions, without penalty. There are no direct benefits or risks to participating, other than the time you could spend doing other things. The information you share with us will be aggregated and analyzed along with information shared by others and will be presented in a report in consolidated format. The report will eventually be made public on USAID’s website but no individual names and information that could identify you will be provided in the report or be shared outside of the evaluation team. With your permission, we would like to record this interview strictly for note-taking purposes and making sure that we have captured the interview notes accurately. Only the evaluation team members will have access to it. After our notetaking and analysis is complete, we will delete the recording. If you would not like to be recorded, we can proceed with the interview without the recorder. If you have any questions or concerns about your rights as a participant, you may contact USAID/Timor￾Leste’s Development Assistance Specialist, Milca Baptista, at mbaptista@usaid.gov or the Social Impact Institutional Review Board at irb@socialimpact.com or +1 703 465 1884. I can leave a copy of this form with you if you would like. Do you have any questions? [ ] Yes [ ] No Do you agree to participate in this study? [ ] Yes [ ] No Agreed to recording? [ ] Yes [ ] No USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 56 KEY INFORMANT INTERVIEW QUESTIONNAIRES Government Agencies and Organizations with Partnerships with the Project Performance Evaluation of USAID/Timor-Leste Tourism for All Project Data Collection Tool: Government Agencies and Organizations with Partnerships with the Project Date: Start Time: ________________________________ Location: Name(s) of Interviewer(s): ____________________ Name of note-taker: Interview Type: ☐KII ☐FGD Respondent Name: ______________ ______ Respondent Position/Profession/Role: ________________ Informant Sex: ☐M ☐F EQ Reference Interview Questions and Probes -- Please describe the ways that you and your organization have interacted with the Tourism for All Project. 57 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV 1 Please briefly describe any changes you feel are attributable to the Tourism For All Project in the following general areas. a. What changes have you observed in tourism policy and regulation? b. What changes have you observed in the Implementation of the National Tourism Policy? c. What changes have you observed in establishing tourism sector advocacy groups? d. What changes have you observed in destination marketing - either domestic and international? e. What changes have you observed in partnerships that promote private sector investment? f. What changes have you observed in sustainable management plans in protected environmental areas? ● Why do you think these changes have taken place? ● What role has the TFA Project played in promoting or causing these changes in each area? ● Why do you think this is the case? ● What barriers remain? ● Were the project’s activities well-targeted to known barriers? ● Ask for examples. As relevant, inquire whether the project is targeting the right partnerships, directed toward the right actors, in the right way. ● Inquire about the factors that helped or hindered in each area. ● How sustainable do you think these changes are likely to be? 1 From your perspective, what are some of the main achievements of the project to date? ● Why do you think the project was successful in these areas? Please provide specific examples. 1 From your perspective, what are some of the main challenges faced by the project to date? ● What do you think is the reason for these challenges? What were the consequences? Please provide specific examples. 1a How well do you think the project worked with each of these groups? 1) Timorese women 2) Timorese youth (ages 18 to 35) 3) Marginalized groups in Timor-Leste such as LGBTI, religious minorities, & persons with disabilities 4) Tourism associations Tourism for All established such as ASTRABEKA, ATKOMA, ATM-TL 5) Dili Municipal Tourism Authority 6) Hotel Owners Timor-Leste Association 7) Government counterparts: Ministry of Tourism, Secretary State of Arts & Culture & Ministry of Agriculture and Fisheries 8) Dili Institute of Technology ● Ask for examples to support the response. support the response. ● What facilitated or prevented success? ● What improvements might be made going forward? USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 58 1,4 How would you describe the results of your interaction or collaboration with the Tourism for All Project? ● Why do you think this is the case? ● If not successful, what were the challenges, and how might they be addressed going forward? 1,4 In your opinion, are Tourism for All activities well-aligned with the national tourism strategy? ● Why do you think this is the case? ● Ask for examples to support the response. ● For non-national agencies (e.g. municipalities), probe regarding alignment with the strategies and plans of that specific agency. 1,4 What has been the most relevant or helpful aspect of the project so far, in terms of supporting your work? ● Least relevant and helpful? Ask for examples to support the response. 1d How do you think the project has supported the objective of increasing private sector investment? Would you say these efforts have been successful? ● Why do you think this is the case? ● Explore respondent’s definition of PSE success. ● What barriers remain? ● Were the project’s activities well-targeted to known barriers? Ask for examples. ● As relevant, inquire whether the project is targeting the right partnerships, directed toward the right private sector actors, in the right way. Inquire about the factors that helped or hindered private sector investment. 3 To what extent has the project provided support to gender equality, women’s empowerment, and social inclusion more broadly into the tourism policy and strategy? ● Why do you think this is the case? ● How does your agency/organization consider gender equality, women’s empowerment, and social inclusion more broadly into the tourism policy and strategy? ● Ask for examples to support the response. 1,3 (For any involved in Tais United Nations Educational, Scientific and Cultural Organization (UNESCO) application): Who was involved in the process of applying to UNESCO for the intangible heritage of Tais? What support did the USAID project provide? ● Was that support effective? Ask for examples to support the response. ● How were communities, women, youth, or other groups involved in the process (e.g. were they consulted)? 59 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV 1,3 (For any involved in Tais UNESCO application): What do you anticipate will be the results of an approved UNESCO application? Who do you anticipate will benefit from that? ● Do you think it will result in the preservation of the Tais tradition? ● Why or why not? Ask for examples to support the response. ● Probe for benefits to women, youth, and other groups. 1,3 (For any involved in Tais UNESCO application): Were there any challenges in this application process, or working with the working group? ● Ask for examples to support the response. 4 Is there duplication of effort by different partners in this sector? Are there challenges in coordination or harmonization by different partners in this sector? ● Why do you think this is the case? ● Ask for examples to support the response. ● What are the consequences of any such occurrences? How might this be improved going forward? 4 Please explain whether you think the TFA Project successfully added value to the GoTL’s implementation of the tourism policy, relative to other donor partners? ● Why do you think this is the case? ● In what specific areas has USAID added value (if any)? 4 What is the role that TFA has played relative to other donors in stimulating tourism in TL? ● Ask for examples to support the response. 3 Are there additional stakeholders with whom USAID Tourism for All should engage in order to support effective implementation of the National Tourism Policy that could stimulate private sector investment in the tourism sector? 2 Do you think the activities of the project will be sustainable? ● Why do you think this is the case? ● What do you think will determine the sustainability of the projects? ● Probe on which specific activities they think might be sustainable or not. -- If you were in a position to recommend how the project could be improved going forward, what would you recommend and why? 5 How has project implementation been impacted by the pandemic? What modifications have you employed? Ask for specific examples. What risks do you anticipate for project success going forward related to the pandemic? -- Is there anything else you’d like to discuss regarding the project that we have not touched on? USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 60 Other notes that fall outside the questions above: General reflections on the interview: END TIME: ___________ 61 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV USAID Staff Performance Evaluation of USAID/Timor-Leste Tourism for All Project Data Collection Tool: USAID Staff Date: Start Time: ________________________________ Location: Name(s) of Interviewer(s): ____________________ Name of note-taker: Interview Type: ☐KII ☐FGD Respondent Name: ______________ ______ Respondent Position/Profession/Role: ________________ Informant Sex: ☐M ☐F EQ Reference Interview Questions and Probes -- Please describe your role with the Tourism for All Project. 1 To what extent is the project meeting its objectives? ● Why? Please provide examples. ● Based on response, probe about tourism policies and regulations, associations and working groups, private sector investment, and sustainable tourism development. 1 What are some of the project’s key achievements to date? ● What do you think enabled these achievements? ● What were the main drivers of the success? 1 What are some of the main challenges faced by the project to date? ● What do you think is the reason for these challenges? What were the consequences? ● How were those challenges ultimately addressed? ● What were the results? ● Do they still remain? ● Will they continue? USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 62 1d How do you think the project has supported the objective of increasing private sector investment? Would you say these efforts have been successful? ● Why do you think this is the case? ● Explore respondent’s definition of PSE success. ● What barriers remain? ● Were the project’s activities well-targeted to known barriers? Ask for examples. ● As relevant, inquire whether the project is targeting the right partnerships, directed toward the right private sector actors, in the right way. Inquire about the factors that helped or hindered private sector investment. 2 Do you think the project will be sustainable as anticipated? ● Why do you think this is the case? ● What aspects of the project, if any, are to be sustained, as agreed by key stakeholders? how? ● What factors do you believe will facilitate or prevent sustainability? ● What, if any, additional support will be required to ensure sustainability? 3 In your view, how is the project benefitting different stakeholder groups? ● Can you provide examples of this? ● Is significant stakeholder benefit codified in the project’s scope? ● Probe for each stakeholder group from Q12 in USAID questions. ● What factors facilitated or prevented effective targeting of benefits? ● What could be done to improve going forward? Have there been any unintended impacts to these stakeholder groups (especially women, youth, and marginalized groups)? 4 Has the TFA Project successfully added value to the GoTL’s implementation of the tourism policy, relative to other donor partners? ● Why do you think this is the case? ● In what specific areas has USAID added value (if any)? ● Ask for examples to support the response. -- Do you think that the assumptions underlying the project theory of change were valid? ● Why do you think this is the case? ● What is different than initially expected, and how has it affected the project? ● Has the change in government since the beginning of the project until now impacted the project theory of change? -- What are the main questions or blind spots you have regarding the project’s progress to date? What are the main information gaps you are hoping for this evaluation to fill? 5 How has project implementation been impacted by the pandemic? Ask for specific examples. What risks do you anticipate for project success going forward related to the pandemic? 63 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV Is there anything else you’d like to discuss regarding the project? How about regarding this evaluation? Other notes that fall outside the questions above: General reflections on the interview: END TIME: ___________ USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 64 Tourism for All Project Staff Performance Evaluation of USAID/Timor-Leste Tourism for All Project Data Collection Tool: Tourism for All Project Staff Date: Start Time: ________________________________ Location: Name(s) of Interviewer(s): ____________________ Name of note-taker: Interview Type: ☐KII ☐FGD Respondent Name: ______________ ______ Respondent Position/Profession/Role: ________________ Informant Sex: ☐M ☐F EQ Reference Interview Questions and Probes -- Please describe your role with the Tourism for All Project. 1 To what extent is the project meeting its objectives? ● Please provide examples. ● Based on response, probe about tourism policies and regulations, associations and working groups, private sector investment, and sustainable tourism development. 1 What are some of the project’s key achievements to date? ● What do you think enabled these achievements? ● What were the main drivers of the success? 1 What are some of the main challenges faced by the project to date? ● What do you think is the reason for these challenges? What were the consequences? ● How were those challenges ultimately addressed? ● What were the results? ● Do they still remain? ● Will they continue? 65 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV 1a How well do you think the project worked with each of these groups? 1) Timorese women 2) Timorese youth (ages 18 to 35) 3) Marginalized groups in Timor-Leste such as LGBTI, religious minorities, & persons with disabilities 4) Tourism associations Tourism for All established such as ASTRABEKA, ATKOMA, ATM-TL 5) Dili Municipal Tourism Authority 6) Hotel Owners Timor-Leste Association 7) Government counterparts: Ministry of Tourism, Secretary State of Arts & Culture & Ministry of Agriculture and Fisheries 8) Dili Institute of Technology ● Ask for examples to support the response. ● What facilitated or prevented success? ● What improvements might be made going forward? 1b To what extent has the project supported implementation of the tourism policy, relative to its objectives? ● Why do you think this is the case? ● Ask for examples to support the response. support the response. ● In what ways has the project most effectively supported implementation of the tourism policy, relative to its objectives? ● In what ways has the project least effectively supported implementation of the tourism policy, relative to its objectives? ● Are these efforts well-aligned with efforts by other government or non-government stakeholders? 1c How effective are the tourism sector associations and working groups established by the project? ● Why do you think this is the case? ● Ask for examples to support the response. ● What factors have led to success (or not) of the different groups? ● What explains any differences in effectiveness between groups? 1c Do you think the associations and working groups will be sustainable? Is planning for sustainability integrated into current efforts? ● Why do you think this is the case? ● Ask for examples to support the response. ● What might be done at this point to boost the likelihood of sustainability? 1d Regarding the partnerships established by the project to stimulate private sector investments: how effective have project efforts in this area been to date, relative to objectives? ● Why do you think this is the case? ● How do these partnerships specifically address constraints to private sector investment? ● Are these efforts well-aligned with efforts by other government or non-government stakeholders? USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 66 1d Do you think the partnerships established by the project have a reasonable likelihood of fulfilling their objectives (stimulating private sector investment)? ● Why do you think this is the case? ● Ask for examples to support the response. ● Probe on differences between partnerships established. What are any major persistent challenges; how might they be addressed? ● As relevant, inquire whether the project is targeting the right partnerships, directed toward the right private sector actors, in the right way. ● Inquire about the factors that helped or hindered private sector investment. 2 Overall, which project activities do you feel have the greatest likelihood of being scaled up or out (future sustainability)? ● Ask for examples to support the response. ● What are the factors you think will facilitate or prevent sustainability? ● Probe to ask about differences between project activities. 3 In your view, which groups or stakeholders have benefitted the most from the project, relative to its objectives? Which groups or stakeholders have benefited the least from the project, relative to its objectives? a) Timorese women b) Timorese youth (ages 18 to 35) c) Marginalized groups in Timor-Leste such as LGBTI, religious minorities, & persons with disabilities d) Tourism associations Tourism for All established such as ASTRABEKA, ATKOMA, ATM-TL e) Dili Municipal Tourism Authority f) Hotel Owners Timor-Leste Association g) Government counterparts: Ministry of Tourism, Secretary State of Arts & Culture & Ministry of Agriculture and Fisheries h) Dili Institute of Technology ● Why do you think this is the case? ● Ask for examples to support the response. ● Probe for each stakeholder group. ● You stated that ____ benefited the least from the project, relative to objectives. Why do you think this is the case? ● What factors facilitated or prevented effective targeting of benefits? ● What could be done to improve going forward? 3 To what extent does the project consider gender equality, women’s empowerment, and social inclusion more broadly into the design and implementation of its activities? ● Why do you think this is the case? ● Ask for examples to support the response. ● Probe for other categories of social inclusion (age, disability status, LGBTI￾identification, and race/ethnicity). 67 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV 3 To what extent do you think that gender equality has been achieved in regard to participation in project activities and overall results? ● Why do you think this is the case? ● Ask for examples to support the response. 4 To what extent has the project collaborated with other donor partners? ● Why do you think this is the case? ● Ask for examples to support the response. ● What are some of the successes (and/or challenges) in this regard? ● Has there been any duplication of effort? What consequences has that had? How is it being addressed? 4 Has the TFA Project successfully added value to the GoTL’s implementation of the tourism policy, relative to other donor partners? ● Why do you think this is the case? ● In what specific areas has USAID added value (if any)? ● Ask for examples to support the response. 3 Are there additional stakeholders with whom USAID Tourism for All should engage in order to support effective implementation of the National Tourism Policy that could stimulate private sector investment in the tourism sector? What has prevented them from participating in the project so far? -- Do you think that the assumptions underlying the project theory of change were valid? ● Why do you think this is the case? ● What is different than initially expected, and how has it affected the project? -- If you were in a position to recommend how the project could be improved going forward, what would you recommend and why? 5 How has project implementation been impacted by the pandemic? What modifications have you employed? Ask for specific examples. What risks do you anticipate for project success going forward related to the pandemic? -- Is there anything else you’d like to discuss regarding the project that we have not touched on? Other notes that fall outside the questions above: General reflections on the interview: USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 68 END TIME: ___________ 69 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV Other Donor Partners Performance Evaluation of USAID/Timor-Leste Tourism for All Project Data Collection Tool: Other Donor Partners Date: Start Time: ________________________________ Location: Name(s) of Interviewer(s): ____________________ Name of note-taker: Interview Type: ☐KII ☐FGD Respondent Name: ______________ ______ Respondent Position/Profession/Role: ________________ Informant Sex: ☐M ☐F EQ Reference Probes -- Please describe the ways that you and your organization have interacted with the Tourism for All Project. USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 70 1 Please briefly describe any changes you feel are attributable to the Tourism For All Project in the following general areas. a. What changes have you observed in tourism policy and regulation? b. What changes have you observed in the Implementation of the National Tourism Policy? c. What changes have you observed in establishing tourism sector advocacy groups? d. What changes have you observed in destination marketing - either domestic and international? e. What changes have you observed in partnerships that promote private sector investment? f. What changes have you observed in sustainable management plans in protected environmental areas? ● Why do you think these changes have taken place? ● What role has the TFA Project played in promoting or causing these changes in each area? ● Why do you think this is the case? ● What barriers remain? ● Were the project’s activities well-targeted to known barriers? ● Ask for examples. As relevant, inquire whether the project is targeting the right partnerships, directed toward the right actors, in the right way. ● Inquire about the factors that helped or hindered in each area. ● How sustainable do you think these changes are likely to be? 1 From your perspective, what are some of the main challenges faced by the project to date? ● What do you think is the reason for these challenges? ● What were the consequences? ● Do you think the project will be able to mitigate those going forward? 1a From your perspective, what are some of the main achievements of the project to date? ● What do you think are the main reasons for these successes? Please provide specific examples. 1,4 How would you describe the results of your interaction or collaboration with the Tourism for All Project? ● Why do you think this is the case? ● If not successful, what were the challenges, and how might they be addressed going forward? 4 Please describe any challenges in coordination or harmonization by different partners in this sector? ● Why do you think this is the case? ● Ask for examples to support the response. ● What are the consequences of any such occurrences? How might this be improved going forward? 71 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV 4 Please explain whether you think the TFA Project successfully added value to the GoTL’s implementation of the tourism policy, relative to other donor partners? ● Why do you think this is the case? ● In what specific areas has USAID added value (if any)? 4 What is the role that TFA has played relative to other donors in stimulating tourism in TL? ● Ask for examples to support the response. 3, 4 In your view, which groups or stakeholders have benefitted the most from the project, relative to its objectives? Which groups or stakeholders have benefited the least from the project, relative to its objectives? ● Why do you think this is the case? ● Ask for examples to support the response. ● Probe for each stakeholder group. a) Timorese women b) Timorese youth (ages 18 to 35) c) Marginalized groups in Timor-Leste such as LGBTI, religious minorities, & persons with disabilities d) Tourism associations Tourism for All established such as ASTRABEKA, ATKOMA, ATM-TL e) Dili Municipal Tourism Authority f) Hotel Owners Timor-Leste Association g) Government counterparts: Ministry of Tourism, Secretary State of Arts & Culture & Ministry of Agriculture and Fisheries h) Dili Institute of Technology ● How does Tourism for All compare in this aspect to other initiatives focused on tourism? ● What factors facilitated or prevented effective targeting of benefits? ● What could be done to improve going forward? 3, 4 To what extent does the project consider gender equality, women’s empowerment, and social inclusion more broadly into the design and implementation of its activities? ● Why do you think this is the case? ● Ask for examples to support the response. ● Probe for other categories of social inclusion (age, disability status, LGBTI￾identification, and race/ethnicity). ● How does Tourism for All compare in this aspect to other initiatives focused on tourism? -- If you were in a position to recommend how the project could be improved going forward, what would you recommend and why? USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 72 5 To what extent has project implementation been impacted by COVID-19? ● How would you evaluate the efficacy of the project in pivoting due to COVID-19? ● Ask for specific examples. ● What risks do you anticipate for project success going forward related to the pandemic? ● Are there additional measures the program could consider to better support the tourism industry during this period? -- What has changed in terms of quality and/or quantity of tourism activities in Timor￾Leste under the Tourism for All project? ● Why? To what extent can these changes be attributed to the Project? ● What are your observations about long-term sustainability of these changes? -- Is there anything else you’d like to discuss regarding the project that we have not touched on? Other notes that fall outside the questions above: General reflections on the interview: END TIME: ___________ 73 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV Village Chief(s) in Geographic Focus Areas Performance Evaluation of USAID/Timor-Leste Tourism for All Project Data Collection Tool: Village Chief(s) in Geographic Focus Areas Date: Start Time: ________________________________ Location: Name(s) of Interviewer(s): ____________________ Name of note-taker: Interview Type: ☐KII ☐FGD Respondent Name: ______________ ______ Respondent Position/Profession/Role: ________________ Informant Sex: ☐M ☐F EQ Reference Questions and Probes -- First, we’d like to begin by asking you to describe what tourism activities happen in this community? -- In general, how is your community impacted by the tourism activities in this area to date? Please provide some examples. ● Probe for positive and negative impacts (social/cultural, environmental and economic). ● Probe for specific ways that women, youth, and other marginalized groups have (or could) be impacted by tourism. -- In general, what is your vision regarding how your community might be involved in or impacted by tourism in the future? Do you have any specific hopes or concerns about the potential for increased tourism? ● Ask for examples to support the response. ● Probe for specific ways that women, youth, and other marginalized groups have (or could) be impacted by tourism. Probe for potential benefits or disadvantages. 1 Please describe the ways that you and your community have interacted with the USAID Tourism for All Project. 1 From your perspective, what are some of the main achievements of the project to date? USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 74 1 From your perspective, what are some of the main challenges faced by the project to date? ● What do you think is the reason for these challenges? What were the consequences? 1a Do you think the project has worked effectively with your community to implement tourism development activities? Why or why not? ● Ask for examples to support the response. ● Probe on which specific activities that have been implemented in their community where effective or not. What facilitated or prevented success? What improvements might be made going forward? 1,4 Would you characterize your working relationship with the project effective? Why or why not? How does this compare with your working relationship with other tourism stakeholders? ● If not successful, what were the challenges, and how might they be addressed going forward? Probe about the working relationship of the project with other community members. 1,3 (For Atauro island only) Are you aware of the designs for an ecolodge on Atauro island? What are your thoughts about the potential benefits or risks of this ecolodge? How do you think it will impact your community? 1,4 In your opinion, are the Tourism for All activities well-aligned with the priorities and needs of this community? Why or why not? ● Ask for examples to support the response. 1,4 What has been the most relevant or helpful aspect of the project so far, in terms of supporting your work? Least relevant and helpful? ● Ask for examples to support the response. 3 In this area, how do you think that all people, men and women, youths and other disadvantaged groups, should be considered in tourism development? To what extent has the project provided support toward these objectives? How? Do you feel this work has been effective? Why or why not? ● Ask for examples to support the response. 4 Is there duplication of effort by different organizations working in tourism development in your community? What are key challenges in coordination or harmonization of different organizations’ working in tourism development in your community? ● Ask for examples to support the response. ● What are the consequences of any such occurrences? ● How might this be improved going forward? 4 In your view, having interacted with several tourism stakeholders, are there particular benefits from the USAID Tourism for All project? What do you think is the value-add of this project relative to the activities of other donor partners? ● Ask for examples to support the response. 75 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV 2 When the Tourism for All project has come to an end, what efforts or activities that the project supported will you continue? Why or why not? What do you think will determine their ability to continue? -- If you were in a position to recommend how the project could be improved going forward, what would you recommend and why? 5 How has project implementation been impacted by COVID-19? What risks do you anticipate for project success going forward related to the pandemic? -- Is there anything else you’d like to discuss regarding the project? Other notes that fall outside the questions above: General reflections on the interview: END TIME: ___________ USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 76 TEDP Grantees Performance Evaluation of USAID/Timor-Leste Tourism for All Project Data Collection Tool: TEDP Grantees Date: Start Time: ________________________________ Location: Name(s) of Interviewer(s): ____________________ Name of note-taker: Interview Type: ☐KII ☐FGD Respondent Name: ______________ ______ Respondent Position/Profession/Role: ________________ Informant Sex: ☐M ☐F EQ Reference Questions and Probes -- Please describe the nature of your business. ● What products or experiences are provided? ● Who are the main customers? ● How long in business? Where in TL? ● In general, how do you consider the needs, opinions, and interests of local communities into your tourism development activities? -- What kinds of products do you source from the local community? ● Has this changed at all in the last two years, relative to before that? Why or why not? -- What are key barriers you face to growing your business? ● What do you think could or should be done to address them? 1 Please describe how you have interacted with the USAID Tourism for All Project in the last two years. ● Have you been involved in any associations and working groups established by the project? 1 Your organization was awarded a grant under the Tourism Enterprise Development Program (TEDP). Please describe the steps involved in the grant process. ● Was the process generally easy or difficult? ● What kinds of challenges were involved in the application process? ● Any places where the process was not transparent, confusing, etc.? ● How many applications have you submitted to the program? 77 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV 1 What have you used the grant funds for? ● How do you monitor the use of these funds and measure the effectiveness of the funds? ● What are your reporting requirements to TEDP? 1 How much of your operating budget does the grant represent? 1 Aside from the funding, have you received any other specific support along with the grant from the project (e.g. technical assistance)? How effective they are or how much they meet your needs? What are other essential support you need to improve your business? ● Please provide examples. 1 If you had not received this funding from USAID, would you still have been able to implement the improvements that you used the grant for? If so, how? If not, why? 1 In your view, has the project worked effectively with enterprises like yours? Why or why not? 1 Have you received other support from other projects, donors, or other organizations? What kind of support? How has the support helped improve your business? 3, 1d (For those who do receive support from others): How does the nature of support provided by the Tourism for All project differ from the support you receive or have received in the past? How would you describe the quality of the support provided by the project? What impact has it had on your business? Please provide specific examples. ● As relevant, inquire whether the project is targeting the right partnerships, directed toward the right private sector actors, in the right way. 3 How would you describe the extent to which the USAID Tourism for All project has adequately considered the needs, opinions, and interests of local communities into its tourism development activities? ● Has the project has positive and negative impacts on women, youth, marginalized groups, the culture, the environment etc.? 3 Were you encouraged or requested by the project to include women, youth, persons with disabilities or other marginalized groups in the development of your tourism product / business? 2 When the Tourism for All project has come to an end, will you sustain the efforts or activities that the project supported? If so, how? If not, why not? 5 How has project implementation been impacted by COVID-19? What risks do you anticipate for project success going forward related to the pandemic? 1a, 3 If you found yourself in a position to make recommendations about improving the project’s interaction with local tourism enterprises, what would you say? Other notes that fall outside the questions above: USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 78 General reflections on the interview: END TIME: ___________ 79 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV Tourism Operators/Hotels/Other Local Enterprises Performance Evaluation of USAID/Timor-Leste Tourism for All Project Data Collection Tool: Tourism Operators/Hotels/Other Local Enterprises Date: Start Time: ________________________________ Location: Name(s) of Interviewer(s): ____________________ Name of note-taker: Interview Type: ☐KII ☐FGD Respondent Name: ______________ ______ Respondent Position/Profession/Role: ________________ Informant Sex: ☐M ☐F EQ Reference Questions and Probes -- Please describe the nature of your business. ● What products or experiences are provided? ● Who are the main customers? ● How long in business? Where in TL? ● In general, how do you consider the needs, opinions, and interests of local communities into your tourism development activities? 1 Please describe how you have interacted with the USAID Tourism for All Project in the last two years. ● Have you been involved in any associations and working groups established by the project? ● Have you received any support from the USAID Tourism for All project (e.g. capacity￾building, technical assistance, training, funding, etc.)? ● Have you participated in any USAID Tourism for All events, conferences, campaigns, etc.? -- Are there barriers your face to growing your business? If so, what are these? ● What do you think could or should be done to address them? 1 In your view, has the project worked effectively with enterprises like yours? Why or why not? ● Probe what their expectations were? Were they met or not met? USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 80 1 Have you received support from other projects, donors, or other organizations? What kind of support? 1d (For those who do receive support from others): How does the nature of support provided by the Tourism for All project differ from the support you receive or have received in the past? How would you describe the quality of the support provided by the project? What impact has it had on your business? Please provide specific examples. ● Reflecting on these other instances of support, probe about whether the project is targeting the right partnerships, directed toward the right private sector actors, in the right way. Inquire about how Tourism for All may further improve its approach to partnerships. 3 How would you describe the extent to which the USAID Tourism for All project has adequately considered the needs, opinions, and interests of local communities into its tourism development activities? 3 Were you encouraged or requested by the project to include women, youth or marginalized in the development of your tourism product / business? 2 When the Tourism for All project has come to an end, will you sustain the efforts or activities that the project supported? If so, how? If not, why not? 1a, 3 If you found yourself in a position to make recommendations about improving the project’s interaction with local tourism enterprises, what would you say? 5 How has project implementation been impacted by COVID-19? What risks do you anticipate for project success going forward related to the pandemic? -- Is there anything else related to the project you would like to discuss? Other notes that fall outside the questions above: General reflections on the interview: END TIME: ___________ 81 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV INFORMED CONSENT STATEMENT- FGD & GROUP INTERVIEW Hello. My name is [name] and I am contacting you on behalf of Social Impact. We are independent evaluators, hired by USAID to conduct an evaluation of the Tourism for All project, funded by USAID and implemented by Chemonics. We are not affiliated with and do not represent either USAID or Chemonics (the Tourism for All project). The information we collect will inform USAID and Chemonics about the project’s progress to date, including any specific successes or challenges faced. We have invited you to participate in a focus group discussion, lasting no more than [one hour], to discuss the Tourism for All project. You were selected given your involvement with the project. If you choose to participate you would be one of approximately [#] stakeholders participating as part of this evaluation. Your participation in this study is completely voluntary, and you are under no obligation to participate. You may leave the discussion at any time for any reason, or can decline to answer any of our questions, without penalty. There are no direct benefits or risks to participating, other than the time you could spend doing other things. The information you share with us will be aggregated and analyzed along with information shared by others and will be presented in a report in summary format. No individual names will be provided within our report findings. The report will eventually be made public on USAID’s website. Your name and information that could identify you will not be shared outside of the evaluation team. We ask that all focus group participants respect each other’s privacy and agree not to share anything that is discussed with anyone outside of this group once this conversation ends. Remember that you are free to refuse to answer any questions. With your permission, we would like to record this discussion strictly for note-taking purposes and making sure that we have captured the discussion notes accurately. Only the evaluation team members will have access to it. After our notetaking and analysis is complete, we will delete the recording. If you would not like to be recorded, we can proceed with the interview without the recorder. If you have any questions or concerns about your rights as a participant, you may contact USAID/Timor￾Leste’s Development Assistance Specialist, Milca Baptista, at mbaptista@usaid.gov or the Social Impact Institutional Review Board at irb@socialimpact.com or +1 703 465 1884. I can leave a copy of this form with you if you would like. Do you have any questions? [ ] Yes [ ] No Do you agree to participate in this study? [ ] Yes [ ] No Agreed to recording? [ ] Yes [ ] No USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 82 FOCUS GROUP DISCUSSION QUESTIONNAIRE Community Members in Geographic Focus Areas Performance Evaluation of USAID/Timor-Leste Tourism for All Project Data Collection Tool: Community Members in Geographic Focus Areas Date: Start Time: ________________________________ Location: Name(s) of Interviewer(s): ____________________ Name of note-taker: Interview Type: ☐KII ☐FGD Respondent Name: ______________ ______ Respondent Position/Profession/Role: ________________ Informant Sex: ☐M ☐F For group interviews or FGDs: Name48 Sex Position/Profession/Role Other 1. 2. 3. 4. 5. 6. 7. 8. 9. 48 Ibid (see footnote above) 83 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV 10. EQ Reference Questions and Probes -- First, we’d like to begin by asking you to describe what tourism activities happen in this community? 1,3 How is your community impacted by the tourism activities in this area? Please provide some examples. ● Probe for potential positive and negative impacts (social / cultural, environmental and economic). ● Probe for specific ways that women, youth, and other marginalized groups have (or could) be impacted by tourism, especially in terms of greater agency or capacity in the tourism sector. Ask for examples. 1,3 How do you think this might change in the future? What do you think will be the impact of those changes? 1,3 In general, what is your ideal vision regarding how your community might be involved in or impacted by tourism in the future? ● Ask for examples to support the response. ● Probe for specific ways that women, youth, and other marginalized groups have (or could) be impacted by tourism. 1,3 Do you know about the USAID Tourism for All project? What do you know about it? What are your experiences with the project so far? How is it affecting your community? ● Probe about the extent to which attendees have been engaged from a private sector standpoint. 1,3 Are you aware of the Turizmu Ba Ema Hotu campaign? What is this campaign about? What are your impressions of the campaign? How do you think it will impact your community? ● Probe for specific ways that women, youth, and other marginalized groups have (or could) be impacted. 1,3 (For Atauro island only) Are you aware of the designs for an ecolodge on Atauro island? What are your thoughts about the potential benefits or risks of this ecolodge? How do you think it will impact your community? 2 When the Tourism for All project has come to an end, will you sustain the efforts or activities that the project supported? Why or why not? What do you think will determine the sustainability of the projects? 5 How has project implementation been impacted by COVID-19? What risks do you anticipate for project success going forward related to the pandemic? -- Is there anything else you’d like to discuss regarding the project? USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 84 Other notes that fall outside the questions above: General reflections on the interview: END TIME: ___________ 85 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV ANNEX V: EVALUATION TEAM MEMBERS A team of highly skilled sectoral and evaluation specialists carried out this evaluation – thoroughly reviewing and analyzing desk review documentation, designing data collection tools, collecting data, analyzing and present findings, and preparing the evaluation report and other deliverables. Members of the ET are: ROLE NAME RESPONSIBILITIES SUMMARY OF QUALIFICATIONS Team Leader & Tourism Specialist Dr. Sara Currie (Australian) Lead the design and conduct of the evaluation; including development of data collection instruments, virtual data collection, analysis, report￾writing, and dissemination. Task ET members with assignments for contributions to fieldwork and deliverables. Serve as main point of contact (POC) for USAID/TL during fieldwork. Within each task area, will add a lens of regional tourism, economic growth, and sustainability expertise, especially to data collection tools and analysis and report-writing. Dr. Sara Currie is a tourism expert with more than a decade of experience in Timor￾Leste’s tourism industry. She has served in the Office of the President of Timor-Leste as a Tourism Advisor to the Minister of Tourism in Timor-Leste, and as a Tourism Consultant for The Asia Foundation. Evaluation Specialist Zofimo Corbafo (Timorese) Contribute to evaluation design, desk review, data collection instruments, analysis, report-writing, and dissemination, as directed by TL. Lead interviews and liaise with Timorese stakeholders during fieldwork as requested by the TL. Mr. Corbafo has extensive experience designing and conducting evaluations in Timor-Leste, including quantitative and qualitative methods, and working with USAID/Timor-Leste as Partnership and Marketing Advisor for the Avansa Agrikultura Project. He has experience working with a wide variety of government officials and donor agencies to provide training and capacity-building for government institutions USAID/Timor-Leste as Partnership and Marketing Advisor. He has experience working with a wide variety of government officials and donor agencies to provide training and capacity-building for government institutions. USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 86 Evaluation Specialist Eugenia Urania da Costa (Zenny) Correia (Timorese) Contribute to evaluation design, desk review, data collection instruments, analysis, report-writing, and dissemination, as directed by TL. Lead interviews and liaise with Timorese stakeholders during fieldwork as requested by the TL. Ms. Zenny Correia is a local researcher with 11 years of extensive experience in qualitative research and monitoring and evaluation projects. She has worked on evaluations in Timor-Leste as a specialist, researcher, and team leader. She holds a bachelor’s degree in Teacher Training and Education Science of Mathematics and a Master’s in Peace in Conflict Studies from the National University of Timor-Leste. Logistician / Notetaker Nazario Dos Santos (Timorese) Maintain and distribute an up-to-date fieldwork schedule on a daily basis; conduct outreach as requested to local stakeholders to set up and confirm appointments before and during fieldwork. Attend interviews during fieldwork to support with note taking as requested. Mr. Dos Santos has qualitative research experience on evaluations and will assist the team when needed with qualitative data collection and notetaking. He holds a bachelor’s degree in Political Science and International Studies from Luther College. 87 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV ANNEX VI: DISCLOSURES OF CONFLICT OF INTEREST USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 88 89 | PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT USAID.GOV USAID.GOV PERFORMANCE EVALUATION OF USAID/TIMOR-LESTE TOURISM FOR ALL PROJECT | 90