MID-TERM PERFORMANCE EVALUATION OF THE USAID/Zrda ACTIVITY IN GEORGIA November 2019 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by ME&A, Inc. MID-TERM PERFORMANCE EVALUATION OF THE USAID/Zrda ACTIVITY IN GEORGIA This publication was produced at the request of the United States Agency for International Development Developed under Request for Task Order No. SOL-72011419R00010 It was prepared independently by: Colin Maclean, Team Leader Nelly Dolidze, Evaluation Specialist Zviad Archuadze, Local Technical Expert Contractor: ME&A, Inc. 4350 East-West Highway, Suite 210 Bethesda, MD 20814 Tel: (301) 652-4334 www.meandahq.com DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS EXECUTIVE SUMMARY.................................................................................................................................. i 1.0 EVALUATION BACKGROUND, PURPOSE AND QUESTIONS........................................... 1 1.1 Evaluation Background................................................................................................................. 1 1.2 Evaluation Purpose........................................................................................................................ 1 1.3 Evaluation Questions.................................................................................................................... 1 2.0 ACTIVITY BACKGROUND............................................................................................................. 2 3.0 EVALUATION METHODS AND LIMITATIONS ........................................................................ 3 3.1 Evaluation Methodology .............................................................................................................. 3 3.2 Evaluation Limitations................................................................................................................... 4 4.0 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS ................................................. 5 4.1 EQ 1: What approaches have been most or least successful in creating jobs and increasing incomes in target communities and why?............................................................ 5 4.2 EQ 2: Do interventions targeted at and/or implemented in partnership with larger companies result in creation of more jobs than interventions involving smaller companies and why?.................................................................................................................... 9 4.3 EQ 3: What are the key lessons learned from implementing PPPs under Zrda?....... 13 4.4 EQ 4: Have Zrda’s sub-awards and TA activities struck optimal balance between supporting minority and non-minority communities? Does Zrda sufficiently integrate and benefit minority communities? ....................................................................................... 18 4.5 EQ 5: Should Zrda provide continued support to DMOs as a means to support the sustainable development of Georgia’s tourism sector and why? What type of support, if any, considering specific organizational needs of each DMO?..................................... 22 4.6 EQ 6: Which of the following activities do Zrda beneficiaries and stakeholders identify as profit centers or keys to future business growth? Activities include: certification programs in hospitality sector; discrete training interventions in tourism; digital marketing—booking.com, TripAdvisor, and development of websites? ...................... 27 ANNEXES ....................................................................................................................................................... 30 Annex 1: Statement of Work........................................................................................................ 31 Annex 2: List of Meetings and Interviews................................................................................... 42 Annex 3: List of Documents Consulted ..................................................................................... 48 Annex 4: Zrda Evaluation Matrix.................................................................................................. 51 Annex 5: FGD Beneficiaries – Participant Responses.............................................................. 58 Annex 6: Zrda Activity M&E Plan Summary .............................................................................. 61 LIST OF FIGURES AND TABLES Figure 1: Distribution of Participants by ADC Training Sessions....................................................... 14 Table 1: FGDs by Location and Business Activity .................................................................................. 20 Table 2: Founders of DMOs in the Samtskhe-Javakheti and Samegrelo-Zemo Svaneti Regions . 23 ACRONYMS Acronym Description ABL Administrative Boundary Line ACED Anaklia Community Economic Development Program ADC Anaklia Development Consortium AER Association of Ecotourism in Romania AIP Annual Implementation Plan AMEP Activity Monitoring and Evaluation Plan ANT Romanian National Authority for Tourism APMA Agriculture Project Management Agency ARDA Agriculture and Rural Development Agency BP British Petroleum BSO Business Support Organization CBO Community-Based Organization CDCS Country Development Cooperation Strategy Chemonics Chemonics International Inc. CSO Civil Society Organization CSR Corporate Social Responsibility CTC Center for Training and Consultancy DCFTA Deep and Comprehensive Free Trade Agreement DMO Destination Management Organization EQ Evaluation Question ET Evaluation Team EU European Union FGD Focus Group Discussion FSC Farmer Service Center FTE Full-Time Equivalent FY Fiscal Year GEL Georgian Lari GFA Georgian Farmers’ Association GFDC Georgian Farmers Distribution Company GHCA Georgia Heritage Crafts Association GITA Georgia Innovation Technology Agency GNTA Georgia National Tourism Agency GoG Government of Georgia HR Human Resources ICT Information and Communications Technology IDP Internally Displaced Person Acronym Description IFAD International Fund for Agricultural Development IP Implementing Partner KI Key Informant KII Key Informant Interview LG Local Government LOP Life-of-Program M&E Monitoring and Evaluation MFO Microfinance Organization MoU Memorandum of Understanding MSME Micro, Small, and Medium Enterprise NAPR National Agency of Public Registry NGO Non-Governmental Organization PPP Public-Private Partnership Q Quarter RDA Regional Development Association SME Small and Medium Enterprise SOW Statement of Work TA Technical Assistance TIC Tourism Information Center ToT Training of Trainers U.S. United States UNDP United Nations Development Programme UNWTO United Nations World Tourism Organization USAID United States Agency for International Development USG United States Government VE Vocational Education VET Vocational and Education Training Center i EXECUTIVE SUMMARY EVALUATION PURPOSE The purpose of the mid-term performance evaluation of the United Stated Agency for International Development Georgia (USAID/Georgia) Zrda Activity was “to determine the relevance of Zrda’s approaches to achieving intended life-of-program (LOP) results; review progress toward achieving LOP targets and identify accomplishments, delays, challenges, and their impact on achieving LOP targets; and provide recommendations on corrective actions and new directions over the remaining LOP.” USAID/Georgia will use the evaluation’s results to improve its rural and tourism sector interventions and public-private partnerships (PPPs), and to support its Country Development Cooperation Strategy (CDCS). The results will be shared with the Government of Georgia (GoG), other local stakeholders, and the United States Government (USG) community in Georgia. ACTIVITY BACKGROUND Zrda was designed to improve the livelihoods of rural farming communities in the Administrative Boundary Lines (ABL) zones, including non-agriculture activities related to developing the potential of the tourism sector. Zrda is a $24,650,000 project implemented by Chemonics International, Inc., that began in March 2016 and is due to end in December 2021. EVALUATION METHODOLOGY The evaluation team (ET) collected data using the following methods: 1. Key Informant Interviews (KIIs): The ET conducted 51 KIIs with 61 individuals in Zrda’s six intervention regions. 1 2. Focus Group Discussions (FGDs): The ET conducted 13 FGDs with 94 participants, including 10 with 76 Zrda beneficiaries in Kvemo Kartli, Samtskhe-Javakheti, Shida Kartli, and Samegrelo￾Zemo Svaneti, and three with 18 municipality officials in Marneuli, Ninotsminda, and Zugdidi. 3. Non-Beneficiary Survey: The ET conducted a survey with 31 non-beneficiaries in Kvemo Kartli. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS EQ 1: What approaches have been most or least successful in creating jobs and increasing incomes in target communities and why? Findings 1. Zrda’s performance monitoring results indicate that it has created 905 jobs against an LOP target of 2,400, with 400 of those in tourism, 317 in agriculture, and the rest in service￾oriented businesses. Beneficiaries in agriculture and tourism uniformly appreciated Zrda’s assistance in providing equipment, training, and marketing/promotional support. 2. Zrda’s Greenhouse Subsidy Program has been particularly successful, with the majority of beneficiaries saying that it helped them create jobs, albeit mostly seasonal and short-term (i.e., under three months), and that it increased their incomes by an average of 25 percent. 3. Contributing to Zrda’s success is Zrda’s delivery of relevant and appropriate training, especially in e-booking and digital marketing, and the availability of grants to improve operations and facilities. Several key informants (KIs) stated that small business owners also 1 Kakheti, Kvemo Kartli, Samtskhe-Javakheti, Samegrelo-Zemo Svaneti, Mtskheta-Mtianeti, and Shida Kartli. ii need training in general business skills, such as preparing business plans, farm management, and even in basic administrative functions, such as filling in forms and completing paperwork. 4. Some KIIs noted that tourism investment in areas like Ganmukhuri should be revised since they are not attractive to tourists, face competition from nearby areas, and are dependent on external factors such as the government not supporting popular events like the Anaklia festival. Conclusions 1. Zrda’s sector-driven approach offering technical assistance (TA) and grants has contributed to an increase in seasonal and short-term jobs and household income among agriculture and tourism beneficiaries. 2. Many of the sectors targeted by Zrda and other private sector development programs remain at a relatively low knowledge and skill level and require ongoing capacity development of sector participants. 3. According to some participants, some areas in the ABL zone are more suitable for agribusiness than for tourism. Recommendations 1. Zrda should concentrate on supporting interventions that help ensure longer term sustainability of small rural enterprises, e.g., through developing training sessions/programs that encourage self-improvement for business owners, whether in agriculture or tourism. 2. Zrda should continue focusing more on agribusiness in some of the targeted areas given their limited tourism potential. EQ 2: Do interventions targeted at and/or implemented in partnership with larger companies result in creation of more jobs than interventions involving smaller companies and why? Findings 1. Activity partners support a variety of initiatives in Zrda’s intervention regions, some of which do not necessarily have job creation as an objective. Beneficiaries tend to be micro￾enterprises or small sole operators with few opportunities to long-term jobs, given that agriculture and tourism are seasonal in nature and employ temporary or seasonal labor. Zrda’s approach in supporting investment in production processes and facilities, however, is a positive development Conclusions 1. The ET found no evidence that interventions implemented in partnership with either larger or smaller companies had a greater impact on job creation than the other, with the exception of Zrda’s investment initiatives, especially in farming communities, which appeared to offer greater job creation opportunities. Recommendations 1. Zrda should leverage its available resources and those of its partners to co-fund joint ventures to help businesses expand (e.g., greenhouses, cold storage units, etc.). If, at the same time, Zrda can link this to offering marketing support to target sectors and businesses, then opportunities for growing targeted beneficiary companies will have a greater likelihood of success. 2. USAID should consider activities similar to Zrda without job creation as their ultimate objective, thereby allowing them to concentrate on interventions in rural communities with less reliance on seasonality and more on developing businesses with greater long-term sustainability prospects as a result of being able to upgrade, expand, or diversify activities. iii EQ 3-3a: What are the key lessons learned from implementing PPPs under Zrda? In addition to the Zrda activity, what could the government, communities, or others do to help businesses be more prosperous in five years? Findings 1. Zrda’s PPP with the Anaklia Development Consortium (ADC) has had limited success to date for reasons outside of Zdra’s control. Its collaboration with British Petroleum (BP) has been more successful, especially with initiatives in Kvemo Kartli and Samtskhe-Javakheti that provide training and grant support to small farmers in minority communities. 2. There exists a need for increased investment in tourism infrastructure, particularly signage street lighting, tourist restrooms/other facilities, and information about tourist attractions. 3. Guesthouse operators require additional capacity development in business and facilities management, particularly in light of Zrda’s interventions to link the Georgian tourism sector to online marketing and information sources. 4. The existing regulatory structure is inappropriate for small guesthouses, which are unable to comply with existing regulations predominately targeted to larger hotels or guesthouses. 5. According to all interviews, Destination Management Organizations (DMOs) currently lack capacity to operate effectively. However, this is expected to be resolved in time with Zrda’s input. 6. There exists a desire among small businesses for statistics and other market information in the form of, for example, marketing information and networking opportunities. 7. Beneficiary selection criteria at times disadvantage smaller businesses that possess the capacity to grow but fail to meet all of the selection criteria. 8. There exists a large, unmet need for increased access to finance for micro and small businesses. Particularly needed are less stringent loan requirements. 9. Private sector development initiatives do not always factor in the comparative advantages of locations or businesses in determining which to support. Conclusions 1. Zrda’s PPP strategy has merit in meeting the aims of all concerned partners. Despite some mixed levels of success (due to factors out of Zdra’s control), the approach of working with these organizations is sound and can act as a springboard for the new PPP initiatives. 2. KIs identified several lessons learned about possible activities to increase business in five years, which are elaborated in the findings and recommendation for EQ 3a. Recommendations 1. Zrda should continue its cooperation with existing partners while continuing to explore and establish new PPP arrangements. 2. Zrda should twin the digital marketing activities with the other tourism development interventions. If the former outpaces the latter, it will possibly blunt positive effects on the development of the tourism sector. 3. Zrda should continue to work with the development partners to generate and disseminate statistics and other market information to small businesses and guesthouse operators. 4. Zrda should take a more targeted approach to supporting sectors and businesses in different locations to leverage their comparative advantages. iv EQ 4: Have Zrda’s sub-awards and TA activities struck optimal balance between supporting minority and non-minority communities? Does Zrda sufficiently integrate and benefit minority communities? Findings 1. Zrda actively integrates minority communities into its diverse interventions in each of the sectors where it works and tailors a package of services to them consistent with their needs. 2. The issues raised by Zrda’s beneficiaries in Kvemo-Kartli and Samtskhe-Javakheti were similar as those experienced by farming communities in non-minority regions. Activity interventions were implemented on a demand-driven basis, irrespective of ethnicity, thus serving all beneficiaries equally and without favor. Conclusions 1. Zrda has struck an optimal balance in supporting minority and non-minority communities and has sufficiently integrated the former into its diverse interventions. 2. The ET observed no disparity in Zrda’s support activities between minority or non-minority communities, nor did KIs identify any types or sources of disparity. 3. Irrespective of minority status, beneficiaries highly valued Zrda’s support, whether TA or grants, without which they would not have been able to improve their businesses. Recommendations 1. Zrda should consider expanding its activities in Kvemo-Kartli and Samtskhe-Javakheti where evidence suggests that many villages appear to be underserved and below Zrda’s radar. Zrda has already made efforts to involve some of those communities and will continue to peruse this activity. In villages where some progress has been made in improving production and household incomes, options may be considered for possible public-private collaboration to establish processing units. This could add value to farmers’ potential income and also reduce food wastage. EQ 5: Should Zrda provide continued support to DMOs as a means to promote the sustainable development of Georgia’s tourism sector and why? What type of support, if any, is needed to make DMOs operational sustainable by the end of the Zrda program, considering specific organizational needs of each DMO? Findings 1. Zrda facilitated the creation of regional DMOs in the Samtskhe-Javakheti and Samegrelo￾Zemo Svaneti regions and will continue to support them during the remainder of the program. Although these DMOs are in their early stages and lack human and financial resources, KIs praised their interventions and argued that Zrda should continue supporting them in order to ensure their sustainability. Conclusions 1. Zrda’s support to DMOs is critical for their sustainability. 2. DMOs do not have sufficient capacity (institutional, human resources, and financial) to produce the technical products outlined by the United Nations World Tourism Organization. Recommendations 1. Zrda should support wider dissemination of information about DMOs to tourism operators and continue support to DMOs to develop their strategic positioning in the tourism sector. 2. Zrda should incorporate international guidelines for strengthening DMOs while supporting their strategy development and implementation of regional and municipal development plans. v 3. Zrda should support schemes to develop DMOs’ capacity to establish regional economic development networks, develop the tourism value chain, and develop tourism products, assets, and destinations. 4. Zrda should help DMOs develop agri-tourism (e.g., wine tourism) by selecting potential touristic sites, developing and marking tourism routes, helping DMOs to disseminate information about tours and sites, and helping finance and organize festivals and events. EQ 6: Which of the following activities do Zrda beneficiaries and stakeholders identify as profit centers or keys to future business growth? These activities include certification programs in the hospitality sector, discrete training interventions in the tourism sector, and digital marketing activities through booking.com, TripAdvisor, and development websites. Findings 1. Training related to tourism, with emphasis on building the capacity of business owners (hotels, restaurants, cafes, tour guides, etc.) to attract customers, was uniformly acknowledged by KIs interviewed as useful for promoting future business growth and increasing their profit margins. 2. Beneficiaries cited training sessions on digital marketing and acquiring certificates in food safety as prerequisites for earning a positive reputation for offering quality services. Conclusions 1. Activity beneficiaries highly valued Zrda’s various interventions to improve and upgrade their capacity in operating tourism-related businesses, which have in turn led to increased business and profit margins. 2. Quality issues have been addressed and self-improvement initiatives have been motivating factors for small business owners to improve performance and attract more customers. 3. Zrda’s collaboration with private service providers to deliver trainings and TA on service quality services has been a successful feature of the activity’s outreach activities. Recommendations 1. Zrda should continue with its capacity-building initiatives with private sector tourism operators to enhance their capacity to improve customer service quality. 2. Zrda should build on previous interventions to build capacity within the tourism sector to market its attractions and services to a wider consumer audience. This recommendation includes the caveat that marketing should be integrated with interventions to improve service quality and provide improved/increased information and amenities for tourists so that marketing efforts do not outpace the capacity of the tourism sector to provide quality tourism experiences. 1 1.0 EVALUATION BACKGROUND, PURPOSE AND QUESTIONS 1.1 EVALUATION BACKGROUND The decision of the United States Agency for International Development Georgia (USAID/Georgia) to support and fund the Zrda Activity, implemented by Chemonics International, Inc. (Chemonics), was based on several important factors, including the Government of Georgia’s (GoG) ongoing commitment to rural development, particularly in recent conflict border regions. The primary aim of Zdra is to improve households’ ability to generate income; develop the capacity of micro, small, and medium enterprises (MSMEs); and create an inclusive economic environment that provides opportunities for all, including ethnic minorities. The mid-term evaluation of Zdra was conducted during the period August – November 2019, by an evaluation team (ET) assembled by ME&A, located in Bethesda, MD. The team included a Team Leader, an Evaluation Expert, and a Local Technical Expert. 1.2 EVALUATION PURPOSE According to the evaluation Statement of Work (SOW), the purpose of the evaluation was “to determine the relevance of ZRDA’s approaches to achieving intended life-of-program (LOP) results; review progress toward achieving LOP targets and identify accomplishments, delays, challenges, and their impact on achieving LOP targets; and provide recommendations on corrective actions and new directions over the remaining LOP.” The evaluation sought to answer the six evaluation questions (EQs) listed below (Section 1.3). The evaluation results will be used by USAID/Georgia to improve ongoing interventions in rural development, tourism sector development, and public-private partnerships (PPPs), and for supporting the Mission’s Country Development Cooperation Strategy (CDCS). The evaluation’s primary audience is USAID, and in particular its Economic Growth Office and the implementing partner (IP), Chemonics. The results of the evaluation will also be shared with the GoG, other local stakeholders, the broader United States Government (USG) community in Georgia, and other development/donor organizations working in the same space. 1.3 EVALUATION QUESTIONS The SOW provided a set of EQs to be answered by the ET, which were subsequently elaborated upon in the ET’s Work Plan and Evaluation Design. They are as follows: 1. What approaches have been most or least successful in creating jobs and increasing incomes in target communities and why? 2. Do interventions targeted at and/or implemented in partnership with larger companies result in creation of more jobs than interventions involving smaller companies and why? 3. What are the key lessons learned from implementing PPPs under Zrda? a. In addition to the Zrda activity, what could the government, communities, or others do to help businesses be more prosperous in five years? 2 4. Have Zrda’s sub-awards and technical assistance (TA) activities struck optimal balance between supporting minority and non-minority communities? Does Zrda sufficiently integrate and benefit minority communities? 5. Should Zrda provide continued support to Destination Management Organizations (DMOs) as a means to promote the sustainable development of Georgia’s tourism sector and why? What type of support, if any, is needed to make DMOs’ operations sustainable by the end of the Zrda program, considering specific organizational needs of each DMO? 6. Which of the following activities do Zrda beneficiaries and stakeholders identify as profit centers or keys to future business growth? These activities include certification programs in the hospitality sector, discrete training interventions in the tourism sector, and digital marketing activities through booking.com, TripAdvisor, and development websites. EQ 3a was added by USAID/Georgia during the fieldwork, as a separate EQ in addition to those specified in the evaluation SOW. However, as this question overlaps significantly with EQ 3, it is addressed in this report as a sub-question to EQ 3. A detailed Zrda Evaluation Matrix highlighting the data collection and analysis methodology employed to answer the above EQs is provided in Annex 4 of the report. 2.0 ACTIVITY BACKGROUND Despite armed conflicts, (specifically the Russian occupation of the separatist regions of Abkhazia and South Ossetia), energy and transport blockages, and the loss of traditional markets, Georgia has taken significant steps in recent years to embracing free market economics, improving its business enabling environment, increasing quality and safety standards, enabling trade to new markets, and enhancing transparency. In support of this progress, Georgia signed an Association Agreement with the European Union (EU) on June 27, 2014, which came into effect on July 1, 2016, and introduced a Deep and Comprehensive Free Trade Agreement (DCFTA) aimed at increasing market access between the EU and Georgia based on better-matched regulations. In the World Bank’s Doing Business Report 2019, Georgia holds the sixth-place ranking regarding its Ease of Doing Business indicator. As part of the Euro-Atlantic Alliance integration process, Georgia is slowly strengthening environmental laws and regulations aimed at supporting businesses in agriculture, manufacturing, and tourism. However, despite these developments, Georgia’s rural households, alongside MSMEs, face continued challenges to improving income generation, growth, and resilience. Against this background, in 2012, the USG allocated development resources to USAID to achieve the goal of strengthening and sustaining Georgia’s democratic, free-market, and Western￾oriented transformation. USAID’s specific objectives and goals are highlighted in its CDCS 2013- 2020. In pursuit of these goals, the Zrda Activity was created to assist economic growth, particularly in the Administrative Boundary Lines (ABLs) areas, by building inter-ethnic networks and increasing communities’ incomes and skills. The focus was to develop opportunities for those communities in both the agricultural and non-agricultural sectors in order to diversify income streams and capitalize on Georgia’s high tourism potential. Zrda began in March 2016 and is due to end (following a recent extension)2 in December 2021. 2 Cooperative Agreement No. AID-114-A-16-00004 – Modification No. 6-27 August 2019. 3 3.0 EVALUATION METHODS AND LIMITATIONS 3.1 EVALUATION METHODOLOGY To conduct the Zrda evaluation, the ET employed a mixed-methods approach, which allowed for triangulation of findings. Data and information needed to respond meaningfully to the EQs were gathered from several qualitative and quantitative sources. Qualitative sources included review of activity documents and fieldwork consisting of key informant interviews (KIIs) and focus group discussions (FGDs) with project beneficiaries and stakeholders. Quantitative sources included review of Zrda monitoring and evaluation (M&E) reports and analysis of official publications, including data and statistics relevant to the ET’s investigations from government ministries and departments, World Bank, EU, International Fund for Agricultural Development (IFAD), and United Nations Development Programme (UNDP), among others. In addition, a short counterfactual survey of non-beneficiaries was conducted in the Shida Kartli region in order to compare their attitudes and perceptions with those who were direct beneficiaries of Zrda’s support in that region. 3.1.1 Qualitative Data Collection and Analysis The qualitative component of the performance evaluation drew on three data collection methods: 1. Structured desk review of Zrda’s available work plans, annual and quarterly reports, annual implementation plans, household surveys, GoG official publications, and regional development strategies, as well as other relevant secondary information, including publications from Zrda’s partners and associations. 2. KIIs conducted with Zrda staff, direct project beneficiaries, project partners, and other project stakeholders, including national and municipal government officials, business associations, and women and youth organizations, among others. 3. FGDs conducted in each of Zrda’s six regions of activity—Kakheti, Kvemo-Kartli, Samtskhe-Javakheti, Samegrelo-Zemo Svaneti, Mtskheta-Mtianeti, and Shida Kartli. 3.1.1.1 Structured Desk Review The ET was provided with project documents3 by USAID/Georgia prior to mobilization, which it subsequently reviewed and retained for reference throughout the in-country Mission. Additional published material and documents—GoG reports and development strategies and documentation on MSMEs in Georgia, particularly related to agriculture and tourism—were accessed by team members via online searches on project-related issues. These documents served to provide the context for Zrda and inform the ET on past, present, and planned activities by interested parties, local and international, on rural economic development in Georgia. 3.1.1.2 Key Informant Interviews The ET conducted a total of 51 KIIs in Zrda’s six regions of activity. They were attended by 61 individuals with a direct or indirect interest or connection with the activity, and included direct 3 See Annex 3 for a complete list of all documents referred to by the ET. 4 beneficiaries, government and municipality officials, project partners, financial institutions, international development organizations, business support organizations, and Zrda staff in Tbilisi and in the field. All KIIs used semi-structured, open-ended questionnaires tailored to the category of project stakeholder interviewed. 3.1.1.3 Focus Group Discussions The ET conducted a total of 13 FGDs—10 with Zrda’s beneficiaries (76 participants in total) and three with local government (LG) municipality officials in Marneuli, Ninotsminda, and Zugdidi (18 participants in total). The FGDs were organized using lists provided by Zrda from the activity’s database of beneficiaries and municipality officials. FGDs used semi-structured, open-ended questionnaires to gather information from participants that would serve to inform the ET’s understanding of the role of municipalities regarding local economic development and the extent to which they collaborated with Zrda in implementing activity interventions. They were also designed to obtain feedback from direct beneficiaries in terms of their individual and/or collective experiences with Zrda. 3.1.2 Quantitative Data Collection and Analysis The ET commissioned a local Azeri professional interpreter and translator to conduct a survey with non-beneficiaries in Kvemo-Kartli, following feedback from the Azeri community that a significant number of villages in the region were experiencing similar issues as those villages currently receiving Zrda’s support, resulting in questions being asked as to why they had not been targeted for support. The ET acknowledges, however, that Zrda employed rigorous criteria for community selection which was conducted within the activity’s available resources and necessarily limited its ability to support more than those selected. The survey was carried out in three villages: Nakhiduri, Tsurtavi, and Khidisxuri. It involved face-to-face interviews with farmers using a short survey questionnaire. A total of 31 farmers were interviewed (24 male and seven female). A complete analysis of the survey is presented as an integral part of EQ 4 (see Section 4) where activity-related issues regarding minority communities are fully explored. 3.2 EVALUATION LIMITATIONS The Zrda evaluation methodology involved the following potential limitations: 1) reliance on, at times, unverifiable perceptions and anecdotes by key informants (KIs); 2) recall bias by KIs commenting on past events; 3) halo bias by KIs incentivized to under-report socially undesirable answers or refrain from criticizing colleagues or people on whom they depend for support; and 4) sampling bias given the purposive (non-random) selection of KIs for KIIs and FGDs. The latter limitation constrained the ET’s ability to make generalizations about the activity and its results on the whole and, together with the absence of a comparison group, limited its ability to attribute cited or observed outcomes to activity interventions. To mitigate the above limitations, the ET sought to verify findings from one data source and method with findings from other data sources and methods through triangulation, and to focus more on the activity’s “contribution” to cited/observed results rather than on trying to attribute them directly to activity interventions. 5 4.0 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 4.1 EQ 1: WHAT APPROACHES HAVE BEEN MOST OR LEAST SUCCESSFUL IN CREATING JOBS AND INCREASING INCOMES IN TARGET COMMUNITIES AND WHY? 4.1.1 Findings According to Zrda’s Fiscal Year (FY) 2019 Annual Implementation Plan (AIP), the activity will continue to employ an overall sector-driven approach to its interventions, which it has pursued since its inception in March 2016. Two sectors were identified as being the primary focus of this approach: agriculture and tourism. The main aim of these interventions is to support the development of high value chains in both sectors (berries, honey, fruit, and vegetables in agriculture and eco-tourism, agro-tourism, food and wine tourism, and adventure tourism in tourism) in order to support job creation opportunities and increase household income potential. Zrda’s target areas are rural communities in the proximity of Georgia’s ABL regions of South Ossetia and Abkhazia. To monitor progress, Zrda has eight purpose-level M&E indicators, the three main and most measurable of which are: 1) sales value generated by MSMEs; 2) number of MSMEs generating new sales; and 3) number of jobs created. All three indicators are inter-linked, since any increase in sales by existing businesses or in the number of businesses increasing sales will likely lead to the creation of new jobs. The number of jobs created, however, has been Zrda’s most challenging target to meet, for two reasons: 1) Zrda’s targeted beneficiaries are micro and small enterprises in rural communities, which are generally family-oriented, subsistence-level businesses with few opportunities to expand and grow without financing, factors that apply to both agriculture and tourism; and 2) additional jobs in the agriculture and tourism sectors are often only part-time and/or seasonal/temporary, in line with seasonal demand. The creation of long-term (i.e., exceeding three months) full-time jobs is not a realistic prospect for either sector absent significant businesses expansion. Zrda’s LOP indicator target for jobs created is 2,400, although this number does not disaggregate between jobs in agriculture or tourism. However, Zrda does track jobs created in both sectors, which collectively contribute towards progress year-on-year in reaching the 2,400 LOP target. Zrda’s implementation modalities focus on two specific intervention approaches: 1) sub-award grants/subsidies (either directly to beneficiaries or via awardee service providers); and 2) TA as indirect support to enterprises with a focus on improving business performance. The former largely relates to micro-enterprises and small businesses, while the latter focuses mainly on slightly larger companies, i.e., those having no more than 50 employees. This could involve classroom-type training sessions or hands-on demonstration plot and field training courses. To date, most of Zrda’s support has been to micro-enterprises as those are the ones operating within the 81 target communities in 17 municipalities where Zrda works. It is against this background that the ET sought to gather data and information on the extent to which the activity has contributed to job creation in agriculture and tourism, particularly taking 6 into account the inherent limitations regarding the nature of Zrda’s beneficiary clients and the seasonality aspect of the jobs in both sectors. Zrda’s job creation is reported through the Purpose Level Indicator 3 (Number of jobs created). The activity’s M&E plan defines jobs as “all types of employment opportunities created during the reporting year in agriculture and non-agriculture-related enterprises, including on- and off-form employment.” The indicator takes into account those who are self-employed and hired as a result of the activity’s interventions. Zrda disaggregates jobs into three sub-categories: 1. Jobs lasting less than one month; 2. Jobs lasting from one to three months (short-term); and 3. Jobs lasting over three months (long-term). Jobs in all categories are calculated using full-time equivalents (FTEs), which means that a person hired for 10 hours per week is counted as 0.25 jobs in one of the three aforementioned categories. 4 Following a review of Zrda’s M&E documentation, the ET prepared a summary table of purpose￾level indicators and illustrative targets (see Annex 6 – Activity M&E Plan Summary), which highlights year-on-year progress of performance against targets. According to the M&E plan, Zrda has, to date, contributed to creating 905 jobs5 against an LOP target of 2,400. Four hundred (400) of those were in tourism-related enterprises, 317 were in agriculture, and the remainder were in service-oriented businesses. FGD participants from 10 regional FGD6 sessions confirmed Zrda’s contribution in helping them create jobs and increase household income (see Annex 5 – FGD Beneficiaries – Participants Responses). Of the 76 FGD participants, 20 from the tourism sector and 30 in agriculture stated that Zrda has helped them create jobs, although most were short-term, seasonal jobs. Regarding household income, the FGD sample is rather small relative to the 13,200 LOP target and 7,527 achieved to date. However, 60 of the 76 participants (79 percent) agreed that their incomes had increased by roughly 25 percent. 4.1.1.1 Tourism Participants in an FGD with hotel owners and family-type hostels in the Samegrelo-Zemo Svaneti region all said that Zrda has contributed to the development of the tourism sector in the region. All participants agreed that the training provided by Zrda in digital marketing and online booking helped make their businesses visible online, and that more tourists began visiting the area after becoming aware of the services and lodging opportunities offered online, which resulted in increased income for hotel and guesthouse owners. According to the FGD participants, after attending Zrda’s training, approximately 70 percent of their booking is now done through Booking.com and the rest through other sites/methods. One guesthouse owner stated that the training she attended on taxation issues and guesthouse management contributed to a 15-20 percent increase in guesthouse income. Other FGD participants mentioned adding new jobs as a result of Zrda’s support, including one who added three new jobs, one who hired two temporary interns, and another who hired one seasonal worker. 4 Zrda Activity in Georgia Activity Monitoring and Evaluation Plan (AMEP), May 2019, p. 26 out of 57. 5 The ET notes that references to jobs refer to FTE jobs. 6 Borjomi, Bolnisi, Ninotsminda, Gori (two), Akhaltsikhe, Zugdidi (two), Anaklia, and Tsalenjikha. 7 Participants in an FGD with Zrda staff noted that Zrda provides support in various areas of the tourism sector, including equestrian tourism, diving, and canyoning. To support these areas, Zrda provides training and equipment and helps organize events that attract tourists. For example, Zrda organized an ethno village in Chkhoria, worked with the Folk House where folklore and culinary arts are taught, and assisted with a book on Megrelian-Abkhazian-Svan cuisine and a handbook about Georgian artisans. Zrda also provided tourism trainings and study tours in EU countries, which have been successful in helping tour guides increase their incomes. KIs highly valued these trainings and tours and considered them to be one of the most beneficial of Zrda’s interventions. Participants in an FGD with tour guides all were very satisfied with the results of Zrda’s assistance and especially praised the training they received in topics like first aid, security, communication with tourists, and presenting products to tourists. Two tour guides stated that they attended 15-day trainings in Italy and France, and one reported getting second level guide certificates in Germany. Internationally recognized certificates, participants claimed, increase their professional credibility, helping them to attract more international tourists and allowing them to increase their tour fees. All tour guides in the FGD confirmed increasing their incomes after attending Zrda’s training sessions, some of them up to 40 percent. Participants noted that Zrda also trained them in how to offer their services through business websites and social media platforms, such as Facebook. However, they added that Zrda’s trainings did not discuss website maintenance or account updating, and they were unclear about how they would manage these mediums in the future. Several informants (from seven KIIs and FGDs) stated that small business owners also need training in general business skills, such as preparing business plans, farm management, and even in basic administrative functions, such as filling in forms and completing paperwork. Discussions with the Georgia National Tourism Administration (GNTA) representatives revealed that Zrda provided the agency with useful support for its tourism outreach campaign, which had been struggling, by developing a commercial webpage that increased tourists’ awareness about available tourist opportunities. GNTA representatives reported that, “Zrda is a flexible and reliable partner that responds quickly to the GNTA needs.” Although these KIs claimed that Zrda support has helped them create new jobs, they conceded that there is no system in place to count the number of new jobs created. While Zrda’s tourism sector interventions have helped increase incomes and create jobs in selected cases, interviews also revealed barriers to success for Zrda’s tourism sector interventions, particularly those along the Abkhazia demarcation line. For example, KIs in the Mayor’s office in Ganmukhuri reported their income has fallen because of the tourism competition they face from neighboring Anaklia, across the demarcation line in Samegrelo-Zemo Svaneti, where there are more tourist attractions and where tourists generally choose to stay. Furthermore, there is no lighting in Ganmukhuri after the bridge connecting Anaklia to Ganmukhuri, so tourists do not want to cross the bridge in the evening. Accordingly, they suggested that it would be better to put more money in agriculture projects in Ganmukhuri rather than tourism projects. In an FGD with eight managers of family hostels in Zugdidi, participants said that, while they have noticed increased interest from tourists in visiting Zugdidi as a result of Zrda’s trainings in online hotel registration, another result has been increased competition from hotels in the area. While 8 bookings in certain cases have increased by 70 percent, the increased earnings have not been significant because room rates must be kept low to beat competition, while booking fees are high. KIs also stated that increased earnings were primarily used to expand the business rather than to create new jobs. Many complained, however, that their visitors and income declined after the government stopped supporting the Anaklia festival. Many also stated that they do not always receive information about events in advance. This leaves them unprepared for the temporary influx of tourists. 4.1.1.2 Agriculture In the agriculture sector, jobs created with Zrda’s contribution have remained steady at just over 100 in each of the three years since 2017. While jobs in open-field production are seasonally limited, Zrda’s support for greenhouse development, requiring potentially more skilled labor, is experiencing an increase in project support for selected beneficiaries as greenhouse production can be year-round. Zrda’s Greenhouse Subsidy Program, which subsidizes the establishment and renovation of existing greenhouses through a fixed rebate subsidy scheme, has been particularly successful. At the time of writing, 17 farmers had already established 12,970 square meters in greenhouses for vegetable and herb production. While it is too early to say how many new jobs will be created as a result, the probability remains high that, if sustained, it will contribute to new jobs created. In 2017, Zdra financed a 65-day training in beekeeping, printed out manuals for beekeepers, and helped to develop the Beekeepers Alliance in Western Georgia (covering Samegrelo, Guria, and Imereti) with which Zrda signed a Memorandum of Understanding (MoU) to provide extension services for UNDP-supported beekeepers. Recently, Zrda funded six beekeepers with training and materials, which has helped them increase their incomes by 20-25 percent and created an opportunity for them to diversify their agricultural production and add jobs. A blueberry farmer in Zugdidi municipality reported that, after unsuccessfully starting his blueberry nursery with defective seedlings from Anasuli, Zrda provided him with two-year seedlings from the United States that were much more successful and provided him with a “substantial” income. Zrda also assisted him with training and marketing, which helped him sell his entire crop to a blueberry retailer from Sulekulu village. The same retailer has expressed interest in buying his entire harvest in the future. 4.1.2 Conclusions The evidence gathered by the ET indicates that Zrda’s sector-driven approach—implemented in line with the activity’s purpose-level indicators and targets and specifically focused on offering small grants/subsidies alongside TA initiatives—has contributed to increased jobs and increased (household) incomes among activity beneficiaries in both the agriculture and tourism sectors. This conclusion supports Zrda’s own indicator performance tracking results, with two caveats. First, jobs created in both sectors have almost universally been seasonal and, therefore, short￾term. Second, the evaluation methodology did not allow for a teasing out of to what extent each of Zrda’s sector-driven approaches (grants or TA) contributed to reported income and job increases, both overall and relative to each other. With that said, simply becoming more productive and being able to sell more locally was a common theme resulting from KIIs and FGDs, which was sufficient to improve participants’ personal circumstances. 9 Many of the sectors targeted by Zrda and other private sector development programs remain at a relatively low knowledge and skill level and require ongoing capacity development of sector participants. According to some participants, some areas in the ABL zone are more suitable for agribusiness than for tourism. 4.1.3 Recommendations • Zrda should continue to concentrate its effort in offering support measures that lead to improved business performance, e.g., through developing robust sales and marketing strategies, including training sessions/programs that encourage self-improvement for business owners, whether in agriculture or tourism. This approach should help ensure longer-term sustainability of small rural enterprises, particularly if marketing strategies and objectives become more of a priority. Companies must gear up to meet market demand and offer the quality of goods and services expected by today’s discerning customers. Zrda can be instrumental in contributing to this and should, therefore, continue with this approach, together with appropriate grants on a case-by-case basis where, for example, the creation of cold storage units, small processing facilities, and greenhouse construction can, alongside TA, significantly improve sales prospect. • Zrda should concentrate on supporting interventions that help ensure longer-term sustainability of small rural enterprises, e.g., through developing training sessions/programs that encourage self-improvement for business owners, whether in agriculture or tourism. • Zrda might consider focusing more on agribusiness in some of the targeted areas given their limited tourism potential. 4.2 EQ 2: DO INTERVENTIONS TARGETED AT AND/OR IMPLEMENTED IN PARTNERSHIP WITH LARGER COMPANIES RESULT IN CREATION OF MORE JOBS THAN INTERVENTIONS INVOLVING SMALLER COMPANIES AND WHY? 4.2.1 Findings In order for the ET to be able to meaningfully answer EQ 2, it was necessary to clarify the definition of “larger” and “smaller” in reference to those companies that may have been recipients of project interventions, as well as the meaning of “larger companies” as they pertain to partnerships. Reference was therefore made to Zrda’s FY 2019 AIP, which provides a clear definition of small, medium, and large enterprises.7 Given this definition, the ET’s investigations found that Zrda’s main client base of rural beneficiaries in both agriculture and tourism only qualified as micro-enterprises, which are defined as those businesses employing fewer than 10 persons, with an average annual turnover less than Georgian Lari (GEL) 200,000. 7 Enterprise Category 1 – Small Enterprise: Comprised of enterprises of any legal form, with an average annual number of employees not exceeding 50 persons and an average annual turnover not exceeding GEL 12 million; Enterprise Category 2 – Medium-Size Enterprise: Comprised of enterprises of any legal form, with an average annual number of employees between 51 and 250 persons and an average annual turnover between GEL 12 million and GEL 60 million; Enterprise Category 3 – Large Enterprise: Enterprise with an average annual number of employees greater than 250 and an average annual turnover exceeding GEL 60 million.; Section 1.4. Enterprise Categorization and Zrda Intervention & Investment Approach, p. 10, FY 2019 Annual Implementation Plan, October 1, 2018 – September 30, 2019. 10 Zrda’s partnership arrangements to date regarding the delivery of interventions to this client base has two strands. The first strand involves working in collaboration with the activity’s two private sector partnerships—British Petroleum (BP) and Anaklia Development Consortium (ADC)— both of which can be classified as large companies. Zrda collaborates with BP through two IPs: Elkana8 in Shida Kartli and Regional Development Association9 (RDA) in Kvemo Kartli and Samtskhe-Javakheti. Interventions are essentially co-funded initiatives, including support for small farmers in minority regions identified by Zrda and BP as target priorities, with a focus on educating farmers to upgrade their skills through training on business development and management and the provision of small grants for infrastructure projects (e.g., greenhouse construction). Zrda’s partnership with ADC focuses on developing local communities and preparing them for the changes associated with a fully functioning Deep Seaport. Previously, ADC had provided trainings to local communities to help them expand their businesses, while Zrda implemented similar activities in the Anaklia region. ADC and Zrda consequently decided to cooperate and merge resources to ensure the efficient provision of training services. Working with ADC, however, has presented Zrda with issues that have prevented progress in line with expectations, due largely to events outside of the activity’s control (mainly political related to Anaklia Deep￾Sea Port construction) and further elaborated in EQ 3 below. Here, job creation opportunities were directly linked with increasing the competitiveness and growth of MSMEs in proximity to the port, now on hold. Zrda’s involvement with the Anaklia Education Hub, an initiative launched in March 2019, together with the MSME Grants Program under the auspices of the Anaklia Community and Economic Development (ACED) program (a joint initiative between ADC and Zrda), have started the process of supporting local economic development and, in particular, upgrading workforce skills and knowledge. Interviews with ADC’s management showed that it appreciates the support from Zrda and “the flexibility that the project provides, especially on training.” According to ADC, the local community needs more jobs; however, to be employed, they need training on how to operate seaports, which ADC cannot do. They were confident that the collaboration with Zrda will provide such training. Job creation opportunities remain, but in the longer term; in the shorter term, they are unlikely to bear fruit until underlying political issues have been resolved. In the second strand, Zrda works with a variety of smaller partners and agencies, mainly to provide trainings and TA related to Zrda’s eight purpose-level indicators (see Annex 6 – Activity M&E Plan Summary) and to promote access to financial services for local beneficiaries. Such delivery is largely through Zrda’s sub-award mechanism used to finance activities via local service providers, including private enterprises, non-governmental organizations (NGOs), associations, business support organizations (BSOs), and LG agencies. All Zrda interventions are implemented under the four sub-purposes shown below and against which all indicators are tracked and measured. • Sub-Purpose 1: Improve MSME Development and Growth • Sub-Purpose 2: Increase Productivity of Rural Households 8 Biological Farming Association “Elkana” is a Georgian NGO founded in August 1994. Elkana has 900 member farmers. 9 RDA is a non-profit organization registered in Georgia in 2010 with head office in Tbilisi and field offices in Akhaltsikhe in Samtskhe-Javakheti and Marneuli and Tsalka in Kvemo Kartli. 11 • Sub-Purpose 3: Facilitate Market Linkages Between Producers and Buyers • Sub-Purpose 4: Promote Local Economic Development by Establishing Networks Below is a brief assessment of Zrda’s collaboration with partner organizations other than the larger BP or ADC partnerships. 1. Greenhouse and Agro-Insurance Promotion: This is a co-sponsorship between Zrda and the Agriculture Project Management Agency (APMA). Reducing risk and financial loss due to failed crops will help farmers stabilize their earnings. Job creation opportunities dependent on sales growth and personal circumstance and objectives of farmers. 2. ICT, Innovation, and Entrepreneurship Activity with the Georgia Innovation and Technology Agency (GITA): The Zrda-contracted service provider, Brain Solutions, completed a training course for GITA’s online training platform. The purpose of this training was to help those in rural areas acquire marketable jobs. Forty (40) percent of GITA and Zrda’s training participants are women, and the majority of participants are between 16-25 years old. Current trainings that show impact potential include e-commerce training to advertise and register products to online customers and digital marketing and website development. KIs interviewed stated that these trainings position participants to be employed as consultants and service providers in existing companies and organizations. Zrda plans to start a new series of trainings in programming (e.g., JAVA). The trainings will be conducted with those local beneficiaries who have the potential to be employed. Jobs created by this collaboration are not monitored but there is a strong likelihood of employment occurring once training is fully promoted in the regions. 3. Restaurant Service Improvement Activity: Zrda-contracted service provider, Gastronaut Ltd., carried out eight training events on food costing and pricing, food and wine pairing, and customer service. The purpose of the training was to upgrade services and improve profitability and income. Several participants praised the trainings, especially the one of food costing and pricing. Job creation prospects are limited depending on size and location of the businesses. 4. Private Sector Extension System Development: Five Zrda-contracted Farmer Service Centers (FSCs) were employed to promote fee-based consultation services and arrange demonstration plots in Zrda’s target communities. Job creation opportunities in these open￾field developments once fully in operational mode have a high probability of being achieved. 5. Tour Guides Training and Certification: Zrda service provider Center for Training and Consultancy (CTC) will develop job and competence frameworks for tour guides. Job creation opportunities are a strong likelihood for those who want to enter the profession having such qualifications. 6. Greenhouse Technology Promotion and Training: Zrda’s Greenhouse Subsidy Program subsidizes the establishment and renovation of existing greenhouses. Particularly relevant is Zrda’s collaboration with the Legal Entity of Public Law (LEPL) Internally Displaced Person (IDP) Livelihood Agency supporting the development of 240 square meter greenhouses. Job creation opportunities will have a high level of probability as staff will be needed to work in the greenhouses, which will have year-round potential. 12 7. Value Chain Financing Scheme in Zrda target regions: Zrda’s successful collaboration with Microfinance Organization (MFO)10 Crystal and subsequent facilitation of a partnership between MFO Crystal and Agro-Com11 in Shida Kartli will be rolled out to other Zrda areas of interest to offer reasonable credit limit loans to farmers according to their production needs. This will allow farmers to buy necessary inputs then repay Crystal post￾harvest. Seventeen (17) micro loans totaling GEL 123,579 were issued in the reporting period Quarter (Q) 3 FY 2019. Job creation prospects are dependent on size and location of farms (open-field or greenhouses) and time of the year (greenhouses can produce year-round so employment prospects are more likely). All KIs interviewed confirmed that none of the respondents, except for the honey producing company Khareba (which is affiliated with the Winery Khareba, one of the biggest wine-producing companies in Georgia), met the definition of a small enterprise set by Zrda but were instead either sole or micro entrepreneurs (guesthouse/local café owners, tour guides) or individual farmers (greenhouse owners, vegetable/berry producers), which were Zrda’s primary target beneficiaries. Participants in an FGD with Zrda staff confirmed that “there are no large businesses in the region with which Zrda works.” They stated that the largest restaurant in Zugdidi, which is already one of Zrda’s beneficiaries, is a relatively large business and has participated in the activity trainings. A large business on the other hand, Hotel Leto, was not interested in the cooperation since the grants offered ($10,000-$15,000) were too small given its size. 4.2.2 Conclusions The ET found no evidence that interventions carried out through either larger or smaller partnerships had a greater impact on job creation than the other, with the exception of Zrda’s investment initiatives, especially in farming communities, which appeared to offer significantly greater job creation opportunities. 4.2.3 Recommendations • Zrda should leverage its available resources and those of its partners to co-fund joint ventures to help businesses with greater long-job creation potential (e.g., greenhouses, cold storage units, etc.). If, at the same time, Zrda can link this to offering marketing support to target sectors and businesses, then opportunities for growing targeted beneficiary companies will have a greater likelihood of success. • USAID should consider activities in the future similar to Zrda that do not have job creation as their ultimate objective. This will allow the activities to concentrate on interventions that lead to real advances for businesses in rural communities with less reliance on seasonality and more towards developing businesses that are sustainable in the longer term as a result of their being able to upgrade, expand, or diversify activities. 10 JSC MFO Crystal was established in 2007 on the basis of the decision of the Crystal Fund according to the Georgian Law on MFOs dated July 18, 2006. As of December 31, 2016, the company’s major shareholder was Crystal Fund with 47.96 percent shareholding. 11 Agro-Com is an FSC operating in Shida Kartli since 2011. It offers specific products to farmers and agriculture organizations such as pesticides, fertilizers, irrigation systems, seedlings, and hail nets, among others. 13 4.3 EQ 3: WHAT ARE THE KEY LESSONS LEARNED FROM IMPLEMENTING PPPS UNDER ZRDA? 4.3.1 Findings In line with USAID’s definition of a PPP arrangement,12 Zrda set up two large PPPs: one with ADC13 and one with BP. In addition, Zrda signed partnership memorandums with the JSC Partnership Fund, and the Georgian Farmers’ Association (GFA), in coordination with the Georgian Farmers Distribution Company (GFDC) and the Adjara Hospitality Group. The partnership arrangement with ADC was an integral part of the ADC’s corporate social responsibility (CSR)14 strategy that aimed to promote the economic development of Anaklia and its surrounding areas, facilitate the development of community infrastructure to meet Anaklia’s needs, support the growth of MSMEs in the region, and develop and sustain a skilled workforce. According to the initial arrangement with ADC, both contracting parties were expected to contribute $1 million to this joint initiative. This arrangement, signed in September 2018, was subject to further modification, which took place in December 2018. As stipulated in the modified agreement, ADC reduced its contribution to $500,000 and reframed the roles and responsibilities of both parties. 15 Meanwhile, the PPP arrangement with BP entailed a contribution of $385,000 from BP and $300,000 from Zrda. ADC was requested to contribute across five initiatives that were aimed at developing the capacity of MSMEs and individuals through ACED. ACED incorporated several components, including the Anaklia training center/educational hub,16 Kolkheti National Park, transformation of Anaklia to a primary sports tourism destination, development of modern businesses, facilitation of a gastronomy festival, and other cross-cutting initiatives. In 2019, ADC faced a number of challenges. Due to certain political tensions, the development of the Deep-Sea Port in Anaklia was suspended. This affected the PPP arrangement between Zrda and ADC, as the activities foreseen under this partnership were delayed for several months. In February-March 2019, ADC delivered training sessions to 77 participants in the following areas: basics of beekeeping, food safety, fish farming, solid waste management, blueberry production agrotechnology, and professional gardening and landscaping (Figure 1). 12- Successful proposals will bring at least a 1:1 resource leveraging, meaning that the partner(s) will collectively contribute resources that are at least equal to, if not greater than, the level contributed by USAID. - Partners are expected to bring significant new resources, ideas, technologies, and/or partners to address development problems in countries where USAID is working. - Partners can be foundations: United States (U.S.) and non-U.S. NGOs; U.S. and non-U.S. private businesses; business and trade associations; international organizations; U.S. and non-U.S. colleges and universities; U.S. cities and states; other USG agencies; civic groups; other donor governments; host country governments; regional organizations; host country parastatals; philanthropic leaders including venture capitalists; public figures; and advocacy groups. Source: https://www.usaid.gov/el-salvador/public-private-partnerships. 13 ADC is a consortium which consists of TBC Holding from Georgia, SSA Marine from the United States, British Wondernet Express working in Central Asia, and G-Star Ltd. from Bulgaria. 14 According to USAID’s Grant and Cooperative Agreement, Modification 06, signed August 29, 2019, Zrda plans to explore establishing a partnership with the CSR Club, which was established by the USAID-funded Advancing CSO Capacities and Engaging Society for Sustainability (ACCESS) project. In addition, a CSR Challenge Fund will be created to engage private sector companies to respond to social and economic challenges in ABL communities. 15 The Amendment was caused by internal (managerial) and external (political tension with the current government) issues the ADC faced. 16 Opened in the Spring of 2019 together with the MSME Support Grants Program under the auspices of ACED. 14 Figure 1: Distribution of Participants by ADC Training Sessions Basics of beekeeping 14% Blueberry production agrotechnology 16% Food safety training 24% Professional gardening and landscaping training 23% Solid waste management 9% Fish farming 14% Source: USAID/Zrda Activity Data, 2019 All FGD participants in Anaklia reported taking part in training sessions. They uniformly rated the training as high-quality and were especially appreciative of the training in online booking. The general sentiment expressed was that, since the training, they “have become familiar with the peculiarities of the online booking” and now do most of their booking (up to 70 percent) online. Participants also noted that after the trainings, the number of guesthouses operating in the area increased significantly. Zrda also offered training in management and services but only one out of the eight participants interviewed mentioned benefiting from this training, stating that he was able to find his current job because of the knowledge and skills he acquired as a result. Three out of eight FGD participants confirmed being unemployed and attending basic English language training. The Anaklia Education Hub is used to deliver different types of activities and training sessions, including English language, computer skills, entrepreneurship, and business administration in tourism and agriculture. ADC was also engaged in developing a concept note and an action plan for a gastronomy festival scheduled for September 2019 in Zugdidi/Ganmukhuri. In addition, within the framework of the ACED, 16 out of 135 applications were selected from among those submitted by local entrepreneurs from Anaklia and the surrounding villages. Overall, GEL 350,000 was allocated to the grant scheme, co-financed equally by Zrda and ADC. The activity received 135 grant applications, out of which 94 were considered eligible, and 40 out of 94 passed the minimum threshold of 70 points. In the end, Zrda selected 16 applications for funding. The total amount of funds disbursed was equal to GEL 162,874. Funded businesses operated mainly in the service sector (hair salon and barber shop, children’s attractions, bakery, confectionery store, etc.). 17 Three out of eight participants in an FGD in Anaklia reported applying for the grant scheme. However, only one of the three applications was approved. All applicants claimed that there was a certain miscommunication about the application selection criteria. “There were rumors that those who already received the financial support were not eligible to take part in it and no clear answers [were] received about the reason for the applications being rejected” reported one participant.18 One participant highlighted that “this miscommunication caused mistrust.” Zrda confirmed that “all applicants will be contacted and informed about the results”; however, the FGD participants (those 17 The source: project reports and the feedback received from the participants of the FGD in Anaklia. 18 The source: the feedback received from the participants of the FGD in Anaklia. 15 who were unsuccessful in the grant scheme) stated that, despite receiving a letter on the status of their applications, the letter was very vague. The PPP arrangement with BP entailed implementing community development initiatives (CDIs) for 19 villages in Kvemo Kartli and Samtskhe-Javakheti regions. Within the framework of this PPP arrangement, there were 5219 applications, of which 33 came from Akhaltsikhe, 18 came from Tsalka, and one came from Adigeni. In addition, 22 villages in Shida Kartli were supported by the Association Elkana, an IP of BP. USAID’s Modification 06 (mentioned above) envisages a continued partnership with BP through Elkana in Shida Kartli and RDA in Kvemo Kartli and Samtskhe￾Javakheti with a view to supporting 41 joint BP-Zrda villages focusing on MSME training and small grant support. To implement this initiative, as well as other proposed ADC activities,20 the activity received an extension from March 14, 2021 to December 14, 2021. This will increase the likelihood of activity objectives being successfully achieved. In 2019, Zrda and the JSC Partnership Fund signed an MoU to facilitate investments in MSMEs in the ABL communities through a co-financing fund—the Startup Georgia Program. Each partner contributed $500,000. The anticipated result of these initiatives by activity’s end is increased MSME sales in ABL communities of $300,000, 35 new jobs created, and $50,000 leveraged from partners. The partnership with the GFA, GFDC, and Adjara Hospitality Group, resulted in creating a mobile match-making platform, Argonavti, for GFA members to sell their products to the hotels and restaurants of the Adjara Hospitality Group. According to the feedback received from KIs, the platform generated $120,000 in sales from GFA members in Zrda-supported municipalities. GFDC planned to invest over $1,000,000 into the supply chain infrastructure to support the platform and deploy the full-cycle supply chain logistics. The Adjara Hospitality Group contributes about $10,000 in cash to support GFA’s organizational development and sustainability and was engaged in contracting the farming schemes jointly with GFA to help farmers generate stable income. 4.3.2 EQ 3a: In addition to the Zrda activity, what could the government, communities, or others do to help businesses be more prosperous in five years? USAID/Georgia requested the ET to ask the following question during its investigations in order to elicit responses from interviewees: “In addition to Zrda project, what could the government, communities or others do to help businesses be more prosperous in five years?” KIIs and FGD participants identified a number of methods, based on lessons learned from past experience, in which the government, communities or others could contribute to helping businesses be more prosperous in five years. The most commonly cited method (seven KIIs and FGDs) was increased government investment in developing local tourist infrastructures. In particular, KIs cited the need for increased investment in signage for street names and domicile addresses so that tourists can more easily find guesthouses and other tourist venues (six KIIs and FGDs). According to KIs, the location of the guesthouses is the first question that tourists ask, and the lack of signage, together with the frequent changing of street names, creates a “serious 19 Three out of the 52 applications failed to meet health, safety, and environmental assessment criteria and were dismissed. 20 The Anaklia Beautification Program (to upgrade public spaces) and Gastronomy Festival (organized in Samegrelo to showcase the best recipes, dishes, and books of regional and national culinary traditions). 16 obstacle” for tourists to Google and find the location of a guesthouse or specific attraction places. For this reason, local guesthouses are receiving negative online reviews (in the section “easy to find”). Also noted by KIs were the need for: 1) improved local government control of feral dogs (two KIIs and FGDs), which are often found in tourist areas (one KI reported an incident of a tourist being bitten by a feral dog); 2) providing tour operators and guesthouses with more information about tourist attractions in the area, or alternatively training them improve their tourism digital marketing skills (four KIIs and FGDs); 3) training and installing a large number of qualified tour guides (three KIIs and FGDs); 4) increasing roadside tourist facilities, especially the number of toilets (two KIIs and FGDs); and 4) improving street lighting (two KIIs and FGDs). KIs also noted the need for additional capacity development for guesthouse operators in the areas of business and facilities management, citing specifically poor quality or missing guesthouse facilities, such as toilets, (suggesting, in turn, the need for financing for facilities’ upgrading) poor service quality, and overpricing, in the latter case notwithstanding poor facilities or service quality. The issues described above take on increased importance in concert with Zrda’s interventions to link the Georgian tourism sector to online marketing and information sources that best facilitate an incipient tourism and guesthouse business environment. Another way in which future interventions might help improve the future prosperity of targeted businesses is through an improved and more appropriate regulatory structure for small guesthouses (three KIIs and FGDs). For example, one small guesthouse operator claimed that the government treated it as if it were a large hotel in that, in order to meet fire safety standards, the guesthouse was ordered to take proper measures within one month at a cost exceeding GEL 5,000. Another guesthouse operator was told she had to treat the guesthouse roof with fire extinguishing liquid and install fire balloons. In neither of these two cases was the guesthouse in a position to comply with the order, nor would it be in the near future given its financial status and business cash flow. Participants in three KIIs cited deficiencies with the DMO development process that could be addressed in future interventions (in addition to Zrda) related to a lack of a well-defined institutional structure or business models, low staff capacity, and insufficient funding such that they are not yet ready to engage the private sector. Yet another area in need of future market interventions is the generation and dissemination of statistics and other market information to market participants (four KIIs and FGDs) in the form of, for example, market and networking opportunities. According to these KIs, business owners, or would-be business owners, do not have access to this type of information and do not know where to turn to get it. In addition, two KIs stressed the absence of youth support programs in the municipalities and the need to expand the number and geographic coverage of youth programs together with training for youth organizations to develop joint strategies on how to engage and encourage youth development initiatives. KIs in an FGD with hotel owners in Zugdidi stated that projects like Zrda need to revise the criteria used to select beneficiaries. For example, Zrda only accepts business plans submitted by guesthouses that have 10 or more rooms. Thus, those having nine rooms or fewer are unable to compete for the grant. In order to be eligible, the owners will need to expand. All FGD participants, however, confirmed that they will never consider loans either from microfinance 17 institutions or banks to be an alternative financial source to expansion because of the high interest rate (about 36 percent) and collateral requirements. When pressed to provide an indication of what they considered important for their businesses and livelihoods, KII and FGD participants generally agreed on the need for improved access to finance to support their ability to, for example, upgrade premises and/or facilities or procure working capital. The primary focus of these financial concerns was for the government and the micro-finance sector in particular to introduce a more beneficial credit scheme to facilitate less stringent loan requirements. A final widely cited factor with potential to contribute to longer-term business growth over the next five years was for private sector development initiatives to take a more targeted approach to support sectors in different locations that have comparative advantages (10 KIIs and FGDs). Together with this observation is the companion observation that some of the existing private sector development initiatives do not always take a strategic approach to determining which sectors and businesses to support in given locations, for example targeting tourism interventions in locations without a reputation or comparative advantage in tourism but with a greater need for agricultural development, or promoting the development of non-traditional or specialty agro￾products in cases where greater need and potential exists for strengthening production and marketing of traditional agro-products. An example is Ganmukhuri where several KIs mentioned that tourism does not provide the desired income since the area is not that attractive to tourists. Instead, they would prefer assistance in processing agricultural products. 4.3.3 Conclusions Zrda has experienced mixed success with its ongoing PPP arrangements. While the partnership with ADC has made a start in reaching out to communities within the Anaklia development zone, progress has been hindered mainly by political challenges connected with the Anaklia Deep-Sea Port construction activities. The BP partnership has been more successful in meeting the needs of mutual BP/Zrda target communities via a number of co-financing initiatives, including MSME training (upgrading and educating farmers), small infrastructure projects, and provision of small grants to improve production facilities. With regard to proposed future PPP activities, it is too early to say at this time what will result from the agreed activity modification related to the JSC Partnership Fund and/or the likely outcome of continued collaboration related to Anaklia, which may yet be influenced by (political) issues out of the PPP’s control. Zrda’s formation of PPP’s with well-established local (ADC) and international (BP) organizations clearly has merit in meeting the aspirations and aims of both sets of partners with regard to addressing the ongoing needs of target activity beneficiaries in their mutual intervention areas. Even with mixed levels of success in meeting agreed objectives, the principle of Zrda working closely with these interested private sector partners is sound. Such collaboration also avoids the potential for duplication of effort in reaching and supporting the same (or similar) target beneficiaries, particularly in the ABL zones. Future PPP initiatives can certainly build on current experience and will have the added advantage of Zrda being extended until December 2021, allowing sufficient time for envisaged joint objectives to be achieved. Feedback from KIs yielded a number of lessons learned related to possible activities that the government, communities, or others might do to help businesses be more prosperous in five years. These included making greater investments in local tourism infrastructure; additional capacity building for guesthouse operators and DMOs; enacting regulations more appropriate for 18 small-sized guesthouses and other tourism operations; revising criteria for selecting beneficiary businesses that are more appropriate for targeted micro and small businesses; designing interventions to increase access to financing for targeted micro and small businesses, particularly in terms of more favorable loan terms; and being more strategic about leveraging the comparative advantages in selected regions and locations. 4.3.4 Recommendations • Zrda should continue its cooperation with existing partners through the existing workstreams set out in the respective MoUs with the ADC and BP. • Zrda should consider putting in place a contingency plan in case unexpected and/or uncontrollable circumstances occur. This could take the form of an early warning system emanating from, say, activity field operatives, offering advice to activity management of their concerns regarding unfolding events that they perceive could affect activity interventions. • Zrda should twin the digital marketing activities with the other tourism development interventions. If the former outpaces the latter, it will possibly blunt any overall positive effect on the development of the tourism sector and tourism operators’ income and employment. • Zrda should work with the GoG, other development projects, and relevant market actors to generate and disseminate statistics and other market information, including information about networking opportunities, to small businesses and guesthouse operators. • Zrda should take a more targeted approach to supporting sectors and businesses in different locations to leverage their comparative advantages. 4.4 EQ 4: HAVE ZRDA’S SUB-AWARDS AND TA ACTIVITIES STRUCK OPTIMAL BALANCE BETWEEN SUPPORTING MINORITY AND NON￾MINORITY COMMUNITIES? DOES ZRDA SUFFICIENTLY INTEGRATE AND BENEFIT MINORITY COMMUNITIES? 4.4.1 Findings 4.4.1.1 General Observations – Completed Activities and Work-in-Progress Zrda’s activities regarding minority communities fall under the activity’s Sub-Purpose 1 and specifically under Activity 1008A: Underserved Minority and Geography Small Grants Activity. The main areas targeted for support under this relatively recent activity, which began in early 2019, focused on the Kvemo-Kartli region (with a predominant Azeri population) and the nearby Samtskhe-Javakheti region (with a predominant Armenian community). According to the National Statistics Office of Georgia’s Ethnic Composition of Georgia 2014 Census, Kvemo-Kartli had 177,032 Azeri inhabitants21 accounting for 41.8 percent of the entire region’s population and 75.9 percent of Georgia’s total Azeri population of 233,024 in 2014. Marneuli with 87,371 Azeri inhabitants and Bolnisi with Azeri 33,964 inhabitants together accounted for 68.5 percent of Kvemo-Kartli’s Azeri population. These two towns were visited by the ET during the field trip part of the evaluation where it conducted KIIs and FGDs with local farmers. 21 The number of inhabitants could have changed since 2014 regarding the ethnic minority populations in Kvemo￾Kartli and elsewhere in Georgia. 19 The same 2014 census provided the following statistics for Georgia’s Armenian population. There were 168,012 Armenians in the country, with 81,089 living in Samtskhe-Javakheti accounting for 50 percent of the total Armenian population. Akhalkalaki had 41,870 Armenian inhabitants (52 percent of the region’s total Armenian population), Ninotsminda had 23,262 inhabitants (29 percent), and Akhaltsikhe had 12,028 inhabitants (15 percent). The Armenian community accounted for 93 percent of Akhalkalaki’s total population. The ET visited all three towns to conduct KIIs and FGDs with beneficiaries and municipal officials. It is against this background of statistical data regarding both the Azeri and Armenian communities in Zrda’s intervention regions that the ET conducted its inquiries in relation to this question. To date, according to Zrda’s FY 2019 Quarterly Report III (N14), 22 applications were submitted for the Bolnisi and Marneuli Small Grants Program (Bolnisi, 16; Marneuli, six). All grant recipients were from ethnic minorities. Following the evaluation process, seven applicants (Bolnisi, five; Marneuli, two) operating in berry production, greenhouses, fruit growing, and other services were selected for award. Procurement is scheduled to follow in the next quarter. Also, under Sub-Purpose 1, Zrda conducted food safety trainings for joint BP and Zrda MSME beneficiaries in Akhaltsikhe; conducted MSME Business Skills Training in Samtskhe-Javakheti, and introduced, in partnership with approved service provider, Arealy LLC, hospitality and retail software presentations in Akhaltsikhe and Marneuli. Under Sub-Purpose 2, Zrda implemented activities focusing on increasing household incomes from both on- and off-farm opportunities. The main interventions included Zrda, in partnership with the CTC, signing an MoU with the Akhaltsikhe Vocational and Education Training Center (VET) “Opizari,” to design and deliver certification training courses and support vegetable farmers in Kvemo-Kartli with an introduction to sweet bell pepper production. Sub-Purpose 3 largely focused on the tourism sector. Zrda, through its sub-contractor Gepra, has supported the development of a promotional website for the Samtskhe-Javakheti region, together with training of local DMO staff. Zrda has also undertaken important work under its agriculture subsidy program to support farmers with post-harvest handling, packaging, storage, and marketing, issues that the ET encountered frequently during its investigations and described in more detail under Section 4.4.1.2 below. Zrda supported a number of initiatives under Sub-Purpose 4 with a focus on establishing new networks in both the public and private sectors. Zrda is currently collaborating with partners and local municipalities in Samtskhe-Javakheti and Kvemo-Kartli regarding support for infrastructure projects with respect to road rehabilitation. 4.4.1.2 General Observations – Minority Communities The issues raised by Zrda’s targeted beneficiaries in Kvemo-Kartli and Samtskhe-Javakheti were essentially the same, particularly in agriculture, as those experienced by rural farming communities in non-minority regions, which the ET defined as all other regions not pre-identified as minority areas. Zones with IDPs were not considered as minority regions under this evaluation. According to the ET’s analysis of the various activities carried out by Zrda under the four main sub-purposes highlighted above, no particular bias was noted in Zrda’s interventions for either minority or non-minority communities. This applied almost equally across all six Zrda regions with regard to Sub-Purpose 1, where the number of activities implemented by the activity was evenly divided between tourism and agriculture. Under Sub-Purpose 2, there was universal 20 support for agricultural activities across all regions. In terms of Sub-Purpose 3, Kakheti, Samtskhe￾Javakheti, and Samegrelo-Zemo Svaneti received predominant support in tourism activities, while the three other regions received almost equal Zrda support in both tourism and agriculture. Regarding Sub-Purpose 4, activities were solely focused on agriculture in Kvemo-Kartli and mainly on tourism in Samtskhe-Javakheti, as in Samegrelo-Zemo Svaneti. Shida-Kartli and Kakheti received equal support in tourism and agriculture activities. 4.4.1.3 Specific Observations KII participants in Kvemo-Kartli and Samtskhe-Javakheti expressed overwhelming appreciation of Zrda’s interventions aimed at supporting local economic development. Mayors and administrative staff believed that Zrda’s involvement and cooperation with them significantly contributed to progress being made in both agriculture and tourism. In both regions, few other support organizations were as involved as Zrda in addressing the problems hindering rural communities’ ability to improve their livelihoods. Similar comments were made by beneficiaries at sites visited by the ET where there was clear acknowledgement of Zrda’s support in helping fund the purchase of needed machinery and equipment. For example, a beekeeper in Nakalakevi Community, Aspindza Municipality, received additional beehive boxes, honey pressing equipment, and other equipment to improve production and storage of his honey together with branding advice for his products, which he claimed would allow him to sell directly to local supermarkets. Feedback from FGDs, in which 47 percent of participants were from Azeri and Armenian minority communities (see Table 1), provided varying responses to interview questions depending on location and sector. In agriculture, the main issues of concern were the continuing battle regarding the availability of both drinking and irrigation water for rural farming communities, lack of support from local municipalities to address ongoing problems, lack of access to finance to fund needed machinery and equipment, inability to deal with pest infestations ruining crops, and lack of protection from natural events, such as hail. Job creation was of limited interest to FGD beneficiaries as they were mainly micro-enterprises, often at the subsistence level with little means or ability to expand activities beyond their smallholdings. Municipalities were not seen as catalysts of change. Instead, participants saw projects such as Zrda as the most likely sources of needed support. Table 1: FGDs by Location and Business Activity No. FGD Location Business Activity No. of Participants M F Total 1 Borjomi (Tourism & Agriculture) Hotels, Tour Guides, Restaurants, Adventure Businesses 8 4 12 2 Bolnisi Azeri Farmers 9 1 10 3 Ninotsminda Farmers 5 1 6 4 Gori (Tourism) Guesthouses 1 7 8 5 Gori (Agriculture) Farmers 6 2 8 6 Akhaltsikhe Hotels & Restaurants 6 2 8 7 Zugdidi Guesthouses 8 8 8 Zugdidi Restaurant/Café Owners 2 1 3 9 Anaklia Guesthouses & Unemployed 3 5 8 10 Tsalenjikha Tour Guides & Guesthouse Owner 4 1 5 Total (Numbers) 44 32 76 Total (%) 58% 42% 100% Note: Green highlight area indicates towns in regions with the Azeri and Armenian communities. 21 4.4.1.4 FGD Data and Analysis The ET conducted separate FGD sessions with beneficiaries in tourism and agriculture, with one in Borjomi attended by participants from both sectors. For those involved in agriculture, the ET noted similar issues of concern by all participants irrespective of location, with universal complaints about the lack of functioning irrigation systems on their farms to ensure annual crop survival, thus restricting their ability to produce sufficient food to feed their families, let alone enough to sell to market. Perennial difficulties with pest infestations, hail, and lack of proper farming equipment have all contributed to low incomes being generated, according to all participants. Yet those beneficiaries who Zrda provided with assistance in setting up irrigation systems (either for open-field or greenhouses) or other support—e.g., training on quality standards, seed selection, crop rotation, modern technology, and storage—all were appreciative of Zrda’s contribution and willingness to address their problems. The ET encountered no issues of concern related to the co-funding of interventions between beneficiary and Zrda with those participants who had benefited from activity support. For those involved in the tourism sector, general issues raised by all participants included the lack of accommodation for tourists, the lack of proper functioning of public toilets, no promotional campaigns to support the region (from the Ninotsminda FGD in Samtskhe-Javakheti), poor infrastructure access to tourism sites, little public-private sector coordination in local development planning, and the lack opportunities for tourists to relax, have fun, and spend money. In specific terms, participants who had received support from Zrda were appreciative and grateful for funding, in particular. For example, in the Akhaltsikhe FGD, a guesthouse owner was able to publish a book on traditional cuisine with 100 percent Zrda funding, while refurbishing her kitchen with 70:30 co-funding. Another restaurant owner attended Zrda training on menu preparation and design, which apparently led to increased business. 4.4.1.5 Mini-Survey With Non-Beneficiary Rural Azeri Farmers in Kvemo-Kartli During the ET’s field trip to the regions, the decision was taken to conduct a mini-survey of non￾beneficiary farmers in Kvemo-Kartli in order to determine whether issues of concern were similar to those experienced by Zrda beneficiaries who had received Zrda support. Three villages were selected for the survey—Nakhiduri, Tsurtavi, and Khidisxuri—and a local Azeri professional translator was commissioned to carry out face-to-face interviews using a brief questionnaire prepared by the ET. Questions were directly related to the types of support Zrda had offered its direct beneficiaries aimed at improving agricultural production and what still remained to be addressed to tackle ongoing concerns. Sixteen (16) farmers were interviewed in Nakhiduri (12 male and three female), and 15 farmers were interviewed in Tsurtavi and Khidisxuri (11 male and four female). In summary, 19 of the 31 respondents in all three villages complained that their livelihoods were significantly affected due to a lack of irrigation for crops and 15 indicated that a lack of drinking water was affecting their families’ lives. Seven advised that poor-quality seeds were problem, and six indicated that the lack of markets to sell to was an issue. Forty-two (42) percent of farmers in both villages had been able to borrow money from a bank, mostly for potato crops and fertilizers, although they complained about high interest rates and, in one case, the inability to pay back the loan. In Nakhiduri village, all interviewed farmers said that the municipality did not support them in any way. A similar story was encountered in Tsurtavi 22 and Khidisxuri, with only two farmers saying they had been assisted. Only one farmer in all three villages had heard of USAID/Zrda. 4.4.2 Conclusions Evidence gathered by the ET indicates that Zrda has struck an optimal balance in supporting minority and non-minority communities and has sufficiently integrated the former into its diverse interventions. No disparity or discrimination in Zrda’s support activities between minority or non-minority was observed by the ET or referred to by either beneficiaries or municipal government official. Irrespective of their minority status, beneficiaries greatly appreciated the support they received from Zrda, whether from grants or TA, without which they would not have been able to improve their businesses on their own account. 4.4.3 Recommendations • Zrda should consider expanding its activities in Kvemo-Kartli and Samtskhe-Javakheti, where evidence suggests that many villages appear to be underserved and below Zrda’s radar. This applies particularly to agriculture where the inability to prosper can be attributed to issues that could potentially be addressed by the activity with minimal disruption to its planned activities for the remaining LOP. • In those villages where progress has been made in improving production and household incomes, Zrda should investigate options for public-private collaborations to establish processing units. This could add to farmers’ potential income while reducing food wastage. 4.5 EQ 5: SHOULD ZRDA PROVIDE CONTINUED SUPPORT TO DMOS AS A MEANS TO SUPPORT THE SUSTAINABLE DEVELOPMENT OF GEORGIA’S TOURISM SECTOR AND WHY? WHAT TYPE OF SUPPORT, IF ANY, CONSIDERING SPECIFIC ORGANIZATIONAL NEEDS OF EACH DMO? 4.5.1 Findings Evidence from Zrda’s reporting and feedback received from key stakeholders confirmed that Zrda facilitated the creation of DMOs in the Samtskhe-Javakheti and Samegrelo-Zemo Svaneti regions. The DMOs will develop projects and ideas, while the GNTA will review them and make a decision about their financing. The ET acknowledges that creating a fully functioning DMO in Zrda’s intervention areas is a long￾term prospect, given the need for all parties concerned in creating the organization to collaborate fully in its establishment and operational modalities. The in-person interviews conducted with key stakeholders indicated that DMOs are still in the process of development; they are understaffed, lack operational procedures, and lack clear roles and responsibilities. The ET found two types of official documents related to the Samtskhe-Javakheti and Samegrelo-Zemo Svaneti DMOs’ functioning: the DMO’s charter and the relevant extracts from the National Agency of Public Registry (NAPR). 23 Table 2: Founders of DMOs in the Samtskhe-Javakheti and Samegrelo-Zemo Svaneti Regions Name Founders Samtskhe-Javakheti DMO Aspindza Municipality, Adigeni Municipality, Ninotsminda Municipality, Borjomi Municipality, Akhaltsikhe Municipality, and Akhalkalaki Municipality22 Samegrelo-Zemo Svaneti DMO Abasha Municipality, Zugdidi Municipality, Martvili Municipality, Mestia Municipality, Senaki Municipality, Chkorotskhu Municipality, Tsalenjikha Municipality, Khobi Municipality, and Poti Municipality23 The charter stipulates the generic mission of the DMO (as well as its governing bodies) is to support tourism development in the respective regions, although no document exists to outline the implementation strategies to achieve this mission. Zrda started actively assisting DMOs in 2019. Pursuant to the charter, the DMO’s membership is open to all public or private organizations, NGOs, and individual members who are eligible to take part in managing the organization. Local municipalities lack strategic planning so there are no statistical data at the municipality level. DMOs are expected to conduct thorough analyses and help generate such data. The Samegrelo-Zemo Svaneti DMO was founded in December 2018 but did not start operating until April 2019. No MoU or other formalized relationship with the governor’s office and Zrda and has been developed; however, a MoU was signed between Zrda and the DMO. As part of the agreement, Zrda purchased furniture, equipment, and appliances. In addition, it funded the website and the brand book. Zrda also funded a foreign expert consultant who prepared a general strategy for the Samegrelo￾Zemo Svaneti region in October-December 2018. Currently, a Georgian expert is working on the Samegrelo strategy and action plan. With her involvement, the strategy and action plan for the DMO is also being developed. Another DMO is being created for the Upper Svaneti and other high mountain regions. According to the KIIs with the municipality council, although recently created, the DMO in Samegrelo is considered very important in positioning the region and the Zugdidi municipality as a tourist destination. Zrda prepared a detailed analysis, taking into consideration the positioning of the region and the Zugdidi municipality. Based on this analysis, Zrda suggested a number of activities for implementation. In addition, it purchased furniture and computers. Some of the most important interventions that Zrda and the DMO supported include: • Numbering streets and houses in Ganmukhuri and Anaklia; • Making hotel and street directions signs; • Placing guesthouses on the e-platforms (booking, Airbnb); • Conducting trainings on posting Ganmukhuri hotels on the e-platforms; • Providing bathrooms; and • Assisting with the preparation of projects for produce in Georgia. Zrda will also help to train staff to open a botanical garden and create a children’s playground. 22 Based on the charter of the Samtskhe-Javakheti DMO. 23 Based on the extract from the Register of Non-Entrepreneurial (Non-Profit) Legal Entity Samegrelo-Zemo Svaneti DMO. 24 When asked about Zrda’s role, KIs stated that Zrda provides significant support to the development of local economy. Besides TA, the activity provides support via equipment purchase, which is considered very important. Notable completed projects include: • Purchased of pavilions during one of the festivals (JemFest), which are still being used; • Purchased canoes for birdwatchers and development of a support infrastructure for the canoes; and • Provided health and safety trainings, which become mandatory by the law as of September 2019. Overall DMOs face two major issues: a lack of qualified human resources (HR) and budgeting constraints. All participants interviewed stated that it is difficult to find highly qualified staff to manage DMOs; for example, in Samtskhe-Javakheti it was hard to recruit the head of the DMO. KIIs with three DMO staff noted that they have no knowledge of how to run the DMO. While the DMOs have furniture and equipment, they do not have adequate funding for the upcoming activities. The involvement of the private sector in the management of the DMO is considered highly important, but staff are not ready to engage the private sector in DMO management. Local officials interviewed said that “they have high expectations for DMOs.” In Zugdidi municipality, they even closed the tourism department after the DMO’s establishment. When asked whether Zrda should continue to support DMOs, KIs uniformly agreed. According to them, “Losing Zrda’s support or replacing it with another project/donor will create discomfort for the municipality as it will take time to get the new partner familiar with what is required.” During its investigations, the ET became aware of the Tourism Cooperation Forum 2017 “Sustainable Mountain Resort Development,” which was organized by GIZ Georgia in the Gudauri mountain resort in northern Georgia with 75 carefully-selected, invited representatives from the Georgian tourism industry private sector (accommodation, transport, guiding, food, and adventure activities providers), national and local government and tourism agencies, and international donors. The event discussed the role of DMOs, among other tourism-related issues. The outcome of the meeting was general agreement among participants that regional DMOs would provide the much-needed leadership and vision for long-term sustainable tourism development, provided the roles and responsibilities of individual DMO members to oversee and implement tourism are clearly defined and that they are appropriately funded. One of the event organizers had recently run a similar event in Poland (in 2019). This event, a practical workshop in Warsaw called “Challenges, Barriers, and Opportunities for Responsible Tourism,” was organized by the Polish Tourism Board and was aimed at the management and staff of the regional and local tourism boards, local government, and DMOs. Key challenges that were identified included: 1) the lack of trust, cooperation, and communication between stakeholders, both between the private and public sectors as well as between tourism businesses; and 2) the lack of long-term vision and responsible tourism strategy, with the lack of leadership in implementing tourism in destinations. This is a common problem of destinations with no effective DMO to oversee the strategic development. Lessons learned from these events in Georgia and Poland clearly suggest similar problems regarding tourism development irrespective of how economically well-developed those countries are. To date, Zrda has made a significant contribution in supporting the development of the DMOs within its sphere of influence, although private sector involvement appears limited thus far. 25 However, now that steps are in place with the activity’s capacity-building measures, some of the issues mentioned above that could inhibit DMO growth could well be alleviated given the lessons learned above and how to ensure they are not repeated in Georgia. A recently completed project in Romania (September 2014 – June 2019), funded by the Romanian Environmental Partnership Foundation and the Romanian-American Foundation, created seven ecotourism destinations recognized by the Romanian National Authority for Tourism (ANT). Each of the ecotourism destinations is coordinated by a DMO, which in turn belongs to a DMO network coordinated by the Association of Ecotourism in Romania (AER), which itself becomes a permanent partner for ANT in developing and promoting ecotourism in Romania. Each local DMO: • Is recognized and treated as the destination manager by all local public and private stakeholders; • Is able to plan, coordinate, and implement local ecotourism policy; and • Runs on a functional financial mechanism. This is a specific approach to tourism development and may be worth following-up by Zrda/USAID. AER is partly funded by USAID, GIZ, EU, Norway Grants, and the Romanian￾American Foundation. 4.5.2 Conclusions The two Zrda-supported DMOs in the Samtskhe-Javakheti and Samegrelo-Zemo Svaneti regions visited by the ET are newly established. As such, they are not fully operational yet. The DMO in Samegrelo is considered to be highly important in supporting local tourism. Through this DMO, Zrda has helped the numbering of streets and houses and erecting street signs. These interventions are considered as critical for improvement of tourism in the area. The existing business model of DMOs established by GNTA with Zrda’s support in the Samtskhe￾Javakheti and Samegrelo-Zemo Svaneti regions is not sustainable unless significant modifications are made to advance its current institutional model and attract private sector interest and investments. Zrda has contributed substantial efforts to prepare their establishment, and its continued support is perceived as critical for their sustainability. DMOs established in the Samtskhe-Javakheti and Samegrelo-Zemo Svaneti regions do not currently demonstrate sufficient capacity (institutional, HR, and financial) to produce the technical products outlined by the United Nations World Tourism Organization (UNWTO), namely: 1. Quality standards in tourism services (improve the overall quality of products and services within the tourism industry, raise the levels of demand at local and international market; promote completeness within the industry); 2. Statistical strengthening and development of a sector (facilitate the understanding of the full social and economic dimension of tourism through the prism of the demand and supply side of the market); 3. Product development and diversification (increase the competitiveness of the tourism sector by evaluating the scope and diversity of existing and potential tourism resources); 4. Marketing and promotion plan development (formulate a detailed marketing and promotional plan for the sector considering the quality and quantity of tourism products); 26 5. Value chain analysis and local economic development (assess the structure of the tourism value chain and the economic value generated, and plan and execute actions to enhance the local economic impact of tourism); and 6. Support for PPPs through identifying and clarifying the present and future roles of the public and private sector organizations in tourism development of the country and making improvements on operations of tourism associations. 4.5.3 Recommendations • In order for the DMOs to justify receiving continued activity support, Zrda should address two major issues: 1) the need for a significantly wider dissemination of information regarding the purpose of DMOs to private sector operators in tourism, in order for them to fully grasp the value and benefit of the DMOs and why they should cooperate with such an organization; and 2) the need for continued support for DMOs to develop their strategic positioning in the tourism sector. • In providing support to DMOs, Zrda, should at least take into consideration the UNWTO Guidelines for Institutional Strengthening of DMOs.24 At the same time, Zrda should continue to finance activities related to the DMOs’ strategies and/or implementation of existing regional and municipal development plans and actively support the DMOs acquire different sources of income particularly from private sector stakeholders. • Zrda should consider supporting schemes developing the institutional capacity of DMOs. Specifically, Zrda should continue to assist DMOs to be supporters of 1) establishing networks for supporting the economic development of the region, 2) developing the tourism value chain, 3) supporting tourism-focused MSMEs and relevant local and national-level government stakeholders, and 4) developing tourism products, assets, and destinations. • Zrda should help DMOs, together with private sector, support develop agri-tourism, including wine tourism; select potential touristic sites; develop tourism routes; mark the routes; and help DMOs disseminate information about such tours and sites. Zrda should continue supporting DMOs—through finance creating touristic products such as festivals reflected in the respective DMOs municipal/regional plans—to prepare and implement rules for the organization of different festivals and events. • Zrda should coordinate its tourism-related DMO initiatives with other donors and development partners (UNDP, World Bank, GIZ, among others) in order to assess the extent of project synergies that could facilitate DMO development and sustainability. 24 https://www.e-unwto.org/doi/pdf/10.18111/9789284420841?download=true. 27 4.6 EQ 6: WHICH OF THE FOLLOWING ACTIVITIES DO ZRDA BENEFICIARIES AND STAKEHOLDERS IDENTIFY AS PROFIT CENTERS OR KEYS TO FUTURE BUSINESS GROWTH? ACTIVITIES INCLUDE: CERTIFICATION PROGRAMS IN HOSPITALITY SECTOR; DISCRETE TRAINING INTERVENTIONS IN TOURISM; DIGITAL MARKETING— BOOKING.COM, TRIPADVISOR, AND DEVELOPMENT OF WEBSITES? 4.6.1 Findings 4.6.1.1 General Observations The ET found that the most meaningful responses to this question resulted from FGDs undertaken in Zrda’s six intervention regions. Zrda was tasked with providing a range of self￾improvement and advisory training programs for beneficiaries to be carried out either directly by activity team members or by IPs. These activities were implemented under Zrda’s four sub￾purposes and aimed at upgrading the skills and knowledge of beneficiaries. FGD participants understood the concept of “keys to growth” but had little or no understanding of what a “profit center” was or how it could impact their businesses. Consequently, the ET redefined its question to ask respondents what actions they believed were still needed to help them increase profits. The same applied to those beneficiaries with whom the ET conducted site visits. IPs and business service providers were more aware of the meaning of “profit center” but, according to them, the issue was not raised with clients as the focus of their interventions was on the provision of self￾improvement measures leading to knowledge improvement and increased expertise in their particular fields of interest. Zrda training events were mainly delivered to invited audiences with similar needs and aspirations. These are highlighted below followed by comments from FGD participants who attended the trainings mentioned. Under Sub-Purpose 1, Zrda conducted a food safety training in collaboration with private sector partner BP in Akhaltsikhe and Tsalka Municipalities. Akhaltsikhe FGD participants who attended the training indicated that it was useful and improved their knowledge regarding public catering business requirements and waste management in line with government legislation. Zrda also initiated a value chain financing scheme in Shida Kartli facilitating a partnership between MFO Crystal and Agro-Com, an FSC operating in Shida Kartli region, with interest-free loans to farmers. As the ET did not encounter any beneficiary who benefited from this initiative, the team is unable to comment. However, the ET recognizes this as a pilot project, which will take time to become fully operational and disseminated to a wider audience. Zrda has also taken steps to increase the information and communications technology (ICT) knowledge and skills of targeted beneficiaries, particularly in the tourism sector. In Zugdidi, all FGD participants attended a two-day training session on online reservations and how to operate the Booking.com and Expedia.com platforms. They also received a detailed step￾by-step guidance manual on the subject. This training was highly appreciated by participants who noted that following the training, a number of guesthouses were created based on the knowledge received on online booking and digital marketing. All received certificates after attending the training and claimed that these certificates helped to attract more tourists. The only concern raised was that criteria to qualify for Zrda grants were too restrictive (one only qualified if he/she had 10 rooms in the guesthouse, according to participants). As borrowing from traditional banks was never an option due to high interest rates, participants believed this was hindering development of local tourism. Under its Restaurant Service Improvement Activity, Zrda- 28 contracted service provider Gastronaut Ltd. carried out eight training events on food costing and pricing in Gori, Zugdidi, Borjomi, and Akhaltsikhe. The training was aimed at catering/accommodation business owners to upgrade their services, generate more income, and improve profitability. FGD participants who attended the trainings uniformly agreed that they were most helpful and provided them with much needed information on how to manage their businesses more effectively (mainly restaurants and cafes). Under Sub-Purpose 2, Zrda is in the process of developing job and competence frameworks for tour guides with the local service provider CTC and also with local VETs. Zrda issued a tender on designing and delivering a short-term certificate program for tour operators/guides. CTC won the contract and should finish this activity by the end of 2019. This certification program is launched on the basis of existing vocational education (VE) colleges. At the moment, CTC is preparing the training of trainers (ToT) sessions in VE colleges and is recruiting participants. CTC signed a MoU with four colleges in (Kakheti, Samtskhe-Javakheti, and Samegrelo) and is now in the process of finalizing the training materials. Zrda engaged an international consultant (an experienced practicing guide) to provide input to the training materials. The consultant will also be delivering training sessions of the ToT component. The overall goal is to inspire participants’ strategic and creative thinking about what can be done at local level for developing new products and services, for example, new thematic tours. Regional educational organizations—such as Zugdidi State Teaching University, Kakheti VET “Aisi” (Akhmeta branch), Gori VET “Gantiadi,” and Akhaltsikhe VET “Opizari”—will be engaged to deliver short training courses. This is a work-in-progress for which the ET has no direct feedback from interested parties other than it is a necessary step forward in attracting and training competent tour guides, especially in Georgia’s touristic regions. Zrda also supports artisans working with handicrafts under Sub-Purpose 3; this is reflected in Zrda’s collaboration with the Georgia Heritage Crafts Association (GHCA), which has conducted business skills development trainings in the Samegrelo region. This initiative, together with the planned pilot program “Artisan as a Destination” in Samegrelo, will support artisans to develop and design packages and marketing and promotional material for their items. The ET did not obtain any direct feedback from either KIIs or FGDs on this initiative possibly due to it being in the early stages of development. The ET did not note or encounter any specific activities under Sub-Purpose 4 that pertained to this question so none are offered. However, data on FGDs in Annex 5 set out participants responses in relation to the “keys to growth” in particular. 4.6.2 Conclusions Evidence gathered from FGD project beneficiaries clearly indicates acknowledgement and overall satisfaction that Zrda’s various interventions aimed at improving and upgrading their knowledge and skills in operating tourism-related businesses have led to increased business and profit margins. Quality issues have been addressed and self-improvement initiatives, such as acquiring certification, have been motivating factors for small business owners to improve performance and increase the opportunity to attract more customers, who often look to comments online (e.g., on TripAdvisor) before booking or considering visiting a tourism-related business. Zrda’s collaboration with private service providers to deliver trainings and provide advice on the importance of offering quality services has been a successful feature of the project’s outreach activities. 29 4.6.3 Recommendations • Zrda should continue with its capacity-building activities and initiatives with private sector business service providers to enhance beneficiaries’ capacity, particularly in tourism, to improve their customer service quality. • Zrda should build on previous interventions to build capacity within the tourism sector to market its attractions and services to a wider consumer audience. Marketing is vital in the tourism industry, which is highly competitive and sensitive to customer perceptions of destinations and the quality of services offered. The development of new and interesting products at competitive prices is also important—the promotion and marketing of which can be significantly supported by the DMOs recently established by municipalities with Zrda support. Highlighting the advantages of particular regions to discerning tourists will play a major role in increasing tourism development. • Zrda should continue to offer grants and TA in the regions of its activity where it can make a lasting and sustainable contribution to the sector. This recommendation comes with the caveat that marketing should be integrated with interventions to improve service quality and provide improved/increased information and amenities for tourists so that marketing efforts do not outpace the capacity of the tourism sector to provide quality tourism experiences. 30 ANNEXES 31 ANNEX 1: STATEMENT OF WORK 32 AID-SOL-OAA-14-000054 / 720-114-19-F-00005 SECTION C – DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK STATEMENT OF WORK MID-TERM PERFORMANCE EVALUATION OF USAID’s ZRDA ACTIVITY IN GEORGIA 1. EVALUATION PURPOSE AND USE Award Number CA No. AID-l 14-A-16-00004 03/15/2016- Award Dates: 03/14/2021 Funding: $19,700,000 Implementing Organization: Chemonics International Inc. Contracts Officer Representative (COR): Philip Greene Alternate Contract Officer Representative (Alternate COR): David Tsiklauri The evaluation Contractor must provide non-personal services for a mid-term performance evaluation of the Zrda Activity in Georgia (Zrda) program. The results of the evaluation will be used by USAID/Georgia for improving ongoing interventions in the areas of rural development, tourism sector development, and public-private partnerships. The primary audience of the evaluation will be USAID, and in particular Economic Growth (EG) office and the implementing partner (Chemonics International Inc.). The results of the study will be shared with local stakeholders (Ministry of Environmental Protection and Agriculture, Ministry of Economic Development, National Tourism Agency, partner NGOs, etc.,) and other donors working in this area. 2. SUMMARY OF SPECIFIC TECHNICAL REQUIREMENTS The evaluation Contractor must ensure that the evaluation team will complete the following tasks and provide the following deliverables within the terms defined by the contract: • Telecon with USAID/Georgia to discuss the upcoming work. • Provide a draft evaluation design and work plan (including meeting schedules and data collection instruments) to USAID for review and comment two weeks prior to the evaluation team’s arrival in country. • Incoming briefing with USAID management to present the detailed evaluation design. • Conduct the evaluation in accordance with the USAID-approved evaluation design and work plan. • Co-creation session with USAID Mission to formulate/refine evaluation recommendations. • Outgoing briefing with USAID management to present the matrix of preliminary findings/conclusions/recommendations of the Evaluation. 33 • Provide an evaluation report to USAID in accordance with Reporting Guidelines under Section 9 – Deliverables. The evaluation report should follow the “Criteria to Ensure the Quality of the Evaluation Report” of the USAID Evaluation Policy. • Submit USAID-approved evaluation report to Development Experience Clearinghouse (DEC) within 30 calendar days following the acceptance of the report by the USAID Contracting Officer Representative (COR.) • Submit quantitative dataset in a machine-readable format to the Development Data Library • (DDL) as per USAID’s Open Data Policy. 3. BACKGROUND On March 15, 2016 USAlD/Georgia signed Cooperative Agreement No. AlD-1l4-A-16-00004 with Chemonics International to implement the five-year Zrda Activity in Georgia. The Zrda Activity promotes inclusive and sustainable economic growth in target regions by promoting Micro, Small, and Medium Enterprises’ (MSMEs) growth, increasing productivity of rural households, facilitating market linkages between producers and buyers, and promoting local economic development by establishing and strengthening networks. Zrda works in 81 communities in proximity to the administrative boundary lines with Georgia’s breakaway regions of Abkhazia and South Ossetia and communities with ethnic minority populations. Zrda will create 2,400 jobs, increase sales for 800 MSMEs, increase incomes of 13,200 households, and generate measurable improvements in community resilience and economic conditions. Zrda uses a sector-driven approach to drive implementation, focusing on high potential value chains in agriculture and tourism. In agriculture, Zrda focuses on high value fruits, vegetables, berries, honey, and other bee products. In tourism, it focuses on rural tourism and development of key sub­sectors that include adventure tourism, eco-tourism, agro-tourism, food and wine tourism, and cultural heritage tourism. Zrda promotes innovation and entrepreneurship in partnership with Georgia’s Innovation and Technology Agency (GITA) and other public and private sector partners. Zrda's approach integrates the entire market system—households, MSMEs, sectoral associations, civil society, and government institutions into networks that generate economic growth, facilitate trust building, and promote resilience. Zrda’s approach to tourism value chain development in Georgia seeks to address identified constraints, specifically by supporting distribution, lodging, excursions, and tourism assets and destinations development, with a focus on target regions. Zrda’s interventions focus on three main pillars: • Product Development & Marketing: Development and promotion of new touristic routes, printed and electronic materials (guides, cultural products—such as cookbooks), handicrafts, destinations/excursions/activities (villages, intangible and tangible heritage sites, museums, agro/eco/adventure sites/activities), festivals and fairs, etc. • Training and Capacity Building: Working with local tourism stakeholders and businesses to improve business performance, quality of service provision, and marketing. Focus areas for training and capacity building are small and medium-sized hotels and rural guesthouses, tour guides, tour operators, restaurants, artisans, national parks, museums, cultural heritage monument operators, and others as opportunities arise. 34 • Regional Coordination, Promotion, & Destination Management: Zrda, in close collaboration with the Georgia National Tourism Agency (GNTA), supports the establishment of regional Destination Management Organizations (DMOs) in Samtskhe￾Javakheti and Samegrelo, supports capacity building of GNTA-coordinated Tourism Information Centers (TICs) in Zrda target regions, and promotes public-private partnerships in tourism. As reported by the Activity, by the end of FY 2018 , new sales generated by Zrda-supported MSMEs reached $11 million, exceeding the life of program target by 31 percent; 689 new jobs were created; $2.2 million was leveraged from program partners; and the number of beneficiaries reached 43,700 individuals. Zrda’s quarterly and annual reports are available at: https://decsearch.usaid.gov (https://pdf.usaid.gov/pdf_docs/PAOOSYC7.pdf; http s:// pdf.usaid.gov/pdf_docs/PAOOSYC5.pdf; https://pdf.usaid.gov/pdf docs/PAOOSVC6.pdf) Two key private sector partnerships were secured during FY 2018 with British Petroleum (BP) and the Anaklia Development Consortium (ADC). Intention of both partnerships is to engage and leverage private sector resources toward community development. 4. KEY EVALUATION QUESTIONS TO BE ADDRESSED The purpose of this mid-term performance evaluation is to determine the relevance (the extent to which the aid activity is suited to the priorities and policies of the target group, offeror, and donor) of Zrda’s programmatic approaches in achieving intended life-of-program results and to provide recommendations on corrective actions and new directions for the remaining years of program implementation. The Contractor must review actual progress toward achieving key expected results and identify accomplishments, delays, challenges, and their impact on achieving Zrda’s Iife-of­program targets. Evaluation shall address the following specific questions: 1. What approaches have been most or least successful in creating jobs and increasing incomes in target communities and why? 2. Do interventions targeted at and/or implemented in partnership with larger companies result in creation of more jobs than interventions involving smaller companies and why? 3. What are the key lessons learned from implementing public-private partnerships under Zrda? 4. Have Zrda’s sub-awards and technical assistance activities struck optimal balance between supporting minority and non-minority communities? Does Zrda sufficiently integrate and benefit minority communities? 5. Should Zrda provide continued support to Destination Management Organizations (DMO) as a means to promote the sustainable development of Georgia’s tourism sector and why? What type of support, if any, is needed to make DMOs operational sustainable by the end of the Zrda program, considering specific organizational needs of each DMO? 6. Which of the following activities do Zrda beneficiaries and stakeholders identify as profit centers or keys to future business growth? These activities include certification programs in the hospitality sector, discrete training interventions in the tourism sector, and digital marketing activities through booking.com, TripAdvisor, and development of websites. 35 5. METHODOLOGY The Mission is looking for the Contractor to suggest the best methods that minimize bias and provide strong evidence. The Contractor is expected to suggest the use of various data collection and analysis methods, both quantitative and qualitative, including document review, key informant interviews with project beneficiaries, the GOG, private sector, focus group discussions, survey instruments (if applicable), and others. The methodology for any evaluation process that involve the select ion of participants (e.g., surveys, focus groups, interviews) must be clearly explained and justified. For example, for a survey or mini-survey (if proposed), the number of respondents and their selection process should be explained and justified. The same is true for key informants, focus group discussions, and other methods as well. Selected respondents should be representative of women, youth, and vulnerable groups, where appropriate. The Contractor must conduct a desk review of Zrda-related documents, which will help identify areas that merit closer attention once the team begins its fieldwork. Reading materials will be available to the team prior to their arrival. The Contractor must develop a detailed evaluation design and a workplan, including data collection plan and drafts of data collection tools. A draft of the workplan and evaluation design must be shared with USAID COR two weeks prior to arrival for review and comment prior to the evaluation team’s in-country arrival. The plan will then be presented to the Mission during the in brief in more detail. The evaluation design must include the evaluation matrix (an illustrative evaluation matrix for this study is given below). The evaluation design must explain how the evaluation Contractor intends to conduct the study in detail, including a detailed description of one or more proposed methodologies as well as limitations of proposed methodologies. It must explain in detail what methods will be used to obtain answers for each evaluation question. The design must also explain how the proposed methodology (mix of methods) to conduct the study generates evidence to ensure rigor and reliability of results; and how and why the proposed methodology will minimize bias. The evaluation design must also include the data analysis plan for each question, draft questionnaires (to be included as an attachment), and other data collection instruments or their main features, criteria for assessing responses to evaluation questions, known limitations, and a dissemination plan. The evaluation design might also include specific sub￾questions for each evaluation question, where needed. Again, the methods described herein are only illustrative and USAID expects that the Contractor will suggest the best methods that would generate most reliable and evidence-based answers to the key evaluation questions. 6. EVALUATION TEAM The evaluation shall be conducted by an external evaluation team composed by international and local experts. The offeror should propose the composition of the evaluation teams. However, it is expected that the team will have at a minimum one expatriate Team Leader and an Evaluation Expert. The Team Leader (international) will have a minimum of 10 years of experience conducting evaluations and assessments including evaluations in democracy and governance, political processes, and elections programming. The expert should have at least master’s level education in the field of public administration, management, political science, law, or other relevant field. Experience in Georgia and/or in the Europe and Eurasia region will be an advantage but is not required. Fluency in English language is required. The team leader will be responsible for the day 36 to day management of the team, data collection and synthesis, presentations and submissions of draft and final reports. Evaluation Expert (international) will have a minimum of seven years of experience conducting evaluations and assessments using various data collection and analysis methodologies preferably in the democracy sector. The expert should have extensive experience in statistics and statistical and social science research and methodologies, should be a key person in developing evaluation design, including data collection and analysis plan. Experience in Georgia and/or in the Europe and Eurasia region will be an advantage but is not required. Fluency in English in required. Locally hired expert/consultant/s with a minimum of five years of experience in the democracy and governance sector as well as experience participating as a team member in a USAID or other donor-funded project assessments and/or evaluations. Experience in election programming with be an advantage. English language knowledge is a requirement. All evaluation team members shall be familiar with USAID’s January 2011 Evaluation Policy.25 All team members are required to provide to USAID a signed statement attesting to a lack of conflict of interest in relation to the ITEP project being evaluated. 25 http:/www.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf Illustrative evaluation matrix: Research Question Data Source Methodology 1. What approaches have been most or least successful in creating jobs and increasing incomes in target communities and why? 2. Do interventions targeted at and/or implemented in partnership with larger companies result in creation of more jobs than interventions involving smaller companies and why ? 3. What are the key lessons learned from implementing public-private partnerships under Zrda? 4. Have Zrda’s sub-awards and technical assistance activities struck optimal balance between supporting minority and non-minority communities? Does Zrda sufficiently integrate and benefit minority communities? Activity documentation: program description, quarterly and annual reports, M&E plan, results framework, workplans, sub-awards documentation. • Project staff • Zrda beneficiaries including grantees, TA and training offerers, Community-based organizations, MSMEs • Employees of the Ministry of Environmental Protection and Agriculture; Ministry of Economic Development; Georgian National Tourism Agency; Georgia’s Innovation and Technology Agency; representatives of local governments of beneficiary municipalities; other relevant state agencies and ministries • Representatives of sub-contractors • Private sector resource partners • Academic institutions Document reviews (e.g., to compare achieved results and set benchmarks) Direct Observation/Key Informant Interviews/Focus Group Discussions and/or Mini￾Survey with identified data sources Research Question Data Source Methodology 5. Should Zrda provide continued support to Destination Management Organizations (DMO) as a means to promote the sustainable development of Georgia's tourism sector and why? What type of support, if any, is needed to make DMOs operationally sustainable by the end of the Zrda program, considering specific organizational needs of each DMO? 6. Which of the following activities do Zrda beneficiaries and stakeholders identify as profit centers or keys to future business growth? These activities include certification programs in the hospitality sector, discrete training interventions in the tourism sector, and digital marketing activities through booking.com, TripAdvisor, and development of websites. Activity documentation: program description, quarterly and annual reports, M&E plan, results framework, workplans, sub-awards documentation • Project staff • Zrda tourism beneficiaries including grantees, TA and training offerers, guesthouses, hotels, other MSMEs involved in tourism • Employees of the Ministry of Economic Development; Georgian National Tourism Agency; Georgia’s Innovation and Technology Agency; representatives of local governments of beneficiary municipalities; other relevant state agencies and ministries • Representatives of sub-contractors • Private sector resource partners • Academic institutions, consulting firms, and other training providers • DMO staff and Tourism Information Center staff • Sector experts Document reviews (e.g., to compare achieved results and set benchmarks) Direct Observation/Key Informant Interviews/Focus Group Discussions and/or Mini￾Survey with identified data sources 39 6. WORK LOCATION Tbilisi, Georgia’s regions and U.S. In order to conduct meetings and interviews, the Contractor must travel to Tbilisi and to some of Zrda­targeted communities located in the Samegrelo, Kvemo Kartli, Shida Kartli, Samtskhe￾Javakheti, and Mtskheta-Mtianeti Regions and Akhmeta municipalities. 7. EVALUATION SCHEDULE The following are the levels of effort of the staff which will be mobilized under this Task Order. Staff No. of Work Days in Country No. of Days for Preparation and Report Writing Total # of Work Days/Consultant International Technical Expert – Team Leader 20 20 40 Evaluation Specialist 20 20 40 Local Technical Expert 30 5 35 Translator/Logistics Specialist 24 0 24 A six-day work week will be authorized in Georgia with no premium pay. The tentative timeline of activities are provided below. Timing (Anticipated Month or Duration) Proposed Activities August 12 Initial teleconference with USA ID/Georgia August 13-August 29 Document review, preparation work and finalization of the evaluation design and workplan August 19 Submission of the draft workplan and evaluation design to USAID August 30 Submission of the final revised workplan/evaluation design to USAID August 31 Team leaves for Georgia September 2 Fieldwork begins September 4 In-brief with USAID September 23 Co-creation meeting with the USAlD Mission September 24 Out-brief with Mission, end of fieldwork, team departs Georgia September 26-October 11 Data analysis and report writing October 15 Submission of the draft evaluation report to USAID November 15 Submission of the revised evaluation report to USAID Note: As needed, there may be several rounds of review of the draft evaluation report by USAID prior to finalizing/approving the report 8. PROJECTS DOCUMENTS FOR REVIEW Relevant reports and other project documentation will be provided by the Mission to the Contractor prior to travel to Georgia. The evaluation contractor shall initiate Washington-based work by reading reports and familiarizing him/herself with the program. These documents are: • Program Description for the Zrda Program • Zrda Work Plan(s) • Zrda quarterly and annual reports • M&E plans and performance data tables 40 • Various program documentation (analytical documents, studies, grants manual, etc.) • Initial Iist of in-country contacts Reporting Guideline The illustrative format for the final evaluation report is as follows: 1. Executive Summary – summarizes key points, concisely states the purpose, background of the project, main evaluation questions, methods, findings, conclusions, recommendations, and any lessons learned; should be sufficiently detailed, yet brief, to serve as a stand-alone product (3-5 pp) 2. Introduction – state the purpose, audience, and outline of the evaluation (1 pp) 3. Background – provide a brief overview of the project and the study implemented (1-2 pp) 4. Methodology – the evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology. Greater detail can be included in the appendices (2-3 pp) 5. Findings/Conclusions/Recommendations – explicitly answer each evaluation question; the report should distinguish between findings (the facts), conclusions (interpretation of the facts), and recommendations (judgements related to possible future programming) (10-20 pp); however it should be clear what the link between them is. 6. Lessons Learned (if not covered in findings, conclusions and recommendations) (2-3 pp) 7. Annexes – annexes must include this statement of work and its modifications (if any); any “statements of differences” regarding significant unresolved difference in opinion by funders, implementers, and/or members of the evaluation team; a glossary of terms; sources of information, properly identified and listed; clear documentation of schedules, meetings, interviews and focus group discussions, and any tools used in conducting the evaluation, such as focus group scripts or questionnaires, checklists, and discussion guides used; and signed disclosures of conflict of interest. The evaluation design should also be attached to the report. The report format should be presented in Microsoft Word and use 12-point type font throughout the body of the report, using page margins I” top/bottom and left/right. The body of the report should ideally be within 20-35 pages, excluding the executive summary, table of contents, references, and annexes. The final report must follow USAID branding and marking requirements. Per the USAID evaluation policy, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report. • The evaluation report should represent a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the projects, what did not and why. • Evaluation reports shall address all evaluation questions included in the statement of work. • The evaluation report should include the statement of work as an annex. 41 • Evaluation methodology shall be explained in detail, and all tools used in conducting the evaluation such as questionnaires, checklists, and discussion guides will be included in an Annex in the final report. • Evaluation findings will assess outcomes and impact on males and females. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. • Sources of information shall be properly identified and listed in an annex. • Recommendations shall be supported by a specific set of findings. • Recommendations shall be action-oriented, practical, and specific, with defined responsibility for the action. 9. LOGISTICAL SUPPORT USAID/Georgia and Zrda Activity will provide the possible list of in-country contacts prior to the team’s arrival but will not assist in the logistics of appointing meetings. Hence, the Mission will not be responsible for arranging logistics for the evaluation team. The Contractor must suggest how they plan to arrange translation, transportation, and logistical support to the evaluation team. The COR, through USAID Zrda AOR, will put the Contractor in contact with Zrda implementing partner. While in Georgia, the Contractor will conduct meetings in Tbilisi. Some meetings will require travel to regions outside Tbilisi to meet with Zrda grant offerors and other beneficiaries, and NGO, private sector and government stakeholders. Zrda program implementing partner may assist with setting those meetings. 10. OTHER REQUIREMENTS The evaluation team must be familiar with USAID’s Human Subject Protection Policy and USAID’s Evaluation Policy (http://www.usaid.gov/evaluation). The evaluation team must provide adequate training for its survey staff on survey methodology, USAID’s survey regulations, other relevant regulations, and the data collection plan. The contractor has the responsibility to safeguard the rights and welfare of human subjects involved in the survey research supported by USAID. USAID has adopted the Common Federal Policy for the Protection of Human Subjects, Part 225 of Title 22 of the Code of Federal Regulations (http://www.usaid.gov/policy/ads/200/200mbe.pdf). Offeror organizations must familiarize themselves with the USAID policy and provide “assurance” that they will follow and abide by the procedures of the Policy. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline, need to be agreed upon in writing by the Contracting officer. 42 ANNEX 2: LIST OF MEETINGS AND INTERVIEWS 43 List of KIIs and FGDs for Zrda Evaluation Team # Contact Person Position Organization Location Date 1. Keti Chogovadze Program Development Specialist USAID/Georgia Tbilisi 4/9/19 2. Peter A. Wiebler Mission Director USAID/Georgia Tbilisi 4/9/19 3. Phillip Greene Senior Economist & Agreement Officer Representative (AOR) – Zrda Program USAID/Georgia Tbilisi 4/9/19 4. Rebecca Black Advisor – Program Office USAID/Georgia Tbilisi 4/9/19 5. Lauren K. Russell Deputy Director (Economic Growth) USAID/Georgia Tbilisi 4/9/19 6. Sharon Valentine Director Economic Growth Office USAID/Georgia Tbilisi 4/9/19 7. Austin Spenzer Development Associate USAID/Georgia Tbilisi 4/9/19 8. Olivia Najedlik Development Associate USAID/Georgia Tbilisi 4/9/19 9. David Tsiklauri Contract Officer Representative (COR) USAID/Georgia Tbilisi 4/9/19 10. Mariam Lashkhi Deputy Chairperson GITA Tbilisi 5/9/19 11. Revaz Javelidze Networking/Youth Engagement Team Leader Association Rural Development for Future Georgia Tbilisi 5/9/19 12. Rusudan Mamatsashvili First Deputy Head GNTA Tbilisi 5/9/19 13. Sulkhan Jorbenadze Crystal Fund Coordinator Agro Products Tbilisi 5/9/19 14. Mark McCord Chief of Party (COP) Economic Security Project (USAID) Tbilisi 6/9/19 15. Ketevan Sharabidze Deputy Project Director of AMMAR International Fund for Agriculture Development (IFAD) Tbilisi 6/9/19 16. Anzor Mantskava ADC Chief Operations Officer/Regional Director ADC Tbilisi 6/9/19 17. Irina Khantadze Executive Director CTC Tbilisi 6/9/19 18. Constantine Chanturia Head of Risk Management and Analytical Service JSC Partnership Fund Tbilisi 7/9/19 19. Zaur Dargali Mayor Marneuli Municipality Marneuli 9/9/19 20. Nino Dekanoidze Head of Administration Marneuli Municipality Marneuli 9/9/19 21. Marneuli Municipality staff (FGD) Local government (LG) officials Marneuli Municipality Marneuli 9/9/19 22. Bolnisi Municipality staff + farmers (FGD) LG officials/Local farmers - Bolnisi 9/9/19 44 List of KIIs and FGDs for Zrda Evaluation Team 23. Bolnisi farmers (FGD)/Zrda beneficiaries Local farmers including Zrda beneficiaries Farmers’ Service Center Bolnisi 9/9/19 24. Valentina Naveriani Deputy Mayor Bolnisi Municipality Bolnisi 9/9/19 25. Family-type hostels (FGD) Representatives of family-type hostels - Zugdidi 9/9/19 26. Culinary group (FGD) Representatives of the culinary group - Zugdidi 9/9/19 27. Zugdidi Municipality (FGD) LG officials Zugdidi Municipality Zugdidi 9/9/19 28. Zugdidi DMO (FGD) Representatives of Zugdidi DMO Zugdidi DMO Zugdidi 9/9/19 29. Lika Akhalaia Representative of the Mayor in Ganmukhuri Municipality of Ganmukhuri Ganmukhuri 10/9/19 30. Nugzar Khufenia Bicycle business owner - Ganmukhuri 10/9/19 31. Nukri Gabelaia Guesthouse owner, Ganmukhuri - Ganmukhuri 10/9/19 32. Armen Amirkhanian Representative “Agro Javakheti” – Rural Advisory Center Ninotsminda 10/9/19 33. Ninotsminda Municipality staff (FGD) LG officials Ninotsminda Municipality Ninotsminda 10/9/19 34. Beneficiaries from Ninotsminda (FGD) Tourism + Agriculture - Ninotsminda 10/9/19 35. Vahe Iritsyan Greenhouse owner - Ninotsminda 10/9/19 36. Guram Zedginidze (site visit) Beekeeper - Aspindza Municipality, Nakalakevi Community 10/9/19 37. Anuki Asatiani Information Officer Technopark Zugdidi 11/9/19 38. Ms. Guliada Arkania Head of the Association Association For Agriculture Development (AFAD) Zugdidi 11/9/19 39. Mrs. Guliada Arkania Greenhouse owner - Shamgona village (Zugdidi municipality) 11/9/19 45 List of KIIs and FGDs for Zrda Evaluation Team 40. Luter Toria Blueberry plantation owner - Darcheli village (Zugdidi municipality) 11/9/19 41. Zurab Sadatierashvili Agricultural Specialist Rural Consultancy Center (RAS) Akhaltsikhe Akhaltsikhe 11/9/19 42. Beneficiaries from Akhaltsikhe (FGD) Tourism + Agriculture - Akhaltsikhe 11/9/19 43. Ruben Karapetyan Deputy Mayor Akhaltsikhe Municipality Akhaltsikhe 11/9/19 44. Zaza Melikidze Governor Akhaltsikhe Municipality Akhaltsikhe 11/9/19 45. Beneficiaries from Borjomi (FGD) Tourism Borjomi Municipality Borjomi 11/9/19 46. Davit Zaalishvili First Deputy Mayor Borjomi Municipality Borjomi 11/9/19 47. USAID Zrda Borjomi staff USAID Zrda Borjomi staff USAID Zrda Borjomi 11/9/19 48. Rusudan Kalichava Guesthouse owner - Zugdidi 12/9/19 49. Nobel Gvasalia Guesthouse owner - Anaklia 12/9/19 50. Mr. Murad Shakladze Strawberry plantation - Darcheli 12/9/19 51. Gela Lemonjava Representative of the Mayor of Zugdidi in Anaklia Municipality of Anaklia Anaklia 12/9/19 52. Konstantine Tavzarashvili Mayor Gori Municipality City Hall Gori 12/9/19 53. Beneficiaries from Gori (FGD) Agriculture - Gori 12/9/19 54. Beneficiaries from Gori (FGD) Tourism/Service - Gori 12/9/19 55. Gelutashvili Family (beneficiaries) Farmers (cold storage) - Dirbi 12/9/19 56. Kakha Zerekidze (beneficiary) Farmer (tractor) - Tergvisi 12/9/19 57. Levan Baghaturia Regional Director Khareba LTD Tsalenjikha, Jvari 13/9/19 58. Domitri Logua Chief Beekeeper Khareba LTD Tsalenjikha, Jvari 13/9/19 59. Irakli Kalandia Information Technology (IT) business - Zugdidi 13/9/19 60. Mamuka Chkhapelia Greenhouse owner - Mujava, Tsalenjikha 13/9/19 46 List of KIIs and FGDs for Zrda Evaluation Team 61. Khatia Tsilosani Deputy Minister Ministry of Environmental Protection & Agriculture Tbilisi 14/9/19 62. Natalia Bakhtadze Co-Founder Georgian Eco-Tourism Association Tbilisi 16/9/19 63. Mzia Giorgobiani Deputy Minister Ministry of Regional Development & Infrastructure of Georgia Tbilisi 16/9/19 64. Nino Kvernadze Head of Infrastructure Policy and Development Partners Relations Department Ministry of Regional Development & Infrastructure Tbilisi 16/9/19 65. Maka Berdzenishvili Chief Specialist at the Division for Infrastructure Policy Ministry of Regional Development & Infrastructure Tbilisi 16/9/19 66. Irina Mania Director Georgian Heritage Crafts Association Tbilisi 16/9/19 67. Rusudan Medzmariashvili Social Responsibility Manager BP Georgia Tbilisi 16/9/19 68. Eteri Kvirikashvili Community Development Initiative Adviser BP Georgia Tbilisi 16/9/19 69. Tornike Kapanadze Project Manager Agriculture and Rural Development Agency (ARDA) (former APMA) Tbilisi 17/9/19 70. Giorgi Noniashvili General Director Georgian Farmers Distribution Company Tbilisi 17/9/19 71. Manana Vardzelashvili Chief Coordinator at the International Relations Unit National Agency for Heritage Protection of Georgia Tbilisi 17/9/19 72. Levan Batiashvili Senior Advisor Public Private Center Georgia Tbilisi 18/9/19 73. Nino Zambakhidze Head Georgian Farmers’ Association (GFA) Tbilisi 18/9/19 74. Tamar Gavasheli Interim Head of the Agency IDP Livelihood Agency Tbilisi 18/9/19 75. Mariam Jorjadze Director ELKANA Biological Farming Association Tbilisi 18/9/19 76. Nodar Kereselidze Improving Rural Development in Georgia – National Project Manager UNDP Georgia/ENPARD Tbilisi 19/9/19 77. Giorgi Nanobashvili Economic Development Team Leader UNDP Georgia Tbilisi 19/9/19 47 List of KIIs and FGDs for Zrda Evaluation Team 78. Shota Brelidze Head of TIC Department Georgian National Tourism Administration Tbilisi 19/9/19 79. Natia Meparishvili Chairwoman, Founder Women Business Council in Georgia (WBCG) Tbilisi 19/9/19 80. Nikoloz Sakvarelidze Program Director RDA Tbilisi 19/9/19 48 ANNEX 3: LIST OF DOCUMENTS CONSULTED 49 Title of Document Date 1. Year 1 Annual Implementation Plan: March 16 – March 17 May 2016 2. Year 1 Activity Monitoring and Evaluation Plan (AMEP) May 2016 3. FY 2017 Annual Implementation Plan October 2016 – September 2017 (Adjusted) September 2016 4. FY 2018 Annual Implementation Plan: October 2017 – September 2018 (Adjusted Project Year 3) – Final Revised Version November 2017 September 2017 5. FY 2019 Annual Implementation Plan: October 2018 – September 2019 October 2018 (revised) 6. FY 2017 Annual Report (Quarterly Report No. 7) October 2017 7. FY 2018 Annual Report October 2018 8. Quarterly Report No. 4: Q1, FY 2017 January 2017 9. Quarterly Report No. 5: Q2, FY 2017 April 2017 10. Quarterly Report No. 6: Q3, FY 2017 July 2017 11. Quarterly Report No. 8: Q1, FY 2018 January 2018 12. Quarterly Report No. 9: Q2, FY 2018 April 2018 13. Quarterly Report No. 10: Q3, FY 2018 July 2018 14. FY 2019 Quarterly Report (N12) January 2019 15. FY 2019 Quarterly Report 11 (N13) April 2019 16. FY 2019 Quarterly Report 111 (N14) July 2019 17. Household Survey and ZRDA Resilience Index Data Collection Report February 2018 18. ZRDA Resilience Index Data Collection Report – Version 1 January 2018 19. Household Survey and ZRDA Resilience Index Data Collection Mid￾Point Report (Version 3) December 2018 20. Pankisi Gorge Action Plan August 2016 21. Market Mapping, Value Chain, and Opportunities-Constraints-Solutions (OCS) Analysis December 2016 22. Zrda Sector Selection Brief No date 23. Zrda Program Description No date 24. USAID Zrda Activity – Pankisi Gorge Action Plan August 2016 25. Market Mapping, Value Chain and Opportunities-Constraints-Solutions (OCS) Analysis December 2016 26. Zrda Sector Selection Brief No date 27. Zrda Target 81 Communities – ZA (1) No date 28. PPP Module FY 2018 29. Mid-Term Evaluation – Georgia’s SME Development Strategy 2016-2020 2018 30. SME Development Strategy 2016-2020 2016 31. Rural Development Strategy of Georgia 2017-2020 No date 32. Rural Development Strategy Action Plan 2018-2020 No date 33. GITA – Report on the year 2018 of the 2018-2020 Action Plan of the Rural Development Strategy of Georgia 2017-2020 No date 34. Georgia Rural Development Strategy (2017-2020) – 2017 Action Plan – Annual Report April 2018 35. Shida-Kartli Regional Development Strategy 2014-2021 2013 36. Samtskhe-Javakheti Regional Development Strategy 2014- 2021 2013 37. Kakheti Regional Development Strategy 2014-2021 2013 38. Regional Development Program of Georgia 2018-2021 No date 39. Strategy for Agricultural Development in Georgia 2015-2020 2015 50 Title of Document Date 40. Grant and Cooperative Agreement – Modification 01 September 23, 2016 41. Grant and Cooperative Agreement – Modification 02 No date 42. Grant and Cooperative Agreement – Modification 03 February 11, 2017 43. Grant and Cooperative Agreement – Modification 04 June 22, 2018 44. Grant and Cooperative Agreement – Modification 05 November 29, 2018 45. Grant and Cooperative Agreement – Modification 06 August 27, 2019 46. USAID – Country Development Cooperation Strategy 2013-2020 July 2012 (extended October 2016 and February 2019) 47. Government of Georgia – Ministry of Regional Development & Infrastructure – “2018-2021 Regional Development Programme of Georgia” June 2018 48. Government of Georgia – Ministry of Regional Development & Infrastructure – “Regional Development Strategies for 2014-2021 for the regions of Georgia” December 2016 49. Government of Georgia – Ministry of Environmental Protection and Agriculture of Georgia – “Strategy for Agriculture Development in Georgia 2015-2020” 2015 50. Georgian National Tourism Administration (GNTA) – Georgian Tourism in Figures (Structure & Industry) 2018 51. Georgia’s Innovation & Technology Agency (GITA) – the “Progress Monitoring Report for 2018 of the Rural Development Strategy of Georgia 2017-2018 Action Plan 2018-2020” March 2019 52. Georgian National Tourism Development Strategy 2015-2025 2015 53. Georgia Rural Development Strategy 2017-2020 – 2017 Action Plan Annual Report April 2018 54. Government of Georgia – Ministry of Economy & Sustainable Development – “SME Development Strategy of Georgia” 2016 55. High Mountain Settlements Development Strategy of Georgia, 2019- 2023 2019 56. Action Plan for the Development of High Mountain Settlements of Georgia, 2019-2020 2019 57. MoU between Zrda and Zugdidi Municipality City Hall July 25, 2019 51 ANNEX 4: ZRDA EVALUATION MATRIX 52 Zrda Evaluation Matrix No. Evaluation Questions and Sub-Questions Data Source Data Collection Method & Sample Size Data Analysis 1. What approaches have been most or least successful in creating jobs and increasing incomes in target communities and why? • Zrda documents: annual and quarterly reports, M&E data, assessments, studies, and surveys • Documents related to Zrda’s direct beneficiaries (recipients of TA, grants & training plus from MSMEs and community-based organizations [CBOs]) • USAID/Georgia staff • Zrda project staff • Pre-identified “stakeholders”— individuals/organizations with direct/indirect knowledge of Zrda including women and youth organizations • Government line ministries: Ministry of Environmental Protection & Agriculture, Ministry of Economy & Sustainable Development, Ministry of Regional Development & Infrastructure • GNTA • GITA • Other relevant sources • Document search & review sourced from USAID & Zrda • Secondary data search & analysis from publications in the public domain—government, donors, private organizations, NGOs, civil society organizations (CSOs) • Feedback from KIIs pre-identified as potential sources of relevant information re: Zrda and its activities • Feedback from KIIs in the regions potentially with local authorities (to be discussed with Zrda) • FGDs with direct beneficiaries— how many, where, and when to be confirmed with Zrda • Site visits to selected direct beneficiaries—subject to logistics • Verification of Zrda reporting • Analysis of Zrda & secondary data • Content analysis of KII and FGD notes from team members and assimilation into the Draft Evaluation Report • Synthesis and triangulation of data gathered to inform findings, conclusions, and recommendations for the Draft Evaluation Report 53 Zrda Evaluation Matrix No. Evaluation Questions and Sub-Questions Data Source Data Collection Method & Sample Size Data Analysis 2. Do interventions targeted at and/or implemented in partnership with larger companies result in creation of more jobs than interventions involving smaller companies and why? • Zrda documents: annual and quarterly reports, M&E data, assessments, studies, and surveys • Documents related to Zrda’s direct beneficiaries (recipients of TA, grants & training plus from MSMEs and CBOs) • USAID/Georgia staff • Zrda project staff • Pre-identified “stakeholders”— individuals/organizations with direct/indirect knowledge of Zrda including women and youth organizations • Government line ministries: Ministry of Environmental Protection & Agriculture, Ministry of Economy & Sustainable Development, Ministry of Regional Development & Infrastructure • GNTA • GITA • Other relevant sources • Document search & review sourced from USAID & Zrda • Secondary data search & analysis from publications in the public domain—government, donors, private organizations, NGOs, CSOs • Feedback from KIIs pre-identified as potential sources of relevant information re: Zrda and its activities • Feedback from KIIs in the regions potentially with local authorities (to be discussed with Zrda) • FGDs with direct beneficiaries— how many, where, and when to be confirmed with Zrda • Site visits to selected direct beneficiaries—subject to logistics • Verification of Zrda reporting • Analysis of Zrda & secondary data • Content analysis of KII and FGD notes from team members and assimilation into the Draft Evaluation Report • Synthesis and triangulation of data gathered to inform findings, conclusions, and recommendations for the Draft Evaluation Report 54 Zrda Evaluation Matrix No. Evaluation Questions and Sub-Questions Data Source Data Collection Method & Sample Size Data Analysis 3. What are the key lessons learned from implementing PPPs under Zrda? • Zrda documents: annual and quarterly reports, M&E data, assessments, studies, and surveys • Documents related to Zrda’s direct beneficiaries (recipients of TA, grants & training plus from MSMEs and CBOs) • USAID/Georgia staff • Zrda project staff • Pre-identified “stakeholders”— individuals/organizations with direct/indirect knowledge of Zrda including women and youth organizations • Government line ministries: Ministry of Environmental Protection & Agriculture, Ministry of Economy & Sustainable Development, Ministry of Regional Development & Infrastructure • GNTA • GITA • PPPs created with BP and ADC • Other relevant sources • Document search & review sourced from USAID & Zrda • Secondary data search & analysis from publications in the public domain—government, donors, private organizations, NGOs, CSOs • Feedback from KIIs pre-identified as potential sources of relevant information re: Zrda and its activities including information provided by both PPPs—BP and ADC • Feedback from KIIs in the regions potentially with local authorities (to be discussed with Zrda) • FGDs with direct beneficiaries— how many, where, and when to be confirmed with Zrda • Site visits to selected direct beneficiaries—subject to logistics • Verification of Zrda reporting • Analysis of Zrda & secondary data • Content analysis of KII and FGD notes from team members and assimilation into the Draft Evaluation Report • Synthesis and triangulation of data gathered to inform findings, conclusions, and recommendations for the Draft Evaluation Report 55 Zrda Evaluation Matrix No. Evaluation Questions and Sub-Questions Data Source Data Collection Method & Sample Size Data Analysis 4. Have Zrda’s sub-awards and TA activities struck optimal balance between supporting minority and non-minority communities? Does Zrda sufficiently integrate and benefit minority communities? • Zrda documents: annual and quarterly reports, M&E data, assessments, studies, and surveys • Documents related to Zrda’s direct beneficiaries (recipients of TA, grants & training plus from MSMEs and CBOs) • USAID/Georgia staff • Zrda project staff • Pre-identified “stakeholders”— individuals/organizations with direct/indirect knowledge of Zrda including women and youth organizations • Government line ministries: Ministry of Environmental Protection & Agriculture, Ministry of Economy & Sustainable Development, Ministry of Regional Development & Infrastructure • GNTA • GITA • Other relevant sources • United Nations Association of Georgia (UNAG) and/or other organizations representing minority interests • Document search & review sourced from USAID & Zrda • Secondary data search & analysis from publications in the public domain – government, donors, private organizations, NGOs, CSOs • Feedback from KIIs pre-identified as potential sources of relevant information re: Zrda and its activities • Feedback from KIIs in the regions potentially with local authorities (to be discussed with Zrda) • FGDs with direct beneficiaries— how many, where, and when to be confirmed with Zrda • Site visits to selected direct beneficiaries—subject to logistics • Mini-survey of non-beneficiaries in Kvemo Kartli villages • Verification of Zrda reporting • Analysis of Zrda & secondary data • Content analysis of KII and FGD notes from team members and assimilation into the Draft Evaluation Report • Analysis of findings resulting from the mini-survey • Synthesis and triangulation of data gathered to inform findings, conclusions, and recommendations for the Draft Evaluation Report 56 Zrda Evaluation Matrix No. Evaluation Questions and Sub-Questions Data Source Data Collection Method & Sample Size Data Analysis 5. Should Zrda provide continued support to DMOs as a means to promote the sustainable development of Georgia’s tourism sector and why? What type of support, if any, is needed to make DMOs operational sustainable by the end of the Zrda program, considering specific organizational needs of each DMO? • Zrda documents: annual and quarterly reports, M&E data, assessments, studies, and surveys • Documents related to Zrda’s direct beneficiaries (recipients of TA, grants & training plus from MSMEs and CBOs) • USAID/Georgia staff • Zrda project staff • Pre-identified “stakeholders”— individuals/organizations with direct/indirect knowledge of Zrda including women and youth organizations • Government line ministries: Ministry of Environmental Protection & Agriculture, Ministry of Economy & Sustainable Development, Ministry of Regional Development & Infrastructure • GNTA • GITA • Other relevant sources • DMOs and TIC personnel • State University of Georgia • Document search & review sourced from USAID & Zrda • Secondary data search & analysis from publications in the public domain—government, donors, private organizations, NGOs, CSOs • Feedback from KIIs pre-identified as potential sources of relevant information re: Zrda and its activities • Feedback from KIIs in the regions potentially with local authorities (to be discussed with Zrda) • FGDs with direct beneficiaries— how many, where, and when to be confirmed with Zrda • Site visits to selected direct beneficiaries—subject to logistics • Data and information from DMOs • Beneficiary survey of targeted households • Verification of Zrda reporting • Analysis of Zrda & secondary data • Content analysis of KII and FGD notes from team members and assimilation into the Draft Evaluation Report • Synthesis and triangulation of data gathered to inform findings, conclusions, and recommendations for the Draft Evaluation Report 57 Zrda Evaluation Matrix No. Evaluation Questions and Sub-Questions Data Source Data Collection Method & Sample Size Data Analysis 6. Which of the following activities do Zrda beneficiaries and stakeholders identify as profit centers or keys to future business growth? These activities include certification programs in the hospitality sector, discrete training interventions in the tourism sector, and digital marketing activities through booking.com, TripAdvisor, and development of websites. • Zrda documents: annual and quarterly reports, M&E data, assessments, studies, and surveys • Documents related to Zrda’s direct beneficiaries (recipients of TA, grants & training plus from MSMEs and CBOs) • USAID/Georgia staff • Zrda project staff • Pre-identified “stakeholders”— individuals/organizations with direct/indirect knowledge of Zrda including women and youth organizations • Government line ministries: Ministry of Environmental Protection & Agriculture, Ministry of Economy & Sustainable Development, Ministry of Regional Development & Infrastructure • GNTA • GITA • Other relevant sources • DMOs and TIC personnel • Document search & review sourced from USAID & Zrda • Secondary data search & analysis from publications in the public domain—government, donors, private organizations, NGOs, CSOs • Feedback from KIIs pre-identified as potential sources of relevant information re: Zrda and its activities • Feedback from KIIs in the regions potentially with local authorities (to be discussed with Zrda) • FGDs with direct beneficiaries— how many, where, and when to be confirmed with Zrda • Site visits to selected direct beneficiaries—subject to logistics • Data and information from DMOs • Verification of Zrda reporting • Analysis of Zrda & secondary data • Content analysis of KII and FGD notes from team members and assimilation into the Draft Evaluation Report • Synthesis and triangulation of data gathered to inform findings, conclusions, and recommendations for the Draft Evaluation Report 58 ANNEX 5: FGD BENEFICIARIES – PARTICIPANT RESPONSES 59 Indicators (result after Zrda support)26 Zrda LOP Target Zrda Actuals to date FGD Beneficiaries – Final Summary Participant Responses Tourism Agriculture Tourism Agri￾culture - Yes No Not Sure Yes No Not Sure - - - M F M F M F M F M F M F Total (Yes) Total (Yes) TOTAL (Yes) 1 Business has generated sales? 800 688 19 20 3 5 4 1 25 4 - - 2 - 29 29 58 2 Business has created jobs? 2400 905 12 8 7 13 7 8 26 4 - - 1 - 20 30 50 3 Household has increased income by 25%? 13,200 7,527 18 13 4 11 7 - 25 4 - - 2 - 31 29 60 - Yes No Not sure Yes No Not sure M F M F M F M F M F M F Identify as profit centers/keys to growth? - TOTAL (Yes) - - 1 Certification programs in hospitality - 51 22 21 3 4 1 1 8 - - - 4 - 2 Discrete training interventions in tourism - 58 21 22 - - 5 4 15 - - - 7 - 3 Digital marketing – booking.com, TripAdvisor, etc. - 38 12 20 11 4 6 4 6 - - - 10 - 4 Other - - - - - - - - - - - - - 26 These are reflected in Zrda’s M&E Plans. 60 # FGD Location Business Activity No. of Participants M F Total 1 Borjomi (Tourism & Agriculture) Hotels, Tour Guides, Restaurants, Adventure Businesses 8 4 12 2 Bolnisi Azeri Farmers 9 1 10 3 Ninotsminda Farmers 5 1 6 4 Gori (Tourism) Guesthouses 1 7 8 5 Gori (Agriculture) Farmers 6 2 8 6 Akhaltsikhe Hotels & Restaurants 6 2 8 7 Zugdidi Guesthouses 8 8 8 Zugdidi Restaurant/Café Owners 2 1 3 9 Anaklia Guesthouses & Unemployed 3 5 8 10 Tsalenjikha Tour Guides & Guesthouse Owner 4 1 5 Total (Numbers) 44 32 76 Total (%) 58% 42% 100% 61 ANNEX 6: ZRDA ACTIVITY M&E PLAN SUMMARY 62 Purpose Level Indicators & Illustrative Targets M&E Plans Indicators 2016 2017 2018 2019 2020 2021 Total targets T A T A T A T A T A T A T A 201627 1 Sales value generated by MSMEs $1,500,000 - 6,000,000 - 9,000,000 - 9,000,000 - 4,500,000 - - - $30m - 2 No. of MSMEs generating sales 30 - 130 - 195 - 195 - 100 - - - 650 - 3 No. jobs created 100 - 400 - 600 - 600 - 300 - - - 2,000 - 4 House + income x 25% - 11,000 - 5 % change in Resiliency Index -10% - 6 No. of stakeholders – risk reducing actions 750 - 7 Total value of GoG leveraged funds $150,000 - 600,000 - 900,000 - 900,000 - 450,000 - - - $3m - 8 No. beneficiaries using improved infrastructure services 400 - 1,600 - 2,400 - 2,400 - 1,200 - - - 8,000 - 201728 LOP 1 Sales value generated by MSMEs 0 0 6,000,000 1,667,484 9,000,000 0 8,000,000 - 4,500,000 - 2,500,000 - $30m $1,667,484 Tourism - 1,246,019 - - - - - - - - - 1,246,019 Agriculture - 258,167 - - - - - - - - - 258,167 Services - 163,298 - - - - - - - - - 163,298 2 No. of MSMEs generating new sales 0 0 170 189 190 0 180 - 90 - 20 - 650 189 Tourism - 141 - - - - - - - - - 141 27 Source: Extracted from Zrda – Activity Monitoring & Evaluation Plan – May 2016. 28 Source: Extracted from Zrda – FY 2017 Annual Report – Annex A – ZRDA Indicator Tracking Matrix. 63 Purpose Level Indicators & Illustrative Targets M&E Plans Indicators 2016 2017 2018 2019 2020 2021 Total targets T A T A T A T A T A T A T A Agriculture - 45 - - - - - - - - - 45 Services - 3 - - - - - - - - - 3 3 No. of jobs created 0 0 400 375 600 0 600 - 300 - 100 - 2,000 375 Tourism - 263 - - - - - - - - - 263 Agriculture - 104 - - - - - - - - - 104 Services - 8 - - - - - - - - - 8 4 No. of households increase incomes by 25% - 11,000 - 5 % change in Resiliency Index +10% - 6 No. of stakeholders – risk reducing actions 750 - 7 Total value of GoG leveraged funds $27,556 27,556 800,000 977,441 900,000 0 800,000 - 350,000 - 122,444 - $3m29 $1,004,997 8 No. beneficiaries using improved infrastructure services 0 0 1,800 77 2,400 0 2,400 - 1,200 - 200 - 8,000 77 201830 1 Sales value generated by MSMEs 0 0 6,000,000 1,667,484 10,000,000 9,321,473 10,000,000 - 6,000,000 - 4,000,000 - $36m $10,988,957 Tourism - 1,246,019 - 4,007,317 - - - - - - - 5,253,336 Agriculture - 258,167 - 3,729,361 - - - - - - - 3,987,528 Services - 163,298 - 1,584,795 - - - - - - - 1,746,093 2 No. of MSMEs generating new sales 0 0 170 189 300 323 220 - 90 - 20 - 800 512 Tourism - 141 - 100 - - - - - - - 241 Agriculture - 45 - 176 - - - - - - - 221 Services - 3 - 47 - - - - - - - 50 3 No. of jobs created 0 0 400 375 600 314 700 - 500 - 200 - 2,400 689 Tourism - 263 - 98 - - - - - - - 361 29 Actual Total = $3,027,556 (rounded up). 30 Source: Extracted from Zrda – FY 2018 Annual Report - Annex A - ZRDA Indicator Tracking Matrix. 64 Purpose Level Indicators & Illustrative Targets M&E Plans Indicators 2016 2017 2018 2019 2020 2021 Total targets T A T A T A T A T A T A T A Agriculture - 104 - 108 - - - - - - - 212 Services - 8 - 108 - - - - - - - 116 4 No. of households increase incomes by 25% - 13,200 - 5 % change in Resiliency Index +10% - 6 No. of stakeholders – risk reducing actions 900 - 7 Total value of GoG leveraged funds $27,556 27,556 800,000 977.441 1,400,000 1,216,938 900,000 - 350,000 - 122,444 - $3.6m $2,221,935 8 No. beneficiaries using improved infrastructure services 0 0 1,800 77 2,500 528 2,500 - 2,200 - 600 - 9,600 605 201931 1 Sales value generated by MSMEs 0 0 6,000,000 1,667,484 10,000,000 9,321,473 10,000,000 10,844,689 6,000,000 - 4,000,000 - $36m $21,833,646 Tourism - 1,246,019 - 4,007,317 - 7,798,528 - - - - - 13,051,864 Agriculture - 258,167 - 3,729,361 - 2,612,579 - - - - - 6,600,107 Services - 163,298 - 1,584,795 - 433,582 - - - - - 2,179,675 2 No. of MSMEs generating new sales 0 0 170 189 300 323 220 156 90 - 20 - 800 688 Tourism - 141 - 100 - 30 - - - - - 271 Agriculture - 45 - 176 - 123 - - - - - 344 Services - 3 - 47 - 3 - - - - - 53 3 No. of jobs created 0 0 400 375 600 314 700 216 500 - 200 - 2,400 905 Tourism - 263 - 98 - 39 - - - - - 400 Agriculture - 104 - 108 - 105 - - - - - 317 Services - 8 - 108 - 72 - - - - - 186 4 No. of households increase incomes by 25% - 13,200 7,527 31 Source: Same data source as FY 2018 to date. 65 Purpose Level Indicators & Illustrative Targets M&E Plans Indicators 2016 2017 2018 2019 2020 2021 Total targets T A T A T A T A T A T A T A 5 % change in Resiliency Index 10% 38% 6 No. of stakeholders – risk reducing actions 900 531 7 Total value of GoG leveraged funds $27,536 27,536 800,000 977,441 1,400,00 1,216,938 900,000 557,580 350,000 - 122,444 - $3.6m $2,779,515 8 No. beneficiaries using improved infrastructure services 0 0 1,800 77 2,500 528 2,500 377 2,200 - 600 - 9,600 982