USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION June 2022 This publication is made possible by the support of the American People through the United States Agency for International Development (USAID) and was prepared by Integra Government Services International LLC for the Learning, Evaluation, and Analysis Project (LEAP III) Activity. USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION Contract Title: LEAP III: Learning, Evaluation, and Analysis Project Contract Number: GS-10F-083CA / 7200AA18M0004 Activity Number: 1009.1064 Submitted: Final Report – June 2022 Contractor: Integra Government Services International LLC 1156 15th Street NW, Suite 800 Washington, DC 20005 USAID Office: USAID/Georgia COR: Katie Qutub, kqutub@usaid.gov Photo Credit: USAID Georgia Economic Security Program gives out National Tourism Award (Photo/USAID Economic Security Program) Prepared by: Brenda Lee Pearson, Maia Giorbelidze, Lasha Kavtaradze, Rati Gabrichidze, Penelope Norton DISCLAIMER This evaluation is made possible by the support of the American people through USAID. The authors’ views expressed in this publication do not necessarily reflect the views of USAID or the United States Government. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 1 TABLE OF CONTENTS ACRONYMS 5 EXECUTIVE SUMMARY 7 1. INTRODUCTION AND BACKGROUND 14 2. METHODOLOGY 18 3. EQ1 FINDINGS, CONCLUSIONS, RECOMMENDATIONS 22 4. EQ2 FINDINGS, CONCLUSIONS, RECOMMENDATIONS 29 5. EQ3 FINDINGS, CONCLUSIONS, RECOMMENDATIONS 37 6. EQ4 FINDINGS, CONCLUSIONS, RECOMMENDATIONS 42 7. EQ5 FINDINGS, CONCLUSIONS, RECOMMENDATIONS 50 8. CROSS-CUTTING FINDINGS AND RECOMMENDATIONS 54 ANNEX 1: STATEMENT OF WORK 57 ANNEX II: EVALUATION WORK PLAN 71 ANNEX III: DOCUMENTS REVIEWED 131 ANNEX IV: KEY INFORMANTS, FOCUS GROUP, AND SURVEY RESPONDENTS 133 ANNEX V: SURVEY RESULTS 139 ANNEX VI: GLOSSARY OF TERMS 152 USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 2 LIST OF TABLES TABLE 1: EVALUATION QUESTIONS 17 TABLE 2: AFFILIATION OF KEY INFORMANTS 19 TABLE 3: EMPLOYMENT IN KEY SECTORS PRIOR TO THE PANDEMIC 43 TABLE 4: ILLUSTRATIVE EVALUATION MATRIX 61 TABLE 5: GETTING TO ANSWERS MATRIX – AGRICULTURE PROGRAM EVALUATION 82 TABLE 6: GETTING TO ANSWERS MATRIX – ECONOMIC SECURITY PROGRAM EVALUATION 86 TABLE 7: STAKEHOLDER CATEGORIES 92 TABLE 8: DELIVERABLES TIMELINE 96 TABLE 9: USAID AOR/COR OF ECONOMIC SECURITY PROGRAM DISCUSSION GUIDE 121 TABLE 10: DAI, SOLIMAR INTERNATIONAL AND PMCG DISCUSSION GUIDE 122 TABLE 11: BUSINESS ASSOCIATIONS/GRANTEES DISCUSSION GUIDE 125 TABLE 12: PUBLIC PRIVATE PARTNERSHIPS AND GDA DISCUSSION GUIDE 127 TABLE 13: FINANCIAL INSTITUTIONS DISCUSSION GUIDE 128 TABLE 14: LIST OF KEY INFORMANTS, FOCUS GROUPS, AND SURVEY RESPONDENTS 133 TABLE 15: EQ1 SURVEY RESULTS 139 TABLE 16: CHALLENGES PER SECTOR 141 TABLE 17: HOW USEFUL WAS THE COOPERATION WITH THE USAID ECONOMIC SECURITY PROGRAM TO ADDRESS THE CHALLENGES 145 TABLE 18: BREAKDOWN OF GRANTS BY VALUE CHAIN 149 TABLE 19: BREAKDOWN OF REQUESTED FUNDS AND COST SHARE BY VALUE CHAIN 149 LIST OF FIGURES FIGURE 1: METHODOLOGY OVERVIEW 20 FIGURE 2: CHALLENGES TO ECONOMIC GROWTH AND COMPETITIVENESS BY SECTOR 23 FIGURE 3: BSO, MSME AND GRANTEES’ RATING OF THE PROGRAM’S APPROACHES 25 FIGURE 4: STAKEHOLDERS’ ASSESSMENT OF PROGRAM SUPPORT AS SUFFICIENT AND RELEVANT 31 FIGURE 5: 95 PERCENT OF PRIVATE SECTOR COMPANIES FOUND WORKFORCE DEVELOPMENT ASSISTANCE RELEVANT OR VERY RELEVANT (N=62) 34 USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 3 FIGURE 6: 79 PERCENT OF INTERNS FOUND THE WORKFORCE DEVELOPMENT #GO4IT PROGRAM USEFUL 34 FIGURE 7: PERCENTAGE FOR GRANT FUNDING TO SPECIFIC VALUE CHAINS 38 FIGURE 8: MORE THAN HALF OF GRANTEES IN SURVEYS FOUND THE APPLICATION PROCESS EASY AND TRANSPARENT 40 FIGURE 9: COVID-19 IMPACT ON THE TOURISM SECTOR SEEN THROUGH COMPARISON OF INTERNATIONAL VISITORS AND ANNUAL TOURISM REVENUES 45 FIGURE 10: PERCENTAGE CHANGE IN EMPLOYMENT FROM IN SHARED INTELLECTUAL SERVICES AND CREATIVE SECTOR FROM 2020 TO Q1 AND Q2 OF 2021 46 FIGURE 11: PERCENTAGE OF CHANGE IN EMPLOYMENT BETWEEN 2020 AND Q1, Q2 2021 48 FIGURE 12: PUBLIC-PRIVATE DIALOGUE AND ADVOCACY: A RESOURCE MANUAL 51 FIGURE 13: SURVEY RESPONSES BY KEY RESPONDENT GROUPS 140 FIGURE 14: KEY CHALLENGES PER SECTOR, MOST COMMONLY ANSWERED TO LEAST COMMONLY ANSWERED 142 FIGURE 15: COMPARATIVE BAR GRAPH ON CHALLENGER PER SECTOR BY CHALLENGE 144 FIGURE 16: COMPARATIVE BAR GRAPH ON CHALLENGER PER SECTOR BY SECTOR 145 FIGURE 17: USEFULNESS RATING OF USAID COOPERATION IN ADDRESSING CHALLENGES 146 FIGURE 18: TO WHAT EXTENT HAVE STAKEHOLDERS RECEIVED SUFFICIENT AND RELEVANT SUPPORT 146 FIGURE 19: SURVEY RESPONSES ON RELEVANCY OF WORKFORCE DEVELOPMENT PROGRAM SUPPORT 147 FIGURE 20: WORKFORCE DEVELOPEMENT PARTICIPANT INTERNS' RESPONSES TO INFORMATION ACCESSIBILITY 147 FIGURE 21: WORKFORCE DEVELOPMENT PROGRAM INTERNS’ SATISFACTION RATES 148 FIGURE 22: ONLINE SURVEY RESPONSES OF WORKFORCE DEVELOPMENT PROGRAM INTERNS’ SATISFACTION RATES 148 FIGURE 23: ASSESSMENT OF GRANT COMPONENTS 150 FIGURE 24: ASSESSMENT OF EQ3 RESULTS 150 USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 4 ACRONYMS BEE - - - - - - - - - - - - - - - - Business Enabling Environment BSO Business Support Organization CSO Civil Society Organization EQ Evaluation Question ET Evaluation Team FDI Foreign Direct Investment GDA Global Development Alliance GNTA Georgia National Tourism Agency GOG Government of Georgia ICT Information Communications and Technology IP Implementing Partner IRI International Republican Institute LEAP III Learning, Evaluation, and Analysis Project III MSD Market Systems Development MSME Micro, Small, and Medium Enterprises OCAP Occupational Customized Assistance Program PLR - - - - - Policy, laws, regulations PMCG Policy and Management Consulting Group PPD Public-Private Dialogue PPP Public-Private Partnership PSE Private Sector Engagement SIS - - - - - - - Shared Intellectual Services SOW Scope of Work STEM Science, Technology, Engineering, and Mathematics SWM Solid Waste Management UNDP United Nations Development Program USAID United States Agency for International Development USMAC U.S. Market Access Center USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 5 ACKNOWLEDGEMENTS The authors would like to acknowledge and thank the numerous stakeholders and beneficiaries of the Economic Security Program, particularly those who shared their time and experiences during remote data collection and during the preparation of this paper. The authors would like to thank USAID/Georgia staff who contributed time and insights to the evaluation, and the LEAP III Contracting Officer Representative, Katie Qutub. We thank Economic Security Program Chief of Party Mark McCord and Deputy Chief of Party Marika Shioshvili and other staff for providing information and support to this evaluation. The authors would like to acknowledge USAID/Georgia staff for their time and efforts in providing project documents, extensive feedback, and reviews during this evaluation. The team wants to thank all our Georgian consultants who made our remote data collection go smoothly. Also, we want to acknowledge the support of our LEAP III home office staff, David Quinn, LEAP III Chief of Party, Ganyapak (Pin) Thanesnant, Director of Operations and Penelope Norton, Senior Associate. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 6 EXECUTIVE SUMMARY INTRODUCTION The United States Agency for International Development (USAID)/Georgia requested that the Learning, Evaluation, and Analysis Project III (LEAP III) team conduct a hybrid (in-person and remote) performance evaluation of the Economic Security Program (the Program) being implemented by DAI and several Georgian and American partners on November 1, 2021. USAID/Georgia undertook this evaluation in the spirit of learning and adapting to better understand how the current program, which was originally designed under a previous strategy and under a very different geo-political and economic environment, could be guided to better advance the current Country Development Cooperation Strategy (CDCS) 2020-2025 and align with current USAID policies. Of note, the Program was conceived and designed prior to USAID’s Private Sector Engagement (PSE) Policy and Digital Strategy, and prior to the outbreak of COVID-19 pandemic which has had a dramatic and lasting impact on the economic sectors targeted by the Program. In light of these changes in the operating context, new strategic priorities, and new Agency policies, USAID/Georgia asked the evaluation team to gather evidence and provide recommendations for improvements in five key areas of interest: 1) private sector engagement, 2) value chain approach, 3) grant components, 4) policy coordination, and 5) COVID-19. METHODOLOGY The Evaluation Team (ET) used a mixed-methods approach, combining qualitative, in-depth remote interviews with key stakeholders, Program partners and informants, and online surveys and group interviews. The purpose of this research was to answer the EQs that focused on PSE; government and business associations’ support of micro, small, and medium enterprises (MSMEs); specific value chains such as tourism, shared intellectual services (SIS), creative industries, light manufacturing, solid waste management (SWM); private investment expansion; skills and business training; and building the sustainability of activities and organizations during the post-COVID-19 recovery. All proposed semi￾structured interviews and group discussions, as well as the surveys, were organized around the EQs and supported with detailed instruments. Each tool (see Annex II) was developed for a specific group of interviewees and includes a mix of shared questions unique to a particular group to obtain a full range of information regarding specific activities, as well as to ensure that data is comparable across all respondent groups. PROGRAM FINDINGS AND RECOMMENDATIONS The ET found that the Program activities evaluated were appropriately designed, relevant, and delivered assistance that aligned with beneficiaries’ needs. The ET also found the Program to be coordinating effectively with other USAID activities, such as the Economic Governance Program, to address the policy barriers facing its priority sectors and value chains. The current modes of implementation that the Economic Security Program uses may require revisions in response to new economic circumstances inherent in the COVID-19 recovery. The selection criteria for future partnerships may need to be revisited once USAID completes its new guidelines for partnership criteria, which will focus more on addressing systemic gaps in industries and market failures. Similarly, grant solicitation themes may need to shift due to disruptions in supply chains affecting sourcing of equipment, changes in consumer behavior USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 7 and the impact of COVID-19 and war in the region. USAID investments across SIS and creative industries may benefit from more pilots with regional and local governments. Overall, the Program has been effective in achieving its goals and creating a supportive ecosystem for established companies and aspiring entrepreneurs, MSMEs, and foreign direct investment (FDI). Each of the current 16 partnerships and 40 grants has, to varying degrees, successfully contributed to strengthening various aspects of the economic ecosystem needed for MSMEs and business support organizations (BSOs) to grow, increase employment, and become competitive. 1 The following recommendations were presented by the ET and discussed with USAID/Georgia during a validation workshop on November 5, 2021. Additionally, the Implementing Partner (IPs) had two opportunities to review these recommendations and provide feedback.2 EQ1: To what extent has the Partnership Development Fund (PDF) targeted and established high-impact (defined as wide-reaching and/or replicable) partnerships with the private sector that have strengthened and catalyzed the development of priority value chains? To what extent are these partnerships sustainable (defined as the establishment of market linkages that will not depend on USAID assistance after the activity ends)? The Program facilitated the creation of 16 Public-Private-Partnerships (PPPs) and Global Development Alliances (GDAs) at an estimated value of $57.4 million. These partnerships leveraged multi-million-dollar resources to create high-value jobs, increase MSME revenues, and diversify value chains. The ET concluded these 16 partnerships contributed to strengthening key value chains in tourism, creative industries, light manufacturing, shared intellectual services, which included ICT and e-commerce. The ET’s data analysis highlighted four main challenges to Georgia’s economic growth influencing competitiveness across value chains: l) lack of qualified staff, 2) access to finance, 3) need for new technologies, and 4) administrative and policy barriers that negatively impact the business enabling environment. The Program partnerships each responded to at least one of these identified challenges, which supports the ET’s findings that the Program has strengthened priority value chains. One PPP, Digital Day with Steller/Georgia National Tourism Agency (GNTA), was assessed very positively because it has already demonstrated high sustainability and the likelihood of replicability. The partnerships in e-commerce with Esty, Creative Industry Masterclasses, training of hospitality workers, and BizLink’s partnership with the Bank of Georgia also strengthened and catalyzed the development of priority value chains. In other cases, it was too soon in the project cycle to measure the extent to which each partnership is likely to be sustainable through scaling, replication and demonstrated impact. The partnership indicators included in the MEL plan, however, did not directly measure systemic and transformational changes with value chains and the degree to which such changes are sustainable.3 Only one regional government, that of the Adjara Region, was currently engaged via a grant for solid waste management in collaboration with the European Innovation Academy. The Key Management Solutions partnership focuses on workforce development training to improve the skills of hospitality workers. 1 The period of performance for this evaluation was completed before the war began in Ukraine, which also may impact international tourism to Georgia. 2 In the period between the drafting of this mid-term evaluation report and its publication in March 2022, many of these recommendations have been considered and adopted according to the Implementing Partner. 3 Implementing Partner introduced new indicators after this mid-term evaluation draft report was shared with USAID/Georgia and the IP. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 8 Further investment in the region was made through a grant to the Adjara Ministry of Finance and Economy that supports the expansion of SWM firms in recycling. Recommendations: ● Consider adapting key milestones in approaches to developing partnerships so results are reported against indicator targets for PPPs and GDAs. Include a transition plan for how the partnerships will continue after USAID funding ends. ● Modify or adapt new selection criteria related for future partnerships. ● Consider expanding the PDF approach to address systemic market failures to increase overall impact across industries. ● Replicate the Adjara Region’s approach to support more regional and municipal locations outside of Tbilisi in the solid waste management and recycling sector through formal partnering arrangements and/or grants. EQ2. To what extent has support to sector associations and government institutions catalyzed priority value chain development? How important a role have these stakeholders played in increasing value chain competitiveness? To what extent have these stakeholders received sufficient and relevant support? The most important stakeholders that need to be involved in increasing the competitiveness of MSMEs in priority value chains are the private sector firms, GOG, financial institutions, educational institutions, and BSO. Program activities have primarily supported the GOG and BSOs - BSOs that serve the business community in Georgia are important conduits for improving MSME competitiveness. The Program’s investments in workforce development activities address the importance of linking educational institutions to employer needs. At the present time, the Program has two formal linkages with financial institutions through its PDF and grants mechanism. Private sector respondents said that BSOs need to develop internal capacity to provide essential services to their members as a first step to increasing value chain competitiveness. According to the ET’s analysis of quantitative and qualitative data, BSOs are not currently positioned to offer critical services such as: business development planning, access to financing, market information, recruitment of qualified workers, financial documentation for loans and grant applications, and linkages across industries and with up-chain enterprises. Prior to COVID-19, the tourism sector was also poised for impressive growth. The Program had designed a multi-pronged approach to improving the competitiveness of the tourism sector. Illustrative examples include launching partnerships to co-invest in a digital media advertising campaign through the Tourism Matching Fund. The partnership with Steller increased the global visibility of Georgia as a tourist destination. Another activity increased the capacity of private sector firms and Mountain Trails Agency staff to offer outdoor sports venues that were competitive with European ones. Other activities focused on training hospitality workers and tour operators. The Program’s facilitation of the COVID-19 recovery plan in tourism was positively received by the public and private sector. Vibrant ICT growth within the SIS sector is seen as critical to generating quality jobs. Georgia’s SIS cluster presents great potential for quick growth and high earning potential, and modernizes the country’s economy, governance, and society. The partnerships between the Government of Georgia (GOG) and USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 9 the private sector that prioritize modernization of the SIS could be a game-changer for the Georgian economy. The Program assisted 50 MSMEs to varying degrees in developing e-commerce sites and 10 micro, small and medium enterprises (MSMEs) in migrating to digital platforms. Overall, the Program supported professionals and businesses to understand best practices and to integrate e-commerce into their operations. Partnerships with the Business and Technology University (BTU) to create the iOS Applications Laboratory, Cinema 13 and N&N studio helped Georgia establish the only film laboratory and post-production facility in the region, which is supported by highly trained technicians in post￾production services. Recommendations: ● Consider providing a wider range of capacity building support to BSOs so that higher-capacity BSOs access training in supply chain linkages and procurement in supply chains, while lower￾capacity BSOs continue training in leadership, change management, marketing, branding, relationship building, and membership services. ● Utilize findings from survey data regarding BSO membership services, such as: 1) more interaction and interest in addressing key business challenges; 2) digital solutions; 3) support for accessing finances; 4) a platform for business associations to collaborate and share experiences; 5) marketing and sales promotion assistance; 6) international legal aid for exports; 7) mechanisms to improve the quality of services in tourism and introduction of innovative products; 8) research and development; 9) engagement with foreign associations for matchmaking; 10) assistance in finding new contacts to expand business networks; and 11) active involvement in legislative drafting and sharing concerns with policy-makers. ● Continue guiding the GOG and affiliated public and private sector tourism entities in implementing the National Tourism Recovery Strategy and Action Plan. ● Align expectations of the partnership between the GOG and the IP so both sides understand the purpose of the activity and what future technical assistance might entail at the strategic and operational levels. ● Utilize PDF, grant and technical assistance to individuals to do more cross-marketing and capacity building in regions beyond the Adjara region; expand efforts to identify and leverage government and the private sector partnerships in the regions and selected municipalities. EQ3. To what extent has the grant component strengthened each priority value chain? To what extent did the grants address gaps or market failures in each value chain? The Program issued 40 grant awards with a total value of $1,352,858 and a cost-share contribution of $1,072,528. Each grant activity has indicators that focus on two outputs: increased revenue and job creation. Nearly all (95 percent) grantees used the funds to expand operations and the reported outputs related to increased revenue. The number of high-value jobs created varies greatly across sectors. In tourism, for example, some grants supported developing Destination Management Organizations (DMOs), which potentially strengthens the tourism value chain. Other tourism-related grants supported hotels in developing their online booking platforms and product development. The grants designated for SIS were viewed by respondents as significant investments in building the capacity of Georgia’s educational institutes and private sector firms to provide state-of-the-art ICT and software. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 10 It is too early to determine the impact of grants for the business-to-consumer and business-to-business activities that are expected to relieve pain points related to complex software development within the e￾commerce value chain. All the grants distributed in the light manufacturing sector were for equipment purchases. At the time of the evaluation, however, there were many disruptions in the supply chain that prevented the timely delivery of the equipment. Presumably, after the equipment is delivered and operational, it will strengthen the industry’s competitiveness. Grants in the creative industry mainly supported the purchase of software licenses, equipment for the new post-production facility and film location database. There were no grants awarded in the solid waste and recycling sector at the time of the evaluation. ● Improve grants’ effectiveness and sustainable gains by requesting that grantees explain how they will leverage the USAID-funding to obtain more financing and support from other sources. ● Encourage more innovative applications from consortia that link education and workforce skills development to balance grant disbursements, which tilt heavily toward equipment purchases for light manufacturing (in total grant value). ● Replicate strategic partnering with education institutions such as the GDA with Sweeft Digital and Ilia State University through targeted grants. ● Provide more customized support to grant applicants in each step of the application process and explain the selection criteria to increase transparency of the award. ● Streamline and improve the processing of grants by working with USAID to adjust selection criteria and post-COVID 19 cost-share obligations. ● Expand network and outreach efforts with educational institutions and BSOs located outside Tbilisi to obtain more gender and geographic diversity in grants and partnering organizations. ● Weigh selection criteria to favor applicants that clearly demonstrate how their proposed activities will address gaps and market failures in each value chain. EQ4. In the context of COVID-19 economic contractions, how can the activity adapt its approaches (e.g., selection of grant solicitation themes, division of USAID investments across sub-sectors, sequencing of interventions, etc.) to improve its ability to achieve its targets: creation of 4,800 jobs and achieving $60 million in new sales? Given the great fluctuations in the Georgian economy due to COVID-19, the war in Ukraine, and global inflation, the Program may need to adapt its approaches to consider the new reality related to investment, higher-value job creation, and increased revenues for MSMEs. The Program utilized an ecosystem approach to supporting each sector and value chain. The program quickly pivoted to adapt new delivery mechanisms and implementation modalities to align with new opportunities and challenges. Due to the economic impact of COVID-19, the Program should closely monitor potential obstacles in achieving its targets of creating 4,800 new jobs. The Program appears to be on track to achieve its $60 million revenue target during the period of performance. Based on its analysis of the economic impact of COVID-19 and qualitative data, the ET concluded that continuing Program support to a revamped tourism strategy and legal framework is crucial to the eventual recovery of the sector, which is also the cornerstone of Georgia’s overall economic recovery model. Georgia needs assistance in adopting new health and safety protocols across the industry for both domestic and international tourists. The Program should continue USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 11 to support GNTA and the private sector in implementing its Post-Recovery Plan and Action Plan. In doing so, it should consider working with selected BSOs in assessing the potential revenue linked to domestic tourism as part of its action plan. The Occupational Customized Assistance Program (OCAP) implementation of strategic plans may need to be adjusted to consider new high-priority actions related to disruptions in supply chains, sourcing equipment, COVID-19 related decline of international tourists and other factors. The e-commerce and SIS sectors have seen large increases in revenue during COVID-19. Similarly, there is evidence to support continued investment in light manufacturing, although perhaps in new industries. According to economic data monitored by the GOG, there are promising trends for growth in light manufacturing and shared intellectual and knowledge-based services. The Program may want to assess whether there are new opportunities in these industries. The Program’s subcontractor, Policy Management and Consulting Group (PMCG), provides quarterly analytical studies of sectors and value chains to analyze the potential for high￾value jobs, increased revenues, and partnerships in the post-COVID economy. Recommendations: ● Continue support to the tourism sector and implementation of the recovery plan that embeds new global safety and health protocols. Assess opportunities in domestic tourism. ● Capitalize on new opportunities in ICT, SIS, and other knowledge-based services that have shown resilience and growth during the pandemic. ● Work with SWM companies and municipalities to pilot initiatives at the regional and local levels. ● Work with BSOs in SWM, light manufacturing, and other industries hit hard by the pandemic to develop industry-wide interventions for job creation while implementing the OCAP. ● Ascertain whether BSOs, educational institutes or innovation centers can ramp up training related to e-commerce, digital content, online learning, educational tutorials, and entrepreneurship to attract more youth to high-paying jobs and provide funding4. EQ5. To what extent has the Economic Security Program coordinated effectively with other USAID activities (managed by both the USAID Economic Growth and Democracy, Rights and Governance offices) to address the policy barriers facing its priority sectors and value chains? To what extent has the absence of a large policy component within USAID’s Economic Security Program helped or hindered its ability to address policy gaps? USAID’s Economic Governance Program was designed to improve economic governance and leadership in Georgia in ways that will enable Georgia to harness investments needed to finance its own development. The Economic Security Program identifies key policy barriers jointly with value chain players and then refers those barriers to the Economic Governance Program. The Economic Security Program and Economic Governance Program demonstrated intentional efforts to collaborate effectively on key policy reforms. The combined impact of their support was greater than individual efforts because they draw on each other’s strengths without duplication. 4 Such as the creation of E-Commerce Academy in partnership with E-Commerce Association-Georgia and TBC Bank. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 12 Recommendations: ● Continue to facilitate public-private dialogue (PPD) initiatives in priority sectors identified by other USAID IPs, such as SIS, creative industries, and tourism to identify key policy gaps, and share these insights with other IPs and partners to amplify reform priorities to GOG institutions and parliamentary committees. ● Continue to work with USAID IPs, such as Economic Governance, Good Governance Initiative, Agricultural Program, Industry-led Skills Development Program, and other donors to identify incentives for strategic cohesion among BSOs to coalesce around priority policy gaps. ● Prioritize policy, laws, and regulation (PLR) gaps with the USAID Economic Governance Program that may have shifted due to the economic impact of COVID-19. ● Continue to encourage close coordination among EG and DRG programs to advocate at the national level to hold GOG accountable by encouraging more policy, legislative, and regulatory reforms. This coordination could help the private sector and NGOs demand better services and accountability through advocacy efforts. ● Public Private Dialogues (PPDs) could elicit suggestions on developing an overall strategy to catalyze citizens and PSE in targeted municipalities to improve capacity and local governments’ understanding of economic growth to adapt supportive local PLRs. CROSS-CUTTING FINDINGS AND RECOMMENDATIONS The ET found that many firms face constraints of limited access to finance, poor business and marketing skills, and insufficient workforce skills that perpetuate strong gender imbalances. Women have less access to capital and networks in Georgia than men do because they lack financial collateral and often work from home . As made evident in Program documents and reinforced by its gender specialist, the Program actively included women as participants in all activities, including training. The Program also evaluates PLRs against 10 criteria, including gender and environmental considerations. It has made an active effort to include women as participants in activities and ensuring the collection of sex-disaggregated data for all activities. In addition, the program developed a Gender and Disability Mainstreaming Guide for organizations and has piloted its use with local partners. 5 PSE and market systems development (MSD) approaches are both key to advancing sustainable, scaled, and inclusive outcomes. Preliminary findings indicated that the Program may need improved cross-cutting analytical framework(s) and assessment tool(s) to support selection criteria for future private sector partners, associations, and grantees to hew more closely to USAID/Georgia’s forthcoming PSE selection criteria for economic growth activities. In its Monitoring, Evaluation, and Learning plan, the Program measured PSE engagement through the number of GDAs established (Indicator 26), amount of dollars leveraged through partnerships (Indicator 27), and the number of PPPs established (Indicator 28). According to USAID PSE and MSD guidance, transformational impact is measured by how MSD can leverage PSE’s strengths in corporate relationship management, diverse investment, and partnership strategies. These outputs, however, do not predict the transformational impact of USAID investment 5 Overview of Women Entrepreneurs and Entrepreneurship- UN Women- April 2021 USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 13 according to USAID respondents, which is a key area of interest. No PDF selection criteria explicitly stated the anticipated overall impact across value chains or sectors to be gained from each partnership. Recommendations: ● Consider increasing designated funding opportunities for women’s organizations, female entrepreneurs, and women-owned businesses in more sectors. 6 ● Utilize a PSE and MSD assessment tool with important definitions and distinctions to establish a common understanding with partners; provide high-level guidance on aligning, customizing, and operationalizing approaches to PSE and MSD with inputs from USAID/Georgia and potentially USAID/Washington PSE and MSD hubs. ● Adapt a strategy for setting indicator targets to report how partnerships and grants address systemic gaps and market failures and/or influence the industry.7 6 Implementing Partner introduced a new EIA partnership and Catapult Fund to address this issue after the mid-term evaluation draft report was shared with USAID/Georgia and the IP. 7 Implementing Partner introduced new indicators to address this issue after this mid-term evaluation draft report was shared with USAID/Georgia and the IP. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 14 1. INTRODUCTION AND BACKGROUND 1.1 PROGRAM BACKGROUND Although Georgia is a global leader in trade and business environment reforms, economic growth has not resulted in greater employment opportunities or higher wages . An aggressive reform agenda and healthy growth rate have not translated into economic dynamism or opportunities for Georgian citizens. Multiple factors influence the reasons why Georgia’s economic growth has not created high-value employment opportunities for its citizens, limiting the benefits of Georgia’s Western orientation. One factor is that Georgian firms still lack access to resources necessary to increase competitiveness and create greater employment opportunities in key sectors, including access to high-value, diverse markets; investment resources; and a workforce that has the skills demanded by the private sector. The recent COVID-19 crisis triggered a major economic recession in Georgia, resulting in the loss of jobs and local currency devaluation, and throwing into stark relief the need for an economy that delivers real gains to its citizens.9 8 The purpose of USAID’s Economic Security Program is to accelerate the broad-based growth of sectors outside of agriculture that show strong potential to create jobs, increase MSME revenues, and support diversification to more productive economic activities in the tourism, creative industries, light manufacturing (including furniture, packaging, personal protective equipment, and construction materials), ICT, SWM, recycled materials, and SIS sectors. The underlying development hypothesis of the program is that IF Georgia’s firms have access to the resources they need (capital, access to high-value markets, skilled workforces, modern technologies, etc.) to improve productivity, sales, and product and service quality, and IF cooperation is strengthened in targeted sectors and value chains, THEN targeted sectors and value chains will become more competitive and will provide greater high-value employment opportunities to its citizens and drive closer integration with the West. The Economic Security Program is organized into four components designed to achieve the stated results: Component 1: Strengthen cooperation in targeted sectors. The Program provides technical assistance and cost-share grants to strengthen linkages and cooperation throughout value chains in targeted sectors and to improve support services intended to enhance growth and productivity across value chains in targeted sectors. In doing so, the Program takes a collaborative approach to development, working with multiple stakeholders including firms, associations, GOG agencies, development partners, regional government and municipalities, and others. Component 2: Support enterprises to improve productivity, sales, and quality and to develop new products and services. Through the identification and exploration of value chains that provide the best opportunities for Georgia to initiate investment that leads to high-value jobs, the Program facilitates entrance into new markets. It also increases and expands product offerings, promotes stronger linkages between enterprises and the organizations that support them, and enhances the overall value chain 8 USAID Georgia Country Development Cooperation Strategy (CDCS)- 2020-2025 – pg. 5 9 Need for a bolstered economy came prior to the war in Ukraine and the rise of inflation on a global scale. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 15 ecosystem to ensure sustainability. This is accomplished through a series of interventions that include technical assistance, cost-share grants, and export enhancement, among others. Component 3: Industry-led workforce development. Alignment of Georgia’s workforce with the needs of industry is critical to the country’s movement toward the development of a prosperous society. To this end, the Program works with the Ministry of Education and Science of Georgia, educational institutions, training providers, and the private sector to prepare Georgians for new and expanded employment opportunities through identified sectors and value chains. The Program’s approach is led by industry, meaning that it focuses on improving knowledge and skills that align with emerging investment and job opportunities. This requires significant re-thinking of educational and vocational models, as well as specific interventions that will link skills development directly with employment, which includes a focus on internships and apprenticeships. Component 4: Building PPPs. Through its $3 million PDF, the Program co-creates and co-funds PPPs, GDA mechanisms, and other investment opportunities that support the growth of identified sectors and value chains and provide high-value employment for Georgians. Approaches under the PDF are collaborative, innovative, and flexible to identify and take advantage of opportunities. 1.2 EVALUATION PURPOSE The United States Agency for International Development (USAID)/Georgia requested that the Learning, Evaluation, and Analysis Project III (LEAP III) team conduct a hybrid (in-person and remote) performance evaluation of the Economic Security Program (the Program) being implemented by DAI and several Georgian and American partners on November 1, 2021. USAID/Georgia undertook this evaluation in the spirit of learning and adapting to better understand how the current program, which was originally designed under a previous strategy and under a very different geo-political and economic environment, could be guided to better advance the current Country Development Cooperation Strategy (CDCS) 2020-2025 and align with current USAID policies. Of note, the Program was conceived and designed prior to USAID’s Private Sector Engagement Policy and Digital Strategy, and prior to the outbreak of COVID￾19 pandemic which has had a dramatic and lasting impact on the economic sectors targeted by the Program. Considering these changes in the operating context, new strategic priorities, and new Agency policies, USAID/Georgia asked the evaluation team to gather evidence and provide recommendations for improvements in five key areas of interest: 1) private sector engagement, 2) value chain approach, 3) grant components, 4) policy coordination, and 5) COVID-19. The full description of the evaluation purpose is found in the Evaluation Work Plan, Annex B. Based on findings and conclusions, the ET outlined actionable recommendations to USAID/Georgia about any necessary adjustments to the implementation of the selected activities and future programming needs and approaches. These recommendations are drawn from the key findings of each EQ, looking at opportunities for change and possible adjustments that could enhance current Program activities and inform the planning for future activities in the USAID/Georgia economic growth portfolio. The recommendations are presented as potential areas of intervention based on the current needs in the post-COVID-19 recovery and country situation. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 16 1.3 EVALUATION QUESTIONS This evaluation assesses the effectiveness of specific programmatic approaches in achieving intended life￾of-program results. To accomplish this task, the ET has developed an evaluation approach to address a set of five EQs outlined in the Evaluation SOW found in Annex 1. TABLE 1: EVALUATION QUESTIONS 1. To what extent has the PDF targeted and established high-impact (defined as wide-reaching and/or replicable) partnerships with the private sector that have strengthened and catalyzed the development of priority value chains? To what extent are these partnerships sustainable (defined as the establishment of market linkages that will not depend on USAID assistance after the activity ends)? 2. To what extent has support to sector associations and government institutions catalyzed priority value chain development? How important a role have these stakeholders played in increasing value chain competitiveness? To what extent have these stakeholders received sufficient and relevant support? 3. To what extent has the grant component strengthened each priority value chain? To what extent did the grants address gaps or market failures in each value chain? 4. In the context of COVID-19 economic contractions, how can the activity adapt its approaches (e.g., selection of grant solicitation themes, division of USAID investments across sub-sectors, sequencing of interventions, etc.) to improve its ability to achieve its targets: creation of 4,800 jobs and achieving $60 million in new sales? 5. To what extent has the Economic Security Program coordinated effectively with other USAID activities (managed by both the USAID Economic Growth and Democracy, Rights, and Governance offices) to address the policy barriers facing its priority sectors and value chains? To what extent has the absence of a large policy component within USAID’s Economic Security Program helped or hindered its ability to address policy gaps? 1.4 EVALUATION AUDIENCE The primary audience of the evaluation will be USAID/Georgia’s Economic Growth (EG) team and the Implementing Partner DAI. USAID/Georgia may also share the results of this evaluation with local stakeholders, such as the Ministry of Economy and Sustainable Development, GNTA, Georgia Innovation and Technology Agency (GITA), partner nongovernmental organizations (NGOs), and other donors working in this area. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 17 2. METHODOLOGY 2.1 EVALUATION METHODS LEAP III conducted two simultaneous mid-term performance evaluations in Georgia; the second evaluation assessed the Agricultural Program. The two ETs worked closely to ensure strong collaboration and knowledge sharing across both evaluations. The ETs conducted joint key informant interviews (KIIs) with selected stakeholders from the GOG, USAID, and other donor partners to collect data efficiently. Both ETs used the same team structure, with an international evaluation lead and two national experts, a Georgian senior evaluation specialist and a PSE specialist. An economist and value chain expert conducted research on the economic impact of COVID-19 for both evaluations. All team members were briefed on USAID’s Human Subject Protection Policy and USAID’s Evaluation Policy. The survey team members were briefed on the rights and welfare of human subjects involved in the data collection protocols during the design phase of the survey interview guides. The ET was trained in survey methodology, USAID’s survey regulations, relevant regulations, and the data collection plan. The team utilized a mixed-methods approach, combining qualitative and quantitative methods to assess programmatic approaches, opportunities, challenges, and sustainability of Program investments. This approach reflects USAID’s similar combination approaches, which use two methods to collect and analyze information, then synthesize them to answer individual EQs. The ET held consultative meetings with the IP and USAID staff to gain inputs and solicit feedback during the design phase of the evaluation. Economic Security Evaluation Team Team Lead: Brenda Pearson Senior Evaluation Specialist: Maia Giorbelidze Private Sector Expert: Rati Gabrichidze Sector/Value Chain Advisor: Lasha Kavtaradze Facilitator: Rusudan Gogibedashvili Associate: Penelope Norton SECONDARY DATA The ET conducted desk research prior to fieldwork to identify and analyze secondary information that could be triangulated with data collected in Georgia. The ET conducted an extensive desk review of key program and external documents, including secondary data and background documents (relevant academic, periodical publications, other donor reports, project surveys, monitoring and evaluation plans, work plans, and quarterly and annual reports). These 30 documents provided a deep dive into the development context, challenges and priorities, economic policies, laws, and regulations (PLRs), as well as insights into the business enabling environment (BEE) and competitiveness in Georgia. The purpose of the desk review was to familiarize the ET with key activities and to build understanding of PSE to situate the evaluation. The ET worked with USAID/Georgia in advance to retrieve program documents and its own PSE strategy and partnership selection criteria. QUALITATIVE DATA COLLECTION The ET conducted fieldwork remotely from September 13 to October 27, 2021. The extended data collection period reflected the difficulties of scheduling KIIs during lockdown periods, when many informants were juggling professional and family responsibilities. The ET did not conduct in-person or on￾site observations due to the safety and security protocols in place. Most interviews were conducted through online platforms, such as Zoom and Google Meet. The primary data collection was conducted in USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 18 the Georgian language for most respondents; survey instruments are presented in English in Annexes II but were translated into Georgian prior to dissemination. Surveys and KIIs with USAID staff and other donor partners were conducted in English. The ET prioritized the main groups of stakeholders and beneficiaries most representative of the public, government, and private sectors across all regions of Georgia for its sampling, in consultation with USAID and the IP. The ET carried out 67 KIIs that included one to three stakeholders from the same organization for a total of 92 respondents drawn across all stakeholder groups. The majority are direct beneficiaries: 27 grantees, BSOs and interns, 16 GOG officials, and 11 partners participating in the PDF. KIIs consisted of nine IPs, eight USAID staff, four donor partners, and three financial institutions. The table below shows the affiliation of each key informant. TABLE 2: AFFILIATION OF KEY INFORMANTS CATEGORY NUMBER OF KEY INFORMANTS USAID 8 Implementing Partners 9 Other USAID-Supported Programs 11 Other Donor-Funded Programs 4 Government of Georgia 16 Beneficiaries (grantees, business associations, interns) 27 PPP and GDAs 14 Financial Institutions 3 Figure 1 shows the number of documents in the desk review and high response rate for the online surveys and KIIs. More than half (56 percent) of KII respondents were female; 44 percent were male. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 19 FIGURE 1: METHODOLOGY OVERVIEW The ET utilized the Program database (supplied by the IP) to establish a sampling frame for the online survey. In total, invitations to the online survey were sent to 222 beneficiaries, and the ET received a response rate of 67 percent for BSOs, 84 percent for grantees, 74 percent for MSMEs, and 62 percent for participants in the #Go4IT internship activity, which provides practical on-the-job learning environments for university students. The ET designed an additional survey to sample the perceptions of youth (ages 17-35) because of the Program interventions specific to youth-focused workforce development interventions. The high response rates were attributed to the timing of data collection and numerous follow-up emails sent to beneficiaries. The ET believes the high response rate increases the usefulness of the survey responses. DATA ANALYSIS The mid-term evaluation includes a comprehensive Getting to Answers matrix in the Evaluation Work Plan (Annex II) that maps the EQs to data sources and data analysis methods. The ET used a manual review process to extract key data such as keywords, quotes, and substantive information about activities from the transcripts and notes. The team sought to visualize results whenever possible. The ET used descriptive statistics to produce a quantitative overview of Economic Security Program activities, including characteristics such as the number of participants, regions, and in-country partners based on survey responses. The team examined qualitative data from KIIs and the online surveys to identify patterns, themes, and trends relevant to each EQ to better understand context and meaning. When the ET found divergence in responses through this thematic and content analysis, it explored possible reasons, using other respondent group interviews and, in some cases, conducting follow-up interviews with IP staff. The ET coded its notes according to key themes of interest across the interviews and summarized the distribution, number, and average responses by theme and respondents. The ET made use of the various data sources through a triangulation process to enhance the credibility of the analysis. Triangulation synthesizes multiple perspectives and leads to a fuller understanding of the issues being studied. Data from various lines of inquiry, including interviews, documents, analytical procedures, and other sources (e.g., the online surveys) were considered separately and together to develop findings and conclusions. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 20 2.2 LIMITATIONS In any evaluation, some inherent challenges and risks must be mitigated to obtain reliable data. The ET consulted biweekly with USAID/Georgia, utilized external resources, and conducted KIIs with organizations that did not directly benefit from the Program to better triangulate data. • Evaluation methodology related to COVID-19. The Asia Development Bank assessed the economic impact of COVID-19, and the ET engaged the same economist who conducted this research. Data collection and analysis related to EQ4 relied upon economic data provided by Geostat10 and other USAID programs. For EQ5 regarding PLR coordination with other USAID programs, the ET used data only from USAID-funded programs. • Selection bias. Since the ET relied on USAID/Georgia and the IP to identify and communicate with specific key stakeholders, selection bias was a risk due to the potential for selecting a large proportion of interviewees who held only positive opinions of the Program. Regardless, the ET reviewed all project documents and made decisions on whom to contact. Beneficiaries selected for KIIs were potentially more likely to fill out online surveys, which presents an overlap between these two groups of respondents. • Sampling representativeness. Many beneficiaries were representatives of SMEs, so some findings may skew toward the concerns of micro and smaller firms. The electronic survey methodology targeted BSOs supporting tourism, SIS, creative industries, light manufacturing, and SWMs rather than many individual firms because it related directly to EQ2 and EQ3. • Response bias. Most respondents have vested interests in maintaining positive relationships with the Program and want to continue to receive funding or other benefits. The ET mitigated this bias by opening every interview by assuring informants that their responses would be anonymous, any comments made would not be attributed to them unless they agreed, and there would be no retaliation against them or direct consequences for their responses. • Recall bias. The ET tried to overcome the bias of respondents attributing impacts to their individual experiences by incorporating best practices for qualitative data collection, such as framing questions that rely less on recall of specific activities and more on the currently perceived implications of those activities. 10 Geostat is the National Statistics Office of Georgia. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 21 3. EQ1 FINDINGS, CONCLUSIONS, RECOMMENDATIONS EQ1. To what extent has the PDF targeted and established high-impact (defined as wide￾reaching and/or replicable) partnerships with the private sector that have strengthened and catalyzed the development of priority value chains? To what extent are these partnerships sustainable (defined as the establishment of market linkages that will not depend upon USAID assistance after the activity ends? 3.1 EQ 1 FINDINGS According to the desk review and confirmed in KIIs, Georgian firms lack access to resources necessary to increase competitiveness and create greater employment opportunities in key sectors. These resources include high-value, diverse markets; investment resources; and a workforce that has the skills demanded by the private sector. The current COVID-19 crisis has led to a major economic recession in Georgia with the loss of jobs and devaluation of local currency. USAID/Georgia is actively working with a cross￾section of citizens and key stakeholder groups to accelerate the country’s transition to a democratic￾based and Euro-Atlantic oriented market economy, while improving the growth and competitiveness of private businesses. To achieve this transformation, especially considering the economic impact of the pandemic, USAID partners need to actively engage and collaborate with the private sector. The Program was established to build high-impact partnerships with the private sector to strengthen and catalyze the development of priority value chains. Although the Economic Security Program has a multi￾pronged approach to private sector engagement (PSE), the evaluation question drafted by USAID/Georgia focused on one aspect of the partnerships, namely the Partnership Development Fund. Therefore, the findings below are in direct response to this narrower PSE approach. Through this PDF activity, the program co-creates and funds PPP and GDA mechanisms. The PDF brought 16 new partners to USAID under the USAID New Partner Initiative, including the Bank of Georgia and the Adjara Group. 11 FINDING: FOUR MAIN CHALLENGES TO ECONOMIC GROWTH AFFECT COMPETITIVENESS AND SUSTAINABILITY ACROSS VALUE CHAINS To understand the current state of the Program’s partnerships and their sustainability, the ET developed extensive survey instruments to assess the factors that influence and hinder MSMEs’ competitiveness, measure the Program’s effectiveness and efficiency in its support to partners, and gauge the status of PSE (see EQ2 for more details). Various value chains showed resilience during COVID-19 while other value chains, such as tourism were adversely affected. See EQ4 for further analysis. The deal notes for each partnership were submitted and approved by USAID. They included an analysis of each of partnership’s return on investment and business goals of the partners. According to the IP respondents, the partnerships’ sustainability is reinforced through their design and financial goals. 11 See Annex 1 for the Mid-term Performance Evaluation Scope of Work and corresponding Evaluation Questions and sub questions. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 22 The ET disaggregated survey results by three stakeholder groups—BSOs, individual MSMEs, and Program grantees—to facilitate comparisons among stakeholder groups. (See Annex E). Figure 2 illustrates the top four challenges to economic growth: lack of qualified staff, access to finance, technologies, and administrative and policy barriers as identified by the ET. MSMEs (67 percent) and grantees (77 percent) cited the lack of qualified staff as the greatest challenge. About 71 percent of BSOs, on the other hand, cited access to finance as the greatest challenge impeding growth and competitiveness. Other noted differences related to marketing challenges, which nearly one-third of MSMEs cited as a major challenge, while BSOs and grantees rated the lack of technology as a greater challenge (64 percent and 27 percent, respectively). Only 7 percent of all respondents cited government PLRs as significant barriers, but a higher percentage of the 14 BSO respondents representing the sectors of light manufacturing, SWM, and SIS stated that GOG business tax rates were a major challenge. FIGURE 2: CHALLENGES TO ECONOMIC GROWTH AND COMPETITIVENESS BY SECTOR Source: Surveys of Grantees, MSMEs and BSOs (92 responses) FINDING: THE ECONOMIC SECURITY PROGRAM REPORTED THE CREATION OF 16 PPP/GDA AND A PIPELINE VALUED AT $57.4 MILLION IN FY2 Under the PDF component, the Program facilitated the conceptualization and development of 16 partnerships that leveraged multi-million-dollar resources to realize shared objectives in high-value job creation, increase MSME revenues, and diversify value chains. The Program integrated two types of partnerships into its activities—PPPs and GDAs. The number of GDAs and PPPs is equally distributed (eight of each). The co-creation process undertaken jointly by the IP and USAID/Georgia characterizes the GDA in its activity design. GDAs must mobilize and leverage private sector assets, expertise, USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 23 capabilities, and resources at a level that at least equals and preferably exceeds the value of resources provided by USAID. Under the Program’s PDF mechanism, the GDA model required extensive co￾creation and shared responsibility between USAID and the private sector. The threshold of private sector contributions mobilized by the partners was not as high for PPPs due to the nature of the PPPs’ leveraged funds. PPPs do not necessarily include GOG as the public sector partner but USAID participation is considered a public contribution. 12 About 42 percent of respondents stated that they had previously partnered with the GOG, and 78 percent of those respondents said they would be open to cooperating with the public sector. They identified five main factors for joining PPPs: 1) the potential to increase their share of the market, 2) greater transparency in obtaining GOG public procurements, 3) increased access to finance, 4) financial support through PPPs, and 5) fulfillment of corporate social responsibility objectives. The Program’s FY2 Annual Report states that its PDF activity will create 870 new high-value jobs in the future and is projected to generate $39.2 million in investment, $57.4 million in sales, and revenue in pipeline development over the contract’s period of performance. The Program’s PSE approaches were primarily evident through the PDF and grants mechanisms. Multiple private sector representatives expressed satisfaction with the Program’s programmatic approaches in addressing the top challenges. Nearly 83 percent of all survey respondents assessed cooperation with the Program as either “very useful” or “useful” (see as Figure 3). A slightly higher number of grantees (89 percent) and 82 percent of BSOs said the Program addressed the main challenges through its PDF activity or grants mechanism. 12 2021 Annual Report and reported indicator outputs. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 24 FIGURE 3: BSO, MSME AND GRANTEES’ RATING OF THE PROGRAM’S APPROACHES Source: Surveys of Grantees, MSMEs and BSOs (92 responses) FINDING: PDF INDICATORS AND TARGETS DO NOT DIRECTLY MEASURE TRANSFORMATIONAL/SYSTEMIC IMPACT OF PARTNERSHIPS ON INDUSTRY OR VALUE CHAINS The narratives in the FY2020 and FY2021 Annual Reports do not explain clearly how the partnerships are achieving key milestones nor do they describe the stages of development for each type of partnership. It was difficult to assess how the currently funded PPPs and GDAs measure their impact within industries and assess at this stage whether the partnership likely would deliver transactional or transformational results. The ET did not find references in the MEL plan to gauge the systemic changes affected by the partnerships through 1) scale of impact, 2) replicability, and 3) addressing market failures. The partnership criteria as understood by the ET, focused more on bringing new private sector partners to Georgia and linking with Georgian firms and institutions. The ET was informed that the IP targeted support to partnerships by utilizing quarterly statistical analyses provided by PMCG to determine the changes in systemic market dynamics. The deal notes had limited explanation how the Program incentivized partnerships for longer-term transformational impact. According to interviews with seven USAID staff, the deal notes do reveal how much each partner contributed financially or with in-kind services but do not illustrate how the specific sector or industry will be affected or influenced by the PPP or GDA. There is not a projection or estimation of how the partnership will influence market dynamics within specific sectors at this mid-point of implementation. This consideration of selection criteria for future partnerships is important because it helps to measure the achievements of each partnership and whether or not the IP’s approach solves systemic problems. Although each partnership is unique, there is a common need to articulate the end vision for scaling, USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 25 FINDING: PDF INTRODUCED NEW MSME PARTNERS AND NEW SECTORS TO USAID PROGRAMMING replication or institutionalization within the GOG or non-public entity and/or any linkages to educational frameworks. The Program’s PDF activity brought new sectors into USAID/Georgia’s economic growth portfolio in knowledge-based services such as SIS, creative industries, and SMW. Investment in these new sectors has been promising because of the potential for high-value job creation and new revenue streams that position Georgia in global markets. In many instances, the PDF activity successfully matched public and private sector interests to create new PPPs—such as the GNTA and Mountainous Tourism Agency to build a world-class snow park. There were evident challenges, however, in finding highly skilled workers to launch and sustain these new partnerships. The top economic growth challenge of skilled labor was more acutely pronounced in these new sectors. A second challenge relates to GOG and BSO capacity to meet the standards of foreign investors and global export markets, addressed under EQ2. High-growth sectors: One way the Program helps partners in high-growth sectors and value chains access finance is through its BizLink activity (see EQ3 regarding grants). The BizLink activity helped participating MSMEs develop a pipeline of nearly $44.2 million in potential sales and investments while increasing firms’ efficiency and export potential. Respondents who participated in the BizLink activity said its support for preparing finance applications and obtaining international certifications was helpful in increasing their market shares locally and internationally. The BizLink clinic, held in conjunction with the Bank of Georgia , provided customized financial, export, and management advisory services to 18 MSMEs. More than 90 percent of respondents described the training in preparing documentation to support loan applications as very helpful and relevant because it enabled rapid submission of applications to multiple lenders. A few respondents noted that the partnership could be improved by explaining how the consulting companies administering the BizLink Clinic protected the confidentiality of their financial data. For example, although Non-Disclosure Agreements were signed by participants and the consulting company (Savvy Consulting), it did not fully alleviate concerns about protecting financial data, as stated by the informants during the interviews. 13 Another example is the Georgia Women in Technology initiative, a GDA between GITA and the U.S. Market Access Center (USMAC). This initiative introduced female entrepreneurs to potential angel investors in Silicon Valley, California (See EQ2). This GDA is one of the Program activities specific to the needs of female entrepreneurs who need to access capital and investors (See section on cross-cutting findings for additional information). High-value tourists: The Program’s multi-faceted approach to attracting high-value tourists requires Georgian firms to raise the country’s visibility as a destination for European, regional, and North American tourists through Western-oriented marketing and promotional activities. It also supports the creation of niche markets for tourists interested in gastronomy, cultural and historical sites, adventure sports, and competitions. The PPPs and GDAs in the tourism sector are working with Georgian educational institutions and international investors to improve the skills of workers in adventure and eco-friendly 13 A first-time activity between USAID and the Bank of Georgia. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 26 tourism (Guria Tea Route, Mountain Adventure Trail) and hospitality (Marriott, Hilton and Georgian￾American Chamber of Commerce). Creative industry: In the creative industry, master classes held in partnership with Universal Pictures/Enkeny Firms, Cluster Films, and N&N studio have trained nearly 200 students in the highly specialized skills needed for the film production industry. All respondents expressed great satisfaction with the quality of the training. A few respondents said that all the courses are offered in Tbilisi and are not accessible to other youth living outside of the capital. The IP respondents, however, said the classes were live streamed and accessible online and that one course was held in Batumi. Thus, the Program may want to review its outreach efforts to ensure people are aware that the live streamed classes are accessible to youth who are not located in Tbilisi. FINDING: OUTREACH TO BOLSTER REGIONAL COOPERATION IS UNDERWAY While the Program was not required to engage with regional governments, there were multiple outreach efforts with regional governments. Only one regional government, that of the Adjara Region, was currently engaged via a grant for solid waste management in collaboration with the European Innovation Academy. The Key Management Solutions partnership focuses on workforce development training to improve the skills of hospitality workers. Further investment in the region was made through a grant to the Adjara Ministry of Finance and Economy that supports the expansion of SWM firms in recycling. Several GOG respondents suggested the Program should establish more local-level collaboration. The Program does interact with regional governments to some degree through the implementation of grant activities in different regions, although regional government level engagement is not the primary focus. 3.2 EQ 1 CONCLUSIONS The Program reported the creation of 16 Public-Private-Partnerships (PPPs) and Global Development Alliances at a projected value of $57.4 million. These partnerships leveraged multi-million-dollar resources to create high-value jobs, increase MSME revenues, and diversify value chains. Based on its desk review, data collection and analysis, the ET concluded these 16 partnerships contributed to strengthening of key value chains in tourism, creative industries, light manufacturing, shared intellectual services, which includes ICT and e-commerce. At the time of the evaluation, a new partnership in the solid waste management and recycling sector was recently launched so it is not possible to assess the longer-term impact of the Program’s investment in the SWM sector. Through the Program, partnerships were established in sectors that were new to USAID programming. One PPP, Digital Day with Steller/GNTA, was assessed very positively because it has already demonstrated high sustainability and the likelihood of replicability. The partnerships in e-commerce with Esty, Creative Industry Masterclasses, training of hospitality workers, and BizLin’s partnership with the Bank of Georgia also strengthened and catalyzed the development of priority value chains. In other cases, it was too soon in the project cycle to measure the extent to which each partnership is likely to be sustainable through scaling, replication and demonstrated impact. The partnership indicators established in the monitoring and evaluation plan, however, do not directly measure systemic and transformational changes within value chains and the degree to which such changes are sustainable. New indicators, however, were added in September 2021 to measure the impact of USAID Intermediate Results. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 27 In discussions with USAID/Georgia, the ET learned that new partnership selection criteria for selected economic growth interventions will include interrelated elements of scale, sustainability, and systemic change, including the assessment of whether partnerships address market failures, whether benefits could create market distortions, and demonstrate impact on the industry or value chain 3.3 EQ 1 RECOMMENDATIONS ● Consider adapting key milestones in approaches to developing partnerships so results are reported against indicator targets for PPPs and GDAs. Include a transition plan for how the partnerships will continue after USAID funding ends. ● Strategize with USAID/Georgia to modify or adapt new selection criteria related for future partnerships. ● Consider expanding the PDF approach to address systemic market failures to increase overall impact across industries. ● Replicate the Adjara Region’s approach to support more regional and municipal locations outside of Tbilisi in the solid waste management and recycling sector through formal partnering arrangements and/or grants. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 28 4. EQ2 FINDINGS, CONCLUSIONS, RECOMMENDATIONS EQ2. To what extent has support to sector associations and government institutions catalyzed priority value chain development? How important a role have these stakeholders played in increasing value chain competitiveness? To what extent have these stakeholders received sufficient and relevant support? 4.1 EQ2 FINDINGS The most important stakeholders that need to be involved in increasing the competitiveness of MSMEs in priority value chains are the private sector firms, GOG, financial institutions, educational institutions, and BSO. Program activities have primarily supported the GOG and BSOs - BSOs that serve the business community in Georgia are important conduits for improving MSME competitiveness. The Program’s investments in workforce development activities address the importance of linking educational institutions to employer needs. At the present time, the Program has two formal linkages with financial institutions through its PDF and grants mechanism. The ET triangulated data from the desk review, KIIs and surveys to assess whether the Program’s support to GOG and BSOs was aligned with priority needs and the degree of satisfaction with the technical assistance. Findings suggest that BSOs, in particular, can facilitate entrance into new markets and promote stronger linkages among enterprises to strengthen and enhance the overall value chain ecosystem. FINDING: OCCUPATIONAL CUSTOMIZED ASSISTANCE PROGRAM (OCAP) PARTICIPANTS REQUIRE MORE CAPACITY BUILDING TO SERVE THEIR MEMBERS Private sector respondents said that BSOs need to develop internal capacity to provide essential services to their members as a first step to increasing value chain competitiveness. According to the ET’s analysis of quantitative and qualitative data, BSOs are not currently positioned to offer critical services such as: business development planning, access to financing, market information, recruitment of qualified workers, financial documentation for loans and grant applications, and linkages across industries and with up-chain enterprises. In response to this need for capacity building, the Program launched the OCAP in December 2019 to build the sustainability of 21 BSOs by assisting them in strengthening their service offerings and membership bases as well as improving governance structures. The OCAP initiative is a three-phased participatory approach that focuses on “systematic institutional capacity building as a way to empower business associations to become key enablers of sector competitiveness.” 14 In 2020, the 21 targeted BSOs participated in at least one of two training sessions offered in Tbilisi and Batumi. Institute participants learned about international best practices for designing sustainability plans. The first phase also included assessment of the BSOs’ current capacity, a workflow development analysis, and one-on-one knowledge-sharing and technical assistance. BSO respondents who attended said that associations should articulate the benefits of membership and design tailored services to meet industry 14 USAID Economic Security Program Annual Report, October 1, 2020 - September 30, 2021, page 18. The OCAP approach is based on international best practices and draws from tools developed by the United States Chamber of Commerce Institute. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 29 needs rather than offering primarily broad advocacy and organizational capacity guidance. The second phase of the OCAP concentrated on the creation of sustainability plans that the organizations developed with Program assistance. After completing the first two phases, a qualified BSO can apply for cost-share grants. The current and final phase of the OCAP technical assistance focuses on implementation of the strategic plans. As per FY 2021 Annual Report, eight BSOs applied for cost-share grants to develop or improve their abilities to serve as competitiveness enablers within their sectors or within the wider Georgian business community. Grants for two of the organizations, Business Association of Georgia (BAG) and the Georgian ICT Cluster, were approved during the year, while four others remain under consideration. Two of the grant proposals did not meet Program criteria and two others also did not meet the technical and financial criteria. According to respondents, their BSOs had low capacity to support members that need financial assistance and access to new technology and equipment. Hence, the BSOs are at very different stages of maturity and a stratified approach to capacity building is likely needed for implementation of the OCAP, according to BSO respondents. BSO survey respondents gave high ratings to the OCAP exercise to improve their associations’ visions and organizational strategies. BSO KII respondents, however, provided more nuanced comments. More than 80 percent said that their BSOs required more specialized technical support for unique sectoral needs and to operationalize OCAPs. These same respondents cited the need for more support in business development processes through mentoring and coaching rather than direct capital investments at this stage. “If we speak about the value added by involvement in the OCAP program, we always knew about our challenges, but we hadn’t [had] them in a systemized form with deep dive analytics, and we hadn’t [had a] strategic action plan to follow up.” (Respondent who participated in OCAP program) “The OCAP is interesting in the component of strategy development, but it will not create any value if [the] implementation process is not supported. Without the support of the implementation process, it makes no sense to develop a strategic development plan. It would be great at the end of the strategy development component to teach us how to get an implementation grant, how to work on a grant application, how to convert our visions in a USAID-acceptable manner. This process is so complicated that companies like us are outsourcing the service of grant application fulfillment, which I think is wrong.” (Respondent who participated in OCAP program) Nearly half (10) of the targeted BSOs represent the tourism sector, three represent the creative industry, three represent SIS, two represent light manufacturing, and one represents SWM. Two of the BSOs, Business Association of Georgia and American Chamber of Commerce, represent all the targeted value chains. Prior to the pandemic, the vast majority of BSOs had a fee-based structure, but many have stopped collecting fees. According to 100 percent of BSO respondents, the Program’s focus on increasing sector competitiveness through the OCAP initiative was relevant (see Figure 4). The same respondents rated the sufficiency of Program support lower (67 percent), because “it was not enough,” as one respondent shared. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 30 FIGURE 4: STAKEHOLDERS’ ASSESSMENT OF PROGRAM SUPPORT AS SUFFICIENT AND RELEVANT Source: Surveys of Grantees, MSMEs, BSOs and #Go4IT interns (154 responses) FINDING: ABOUT 57 PERCENT OF MSMES ARE SATISFIED WITH TECHNICAL SUPPORT The ET found through its surveys that about half (54 percent) of the MSMEs cooperate with BSOs as members or customers. Of these respondents, slightly over half (57 percent) are satisfied or very satisfied with the services offered. In surveys and KIIs, respondents had many suggestions for the types of services that BSOs should offer to members. These include: 1) more interaction and interest in addressing key business challenges; 2) digital solutions; 3) support for accessing finances; 4) a platform for business associations to collaborate and share experiences; 5) marketing and sales promotion assistance; 6) international legal aid for exports; 7) mechanisms to improve the quality of services in tourism and introduction of innovative products; 8) research and development; 9) engagement with foreign associations for matchmaking; 10) assistance in finding new contacts to expand business networks; and 11) active involvement in legislative drafting and sharing concerns with policy-makers. Program support to the BSOs was designed to spur systematic changes to improve their capacity, relevance, and value to members (especially MSMEs) through support services, certification assistance, and marketing advice. Overall, only seven percent of respondents identified other aspects of the business enabling environment as a barrier with the important exception of BSOs supporting light manufacturing, SWM, and SI. A significant percentage of these respondents stated that GOG business tax rates were a major challenge. As noted in EQ1 findings, access to finance was the top challenge impeding value chain competitiveness but the Program is mainly focused on enterprise-level work and capacity building. Other USAID mechanisms and programs are designed to address regulatory issues and access to finance issues. Several BSO respondents expressed greater interest in receiving technical support to create market linkages and obtain more technical assistance in areas such as gaining knowledge of export procedures, which affect access to international markets. Three BSO respondents described the immediate need for USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 31 diagnostic and forecasting tools for job creation in SWM, e-commerce, and SIS in the post-pandemic economy. Given the varying degrees of organizational capacity and sector-specific support services across the 21 BSOs, there may need to be customized capacity building assistance that considers the varying degrees of organizational capacity. The ET found evidence that BSOs are fragmented and discordant at the national level. USAID and donor respondents observed that the existing dynamics between BSOs minimize the effectiveness of BSOs as agents of change to advocate for BEE reforms. The Program developed the Advocacy and PPD Resource Manual for BSOs and planned to conduct the Advocacy and PPD Course for BSOs, including OCAP beneficiaries, in year three. The activity will be conducted in cooperation with USAID’s Economic Governance Program. FINDING: ACCESS TO FINANCE ACTIVITIES ARE IN GREAT DEMAND GOG and private sector respondents unanimously connected limited opportunities in certain supply chains to a lack of skilled labor and competitive financing. MSMEs that participated in the surveys stated that they mostly utilize bank loans with high interest rates for financing and have low awareness of financial instruments or practices such as hedging exchange rates, factoring, and trade-financing of export products in international markets. The BizLink Clinic benefited 18 MSMEs, of which 13 companies were in light manufacturing, three were in creative industries, one was in solid waste management, and one was in the tourism sector. Of the participating MSMEs, nine firms improved financial management and business practices, 11 firms developed and pursued export facilitation strategies, and six firms expanded sales on local markets. The support has helped these MSMEs attract up to $1 million in onward investment. Multiple businesses are expected to receive expansion loans from the bank in addition to targeted technical assistance. As noted in Section 4, the BizLink program helped MSME participants prepare loan applications to the Bank of Georgia after some of them overcame concerns about the confidentiality of their financial information. Some of the same participants said that another entity, Enterprise Georgia, scored their applications favorably because of the superior preparation of their documentation and financial plans. FINDING: TARGETED SUPPORT INCREASED FEMALE PARTICIPATION IN THE ICT AND HOSPITALITY VALUE CHAINS The Program has two innovative initiatives to increase the participation of female entrepreneurs and women in STEM to raise capital and improve digital skills in an industry traditionally dominated by men. One such activity was the Grace Hopper ICT Awards, which was developed by the Program and supported by five private sector and donor partners. These awards highlight women’s involvement in the ICT sector by showcasing Georgian women and companies that have achieved leadership positions. In addition, the Program supported the Georgia Women in Technology initiative in partnership with the U.S. Market Access Center (USMAC) and in collaboration with GITA. These activities were designed to reduce the gender gap and promote STEM. Five women-owned businesses received technical assistance to improve their management practices, increase sales, expand and hire new employees, and attract new investments. At the time of the evaluation, the outputs from this partnership had not been reported. In another partnership with USMAC, three Georgian start-ups developed innovative products and improved their management practices. The beneficiary companies increased their sales by approximately $232,000, hired 11 new employees, and attracted approximately $400,000 in new investments. In addition, this USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 32 partnership developed relationships between Georgian entrepreneurs and successful Silicon Valley investors. The Program supported two women’s business organizations through the OCAP support to BSOs and grants. The Association of Businesswomen of Adjara (ABWA) was one of the 21 BSOs that received capacity building support. Of the 40 grants awarded to date, 8 were owned and 12 were managed by women although there may also be female senior directors or female key management staff. In addition to the aforementioned, the grants were awarded to BSOs that are led by women such as Georgian Arts and Cultural Center, Georgian Heritage Crafts Association, the Furniture Cluster, etc. The grant to the Women’s International Shipping and Trading Association (WISTA) focused on developing a regulatory framework and skills development for male and female maritime shipping agents, who participate in certificate-based training courses. In the wine value chain, the Program supported the training of 30 female, 40 male and 21 youth as cultural and wine guides. There was an equal ratio of women to men (11 women and 12 men) who participated in training related to SWM and environmental protection The tourism sector and production of artisanal products are important growth value chains for women’s employment and economic empowerment, especially in underserved rural areas. There were 248 female beneficiaries out of the 340 who participated in hospitality training conducted by Key Management Solutions (KMS) and the Alliance Group in the Adjara region, which should foster rural employment opportunities and create a sustainable industry that is attractive to international tourists. Women dominated the composition of training participants in the crafts, 135 out of 163 artisans, who participated in Future Workforce for Crafts Industry in 2021 as well as females comprising 16 out of 20 digital designers who participated in the iAtelier training program. The reporting did not, however, indicate whether the women involved in these activities were new entrants into these activities. FINDING: YOUTH WORKFORCE DEVELOPMENT ACTIVITIES ARE HIGHLY RELEVANT AND SUCCESSFUL Respondents said that the Program’s workforce development component enabled private sector companies to further develop the capacity of current staff or train youth as potential employees. The linkages between the private sector, government, and educational institutions were important for building a sustainable pipeline of highly skilled workers. Through its Training for Life activity, the Program facilitated partnerships between Georgian educational institutions and international universities and NGOs to improve curricula and better align education with the employment needs of targeted sectors. GOG respondents saw these education-oriented partnerships as important investments, because graduates of these certification and degree programs ensure that Georgian industries and workers comply with international standards. Included in this approach is the recognition that educational and vocational models need to link skills development directly with employment opportunities for youth. 15 Among the total sample of workforce development participant interns (#Go4it), 74 percent indicated they were “very satisfied” or “satisfied” with workforce development assistance. About 68 percent said that they were offered full-time jobs after their training and/or internships. In accessing information, 39 percent learned about this opportunity from their universities, 29 percent from Facebook, 23 percent from friends, 15 In Georgia, youth are designed as people between the ages of 18-35 years old. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 33 and only 5 percent from the announcement on www.jobs.ge. Figure 5 shows the level of satisfaction of private sector companies that participated in workforce development program collaborations. FIGURE 5: 95 PERCENT OF PRIVATE SECTOR COMPANIES FOUND WORKFORCE DEVELOPMENT ASSISTANCE RELEVANT OR VERY RELEVANT (N=62) Source: Surveys of #Go4IT internship provider companies (62 responses) Furthermore, as shown in Figure 6, 79 percent of survey respondent interns shared that they found participation in the program useful for acquiring new skills, and 73 percent expected these new skills to help them find full-time employment. FIGURE 6: 79 PERCENT OF INTERNS FOUND THE WORKFORCE DEVELOPMENT #GO4IT PROGRAM USEFUL Source: Surveys of #Go4IT internship provider companies (62 responses) FINDING: THE GOG WAS VERY SATISFIED WITH THE PRIORITIZATION OF KNOWLEDGE-BASED SERVICES AND TARGETED GLOBAL TOURISM GOG respondents said they were very satisfied with Program cooperation, prioritization of knowledge￾based services, and targeted global tourism activities. One GOG respondent stated that the Program’s USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 34 most significant support related to introducing modernized marketing, trade promotion approaches, and products that would appeal to Euro-Atlantic consumers and tourists. The GNTA and tourism-related agencies within the MoESD benefited during the pandemic through coordinated social media posts and marketing campaigns and gave high marks for the Digital Day in the Life of Georgia initiative, which streamed content and reached 7.8 million potential visitors. Program staff said this campaign could potentially result in a global media boost with estimated revenue in the range of $20 million in tourism￾related dollars, which may not be realized in 2022 due to COVID-19. The Program made significant contributions to the development of Georgia’s National Tourism Recovery Strategy and Action Plan. In essence, the Program helped craft a one-year “reboot” marketing strategy to position Georgia as a safe and viable tourist destination. The Program facilitated the formation of a team of industry experts to guide and support GOG agencies over the next year. Through its partnership with Green Team Global, the Program supported the development of a crisis communications toolkit and other marketing-related promotions. Through 2021, 32 tourism companies signed agreements to participate in the Tourism Matching Fund, which provides customized social media outreach to target markets. According to Noxtton, the public relations and marketing firm that is working with Fund participants, 14 of the participating companies started their campaigns in July 2021. It is estimated that about 3.1 million potential customers in target markets have been reached with a low Cost Per Result (CPR) of 0.11. Through the partnership initiative with Key Management Solutions (KMS) in the Adjara Region, 430 individuals were trained, providing a steady supply of skilled workers for hotels in West Georgia. KMS conducted nine training courses, four of which are certified by the American Hotel & Lodging Education Institute. The partnership was designed to place at least 85 percent of training graduates in high-value hospitality jobs in Adjara and the surrounding regions. KMS, together with the Program, partially absorbed training costs and invested staff time to build relationships with key employers and hotels in the region. 4.2 EQ2 CONCLUSIONS Vibrant ICT growth within the SIS sector is viewed by respondents as critical to generating quality jobs. Georgia’s SIS cluster presents great potential for quick growth and high earning potential, and modernizes the country’s economy, governance, and society. The ICT sub-sector under shared intellectual services recorded exponential development and is estimated to have achieved nearly 33 percent growth in real outputs per worker prior to the pandemic. Support to the SIS and creative industries through partnerships, grants and BSOs resulted in increased revenues and new job creation for partners and grant recipients. Many respondents noted that PPPs in these two sectors could be a game-changer for the Georgian economy. The Program initiated multiple efforts to promote the development of Georgia’s e-commerce sector and assisted 50 MSMEs to varying degrees in developing e-commerce sites and 10 MSMEs in migrating to digital platforms. The partnerships formed between GOG and the private sector, which have joined forces to prioritize SIS modernization, could be a game-changer for the Georgian economy. During the recovery period, the Program may want to assess whether other approaches to the SIS and creative industries would increase its potential. Similar programs in the region (Ukraine and Moldova) have focused on creating development synergies to support continued growth of SIS and competitiveness in foreign markets by developing a sustainable talent pool to match industry requirements. This is done by raising USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 35 the profile of STEM subjects and positioning the country as a leading exporter of high-value services while crafting an enabling business and entrepreneurial environment. Overall, the Program supported professionals and businesses to understand best practices and integrate e-commerce into their operations. The Program partnered with the Business and Technology University (BTU) to create the iOS Applications Laboratory, which invests in youth. Partnerships with Cinema 13 and N&N studio will help Georgia establish the only film laboratory and post-production facility in the region that is supported with highly trained technicians in post-production services. As noted in the finding, the Program should consider adjusting its support to BSOs so they can partner with educational or innovation centers to deliver digital content, online learning solutions, educational tutorials, and entrepreneurship training to attract more youth to high-paying jobs. Prior to COVID-19, the tourism sector was poised for impressive growth in terms of attracting high value tourists. The Program had designed a multi-pronged approach for partnerships to co-invest in a digital media advertising campaign through the Tourism Matching Fund, increase the capacity of private sector firms and Mountain Trails Agency staff to offer outdoor sports venues that were competitive with European ones, and provide training to hospitality workers and tour operators. The partnership with Steller also increased the global visibility of Georgia as a tourist destination. As pandemic-related travel restrictions are eased, these early investments will benefit from continued technical support and engagement with the GNTA. 4.3 EQ2 RECOMMENDATIONS ● Consider providing a wider range of capacity building support to BSOs so that higher-capacity BSOs access training in supply chain linkages and procurement in supply chains, while lower￾capacity BSOs continue training in leadership, change management, marketing, branding, relationship building, and membership services. ● Utilize findings from survey data regarding BSO membership services, such as: 1) more interaction and interest in addressing key business challenges; 2) digital solutions; 3) support for accessing finances; 4) a platform for business associations to collaborate and share experiences; 5) marketing and sales promotion assistance; 6) international legal aid for exports; 7) mechanisms to improve the quality of services in tourism and introduction of innovative products; 8) research and development; 9) engagement with foreign associations for matchmaking; 10) assistance in finding new contacts to expand business networks; and 11) active involvement in legislative drafting and sharing concerns with policy-makers. ● Continue guiding the GOG and affiliated public and private sector tourism entities in implementing the National Tourism Recovery Strategy and Action Plan. ● Align expectations of the partnership between the GOG and the IP so both sides understand the purpose of the activity and what future technical assistance might entail at the strategic and operational levels. ● Utilize PDF, grant and technical assistance to individuals to do more cross-marketing and capacity building in regions beyond the Adjara region; expand efforts to identify and leverage government and the private sector partnerships in the regions and selected municipalities. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 36 5. EQ3 FINDINGS, CONCLUSIONS, RECOMMENDATIONS EQ3. To what extent has the grant mechanism strengthened each priority value chain? To what extent did the grants address gaps or market failures in each value chain? 5.1 EQ3 FINDINGS The USAID Economic Security Program issued 40 grant awards with a total value of $1,352,858 and a cost-share contribution of $1,072,528. The Program adjusted the targets of its grant mechanism in response to challenges created by COVID-19, especially in sourcing equipment and meeting the 60 percent cost-share threshold. Indeed, the Program worked closely with USAID/Georgia to reduce the beneficiary cost-share to 30 percent of the grant award. The grants cut across all sectors and value chains, although not equally. Grants to value chains in emerging knowledge-based services received significantly less funding than light manufacturing, and four grants were disbursed to tourism value chains. The grant mechanism was recalibrated to address challenges in the tourism sector due to the pandemic. The available grant data provided insights on the distribution of grant funding but there is limited data measuring how grant use or expenditures generated outputs to achieve desired outcomes for greater impact in value chains at this mid-point of implementation. The limited data available also makes it difficult to discern the extent to which grants may have been a catalyst for growth across sectors during the pandemic. The Program MEL plan does not have standard or custom indicators to measure how grantees’ programmatic interventions and approaches address gaps or known market failures in value chains. For example, including a qualitative custom indicator measuring the extent to which the grant support was sufficient to overcome identified systemic challenges would improve reporting. Without this type of routine monitoring and reporting, it was difficult for the ET to fully answer this evaluation question specific to each value chain. The findings presented below are based on available data and what the ET was able to ascertain from its review of the Program MEL plan and use of evaluation tools. FINDING: COVID-19’S ECONOMIC IMPACT MAKES IT DIFFICULT TO MEASURE HOW GRANTS STRENGTHENED THE CREATIVE INDUSTRY Due to the economic impact of COVID-19, it was difficult to measure the extent to which the grants strengthened each targeted value chain. Equally problematic was assessing how the 40 grant awards addressed gaps or market failures because of the pandemic. For example, many gains in tourism were wiped out due to national and global lockdowns, restricted air travel, and supply chain disruptions. Grantee respondents in the tourism and creative industries were the hardest hit by the pandemic. Grantees in the creative industries stated that the grants helped finance the purchase of technical equipment and software and supported the animators’ salaries. Expectations for the effectiveness of technical support in the creative industry were high, perhaps due to the newness of USAID engagement in this sector. The Program, through a grant with Cinema 13 for example, provided support for the development of a Trusted Partner Network to establish a post-production company and other employment opportunities. In Year Three, these grantees expressed a need for significantly increased USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 37 funding. The ET found that most of these grantees had expectations that funding levels would be higher. Three respondents from the creative industry stated: “Existing support by USAID isn’t enough to cover the full scope, which prepares staff with large-scale and versatile skills to enter the local and international market. To create qualified staff in this industry, it is important to reach large numbers [and] increase the intensity as soon as possible, because the demand for this type of staff is quite high.” “What has been done in the film industry in this regard is very minimal, and it is difficult to say that it will contribute to the development of the industry. There is a need of a holistic approach.” “The program in this project supported just the execution of the animation process, which is valuable. But there are still several challenges that Georgian film has in different directions and need to be supported: new generation education enhancement, staff qualification, regional coverage, absence of film distribution system, etc.” According to respondents, the services industries suffered more revenue losses than did producers of goods. SWM industry respondents reported a 100 percent increase in revenue, while creative industry respondents reported that half of their industry’s employees lost their jobs in 2020. FINDING: NEARLY HALF OF GRANTS SUPPORTED THE LIGHT MANUFACTURING VALUE CHAIN Grants were not distributed equally across sectors. Figure 7 shows the disbursement to each value chain, which is dominated by equipment purchases for light manufacturing (42 percent of the total grant value). In the overall distribution of grants, only a small percentage of grants went to value chains that employ high-value skilled workers, such as software developers, IT specialists, or film industry animators. This distribution of grants may affect the number of high value jobs created in these sectors but as noted earlier, this was not the priority selection criteria for grants. FIGURE 7: PERCENTAGE FOR GRANT FUNDING TO SPECIFIC VALUE CHAINS Source: List of grantees provided by USAID Economic Security Program USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 38 Analysis of quantitative and qualitative data both showed that nearly all grant recipients used the financial support to expand their operations. One grant recipient noted, “We could not really implement this project with our own resources because it is quite an expensive service, and the cluster member companies could not pay the fee. So USAID financial support was very important.” The size of the grant award was sufficient to expand their operations, according to 63 percent of grant recipients. A little over half (53 percent) of grant recipients started selling their products to new markets, and 70 percent established new partnerships with other private sector companies. FINDING: THE GRANT APPLICATION PROCESS COULD BE STREAMLINED AND WOULD BENEFIT FROM THE INVOLVEMENT OF BSOs Many grantee respondents commented on the lengthy period from grant application submission to agreement signing as the most cumbersome aspect of the process. About one-third of respondents said the process was usually burdensome due to frequent communication with Program staff, which prolonged the process. Delays caused financial consequences for some applicants, and budgets required repeated revision due to exchange rate devaluations caused by the pandemic. About one-half of the respondents who received grants said the Program did not provide sufficient support during each step of the grant process. These same respondents expressed concerns about the selection criteria for awards and said the process would benefit from more transparency. They suggested the application process should be simplified, which could potentially be done through modifications to its grants manual in consultations with USAID/Georgia. About 71 percent of grantee respondents indicated that bureaucratic procedures delayed the launch and/or implementation process. “The worst thing we remembered in the engagement process was that there were new questions coming up in non-stop mode. It would be great if a unified list of these questions were prepared at the beginning of the process, and we could complete the review process in a maximum of two months.” “I would recommend optimizing the grant disbursement process, as it’s the slowest process I’ve ever met during my grant funding experience.” Figure 8 depicts grantees’ observations regarding the grant application process. The online survey data responses were more favorable than key informants who were interviewed, which is often the case in mixed methods approaches. Interviewees provided more detailed criticism of the length of the application process. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 39 FIGURE 8: MORE THAN HALF OF GRANTEES IN SURVEYS FOUND THE APPLICATION PROCESS EASY AND TRANSPARENT Source: Surveys of Grantees (26 responses) As noted by the respondents from larger scale associations during the interviews, potentially relevant clusters and associations could support smaller-scale enterprises with grant proposal preparation by explaining the requirements for grant disbursement and reporting requirements that are part of the application process. The BSOs were limited in their technical capacity to provide support in drafting grant applications according to respondents, as illustrated below: “We [larger-scale and more developed associations and clusters] have more capacity and knowledge to develop the winning-grant proposals. But we can also help our members, who are not capacitated enough to fill out the forms. and the questions asked in the proposal template are usually too complicated for them. We have a role, to sit down with them and explain the required information in a more user-friendly way” 5.2 EQ3 CONCLUSIONS The USAID Economic Security Program issued 40 grant awards with a total value of $1,352,858 and a cost-share contribution of $1,072,528. Each grant activity has indicators that focus on two outputs: increased revenue and job creation. Nearly all (95 percent) of grantees used the funds to expand operations and the reported outputs related to increased revenue. The number of high-value jobs created varies greatly across sectors. In tourism, for example, some grants supported developing Destination Management Organizations (DMOs), which has potentially a modest impact on strengthening the tourism value chain. Other tourism-related grants supported hotels in developing their online booking platforms and product development. On the other hand, the grants designated for SIS were viewed by respondents as significant investments in building the capacity of Georgia’s educational institutes to provide state-of￾the-art ICT and software development training through the Ilia State University and Georgian American University. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 40 All the grants distributed in the light manufacturing sector were for equipment purchases. At the time of the evaluation, however, there were many disruptions in the supply chain that prevented the timely delivery of the equipment. Presumably, after the equipment is delivered and operational, it will strengthen the industry’s competitiveness. Grants in the creative industry mainly supported the purchase of software licenses and equipment for the new post-production facility, which will be the only one in the region. There were no grants awarded in the solid waste and recycling sector at the time of the evaluation. The data outlined in the Annual Reports indicate that several grantees have already generated employment (Wine Club, E-commerce Association of Georgia, Griffin), however, there is limited evidence that indicates recipients used grants in part or wholly to address main gaps or market failures in value chains related to finance, skilled workforce, technology, and equipment. One significant example was a grant initiative to fund the business-to-consumer and business-to-business activities that are expected to relieve pain points related to complex software development within the e-commerce value chain. It is too early to determine the impact of these grants. The Program’s MEL plan and reporting provided limited information on how the grants were used wholly or in part to address main gaps or market failures in value chains related to finance, skilled workforces, technology, and equipment, except in the e-commerce value chain . Although these outcomes were not explicitly stated in the original SOW, the USAID Mission is increasingly looking at how USAID-funded grants can close gaps and/or have an impact on a specific industry. 16 5.3 EQ3 RECOMMENDATIONS ● Improve grants’ effectiveness and sustainable gains by explicitly requesting that grantees explain how they will leverage the USAID-funding to obtain more financing and support from other sources. ● Encourage more innovative applications from consortia that link education and workforce skills development to balance grant disbursements, which tilt heavily toward equipment purchases for light manufacturing (in total grant value). ● Replicate strategic partnering with education institutions such as the GDA with Sweeft Digital and Ilia State University through targeted grants. ● Provide more customized support to grant applicants in each step of the application process and explain the selection criteria to increase transparency of the award. ● Streamline and improve the processing of grants by working with USAID to adjust selection criteria and post-COVID 19 cost-share obligations. ● Expand network and outreach efforts with educational institutions and BSOs located outside Tbilisi to obtain more gender and geographic diversity in grants and partnering organizations. ● Weigh selection criteria to favor applicants that clearly demonstrate how their proposed activities will address gaps and market failures in each value chain. 16 Implementing Partner introduced new indicators to address this issue after this mid-term evaluation draft report was shared with USAID/Georgia and the IP. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 41 6. EQ4 FINDINGS, CONCLUSIONS, RECOMMENDATIONS EQ4. In the context of COVID-19 economic contractions, how can the activity adapt its approaches (i.e., selection of grant solicitation themes, division of USAID investments across sub-sectors, sequencing of interventions, etc.) to improve its ability to achieve its targets: creation of 4,800 jobs and achieving $60 million in new sales? 6.1 EQ 4. FINDINGS The past two years (FY2020 and FY 2021) have been challenging for Georgia and the world. The COVID￾19 crisis led to an economic recession in Georgia, resulting in loss of jobs and devaluation of the local currency. It is estimated that the country’s economy contracted by 6.2 percent in 2020. At the height of the crisis, in April and May 2020, about nine percent of formally employed workers lost their jobs and about 370,000 self-employed workers registered for unemployment assistance. The GOG swiftly mobilized about $2.3 billion from the International Monetary Fund and other international financial institutions to bolster vulnerable sectors including tourism, agriculture, and real estate. Other support came from an influx of remittances, along with fiscal stimulus measures. 17 The GOG’s mitigation of the catastrophic impact included lockdown measures, immediate travel restrictions, and border closures in early 2020. When COVID-19 was officially declared a global pandemic, the GOG banned flights to and from China, Iran, and other countries. In March, the country declared a state of emergency and introduced a strict quarantine, along with other mobility restrictions that led to breakdowns in supply chains and economic structures (similar to the global experience). Domestic travel during the summer months, and the parliamentary election, led to a rapid surge in COVID-19 cases, and the GOG imposed additional lockdown measures from November 2020 to February 2021. Inflation rose to 5.2 percent above the National Bank of Georgia’s 3.0 percent target in 2020. Georgia’s consolidated budget deficit widened from 2.1 percent of gross domestic product (GDP) in 2019 to 9.3 percent of GDP in 2020, and government debt increased 40.4 percent in 2019 to 60.0 percent of GDP in 2020. The pandemic negatively affected nearly all aspects of Georgia’s economy, especially the tourism sector, which remains a key component of contract and Program targets. At the same time, the shift toward utilizing SIS, especially in the e-commerce, SWM, and ICT value chains created opportunities for Georgian businesses to adapt. GOG and private sector respondents noted that the Program remained operational, pivoted rapidly to adjust its activities, and delivered interventions effectively through online platforms while supporting partners and grantees. FINDING: COVID-19 CREATED LOSSES AND OPPORTUNITIES IN KEY SECTORS SUPPORTED 17 The ET conducted research on the overall impact of COVID-19 on the sectors supported by USAID’s Economic Security Program and Agriculture Program. The research was based on data provided by the National Statistics Office of Georgia (Geostat). The ET’s findings aligned with ongoing research and economic analysis conducted by the Asia Development Bank. The ET presented its analysis to the USAID/Georgia Mission on October 28, 2021. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 42 The Program’s targeted industries were among the most severely affected in terms of lost jobs and revenue. As seen in Figure 11, the number of formally employed workers in all sectors fluctuated from 2017 to 2019 as workers left agricultural jobs for employment in sectors related to tourism, such as real estate, accommodation and food service, construction, transportation, and storage. The ICT sector also contracted slightly prior to the pandemic. Georgia’s emerging creative sector is also vulnerable, as all cultural and educational events were canceled. TABLE 3: EMPLOYMENT IN KEY SECTORS PRIOR TO THE PANDEMIC 2017 2018 2019 2018 2019 000' persons % change year over year Agriculture 289 254 247 -12.3% -2.5% Industry 154 154 147 0.0% -4.4% Construction 84 99 101 17.3% 2.7% Trade 175 185 196 5.9% 5.9% Transportation and storage 70 78 82 11.5% 4.9% Accommodation and food service activities 38 44 49 16.9% 10.1% Information and communication 22 21 19 -3.9% -8.8% Real estate activities 3 4 4 43.8% -11.9% Arts, entertainment and recreation 27 28 30 5.8% 5.2% Other 425 429 421 0.8% -1.9% Total 1,287 1,296 1,296 0.7% 0.0% Source: Geostat FINDING: TOURISM SECTOR IS HARDEST HIT BY COVID-19 According to the desk review and KIIs, tourism was one of the fastest growing sectors in Georgia in the pre-COVID-19 period. The sector benefited from visa-free travel from more than 100 countries, improved accessibility, and government support and services. Georgia offers beach, winter ski, and four￾season resorts; health and medical destinations; wine tourism; cultural attractions; and gaming facilities, which transformed tourism into the country’s key service sector. Prior to the pandemic, according to USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 43 USAID respondents, it was anticipated that Program support to the tourism sector would be “graduated” from the Program because the sector no longer required USAID funding. The Program took a five-fold approach to the sector: 1) supporting multiple awareness-raising and digital media campaigns to raise the global visibility of Georgia’s tourism sector; 2) developing partnerships with 32 tourism companies to improve capacity, product development, and niche marketing; 3) designing skills training for hospitality workers; 4) enhancing the BEE for competitiveness through support to 21 BSOs in all sectors; and 5) guiding the GNTA to better plan and target key markets. These interventions, however, likely require recalibration going into Year 3 of the contract to align with Georgia’s recovery. The COVID-19 pandemic has been disastrous for Georgia’s tourism industry. According to the estimates, the country lost around $2.8 billion in tourism revenues in 2020, based on revenue projections for the year without considering COVID-19 and the acceleration of domestic tourism growth in the third quarter of 2020. The steep decline in international tourists visiting Georgia from April to December 2020 had a strong impact on hospitality and food services, construction, transport, and real estate. Revenues in the accommodation and food service sectors decreased by 61.0 percent (year over year) in 2020. The tourism sector’s value-added contribution to overall GDP shrank by 38 percent. In 2020, according to Geostat, 12,800 people lost jobs in the hospitality industry, and corresponding productivity fell by 15.8 percent. Foreign investors sold their shares in Georgian hotels and restaurants; it is estimated that about $249.5 million in foreign direct investment was withdrawn from the tourist industry in 2020. Georgia saw a nearly 96 percent reduction in tourism revenue and 80 percent decline in the number of tourists from 2019 to 2020 (see Figure 9). As the pandemic progressed, the steep decline in the number of international tourists visiting Georgia in 2021 compared to 2019 became partially offset by a marked increase in domestic travel as people relocated temporarily. According to the GNTA in September 2021: “Georgia hosted 266,544 visitors in August 2021 which is a 510% increase compared to the same month of 2020, however it is still 75% less compared to August 2019, said. the Georgian National Tourism Administration.” In July 2021 Georgia received $205.3 million. In total in the first seven months of 2021 Georgia received $505.1 million from international tourism.”18 18https://agenda.ge/en/news/2021/2552#:~:text=Georgia%20hosted%20266%2C544%20visitors%20in,the%20Georgian%20Natio nal%20Tourism%20Administration USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 44 Source: National Bank Georgia, Geostat FIGURE 9: COVID-19 IMPACT ON THE TOURISM SECTOR SEEN THROUGH COMPARISON OF INTERNATIONAL VISITORS AND ANNUAL TOURISM REVENUES Source: GNTA The situation would have been worse but for GOG support that deferred income taxes for all affected businesses, subsidized loan interest rates for small hotels for six months, supported travel agencies and guides, exempted the tourism industry from property taxes, and made payments to employers who retained their employees. These interventions prevented massive closures, but there is no guarantee that hotels will remain operational when the benefits end. About three-quarters of the total contraction of the Georgian economy is either directly attributable to changes in the tourism sector or closely linked to transport, trade, arts and entertainment, real estate, and construction, jeopardizing the economic growth model that was the driver of FDI in Georgia. GOG and private sector stakeholders characterized Program support to GNTA as timely and critical in creating a Post-Recovery Plan and Action Plan. Three GOG respondents said that Program support helped them navigate challenges and assess responses to the Russian air embargo during the pandemic. Continued support to the GOG and PSE, including BSOs, during this critical period filled an important gap for the tourism sector and contributed more broadly to Georgia’s economic recovery model, according to GOG and donor respondents. Stakeholders in the tourism sector, BSOs, grantees and partners stated their beliefs that a revamped tourism strategy and legal framework would be crucial to the eventual recovery of the sector. Two respondents noted that Georgia requires assistance in adopting new health and safety protocols across the industry for both domestic and international tourists. Three respondents representing light manufacturing and SWM cited the urgent needs for equipment and skilled workers. FINDING: SIS AND CREATIVE INDUSTRIES ARE POTENTIALLY IMPORTANT SECTORS FOR JOB CREATION AND ENHANCED REVENUE STREAMS According to respondents in the SIS and creative industries sectors, the most pressing challenge is to help MSMEs and new clients maintain their digital presence, develop online commerce, and strengthen their digital marketing to meet domestic consumers’ demands during this downturn in the economy. Several private sector respondents noted that Georgian businesses benefited from support that integrated real￾time ordering and payment solutions that facilitated interaction with customers and improved the efficiency of online purchases. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 45 In the past two years, many companies migrated to the digital environment because the pandemic accelerated the need for industries and businesses to embrace digital technologies as critical tools for business continuity and sales recovery. Prior to the onset of COVID-19, the Program had deployed high￾impact digital promotion in tourism and creative industries to increase international awareness of Georgian products in European Union and global markets. The SIS sector plays a significant role in the Georgian economy. Its contribution to real GDP growth in 2019 of 11.2 percent has doubled since 2015. This sector contracted significantly in 2020 as evidenced by a decline in revenue as a percentage of GDP. The emerging creative sector is vulnerable, as the small or micro firms that dominate the industry experienced about 25 percent overall job losses (see Figure 11). According to Geostat, the creative industries experienced an increase in revenue of nearly 108 percent in 2019 but contracted significantly in 2021. Wages in this sector increased in 2020, most notably in computer programming and information services. The potential for growth in terms of skilled labor outputs seems most promising in broadcasting, telecommunications, and information service activities, including e-commerce. FIGURE 10: PERCENTAGE CHANGE IN EMPLOYMENT FROM IN SHARED INTELLECTUAL SERVICES AND CREATIVE SECTOR FROM 2020 TO Q1 AND Q2 OF 2021 Source: Geostat According to the evaluation, use of e-commerce for shopping has increased by 2.2 percentage points to 20.8 percent of the total adult population since 2016. Importantly, there has been a sharp increase in internet use in rural areas, which nearly doubled from 2016 to 2019. Domestic sales utilizing e-commerce and digital technology have risen, but there has been scant evidence of regional and international sales due to cross-border price competitiveness and poor logistics. The Program initiated multiple efforts to promote the development of Georgia’s e-commerce sector and assisted 50 MSMEs to varying degrees in developing e-commerce sites and 10 MSMEs in migrating to digital platforms. One grantee stated that its operations increased by 150 percent during the pandemic. The e-commerce initiative with the E￾Commerce Association—Georgia and TBC Bank led to the launch of the country’s first e-commerce academy, which was expected to create a pipeline of skilled labor. There were no new employment figures available during the evaluation period. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 46 The Program invested in partnerships with Sweeft Digital and the Business and Technology University to provide apprenticeships in SIS applications for approximately 1,100 youth. Of these, it is anticipated that 800 will gain full-time employment, which would generate more than $4 million in new payroll through 2024. Currently, the first cohort of 100 apprentices has been recruited and is undergoing a six month of intensive, hands-on training. Based on KIIs with partners and participants of this training, it is not clear at this juncture during the pandemic, however, whether the Program will meet its target goal of youth who will find full-time employment after their apprenticeships. Overall, the Program supported professionals and businesses to understand best practices and integrate e-commerce into their operations. The Program partnered with the Business and Technology University (BTU) to create the iOS Applications Laboratory, which is expected to create 65 high-value jobs and generate a new payroll of $431,000. By investing in youth, this partnership could potentially strengthen Georgia’s ICT sector and create high-value jobs well beyond the partnership. Although in the early stages, the partnership with Cinema 13 potentially has a multiplier effect by positioning Georgia as a provider of post-production services. The partnership will employ 20 new staff in the first year of full operation and is expected to generate approximately $4.5 million in sales over the next three year through film restoration and post-production services. The aim is to create a film laboratory that will have the capacity to restore classic movies and other video content shot on 16mm and 35mm film. This will be the only center in the region that can provide high quality post-production services to local and international production companies, film centers, and archives. On a smaller scale, the Program partnered with N&N Studio, a Georgian film production company, to fund on-the-job training for eight Georgian animators Two BSOs suggested the Program should fund BSOs to partner with educational or innovation centers that can deliver digital content, online learning solutions, educational tutorials, and entrepreneurship training to attract more youth to high-paying jobs. These comments strengthen the earlier findings that partnerships with educational institutions are valued highly by private sector respondents. FINDING: NEW OPPORTUNITIES IN THE LIGHT MANUFACTURING AND SWM SECTORS Prior to the onset of COVID-19, USAID/Georgia prioritized expanding interventions in the e-commerce, tourism, light industry and ICT sectors. As noted in the Introduction chapter, the IP’s June 2020 contract modification selected the additional sector of solid waste management and recycling for assistance. Through its BizLink activity, the Program supported 11 light manufacturing companies with grants to purchase equipment and obtain international certifications. According to the IP respondents and confirmed by grantees working in these sectors, the Program prioritized skills development in this value chain to generate new employment in packaging, furniture, and personal protective equipment. Early support for the SWM sector consisted primarily of designing a joint grant venture with the Adjara regional government to develop a pilot initiative for recycling tires, wood, and waste. SWM, sewerage, and the recycling sector’s share of GDP was minimal, accounting for only 0.8 percent of GDP from 2015 to 2019. There is not enough data to analyze the impact of COVID-19 on the sector but several GOG respondents said that the government would prioritize these sectors in the near future. Manufacturing was an important sector of the formal Georgian economy in the pre-COVID period, accounting for ten percent of GDP in 2015–2019 and employing nearly eight percent of the total workforce in Q1 2020. However, employment in the sector decreased sharply as COVID-19 caused global supply chain disruptions and reduced demand for manufactured products. Overall, there has been mixed USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 47 performance across the manufacturing subsectors. The share of FDI in the sector is low, and products are not sufficiently diversified for export because the manufacturing process is fragmented, and companies need to adapt new technologies to reduce costs and improve quality. FIGURE 11: PERCENTAGE OF CHANGE IN EMPLOYMENT BETWEEN 2020 AND Q1, Q2 2021 Source: Geostat 6.2 EQ 4 CONCLUSIONS Given the great fluctuations in the Georgian economy, the Program may need to pivot activities as necessary to consider the new reality related to investment, higher-value job creation, and increased revenues for MSMEs. The Program utilized an ecosystem approach to supporting each sector and value chain and quickly pivoted to adapt new delivery mechanisms and implementation modalities to align with new opportunities and challenges. Due to the economic impact of COVID-19, the Program should closely monitor potential obstacles in achieving its targets of creating 4,800 new jobs. The Program appears to be on track to achieve its $60 million revenue target during the period of performance. Based on its analysis of the economic impact of COVID-19 and qualitative data, the ET concludes that continuing Program support to a revamped tourism strategy and legal framework is crucial to the eventual recovery of the sector, which is also the cornerstone of Georgia’s economic recovery model. Georgia needs assistance in adopting new health and safety protocols across the industry for both domestic and international tourists. The Program should continue to support GNTA and the private sector in implementing its Post-Recovery Plan and Action Plan. In doing so, it should consider working with selected BSOs in assessing the potential revenue linked to domestic tourism as part of its action plan. The OCAP implementation of strategic plans may need to be adjusted to consider new high-priority actions that account for the steep decline of international tourists. There is evidence to support continued investment in light manufacturing, although perhaps in new industries. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 48 According to economic data monitored by the GOG, there are promising trends for growth in light manufacturing and shared intellectual and knowledge-based services. The Program may want to assess whether there are new opportunities in these industries. The Program’s subcontractor, PMCG, provides quarterly analytical studies of sectors and value chains to analyze the potential for high-value jobs, increased revenues and partnerships in the post-COVID economy. Based on the ET’s research of economic trends monitored by the GOG and its own data collection and analysis, some of the most promising sectors for future USAID programming in the post-COVID economy may be in light manufacturing of automobile parts, home electronics, textiles, and apparel and leather, and production of pharmaceuticals (see Figure 11). Some of these industries are aligned with the Program’s current sectoral focus through partnerships and grants but some industries may be more appropriate for a follow-on program. 6.3 EQ 4 RECOMMENDATIONS ● Continue support to the tourism sector and implementation of the recovery plan that embeds new global safety and health protocols. Assess opportunities in domestic tourism. ● Capitalize on new opportunities in ICT, SIS, and other knowledge-based services that have shown resilience and growth during the pandemic. ● Work with SWM companies and targeted municipalities to pilot initiatives at the regional and local levels. ● Work with BSOs in SWM, light manufacturing, and other industries hit hard by the pandemic to develop industry-wide interventions for job creation while implementing the OCAP. ● Ascertain whether BSOs, educational institutes or innovation centers can ramp up training related to e-commerce, digital content, online learning, educational tutorials, and entrepreneurship to attract more youth to high-paying jobs and provide funding19. 19 Such as the creation of E-Commerce Academy in partnership with E-Commerce Association-Georgia and TBC Bank. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 49 7. EQ5 FINDINGS, CONCLUSIONS, RECOMMENDATIONS EQ.5 To what extent has the Economic Security Program coordinated effectively with the Economic Governance Program to address the policy barriers facing its priority sectors and value chains? To what extent has the segregation of USAID’s economic policy reform efforts into a separate activity helped or hindered the Economic Security Program in achieving results? 7.1 EQ 5 FINDINGS Interviewed stakeholders, donor-funded program and activity staff, and partners recognized USAID’s strategy of working closely with donor partners to help the GOG establish a culture of public-private dialogues (PPD) and consultations to improve the quality of economic PLRs. Moreover, USAID/Georgia is seen as a critical partner in improving the BEE by promoting the formulation, adoption, and implementation of priority economic reforms. USAID programs fill a critical development niche by empowering the private sector to contribute to the GOG’s economic reform efforts in financially sustaining ways. The USAID-funded Economic Governance Program, a five-year, $19 million activity award, was designed to improve the enabling environment for MSMEs; Multiple other USAID-funded programs, including Economic Security Program, identify key policy barriers jointly with value chain players and then refer those barriers to the Economic Governance Program. The Economic Security Program includes very modest funding for policy work, and its SOW is limited to facilitating PPD to identify sectoral constraints and to propose potential solutions. It does not support legal drafting or implementation of reforms. Analysis of quantitative and qualitative data collected from the staff of the Economic Governance Program and the Economic Security Program confirmed that they are cooperating closely and adhering to the intended program designs. The two programs coordinated efforts in supporting the GOG in several sectors, including the development of a new tourism law and digitalization, especially in terms of e￾commerce initiatives. Key informants from the private sector, academic institutions, BSOs, NGOs, GOG entities, and other donors noted that the policy coordination was especially effective in the tourism sector. The Economic Security Program also is coordinating with the National Democratic Institute for International Affairs (NDI), International Republican Institute (IRI) and Good Governance Initiative (GGI) that are implementing DRG activities. For example, the Economic Security Program leveraged its experience in working with the private sector to support other civil society organizations (CSOs) to provide training and explain economic growth issues to legislators, political leaders, and their staff. At the time of the award, the Program established early relationships with other key donors, including the European Union and European Bank for Reconstruction (EBRD). The EBRD hosts the Tbilisi-based Investment Council which, prior to COVID-19 restrictions, met regularly and was an effective forum for donors and the GOG to prioritize improvements and reforms to the BEE. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 50 FINDING: EXTENSIVE COORDINATION WITH THE ECONOMIC GOVERNANCE PROGRAM FIGURE 12: PUBLIC-PRIVATE DIALOGUE AND ADVOCACY: A RESOURCE MANUAL The Economic Governance Program and Economic Security Program have collaborated extensively through weekly meetings (prior to the pandemic) and a working group dedicated to adapting a new tourism law and developing a Tourism Recovery Strategy. Both programs support the GNTA to mitigate the effects of the pandemic and contribute to the GNTA platform that enables key stakeholders to engage in ongoing tourism policy dialogue with the GOG. Each program has a unique lead responsibility for bolstering the tourism sector. The Economic Governance Program supports the GOG in the finalization of its 2019 draft Law on Tourism and improving the GNTA’s public policy advocacy capabilities. The Economic Security Program primarily supports the GNTA in capacity building and overall institutional development by guiding BSOs in putting in place effective governance structures, which were identified as weak in the Program’s OCAP activity. These two programs and the Agriculture Program partnered to conduct a three-day workshop to share knowledge to enhance the competitiveness of BSOs and CSOs in the Adjara region. The Economic Governance Program and Economic Security Program are collaborating in a digital transformation that will deliver more efficient and transparent government services and programs. According to the document review and USAID/Georgia respondents, the GOG’s digital infrastructure is outdated and vulnerable to cyberattacks and malfunction. In addition, the GOG agencies lack coordination mechanisms and are severely under-resourced to provide comprehensive digital solutions to increase MSMEs’ competitiveness in global markets. The Economic Security Program is providing inputs to the Economic Governance Program’s assessment of GOG digital transformation of government services. The two programs also jointly engage in PPD to address anti-piracy policy and intellectual property rights, which is crucial to the creative industry sector’s competitiveness. The programs are working with sector specific BSOs to develop a code of conduct for public and private sector stakeholders to improve e￾commerce governance. This collaboration is in its early stages but aims to support the industry, users, and GOG authorities to agree on time limits for e-commerce transactions and the rapid detection and removal of harmful and illegal online content. Another important area of effective collaboration relates to improving the currently limited access to equity and non-bank finance for local companies. The two programs have initiated discussions with GITA and National Bank of Georgia to explore possible sources of alternative financing, such as increasing the fintech institutional environment and regulating shadow banking. According to donor respondents supporting BEE and entrepreneurs, fintech could advance the delivery and automation of financial services and ease access to financing for companies. Existing shadow banking practices or unformalized lending, however, pose risks that need to be mitigated. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 51 FINDING: TARGETED COLLABORATION WITH USAID DRG PROGRAMS The ET found that the Economic Security Program and Economic Governance Program demonstrated intentional efforts to collaborate effectively on key policy reforms to support the tourism sector’s competitiveness and advance digital solutions. Both programs reported that the combined impact of their support was greater than individual efforts because they draw on each other’s strengths without duplication. The publication, developed by the Economic Security Program, Public-Private Dialogue and Advocacy: A Resource Manual (Figure 12) is a good example of each IP leveraging its expertise to reach the GOG, the private sector, and CSOs. However, the two programs use different terminology to describe the same stakeholders and key partners in their publications and conversations, which can lead to confusion in comparing outputs that contribute to cross-sectoral outcomes. For example, the Economic Security Program refers to associations as BSOs, and the Economic Governance Program refers to the same entities as private sector associations. Many respondents used the terms interchangeably in interviews but BSO respondents participating in the OCAPs generally referred to themselves as members of an association. The Program facilitated the development of sound economic platforms at the political party and government levels through continued collaboration with other USAID EG and DRG IPs. According to Program staff, and confirmed by GOG respondents, the Program presented PPD initiatives that were integrated into the Parliamentary Economic Policy Committee’s Action Plan for 2021. Consistent collaboration with USAID DRG programming resulted in effective PPDs and informal information exchanges for transferring EG and sectoral knowledge to parliamentary committees, elected officials, and political party leaders. Program staff contribute technical expertise in economic growth to ongoing DRG activities through its PPD sessions with Parliament and MoESD on topics such as the SIS sector and the furniture and packaging value chains of light manufacturing. FINDING: DONOR COORDINATION ENHANCES ECONOMIC REFORMS AND BEE According to respondents, the Program is effectively collaborating with the European Union and the United Nations Development Program (UNDP) on post-COVID recovery; the EBRD, Asian Development Bank, and Mashav/Embassy of Israel on MSMEs and access to finance; the European Union Water Initiative Plus and UNDP on solid waste water resource management; USAID Good Governance Initiative; the European Union; the German Development Agency; World Bank; the USAID Zrda Activity and USAID Good Governance Initiative on tourism policy; the USAID Economic Governance Program on fintech and crowdfunding; and the USAID Industry-Led Skills Development Program on youth empowerment. To facilitate collaboration, the Program participates in quarterly coordination meetings with USAID programs. No respondents mentioned that the Program was unwilling to collaborate and coordinate efforts to address critical PLRs in support of MSMEs. All respondents stated that these coordination efforts between USAID partners and with other donors enhanced reform processes. 7.2 EQ 5 CONCLUSIONS USAID’s Economic Governance Program was designed to improve economic governance and leadership in Georgia in ways that will enable Georgia to harness investments needed to finance its own development. The Economic Security Program identifies key policy barriers jointly with value chain players and then refers those barriers to the Economic Governance Program. The Economic Security Program and USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 52 Economic Governance Program demonstrated intentional efforts to collaborate effectively on key policy reforms. The combined impact of their support was greater than individual efforts because they draw on each other’s strengths without duplication. 7.3 EQ 5 RECOMMENDATIONS ● Continue to facilitate PPD initiatives in priority sectors identified by other USAID IPs, such as SIS, creative industries, and tourism to identify key policy gaps, and share these insights with other IPs and partners to amplify reform priorities to GOG institutions and parliamentary committees. ● Continue to work with USAID IPs, such as Economic Governance, Good Governance Initiative, Agricultural Program, Industry-led Skills Development Program, and other donors to identify incentives for strategic cohesion among BSOs to coalesce around priority policy gaps. ● Prioritize policy, laws, and regulation (PLR) gaps with the USAID Economic Governance Program that may have shifted due to the economic impact of COVID-19. ● Continue to encourage close coordination among EG and DRG programs to advocate at the national level to hold GOG accountable by encouraging more policy, legislative, and regulatory reforms. This coordination could help the private sector and NGOs demand better services and accountability through advocacy efforts. ● Public Private Dialogues (PPDs) could elicit suggestions on developing an overall strategy to catalyze citizens and PSE in targeted municipalities to improve capacity and local governments’ understanding of economic growth to adapt supportive local PLRs. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 53 8. CROSS-CUTTING FINDINGS AND RECOMMENDATIONS 8.1 CROSS-CUTTING FINDINGS FINDING: INCLUSION OF WOMEN AND YOUTH IN ACTIVITY TARGETS IS EVIDENT The ET found that many value chains face constraints of limited access to finance, poor business and marketing skills, and insufficient workforce skills that perpetuate strong gender imbalances. Women have less access to capital and networks in Georgia than men do because they lack financial collateral and often work from home. The ET assessed gender equity as a cross-cutting theme and tried to determine its role in the sustainability of the Program. Throughout the evaluation, female respondents stated that more Program activities should use a specific gender lens that advances women’s economic empowerment (WEE) through incentives and weighted scoring for female grant applicants. As made evident in Program documents and reinforced by its gender specialist, the Program actively included women as participants in all activities, including training. The Program also evaluates PLRs against 10 criteria, including gender and environmental considerations. It has made an active effort to include women as participants in activities and ensuring the collection of sex-disaggregated data for all activities. It reports on four gender indicators related to WEE, gender inclusion in policy, and CSO and private sector association leadership. The Program has specific initiatives to create equality opportunities for women and men along the target value chains. One such activity was the Grace Hopper ICT Awards, which was developed by the Program and supported by five private sector and donor partners. The awards highlight women’s involvement in the ICT sector by showcasing Georgian women and companies that have succeeded in achieving leadership positions. The Program also supported the Georgia Women in Technology initiative in partnership with the USMAC, which supported female entrepreneurs’ marketing visits to Silicon Valley. Overall, in Year 1, the Program increased women and youth access to productive economic resources by engaging more than 500 youth of which 400 were women in its various activities and initiatives. In total, 304 youth of which 99 were women found new full-time employment because of the Program’s efforts, and 12 women increased their livelihoods by gaining better employment opportunities. In addition, the Program’s initiatives increased the soft skills of 490 youth and 404 women. During year two, the program increased women and youth access to economic resources by engaging more than 123 youth and ninety (90) women in various activities and initiatives. In total, because of the Program’s efforts 633 people found full-time employment (375 youth, 331 women); and five (5) women￾led startups have participated in the international acceleration initiative. In addition, the Program initiatives increased soft skills in the targeted sectors and value-chains of 643 youth and 770 women. In addition, the program developed a Gender and Disability Mainstreaming Guide for organizations and has piloted it with local partners. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 54 FINDING: PSE SELECTION CRITERIA MAY NEED ADJUSTMENTS FOR FUTURE PARTNERSHIPS PSE and market systems development (MSD) approaches are both key to advancing sustainable, scaled, and inclusive outcomes. Preliminary findings indicated that the Program may need improved cross-cutting analytical framework(s) and assessment tool(s) to support selection criteria for future private sector partners, associations, and grantees to hew more closely to USAID/Georgia’s forthcoming PSE selection criteria for economic growth activities. The Program’s approach is to leverage PDF and collaboration with public, private, and international stakeholders to scale existing and new opportunities. In its Monitoring, Evaluation, and Learning plan, the Program measured PSE engagement through the number of GDAs established (Indicator 26), amount of dollars leveraged through partnerships (Indicator 27), and the number of PPPs established (Indicator 28). According to USAID PSE and MSD guidance, transformational impact is measured by how MSD can leverage PSE’s strengths in corporate relationship management, diverse investment, and partnership strategies. The qualities of transformation include the interrelated elements of scale, sustainability, and systemic change. These outputs, however, do not predict the transformational impact of USAID investment according to USAID respondents. No PDF selection criteria explicitly stated the anticipated overall impact across value chains or sectors to be gained from each partnership. Without more evidence-based data regarding how these partnerships advance MSD approaches and generate enduring changes in incentives, rules, or norms or support the functioning of the system, the ET could not assess the extent to which these partnerships are transformational or limited transactions. The ET assumed that the PDF partnerships could be transformational, but there was insufficient evidence at this stage of the project cycle to state this finding. FINDING: PPD AND BSOs HIGHLIGHT MSME INTERESTS IN IMPROVED BEE The role of BSOs in promoting and protecting the interests of MSME interests through improvements to the BEE is critically important, although the capacity of many associations in Georgia is low. Survey and KII respondents representing BSOs confirmed the need to develop effective PPD mechanisms to work with central and local authorities. GOG counterparts have widely varying experience and levels of trust regarding interactions with private sector partners. The Program developed the PPD initiative to address policy gaps in the targeted sectors, which included the involvement of BSOs. The PPDs established new avenues of cooperation with parliamentary committees to address important policy gaps and the need to promulgate economic policies in all five targeted sectors. Three respondents from other USAID and donor-funded programs said the Economic Security Program made valuable contributions to improving PLRs through its in-person and online PPDs, which focused on enhancing the overall enabling environment for competitiveness through supporting digitalization of public services, capacity build of key ministries and agencies, PPDs and information exchanges with parliamentarians, political leaders and their staff. Through its OCAP, the Program continued its work with 21 BSO partners to address BEE issues as well. The Program provides capacity support to many government institutions, including the MoESD, Ministry of Regional Development and Infrastructure, Enterprise Georgia, National Agency for State Property, Georgia Innovation and Technology Agency, and many agencies related to the tourism sector: GNTA, Mountain Trails Agency, National Agency for Cultural Heritage Preservation, and United Airports of USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 55 Georgia, among others. Currently, the MoESD, Ministry of Finance, GNTA, and GITA are engaged in PPPs in the tourism, SWM, SME development, and ICT sectors that the Program leveraged. The ET recommends that the Program monitor its types of support to GOG institutions that are not formally engaged in PPPs so that any changes noted at a later stage can be attributed to the Program. 8.2 CROSS-CUTTING RECOMMENDATIONS ● Consider increasing designated funding opportunities for women’s organizations, female entrepreneurs, and women-owned businesses in more sectors.20 ● Utilize a PSE and MSD assessment tool with important definitions and distinctions to establish a common understanding with partners; provide high-level guidance on aligning, customizing, and operationalizing approaches to PSE and MSD with inputs from USAID/Georgia and potentially USAID/Washington PSE and MSD hubs. ● Adapt a strategy for setting indicator targets to report how partnerships and grants address systemic gaps and market failures and/or influence the industry.21 ● Continue the Program’s systemic reform initiatives for BSOs and PPDs with other USAID and international partners that support BEE reforms at the national and sub-national levels to identify synergies with USAID and avoid duplication of activities. 20 Implementing Partner introduced a new EIA partnership and Catapult Fund to address this issue after the mid-term evaluation draft report was shared with USAID/Georgia and the IP. 21 Implementing Partner introduced new indicators to address this issue after this mid-term evaluation draft report was shared with USAID/Georgia and the IP. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 56 ANNEX 1: STATEMENT OF WORK STATEMENT OF WORK MID-TERM EVALUATION OF USAID’S ECONOMIC SECURITY PROGRAM 1. EVALUATION PURPOSE The purpose of this mid-term performance evaluation is to determine the effectiveness of specific Programmatic approaches, which are referenced in the evaluation questions below, in achieving intended life-of-Program results and to provide recommendations on corrective actions and new directions for the remaining years of Program implementation. The primary audience of the evaluation will be USAID/Georgia’s Economic Growth (EG) office and USAID’s Economic Security Program implementing partner (DAI). The results of the study may be shared with local stakeholders (Ministry of Economy and Sustainable Development and its agencies, private sector stakeholders, partner NGOs, etc.,) and other donors working in this area. 2. SUMMARY INFORMATION Award Number Contract No. 72011419C00001 Award Dates: 17-Apr-2019 - 16-Apr-2024 Funding: $23,033,752 Implementing organization: DAI Global LLC. Contracting Officer’s Representative (AOR): David Tsiklauri Alternate AOR: Philip Greene : 3. BACKGROUND A. Description of the Problem, Context, and Theory of Change Despite Georgia being a global leader in trade and business environment reforms, growth has not resulted in employment opportunities or higher wages. An aggressive reform agenda and healthy growth rate have not translated into economic dynamism or opportunities for Georgian citizens. There are multiple causes why Georgia’s economic growth has not created high-value employment opportunities for its citizens, limiting the benefits of Georgia’s Western orientation. One such cause is that Georgian firms still lack access to resources necessary to increase competitiveness and create greater employment opportunities in key sectors, including access to high-value, diverse markets; investment resources; and a workforce that has the skills demanded by the private sector. The recent COVID-19 crisis has led to a major economic recession in Georgia, resulting in the loss of jobs and local currency devaluation and has thrown into stark relief the need for an economy that delivers real gains to its citizens. B. Description of the Intervention to be Evaluated USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 57 On April 17, 2019, USAID/Georgia awarded a five-year, $17,833,752 contract to DAI Global to implement USAID’s Economic Security Program The purpose of USAID’s Economic Security Program is to accelerate broad-based growth of sectors outside of agriculture that show strong potential to create jobs, increase micro, small, and medium enterprise (MSME) revenues, and support diversification to more productive economic activities in tourism, creative industries, light manufacturing, ICT, and shared intellectual services sectors. The underlying hypothesis of the Program is that IF Georgia’s firms have access to the resources they need (capital, access to high-value markets, skilled workforce, modern technologies, etc.) to improve productivity, sales, and product and service quality, and IF cooperation is strengthened in targeted sectors and value chains, THEN targeted sectors/value chains will become more competitive and will provide greater high-value employment opportunities to its citizens and drive closer integration with the West. In June 2020, USAID amended the subject contract, expanding its scope and increasing the life of activity funding by $5,200,000, from $17,833,752 to $23,033,752. Through this cost-extension, the Program was tasked to address the challenges and opportunities created by COVID-19 pandemic by: (a) expanding interventions in the e-commerce, tourism, light industry, and ICT sectors; (b) selecting an additional sector for assistance - the solid waste management and recycling sector; and (c) expanding the Program’s public￾private partnerships component. The contract is organized by four components designed to achieve the stated results: Component 1: Strengthen cooperation in targeted sectors: The Program provides technical assistance and cost-share grants to strengthen linkages and cooperation throughout value chains in targeted sectors and improve support services intended to enhance growth and productivity across targeted value chains in target sectors. In doing so, the Program takes a collaborative approach to development, working with a plethora of stakeholders including firms, associations, Government of Georgia (GOG) agencies, development partners, regional government and municipalities, and other stakeholders, Component 2: Support Enterprises to improve productivity, sales, quality, and develop new products and services: Through identification and exploration of value chains that provide the best opportunity for Georgia to initiate investment that leads to high-value jobs, the Program facilitates entrance into new markets. It also increases and expands product offerings, promotes stronger linkages between enterprises and the organizations that support them, and enhances the overall value chain ecosystem to ensure sustainability. This is accomplished through a series of interventions that include technical assistance, cost￾share grants, and export enhancement, among others. Component 3: Industry-led workforce development: Alignment of Georgia’s workforce with the needs of industry is critical to the country’s movement toward the development of a prosperous society. As such, the Program works with the Ministry of Education, Science, Culture and Sport (MoESCS), educational institutions, training providers, and the private sector to prepare Georgians for new and expanded employment opportunities through identified sectors and value chains. The Program’s approach is led by industry, meaning that it focuses on improving knowledge and skills that align with emerging investment and job opportunities. This requires significant re-thinking of educational and vocational models, as well as specific interventions that will link skills development directly with employment. Component 4: Building public-private partnerships: Through its Partnership Development Fund (PDF) with the total value of $3 million, the Program co-creates and co-funds Public Private Partnerships (PPPs), Global Development Alliance (GDA) mechanisms, and other investment opportunities that support the USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 58 growth of identified sectors/value chains and that provide high-value employment for Georgians. Approaches under PDF are collaborative, innovative, and flexible to identify and take advantage of emerging opportunities. USAID’s Economic Security Program includes very modest funding for policy work. Specifically, the Program’s role is limited to facilitating public-private dialogue among sector stakeholders to jointly identify policy and regulatory constraints to the development of targeted sectors and to propose potential solutions to mitigate these constraints. The Program does not support legal drafting or implementation of reforms. The underlying reason why the Program does not have a robust policy component is that USAID/Georgia supports major legislative, policy and regulatory reforms through a standalone Economic Governance Program, specifically designed to improve enabling environment for MSMEs. The Mission designed these two Programs with a logic that both Programs would cooperate closely, and that USAID’s Economic Security Program would serve as an extension agent to identify key policy barriers jointly with value chain players and then refer these barriers to USAID’s Economic Governance Program for further action. USAID’s Economic Governance Program is a five year, $19 million activity awarded in December 2019, which is designed to improve economic governance and leadership in Georgia in ways that will enable Georgia to harness investments it needs to finance its own development. Specifically, this activity is designed to increase Georgia’s ability to attract private sector investment by building the sustainability of the consultative economic reform-making process and promoting the formulation, adoption, and implementation of priority economic reforms that will make Georgia’s business environment more transparent, predictable, consistent, inclusive, and cost-effective. 4. EVALUATION QUESTIONS The evaluation should address the following specific questions: ● Private sector engagement: To what extent has the Partnership Development Fund (PDF) targeted and established high-impact (defined as wide-reaching and/or replicable) partnerships with the private sector that have strengthened and catalyzed the development of priority value chains? To what extent are these partnerships sustainable (defined as the establishment of market linkages that will not depend on USAID assistance after the activity ends)? ● Value chain approach: To what extent has support to sector associations and government institutions catalyzed priority value chain development? How important a role have these stakeholders played in increasing value chain competitiveness? To what extent have these stakeholders received sufficient and relevant support? ● Grant component: To what extent has the grant component strengthened each priority value chain? To what extent did the grants address gaps or market failures in each value chain. ● Coordination on policy: To what extent has the Economic Security Program coordinated effectively with other USAID activities (managed by both the USAID Economic Growth and Democracy, Rights and Governance Offices) to address the policy barriers facing its priority sectors and value chains? To what extent has the absence of a large policy component within USAID’s Economic Security Program helped or hindered its ability to address policy gaps. ● COVID-19: In the context of COVID-19 economic contractions, how can the activity adapt its approaches (e.g. selection of grant solicitation themes, division of USAID investments across sub￾USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 59 sectors, sequencing of interventions, etc.) to improve its ability to achieve its targets: creation of 4,800 jobs and achieving $60 million in new sales? 5. EVALUATION DESIGN AND METHODOLOGY This evaluation will utilize a non-experimental, mixed-methods evaluation design. The Contractor is expected to suggest the use of appropriate data collection and analysis methods, both quantitative and qualitative, including document review, key informant interviews, focus group discussions, direct observation, survey instruments (if applicable), etc., with Program stakeholders, beneficiaries, the GOG, the private sector, and other players. The methodology for any evaluation process that involves the selection of participants (e.g. surveys, focus groups, interviews) must be clearly explained and justified. For example, for a survey or mini-survey (if proposed), the number of respondents and their selection process should be explained and justified. The same is true for key informants, focus group discussions, and other methods as well. Selected respondents should be representative of women, youth, and vulnerable groups, where appropriate. The Contractor must conduct a desk review of USAID’s Economic Security Program related documents, which will help identify areas that merit closer attention once the team begins its fieldwork. Reading materials will be available to the team shortly after signing the Contract. The Contractor must develop a detailed evaluation design and a workplan, including data collection plan and drafts of data collection tools. A draft of the work plan and evaluation design must be shared with USAID/Georgia for review prior to the fieldwork. The plan will then be presented to the Mission during the in brief in more detail. The evaluation design must include the evaluation matrix (an illustrative evaluation matrix for this study is given below). The evaluation design must explain how the evaluation Contractor intends to conduct the study in detail, including a detailed description of one or more proposed methodologies as well as limitations of proposed methodologies. It must explain in detail what methods will be used to obtain answers for each evaluation question. The design must also explain how the proposed methodology (mix of methods) to conduct the study generates evidence to ensure rigor and reliability of results; and how and why the proposed methodology will minimize bias. The evaluation design must also include the data analysis plan for each question, draft questionnaires (to be included as an attachment), and other data collection instruments or their main features, criteria for assessing responses to evaluation questions, known limitations, and a dissemination plan. The evaluation design might also include specific sub-questions for each evaluation question, where needed. Again, the methods described herein are only illustrative and USAID expects that the Contractor will suggest the best methods that would generate most reliable and evidence-based answers to the key evaluation questions. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 60 TABLE 4: ILLUSTRATIVE EVALUATION MATRIX Research Question Data Source Methodology Private sector engagement: To what extent has the Partnership Development Fund (PDF) targeted and established high-impact (large-scale, wide￾reaching, and/or replicable) partnerships with the private sector that have strengthened and catalyzed development of the priority value chains? To what extent are these partnerships sustainable? Activity documentation: Program description, quarterly and annual reports, M&E plan, results framework, workplans, sub-awards documentation, PDF solicitation documents, PDF applications, selection criteria and selection documentation, value chain assessments. ● Project staff ● Representatives of sub-contractors ● Private sector ● Academic institutions ● Associations and NGOs ● GOG entities ● USAID staff Document reviews/Direct Observation/ Key Informant Interviews / Focus Group Discussions and/or Mini-Survey with identified data sources. Value chain approach: To what extent has the activity’s support to sector associations and government institutions catalyzed priority value chain development? How important a role have these stakeholders played in increasing value chain competitiveness? To what extent have these enabling institutions received the optimal type and level of support? Activity documentation: Program description, quarterly and annual reports, M&E plan, results framework, workplans, sub-awards documentation, third party publications related to target value chains, etc. ● Project staff ● Representatives of sub-contractors ● Private sector ● Academic institutions ● Associations and NGOs ● GOG entities ● USAID staff Document reviews/Direct Observation/ Key Informant Interviews / Focus Group Discussions and/or Mini-Survey with identified data sources. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 61 Grant component: To what extent has the grant component strengthened each priority value chain? To what extent did the grants address the most important gaps or market failures in each value chain. Activity docs: Program description, quarterly and annual reports, M&E plan, results framework, workplans, sub-awards documentation, grant solicitation documents, grant applications, selection criteria & selection documentation, value chain assessments. ● Project staff ● Representatives of sub-contractors ● Grantees ● Other private sector firms ● Academic institutions ● Associations and NGOs ● GOG entities ● USAID staff Document reviews/Direct Observation/ Key Informant Interviews / Focus Group Discussions and/or Mini-Survey with identified data sources. Research Question Data Source Methodology COVID-19: In the context of COVID-19 economic contractions, how can the activity adapt its approaches (e.g. selection of grant solicitation themes, division of USAID investments across sub￾sectors, sequencing of interventions, etc.) to improve its ability to achieve its targets: creation of 4,800 jobs and achieving $60 million in new sales? Activity documentation: Program description, quarterly and annual reports, M&E plan, results framework, workplans, sub-awards documentation, grant and PDF solicitation documents, grant and PDF applications, selection criteria and selection documentation, value chain assessments. ● Project staff ● Representatives of sub-contractors ● Private sector ● Academic institutions ● Associations and NGOs ● GOG entities ● USAID staff Document reviews/Direct Observation/ Key Informant Interviews / Focus Group Discussions and/or Mini-Survey with identified data sources. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 62 Coordination on policy: To what extent has the Economic Security Program coordinated effectively with the Economic Governance Program to address the policy barriers facing its priority sectors and value chains? To what extent has the segregation of USAID’s economic policy reform efforts into a separate activity helped or hindered the Economic Security Program in achieving its results. Activity documentation: Program description, quarterly and annual reports, M&E plan, results framework, workplans, sub-awards documentation, grant and PDF solicitation documents, grant and PDF applications, selection criteria and selection documentation, value chain assessments. Document reviews/Direct Observation/ Key Informant Interviews / Focus Group Discussions and/or Mini-Survey with identified data sources. ● USAID staff ● Economic Security Project staff ● Economic Governance Project staff ● Representatives of sub-contractors of both Programs ● Private sector ● Academic institutions ● Associations and NGOs ● GOG entities USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 63 6. DELIVERABLES The contractor will be required to provide USAID with the following deliverables: a. Final Work Plan and Evaluation Design: Final Work Plan and Evaluation Design document for the evaluation shall be completed by Contractor and presented to the COR prior to the commencement of fieldwork. The evaluation design will include a detailed evaluation design matrix (including the key questions, methods and data sources used to address each question and the data analysis plan for each question), draft questionnaires and other data collection instruments or their main features, known limitations to the evaluation design, and a dissemination plan. The final design requires USAID/Georgia approval. The work plan will include the anticipated schedule and logistical arrangements and delineate the roles and responsibilities of members of the evaluation team. b. In-brief with the mission: will be held prior to starting the fieldwork. This will be a maximum 30- minute PowerPoint presentation of the plan, namely, how the questions asked in SOW will be answered. Prior to in brief, the evaluation team may have working meeting/s with USAID/Georgia Evaluation Contract activity manager and USAID’s Economic Security Program COR to discuss the details of the design. c. Conduct fieldwork: The in-country evaluation must expand upon the analysis in the desk review and in the facilitated discussion through methods proposed by the evaluation team that might include interviews with focus groups of sub-contractors, beneficiaries or end-users, Georgian government, private sector entities, field visits, and mini-survey, if proposed. d. Recommendations co-creation meeting: After finishing the fieldwork, the evaluation team must participate in a co-creation session with USAID Mission, including the Evaluation Contract activity manager at USAID/Georgia and USAID’s Economic Security Program COR, to produce a set of recommendations. Prior to the co-creation session/meeting, the evaluation team must summarize and submit to USAID a matrix of preliminary evaluation findings/conclusion/recommendations. The co-creation meeting/session will serve to review the matrix and jointly formulate/refine evaluation recommendations. However, the evaluation team maintains complete editorial authority with regard to the evaluation recommendations section of the Final Evaluation Report (see deliverable (g) below). e. Mission out-brief: The evaluation team must present an outline (in bullets, possibly in power point or as a handout) of the evaluation report with general findings, conclusions, and anticipated recommendations to USAID Mission Management and other interested USAID staff at the end of their fieldwork. f. Draft reports: The Contractor must submit to USAID/Georgia a draft report within 20 working days of completing the out-briefing with USAID. This document must explicitly respond to the requirements of the SOW, answer the evaluation questions, be logically structured, and adhere to the standards of the USAID Evaluation Policy. g. Final Evaluation Report: The Contractor must incorporate USAID/Georgia’s comments and submit the final report to USAID/Georgia within five (5) working days following receipt of the final batch of USAID’s comments on the draft report. The Contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse at http://dec.usaid.gov within 30 calendar days of final approval of the formatted report with USAID/Georgia consent. In case it is determined that the full report includes sensitive information, USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 64 the Contractor must produce a public version for submission to DEC; the latter also requires USAID/Georgia’s clearance. h. All records from the evaluation (e.g. interview transcripts and summaries, focus group transcripts, code books, etc.) must be provided to USAID/Georgia as requested. All quantitative data collected by the evaluation team must be provided in an electronic file in a machine-readable format. The data should be organized and fully documented for use by those not fully familiar with the Program or the evaluation. USAID will retain ownership of the survey and all datasets developed. In addition, the dataset must be submitted to the Development Data Library (DDL) as part of USAID’s Open Data Policy. 7. EVALUATION TEAM Proposed evaluation team: The evaluation must be conducted by a team composed of experts. The contractor has to demonstrate that proposed team members have sufficient expertise to carry out the task at a high standard. The team collectively must have demonstrated prior experience of working in and/or evaluating/assessing private sector competitiveness, economic governance, access to finance, public-private partnerships, and workforce development areas. The Contractor must justify and explain proposed team configuration and distribution of roles among team members. The Team Leader (international) must have extensive, demonstrated experience leading development assistance Programs and/or similar evaluations or assessments focused on private sector competitiveness and value chain development. Experience in private sector development in Georgia and/or in the Europe and Eurasia region will be an advantage but is not required. The team leader will be responsible for the day-to-day management of the team, data collection and synthesis, presentations, and drafting of the interim/final reports. Fluency in English language is required. Excellent writing skills and the demonstrated ability to produce well written and sound evaluation deliverables are required. Evaluation Expert must have a justifiable experience in planning and conducting evaluations using various data collection and analysis methodologies. Prior work experience evaluating economic growth activities is also required. The evaluation expert will lead the team in evaluation design, data collection and synthesis, and also assist in report writing. The evaluation expert’s role will also include document review and instrument development. Fluency in English is required. Excellent writing skills and the demonstrated ability to produce well written and sound evaluation deliverables are required. Locally-hired private sector expert must have extensive, justifiable experience working in the private sector development assistance Programs and deep knowledge of value chain development. Experience of participating as a team member in conducting evaluations is preferable but not required. English language knowledge and good writing skills are required. Fluency in Georgian – both speaking and reading is required. The Contractor may be asked to provide 1-2 examples of their proposed team leader’s past work. The Contractor must provide information about the selected evaluation team members including their CVs and explain how they meet the requirements set forth in the evaluation SOW. All evaluation team members must be familiar with USAID’s Evaluation Policy. USAID may request an interview with any of the proposed evaluation team member/s via conference call/google hangouts/Zoom or any other means available. 8. EVALUATION SCHEDULE USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 65 The following levels of effort are illustrative and should serve only as an example of the staff which may be mobilized under this Contract. These levels may not reflect the actual level of effort contracted, and the Contractor will be expected to submit its own estimate of the level of effort needed to fulfill the objectives. Team Member Estimated # of Days International Technical Expert – Team Leader 50 Evaluation Expert 48 Local Private Sector Expert 48 Timing (Anticipated Month or Duration) Proposed Activities O/A May 2021 Initial teleconference with USAID/Georgia TBD Document review, preparation work and finalization of the evaluation design and work plan TBD Submission of the draft work plan and evaluation design to USAID TBD In-brief with USAID Mission to collect feedback TBD Submission of the final work plan/evaluation design to USAID TBD Fieldwork begins Weekly check-ins with USAID TBD Recommendations workshop with the USAID Mission TBD Out-brief with Mission, end of fieldwork TBD Data analysis and report writing TBD Submission of the draft evaluation report to USAID TBD Submission of the final evaluation report to USAID Note: As needed, there may be several rounds of review of the draft evaluation report by USAID prior to finalizing/approving the report 9. WORK LOCATION Tbilisi, Georgia’s regions. In order to conduct meetings and interviews, the Contractor may need to travel to Tbilisi and to Program￾targeted communities located in the Kutaisi, Telavi, Zugdidi, Gori, and Batumi municipalities. Due to USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 66 COVID-related travel and other restrictions, virtual meetings, remote data collection, and remote supervision by the evaluation team lead may be authorized. 10. LOGISTICAL SUPPORT USAID/Georgia and USAID’s Economic Security Program will provide the list of in-country contacts prior to the commencement of fieldwork but will not assist in the logistics of appointing meetings. Hence, the Mission will not be responsible for arranging logistics for the evaluation team. The Contractor must suggest how they plan to arrange translation, transportation, and logistical support to the evaluation team. USAID/Georgia will put the Contractor in contact with USAID’s Economic Security Program implementing partner. The Contractor will conduct meetings in Tbilisi. Some meetings will require travel to regions outside Tbilisi to meet with grant recipients and other beneficiaries, and NGO, private sector and government stakeholders. USAID’s Economic Security Program implementing partner may assist with setting those meetings. However, due to COVID-19 considerations, remote data collection and online KIIs may be authorized in lieu of in-person meetings. 11. OTHER REQUIREMENTS The evaluation team must be familiar with USAID’s Human Subject Protection Policy and USAID’s Evaluation Policy (http://www.usaid.gov/evaluation). The evaluation team must provide adequate training for its survey staff on survey methodology, USAID’s survey regulations, other relevant regulations, and the data collection plan. The contractor has the responsibility to safeguard the rights and welfare of human subjects involved in the survey research supported by USAID. USAID has adopted the Common Federal Policy for the Protection of Human Subjects, Part 225 of Title 22 of the Code of Federal Regulations (http://www.usaid.gov/policy/ads/200/200mbe.pdf). Recipient organizations must familiarize themselves with the USAID policy and provide “assurance” that they will follow and abide by the procedures of the Policy. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline, need to be agreed upon in writing by the USAID/Georgia. 12. SUMMARY OF SPECIFIC TECHNICAL REQUIREMENTS The Contractor must ensure that the evaluation team completes the following tasks and provides the following deliverables within the terms defined by the contract: ● Conduct initial teleconference with USAID/Georgia to discuss the upcoming work. ● Provide a draft evaluation design and work plan (including meeting schedules and data collection instruments) to USAID for review and comment. ● Incoming briefing with USAID management to present the detailed evaluation design. ● Conduct the evaluation in accordance with the USAID-approved evaluation design and work plan. This includes weekly check-in calls with USAID/Georgia to ensure all work is proceeding smoothly and address any outreach challenges the evaluation team may be experiencing. ● Recommendations co-creation session with USAID Mission to formulate/refine evaluation recommendations. ● Outgoing briefing with USAID management to present the matrix of preliminary findings/conclusions/recommendations of the evaluation. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 67 ● Provide a final evaluation report to USAID in accordance with Reporting Guidelines under Section 9 - Deliverables. The evaluation report should follow the “Criteria to Ensure the Quality of the Evaluation Report” of the USAID Evaluation Policy. ● Submit USAID-approved evaluation report to Development Experience Clearinghouse (DEC) within 30 calendar days following the acceptance of the report by the USAID Contracting Officer’s Representative (COR.) ● Submit quantitative dataset, if collected, in a machine-readable format to the Development Data Library (DDL) as per USAID’s Open Data Policy at least five work days prior to the end date of the evaluation contract. ANNEXES ANNEX 1. ACTIVITY DOCUMENTS FOR REVIEW Relevant reports and other project documentation will be provided by the Mission to the Contractor prior to the commencement of fieldwork. The evaluation contractor shall initiate Washington-based work by reading reports and familiarizing him/herself with the Program. These documents are: ● SOW for USAID’S Economic Security Program ● Work plans ● Quarterly and annual reports ● Grants Manual ● Grant solicitations, grant applications, and grant selection documentation ● M&E plans and performance data tables ● Initial list of in-country contacts ● Value chain analysis and assessments ● Other reports and papers, as applicable. ANNEX 2. REPORTING GUIDELINE The illustrative format for the final evaluation report is as follows: 1. Executive Summary—summarizes key points, concisely states the purpose, background of the project, main evaluation questions, methods, findings, conclusions, recommendations and any lessons learned; should be sufficiently detailed, yet brief, to serve as a stand-alone product (3-5 pp) 2. Introduction—state the purpose, audience, and outline of the evaluation (1 pp) 3. Background—provide a brief overview of the project and the study implemented (1-2 pp) 4. Methodology— the evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology. Greater detail can be included in the appendices (2-3 pp); 5. Findings/Conclusions/Recommendations—explicitly answer each evaluation question; the report should distinguish between findings (the facts), conclusions (interpretation of the facts), and recommendations (judgments related to possible future programming) (10-15 pp); however it should be clear what is the link between them; USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 68 6. Lessons Learned (if not covered in findings, conclusions and recommendations) (2–3 pp); 7. Annexes—annexes must include this statement of work and its modifications (if any); any “statements of differences” regarding significant unresolved difference in opinion by funders, implementers, and/or members of the evaluation team; a glossary of terms; sources of information, properly identified and listed; clear documentation of schedules, meetings, interviews and focus group discussions, and any tools used in conducting the evaluation, such as focus group scripts or questionnaires, checklists and discussion guides used; and signed disclosures of conflict of interest. The evaluation design should also be attached to the report. The report format should be presented in Microsoft Word and use 12-point type font throughout the body of the report, using page margins 1” top/bottom and left/right. The body of the report should ideally be within 20-25 pages, excluding the executive summary, table of contents, references and annexes. The final report must follow USAID branding and marking requirements. Per the USAID evaluation policy, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report. ● The evaluation report should represent a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the projects, what did not and why. ● Evaluation reports shall address all evaluation questions included in the statement of work. ● The evaluation report should include the statement of work as an annex. ● Evaluation methodology shall be explained in detail, and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report. ● Evaluation findings will assess outcomes and impact on males and females. ● Limitations to the evaluation shall be disclosed in the report, with attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). ● Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. ● Sources of information shall be properly identified and listed in an annex. ● Recommendations shall be supported by a specific set of findings. ● Recommendations shall be action-oriented, practical and specific, with defined responsibility for the action. ANNEX 3. REPORTED RESULTS TO DATE To date, the Program has assisted 139 organizations, created 349 high-value jobs, achieved $3,215,554 in increased sales/revenues for MSMEs, and leveraged $1.3 in funding for partnership initiatives. Life of Program (LOP) targets for these indicators are 720 organizations, 4,800 jobs, $60 million in sales, and $18 million in external funding leveraged, respectively. The Program awarded 20 grants worth $867,000 and leveraged cost share from grantees worth $840,000. These grants support skills development; strengthen sectoral organizations; improve productivity, quality and sales; and develop new products and services in USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 69 the packaging, furniture, and other priority sectors. The Program has forged 10 public-private partnerships, leveraging more than $1.3 million in private sector funding. The Program worked with the Georgian Tourism Agency to respond to the Russian Air Embargo and challenges related to the COVID-19 pandemic; supported an update to the Agency’s strategy and action plan; facilitated better planning and targeting of key markets; spearheaded tourism donor coordination; and initiated multiple awareness raising and product development actions to encourage high-value, low(er) volume tourism. In light of COVID-19 pandemic, the Program initiated multiple efforts to promote the development of Georgia’s e-commerce sector and supported professionals and businesses to understand and integrate e-commerce into their operations. It strengthened the Georgian e-commerce association and assisted 50 MSMEs to develop e-commerce sites and 10 MSMEs to migrate to digital platforms. While COVID-19 delayed some interventions, it opened the door for many others, notably in e￾commerce, solid waste management, and ICT value chains. As the pandemic continues, the activity faces a major challenge in the decline of Georgia’s tourism sector, which remains a key component of the contract and the activity’s targets. USAID’s Economic Security Program is responding to this challenge by assisting the GOG and private sector partners to develop future actions, seize domestic tourism opportunities, and comply with new health and safety rules. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 70 ANNEX II: EVALUATION WORK PLAN 1. INTRODUCTION Upon successful completion of the mid-term performance evaluation of the Youth Entrepreneurial Skills for Advancing Employability and Income Generation Program in Georgia, or YES-Georgia, USAID/Georgia requested the LEAP III team to conduct mid-term evaluations of two additional Programs, USAID’s Agriculture Program implemented by Cultivating New Frontiers in Agriculture (CNFA) and USAID’s Economic Security Program implemented by DAI. The purpose of the performance evaluations is to: 1) determine the effectiveness of specific Programmatic approaches in achieving intended life-of-Program results; 2) to provide recommendations on corrective actions and new directions for the remaining years of Program implementation; and 3) inform the design of future programs. This Evaluation Work Plan describes the approach that the LEAP III team will take to this evaluation, along with protocols for electronic surveys and semi-structured interviews (Annexes A-E), proposed timeline (Section 6), Getting to Answers Matrix (Section 4), and List of Resource Documents (Annex G), to implement the mid-term performance evaluation. The Evaluation Team (ET) is closely monitoring the COVID-19 situation and related risks and will adjust its data collection protocols according to prevailing circumstances. The ET exit briefings and workshops may be done in person, depending upon the situation in consultation with USAID/Georgia regarding country and local safety protocols. 2. BACKGROUND 2.1 AGRICULTURE PROGRAM Agriculture is one of three sectors, along with tourism and light manufacturing, identified as a key driver for Georgia’s economic growth and employment. While agriculture is not a large contributor to Georgia’s Gross Domestic Product (GDP), it provides a safety net to 50 percent of Georgia’s population. Georgia’s Euro-Atlantic aspirations include becoming a major trade partner with the West; however, Georgia still depends on Russia as a destination market for their agricultural products. USAID/Georgia has a robust agricultural portfolio that facilitates firm-level investments in entrepreneurial and market-driven enterprises and associations, by supporting the private sector to meet the sector’s needs, and by enhancing the capacity of relevant government stakeholders to catalyze agricultural development. The Programming focuses on improving the competitiveness, diversity, value, and market access for identified value chains, and using those value chains as providers of high-value employment. USAID’s Agriculture Program aims to accelerate the growth of agricultural sub-sectors that demonstrate strong potential to create jobs, increase incomes and revenues of micro, small, and medium enterprises (MSMEs), and diversify export markets away from malign countries. These horticulture sub-sectors include berries (including kiwi fruit), culinary herbs, stone fruits, perishable vegetables, pome fruits (apples), table grapes, mandarins, and nut crops (pistachios, almonds, walnuts). The development hypothesis for the Program is that increased competitiveness of these key sub-sectors and value chains will advance inclusive high-value employment opportunities for Georgians and help diversify export markets away from Russia. The Program is implemented through two integrated, mutually USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 71 reinforcing components. Under Component One, the Program provides cost-share grants to MSMEs, cooperatives, service/information/extension providers, and associations, while Component Two focuses on demand-driven technical assistance, including technical and/or business trainings, and a wide spectrum of customized consultancies aimed at building the capacity of targeted value chain actors. 2.2 ECONOMIC SECURITY PROGRAM Despite Georgia being a global leader in trade and business environment reforms, economic growth has not resulted in employment opportunities or higher wages. An aggressive reform agenda and healthy growth rate have not translated into economic dynamism or opportunities for Georgian citizens. There are multiple causes why Georgia’s economic growth has not created high-value employment opportunities for its citizens, limiting the benefits of Georgia’s Western orientation. One such cause is that Georgian firms still lack access to resources necessary to increase competitiveness and create greater employment opportunities in key sectors, including access to high-value, diverse markets; investment resources; and a workforce that has the skills demanded by the private sector. The recent COVID-19 crisis triggered a major economic recession in Georgia, resulting in the loss of jobs, local currency devaluation and has thrown into stark relief the need for an economy that delivers real gains to its citizens. The purpose of USAID’s Economic Security Program is to accelerate broad-based growth of sectors outside of agriculture that show strong potential to create jobs, increase MSME revenues, and support diversification to more productive economic activities in tourism, creative industries, light manufacturing (including furniture, packaging, personal protective equipment, and construction materials), information and communications technology (ICT), solid waste management, recycled materials, and shared intellectual services sectors. The underlying development hypothesis of the Program is that IF Georgia’s firms have access to the resources they need (capital, access to high-value markets, skilled workforce, modern technologies, etc.) to improve productivity, sales, and product and service quality, and IF cooperation is strengthened in targeted sectors and value chains, THEN targeted sectors/value chains will become more competitive and will provide greater high-value employment opportunities to its citizens and drive closer integration with the West. The contract is organized by four components designed to achieve the stated results: Component 1: Strengthen cooperation in targeted sectors: The Program provides technical assistance and cost-share grants to strengthen linkages and cooperation throughout value chains in targeted sectors and improve support services intended to enhance growth and productivity across targeted value chains in target sectors. In doing so, the Program takes a collaborative approach to development, working with a plethora of stakeholders including firms, associations, Government of Georgia (GOG) agencies, development partners, regional government and municipalities, and other stakeholders, Component 2: Support Enterprises to improve productivity, sales, quality, and develop new products and services: Through identification and exploration of value chains that provide the best opportunity for Georgia to initiate investment that leads to high-value jobs, the Program facilitates entrance into new markets. It also increases and expands product offerings, promotes stronger linkages between enterprises and the organizations that support them, and enhances the overall value chain ecosystem to ensure sustainability. This is accomplished through a series of interventions that include technical assistance, cost￾share grants, and export enhancement, among others. Component 3: Industry-led workforce development: Alignment of Georgia’s workforce with the needs of industry is critical to the country’s movement toward the development of a prosperous society. As such, the Program works with the Ministry of Education, Science, Culture and Sport (MoESCS), educational USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 72 institutions, training providers, and the private sector to prepare Georgians for new and expanded employment opportunities through identified sectors and value chains. The Program’s approach is led by industry, meaning that it focuses on improving knowledge and skills that align with emerging investment and job opportunities. This requires significant re-thinking of educational and vocational models, as well as specific interventions that will link skills development directly with employment. Component 4: Building public-private partnerships: Through its Partnership Development Fund (PDF) with the total value of $3 million, the Program co-creates and co-funds Public Private Partnerships (PPPs), Global Development Alliance (GDA) mechanisms, and other investment opportunities that support the growth of identified sectors/value chains and that provide high-value employment for Georgians. Approaches under PDF are collaborative, innovative, and flexible to identify and take advantage of emerging opportunities. 3. EVALUATION PURPOSE, AUDIENCE, AND QUESTIONS 3.1 PURPOSE The purpose of the performance evaluations is to: 1) determine the effectiveness of specific Programmatic approaches in achieving intended life-of-Program results; 2) to provide recommendations on corrective actions and new directions for the remaining years of Program implementation; and 3) inform the design of future programs. The performance evaluation will: ● Assess the relevance, effectiveness, and efficiency of the Programs; ● Analyze the status of the activities in relation to the set objectives, activities’ successes and weaknesses; ● Assess the implementing organization’s performance in achieving Program objectives, including a special focus on uptake of principles and approaches in line with USAID’s Private Sector Engagement (PSE) policy and Digital Strategy, utilization of partnerships and market systems development (MSD) approaches, as well as it’s processes, implementation team performance, relations with stakeholders, performance feedback loops, reporting, timely management decisions, etc.; ● Identify any external factors which might have impacted activity performance and/or created new opportunities, such as political, economic, sector dynamics, as well as COVID-19; ● Provide recommendations on adjustments and/or corrective actions and new directions for the remaining years of Program implementation; ● Inform USAID/Georgia on future Programming needs and approaches, in particular the design of future follow-on projects. 3.2 AUDIENCE The primary audience of both evaluations will be USAID/Georgia’s Economic Growth team and the prime implementing partners (IPs) – CNFA for the Agriculture Program and DAI for the Economic Security Program. USAID/Georgia may also share the results of this evaluation with local stakeholders such as the Ministry of Economy and Sustainable Development, Rural Development Agency, partner non￾governmental organizations (NGOs), and other donors working in this area. Additionally, the results of USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 73 the Agriculture Program evaluation may be shared with the Ministry of Environmental Protection and Agriculture. 3.3 EVALUATION QUESTIONS AGRICULTURE PROGRAM The evaluation will address the following evaluation questions (EQs): EQ1. Diversification of export markets: To what extent have the export capacity building interventions with firms, and export promotion interventions with the government, been necessary and sufficient to diversify agricultural exports of target products to United States Government (USG) preferred markets (i.e. outside of Russia)? What have been the most pressing challenges in each priority value chain hindering the diversification of export markets? How sustainable are the USG-supported market linkages? ● What are the main challenges to expanding export diversification for MSMEs away from Russia in targeted value chains? (Probe: capacity building, export promotion, technologies, finding skilled labor, business enabling environment [BEE], market information, quality and safety standards, etc). ● What are the top business opportunities from the perspective of target MSMEs? (Probe: does it involve export diversification). ● What are the most important factors that influence decisions to diversify export markets? ● How important are prevailing cultures, attitudes, and/or perceptions in driving export decisions? How successful has the USAID Agricultural Program been in shifting these factors (probe: what are the perceived strengths and weaknesses of the Russian market)? ● What support has been provided by the USAID Agricultural Program to MSMEs in entering international markets, particularly in Western Europe? How effective has this support been? (Probe: have target MSMEs entered Western Markets, what are the determining factors for this, what challenges or opportunities are faced). ● What additional support is needed to facilitate expanded access to non-Russian export Markets? ● How effective are similar activities funded by other donors or the GOG in capacity building and export promotion? ● Is USAID's Agriculture Program's support to GOG to develop a "Georgian brand" for international markets and promote agriculture exports through trade shows, digital platforms linking MSMEs to potential export markets, etc., yielding positive results? ● Are linkages to non-Russian export markets sustainable? (probe: explore sustainability factors) ● Can services supporting sustained exports to non-Russian export markets be cost-shared or offered as paid services? (Probe: any currently available services from the private sector) EQ2. Value-chain approach: To what extent has the activity’s support to sector associations, cooperatives and government institutions catalyzed priority value chain development? How important a role have these stakeholders played in increasing value chain competitiveness? To what extent have these stakeholders received sufficient and relevant support? ● Who are the most important stakeholders that need to be involved in value chain development and increasing competitiveness of MSMEs in priority value chains? USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 74 ● What services do these stakeholders provide to MSMEs and other sector stakeholders (e.g. individuals, GoG institutions, NGOs, education institutions, etc.)? (Probe: Are these services of acceptable quality? Are there gaps? How can these services be improved or expanded?) ● Which associations, cooperatives and/or other stakeholders have been most effective in catalyzing value chain development and increasing value chain competitiveness and why? (Probe: How have USAID Programs and partnerships supported them in these efforts and was the support received sufficient and relevant?). ● How can associations, cooperatives, other private sector organizations inform and support the GOG as it improves the business enabling environment and encourages market systems strengthening? ● Does USAID support to GOG agencies help address value chain gaps? (Probe: certifications (i.e. nursery, phytosanitary, HAACP, ISO, etc.) ● Which activities by USAID, if any, encouraged greater female participation and youth in these targeted sectors? ● To what extent did USAID Programs support businesses in utilizing digital tools, including those that facilitate access to information, services and markets, either administered by the state or regional authorities, or by private sector organizations? (Probe: access to finance, certification, customs, public procurements, etc.). ● How does the Agriculture Program encourage grant applications from new partners under USAID’s New Partnerships Initiative (NPI)? What types of additional technical, managerial and operational support does DAI provide to new partners receiving grants? ● What PSE opportunities have been facilitated by the Agricultural Program and how have these partnerships supported priority value chain development (Probe: the role of PSE in value chain development across different Program activities, the types and number of engagements that have taken place, and the outcomes associated with engagements)? EQ3. Grant component: To what extent has the grant component strengthened each priority value chain? To what extent did the grants address gaps or market failures in target value chains? ● What were the key market gaps in each target value chain at the inception of the Agricultural Program? (Probe: consolidation facilities, quality inputs, cold storage, distribution infrastructure, certification, packing and labeling, logistics, etc.) ● To what degree did grants transform the priority value chain by addressing these gaps? (Probe: access to finance, increased sales, hiring talent, product differentiation, research and development (R&D), knowing customer demand, equipment, new varieties, expansion of production facilities, supply chain infrastructure). ● From the perspective of target MSMEs and market actors/stakeholders they work with, what types of grants and in which areas of business operations would grant support be most impactful? EQ4. COVID-19: In the context of COVID-19 economic contractions, how can the activity adapt its approaches (e.g. selection of grant solicitation themes, division of USAID investments across sub-sectors, sequencing of interventions, etc.) to improve its ability to USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 75 achieve its targets: creation of 3,680 jobs and increase agricultural sales by $70 million, including $23 million in new exports? ● What have been the biggest challenges and opportunities in the agriculture sector due to COVID￾19? ● How has the Agricultural Program responded to these challenges and opportunities? (Probe: using distance communication tools, introducing post-COVID recovery measures or Programs, organizing capacity building in COVID related topics, supporting with hygiene tools and stocks, etc.)________ ● What are best practices in how MSMEs and the market actors/stakeholders they work with have successfully adapted amid COVID-19 (Probe: issues related to labor, new regulations, shifting to online sales) ● What further opportunities are there for the Agricultural Program to target? ECONOMIC SECURITY PROGRAM The evaluation will address the following specific questions: EQ1. Private sector engagement: To what extent has the PDF targeted and established high￾impact (defined as wide-reaching and/or replicable) partnerships with the private sector that have strengthened and catalyzed the development of priority value chains? To what extent are these partnerships sustainable (defined as the establishment of market linkages that will not depend on USAID assistance after the activity ends)? ● Does the PDF engage with high-impact private sector partners to achieve its goal of supporting the growth of identified sectors/value chains and provide high-value employment for Georgians (Probe: what are the challenges and approaches used by the project to attract the right mix of partners to achieve Program objectives and what indicators is the project using to track current impact and the potential for sustained impact beyond the life of partnerships?) ● Are the types of PDF partnerships sufficient to achieve its goal of supporting the growth of identified sectors/value chains and provide high-value employment for Georgians (Probe: size, scalability, innovation) ● What factors are influencing the decisions of the private sector to co-fund with the PDF? (Probe: is the Economic Security Program’s value proposition to the private sector working, do private sector firms contribute enough resources) ● What results has the PDF had to date (Probe: How does this differ from expectations?) ● What other PSE approaches (in line with the PSE policy) and private sector collaboration mechanisms are active in the priority value chains (Probe: how do stakeholders perceive the PDF versus these mechanisms and the value proposition of USAID beyond a source of funding). EQ2. Value chain approach: To what extent has support to sector associations and government institutions catalyzed priority value chain development? How important a role have these stakeholders played in increasing value chain competitiveness? To what extent have these stakeholders received sufficient and relevant support? ● Who are the most important stakeholders that need to be involved in increasing competitiveness of MSMEs in priority value chains? USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 76 ● Which associations and/or other stakeholders have been most effective in catalyzing value chain development and increasing value chain competitiveness and why? (Probe: How have USAID Programs supported them in these efforts and was the support received sufficient and relevant?). ● What services do these stakeholders provide to MSMEs and other sector stakeholders (e.g. individuals, GoG institutions, NGOs, education institutions, etc.)? (Probe: Are these services of acceptable quality? Are there gaps? How does the fee structure look like? How can these services be improved or expanded?) ● Has USAID’s support to GOG entities, including Enterprise Georgia, Georgia’s Innovation and Technology Agency (GITA), and the Georgian National Tourism Administration (GNTA), been effective? ● Which associations have been most effective in increasing value chain competitiveness and why? (Probe: access to finance, support services, export enhancement, etc.). ● To what degree are gender and youth considerations integrated into USAID Economic Security Program activities (Probe: which activities, if any, encouraged greater female and youth participation in these targeted value chains?) EQ3. Grant component: To what extent has the grant component strengthened each priority value chain? To what extent did the grants address gaps or market failures in each value chain? ● What are the main gaps in the targeted value chains? (Probe: capacity building, export promotion, lack of technologies, unskilled workforce, lack of distribution channels). ● To what degree did grants transform the priority value chain by addressing these gaps? (Probe: access to finance, increased sales, hiring talent, product differentiation, R&D, knowing customer demand, equipment, expansion of production facilities, supply chain infrastructure). ● From the perspective of target MSMEs and market actors/stakeholders they work with, what types of grants and in what operational areas would grant support be most impactful? ● What PSE opportunities have been facilitated by the Economic Security grants (Probe: types of engagement, number of engagements, outcome of engagements) ● How does the Economic Security Program encourage grant applications from new partners under USAID’s New Partnerships Initiative (NPI)? What types of additional technical, managerial and operational support does DAI provide to new partners receiving grants? EQ4. Coordination on policy: To what extent has the Economic Security Program coordinated effectively with other USAID activities (managed by both the USAID Economic Growth and Democracy, Rights and Governance Offices) to address the policy barriers facing its priority sectors and value chains? To what extent has the absence of a large policy component within USAID’s Economic Security Program helped or hindered its ability to address policy gaps. ● What public-private dialogue activities do the Economic Security Program either host, or participate in collaboration with other USAID activities facilitate to ensure that the private sector’s voice is heard during the formulation of key regulations/policies (Probe: does the private sector believe their contribution is meaningful) USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 77 ● Despite not having a large policy component, is the Economic Security Program able to address policy issues raised through this public-private dialogue? (Probe: how many policy issues has the Program addressed and/or initiated) ● To what degree has the Economic Security Program been successful in referring policy issues to other USAID Programs, including the Economic Governance Program? (Probe: how many policy issues have been referred) ● Has the absence of this policy component constrained the effectiveness of the Program to address policy issues central to achieving its objectives? EQ5. COVID-19: In the context of COVID-19 economic contractions, how can the activity adapt its approaches (e.g. selection of grant solicitation themes, division of USAID investments across sub-sectors, sequencing of interventions, etc.) to improve its ability to achieve its targets: creation of 4,800 jobs and achieving $60 million in new sales? ● What are the biggest challenges and opportunities, if any, that MSMEs face amid the current COVID-19 pandemic? (Probe: issues related to labor, new regulations, shifting to online sales) ● What have been the biggest challenges and opportunities in the tourism sector due to COVID￾19? (Probe: how has the Economic Security Program responded to these challenges and opportunities) ● What have been the biggest challenges and opportunities in the ICT due to COVID-19? (Probe: how has the Economic Security Program responded to these challenges and opportunities) ● What further opportunities are there for the Economic Security Program to target? 4. EVALUATION DESIGN AND METHODOLOGY For these evaluations, primary and secondary data collection will be conducted using a mixed-methods approach. Whenever possible, existing quantitative data will be utilized. Survey-based instruments will be developed to collect quantitative data to fill existing knowledge gaps. Qualitative data will be collected primarily through remotely conducted key informant interviews (KIIs) with USAID staff, including the Contracting Officer’s Representative (COR), Activity Managers, mission staff points of contact, the IP, project partners and beneficiaries, and other actors and stakeholders relevant to the Program and to informing evaluation findings, insights, and recommendations. LEAP III has formed two ETs that will ensure strong collaboration and knowledge sharing across both evaluations. Both ETs will follow the same team structure with an international evaluation lead, a local senior evaluation specialist and a local subject matter expert. Further details on the team members can be found below. A central activity management team which includes the LEAP III core staff will be responsible for client management, quality control, operations, and coordination across teams. Activity Manager, David Quinn will be responsible for quality assurance. Ms. Pin Thanesnant will serve as the Operations Lead and will support the management of the activity. She will also support the development of the Evaluation Work Plan, assist in data collection as needed, ensure all work is streamlined, and provide inputs for the draft and final reports. LEAP III Associate, Ms. Liesl Kim, will provide administrative, logistical, and operations support. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 78 Economic Security Evaluation Team Team Lead: Brenda Pearson Senior Evaluation Specialist: Maia Giorbelidze Private Sector Expert: Rati Gabrichidze Sector/Value Chain Advisor: Lasha Kavtaradze Facilitator: Rusudan Gogibedashvili Agriculture Evaluation Team Team Lead: Nikolaus Eichman Senior Evaluation Specialist: Mikheil Pakatsoshvili Agriculture Expert: Grigol Modebadze Facilitator: Ani Chokhonelidze 4.1 USE OF BEST PRACTICES The evaluation will use methods that generate quality data and credible evidence that correspond to the questions being asked, taking into consideration time, budget, and other practical considerations. The evaluation will use sound social science methods and include the following basic features: 1. Establish a team with the appropriate methodological and subject matter expertise to conduct an excellent mid-term performance evaluation; 2. Ensure transparency and dissemination of the evaluation design and final report, including briefings and presentations to the Missions and the posting of the final report through USAID￾funded information dissemination websites; 3. Use data collection and analytic methods that ensure, to the maximum extent possible, that if a different, well-qualified evaluator were to undertake the same evaluation, he or she would arrive at the same or similar findings and conclusions; 4. Communicate and present separately the credible findings, conclusions and recommendations so the progression is clear and easy to follow in relation to each of the evaluation questions included in the Evaluation scope of work; and 5. Remain vigilant and flexible to the changing environment caused by the COVID-19 pandemic that impacts travel to and within Georgia. The ET will consult with USAID as it determines how to safeguard the health and safety of its team members. 4.2 EVALUATION DESIGN These two mid-term performance evaluations will be cross-sectional, descriptive, and analytical evaluations employing mixed methods of data collection, combining qualitative information collected from interviews, quantitative results of electronic surveys, and the collection of quantitative data from project monitoring and the verification of reported results to assess the success, challenges, and sustainability of both the Agriculture and Economic Security Programs. The data collection methodology includes the following: a) document review; b) performance indicator assessments ; c) electronic surveys; d) KIIs and 22 22 The ET will review performance indicators found in project documentation (e.g., contract agreement, work plans, annual reports) and incorporate as appropriate to address the evaluation questions. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 79 group interviews; and e) strategic consultations with IPs and the private sector to inform recommendations related to Program uptake of PSE and MSD principles and approaches. The data collection and analysis efforts are framed to assess activity implementation and how it affects activity outputs and results to date (September 2018 - September 2021 for CNFA and April 2019- September 2021 for DAI). The EQs are intended to highlight best practices and to identify challenges to the implementation of activities’ objectives. Based on consultations with USAID and the ET’s desk review, the ET will select appropriate key informants for interviews and determine the optimal use of group interviews if feasible. The ET will develop interview protocols for KIIs with USAID/Georgia staff and IP staff as well as in-country group and individual interviews with local partners and beneficiaries involved in the activities. The data collection plan includes a comprehensive Getting to Answers Matrix in Tables 5 and 6 (see below) that map the EQs and sub-questions to data sources and data methods. Data sources include the USAID/Georgia database, original documents such as activity reports, monitoring, evaluation, and learning (MEL) plans, and activity-generated resources. Data analysis methods will include refining the descriptive statistics and content analysis. In addition to the evaluation questions and proposed sub questions, the ET will probe cross-cutting issues that are important to USAID/Georgia, such as the impact of COVID-19 on job creation and sales/exports, PSE, business enabling environment, and inclusive economic growth interventions supporting women, youth, and vulnerable populations. A summary of how the agriculture and economic security Programs address cross-cutting issues will be included in the narratives of both final evaluation reports. IMPACT OF COVID-19 ON JOB CREATION AND SALES/EXPORTS IN SELECTED SECTORS Both performance evaluations contain a similar EQ related to the impact of COVID-19 on achieving targets in job creation and revenue in new sales and/or exports. The ET will analyze the impact of the COVID-19 pandemic on targeted sectors including agriculture, e-commerce, tourism, light manufacturing, ICT, waste management and recycling. The ET will conduct a benchmark assessment to examine the dynamics of each sector and their contributions to GDP and employment for the period of 2015-2019. This assessment will measure the losses and gains of the targeted sectors at the beginning of the pandemic (second quarter of 2020) and measure the effects on employment.23 Next, the ET will identify whether any of these sectors have received one-off government support and estimate how this support may have helped the sector to minimize economic losses or maximize gains and maintain or increase employment. The ET’s senior macroeconomic advisor will use the Leontief Input￾Output Model and estimated multipliers for each sector to evaluate the development of the sectors in the medium to long-term. Our research approach will utilize the economic modeling, findings and forecasts recently presented in a similar study by the Asia Development Bank. 23 GDP and employment data by sector is available on the Geostat website and additional employment data can be obtained from the GOG Revenue Service. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 80 PRIVATE SECTOR ENGAGEMENT AND MARKET, SYSTEMS AND PARTNERSHIPS The ET will engage the USAID PSE Hub and USAID’s Market, Systems and Partnerships Program for PSE and MSD best practices that can guide and inform strategic consultations and resulting recommendations. The ET will also probe during KIIs to identify practical approaches to shift PSE from transactional to transformational engagement with the private sector and explore MSD approaches that can facilitate efficient and effective PSE in the targeted sectors. BUSINESS ENABLING ENVIRONMENT During desk review and qualitative data collection, the ET will consider broadly the norms, customs, laws, regulations, policies, international trade agreements and public infrastructure that facilitate or hinder specific products and services along the targeted value chains included in the two evaluations. The focus will be on the constraints and opportunities facing MSMEs. GENDER, YOUTH AND VULNERABLE GROUPS During the desk review stage, project documents and early communications with USAID/Georgia revealed that both projects’ activities should be implemented with consideration of gender equality, youth, and vulnerable groups. Inclusive development is important for USAID/Georgia as a cross-cutting issue for all projects and recognizes the importance of reflecting and understanding how socially vulnerable groups are engaged in entrepreneurship and private business development activities. The ET will probe during KIIs to identify both positive and negative unintended consequences of Program activities for women and youth within the local contexts and norms concerning employment and income generation in which they operate. This will be done in combination with other criteria such as age, income, urban/rural divide, etc. The ET will also consider to what extent inclusive development approaches are part of technical assistance provided by the Agriculture and Economic Security Programs. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 81 TABLE 5: GETTING TO ANSWERS MATRIX – AGRICULTURE PROGRAM EVALUATION EVALUATION QUESTIONS TYPE OF ANSWER/ EVIDENCE NEEDED (CHECK ONE OR MORE, AS APPROPRIATE) METHODS FOR DATA COLLECTION, E.G., RECORDS, KIIS, SURVEYS24 SAMPLING OR SELECTION APPROACH DATA ANALYSIS METHODS: FREQUENCY DISTRIBUTION TREND, CONTENT ANALYSIS YES/NO SOURCES SPECIFIC METHODS EQ1. To what extent have the export capacity building interventions with firms, and export promotion interventions with the government, been necessary and sufficient to diversify agricultural exports of target products to USG preferred markets (i.e. outside of Russia)? What have been the most pressing challenges in each priority value chain hindering the diversification of export markets? How sustainable are the USG-supported market linkages? Yes Description Key Performance Indicators Interviews KIIs Survey Quantitative: Performance Indicators Qualitative: Interviews Quantitative: Online Survey Survey of grantees and participants of capacity building interventions KIIs with IPs, COR, Activity Manager KIIs with the Ministry of Economy and Sustainable Development (MoESD), Enterprise Georgia, Ministry of Environmental Protection and Agriculture of Georgia (MEPA), Regional Development Trend analysis Content analysis Data disaggregated by sex and age Comparative analysis with baseline data Yes Comparison Yes Explanation 26 24 Data from evaluations are a deliverable and methods should indicate how data would be captured, i.e., for focus groups USAID requires a transcript. 25 25 Comparison – to baselines, plans/targets, or to other standards or norms 26 Explanation – for questions that ask “why” or about the attribution of an effect to a specific intervention (causality) USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 82 – Association (RDA), National Food Agency (NFA), laboratories, SRCA KIIs with private sector actors, including sector associations and cooperatives KIIs with other USAID projects KIIs with other donors EQ2. To what extent has the activity’s support to sector associations, cooperatives and government institutions catalyzed priority value chain development? How important a role have these stakeholders played in increasing value chain competitiveness? To what extent have these stakeholders Yes Description Data collection Interviews with key stakeholders Survey Quantitative: Data collection Qualitative: Interviews Quantitative: Phone survey KIIs with MoESD, Enterprise Georgia, MEPA, RDA KIIs with IPs, COR, Activity Manager KIIs with private sector actors, including sector associations and cooperatives Trend analysis Content analysis Yes Comparison Yes Explanation USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 83 received sufficient and relevant support? KIIs with the MoESD, Enterprise Georgia, MEPA, RDA, NFA, laboratories, SRCA Phone survey of farmers in priority value chains KIIs with grantees KIIs with other USAID projects KIIs with other donors EQ3. To what extent has the grant component strengthened each priority value chain? To what extent did the grants address gaps or market failures in target value chains? Yes Description Performance indicators Data collection Interviews Survey Market survey Quantitative: Performance Indicators Data collection Qualitative: Interviews Quantitative: Survey KIIs with IPs, COR, Activity Manager KIIs with private sector actors, including sector associations and cooperatives KIIs with the MoESD, Enterprise Georgia, MEPA, RDA, NFA KIIs with grant applicants and grantees Trend analysis Content analysis Data disaggregated by sex and age Yes Comparison Yes Explanation EQ4. COVID-19: In the context of COVID-19 economic contractions, how can the activity adapt its approaches (e.g. selection of grant Yes Description Performance indicators KIIs Interviews Quantitative: Performance Indicators, Online-Survey, KIIs with IPs, COR, Activity Manager KIIs with private sector actors, Content Analysis Trend Analysis Yes Comparison USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 84 solicitation themes, division of USAID investments across sub￾sectors, sequencing of interventions, etc.) to improve its ability to achieve its targets: creation of 3,680 jobs and increase agricultural sales by $70 million, including $23 million in new exports? Yes Explanation Official statistics from GeoStat / Revenue Service Qualitative: Interviews including sector associations and cooperatives KIIs, with GOG and other relevant stakeholders KIIs with the MoESD, Enterprise Georgia, MEPA, RDA, NFA KIIs with grantees Comparative analysis with pre-COVID-19 baseline data USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 85 TABLE 6: GETTING TO ANSWERS MATRIX – ECONOMIC SECURITY PROGRAM EVALUATION EVALUATION QUESTIONS TYPE OF ANSWER/ EVIDENCE NEEDED (CHECK ONE OR MORE, AS APPROPRIATE) METHODS FOR DATA COLLECTION, E.G., RECORDS, KIIS, SURVEYS SAMPLING OR SELECTION APPROACH DATA ANALYSIS METHODS: FREQUENCY DISTRIBUTION TREND, CONTENT ANALYSIS YES/NO SOURCES SPECIFIC METHODS EQ 1: Private sector engagement: To what extent has the PDF targeted and established high-impact (defined as wide-reaching and/or replicable) partnerships with the private sector that have strengthened and catalyzed the development of priority value chains? To what extent are these partnerships sustainable (defined as the establishment of market linkages that will not depend on USAID assistance after the activity ends)? Yes Descripti on Key Performance Indicators Interviews KIIs Survey Quantitative: Performance Indicators Qualitative: Interviews KIIs with USAID, DAI, Solimar International, PMCG KIIs with Ministry of Economy and Sustainable Development KIIs with Ministry of Finance Interviews with all partnerships KIIs with Enterprise Georgia KIIs with GITA KIIs with GNTA Online survey of business associations Trend analysis Direct attribute/linkages Content analysis Comparative analysis with baseline data Yes Comparis on Yes Explanati on USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 86 – Online survey of MSMEs (including Biz￾link beneficiaries) Administrative data from National Statistics Office of Georgia Revenue Service EQ 2: Value chain approach: To what extent has support to sector associations and government institutions catalyzed priority value chain development? How important a role have these stakeholders played in increasing value chain competitiveness? To what extent have these stakeholders received sufficient and relevant support? Yes Descripti on Data collection Interviews with key stakeholders Survey Quantitative: Data collection Survey Qualitative: Interviews KIIs with USAID, DAI, Solimar International, PMCG KIIs with Ministry of Economy and Sustainable Development KIIs with Ministry of Finance Interviews with all partnerships KIIs with Enterprise Georgia KIIs with GITA KIIs with GNTA KIIs with sampled business associations KIIs with beneficiaries of business associations Cross tabulations Trend analysis Content analysis Descriptive statistical analysis Direct attribution/linkages Data disaggregated by gender, age No Comparis on Yes Explanati on USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 87 Online survey of business associations Online survey of MSMEs (including Biz￾link beneficiaries) Online survey of #Go4It Interns EQ 3: Grant component: To what extent has the grant component strengthened each priority value chain? To what extent did the grants address gaps or market failures in each value chain? Yes Descripti on Data collection Interviews with key stakeholders Online survey Quantitative: Data collection Survey Qualitative: Interviews KIIs with USAID KIIs with DAI staff (Chief of Party (COP), Deputy Chief of Party (DCOP), Grants Director, Monitoring and Evaluation Manager) Online survey of Grantees Interviews with sampled grantees Online survey of MSMEs (including Biz￾link beneficiaries) Cross tabulations Descriptive statistical analysis Trend analysis Content analysis Direct attribute/linkages Gap analysis Data disaggregated by sex Yes Comparis on Yes Explanati on EQ4. Coordination on policy: To what extent has the Economic Security Program coordinated effectively with other USAID activities (managed Yes Descripti on Data collection Interviews with key stakeholders Online survey Qualitative: Interviews KIIs with USAID, DAI, PMCG KIIs with other USAID funded Programs Content analysis Gap analysis Direct attribution/linkages Yes Comparis on USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 88 by both the USAID Economic Growth and Democracy, Rights and Governance Offices) to address the policy barriers facing its priority sectors and value chains? To what extent has the absence of a large policy component within USAID’s Economic Security Program helped or hindered its ability to address policy gaps? Yes Explanati on KIIs with other international-aid provider agencies KIIs with Parliament Online survey of business associations Online survey of grantees KIIs with sampled grantees and business associations Gap analysis EQ5. COVID-19: In the context of COVID-19 economic contractions, how can the activity adapt its approaches (e.g. selection of grant solicitation themes, division of USAID investments across sub￾sectors, sequencing of interventions, etc.) to improve its ability to achieve its targets: creation of 4,800 jobs and achieving $60 million in new sales? Data collection Interviews with key stakeholders Online survey Quantitative: Data collection Qualitative: Interviews KIIs with DAI COP, DCOP, Grants Director KIIs with Solimar International, PMCG KIIs with USAID COR/Agreement Officer’s Representative (AOR) KIIs with GNTA KIIs with GITA Online survey of MSMEs (including Biz￾link beneficiaries) Online survey of business association Online survey of #Go4It Interns Content analysis Comparative analysis with pre￾COVID-19 baseline data Systematic Document Review USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 89 4.3 DOCUMENT REVIEW The ET will conduct a focused document review of relevant resources as well as activity documents. Key variables of the review will include the purpose, goals and objectives, interventions, results, and sustainability of each activity, as well as best practices in PSE and MSD approaches that could be applied by these and future programs in Georgia. The ET will enter this information into Excel files to serve as display tables for analysis. The ET will review USAID and IP documents in an iterative process of data analysis and writing, including the following resources. 4.4 PRIMARY DATA: SURVEYS AND INTERVIEWS To accommodate the COVID-19 situation as well as harvest season, the team will first disseminate online surveys early in the data collection process. The analysis and initial findings from the surveys can be used to inform KIIs and focus group discussions (FGDs). This two-step approach will allow for probing deeper for insights during the KIIs and group interviews. The primary data collection will be conducted in the Georgian language for all respondents (unless a respondent requests the interview to be conducted in English). The electronic survey and KIIs with USAID staff and other donor partners will be conducted in English. The survey instruments are presented in English in Annexes A and B but will be translated into Georgian prior to dissemination. ELECTRONIC SURVEYS The ET will develop and conduct seven online surveys, which may be found in Annexes A and B, as a pre￾screening and data collection instrument before conducting the KIIs. Based upon the desk review, discussions with USAID staff and preliminary consultations with the two IPs, the electronic surveys will target the following stakeholders: 1. Agricultural Program grantees; 2. Agricultural Program recipients of technical assistance; 3. Agricultural Program and Economic Security grant applicants who did not receive grants; 4. Economic Security Program affiliated Business Associations; 5. Economic Security Program affiliated MSMEs; 6. Economic Security Program grantees; and 7. Economic Security Program interns. These surveys will be structured and utilize a combination of dichotomous questions, i.e. Yes/No/Don’t Know, Likert scale (using a 5-point rating scale), and open-ended responses. The online surveys will be hosted using the online platform, Survey Monkey. Respondents will be requested to complete the survey within one week of receipt, and reminder emails will be sent to those who do not complete the survey. After piloting the electronic surveys, adjustments may be necessary, and the team will determine whether incomplete surveys will be accepted. Based on previous experience, the ET anticipates an estimated 20- 25 percent response rate. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 90 The sample size for each of the seven surveys will be approximately 10 percent of key stakeholders, which is derived by power calculations using a power factor of 1.0 for determining the probability of significance, using the formula below: Where α is the selected level of significance and Z 1-α /2 is the value from the standard normal distribution holding 1- α/2 below it. For example, if α=0.05, then 1- α/2 = 0.975 and Z=1.960. 1- β is the selected power, and Z 1-β is the value from the standard normal distribution holding 1- β below it. ES - Effect Size. KEY INFORMANT INTERVIEWS The ET will use a purposive sampling for selecting key informants who will be chosen specifically for their relationship to the project. These will include IP staff, USAID staff, including the COR, Program staff and/or other USAID Economic Growth staff that have been involved in the projects and Program Office staff that have been involved in design and monitoring and evaluation. Other KIIs may include private sector partners and stakeholders, associations, other donor partners, NGOs, and GOG officials. The ET will conduct qualitative, in-depth individual interviews with key informants. In rare instances, in-person interviews may be possible. However, it is expected that most interviews will be conducted via video conferencing using the Google Meet platform (for all USAID staff), Zoom, Skype, and WhatsApp platforms. Both ETs will follow the same structure during data collection, including streamlined stakeholder outreach to ensure no duplication in outreach and frequent technical dialogue between both ETs. All travel and interview schedules will be centralized and managed by the local coordinators to ensure all team members have access and can plan each day accordingly. Throughout data collection, the teams will have frequent check-in meetings to discuss preliminary findings and lessons learned from each day, as well as plan for the days ahead (i.e., addressing schedule changes, coordinating meetings, organizing updated stakeholder lists, etc.) The teams will conduct three weeks of KIIs and FGDs in Tbilisi and other parts of Georgia as needed to better understand Program impact at the level of the operating environments of target beneficiaries. If the ET believes it is feasible to conduct in-person interviews, USAID will have a chance to approve proposed travel itineraries of the teams before deployment. SITE VISITS/DIRECT OBSERVATION The ET will consult with DAI and CNFA staff to assess opportunities to conduct either virtual or in￾person site visits and direct observations in accordance with prevailing circumstances to obtain additional insights in assessing the quality of services or training provided, way of event organization, beneficiaries’ skills, and communications channels. If the ET elects to conduct direct observations, the ET will follow the USAID protocol guide for conducting site visits and develop a site summary report based on this guidance (Annex F: Site Visit Summary). For example, the ET will plan to observe a business clinic workshop for BizLink beneficiaries of the Economic Security Program and has coordinated with the DAI team for access to this event. Other direct observation opportunities will be explored during the data collection period for this evaluation. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 91 All proposed KIIs and FGDs, as well as the surveys and direct observations, are organized around key evaluation questions and supported with detailed questionnaires. Each questionnaire will be developed for each specific group of interviewees and includes both common questions as well as questions unique to each group (clearly marked), which will allow the team to obtain the full range of opinions regarding specific projects but also to ensure that data is comparable across all the respondent groups. The ET will take detailed field notes in support of any direct observations consistent with USAID’s ADS (Chapters 201, 320, and 578 as well as relevant mandatory references) and USAID’s Evaluation Policy (January 2016). (See Annex C-E for more information regarding the data collection protocols). TABLE 7: STAKEHOLDER CATEGORIES STAKEHOLDER CATEGORY JOINT AG + ECON SECURI TY ETS AG PROGRAM ET ECONOMIC SECURITY PROGRAM ET COMMENT USAID X Both ETs will conduct joint interviews with senior leadership of USAID. ETs will facilitate a separate Program-specific discussions with AOR/COR of respective Program Implementing Partners X X ETs will conduct KIIs with Implementing Partners of respective Programs. Other USAID￾supported Programs X Both ETs will conduct joint interviews with other USAID￾supported Programs Other donor-funded Programs X X (Food and Agriculture Organization (FAO), United Nations Development Programme) X Both ETs will conduct joint interviews with other donor-funded Programs with whom both Programs cooperated with. In addition, a separate Program￾specific discussions will be facilitated by respective ET as needed Government of Georgia X (Enterpris e Georgia, GITA, RDA, which oversees Informati on X (MEPA,RDA, NFA, laboratories, SRCA) X (MoESD, Ministry of Finance, Parliament of Georgia) Both ETs will conduct joint interviews with the representatives of Government of Georgia. In addition, a separate Program￾specific discussions will be facilitated with by respective ET as needed USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 92 Consultat ion Centers) Beneficiaries (grantees, business associations, interns) X X ETs will conduct KIIs and online surveys of beneficiaries of respective Programs. PPP and GDAs X ETs will conduct KIIs with PPPs and GDAs established within Economic Security Program Private Sector Actors X X ETs will conduct KIIs and online surveys of private sector actors to collect the responses on Evaluation Questions for each Program. Private Sector Partners (producers, buyers, service providers) X X ETs will conduct KIIs with private sector partners of Agricultural Program Financial Institutions X X ETs will conduct KIIs with Financial Institutions to measure accessibility to finances within Economic Security Program STRATEGIC CONSULTATIONS The ET will conduct strategic consultations with IPs and the private sector to better understand constraints and opportunities related to Program objectives. Strategic consultations will also inform an assessment of Program uptake of principles and approaches in line with USAID’s PSE policy and Digital Strategy, utilization of partnerships and market systems development approaches and assist in identifying related and actionable recommendations for the remaining years of Program implementation and future USAID/Georgia Programming. 4.5 DATA ANALYSIS METHODS The mid-term performance evaluations include comprehensive Getting to Answers matrices (see Table 5 and Table 6) that map the EQs to data sources and data analysis methods. Once the ET’s data plan is developed fully, the ET will use a document review instrument to guide data collection along with interview guides and protocols for conducting the video conferencing interviews. The ET will then collect data from the document review and interviews, then analyze it using descriptive statistics or content analysis to develop the findings to answer the EQs. The ET will also use descriptive statistics to produce a quantitative overview of both activities, including characteristics such as the number of participants, regions, and in-country partners. The ET will use standard qualitative analysis to review the data summaries and data display tables described above. Secondary priority will be given to less common themes and patterns that illustrate key characteristics relevant to the EQs. The ET will conduct semi-structured interviews with USAID staff, IPs, their partners USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 93 and stakeholders and selected beneficiaries to gather their perspectives and additional information about the activities. The team proposes to use a manual review process to extract key data such as keywords, quotes, or substantive information about activities from the transcripts. A core technical approach will be triangulation: the systematic, evidence-based, careful synthesis of disparate findings (from a broad variety of data sources) to discern consistent themes, trends, and patterns. Because the ET will be synthesizing data from multiple sources, it is imperative that the ET employs a broad variety of analytical technical techniques throughout the mid-term performance evaluation. These techniques will be customized to fit both the available data sources and address the EQs provided in this plan. 4.6 POTENTIAL LIMITATIONS Selection bias: As some key informants may decline to be interviewed, there is a possibility of selection bias. Those respondents who chose to be interviewed might differ from those who did not in terms of their attitudes and perceptions, affiliation with government/non-government structures, and socio￾demographic characteristics and experience. The ET will mitigate by developing a purposive sampling of key informants. Limited Fieldwork: Due to the COVID-19 situation, both evaluations will be primarily conducted remotely. There is a disadvantage that ET members cannot be in-country to speak with stakeholders in￾person and experience the activities on the ground. The ET will work to mitigate all data collection issues by planning in advance and working with local team members to help coordinate in-country. The ET will take into consideration lessons learned during recent evaluations using remote data collection methods. Instrumental Bias: Guarding against instrumental bias is a consideration because many beneficiaries and in-country partners were exposed to or participated in more than one intervention and their responses may be influenced by participation in multiple interventions. The ET will note if key stakeholders participated in multiple activities. Difficulty assessing progress in addressing gaps: The evaluation will be conducted while the reform implementation and capacity building interventions are ongoing, and the influences on the relevant value chain may take years to deliver intended results. The ET will assess progress to date and highlight potential gaps that should be addressed in end line evaluations. Availability of respondents for key informant interviews: Due to complications related to COVID￾19, some respondents may not be available due to precautions, government restrictions or limited internet connectivity. Scheduling interviews with farmers during harvesting season may pose challenges in sampling size and scheduling, therefore the ET will adjust sampling size as needed. Complexity of questions during phone interviews: Due to challenges posed by COVID-19, the ET will rely more extensively on video and telephone interviews. This may cause the evaluators to simplify and shorten the duration of KIIs, thus resulting in somewhat limited data availability. The ET will adjust its interview techniques to accommodate respondents’ abilities to elaborate. 4.7 COLLABORATION, LEARNING AND ADAPTING The ET will work closely with USAID/Georgia to present its preliminary findings and recommendations with USAID mission staff and IPs. The ET proposes that two exit briefings and one recommendations and validation workshop be held after the data analysis has been completed and prior to report writing. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 94 Exit Briefings: Upon the conclusion of fieldwork, the teams will deliver exit briefings to report on initial findings and observations. The ET proposes conducting two separate exit briefings with relevant mission staff. The Exit Briefings will include general findings, conclusions, and anticipated recommendations on Programs, as well as high-level comparisons of cross-cutting lessons. These cross-cutting lessons will be derived from a comparative analysis of the findings from both evaluations. Recommendations and Validation Workshop: The Chief of Party and two Team Leaders will facilitate a 90-minute validation workshop with selected staff from USAID/Georgia to include CORs, representatives from the economic growth and Program office and senior leadership. The purpose of this validation workshop is to improve the evaluation learning and utilization through group discussion and shared understanding of the findings, recommendations, and key learning points. The format of the workshop will be determined two weeks in advance of the date, which will allow preparations for either an in-person or virtual discussion. Any feedback will be taken into consideration for the evaluation report. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 95 5. DELIVERABLES Under these evaluations, the LEAP III team will submit the following deliverables: TABLE 8: DELIVERABLES TIMELINE DELIVERABLE DUE DATE Evaluation Work Plan: This document outlines the methodology, limitations, timeline, and travel logistics for USAID/Georgia’s review and approval. Mission in-brief: Discuss evaluation design and questions With USAID office directors and senior leadership. August 23, 2021 September 15, 2021 Remote Data Collection: The evaluation team will utilize electronic surveys and online meeting methods to conduct KIIs and FGDs. Weeks of August 30 - September 27, 2021 Exit Briefings and Recommendations and Validation Workshop: The evaluation team will conduct separate presentations for USAID/Georgia on its preliminary findings on an agreed upon date in mid-October 2021. The team will also facilitate a validation workshop. These dates will depend on the schedules of USAID/Georgia and will be determined later. Exact date TBD ~week of October 11-15, 2021 Draft Evaluation Report: The draft evaluation report will adhere to USAID Evaluation Policy guidelines. Within 21 working days after data collection, the LEAP III team will provide to USAID/Georgia a draft of the report. November 1, 2021 Final Evaluation Report: Upon the receipt of the Mission’s comments on the draft report, the LEAP III team will finalize the report for submission. Within 10 days of receiving comments on the final report. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 96 6. TIMELINE Task J u n 2 8 J u l 5 J u l 1 2 J u l 1 9 J u l 2 6 A u g 2 A u g 9 A u g 1 6 A u g 2 3 S e p 6 S e p 1 3 S e p 2 0 S e p 2 7 O c t 4 O c t 1 1 O c t 1 8 O c t 2 5 N o v 1 N o v 8 N o v 1 5 N o v 2 2 N o v 2 9 1. Develop ment of AAR (USAID Approval) 2. Desk Review /Evaluatio n Design/W ork Plan 2.1. Initial Documen t And Desk Review USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 97 Task J u n 2 8 J u l 5 J u l 1 2 J u l 1 9 J u l 2 6 A u g 2 A u g 9 A u g 1 6 A u g 2 3 S e p 6 S e p 1 3 S e p 2 0 S e p 2 7 O c t 4 O c t 1 1 O c t 1 8 O c t 2 5 N o v 1 N o v 8 N o v 1 5 N o v 2 2 N o v 2 9 2.2. Evaluation Work Plan Develop ment (USAID Approval) 3. Mission In-Brief 4. Remote Data Collectio n 5.Exit Briefings and Recomme ndations USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 98 Task J u n 2 8 J u l 5 J u l 1 2 J u l 1 9 J u l 2 6 A u g 2 A u g 9 A u g 1 6 A u g 2 3 S e p 6 S e p 1 3 S e p 2 0 S e p 2 7 O c t 4 O c t 1 1 O c t 1 8 O c t 2 5 N o v 1 N o v 8 N o v 1 5 N o v 2 2 N o v 2 9 and Validation Worksho p 6. Draft Evaluation Report (USAID approval) 7. Final Evaluation Report (USAID approval) USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 99 ANNEX II.A. ONLINE SURVEY QUESTIONNAIRES - ECONOMIC SECURITY PROGRAM ANNEX II.A.1. ELECTRONIC SURVEY OF BUSINESS ASSOCIATIONS Thank you for participating in the electronic survey. The survey should take approximately 20 minutes to complete. Please answer as completely as you can. If you have any questions, please contact Dr. Maia Giorbelidze, giorbelidzemaia@gmail.com. 1. Sex a. Female b. Male c. Prefer not to state 2. Youth: Yes or No (18 - 29 years) 3. Region: 4. Please indicate name of your business association: 5. How many members does your business association include? a. 1-9 b. 10-19 c. 20-29 d. 30-39 e. 40-49 f. 50-59 g. 60-69 h. More than 70 6. In which value chain does your business association operate in? a. Tourism (Mountain/Adventure; Gastronomic; Culture/Heritage) b. Creative Industries (Production, Post-Production, Architecture, High-Value Artisans) c. Shared Intellectual Services (BPO, Nearshoring, Digital Transformation, e-commerce and ICT) d. Light Manufacturing (Furniture, Packaging, Personal Protective Equipment, and Construction Materials) USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 100 My organization’s relative strength and potential for sustainability were assessed e. Solid Waste Management f. g. Other, please specify 4. What are some of the challenges in your industry/business area (select all relevant responses) a. Lack of technologies b. Access to finance c. Lack of qualified staff d. Lack of distribution channels e. Research and development, f. Marketing g. Demand estimation h. Access to equipment i. Supply chain infrastructure j. Exchange rate dependency k. Administrative/policy barriers l. taxes m. Other, please specify 5. From which USAID Program did you receive support? a. Economic security b. Agriculture 6. What type of support have your business association received through Economic Security Program? (Mark all as relevant) a. b. Development of a customized strategy for sustainability c. Support in implementation of a customized strategy d. Grant support e. Business training f. Mentoring g. Other, please specify 7. What type of support have your business association received through the Agriculture Program? (Mark all as relevant) a. Individual consultancies (please specify which consultancies) USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 101 No No I don’t know b. Group trainings (please specify which trainings) c. Cost-share ISO 22 000 certification consultancy d. Gap Analysis e. In obtaining GlobalGAP certification f. GRASP certification g. HACCP certification h. Organic certification i. Participated in study tour / fair abroad (please specify which study tour / fair) j. Support in branding 8. Did Econ Security Program cooperate with you to identify policy barriers you are experiencing in your value chain? Yes ☐ Please describe the cooperation and its outcome 9. Did Economic Security Program create the possibility for your association to participate in public￾private dialogue? a. Yes ☐ Please describe the cooperation and its outcome 10. How would you evaluate the relevance of the support provided by the program? a. Very relevant b. Relevant c. Neutral d. Irrelevant e. Very irrelevant f. I don’t know 11. How would you evaluate the sufficiency of the support provided by the program? a. Very sufficient b. Sufficient c. Neutral d. Insufficient e. Very insufficient f. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 102 12. How useful was the cooperation with the USAID Econ Security Program to address the challenges identified above? A. Very useful B. Useful C. Neutral D. Not useful E. Not at all useful 13 To what extent do you agree/disagree to the statements below regarding the benefits of cooperating with the Economic Security Program Totally Disagree Disagree Neutral Agree Totally Agree Don’t know Assessment of my organization’s strength and potential for sustainability enabled me to have a clear picture on strengths and weaknesses of my organization The strategy for sustainability was developed with active participation of the representatives of my business association The support provided by Economic Security Program enabled us to have a clear development vision for future USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 103 The partnership with Economic Security Program did not impact our operations neither in a positive, nor negative way The partnership with Economic Security Program was a loss of time 14. What was the result of your cooperation with the Economic Security / Agriculture Program (please name any additional service, any new product that was created as a result of cooperation)? 15. How would you rate your overall satisfaction with the support you received from the program? a. Very satisfied b. Satisfied c. Neutral d. Unsatisfied e. Very unsatisfied f. I don’t know 16. How would you rate your effectiveness of your partnership with the program? a. Very effective b. Effective c. Neutral d. Ineffective e. Very ineffective f. I don’t know 17. To what extent did COVID-19 impact on the operations of your business association? a. Very significant b. Significant c. Neutral d. Insignificant USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 104 e. Did not have any impact f. I don’t know 18. To what extent did COVID-19 impact on the operations of members of your association? a. Very significant b. Significant c. Neutral d. Insignificant e. Did not have any impact f. I don’t know 19. Did COVID-19 create any new opportunities for you and your members? Yes ☐ Please explain No 20. How did the employment figures change in your organization due to COVID? a. Increased, indicate # b. Decreased, indicate # c. Has not changed How did the employment figures change within your members due to COVID? a. Increased, indicate # b. Decreased, indicate # c. Has not changed 20. How would you assess the sustainability of your business operations after finishing the engagement with Economic Security Program? a. Very sustainable b. Sustainable c. Unsustainable d. I don’t know 21. How do you see the performance of your business association during the next several years? a. Significantly improving b. Improving c. Same USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 105 d. Worsening e. Significantly worsening 22. Have you received any support from Economic Security Program regarding enhancing business enabling environment? a. Yes, please specify b. No 23. Do you have experience of cooperating with government in public-private partnership modality? a. Yes b. No 24. Are you willing to cooperate with government and work on public-private partnership with them? a. Yes b. No 25. What factors would influence your decision to be engaged in public-private partnership? 26. Which mechanisms/models could be employed in this type of partnerships? USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 106 ANNEX II.A.2. ELECTRONIC SURVEY OF MSMEs Thank you for participating in the electronic survey. The survey should take approximately 20 minutes to complete. Please answer as completely as you can. If you have any questions, please contact Dr. Maia Giorbelidze, giorbelidzemaia@gmail.com. 1. Sex a. Female b. Male c. Prefer not to state 2. Youth: Yes or No (18 - 29 years) 3. Region: 4. Type of your business a. Community-based private business b. National private business c. Financial institution d. International private sector company e. Other, please specify 5. How many people are employed at your company? a. 1-10 b. 11-50 c. 51-99 d. 100-150 e. More than 150 6. In which value chain does your company operate in? a. Tourism (Mountain/Adventure; Gastronomic; Culture/Heritage) b. Creative Industries (Production, Post-Production, Architecture, High-Value Artisans) c. Shared Intellectual Services (BPO, Nearshoring, Digital Transformation, e-commerce and ICT) d. Light Manufacturing (Furniture, Packaging, Personal Protective Equipment, and Construction Materials) e. Solid Waste Management f. Agriculture (if checked, the respondents answer questions 1-9) g. Other, please specify USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 107 7. What are some of the challenges in your industry/business area (select all relevant responses) a. Lack of technologies b. Access to finance c. Lack of qualified staff d. Lack of distribution channels e. Research and development, f. Marketing g. Demand estimation h. Access to equipment i. Supply chain infrastructure j. Exchange rate dependency k. Administrative/policy barriers l. Taxes m. Other, please specify 8. How useful was the cooperation with the USAID Econ Security Program to address the challenges identified above? A. Very useful B. Useful C. Neutral D. Not useful E. Not at all useful 9. Which national and/or foreign stakeholders would you cooperate with in order to increase your competitiveness on local and international markets? 10. How would you assess the impact of COVID-19 on the operations of your company? a. Very positive b. Positive c. Neutral d. Negative e. Very negative f. I don’t know 11. How did the employment figures change in your organization due to COVID? a. Increased, indicate # USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 108 b. Decreased, indicate # c. Has not changed d. Other, please specify 12. How did your income of your organization change due to COVID-19? a. Increased, indicate # and % b. Decreased, indicate # and % 13. Which factors had an influence on your organization due to COVID? (please select all as relevant) a. The limitation of international transportation b. The limitation of inter-country transportation c. Exchange rate depreciation d. Increase of prices on raw materials/inputs e. Access to finance f. Decrease in demand g. Other, please specify 14. Do you have experience of cooperating with government in public-private partnership modality? a. Yes b. No 15. Are you willing to cooperate with government and work on public-private partnership with them? a. Yes b. No 16. What factors would influence your decision to be engaged in public-private partnership? 17. Which mechanisms/models could be employed in this type of partnerships? 18. Have you received any support from Economic Security Program regarding enhancing business enabling environment? a. Yes, please specify b. No 19. Which policy/legal barriers exist that hinder operations of your company? 20. Have you received support from the Economic Security Program? a. Yes, to Question 21 b. No, to question 32 21. What type of support have you received? a. Grant, jumps to questionnaire for grantees USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 109 b. Biz, Link question 22 c. Training of staff à question 29 22. Which part of Biz Link did you participate in? a. Business Clinic b. Guided growth 23. Please outline the examples of results achieved with engagement of Economic Security Program 24. How would you evaluate the relevance of the support provided by the program? a. Very relevant b. Relevant c. Neutral d. Irrelevant e. Very irrelevant f. I don’t know 25. How would you evaluate the sufficiency of the support provided by the program? a. Very sufficient b. Sufficient c. Neutral d. Insufficient e. Very insufficient f. I don’t know 26. How would you rate your overall satisfaction with the support you received from the program? a. Very satisfied b. Satisfied c. Neutral d. Unsatisfied e. Very unsatisfied f. I don’t know 27. How would you rate your effectiveness of your partnership with the program? a. Very effective b. Effective c. Neutral USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 110 d. Ineffective e. Very ineffective f. I don’t know 28. How would you assess the sustainability of your business operations after finishing the engagement with Economic Security Program? a. Very sustainable b. Sustainable c. Unsustainable d. I don’t know 29. Did you hire the staff trained by the Economic Security Program? a. Yes, please name the qualifications of staff b. No 30. How would you assess the satisfaction with the employees trained by the program? a. Very satisfied b. Satisfied c. Neutral d. Unsatisfied e. Very unsatisfied f. I don’t know 31. Describe the advantages or disadvantages to having the staff trained by the Economic Security Program in the qualifications relevant to your value chain? a. Advantages b. Disadvantages c. I don’t know 32. What type of hard skills do you require from your employees? a. Microsoft Office suite b. Specific software c. Other, please specify 33. What type of soft skills do you require from your employees? a. Communication b. Problem solving c. Negotiation USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 111 d. Team spirit 34. Do you cooperate with any business association? (please select all relevant responses) a. Yes, I am a member of a business association b. Yes, I am a recipient of services provided by business association à question 35 c. No 35. Have you received the service from the following business associations? a. List of Economic Security Program BSO beneficiaries will be added here 36. How would you assess your satisfaction with the quality of the services provided by these BSOs? a. Very satisfied b. Satisfied c. Neutral d. Dissatisfied e. Very dissatisfied f. I don’t know 37. What additional support is needed to expand/improve the services and/or close the existing gaps? USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 112 ANNEX II.A.3. ELECTRONIC SURVEY OF GRANTEES Thank you for participating in the electronic survey. The survey should take approximately 20 minutes to complete. Please answer as completely as you can. If you have any questions, please contact Dr. Maia Giorbelidze, giorbelidzemaia@gmail.com. 1. Sex a. Female b. Male c. Prefer not to state 2. Youth: Yes or No (18 - 29 years) 3. Region: 4. Type of your business a. Community-based private business b. National private business c. Financial institution d. International private sector company e. Other, please specify 5. How many people are employed at your company? a. 1-10 b. 11-50 c. 51-99 d. 100-150 e. More than 150 6. Please indicate the location of your business company: 7. In which value chain does your company operate in? a. Tourism (Mountain/Adventure; Gastronomic; Culture/Heritage) b. Creative Industries (Production, Post-Production, Architecture, High-Value Artisans) c. Shared Intellectual Services (BPO, Nearshoring, Digital Transformation, e-commerce and ICT) d. Light Manufacturing (Furniture, Packaging, Personal Protective Equipment, and Construction Materials) e. Solid Waste Management f. Other, please specify USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 113 8. What are some of the challenges in your industry/business area (select all relevant responses) a. Lack of technologies b. Access to finance c. Lack of qualified staff d. Lack of distribution channels e. Research and development f. Marketing g. Demand estimation h. Access to equipment i. Supply chain infrastructure j. Exchange rate dependency k. Administrative/policy barriers l. Taxes m. Other, please specify 9. How useful was the cooperation with the USAID Econ Security Program to address the challenges identified above? A. Very useful B. Useful C. Neutral D. Not useful E. Not at all useful 10. What type of support did you receive from Economic Security Program? a. Financial through grant b. Mentoring/coaching c. Training in business development d. Other, please specify 11. To what extent to do you agree/disagree to the statements below Totally Disagree Disagree Neutral Agree Totally Agree Don’t know USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 114 Preparing a grant proposal was easy The selection criteria were clear The selection process of transparent The engagement of Economic Security Program team during the grant implementation process was balanced The bureaucratic procedures of grant disbursement delayed the launch/and or implementation process 12. To what extent to you agree/disagree to the statements below regarding the benefits of cooperating with the Economic Security Program Totally Disagree Disagree Neutral Agree Totally Agree Don’t know The grant support enabled me to start the operations The grant support enabled me to expand the operations USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 115 The grant support enabled me to diversify my business operations The grant support was not sufficient to start a business The grant support was not sufficient to expand a business I started selling products/services to new markets I established new partnerships with other private sector companies 13. To what extent/component has your access to finance increased from participating in Economic Security Program (Other than direct benefit from grant)? Totally Disagree Disagree Neutral Agree Totally Agree Don’t know I am more skilled in preparing applications for different financial products and programs. I have better awareness of financial products / programs. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 116 New financial products are offered to me as a consequence of the Economic Security Program efforts. I have greater access (eligibility) to financial products / programs. Other please specify 14. How would you assess the increase in your business operations after receiving the support from Economic Security Program? a. My business income increased by 5-10% compared with last year b. My business income increased by 15-35% compared with last year c. My business income increased by 35-50% compared with last year d. My business income increased by more than 50% compared with last year e. My business income reduced due to business related expenses f. My business income remained the same 15. How would you evaluate the relevance of the support provided by the program? a. Very relevant b. Relevant c. Neutral d. Irrelevant e. Very irrelevant f. I don’t know 16. How would you evaluate the sufficiency of the support provided by the program? a. Very sufficient b. Sufficient c. Neutral d. Insufficient e. Very insufficient USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 117 f. I don’t know 17. How would you rate your overall satisfaction with the support you received from the program? a. Very satisfied b. Satisfied c. Neutral d. Unsatisfied e. Very unsatisfied f. I don’t know 18. How would you rate your effectiveness of your partnership with the program? a. Very effective b. Effective c. Neutral d. Ineffective e. Very ineffective f. I don’t know 19. How would you assess the sustainability of your business operations after finishing the engagement with Economic Security Program? a. Very sustainable b. Sustainable c. Unsustainable d. I don’t know 20. How would you assess the impact of COVID-19 on the operations of your company? a. Very positive b. Positive c. Neutral d. Negative e. Very negative f. I don’t know 21. How did the employment figures change in your organization due to COVID? a. Increased, indicate # b. Decreased, indicate # c. Has not changed USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 118 d. Other, please specify 22. Which factors had an influence on your organization due to COVID? (please select all as relevant) a. The limitation of international transportation b. The limitation of inter-country transportation c. Exchange rate depreciation d. Increase of prices on raw materials/inputs e. Access to finance f. Decrease in demand g. Other, please specify 23. Have you received any support from Economic Security Program regarding enhancing business enabling environment? a. Yes, please specify b. No 24. Which policy/legal barriers exist that hinder operations of your company? 25. How would you assess the level of impact of this grant support to your business operations? a. High b. Moderate c. Low d. No impact 26. What type of support would have been more impactful? 27. How do you see the performance of your business during the next several years? a. Significantly improving b. Improving c. Same d. Worsening e. Significantly worsening USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 119 ANNEX II.B. KEY INFORMANT INTERVIEW PROTOCOLS - ECONOMIC SECURITY PROGRAM INTERVIEW TRACKING DATA This section will complete this section prior to conducting the KII. Date of Interview Location of Interview Name of Data Collector Name of Respondent Role or Position/Title of Respondent Male/Female Respondent Affiliation USAID, Implementing Partners, Grantees, Business Associations, Government of Georgia, Private Sector companies SCRIPT FOR START OF THE INTERVIEW Hello, My name is and I am working with Integra to conduct an evaluation of USAID’s Economic Security Program. The purpose of this evaluation is to help you and USAID/Georgia gain a better understanding of how the Activity has worked, what results have been achieved to date, and how it might be improved going forward. __________. __________ ☐Consent to the Interview and Recording the Interview Your participation is voluntary. No one will know your responses to the questions. Let me know if you want to pause or stop the interview at any time. Would you be willing to allow the interview to be recorded? Y/N Sex a. Female b. Male c. Prefer not to state Youth: Yes or No (18 - 29 years) USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 120 Region: You have the right to participate in the interview without being recorded, Do you have any questions? Thank you. ANNEX II.B.1. KEY INFORMANT INTERVIEWS FOR USAID AOR/COR OF ECONOMIC SECURITY PROGRAM TABLE 9: USAID AOR/COR OF ECONOMIC SECURITY PROGRAM DISCUSSION GUIDE 1. What was your role in the project? 2. What period of time were you engaged with the project? 3. What are some of the binding constraints hindering the private sector’s access to markets, resources and their ability to engage with the GOG through public-private dialogue? Do the constraints differ based on value chains? Probe: How is the Economic Security Program addressing these constraints? 4. How did COVID-19 influence the market dynamics in this regard? How have challenges and opportunities related to economic growth shifted amid COVID? 5. From your point of view, what prevents the vulnerable groups (women, rural poor, youth) from having access to high-value employments and/or income generating opportunities? 6. How would you describe the startup of these activities? What were the expectations from this program from initial phase? Probe: What were early challenges and how were they addressed? Implementing Partners capacity, partnership agreements, adjusting to the cultural, social or political context, activity budget, etc. 7. How would you describe the cost-efficiency of this activity? Probe: Were some components more efficient than others? 8. To what extent have the activities achieved the contract’s specified results? What were the most significant factors that led to results? How do these differ from expectations? Probe: Project design, management approach, relationship with stakeholders, human resources availability, and sub-national versus national stakeholder engagement. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 121 9. What have been the most significant achievements related to this project? Probe: Ask for achievement per component: strengthen cooperation in targeted sectors, support enterprises to improve productivity, sales, quality, and to develop new products and services, industry-led workforce development, building public-private partnerships 10. To what extent do you believe that the Economic Security Program was able to target and establish high￾impact partnerships through the PDF? Were there any cases, when the partnership was being negotiated, but the deal did not go through? What were the underlying causes? Probe: Use of PSE and MSD approaches. 11. How would you evaluate the effectiveness of Economic Security Program with regards to catalyzing priority value chain developments? Probe: Through business association development, grants programs, biz-link, industry led workforce 12. How would you evaluate the effectiveness of policy coordination efforts to address the policy barriers facing its priority sectors and value chains? Could you outline examples of challenges hindering the effectiveness of this direction? 13. To what extent do you believe the program was able to adapt its operations during pandemic? Could you name specific examples of adaptation/reprogramming? 14. Did the MEL plan contribute to adaptive management in terms of adjusting Economic Security Program’s technical approaches and interventions? 15. What are some of the challenges affecting the partnerships with the implementing partners? Any suggestions to make these partnerships more effective or efficient? 16. What type of programs are currently implemented by USAID, which target development of public-private partnerships? 17. Are there other groups or people we should talk to about this project? ANNEX II.B.2. KEY INFORMANT INTERVIEWS FOR DAI, SOLIMAR INTERNATIONAL AND PMCG TABLE 10: DAI, SOLIMAR INTERNATIONAL AND PMCG DISCUSSION GUIDE 1. What is your role in the activity? USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 122 2. What are some of the binding constraints hindering the private sector’s access to markets, resources and their ability to engage with the GOG through public-private dialogue? Do the constraints differ based on value chains? Probe: How is the Economic Security Program addressing these constraints? Probe: Are these challenges common for all vulnerable groups? Gender/youth/rural poor. If not, what are the specificities per group? Do the challenges differ based on value chain? 3. What are the general barriers and business enabling environment in this regard? 4. How did COVID-19 influence the market dynamics in this regard? How have challenges and opportunities related to economic growth shifted amid COVID? Probe: Access to finance, decreased sales, exchange rate fluctuations, decreased demand, etc. 5. From your point of view, what prevents the vulnerable groups (women, rural poor, youth) from having access to high-value employments and/or income generating opportunities? 6. What were some of the factors that led to success or not meeting expected goals? Probe: PSE and MSD approaches 7. As you adapted your operations during COVID, what was the response from the stakeholders’ side? Did the adaptation/reprogramming influence on participation level? Which component was mostly impacted by pandemic in a both negative and positive context? 8. To what extent have the activities achieved the contract’s specified results? How does the reality differ from expected results? Probe: Refer to each component separately 9. What have been the most significant achievements of the activities related to the project? Probe: Ask for achievement per component: strengthen cooperation in targeted sectors, support enterprises to improve productivity, sales, quality, and to develop new products and services, industry-led workforce development, building public-private partnerships 10. From your point of view, which interventions/approaches worked well, and which did not? Why? Why not? 11. What are the criteria applied to select partnerships? How are these partnerships categorized: by sector, size of investment? What other factors are considered in PPP decision making? Are there / Were there other private sector mechanisms considered? Were there any cases, when the partnership was being negotiated, but the deal did not go through? 12. How does project assess the sustainability, replicability, market failure (that partnership is solving), high-value employment for Partnerships? Can you name specific examples when the model has been replicated by other companies? USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 123 13. Can you please elaborate a process of grant management (designing the RFP, announcement, selection, contracting, implementation, monitoring, evaluation and reporting)? Probe: Ask for each management process separately 14. What types of technical, managerial and operational support does project provide to new partners receiving grants? Name specific examples 15. From your perspective, which market failures did the grant mechanism address the most/least effectively? Name specific examples 16. To what extent do you believe the program was able to catalyze developments in selected value chains? Can you name specific examples? 17. Which associations or other stakeholders have been most effective in value chain development? 18. Are there interventions that are specific to supporting women, youth or vulnerable populations? Are there any activities that support women’s economic empowerment objectives such as increased labor market participation, females entering male-dominated industries, changes to the business enabling environment? 19. To what extent was the program able to deepen cooperation with the government on national and local levels? What are the examples in this regard? What were the challenges in this regard? 20. How did the program cooperate with stakeholders to enable policy change in respective areas? How did the collaboration look like? Can you name the specific examples? Probe: With government, with business association, with other USAID-programs 21. How likely will the activities (e.g. public-private partnerships, industry-led workforce development, supported grantees in value chains) be sustained after the Economic Security Program ends? 22. What are some of the challenges affecting these partnerships with USAID and other donor-funded projects? Any suggestions to make these partnerships more effective or efficient? 23. Are there other groups or people we should talk to about this project? USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 124 ANNEX II.B.3. KEY INFORMANT INTERVIEWS FOR BUSINESS ASSOCIATIONS/GRANTEES TABLE 11: BUSINESS ASSOCIATIONS/GRANTEES DISCUSSION GUIDE 1. Can you please tell us a brief information about your organization? 2. Which services does your organization provide to MSMEs? Do you have any feedback about the quality of your services? What are the gaps in this regard? Probe: Particularly important for business associations 3. What are some of the challenges in your industry/business area (capacity building, lack of technologies, access to finance, lack of qualified staff, lack of distribution channels, research and development, marketing, demand estimation, equipment, supply chain infrastructure, exchange rate depreciation) Probe: Are these challenges common for all vulnerable groups? Gender/youth/rural youth. If not, what are the specificities per group? 4. What are the general barriers and business enabling environment in this regard? 5. How did COVID influence the market dynamics in this regard? Probe: Access to finance, decreased sales, exchange rate fluctuations, decreased demand, etc. 6. How did you get engaged in Economic Security Program? How did you hear about it? 7. How would you describe the support provided by Economic Security Program? What was the added value of this support? Did the expectation differ from the reality? Probe: Diversified products/services, diversified markets, acquired qualified trained staff, increased access to raw materials, increased sales 8. To what extent was the program able to overcome the above-stated challenges? Probe: PSE and MSD approaches 9. Can you please describe the process of engagement in this program? Was it transparent? What were the selection criteria? What was your incentive to be engaged in this program? 10. To what extent did you benefit from this support? Can you name specific examples? How would you evaluate the partnership with Economic Security Program? 11. What were some of the challenges during the implementation process? 12. How would you assess the relevance and sufficiency of the funding provided by the program? USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 125 Probe: Ask specifically on relevance and sufficiency 13. What type of support would be most impactful? 14. How did COVID affect your operations? Please name specific examples Probe: Logistics, access to finance, decrease in demand, increased prices on raw materials/input, exchange rate depreciation 15. What support did you receive to tackle the challenges related to COVID from USAID economic security program? 16. Which interventions/approaches did work/did not work well? Why? 17. Have you received any support from Economic Security Program regarding enhancing business enabling environment? Have you address the program with such challenges/gaps in legislation? What was the response from the program? 18. Are there other donor or GOG interventions that support value chain development, market systems development and export of agricultural products from Georgia? 19. To what extent are you willing to cooperate with government and work on public-private partnership with them? Which mechanisms/models could be employed in this type of partnerships? What factors would influence your decision to be engaged in public-private partnership? Probe: Did you have any previous PPP experience? 20. How would you assess the sustainability of your business operations after finishing the engagement with Economic Security Program? How does your sustainability plan look like? 21. If you were to restart the engagement with the program, what would you do differently? 22. What should the program do better/differently in order to increase the effectiveness of its support? 23. From your point of view, would you be able to achieve the results indicated above without the involvement from the Economic Security Program? Why? Why not? By what means/with which stakeholders’ support 24. What additional support does your organization need to expand/improve the services and close the existing gaps? USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 126 ANNEX II.B.4. KEY INFORMANT INTERVIEWS FOR PUBLIC PRIVATE PARTNERSHIPS AND GDA TABLE 12: PUBLIC PRIVATE PARTNERSHIPS AND GDA DISCUSSION GUIDE 1. Can you please tell us a brief information about your organization? 2. Could you please describe the PPP/GDA cooperation model that was established with support of Economic Security Program? What were your incentives to participate in this partnership? 3. Could you describe the process of establishment of this partnership (acquiring the information, applying, negotiation, deal making, implementation)? What were the selection criteria? What factors influenced your decision to participate in this partnership? 4. How would you describe the support provided by Economic Security Program? What was the added value of this support for your organization/for the industry or overall economy? Did the expectation differ from the reality? Probe: Created high-value employment Probe: Have you received any other type of support from other programs and /or Government? If yes. please specify? 5. To what extent did you benefit from this support? Can you name specific examples? How would you evaluate the partnership with Economic Security Program? 6. What were some of the challenges during the implementation process? Probe: Interaction with government? 7. How would you assess the relevance and sufficiency of the support provided by the program? Probe: ask specifically on relevance and sufficiency 8. To what extent do you think the partnership is wide reaching, impactful and replicable? What is the rationale behind? 9. How did COVID affect your operations/the partnership? Please name specific examples Probe: Logistics, access to finance, decrease in demand, increased prices on raw materials/input, exchange rate depreciation 10. What support did you receive to tackle the challenges related to COVID from USAID economic security program? 11. Which interventions/approaches did work/did not work well? Why? USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 127 Probe: Are these challenges common for all vulnerable groups? Gender/youth. If not, what are the specificities per group? Do the challenges differ based on the value? 12. What type of changes have been occurred/expecting to be occurred in your industry as a result of this PPP/GDA? Name specific examples 13.. Have you had any legal/policy barriers during this process? Was it addressed by the program? How would you evaluate the efficiency of this response? 14. Which other mechanisms/models could be employed in this type of partnerships? What are their advantages and disadvantages? Probe: Did you have any previous PPP experience? 15. How would you assess the sustainability of this partnership after finishing the engagement with Economic Security Program? How does your sustainability plan look like? 16. If you were to restart the engagement with the program, what would you do differently? 17. What should the program do better/differently to increase the effectiveness of its support? 18. From your point of view, how did the Economic Security Program achieve results, if at all? ANNEX II.B.5. KEY INFORMANT INTERVIEWS FOR FINANCIAL INSTITUTIONS TABLE 13: FINANCIAL INSTITUTIONS DISCUSSION GUIDE 1. What is your role in this financial institution? 2. What are some of the challenges hindering the private sector’s access to markets, resources, their ability to engage with the GOG through public-private dialogue and related development of targeted value chains? What are the general barriers related to the business enabling environment in this regard? 3. What type of new financial products is your institution developing / offering? 4. What do you know about the USAID Economic Security program? 5. Have you had any cooperation with USAID Economic Security program? If yes, what type of cooperation have you had? Probe: Financial product development, capacity development, policy discussion, etc. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 128 6. Do you observe any changes in the skillset of the clients (because of the technical assistance) / business patterns (vertical integration projects, more cooperative / partnership applications) or any other structural changes on the demand side? 7. What type of cooperation are you engaged with international donor projects (USAID, EU, ADB, EBRD, other IFIs, etc.) or Government of Georgia? 8. Could you describe the latest market developments in terms of access to finance? 9. What are the changes on the market that COVID-19 has caused? What has your organization done to adapt to new reality? From your point of view, which sectors / groups were affected the most? Have you received any support from government or international stakeholders to mitigate the socio-economic impact of COVID? 10. To what extent are you willing to cooperate with government and work on public-private partnership with them? Which mechanisms/models could be employed generally in PPPs? What factors would influence your decision to be engaged in public-private partnership? Probe: Did you have any previous PPP experience? 11. What type of support (e.g. grant mechanism) would be most impactful for MSMES? In which value chains? 12. Are there other groups or people we should talk to about this project? USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 129 ANNEX II.C. SITE VISIT SUMMARY REPORT SITE VISIT SUMMARY REPORT This template follows guidance from USAID/PPL Program Cycle How-to-Note: Planning and Conducting Site Visits Date of site visit Location of the event Name of Observer(s) Name of site, learning event, workshop, exposition, etc. Sponsor(s) of the event Format of the event: workshop, remote online, one-stop services, public sector institution, public or private sector firm Approximate number of participants Characteristics of participants: SMEs, government, women, youth, mixed, etc. Type of USAID affiliation: implementing partner, grantee, mixed, etc. Brief explanation of findings: Positive: Negative: Observations about USAID partner collaboration: Did the event/material support meet the expectations of the USAID implementing partner: Feedback observed or heard directly from participants/beneficiaries: Evidence of USAID marketing and branding of promotional and learning materials: Follow-up recommendation USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 130 ANNEX III: DOCUMENTS REVIEWED DAI. USAID Economic Security Program Activity Overview BIZ-LINK 1: Business Clinic. Tbilisi, Georgia: USAID, n.d. DAI. USAID Economic Security Program Annex 1: Tourism Value Chain Assessment. Tbilisi, Georgia: USAID, 2019. DAI. USAID Economic Security Program Annex 2: Business Process Outsourcing Value Chain Assessment. Tbilisi, Georgia: USAID, 2019. DAI. USAID Economic Security Program Annex 3: Film and Post-Production Value Chain Assessment. Tbilisi, Georgia: USAID, 2019. DAI. USAID Economic Security Program Annex 4: Light Manufacturing Sector: Furniture Value Chain Assessment. Tbilisi, Georgia: USAID, 2019. DAI. USAID Economic Security Program Annex 5: Light Manufacturing Sector: Packaging Value Chain Assessment. Tbilisi, Georgia: USAID, 2019. DAI. USAID Economic Security Program Annex 6: Mapping of Private Sector Investment Funds. Tbilisi, Georgia: USAID, 2019. DAI. USAID Economic Security Program Annex 7: CAM Scoring Definitions and Ratings. Tbilisi, Georgia: USUAID, n.d. DAI. USAID Economic Security Program Annual Report October 1, 2019 – September 30, 2020. Tbilisi, Georgia: USAID, 2019. DAI. USAID Economic Security Program Business Association Resource Manual: A Guidebook to Effectiveness and Sustainability. Tbilisi, Georgia: USAID, n.d. DAI. USAID Economic Security Program Grants Under Contract Manual. Tbilisi, Georgia: USAID, 2019. DAI. USAID Economic Security Program Monitoring, Evaluation, and Learning Plan Revised Version. Tbilisi, Georgia: USAID, 2021. DAI. USAID Economic Security Program Partnership Development Fund Manual. Tbilisi, Georgia: USAID, n.d. DAI. USAID Economic Security Program Quarterly Report FY20 Q1: October – December 2019. Tbilisi, Georgia: USAID, 2020. DAI. USAID Economic Security Program Quarterly Report FY20 Q3: April – June 2020. Tbilisi, Georgia: USAID, 2020. DAI. USAID Economic Security Program Quarterly Report Quarter 1 FY21: October 2020 – December 2020. Tbilisi, Georgia: USAID, 2021. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 131 DAI. USAID Economic Security Program Quarterly Report Quarter 2 FY21: January 2021 – March 2021. Tbilisi, Georgia: USAID, 2021. DAI. USAID Economic Security Program Sector and Value Chain Analytics The First Analytical Report. Tbilisi, Georgia: USAID, 2021. DAI. USAID Economic Security Program Sector and Value Chain Analytics The Second Analytical Report. Tbilisi, Georgia: USAID, 2021. DAI. USAID Economic Security Program Value Chain Prioritization and Gaps Assessment. Tbilisi, Georgia: USAID, 2019. DAI. USAID Economic Security Program Year One Workplan. Tbilisi, Georgia: USAID, 2019. DAI. USAID Economic Security Program Year Two Workplan. Tbilisi, Georgia: USAID, 2020. ISET Policy Institute. Report on Methodology Development of Indirect Impact Assessment Methodology and Multipliers. Tbilisi, Georgia: International School of Economics at TSU, 2020. USAID, Digital Strategy. USAID.gov Last updated August 16, 2021. https://www.usaid.gov/usaid-digital￾strategy. USAID. Private-Sector Engagement Policy. Washington, D.C.: USAID, 2019.https://www.usaid.gov/sites/default/files/documents/1865/ usaid_psepolicy_final.pdf. The USAID Economic Governance Program Post-COVID Advisory Report. Tbilisi, Georgia: USAID, 2020. The USAID Youth Policy. October 2012 The USAID Gender Equality and Women’s Empowerment Policy. 2020 USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 132 ANNEX IV: KEY INFORMANTS, FOCUS GROUP, AND SURVEY RESPONDENTS TABLE 14: LIST OF KEY INFORMANTS, FOCUS GROUPS, AND SURVEY RESPONDENTS # DATE INTERVIEWEE FULL NAME GENDER POSITION NAME, ORGANIZATION SECTOR # OF RESPON DENTS 1 02.09.2021 David Gvenetadze Male M&E Manager, DAI Implementing Partner 1 Nato Ardishvili Female Grants Director, DAI Implementing Partner 2 2 02.09.2021 Natia Vepkhvadze Female Component 3 Lead, DAI Implementing Partner 3 Natia Kapanadze Female Gender and Youth Specialist, DAI Implementing Partner 4 3 02.09.2021 Ketevan Chogovadze Female Program Development Specialist, USAID Georgia USAID 5 4 20.09.2020 Mark McCord Male Chief of Party, DAI Implementing Partner 6 Marika Shioshvili Female Deputy Chief of Party, DAI Implementing Partner 7 5 21.09.2021 Georgia Darchia Male Component 1 Lead, DAI Implementing Partner 8 6 21.09.2021 Georgia Akhalaia Male Component 2 Lead, DAI Implementing Partner 9 7 27.09.2021 Tatia Samkharadze Female Head of VET Department, Ministry of Education Government 10 8 30.09.2021 Nino Veltauri Female Director, Employment Agency Government 11 9 04.10.2021 Maya Eristavi Female Component Lead, USAID Economic Governance Program USAID Partner 12 Natalia Beruashvili Female Chief of Party, USAID Economic Governance Program USAID Partner 13 USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 133 Tamar Buadze Female Component Lead, USAID Economic Governance Program USAID Partner 14 Giorgi Giorgobiani Male Component Lead, USAID Economic Governance Program USAID Partner 15 10 05.10.2021 Saba Sarishvili Male Deputy Chief of Party, IESC USAID Partner 16 11 05.10.2021 William Baringer Male Strategy and Programming Development Associate, USAID Georgia USAID 17 12 06.10.2021 Irine Salukvadze Female Organizational Capacity Development Manager, CNFA Other USAID Supported Programs 18 13 06.10.2021 Konstantine Kobakhidze Male Agriculture Project Management Specialist, USAID Georgia USAID 19 14 07.10.2021 Ana Chikovani Female Executive Manager, United Airports of Georgia Government 20 15 07.10.2021 Beverly Hoover Female Private Sector Engagement Coordinator, USAID Georgia USAID 21 16 08.10.2021 Lela Akiashvili Female Team Lead of Gender and Social Inclusion, UNDP Other Donor 22 17 08.10.2021 David Dzebisashvili Male Program Manager/Gender Focal Point, USAID Georgia USAID 23 18 11.10.2021 Tornike Zirakishvili Male Deputy Head, Enterprise Georgia Government 24 19 11.10.2021 Tamar Koriauli Female First Deputy Head, National Tourism Administration of Georgia Government 25 20 12.10.2021 Philipp Steinheim Male Project Team Leader, GIZ Other Donor 26 21 13.10.2021 Siobhan Pangerl Female Foreign Service Officer, USAID Georgia USAID 27 22 13.10.2021 David Songhulashvili Male Chairman of the Sector Economy and Economic Policy Committee, Parliament of Georgia Government 28 USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 134 Tamila Shabashvili Female Head of the Committee Staff, Parliament of Georgia Government 29 23 15.10.2021 David Tsiklauri Male Senior Private Sector Development Advisor, USAID Georgia USAID 30 24 15.10.2021 Marika Olson Female Economic Growth Office Director, USAID Georgia USAID 31 25 18.10.2021 Levan Tsulaia Male Executive Director, Destination Management Organization (Samegrelo region) Government 32 Tinatin Khanjaleishvili Female Director, Destination Management Organization (Kakheti region) Government 33 Davit Mumladze Male Executive Director, Destination Management Organization (Samtskhe￾Javakheti region) Government 34 26 18.10.2021 Salome Mekvabishvili Female Head of the Strategic Development Department, Ministry of Economy and Sustainable Development of Georgia (MoESD) Government 35 27 18.10.2021 Zurab Alavidze Male Co-Founder & Managing Partner, Fabrica 1900 Beneficiary 36 Natia Bolkvadze Female Financial Director, Fabrica 1900 Beneficiary 37 28 18.10.2021 Giorgi Ketiladze Male Georgia Capital, Managing Director Financial Institution 38 29 19.10.2021 Irakli Shengelia Male Founder/CEO, VTOL Raven Public Private Partnership 39 30 19.10.2021 Guram Tateshvili Male Founder, Dataninja Public Private Partnership 40 Malkhaz Dartsmelidze Male Full Stack Software Engineer, Dataninja Public Private Partnership 41 31 19.10.2021 Giorgi Noniashvili Male Co-founder, Wenu Public Private Partnership 42 Tornike Okrostsvaridze Male Co-founder, Wenu Public Private Partnership 43 USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 135 32 19.10.2021 Lela Potskhverashvili Female Head of International Relations and Analytics Department, Mountain Trails Agency Government 44 33 20.10.2021 Zaali Patchkoria Male Co-founder, Irnero Public Private Partnership 45 Mari Gelashvili Female Marketing and Public Relations Manager, Irnero Public Private Partnership 46 34 20.10.2021 Sophio Chelidze Female Head of Sales Division, National Agency of State Property of Georgia Government 47 35 20.10.2021 Maka Dvalishvili Female Executive Director, Georgian Arts and Culture Center Beneficiary 48 Tea Gotsiridze Female Gallery Manager, Georgian Arts and Culture Center Beneficiary 49 Besarion Kacharava Male Founder/Director, LTD PostRed Beneficiary 50 Tinatin Babakishvili Female Project Manager, LTD PostRed Beneficiary 51 Nino Kvirtia Female Founder/Director, Ltd ArtStyle Beneficiary 52 36 20.10.2021 Veronika Gogokhia Female Deputy CEO, Georgian Packaging LLC Beneficiary 53 37 20.10.2021 Richard Bangs Male Chief Adventure Officer, Steller Public Private Partnership 54 38 21.10.2021 Annie Vashakmadze Female Head of International Relations, Gita Government 55 39 21.10.2021 Nana Dikhaminjia Female Vice Rector for Innovations and Science Popularization, llia State University LEPL Beneficiary 56 David Gegechkori Male Head of Foreign Affairs and Strategic Development Office, Akaki Tsereteli State University LEPL Beneficiary 57 40 21.10.2021 Vano Vashakmadze Male Tourism Lead, DAI Implementing Partner 58 USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 136 41 21.10.2021 Alfredo Coppola Male Co-CEO, USMAC Public Private Partnership 59 42 22.10.2021 Tina Kavadze Female External Communications Lead, TBC Bank Financial Institution 60 43 22.10.2021 Severian Ghvinepadze Male Principal Manager, Advice for Small Businesses Georgia & Azerbaijan, EBRD Other Donor 61 Irakli Toloraia Male Associate, EBRD Other Donor 62 44 22.10.2021 Natia Goliadze Female Partner, SavvY JSC Beneficiary 63 45 25.10.2021 Ana Lomtadze Female Marketing Specialist, Sweeft Digital LLC Public Private Partnership 64 46 25.10.2021 Olga Nakashidze Female SME Value Added Services Manager, Bank of Georgia Financial Institution 65 47 25.10.2021 Mariam Sumbadze Female Managing Director, Georgian ICT Cluster Beneficiary 66 48 25.10.2021 Kesi Katsitadze Female Development Department | Manager, Business and Technical University Beneficiary 67 49 25.10.2021 Davit Isakadze Male Producer, N&N Studio Public Private Partnership 68 50 25.10.2021 Maia Kheladze Female Co-Founder, Georgian E￾commerce Association Beneficiary 69 51 25.10.2021 Oral DEMİRCİOĞLU Male Partner, Karina Tasarım Danışmanlık ve Eğitim Hizmetleri Ltd. Şti. Public Private Partnership 70 52 25.10.2021 Amiran Ivanidze Male President, Georgian Business Tourism Association Public Private Partnership 71 53 25.10.2021 Ketevan Vachiberidze Female CEO, Griffin LLC Beneficiary 72 54 26.10.2021 Davit Mizandari Male CEO/Founder, Qvevry Research Company Beneficiary 73 55 26.10.2021 Tamar Sikharulidze Female Chair of the Board, Gastronomic Tourism Business Association Beneficiary 74 56 26.10.2021 Mariam Kandelaki Female Program Manager, Georgian Animation Association “Saqanima“ Beneficiary 75 USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 137 Lika Mezvrishvili Female Program Manager, Georgian Animation Association “Saqanima“ Beneficiary 76 57 26.10.2021 Nata Kvatchantiradze Female Chairperson, Georgian Tourism Association (GTA) Beneficiary 77 58 27.10.2021 Keta Buachidze Female Deputy Chairwoman, Design Georgia Beneficiary 78 59 27.10.2021 Giorgi Guliashvili Male President, Waste Management Association of Georgia Beneficiary 79 60 27.10.2021 Natia Nikoleishvili Female Co-Founder, Georgian Film Cluster Beneficiary 80 Davit Vashadze Male Executive Director, Georgian Film Cluster Beneficiary 81 61 28.10.2021 Nino Tskhadaia Female Director, Strategic Planning, Adjara Group Public Private Partnership 82 62 28.10.2021 John P. DiPirro Male Resident Program Director, International Republican Institute Other USAID Supported Programs 83 63 01.11.2021 Tamar Kaikatsishvili Female First Deputy Chairman of Department of Tourism and Resorts of Adjara A. R Government 84 64 02.11.2021 Revaz Topuria Male Managing Partner, Packaging Manufacturers’ Association of Georgia (PMAG) Beneficiary 85 65 03.11.2021 Giorgi Chugoshviii Male Tourism Lead, DAI Beneficiary 86 66 10.11.2021 Dominik Papenheim Male Team Leader Economic Development and Market Opportunities, Budget Support Coordination, EU to Georgia Other Donor 87 Georges Dehoux Male Programme Officer Agriculture, Fisheries and Food Safety, EU to Georgia Other Donor 88 USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 138 ANNEX V: SURVEY RESULTS EQ1. Private sector engagement: To what extent has the PDF targeted and established high￾impact (defined as wide-reaching and/or replicable) partnerships with the private sector that have strengthened and catalyzed the development of priority value chains? To what extent are these partnerships sustainable (defined as the establishment of market linkages that will not depend on USAID assistance after the activity ends)? Access to finance, access to finance and lack of technologies have been indicated as the most pressing challenges by the online survey participants. The only exception was the MSME respondents, who mentioned “Marketing” as a critical challenge hindering their business operations. TABLE 15: EQ1 SURVEY RESULTS BUSINESS ASSOCIATIONS CHALLENGE GRANTEES MSMES AVERAGE Lack of qualified staff 57% 76.92% 66.67% 67% Access to finance 71% 50.00% 50.98% 57% Lack of technologies 64% 26.92% 27.45% 39% Research and development, 50% 23.08% 19.61% 31% Administrative/policy barriers 50% 23.08% 17.65% 30% Marketing 43% 15.38% 31.37% 30% Taxes 36% 23.08% 21.57% 27% Exchange rate dependency 29% 23.08% 17.65% 23% Supply chain infrastructure 43% 7.69% 11.76% 21% Lack of distribution channels 14% 11.54% 27.45% 18% Access to equipment 14% 15.38% 11.76% 14% Demand estimation 0% 3.85% 13.73% 6% USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 139 FIGURE 13: SURVEY RESPONSES BY KEY RESPONDENT GROUPS If we analyze the data in light of challenges per sector, lack of qualified staff was indicated as common challenge for all sector representatives. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 140 Lack of technologies 36% 29% 56% 21% 40% Access to finance 64% 58% 50% 38% 100% TABLE 16: CHALLENGES PER SECTOR Light Manufacturing (Furniture, Packaging, Personal Protective Equipment, and Construction Materials) Shared Intellectual Services (BPO, Nearshoring, Digital Transformatio n, e-commerce and ICT) Creative Industries (Production, Post-Production, Architecture, High-Value Artisans) Tourism (Mountain/Adven ture; Gastronomic; Culture/Heritage ) Challenges per sector Solid Waste Management Lack of qualified staff 64% 67% 81% 67% 40% Lack of distribution channels 18% 21% 13% 29% 0% Research and development, 45% 21% 6% 33% 40% Marketing 55% 33% 6% 38% 20% Demand estimation 9% 4% 0% 21% 0% Access to equipment 9% 21% 19% 4% 20% Supply chain infrastructure 55% 21% 0% 8% 20% Exchange rate dependency 18% 25% 50% 8% 20% Administrative/po licy barriers 0% 17% 31% 21% 40% USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 141 Taxes 18% 25% 25% 17% 0% FIGURE 14: KEY CHALLENGES PER SECTOR, MOST COMMONLY ANSWERED TO LEAST COMMONLY ANSWERED USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 142 USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 143 USAID/GEORGIA’S ECO FIGURE 15: COMPARATIVE BAR GRAPH ON CHALLENGER PER SECTOR BY CHALLENGE USAID.GOV NOMIC SECURITY PROGRAM MID-TERM EVALUATION | 144 FIGURE 16: COMPARATIVE BAR GRAPH ON CHALLENGER PER SECTOR BY SECTOR Noteworthy, that 83% of all survey participants assessed the usefulness of cooperation with USAID Economic Security Program as either “very useful” or “useful”. TABLE 17: HOW USEFUL WAS THE COOPERATION WITH THE USAID ECONOMIC SECURITY PROGRAM TO ADDRESS THE CHALLENGES MSMEs Business Associations Grantees Very Useful 21% 35% 58% Useful 56% 47% 31% Neutral 15% 13% 8% Not Useful 0% 5% 4% Not at all Useful 2% 0% 0% Don‘t know 6% 0% 0% USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 145 FIGURE 17: USEFULNESS RATING OF USAID COOPERATION IN ADDRESSING CHALLENGES EQ2. Value chain approach: To what extent has support to sector associations and government institutions catalyzed priority value chain development? How important a role have these stakeholders played in increasing value chain competitiveness? To what extent have these stakeholders received sufficient and relevant support? FIGURE 18: TO WHAT EXTENT HAVE STAKEHOLDERS RECEIVED SUFFICIENT AND RELEVANT SUPPORT USAID Economic Security Program encompassed workforce development component which enabled the private sector companies to develop the capacity of existing staff or train new potential employees. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 146 The online survey respondent workforce development program participant private sector companies assessed the support provided by the program either “relevant” or “very relevant” and expressed their satisfaction with the collaboration results. FIGURE 19: SURVEY RESPONSES ON RELEVANCY OF WORKFORCE DEVELOPMENT PROGRAM SUPPORT As for the workforce development participant interns (#Go4it), 68% of them indicated that they were offered a full-time job after the training and/or internship. As for the accessibility to the information regarding this opportunity, 39% of them heard it from the University, 29% - from Facebook, 23% - from Friends and only 5% of them saw the announcement on www.jobs.ge. FIGURE 20: WORKFORCE DEVELOPEMENT PARTICIPANT INTERNS' RESPONSES TO INFORMATION ACCESSIBILITY USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 147 Generally, these interns are very satisfied with participation in this program and 74% of them indicated their satisfaction level as “very satisfied” or “satisfied”. FIGURE 21: WORKFORCE DEVELOPMENT PROGRAM INTERNS’ SATISFACTION RATES Furthermore, 89% of online survey respondent interns indicated, that the participation in the program was useful for acquiring new skills and 73% of them noted, that these skills will help them in finding full￾time employment. FIGURE 22: ONLINE SURVEY RESPONSES OF WORKFORCE DEVELOPMENT PROGRAM INTERNS’ SATISFACTION RATES USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 148 EQ3. Grant component: To what extent has the grant component strengthened each priority value chain? To what extent did the grants address gaps or market failures in each value chain? - 71% of grant recipients indicated, that the bureaucratic procedures delayed the launch/and or implementation process - 86% of grant recipient indicated, that the grant proposal preparation of easy - 95% of grant recipients outlined, that the financial support enabled them to expand their operations, while 60% of them noted, that it enabled them to diversify their operations - 63% of grant recipients noted, that the allocated amount was sufficient to expand their operations - Almost 53% of grant recipients started selling their products to new markets, as a result of financial support from the program - 70% of grant recipients established new partnerships with other private sector companies - 95% of grant recipient evaluated the support provided by the program as “relevant” or “very relevant” TABLE 18: BREAKDOWN OF GRANTS BY VALUE CHAIN Value Chain Share of Grant Initiatives Creative industry 15% Cross cutting 9% E-commerce 6% Light Manufacturing 43% Shared intellectual service 15% Tourism 12% TABLE 19: BREAKDOWN OF REQUESTED FUNDS AND COST SHARE BY VALUE CHAIN Row Labels Count of Applicant Organization / Grantee Average of Funds Requested from the USAID/Program Average of Grantee cost share contribution Creative industry 15.15% $29,687.82 $24,491.21 Cross cutting 9.09% $49,988.58 $25,036.18 E-commerce 6.06% $16,849.79 $33,699.59 Light Manufacturing 42.42% $40,899.50 $60,929.85 USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 149 Shared intellectual service 15.15% $27,723.80 $17,467.35 Tourism 12.12% $22,174.00 $10,003.35 Grand Total 100.00% $34,303.41 $37,737.33 FIGURE 23: ASSESSMENT OF GRANT COMPONENTS FIGURE 24: ASSESSMENT OF EQ3 RESULTS USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 150 EQ4. Coordination on policy: To what extent has the Economic Security Program coordinated effectively with other USAID activities (managed by both the USAID Economic Growth and Democracy, Rights and Governance offices) to address the policy barriers facing its priority sectors and value chains? To what extent has the absence of a large policy component within USAID’s Economic Security Program helped or hindered its ability to address policy gaps. 61% of MSME representatives indicated, that they have received the support from the USAID’s Economic Security Program for enhancing business enabling environment. EQ5. COVID-19: In the context of COVID-19 economic contractions, how can the activity adapt its approaches (e.g., selection of grant solicitation themes, division of USAID investments across sub-sectors, sequencing of interventions, etc.) to improve its ability to achieve its targets: creation of 4,800 jobs and achieving $60 million in new sales? ● Tourism was the most affected sector by pandemic, while ICT and e-commerce (fintech, healthtech, edutech, toursimtech) bloomed during this period. E.g. one of the grantees indicated, that their operations boosted by 150% after pandemic (e-commerce provider enterprise) ● 19% of grant applicants indicated, that COVID had positive influence on their operations (shared intellectual services, tourism, creative industries) USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 151 ANNEX VI: GLOSSARY OF TERMS BUSINESS ENABLING ENVIRONMENT The context in which commercial firms operate. It includes laws, regulations, policies, international trade agreements, and public infrastructure that affect the movement of a product or service along its value￾chain. The business-enabling environment at the national and local level encompasses policies, administrative procedures, enacted regulations, and the state of public infrastructure. In addition to these more-formal factors, social norms, workforce-skill levels, business culture, and local expectations can be powerful aspects of the business-enabling environment. BUSINESS SUPPORT ORGANIZATIONS Nonprofit, public and for-profit resource organizations that serve local businesses and support their growth and success ENTERPRISE-DRIVEN DEVELOPMENT aligning with the private sector as co-creators of market-oriented solutions, with shared risk and shared reward. MARKET SYSTEMS DEVELOPMENT Focuses on building the capacity and resilience of local systems, leveraging the incentives and resources of the private sector, ensuring the beneficial inclusion of the very poor, and stimulating change and innovation that continues to grow beyond the life of the project. PRIVATE SECTOR ENGAGEMENT A strategic approach to planning and programming through which USAID consults, strategizes, collaborates, and implements with the private sector for greater scale, sustainability, and/or effectiveness of outcomes. PRIVATE SECTOR DEVELOPMENT A discipline and area of programmatic work focused on strengthening the business-enabling environment for the private sector to drive inclusive economic growth in the countries in which USAID operates. PSD often focuses on supporting regulatory reforms that improve business and investment climates, providing public goods that help strengthen the broader private sector, and/or facilitating investment from companies. PUBLIC- PRIVATE PARTNERSHIPS Formal arrangements between public and private counterparties to share risks and rewards in the delivery of services and infrastructure. Characterized by joint planning, joint contributions, and shared risk, PPPs in development are an opportunity to leverage resources, mobilize industry expertise and networks, and bring fresh ideas to projects WOMEN’S ECONOMIC EMPOWERMENT When we increase incomes and opportunities for women, entire communities, economies, and countries benefit. USAID.GOV USAID/GEORGIA’S ECONOMIC SECURITY PROGRAM MID-TERM EVALUATION | 152