SUPPORTING YOUTH AND WOMEN ENTREPRENEURSHIP (YES-GEORGIA) MID-TERM PERFORMANCE EVALUATION March 2021 This publication is made possible by the support of the American People through the United States Agency for International Development (USAID) and was prepared by Integra Government Services International LLC for the Learning, Evaluation, and Analysis Project (LEAP III). SUPPORTING YOUTH AND WOMEN ENTREPRENEURSHIP (YES-GEORGIA) MID-TERM PERFORMANCE EVALUATION Contract Title: LEAP III: Learning, Evaluation, and Analysis Project Contract Number: GS-10F-083CA / 7200AA18M0004 Activity Number: LEAP III 2020 – 1009.1053 Submitted: March 23, 2021 Contractor: Integra Government Services International LLC 1156 15th St NW, Suite 800 Washington, D.C. 20005 USAID Office: USAID/Georgia COR: Katie Qutub, kqutub@usaid.gov Photo Credit: W-GDP Annual Report 2020 DISCLAIMER This report is made possible by the support of the American people through USAID. The authors’ views expressed in this publication do not necessarily reflect the views of USAID or the United States Government. II LIST OF TABLES TABLE 1. KEY FINDINGS, CONCLUSIONS AND RECOMMENDATIONS 4 TABLE 2. DESCRIPTIVE STATISTICS OF QUALITATIVE DATA 15 TABLE 3. COMPARISON OF LENDING FACILITIES IN TERMS OF AFFORDABILITY AND ACCESSIBILITY OF LOANS 26 LIST OF FIGURES FIGURE 1. DOCUMENT REVIEW OF YES-GEORGIA MATERIALS 12 FIGURE 2. STAKEHOLDER AFFILIATIONS 13 FIGURE 3. RESPONDENTS OF ONLINE SURVEYS DISAGGREGATED BY SEX 13 FIGURE 4. INTERNSHIP RESPONDENTS OF ONLINE SURVEYS AND STAKEHOLDER AFFILIATIONS 14 FIGURE 5. SEX DISAGGREGATION OF ALL RESPONDENTS 16 FIGURE 6. MOST FREQUENTLY MENTIONED BARRIERS TO STARTING NEW BUSINESSES AND JOB CREATION 19 FIGURE 7. A HIGHER RATIO OF MALES THAN FEMALES RECEIVED BUSINESS STARTUP LOANS AND LARGER LOANS WHEN THE RESPONDENTS ARE DISAGGREGATED BY SEX 21 FIGURE 8. A HIGHER RATIO OF FEMALES THAN FEMALES SUBMITTED BUSINESS PROPOSAL BUT A HIGHER RATIO OF MALES RECEIVED BUSINESS STARTUP LOANS, AS DISAGGREGATED BY SEX 21 FIGURE 9. OPERATIONAL STATUS OF NEW BUSINESSES DISAGGREGATED BY SEX 22 FIGURE 10. PERCENTAGE OF CHANGE IN INCOME AS A RESULT OF TRAINING ACCORDING TO RESPONDENTS 22 FIGURE 11. PERCEIVED CHANGES AS A RESULT OF BUSINESS STARTUP 23 FIGURE 12. BENEFICIARIES’ ASSESSMENT OF THE TRAINING CURRICULUM 24 FIGURE 13. BENEFICIARIES PERCEIVED PROGRAM BENEFITS 25 III FIGURE 14. AFFORDABILITY OF LOANS (BOTH YES-GEORGIA AND OTHER FINANCIAL FACILITIES) 30 FIGURE 15. ACCESSIBILITY OF LOANS (BOTH YES-GEORGIA AND OTHER EXTERNAL FINANCIAL FACILITIES) 31 FIGURE 16. SURVEY OF INTERNSHIP SPONSORS BY CATEGORY OF RESPONDENTS 32 FIGURE 17. SURVEY OF THE SOFT SKILLS MOST VALUED BY INTERNSHIP SPONSORS 33 FIGURE 18. SURVEY OF TECHNICAL SKILLS MOST VALUED BY INTERNSHIP SPONSORS 33 FIGURE 19. INTERNSHIP SPONSORS’ RATING OF THE EFFECTIVENESS OF INTERNSHIP PROGRAMS 34 IV ACRONYMS ADS - Automated Directives System DDI - Bureau for Democracy, Development, and Innovation DDL - Data Development Library DEC - Development Experience Clearinghouse E3 - Bureau for Economic Growth, Education and Environment EBRD - European Bank for Reconstruction and Development EDR - Evaluation Design Report EG - Economic Growth Office EQ - Evaluation Question ET - Evaluation Team FGD - Focus Group Discussions GenDev - USAID/E3/Office of Gender Equality and Women’s Empowerment GOG - Government of Georgia ICT - Information and Communication Technologies IP - Implementing Partner JAG - Junior Achievement Georgia KII - Key Informant Interview LEPL - Legal Entity of Public Law MFO - Micro Finance Organization NDA - Non-Disclosure Agreement NPI - New Partner Initiative PH - Project Harmony International POC - Point of Contacts V PPP - Public-Private Partnership PSE - Private Sector Engagement SME - Small Medium Enterprise SOO - Statement of Objectives SOW - Statement of Work SPO - Strategic and Program Office TOT - Trainers of Trainers USAID - United States Agency for International Development USD - United States Dollars WEE - Women’s Economic Empowerment W-GDP - Women’s Global Development Prosperity Initiative YES-Georgia - Supporting Youth and Women Entrepreneurship Program 1 ACKNOWLEDGMENTS The evaluation team (ET) thanks all interviewees who took the time to share their perspectives and experiences with the team while adjusting to work and life during a pandemic. The team could not have implemented the evaluation without the ongoing and patient support program implementers the Crystal Fund, particularly Ms. Nino Lagvilava and Mr. Archil Bakuradze. The team appreciates the technical review and support for this report from Integra’s LEAP III project team including Chief of Party, David Quinn, and Director of Operations, Kimberly Hamilton. The team also thanks Anna Chaus and Ketevan Chogovadze of the USAID/Georgia Mission, as well as LEAP III Contracting Officer Representative (COR) Katie Qutub (USAID/DDI) for her continued support of this study. 2 EXECUTIVE SUMMARY This report presents the results of a mid-term performance evaluation of the United States Agency for International Development’s (USAID) Supporting Youth and Women Entrepreneurship Program (YES￾Georgia). The USAID/Georgia Economic Growth (EG) Office and Strategic and Program Office (SPO) requested the USAID-funded Learning, Evaluation, and Analysis Project (LEAP III) conduct a mid-term performance evaluation of YES-Georgia, implemented by Crystal Fund. The Crystal Fund is a local entity based in Tbilisi, Georgia and meets all criteria under USAID’s New Partner Initiative (NPI). YES-GEORGIA BACKGROUND In September 2015, USAID/Georgia signed Cooperative Agreement No. AID-114-A-15-00009 with Crystal Fund to implement the “Youth Entrepreneurial Skills for Advancing Employability and Income Generation Program in Georgia” (YES-Georgia) program. The activity was a joint initiative of Crystal Fund, Project Harmony (PH) International, and JSC MFO Crystal with US$350,000 in USAID funding and US$93,000 in cost share from JSC MFO Crystal, with a 30-month implementation period, until January 2018. In July 2018, USAID approved a 30-month, US$350,000 time and cost-extension of the activity to increase opportunities for self-employed or employees ages 14 to 28. In July 2019, the activity was competitively selected to receive additional funding of US$840,000 through the White House-led Women’s Global Development and Prosperity Initiative (W-GDP). Thus, the activity received another cost extension to further Georgia's economic development targeting women with an expanded age range of 28 to 49 years old. The program was renamed to “Supporting Youth and Women Entrepreneurship Program” (YES-Georgia) and has a new period of performance through January 2024. EVALUATION PURPOSE AND DESIGN The purpose of this evaluation is to summarize what has worked well, what could have been improved over YES-Georgia’s first four years, and provide recommendations for the new iteration of the YES￾Georgia activity—which will focus exclusively on female beneficiaries. The planned field work and data collection was impacted by COVID-19 restrictions, which limited primary data collection. The ET adjusted its methodology (Annex B) to conduct remotely all key informant interviews (KIIs) and focus group discussions (FGDs) by utilizing video conferencing platforms. The evaluation took a mixed-methods approach and used four main data collection methods: ● A desk review of YES-Georgia performance reports, statements of objectives, procurement documents, field reports, promotional materials, communication and learning documents, and other available YES-Georgia documents and data; ● Online surveys of 188 different respondents who are direct and indirect beneficiaries and local partners to understand how they engaged with and used YES-Georgia training, learning materials, and micro equity loans; ● An online survey of a partner network of private sector firms to understand how the private sector views internships; and 3 ● The team conducted 49 KII’s with targeted stakeholders including the implementing partner’s (IP) staff and trainers, Government of Georgia public sector officials, staff from financial institutions that have not worked with Crystal Fund, and USAID staff who were familiar with YES-Georgia. KEY FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS The YES-Georgia youth enterprise and Buzz Georgia training components were designed to develop the necessary skills for innovative, small-scale enterprises in seven rural Georgia municipalities in which the entrepreneurial schools are located. The business model specifically targets rural youth and female entrepreneurs, which is a unique target group. Beginning in January 2021, the direct beneficiaries of the training activities will be only females due to the change in funding streams. The activity has benefitted about 1,500 youth and women through supporting new businesses and job creation. There was scant evidence of job creation by youth entrepreneurs other than self-employment. According to KIIs conducted with USAID and IP staff, there are very few small or medium enterprises (SMEs) that can drive job creation. The key informants and respondents noted many challenges related to youth entrepreneurship and job creation because of low skills, poor infrastructure in rural communities, lack of affordable and accessible financing, and minimal networking opportunities. The YES-Georgia’s overall goal is to train budding entrepreneurs and to provide the necessary skills to obtain finance for startup businesses. To this end, the training had an overall positive impact on the participants that went beyond their abilities to start new businesses. The respondents highly valued YES-Georgia’s training and its perceived impact on attitudes, status, and confidence in addition to changes in income. In total, about 400 graduates of the training components submitted requests for financing of which 67 businesses were funded across Georgia as of September 30, 2020; 72 percent are still operating, and 23 percent have failed. YES-Georgia adapted and improved its training curriculum to better reach rural youth and females. In its first two years, the instructors of the Youth Entrepreneurial Training Component were volunteers seconded from Crystal Fund staff who were later replaced by experienced instructors and mentors. Buzz Georgia adapted its curriculum in another important way. The most active women were selected to serve as Ambassadors who helped recruit and mentor new participants. The curriculum also changed to include both soft skills training and business development and planning. The majority of the participants rated the training curriculum as very strong. Georgia has a well-developed banking sector, but these traditional banks are risk-averse. It is estimated that all but six percent of available financial capital is held by banks, which means there are very few alternatives or financial institutes that offer venture capital and other financial products. The Micro Equity Fund and Crystal MFO standard loan facility are both accessible and affordable to the targeted beneficiaries based on data from the respondents. The Micro Equity Fund is directly linked to the educational component of the YES-Georgia trainings. The Crystal MFO loan facility only began making loans recently and less data is available regarding its loan utilization rates and beneficiaries’ perceptions of affordability and accessibility. There is no financial product on the market offering zero percent interest loans. Mentors and some other KII respondents contended that providing zero percent interest loans had advantages compared to grants because they do not distort the market. 4 The YES-Georgia activity also supported on-the-job training and internship programs with 90 private sector companies; it is estimated that about 200 youth participants obtained full-time employment following the completion of the internships. To gauge the perception of companies on internship practices, the ET initiated the online survey and disseminated the questionnaire through the HRHub network. The report’s findings, conclusions, and recommendations are summarized according to the four Evaluation Questions (EQs), which can be found on pages 16 and 17 of this Report. TABLE 1. KEY FINDINGS, CONCLUSIONS AND RECOMMENDATIONS FINDINGS/CONCLUSIONS RECOMMENDATIONS How did Entrepreneurship Training Help Start New Business and Increase Incomes? About 400 loan applications were submitted and 67 businesses were funded as of 9/30/2020. Seventy￾two percent are still operational, while 23 percent failed but there was no information regarding the reasons for successes and failures. Conduct outreach and solicit feedback to understand the circumstances and variables related to business successes and failures. Most frequently cited barriers to new business startup and job creation are lack of skills, underdeveloped rural infrastructure and networks, and lack of access to affordable loans. Adjust the training curriculum to increase the time devoted to business planning, investment strategies, and identifying of financing while compressing the soft skills training. Integrate feedback from the 2020 online training participants and consider offering blended training options. Although more women participated in the trainings, a higher ratio of males submitted loan applications and received funding than female participants who submitted business proposals. Make the training more gender-sensitive and inclusive; ensure that trainers and mentors are familiar with women-focused businesses and those that cater to females and more diverse customers. Develop transparent selection criteria for accessing finance. About 43 percent of participants reported increases in family income as a result of the training and obtaining loans. More than half of the participants, 56 percent, who did not receive loans reported no change in family income. Consider adopting a better monitoring and tracking system and conduct follow-up surveys with past participants to gain insights into the reasons some participants’ incomes increased by sourcing local organizations and experts familiar with rural development. All participants reported perceived feelings of appreciation by friends and family and reported increased levels of self-confidence. As noted above, conduct follow-up surveys to measure the correlation between perceived improvements in appreciation and confidence leading to better business decisions. 5 What Changes are needed to Sustain the Business Model of Youth Entrepreneur Training? The training curriculum was adapted in two ways: the IPs’ in-house staff was replaced by more experienced trainers and the learning platform switched from in-person courses to an online platform due to Covid-19. Female participation increased as a result of these two changes. Adjust the training curriculum to integrate the important lessons learned during Covid-19 regarding the online platform, which increased female participation. All beneficiaries (95 percent) reported exceptionally high satisfaction with the training; 91 percent of participants said the curriculum was balanced and 97 percent commented positively on the qualifications of the trainers. Continue to provide feedback to the instructors, mentors, and Crystal Consulting to improve the curriculum as needed, perhaps focusing on more sector-specific offerings. The participants noted a range of perceived benefits from the trainings, ranging from incentives to start businesses, acquisition of new skills, networking, and mentoring support. Participants who received loans reported higher degrees of satisfaction. Brainstorm with the Crystal Consultants and other stakeholders to identify ways to improve the loan applicants' abilities to access affordable loans. Develop clear guidelines of the selection criteria and more practical exercises related to loan applications. Yes-Georgia training is unique in Georgia with no other competitors targeting the same rural youth and women. Specific cost-recovery changes to the business model are not supported by evidence at this time. It is likely that a fee-based training will not attract enough of the targeted participants to be viable. YES-Georgia will need to explore other options such as introducing corporate sponsorships or corporate social responsibility incentives. Extent of Affordability and Accessibility of YES-Georgia Loans. About 60 percent of the participants found the YES￾Georgia loans to be accessible. Integrate a gender lens to ensure that instructors, mentors, and loan reviewers are more aware of gender biases when evaluating credit worthiness and business viability. Almost all of the participants who obtained loans through YES-Georgia found the zero-interest loans and repayment terms to be very affordable. Evidence suggests that zero-interest loans are preferable to grants; YES-Georgia should conduct a landscape analysis to identify potential partners that offer grants to expand the pools of applicants for both loans and grants. What Design Considerations could have improved the Internship Program? There was scant data available from former YES￾Georgia interns, so it is not possible to assess their feedback. Hire short-term consultants to design and integrate a more robust monitoring and evaluation plan moving forward. In general, the non-financial sector hires the most Increase focus on the inputs from the private 6 interns (42 percent of internships), followed by the financial sector at 19 percent, and the international organizations and the Government of Georgia at about 15 percent each. sector, financial sector, international organizations, and the Government of Georgia to identify desirable functional and sectoral skill sets. Develop an online platform for candidates to submit their resumes and make it accessible to potential internship sponsors. Both soft skills like problem solving, communications, and work attitude, and technical skills related to software applications and accounting are equally sought skill sets. Place greater emphasis on understanding the corporate culture, career mobility, human resource policies, and how to navigate the work environment. 7 1. INTRODUCTION This document describes the design, research methodology, and detailed findings, conclusions, and recommendations for a mid-term performance evaluation of the United States Agency for International Development’s (USAID) Supporting Youth and Women Entrepreneurship Program (YES-Georgia). The USAID Mission in Georgia (USAID/Georgia) commissioned, designed, and implemented this evaluation utilizing the LEAP III mechanism through the Center for Economics and Market Development in the Bureau for Development, Democracy, and Innovation. Annex A provides USAID’s approved statement of work (SOW) for this evaluation. YES-GEORGIA BACKGROUND In September 2015, USAID/Georgia signed Cooperative Agreement No. AID-114-A-15-00009 with Crystal Fund to implement the “Youth Entrepreneurial Skills for Advancing Employability and Income Generation Program in Georgia” (YES-Georgia) program. The activity was a joint initiative of Crystal Fund, PH International, and JSC MFO1 Crystal with US$350,000 in USAID funding and US$93,000 in cost share from MFO Crystal, with a 30-month implementation period, until January 2018. In July 2018, USAID approved a 30-month, US$350,000 time and cost-extension of the activity to increase opportunities for self-employed or employees age 14 to 28. In July 2019, the project was competitively selected to receive additional funding of US$840,000 through the White House-led Women’s Global Development and Prosperity Initiative (W-GDP). Thus, the project received another cost extension to further Georgia's economic development targeting women with an expanded age range of 28 to 49 years old. The program was renamed to “Supporting Youth and Women Entrepreneurship Program” (YES-Georgia) and has a new period of performance through January 2024. Reforms in the educational system, labor market, and wider economy have improved the conditions for young people in recent years, but Georgia’s youth still struggle to participate in the economy. The youth unemployment rate is three times higher than that for those age 30 and older, and much of what counts as youth employment is rural, agricultural, and subsistence in nature. YES-Georgia was designed to be a national, demand-driven activity to create economic opportunities and empower Georgia’s young people (age 17 to 25) by enabling youth-led enterprises, encouraging innovation, promoting entrepreneurship, and enhancing youth employability through skills training, confidence building, and mentoring. This was achieved through the following interventions: 1) Youth entrepreneurship training program that included “Training of Trainers” and training of beneficiaries; 2) Youth-led start-up financing program that used zero-interest equity convertible loans financing tool; 3) Youth internship program within private companies; and 4) JobFest program, a joint initiative with Georgia’s line ministries, which brought hundreds of youth representatives and dozens of public and private companies together to offer young people internship and job opportunities. 1 PH was responsible for building curriculum, monitoring, and adoption of the key components of YES-Georgia: “Youth Enterprise” and “Employment Guide to Internships” modules. Crystal Fund was responsible for overall program management, implementation, monitoring, evaluation and financial reporting. JSC MFO Crystal was responsible for adopting Youth Enterprise module, establishing Youth Micro Equity Facility, co-financing of projects, and providing follow-up mentoring. 8 The Crystal Fund is a local entity based in Tbilisi, Georgia and meets all criteria under USAID’s New Partner Initiative (NPI). YES-Georgia was structured so Crystal Fund, as the prime implementer, would provide training in entrepreneurial and business skills and operate a Youth Micro Equity Fund providing capital to young people to establish and operate small businesses. YES-Georgia is an innovative institutional model for the delivery of entrepreneurial skills and startup financing to youth and women by institutionalizing a network of Young Entrepreneurs’ Schools in seven cities of Georgia (Tbilisi, Gori, Kutaisi, Zugdidi, Batumi, Akhaltsikhe, and Telavi). The activity established a viable youth mentorship program that provides continued support to YES-funded start-ups. The activity also established an entrepreneurship curriculum that is taught in 30 schools and approximately 3,000 entrepreneurs received training, and about two-thirds of them are female. The activity mobilized US$230,000 from private and public sector funds to support youth-owned enterprises and start-up businesses. 9 2. METHODOLOGY EVALUATION PURPOSE, AUDIENCES, AND INTENDED USES This evaluation was conducted at an opportune time. USAID and the international community are looking for ways to leverage private sector capital and opportunities to promote sustainable development outcomes targeting youth and women. USAID and its implementing partners from international development, the Government of Georgia, and non-governmental organizations have been exploring different approaches to supporting workforce development and entrepreneurial activities. USAID also seeks to engage new and underutilized partners to work with USAID under the NPI. PURPOSE The purpose of this mid-term performance evaluation is to determine the effectiveness of specific programmatic approaches, which are referenced in the EQs below, in achieving intended life-of-program results in some discontinued interventions and to provide recommendations on corrective actions and new directions for the remaining years of program implementation. This report summarizes what has worked well during YES-Georgia’s first four years and provides recommendations for the new iteration of the YES-Georgia activity focused exclusively on female beneficiaries. AUDIENCES AND INTENDED USES The primary audience of the evaluation will be USAID/Georgia’s Economic Growth and Strategy and Program Office and the Crystal Fund. USAID/Washington and the private sector firms and public institutions involved with the YES-Georgia network are the secondary audiences, as well as the broader donor community—particularly those working on youth enterprise training, women’s economic empowerment, and access to financing. The results of the study will be shared with local stakeholders such as the Ministry of Education, Ministry of Economy and Sustainable Development, local not-for￾profit organizations, and other partners working in this sector. USAID/Georgia will use the evaluation findings, conclusions, and recommendations as learning opportunities to understand what the Crystal Fund activity has achieved, how it is perceived and valued, what opportunities for collaboration were available and utilized, and whether USAID should continue with the model. Crystal Fund and other YES￾Georgia stakeholders will have an opportunity to learn about and utilize evaluation results regarding the mechanism’s strengths and areas for improvement. EVALUATION DESIGN The ET conducted the mid-term performance evaluation in two phases from September 2020 to March 2021. The desk review, secondary data review, and primary data collection were completed in December 2020, before the one-month lockdown in Georgia due to the Covid-19 pandemic. The data analysis, presentation of preliminary findings and recommendations to USAID/Georgia, and report writing took place from January through March 2021. The ET was comprised of an international Team Leader based in Washington, D.C., and three Georgia-based team members: senior evaluator, subject matter expert, and logistician. 10 For this evaluation, primary and secondary data collection was conducted using a mixed-methods approach. Whenever possible, existing quantitative data such as YES-Georgia performance indicators were used. Additional survey-based instruments were developed to collect quantitative data to fill existing knowledge gaps. Qualitative data was collected primarily through online surveys and remotely conducted KIIs with direct and indirect beneficiaries, the IP, Government of Georgia officials, the private sector, academia, and USAID staff. Other USAID staff participated in the ET’s in-briefs and out-briefs. EVALUATION QUESTIONS (EQS) The evaluation was designed to respond to the following EQs, which are identical to those in USAID’s approved evaluation SOW (Annex A). These four EQs included probes that were finalized in discussions with USAID staff during the in-brief on November 16, 2020. EQ 1: To what extent has the entrepreneurship training component helped to start new businesses and increase the incomes of the trainees who did not receive financing through YES-Georgia’s micro equity fund (and who account for approximately 90 percent of beneficiaries)? ● Compare income rates of trainees who received financing and those who did not; ● Changes in income as a result of training and financial support (adjusted for inflation); ● Influence of geographic location and gender affecting submission of applications to Micro Equity Fund; and ● Trainees’ perceived impact of training and financial support. EQ 2: What changes, if any, are needed to the business model of the seven youth entrepreneur schools to ensure their financial viability and continuity after the life of the program? ● Perceived benefits of activity by beneficiaries; ● Ranked value of curriculum content by beneficiaries; ● Comparable services offered by other organizations, government, or financial facilities; ● Forecasted expenses for different business models; and ● Available resources for subsidizing the schools. EQ 3: To what extent are the two YES-Georgia financing facilities - the Crystal MFO’s loan facility (US$2.5 million) and micro equity facility (US$75,500)— accessible to and affordable for their respective target beneficiaries? What changes, if any, are needed to the design and size of the aforementioned two financing facilities (e.g., interest rates, repayment schedule, average loan amount, etc.) to increase their accessibility and affordability? Note: accessibility refers to the beneficiary's ability to obtain a loan in support of a promising business idea despite the lack of collateral, documented revenues, and credit history. 11 Affordability refers to the beneficiary’s ability to repay a loan using revenues generated from a startup/existing business. ● Characteristics of Crystal MFO Loan Facility and Micro Equity Facility; ● Comparable financial facilities targeting the same beneficiaries; ● Comparable financial products offered in Georgia; ● Interest rates offered by Crystal MFO and Micro Equity Fund; and ● Percentage of associated costs (capital, inflation, default risks, loan administration) included in subsidized loans. EQ 4: To what extent could the internship program (which was discontinued in 2018) have been designed differently to better meet the needs of the private sector? How could the private sector be encouraged to participate in similar internship programs in the future and contribute resources toward youth development/employment goals? ● Involvement of private sector and not-for-profit organizations in the design of internship program; ● Curriculum matching the needs of demand for internships; ● Percentage of interns receiving offers of employment from the firms after completing an internship; and ● Types of incentives important to sponsors of the internships. DATA COLLECTION This section briefly describes the multiple methods the team employed to triangulate information and to fill gaps based on the approved evaluation design report (EDR) (Annex H). To collect evidence to develop the findings and conclusions to answer these EQs, the ET employed a mainly qualitative approach. Annex C provides a “Getting to Answers” matrix with more details on the team’s approach for each EQ. The primary data collection method yielded 49 in-depth semi-structured KII’s and 188 responses from three online surveys with YES-Georgia stakeholders, including USAID staff from the mission. DESK REVIEW The ET created a document library and collected relevant documents such as internal USAID concept papers, activity reports, annual reports, the monitoring, evaluation and learning plan, the USAID/Georgia Country Development Cooperation Strategies (CDCS), and other donor partners’ publications. The quantitative research was carried out through a structured desk review of pertinent documents including readily available literature, studies, and assessments conducted by donors, civil society, and the host government. Figure 1 shows the distribution of materials reviewed by the ET during the desk 12 review segment of this evaluation. The ET reviewed 29 documents provided by USAID/Georgia as well communications provided by the IP, including internal Facebook groups, photographs, and training curriculum. The ET reviewed these materials to identify constraints in entrepreneurial activities, youth employment, and women’s economic empowerment. The majority of publications contained in the bibliography and relevant references were published within the last five years. Along with the desk review, the assessment team generated a set of descriptive statistics for the YES-Georgia activity. Following the completion of the desk review, the ET developed an EDR that was approved by USAID/Georgia. Figure 1. Document Review of YES-Georgia Materials PRIMARY DATA COLLECTION The primary data collection was conducted in the Georgian language for all respondents unless a respondent requested the interview to be conducted in English. The electronic survey and KIIs with USAID staff were conducted in English. The ET developed and conducted three online surveys as a pre￾screening and data collection instrument before conducting the KIIs. These surveys were structured and utilized a combination of dichotomous questions i.e. Yes/No/Don’t Know, Likert scale (using a 5- point rating scale), and open-ended responses. The online surveys utilized the online platform Survey Monkey. Two of the online surveys assessed and compared the experiences of beneficiaries of the three training courses and mentoring support: Youth Enterprise Component, Buzz Georgia, and Crystal Consulting. These surveys had two tracks of questions specific to beneficiaries who received micro financing and beneficiaries of the training courses who did not receive micro financing. The third online survey gathered data about prevailing internship policies, and staffing needs of the private sector, but not necessarily through YES-Georgia or the HRHub network. The ET conducted semi-structured video interviews with USAID staff, IPs, and selected beneficiaries to gather their perspectives and additional information about the YES-Georgia process and activities. The list of key informants can be found in Annex D. SAMPLE SIZE 13 Sampling was not intended to be random, but rather, participants were chosen specifically for their relationship to the activity, as in the case of USAID staff, partner staff, and Government of Georgia officials. The YES-Georgia activity did not keep contact information for training participants nor the internship participants, which meant the ET had to adjust its methodology to rely more extensively on social media platforms to reach potential respondents. Thus, the majority of respondents were more recent graduates of the trainings conducted from 2018–2020 rather than early participants from 2015– 2017 who are not as active on social media and group Facebook pages. Figures 2–4 below highlight the diversity of participants, capturing their group affiliations, self-identified gender and sectoral focuses. Figure 2. Stakeholder Affiliations Figure 3. Respondents of Online Surveys Disaggregated by Sex 14 Figure 4. Internship Respondents of Online Surveys and Stakeholder Affiliations ANALYSIS APPROACH The mid-term performance evaluation includes a comprehensive Getting to Answers matrix (Annex C) that maps the EQs to data sources and data analysis methods. The ET used descriptive statistics to produce a quantitative overview of the YES-Georgia activities, including characteristics such as the number of participants, regions, and in-country partners. The team used a manual review process to extract key data such as keywords, quotes, or substantive information about activities from the transcripts and notes. THEMATIC AND CONTENT ANALYSIS The team examined qualitative data from KIIs and the online surveys to identify patterns, themes, and trends relevant to each EQ to better understand context and meaning. When the ET found a divergence in responses, it explored possible reasons for the divergence, using other respondent group interviews and in some cases, conducting follow-up interviews with the IP staff and training instructors/mentors. The ET coded its notes according to key themes of interest across the interviews. Using these coded notes, the ET summarized the distribution, number, and average responses by theme and respondents. DESCRIPTIVE STATISTICS Descriptive statistics provided simple summaries of the analyses the ET conducted as seen in Table 2 on the following page. The team sought to visualize results whenever possible. Quantitative data sources for descriptive analysis included the online surveys conducted to respond to all four EQs. 15 TABLE 2. DESCRIPTIVE STATISTICS OF QUALITATIVE DATA DESCRIPTIVE STATISTICS (QUALITATIVE) Male 15 Female 34 Total Number of Participants 49 Total Number of Interviews 32 Implementing Partner 17 Intern 6 Government 6 Financial Institution 9 Private Sector 5 EU 2 Academia 1 USAID 3 TRIANGULATION The ET made use of the various data sources through a triangulation process to enhance the credibility of the analysis. Triangulation synthesizes multiple perspectives and leads to a fuller understanding of the issues being studied. Data from various lines of inquiry including interviews, written documents, analytical procedures, and other sources (e.g., the online surveys) were considered both separately and together to develop findings and conclusions. GENDER ASPECTS OF EVALUATION QUESTIONS USAID evaluation policy guidance calls on Agency staff and ETs to examine EQs and processes from a gender perspective. Gender aspects must be considered and incorporated during all phases of an evaluation. The nature of the YES-Georgia mechanism means that direct beneficiary interventions were directly correlated with gender outcomes because the Buzz Georgia activity was specific to females. More than three-fourths of the KII respondents and survey respondents, as seen in Figure 2 and Figure 3 above, were female. In line with USAID’s Gender Equality and Female Empowerment Policy and 16 Automated Directives System (ADS) 201, the ET considered gender-specific and differential effects of the YES-Georgia mechanism, where relevant. Of the 49 participants of the qualitative study, 34 were females and 15 were males. Furthermore, of the 136 respondents of YES and Buzz training participants, 108 were female and 26 were male (two respondents preferred not to state their sex). Figure 5. Sex Disaggregation of all Respondents STRENGTHS AND LIMITATIONS OF THE EVALUATION DESIGN COVID-19 RESTRICTIONS LIMITED KEY INFORMANT INTERVIEWS AND FOCUS GROUP DISCUSSIONS The safety and security of the evaluation team and key stakeholders was of the utmost importance. The planned field work and data collection was impacted by COVID-19 restrictions, which limited primary data collection. No KIIs or FGDs were conducted face-to-face with direct beneficiaries, key stakeholders from host governments, private sector or not-for-profit organizations, or civil society leaders in Georgia. The ET developed a remote data collection strategy that was included in the Evaluation Design Report and approved by USAID/Georgia prior to conducting the interviews. STRENGTHS ● Coverage: The 49 KIIs and 188 survey respondents covered a broad range of stakeholders from the 2015–2020 period of performance. The KIIs provided perspectives from eight groups of stakeholders: IP, interns, government, financial institutions, private sector firms, European Union staff, academia, and USAID; ● Proximity and availability of respondents: All of YES-Georgia’s organizational and individual stakeholders are based in Georgia so scheduling and accessibility did not present any challenges and the ET was able to interview remotely almost all key informants selected through the sampling process; 17 ● Mixed methods: The evaluation design combined qualitative interview data, quantitative survey data, quantitative activity data, formal process mapping, and thorough document review to respond to the EQs. In coordination, the range of types of evidence fills gaps in understanding, provided a robust understanding of YES-Georgia’s first five years, and presented detailed evidence on which to make actionable recommendations; and ● Anonymity: As discussed in the background section, the ET granted each respondent anonymity to gather open, honest feedback. This led to balanced conversations, with few respondents sharing only positive or negative perspectives, and provided a nuanced view into a complex mechanism. LIMITATIONS Challenges emerged in implementing the evaluation that limited the team’s ability to adhere to its sampling framework of direct beneficiaries because the Crystal Fund did not have the financial and human resources necessary to develop and maintain a monitoring or tracking system that allowed for contacting 1,500 past participants of its training activities and internships to participate in the survey. Thus, it was not possible to learn how many participants received funding for their businesses, whether businesses remained operational, or if interns gained employment after the completion of their trainings. The team made multiple efforts to mitigate these challenges and limitations, as noted below. ● Response bias: Key informants and online survey respondents were the primary sources of information to answer the EQs. Although the ET attempted to triangulate as much of this testimonial evidence as possible, these data are subject to personal biases, opinions, and recollection. The ET identified, selected, and interviewed a broad range of stakeholders to address this challenge. However, not every statement could be weighed against a similar or counterexample to address potential response bias or gauge the magnitude of such bias; ● Information availability: To address the EQ related to the uniqueness and sustainability of the YES-Georgia financing model, several variables were considered (including interest rates, repayment schedules, collateral, and risk assessment), and the number of comparative variables was estimated because of other lending facilities’ proprietary client information. However, the ET attempted to collect information in a way that allowed for this type of comparative analysis. To address the EQ related to internships, only six interns could be identified and interviewed, which is a small sampling; ● Complex Environment: The activities included in the YES-Georgia activity represent a wide array of socio-economic complexities that were viewed at a macro level. Entrepreneurial and economic empowerment issues are highly complex and often fluctuate depending upon the region, sector, and interlocutor. Youth and gender issues intersect with and sometimes exacerbate other forms of social exclusion based on geography, ethnicity, sexual identity, disability, and others; and ● Instrumental Bias: Guarding against instrumental bias was a consideration because many beneficiaries and in-country partners were exposed to or participated in more than one intervention, and their responses may have been influenced by participation in multiple interventions. 18 EVALUATION TEAM The ET consisted of the following members: Brenda Lee Pearson as Team Leader brings more than 30 years of experience leading and conducting highly technical evaluations and sectoral assessments for USAID, MCC, and World Bank. She is a globally recognized youth and gender subject matter expert and has previously worked in Georgia. Dr. Maia Giorbelidze, Senior Evaluator, holds a Ph.D. in Business Administration and possesses extensive experience in monitoring and evaluation, data analysis, program design, planning, and implementation; she facilitated data analysis. Ms. Rati Gabrichidze, Subject Matter Expert, is a senior finance expert with expertise in micro and SME financing, public finance (municipalities as well as central government), developing business strategies and market surveys, entrepreneurial trainings, and consulting and financing of micro-businesses. Ms. Rusudan Gogibedashvili, Logistician, supported the ET in scheduling informant interviews, disseminating online surveys through social networks. 19 3. FINDINGS AND CONCLUSIONS This section discusses the evaluation’s major findings for each of the four EQs. Supporting evidence may be found in the data visualizations generated through the KIIs and surveys. The last section of the report offers conclusions and recommendations based on these findings presented by evaluation question. EQ 1: To what extent has the entrepreneurship training component helped to start new businesses and increase the incomes of the trainees who did not receive financing through YES-Georgia’s micro equity fund (and who account for approximately 90 percent of beneficiaries)? The YES-Georgia mechanism was intended to support and enhance USAID actions to create economic opportunities and empower Georgia’s young people (ages 17–25) by enabling youth-led enterprises, encouraging innovation, promoting entrepreneurship, and enhancing youth employability through skills training, confidence building, and mentoring. The YES-Georgia youth enterprise and Buzz Georgia training components were designed to develop the necessary skills for innovative, small-scale enterprises in seven rural Georgia communities. The activity benefitted about 1,500 youth and women through supporting new businesses and job creation. There was scant evidence of job creation by youth entrepreneurs other than self-employment. According to KIIs conducted with USAID and IP staff, there are very few small to medium enterprises (SMEs) among the Crystal beneficiaries that can drive job creation. The key informants and respondents noted many challenges related to youth entrepreneurship and job creation related to skills, poor infrastructure in rural communities, lack of affordable and accessible financing, and minimal networking opportunities. These challenges are summarized and visually represented in the word cloud found in Figure 6. Figure 6. Most Frequently Mentioned Barriers to Starting New Businesses and Job Creation 20 The YES-Georgia’s overall goal is to train budding entrepreneurs and to provide the necessary skills to obtain finance for startup businesses. To this end, the training had an overall positive impact on the participants that went beyond their abilities to start new businesses. The respondents highly valued YES-Georgia’s training and its perceived impact on attitudes, status, and confidence in addition to changes in income. In total 67 businesses were funded across Georgia as of September 30, 2020; of these businesses, 72 percent are still operating and 23 percent have failed. The table in Annex D summarizes the key data related to how many businesses were started in each of the targeted municipalities and the operational status of these businesses. An important finding of the data analysis is that a higher ratio of males received startup loans and larger amounts of loans, despite the fact that the majority of training participants are female. In other words, there were a disproportionate number of female respondents included in the KIIs and surveys. This finding was confirmed in KIIs with the IP staff, trainers, and Crystal Consulting. “During the trainings, the participants gradually transform their idea into business proposal, and at the end of the session, we have a pitching session. As observed, those who are more self-confident and can better present the idea, they are selected for funding and in many cases, men dominate in this regard.” Mentor, YES-Georgia The ET conducted two sets of KIIs with the training instructors and mentors to understand this discrepancy. First, female participants performed worse in pitching their business proposals to prospective investors and loan officers. According to the KIIs, female youth were “less confident” and “less forceful” in presenting their business plans. Second, there can be an inherent bias against female￾owned businesses or female-focused services and products, such as daycare, children’s activities, grooming or hair salons, etc. According to the ET’s calculations, about 67 percent of the total contractual loan amount of 403,684 GEL, was allocated to male participants of the training component, as shown by ratio percentages in Figures 7 and 8 below. The Importance of Monitoring and Tracking Training Participants The ET was not able to quantify the degree to which the entrepreneurship training helped to start new businesses because the sampling frame was skewed to more recent participants of the eight rounds of training. Again, the lack of a monitoring or tracking system of participants who graduated from the training limited overall data collection that would indicate what percentage of participants started new businesses or received funding, thus the statistical analysis is limited to respondents. 21 Figure 7. A Higher Ratio of Males than Females Received Business Startup Loans and Larger Loans when the Respondents are Disaggregated by Sex Figure 8. A Higher Ratio of Females than Females Submitted Business Proposal but a Higher Ratio of Males Received Business Startup Loans, as Disaggregated by Sex Figure 9 on the following page illustrates the operational status of new businesses as disaggregated by sex. These statistics are derived from the ratio of male and female participants because overall, more females are participating in the entrepreneurship training, e.g., the Buzz Georgia component was comprised entirely of female beneficiaries. The sex-disaggregated data in the figure shows the number of business startups that have failed, not yet started, operating, or in the early stages of minimum viable product (MVP) development. 22 Figure 9. Operational Status of New Businesses Disaggregated by Sex Only 10 percent of the training participants received financing through the Crystal Fund’s micro equity loan facility through 2020.2 The ET found that about 77 percent of participants who were able to access loans saw an increase in their family income, whereas only 46 percent of participants who did not receive financing saw an increase in family income as a result in the training. See Figure 10 below for an indication of the percentage of increased income, which is followed by Figure 11, highlighting the gender disaggregated changes in perception as a result of individuals’ business startups. Figure 10. Percentage of Change in Income as a Result of Training according to Respondents 2 The MFO Crystal Fund did not make any loans prior to September 1, 2020, which was the period of performance included in this mid-term evaluation. The subsequent first quarter of fiscal year 2021 included 22 loan disbursements, but these are not included in this evaluation. 23 Figure 11. Perceived Changes as a Result of Business Startup EQ 2: What changes, if any, are needed to the business model of the seven youth entrepreneurs schools to ensure their financial viability and continuity after the life of the program? Currently, the IP does not have any specific plans for changing its business model after USAID funding ends. The business model specifically targets rural youth and female entrepreneurs, which is a unique target group. Beginning in January 2021, the direct beneficiaries of the training activities will be only females due to the change in funding streams. Based on feedback from training participants, the beneficiaries highly value the training curriculum. The instructors, however, noted that female participants were less comfortable presenting their business plans in internal pitching sessions, which may account for the lower proportion of females that received funding for their business startups. Several key informants suggested that the curriculum would benefit from a more balanced integration of soft skills topics and business proposal development that included more role-playing related to obtaining financing. A summary of beneficiaries’ assessments of the training program can be found in Figure 12 on the following page. YES-Georgia adapted and improved its training curriculum to better reach rural youth and females. In its first two years, the instructors of the Youth Entrepreneurial Training Component were volunteers seconded from Crystal Fund staff who were later replaced by experienced instructors and mentors. The curriculum also changed to include both soft skills training and business development and planning. YES-Georgia’s annual and quarterly reports describe a range of achievements including how Crystal Fund aligned its approach by integrating feedback from students and instructors; the IP also brought necessary private sector expertise from the Buzz Georgia franchise to target the business needs of rural women. The majority of the participants rated the training curriculum as very strong. Buzz Georgia adapted its curriculum in another important way. The most active women were selected to serve as Ambassadors who helped recruit and mentor new participants. These Ambassadors received additional financial training, but the lack of an overall monitoring system meant that the ET could not assess whether these Ambassadors received more loans than non-Ambassadors. 24 “Even those youth, whose businesses failed, are the examples of success for me. They have learned lessons and found a new sparkle to restart the journey. They started knocking on new doors.” Mentor, YES-Georgia A second factor that improved the training experience for some participants was the change in the training platform and venue. From 2015–2020, trainings were held at Crystal Fund offices after hours, which may have limited participation. In response to the Covid-19 pandemic, the training switched to an online platform, which increased participation, especially among females. The Buzz Georgia participants are older women (ages 16–49) and they found the new online platform easier for them because they could download the curriculum at their convenience. The online platform has distinct advantages in terms of women’s mobility and time-use considerations, although it may have adverse effects too. Networking was consistently listed as an important catalyst to business development plans and remote learning may diminish these important interactions with fellow entrepreneurs. “In Buzz Georgia, we outline from the very beginning, that professional development and the drive for personal growth is critical to business success, therefore, we aim to boost these women’s personal confidence.” Mentor, Buzz Georgia “As youth entrepreneurship school will be fully oriented on female participants, we might think to test paid training program for males.” Chairman, Crystal Fund Figure 12. Beneficiaries’ Assessment of the Training Curriculum Based on its findings, the ET summarizes there is a very high level of participant satisfaction with the free training. During the last five rounds of trainings, the number of beneficiaries per training round ranged from 107 to 127 participants in 2019 and 2020. The participants who received financing experienced much higher levels of satisfaction than those who did not receive financing. The greatest gaps in benefits of those who received funding are related to networking advantages, where the deviation is about 40 percent to 26 percent and the ongoing benefits of mentoring and continued advisory services experienced by those who received financing was more than double than those that did not receive financing, as demonstrated in Figure 13 on the following page. 25 Figure 13. Beneficiaries Perceived Program Benefits The ET identified three types of entrepreneurial trainings offered in Georgia, but none of these trainings are free and available to the YES-Georgia targeted beneficiaries. The key informants confirmed the unique characteristics of the YES-Georgia activity. “Our bank is oriented to offer one-day training to bank clients or their employees on specific narrow topics, but it is not linked to additional funding or any other trainings.” Product owner, Business Support Squad, MSME tribe, TBC Bank “YES-Georgia is very unique program, I have not heard about other financial institutions or training centers that have a comparable business model.” Operations Director, MFO MBC “There is quite high competition for our grants . . . our training component is specifically oriented on grant proposal development and we also have short trainings on specific topics like foreign markets/export.” Head of Strategic Development and communication Department, Enterprise Georgia There are a few organizations that offer limited types of entrepreneurial training and support, which include the following: ● Theoretical business administration course offered by accredited educational institutions; ● Support in developing grant proposals and narrowly defined grant opportunities offered by government entities such as Legal Entity of Public Law (LEPL) Enterprise Georgia and LEPL Georgia's Innovation and Technology Agency offered a one-time entrepreneurial training in Zugdidi; ● Specific advisory services related to taxation issues, agricultural issues, other specific industry or sectoral support for banking clients offered by banks and Micro-Finance Organizations (MFOs); ● Fee-based business advisory services are offered by private educational institutions; ● Internally displaced women can access entrepreneurial training offered by UN Women; 26 ● Social entrepreneurial training is recently being offered by some international donors; and ● Entrepreneurship curriculum and business accelerators for students (mostly in Tbilisi) are primarily focused on theoretical and administrative aspects and less focused on financing. Table 3 below compares the financial products offered by public, private, and non-profit institutions. TABLE 3. COMPARISON OF LENDING FACILITIES IN TERMS OF AFFORDABILITY AND ACCESSIBILITY OF LOANS N NAME OF FINANCIAL FACILITY ACCESSIBILITY AFFORDABILITY CAPACITY DEVELOPMENT AVERAGE SCORE 1 YES Georgia micro equity facility 3.89 6.00 7.14 5.68 2 YES Georgia loan facility 2.50 2.00 4.29 2.93 3 Enterprise Georgia grant facility 3.33 8.00 1.43 4.25 4 Enterprise Georgia cheap loan facility 1.11 4.00 1.43 2.18 5 Gita Grant facility 1.67 8.00 4.29 4.65 6 TBC-EBRD (female-owned startup) facility 3.89 0.00 1.43 1.77 7 BOG-EBRD (female-owned startup) facility 3.89 0.00 1.43 1.77 8 MFO Crystal standard loan facility 3.33 0.00 0.00 1.11 9 MFO MBC standard loan facility 3.33 0.00 0.00 1.11 10 MFO Credo standard loan facility 3.33 0.00 1.43 1.59 11 TBC bank standard loan facility 3.33 0.00 1.43 1.59 12 Bank of Georgia standard1oan facility 3.33 0.00 1.43 1.59 This judgment is based on perception of respondents interviewed during the evaluation process 0=Least, 10=Most 27 The ET discussed two alternative business models with selected key informants: IP, training instructors and mentors, and direct beneficiaries. 1. ALTERNATIVE BUSINESS MODEL: FEE-BASED COST RECOVERY MODEL This alternative business model introduces potential cost-recovery through fee-based training services that would be offered to a more diverse target audience and not limited to rural youth. Very roughly, the estimated training cost per participant is 800 GEL (US$240). This cost does not, however, take into consideration the value of Crystal Fund’s in-kind contributions related to administrative costs such as rent, internet expenses, and trainers. The Crystal Fund staff served as Trainers of Trainers (TOT), which has an estimated value of about three GEL (US$0.90) per round of training per participant, according to the IP. The ET then tried to calculate the cost savings if the curriculum continued to be offered online rather than returning to in-person courses based on the recent experience in 2020. As a result of switching to an online platform, the operational costs of YES-Georgia youth trainings were reduced in 2020 because the costs of renting training facilities, trainers’ payroll expenses, transportation, and hotel accommodation for trainers decreased almost three-fold. In discussions with the Crystal Fund staff, the ET estimated that the average cost per participant decreased from 31 GEL (US$9.40) to 12 GEL (US$3.60) per round of training in 2020. The savings in indirect costs are harder to estimate, but there were reductions in back-office administration, mentors’ payroll expenses, communications, and related administrative costs. The ET roughly estimates that indirect costs were reduced from about 77 GEL (US$23.30) to 56 GEL (US$17) per round of training per participant. Based on available data provided by the IP, the ET estimates the total cost per beneficiary has decreased from about 870 GEL (US$264) to about 635 GEL (US$192).3 For comparison purposes, the average monthly income per household in the same targeted rural areas is 961 GEL (US$291) and the average monthly income per person in rural areas is 301.6 GEL (US$91.4).4 The median cost of university tuition in Georgia is 2250 GEL (US$682) annually. Students are eligible to get a grant from the government based on their performance on Unified National Examinations of Georgia (30 percent to 100 percent of tuition fee for all four years of bachelor’s studies). In conclusion, the estimated cost per beneficiary (635 GEL or US$192) is very expensive for the target group (rural youth and females) and would lead to a significant decrease of quantity demanded (of training services) and the remaining demand would probably be from a different target group in Tbilisi who have much higher levels of financial resources. There is also no evidence that increased marketing and its associated expenses would increase participation. Fee-based training based on real costs is out￾of-reach for the currently targeted beneficiaries as was noted by the IP and its mentors, as noted below: “If this program requires payment, the demand might drop dramatically, or program might end-up working with different segments of the market.” Mentor, YES-Georgia 3 The ET arrived at this calculation based on inputs from the Crystal Fund Business Manager and other staff. 4 Source: GEOSTAT (https://www.geostat.ge/en/modules/categories/50/households-income). 28 2. ALTERNATIVE BUSINESS MODEL: CRYSTAL FUND EARNS DIVIDENDS BY SWAPPING LOANS FOR EQUITY IN HIGH PERFORMING BUSINESSES This second model is hypothetical at this early stage of investment in startup businesses. Crystal Fund potentially could convert a portion of its loans into equity purchases of the highest performing businesses, thereby earning money through its partial ownership of the businesses. For example, if Crystal Fund transferred a portion of its loan in exchange for 10 or 25 percent equity of the small business, there would be three options available for leveraging this equity: 1) Crystal sells its equity stake to an outside investor, 2) the business “buys” business services and products from Crystal in exchange for equity shares, and 3) Crystal increases its share of equity by providing second or third rounds of funding in exchange for greater ownership of the businesses. “The only way Crystal Fund can monetize on this program is through developing a new portfolio by leveraging micro-equity funds or by offering scale-up loans at a higher interest rate.” Mentor, YES-Georgia In conclusion, two factors make it difficult to forecast the potential earnings under this business model. First, the Covid-19 pandemic has caused great economic uncertainty and many small entrepreneurial startups may have difficulty repaying their loans in 2021. Second, the medium-term repayment schedule of micro-equity loans is unknown so there is not enough financial data available to estimate the potential value of a debt-to-equity swap. EQ 3: To what extent are the two Yes-Georgia financing facilities—the Crystal MFO’s loan facility (USD 2.5 million) and micro equity facility (USD 75,500)— accessible to and affordable for their respective target beneficiaries? What changes, if any, are needed to the design and size of the aforementioned two financing facilities (e.g., interest rates, repayment schedule, average loan amount, etc.) to increase their accessibility and affordability? Note: accessibility refers to the beneficiary's ability to obtain a loan in support of a promising business idea despite the lack of collateral, documented revenues, and credit history. Affordability refers to the beneficiary’s ability to repay a loan using revenues generated from a startup/existing business. Based on a sample size of 84 respondents, the YES-Georgia participants stated that financing from the Micro Equity Fund and Crystal MFO standard loan facility is both accessible and affordable to the targeted beneficiaries based on data from the respondents. Participants who successfully complete the YES-Georgia trainings are eligible to apply for zero interest loans with a grace period of three years before repayment from the Micro Equity Fund. As seen in Figure 13, there is no comparable financial product on the market that offers zero percent interest loans. Mentors and some other KII respondents contended that providing zero percent interest loans had advantages compared to grants because they do not distort incentives and the behavior of beneficiary, “Zero percent interest rate loan is healthier than grants because young entrepreneurs behave more responsible . . . the zero percent interest rate loan forces the entrepreneurs to develop good habits and adhere to a repayment schedule that is affordable too.” Mentor, YES Georgia 29 The Crystal MFO loan facility issued its first loans in the last quarter of FY2020, which was not included in the period of performance covered by this evaluation. These 22 loans were so recent that the ET could not access the viability of the business startups and whether the amounts of the loans were sufficient to meet expressed business needs. Thus, the utilization rate of loans is difficult to calculate due to Covid-19 and other factors. According to the Chairman of MFO Crystal and Crystal fund Mr. Archil Bakradze, it is difficult to delineate the composite costs of these two financial products to specify capital, inflation, default risks, and loan administration costs, because interest rates are quite dynamic. Further, it was especially risky to make loans to new businesses during the Covid-19 pandemic and ensuing economic downturn. “The loan facility is in process of development . . . our approach is to disburse loans after we mentor and educate our clients through consulting.” Executive Director, Crystal Consulting Crystal Fund offers loans at the rate of 23 – 25 percent for the graduates of its training program. Based on the ET’s research, these interest rates are competitive in the Georgian microfinance market, which has rates ranging from 23 – 27 percent. The interest rate of the loan also depends upon the riskiness of each client, but because the borrowers are graduates of the training program, their loan should reflect a lower default risk and loan administration cost than the average or standard loan offered by MFO Crystal. Two factors determine the risk factors for lenders that influence interest rates: 1. These female clients will be extensively mentored by Crystal Consulting, which will improve their decision-making skills and practices. Crystal Fund also will engage in more intensive communication with these clients and therefore, monitor the progress of their businesses to reduce the risk for Crystal MFO. 2. MFO Crystal expects demand for its financial products to grow and to disburse larger amounts of loans to these clients compared to their current average loan size. As loan administration is mostly fixed cost per loan, the processing cost per dollar value of loans is expected to decrease. As seen in Figures 14 and 15 on the following pages, the sampling size of 84 respondents found the Micro Equity Loans affordable and accessible as evidenced by 83 percent of the loan recipients who said the process was simple. Some of the respondents received zero-interest loans from YES-Georgia and other financial facilities. Of those that received these loans from other lenders, about 14 percent said the interest rates were high and 10 percent said the collateral was high. 30 Figure 14. Affordability of Loans (Both YES-Georgia and other financial facilities) Georgia has a well-developed banking sector, but these traditional banks are risk-averse. It is estimated that all but six percent of available financial capital is held by banks, which means there are very few alternatives or financial institutes that offer venture capital and other financial products. The Bank of Georgia and TBC Bank are offering new programs targeted towards female entrepreneurs to support the development and implementation of business proposals; this program is financed by European Bank for Reconstruction and Development (EBRD). Under this initiative, these two banks offer startup and standard business loans for women without collateral. EBRD is sharing the risk of default with Bank of Georgia and TBC Bank and as a result startup loan rates are the same as standard business loans. The grace period is up to one year before repayment and the maximum loan term is five years. Loans up to 40,000 GEL (US$12,121) can be disbursed without providing collateral. The Asian Development Bank provided 60 million GEL (US$18.2 million) to Credo bank in 2019 to capitalize a loan program in local currency that targets inhabitants of rural areas. The interest rates ranged from 23 – 27 percent from these lenders. As noted in Figure 15 on the following page, about 40 percent of respondents experienced no change in access to financing while 57 percent experienced increased access to financing through donor-funded programs or commercial banks. It is important to note, however, that half of the youth respondents participated in only one of the two training curriculum modules that focused exclusively on basic education and soft skills development. Therefore, the increase in levels of income may be a higher percentage among those participants who completed both training modules. 31 Figure 15. Accessibility of Loans (Both YES-Georgia and other external financial facilities) The ET frequently saw MFOs offering some types of discounts to their clients, but they are not reflected in strictly “isolated” programs from the perspective of financial institutions. As financial products have flexible prices these strategies are reflected “within these products” interest rate range and thus when a company treats clients as less risky, price (annual effective percentages) are negotiated in favor of the client as reported in the KIIs: “In our industry it is common to negotiate interest rate on the loan products we are offering based on the custom properties of the client and particular loan.” Operations Director, MFO MBC “In my opinion there is no shortage to micro equity fund financing, those who are determined to start business and show adequate effort and still get financing. However, having another round of financing for successful startups could be interesting.” Trainer, YES-Georgia In summary, the ET found that financial products offered in Georgia are somewhat comparable to Crystal MFO’s US$2.5 million loan facility in very broad terms. Several financial institutions (more commonly offered by MFOs for their clients and offered by some banks for their SME clients) have adopted a strategy of providing in-house business startup consulting services to clients or outsourcing similar services for specific sectors. As a result, the estimated riskiness of these clients decreases, and the scale of the business loans increases. These changes lead to lower risk assessments of the clients and lower per dollar administration costs of these loans. Further benefits compound because these same clients become more attractive to other financial institutions and have greater bargaining power. EQ 4: To what extent could the internship program (which was discontinued in 2018) have been designed differently to better meet the needs of the private sector? How could the private sector be encouraged to participate in similar internship programs in the future and contribute resources toward youth development/employment goals? YES-Georgia supported on-the-job training and internship programs with 90 private sector companies; it is estimated that about 200 youth participants obtained full-time employment following the completion of the internships. To gauge the perception of companies on internship practices, the ET initiated the 32 online survey and disseminated the questionnaire through the HRHub network. Figure 16 below outlines the internship sponsor categories as selected by the 52 respondents. Figure 16. Survey of Internship Sponsors by Category of Respondents According to the 52 respondents who participated in the online survey and KIIs, the internship activity was not especially well designed, but at the same time was characterized as a very good initiative. The internship activity was designed exclusively by the staff of Crystal Fund without inputs from the private sector. These staff did not have insight into the specific employment needs of the private sector beyond their own narrow knowledge of the financial sector. The second design flaw was the lack of preparation of the interns, who had limited experience in professional work environments. There was not a good match between the expectations of the employers and the interns. The private sector survey respondents stated their need for qualified entry level employees (interns) but also needed these individuals to possess a minimal level of technical skills in software applications like Microsoft Word and Excel. The private sector respondents expressed a willingness to support internship programs but suggested there needs to be changes in the design before committing their participation. The private sector respondents said that the “match-making” between sector specific needs and interns skills has to be improved. They suggested that it would be valuable to produce a short-list of three candidates that could be interviewed for selection, rather than just accepting one candidate. Secondly, there should be an effort to verify the computer literacy skills of candidates. The respondents were more motivated by the potential access to inexpensive labor than any potential corporate social responsibility benefits. According to the online survey, there are two skill sets sought by sponsors of the internships. The first set of soft skills is related to how the interns interact with colleagues. Private sector firms rated the skills of being a team player and being able to communicate as highly important as 83 percent and 81 percent respectively. Other important skills related to problem solving, creating presentations, and decision-making were also important, as seen in Figure 17 on the following page. 33 Figure 17. Survey of the Soft Skills most Valued by Internship Sponsors Prospective employers and sponsors of internships identified specific technical skills and familiarity with standard software applications, such as Microsoft Excel, Word, and PowerPoint, as seen in Figure 18 below. Figure 18. Survey of Technical Skills most Valued by Internship Sponsors The program developed an internship guide for business companies to support them in institutionalizing the internship practices at their own companies. The workshop held for this purpose was valued by the respondents. “There was a three-day workshop that I participated in. It was very interesting and very intensive. We worked on elaboration of the internship guide. Later, the document was shared with us for feedback as well.” HR representative, private sector company 34 “The document integrating specific steps for introduction of internship at business companies guided the private sector HRs to launch effective internship programs at their organizations.” Representative of HR Hub Furthermore, the project created the platform for the private sector and university career development centers to sit together and discuss the existing gaps in internship supply-demand and matching. The respondents emphasized the importance of such opportunities for further promotion of internship practices among students. “Participation in YES-Georgia gave us, the University Representatives, the possibility to sit together with private sector companies and negotiate on terms of internship. It has never happened before, as private sector companies have never compromised on their offers before. This was the case, when we, both parties, attempted to match the demand of students and employers and reduce the gap as much as possible.” Head of Career Development Center “The internship component was supplemented by the campaign in media, which supported the information dissemination to a greater extent. For example, I can recall of several cases, when private sector representatives, after listening the TV footage, approached the YES-Georgia project representatives about the ways of being engaged in an internship program.” IP staff As for the educational background demanded from interns, 92 percent of companies request at least a bachelor’s degree from interns, and the roles interns are recruited for include assistants, credit officers, administrative roles, etc. Of the 52 respondents who participated in the online survey, about 86 percent of them stated that the internship programs offered by their firm or organization were either very effective or effective, as seen in Figure 19 below. Figure 19. Internship Sponsors’ Rating of the Effectiveness of Internship Programs In summary, the IP did not have a monitoring system in place that allowed the ET to assess the merits of the YES-Georgia internship program in terms of matching interns’ skills to the work placement. Nor was the ET able to assess how many of the YES-Georgia interns received employment upon the 35 completion of the internship program. Generally, based on the online survey, 92 percent of respondents have hosted an intern in the last 10 years and 67 percent of the interns were placed in Tbilisi and the urban peripheral. Furthermore, more than half of the respondents (53 percent) offer paid internships, while 39 percent of the respondents offer both paid and unpaid internships. It is significant that 42 percent of the respondents hire one to five interns annually and 11 percent of the respondents said they hire six to ten interns annually. 4. RECOMMENDATIONS Based on these findings and conclusions, the ET proposes the following recommendations for each of the EQs. EQ 1: To what extent has the entrepreneurship training component helped to start new businesses and increase the incomes of the trainees who did not receive financing through YES-Georgia’s micro equity fund (and who account for approximately 90 percent of beneficiaries)? The ET had great difficulty obtaining data that showed how many new businesses were started and continued to be operational because the IP does not maintain a monitoring system that tracks new businesses or job creation that may have been created after the participants completed the training. The ET estimates that 67 businesses were funded as of September 30, 2020 and 72 percent are still operational while 23 percent failed. The next iteration of YES-Georgia should conduct semi-annual surveys to solicit feedback from participants to understand the circumstances and variables related to business successes and failures. Based on KIIs and online surveys, the majority of the respondents reported increased incomes, especially those who received financing from YES-Georgia or other MFOs. Specific recommendations: ● Consider adapting a better monitoring and tracking system and conduct follow-up surveys with past participants to gain insights into the reasons why some participants’ incomes increased, and to understand the circumstances and variables related to business successes and failures; and ● Utilize the current Facebook groups to develop a platform to host and curate a Community of Practice comprised of YES-Georgia alumni to build an ecosystem of rural entrepreneurs who can support each other. EQ 2: What changes, if any, are needed to the business model of the seven youth entrepreneur schools to ensure their financial viability and continuity after the life of the program? The YES-Georgia business model delivers training to targeted groups and makes loans and financial services available to participants who successfully complete the training. Related to the first component, the training curriculum was adapted in two ways: 1) Crystal Fund’s in-house staff were replaced by more experienced trainers and 2) the learning platform switched from in-person courses to an online platform due to Covid-19, which resulted in higher female participation. If there is indeed an inherent gender bias, it is critical to address this issue because all beneficiaries are female as of January 2021. As 36 noted in the report, adjustments to the training curriculum should be an iterative and collaborative process that addresses potential gender biases and weaknesses that may have prevented female entrepreneurs from obtaining a proportional ratio of loans compared to their male counterparts. The ET recommends that the IP modify its training curriculum and offer a blended learning approach of using both online and in-person learning experiences when teaching students. Blended learning provides more flexibility to working women who often have a heavy burden of domestic responsibilities during the day. Crystal Fund should: ● Make the training more gender-sensitive and inclusive; ensure that trainers and mentors are familiarized with women-focused businesses and businesses that cater to females and a diverse customer base; ● Offer blended in-person and online courses to attract more female entrepreneurs; and ● Make the selection criteria for obtaining loans more transparent and standardized. Although the participants expressed high levels of satisfaction with the training curriculum and instructors, the IP should provide feedback to the instructors, mentors, and Crystal Consulting to improve the curriculum as needed, perhaps focusing on more sector-specific offerings. ● Conduct a landscape survey of rural and Tbilisi-based private sector firms to assess the level of demand for sector-specific trainings; ● Brainstorm with Crystal Consultants and other stakeholders to identify ways to improve the loan applicants’ abilities to access affordable loans and adjust repayment schedules as needed; and ● Conduct follow-up surveys to assess the status of participants who only participated in the component of the training that emphasized basic education and soft skills and determine whether participation in only this component led to business startups or job creation. YES-Georgia training is unique in Georgia because no other competitors target the same rural youth and women. Specific cost-recovery changes to the business model are not supported by evidence at this time. The estimated actual cost per participant for the YES-Georgia training is too high to be recovered by introducing a fee-based business model. Aside from charging training fees to participants, there are other cost-recovery innovations that could offset expenditures, but it is difficult to access accurately due to the Covid-19 pandemic and the unknown severity of economic repercussions in rural communities. The following interventions could reduce the cost of trainings per participant but more research is required: ● Consider creating a corporate social responsibility incentive and outreach to the private sector and other donors to contribute annual scholarships for this unique target group, which would defray costs, increase visibility, and contribute to sustainability; ● Explore the feasibility of debt-equity swap for Crystal Fund’s high performing businesses; 37 ● Develop a customized training curriculum in specific sectors in coordination with private sector firms could be an incentive for investment in the future talent pool of employees in addition to supporting entrepreneurs. EQ 3: To what extent are the two YES-Georgia financing facilities—the Crystal MFO’s loan facility (US$2.5 million) and micro equity facility (US$75,500)— accessible to and affordable for their respective target beneficiaries? What changes if any are needed to the design and size of the aforementioned two financing facilities (e.g., interest rates, repayment schedule, average loan amount, etc.) to increase their accessibility and affordability? Note: accessibility refers to the beneficiary's ability to obtain a loan in support of a promising business idea despite the lack of collateral, documented revenues and/or credit history. Affordability refers to the beneficiary’s ability to repay a loan using revenues generated from a startup/existing business. As noted in the findings, about 60 percent of the participants found the YES-Georgia micro equity loans to be accessible.5 Almost all of the participants who obtained loans though YES-Georgia found the zero interest loans and repayment terms to be very affordable (before Covid-19). They also stated that the size of the loan was adequate for their needs. Evidence suggests that zero interest loans are preferable to grants because the grants issued by international organizations can lead to market distortions. The ET does not recommend changes to the current zero interest rates or loan amount. As noted elsewhere, the ET does recommend adjusting the loan repayment schedules due to the current context of Covid-19 and economic downturn. The findings indicate the zero interest micro equity loans and corresponding low collateral requirements are a unique product in Georgia. The ET canvassed 12 financial institutions in Georgia and did not find comparable alternatives to the YES-Georgia loan products and services. There are no comparable models that specifically target rural youth and offer zero-interest loans, which makes it important to communicate and market the uniqueness of this activity to other potential investors or donors. The ET does not have any specific recommendations for the Crystal MFO business model because there was not sufficient data available; the first round of loans were issued after the period of performance covered in this evaluation. The ET estimates that additional funding from USAID and other donor partners in the amount of US$500,000 would allow for significant changes to the business model. If such funding were to become available, the ET recommends the following: ● Introduce two product lines of loans to distinguish between start-up and scale-up loans, which vary in degrees of risk and adjust interest rates and repayment terms for business that are operational for three years; 5 The findings and related recommendations are limited to the disbursement of micro equity loans because the Crystal Loan Facility did not issue loans until after the period of performance covered in this mid-term evaluation. 38 ● Introduce new financial products such as blended loans/grants and provide assistance in leveraging the Crystal Fund loans with grants offered by the Government of Georgia and international donor partners; and ● Consider converting loans to grants in priority sectors like green energy, infrastructure, and Information and Communication Technologies (ICT). EQ 4: To what extent could the internship program (which was discontinued in 2018) have been designed differently to better meet the needs of the private sector? How could the private sector be encouraged to participate in similar internship programs in the future and contribute resources toward youth development/employment goals? More engagement with private sector firms and academia in designing the internship activity is critical in developing a multi-sectoral approach for internships. The respondents provided very positive feedback regarding the internship initiative but believed the activity was under capacitated. The private sector stakeholders emphasized two important aspects of internships. First, the candidates should have some understanding of how to work in a professional environment, which can be attained through soft skills training. Second, the internships would be more results-driven if the placements better matched technical skills to specific industry needs. Other suggestions include: ● Develop an online platform for candidates to submit their resumes and make it accessible to both university career centers and selected private sector firms (adding some features and preferences from both employers’ and employees’ side could help to increase the level of matching automation); ● Host semi-annual job fairs that introduce potential interns to various public and private job opportunities, which were great networking opportunities according to former interns; ● More consideration should be given to understanding corporate culture, career mobility within targeted sectors, human resource policies, and the work environment; and ● Link corporate social responsibility incentives to firms that sponsor internships such as recognition and annual awards, tax incentives, etc. GENERAL RECOMMENDATIONS In addition to the recommendations stated above, the ET strongly urges that the IP hire a monitoring, evaluation, and learning (MEL) senior consultant or fund a dedicated staff position to comply with contractual obligations. The performance monitoring reporting requirements for the new W-GDP funding are stringent and highly visible in USAID/Washington. It will be incumbent upon the IP to provide timely performance monitoring data and analysis. Further, an experienced MEL specialist could develop the platform for a Community of Practice comprised of YES-Georgia alumni. 39 The ET shared examples with USAID/Georgia of highly regarded W-GDP reports prepared by other IPs that could serve as new reporting templates; the ET suggests that Crystal Fund modify its reporting templates. Two recommendations: ● Encourage the IP to align future quarterly and annual reports more closely to the W-GDP’s three pillars so that key performance indicators and results are more clearly reported, and successes can be amplified within USAID and externally; see examples from current W-GDP activities; and ● There is a missed opportunity to highlight Crystal Fund as a new partner for USAID under NPI, therefore the NPI should be prominently mentioned in all reports. 40 ANNEX A: SCOPE OF WORK STATEMENT OF WORK MID-TERM EVALUATION OF USAID’S YES-GEORGIA PROGRAM IN GEORGIA 1. EVALUATION PURPOSE AND USE Award Number: Cooperative Agreement No. AID-114-A-15-00009 Award Dates: 09/02/2015-02/01/2024 Funding: $1,540,000 Implementing organization: Crystal Fund Agreement Officer’s Representative (AOR): Anna Chaus Alternate AOR: Phillip Greene The evaluation Contractor must provide non-personnel services for a mid-term performance evaluation of the Supporting Youth and Women Entrepreneurship (YES-Georgia) Program in Georgia. The results of the evaluation will be used by USAID/Georgia for improving ongoing interventions in the areas of youth and women economic empowerment, enterprise development, and public-private partnerships. In addition to helping refine the implementation approaches in the remaining years of the YES-Georgia activity, the evaluation will also inform the implementation of the Mission’s new industry￾led skills development program. The primary audience of the evaluation will be USAID/Georgia’s Economic Growth (EG) office and the YES-Georgia implementing partner (Crystal Fund). The results of the study will be shared with local stakeholders (Ministry of Education, Ministry of Economy and Sustainable Development, partner NGOs, etc.,) and other donors working in this area. 2. SUMMARY OF SPECIFIC TECHNICAL REQUIREMENTS The Contractor must ensure that the evaluation team completes the following tasks and provides the following deliverables within the terms defined by the contract: 41 ● Conduct initial teleconference with USAID/Georgia to discuss the upcoming work. ● Provide a draft evaluation design and work plan (including meeting schedules and data collection instruments) to USAID for review and comment two weeks prior to the fieldwork. ● Incoming briefing with USAID management to present the detailed evaluation design. ● Conduct the evaluation in accordance with the USAID-approved evaluation design and work plan. This includes weekly check-in calls with USAID/Georgia to ensure all work is proceeding smoothly and address any outreach challenges the evaluation team may be experiencing. ● Recommendations Co-creation session with USAID Mission to formulate/refine evaluation recommendations. ● Outgoing briefing with USAID management to present the matrix of preliminary findings/conclusions/recommendations of the evaluation. ● Provide a final evaluation report to USAID in accordance with Reporting Guidelines under Section 9 - Deliverables. The evaluation report should follow the “Criteria to Ensure the Quality of the Evaluation Report” of the USAID Evaluation Policy. ● Submit USAID-approved evaluation report to Development Experience Clearinghouse (DEC) within 30 calendar days following the acceptance of the report by the USAID Contracting Officer’s Representative (COR.) ● Submit quantitative dataset, if collected, in a machine-readable format to the Development Data Library (DDL) as per USAID’s Open Data Policy at least five workdays prior to the end date of the evaluation contract. 3. BACKGROUND Reforms in the educational system, labor market, and wider economy have improved the conditions for young people in recent years, but Georgia’s youth still struggle to participate in the economy. Young people experience an unemployment rate that is three times higher than that for those 30 and older, and much of what counts as youth employment is rural, agricultural, and subsistence in nature. To respond to the challenge, on September 2, 2015 USAID/Georgia signed Cooperative Agreement No. AID-114-A-15-00009 with Crystal Fund to implement the “Youth Entrepreneurial Skills for Advancing Employability and Income Generation Program in Georgia” (YES-Georgia) program. The program was a joint initiative of Crystal Fund, PH International, and JSC MFO6 Crystal with $350,000 in USAID funding and $93,000 in cost share from MFO Crystal, with a 30-month implementation period, until January 2018. 6 PH was responsible for building curriculum, monitoring and adoption of the key components of YES-Georgia: “Youth Enterprise” and “Employment Guide to Internships” modules. Crystal Fund was responsible for overall program management, implementation, monitoring, evaluation, and financial reporting. JSC MFO Crystal was responsible for adopting Youth Enterprise module, establishing Youth Micro Equity Facility, co-financing of projects and providing follow-up mentoring. 42 At the inception, the goal of the program was to create economic opportunities and empower Georgia’s young people (aged 17–25) by enabling youth-led enterprises, encouraging innovation, promoting entrepreneurship, and enhancing youth employability through skills training, confidence building, and mentoring. This was achieved through the following interventions: 1) Youth entrepreneurship training program that included “Training of Trainers” and training of beneficiaries; 2) Youth-led start-up financing program that included no-interest and semi-equity financing; 3) Youth internship program within private companies; and 4) JobFest program, a joint initiative with Georgia’s line ministries, which brought hundreds of youth representatives and dozens of public and private companies together to offer young people internship and job opportunities. On September 30, 2017, PH International’s contract ended, and Crystal Fund and JSC MFO Crystal continued the program implementation. In July 2018, USAID approved a 30-month, $350,000 time- and cost-extension of the program. This extension expanded the scope to also target public and private high school students to implement the Junior Achievement Georgia (JAG)’s business skills education curriculum. JAG joined the program as a subcontractor to work with schoolchildren on business educational issues. Thus, the program became a 60-month activity with a total USAID estimated cost of $700,000 and $155,500 cost-share from MFO Crystal. The expanded program had a new scheduled completion date of January 1, 2021 and was called YES-Georgia Phase II. Accordingly, the goal of the YES-Georgia program II Phase was broadened to increase opportunities for self- or waged employment of Georgia’s youth aged 14–28. With the extension, the program continued implementing a targeted system of training in entrepreneurial and business skills and operating a Youth Micro Equity Fund providing capital to young people to establish and develop new small businesses. The course curriculum was presented in a classroom setting, packaged along with take-home materials, and subsequently incorporated into an online web-portal. Those that were not able to participate in a classroom program were able to access the online Youth Academy (http://crystalcrowd.ge), which combined the Youth Entrepreneurs’ Network and crowd-funding platform. Following these intensive training courses, graduates prepared and submitted their business proposals to an investment committee. As a result, promising start-up enterprises received funding from the Youth Micro Equity Fund, launched new businesses, and provided employment opportunities to other young people. It organized special workshops for human resource managers to enhance their knowledge and motivation to design sustainable youth internship programs within private businesses. The Government of Georgia-supported JobFest events informed youth about this major internship initiative. These efforts linked young professionals to over hundred private sector companies and resulted in new employment and internship opportunities. YES-Georgia’s partner, JAG, supported the development of business skills of high school students ages 14–16. Since the start of the component in 2018, JAG has developed Georgian version of the “Be Entrepreneurial” program and introduced it to 30 high school teachers from various regions of Georgia. As a follow up to the training, teachers designed their own schedule and conducted the full course of the “Be Entrepreneurial” training at their schools. Overall, more than 500 schoolchildren benefited from this component. Since the start of the project, YES-Georgia has achieved the following results: 43 ● Established an innovative institutional model for the delivery of entrepreneurial skills and startup financing to youth and women by institutionalizing a network of Young Entrepreneurs’ Schools in seven cities of Georgia (Tbilisi, Gori, Kutaisi, Zugdidi, Batumi, Akhaltsikhe, and Telavi). ● Supported more than 3,000 young entrepreneurs and professionals, roughly 60 percent of whom are women. ● Provided professional skills training to nearly 1,000 young people. ● Assisted more than 200 young people (following the completion of the intensive entrepreneurship training program) to develop business plans and submit their business proposals to Crystal’s investment committee for funding. ● Created 55 start-up businesses run by young entrepreneurs by providing no-interest loans (the first payment is due from entrepreneurs to the MFO Crystal after two years since business startup, and fully paid back by the end of year four) and equity investment through YES-Georgia’s Youth Micro Equity Fund. All businesses are currently operational and employ 63 people. ● Mobilized $230,000 from private and public funds to support youth-owned microenterprises and startups. ● Developed a viable youth mentorship program that provides continued support to YES￾funded start-ups. ● Established a special program to promote entrepreneurship in 30 schools, benefitting more than 500 schoolchildren. ● Supported on-the-job training and internship programs with 90 private sector companies. As a result, more than 200 youth participants have obtained permanent employment. In July 2019, the project was competitively selected to receive additional funding of $840,000 through the White House-led Women’s Global Development and Prosperity Initiative (W-GDP). Thus, the project received another cost extension to further Georgia's economic development with the participation of women. The program was renamed to Supporting Youth and Women Entrepreneurship Program (YES-Georgia), now focusing on women (with an expanded age range of 28 to 49) and aiming to increase women’s soft skills necessary to plan and manage a business. JSC MFO Crystal cost-share increased by 195,000, reaching $350,500. The cost-share from Crystal MFO is used for co-financing young women entrepreneurs’ business initiatives, co-funding the development of advisory and electronic services for women and supporting women’s capacity-building initiatives. The program’s time span is now extended until the end of January 2024. The newly expanded YES-Georgia supports economic growth and empowers Georgia’s youth and women by encouraging innovation, promoting entrepreneurship, and enhancing youth and women employability. The program provides emerging entrepreneurs and professionals with skills development, business training, mentoring, and access to finance. The activity also addresses the gender disparities prevailing in financing of the small and medium enterprise (SME) sector. With the additional funding from W-GDP, the program has renewed its focus toward supporting women, launching new activities 44 designed to meet the needs of women entrepreneurs. Through 2024, YES-Georgia will provide more than 2,500 women with business management training, access to legal and accounting services, mentoring and networking opportunities. The program’s goal is to increase the number of women who become and succeed as entrepreneurs. The recently expanded program supports the following three components: Youth Enterprise component will support young women’s (ages 16–28) entrepreneurship development and capacity-building by improving their access to finance, knowledge, and expertise, and by delivering “Youth Enterprise” training to women from all regions of Georgia. Authors of best business proposals will compete for equity financing and receive mentorship and capacity strengthening support from MFO Crystal and Supporting Youth and Women Entrepreneurs program. The total size of the Micro Equity Facility is $75,000, which will benefit 30 women. The Buzz-Georgia component will increase women’ soft skills necessary to plan and manage a business through “Buzz-Georgia” facility, specifically targeting women in remote regions of Georgia. Buzz-Global is a fully equipped, traveling mini-van that offers training sessions in Georgia’s remote areas, making education mobile and localized. It introduces women to concepts of savings, financial planning, networking, business skills, and personal development. Advisory/Mentorship Services component will provide access to finance as well as business advisory, mentoring and coaching services to women entrepreneurs receiving financing from MFO Crystal. Business loans to women will include the elements of management, sales, business development, coaching, personal empowerment, and financial education. Interest rates on these loans will be at least 5 percent less than the market rate currently offered to micro-entrepreneurs. Beneficiaries will receive advice and mentoring not only at the MFO Crystal’s branches, but also on the spot as the mentors will be visiting women entrepreneurs at the location of their businesses. Beneficiary women will also benefit from networking sessions and workshops specifically designed for their needs. The component will also include the development of a special web and mobile application, which will increase entrepreneurs’ productivity and save their time for record-keeping, accounting and reporting tasks. The program will achieve the following goals by FY 2024: ● Increase access to quality education, training, and market opportunities for 2,500 women (1,500 women from Buzz-Georgia component, 500 young women from YES-Youth Enterprise component, and 500 recipients of Advisory/Mentorship Services component) ● Increase access to finance for 530 women entrepreneurs (30 from YES-Youth Enterprise component and 500 from Buzz-Georgia and/or Advisory Services component) ● Increase the volume of capital lent to female entrepreneurs by MFO Crystal ($2.5 million in total) ● Increase the size and significance of capital accessed by female entrepreneurs (from an average loan size of less than $2,000 to an average loan size of $5,000) ● Empower women as reflected by growing acceptance among men and women of female 45 entrepreneurship (80 percent of participants reporting increased agreement with the concept that males and females should have equal access to social, economic, and political resources and opportunities) 4. KEY EVALUATION QUESTIONS TO BE ADDRESSED The purpose of this mid-term performance evaluation is to determine the effectiveness of specific programmatic approaches, which are referenced in the evaluation questions below, in achieving intended life-of-program results and to provide recommendations on corrective actions and new directions for the remaining years of program implementation. Evaluation should address the following specific questions: 1. To what extent has the entrepreneurship training component helped to start new businesses and increase the incomes of the trainees who did not receive financing through YES Georgia’s Micro Equity Fund (and who account for approximately 90 percent of beneficiaries)? 2. What changes, if any, are needed to the business model of the seven Youth Entrepreneurs Schools to ensure their financial viability and continuity after the life of the program? 3. To what extent are the two YES Georgia financing facilities—the Crystal MFO’s loan facility ($2.5 million) and Micro Equity Facility ($75,500)—accessible to and affordable for their respective target beneficiaries? What changes if any are needed to the design and size of the aforementioned two financing facilities (e.g., interest rates, repayment schedule, average loan amount, etc.) to increase their accessibility and affordability? Note: Accessibility refers to the beneficiary's ability to obtain a loan in support of a promising business idea despite the lack of collateral, documented revenues and/or credit history. Affordability refers to the beneficiary’s ability to repay a loan using revenues generated from a startup/existing business. 4. To what extent could the internship program (which was discontinued in 2018) have been designed differently to better meet the needs of the private sector? How could the private sector be encouraged to participate in similar internship programs in the future and contribute resources toward youth development/employment goals? 5. METHODOLOGY The Mission is looking for the Contractor to suggest the best methods that minimize bias and provide strong evidence. The Contractor is expected to suggest the use of various data collection and analysis methods, both quantitative and qualitative, including document review, key informant interviews, focus group discussions, direct observation, survey instruments (if applicable), etc., with program stakeholders, beneficiaries, the Government of Georgia, the private sector, and other players. The methodology for any evaluation process that involves the selection of participants (e.g. surveys, focus groups, interviews) must be clearly explained and justified. For example, for a survey or mini-survey (if proposed), the 46 number of respondents and their selection process should be explained and justified. The same is true for key informants, focus group discussions, and other methods as well. Selected respondents should be representative of women, youth, and vulnerable groups, where appropriate. The Contractor must conduct a desk review of YES-Georgia related documents, which will help identify areas that merit closer attention once the team begins its fieldwork. Reading materials will be available to the team prior to their arrival. The Contractor must develop a detailed evaluation design and a workplan, including data collection plan and drafts of data collection tools. A draft of the workplan and evaluation design must be shared with USAID COR for review two weeks prior to the fieldwork. The plan will then be presented to the Mission during the in brief in more detail. The evaluation design must include the evaluation matrix (an illustrative evaluation matrix for this study is given below). The evaluation design must explain how the evaluation Contractor intends to conduct the study in detail, including a detailed description of one or more proposed methodologies as well as limitations of proposed methodologies. It must explain in detail what methods will be used to obtain answers for each evaluation question. The design must also explain how the proposed methodology (mix of methods) to conduct the study generates evidence to ensure rigor and reliability of results; and how and why the proposed methodology will minimize bias. The evaluation design must also include the data analysis plan for each question, draft questionnaires (to be included as an attachment), and other data collection instruments or their main features, criteria for assessing responses to evaluation questions, known limitations, and a dissemination plan. The evaluation design might also include specific sub-questions for each evaluation question, where needed. Again, the methods described herein are only illustrative and USAID expects that the Contractor will suggest the best methods that would generate most reliable and evidence-based answers to the key evaluation questions. 47 Research Question Illustrative evaluation matrix: Data Source Methodology 1. helped to start new busi entrepreneurship training component To what extent has the nesses and/or YES who did not receive financing through increase the incomes of the trainees -Georgia’s Micro Equity Fund (and percent of beneficiaries)? who account for approximately 90 rep Activity documentation: program description, quarterly and annual orts, M&E plan, results framework, workplans, sub-awards documentation. ● Project staff ● Crystal MFO staff ● YES-Georgia TA and training beneficiaries ● Trainers and mentors ● Representatives of sub-contractors ● Private sector partners ● Academic institutions Document reviews and/or Mini Group Discussions Interviews / Focus Key Informant Direct Observation/ benchmarks.) and set achieved results (e.g. to compare -Survey sources. with identified data 2. the business model of the seven What changes, if any, are needed to Youth the life of the program? financial viability and continuity after Entrepreneurs Schools to ensure their framework, workplans, sub Program description, quarterly and annual reports, M&E plan, results -awards documentation, financial statements, projections, enrollment and attendance records and other documentation for Youth Entrepreneurs Schools ● Project staff ● Sub-contractor staff ● Crystal MFO staff ● finance/administration staff and instructors Youth Entrepreneurs Schools staff, including and/or Mini Group Discussions Interviews / Focus Informant Observation/Key Direct Document reviews -Survey sources. with identified data 48 ● Youth Entrepreneurs School beneficiaries/students ● Ministry of Education and municipal governments ● Private sector and academia 3. To what extent are the two YES￾Georgia financing facilities—the Crystal Micro Equity Facility ($75,500) MFO’s loan facility ($2.5 million) and — design changes if any are needed to the respective target beneficiaries? What accessible to and affordable for their and size of the aforementioned accessibility and affordability? loan amount, etc.) to increase their rates, repayment schedule, average two financing facilities (e.g., interest framework Program description, quarterly and annual reports, M&E plan, results , workplans, sub-awards documentation, financial rejected grant applications and business plans submitted by YES loan capital and related manuals, documentation for funded and statements and projections for Youth Micro Equity Facility and Crystal’s - Georgia beneficiaries, external reports about Georgia’s financial and non-bank financial sector including demand and supply/statistics/rates, etc. ● Project staff ● Sub-contractor staff ● Crystal MFO staff, including loan officers ● Micro Equity Fund staff, including fund managers and others ● Commercial banks and microfinance organizations ● Private sector ● YES Georgia’s sub-grant, loan and equity financing beneficiaries ● rejected loan applicants, as well as those who com YES Georgia TA and training beneficiaries, including pleted or applied for the training but did not submit a loan application ● Financial sector experts and/or Mini Group Discussions Interviews / Focus Informant Observation/Key Direct Document reviews -Survey sources. with identified data 49 4. To what extent could the internship contribute internship programs in the future and be encouraged to participate in similar sector? How could the private sector to better meet the needs of the private 2018) have been designed differently program (which was discontinued in resources toward youth development/employment goals? reports, M&E plan, results framework, workplans, sub Activity documentation: program description, quarterly and annual -awards documentation. ● Project staff ● Sub-contractor staff ● Beneficiaries of YES Georgia’s internship program ● YES-Georgia beneficiaries including grantees, TA and training recipients, Community-based organizations ● of beneficiar Ministry of Education, representatives of local governments Employees of the Ministry of Economic Development, y municipalities, other relevant state agencies and ministries ● program, including HR staff and management hosted or participated in the YES Georgia’s internship Private sector businesses and public sector partners who ● Other private and public sector organizations ● program Partner academic institutions participating in the internship and/or Mini Group Discussions Interviews / Focus Key Informant Direct Observation/ benchmarks.) and set achieved results (e.g. to compare Document reviews -Survey with identified data sources. 50 6. WORK LOCATION Tbilisi, Georgia’s regions To conduct meetings and interviews, the Contractor may need to travel to Tbilisi and to some of YES￾Georgia-targeted communities located in the Samegrelo, Kvemo Kartli, Shida Kartli, Samtskhe-Javakheti, Mtskheta-Mtianeti Regions and Ajara municipalities. Due to COVID-related travel and other restrictions, virtual meetings and remote supervision by the Evaluation Team Lead may be authorized. 7. EVALUATION SCHEDULE The following levels of effort are illustrative and should serve only as an example of the staff, which may be mobilized under this Contract. These levels may not reflect the actual level of effort contracted, and the Contractor will be expected to submit its own estimate of the level of effort needed to fulfill the objectives. # of Work Days in Country # of Days for Preparation and Report Writing Total # of Work Days /Consultant International Technical Expert – Team Leader 25 30 55 (plus 2 travel days) Evaluation Expert 25 30 55 (plus 2 travel days) Local Consultant 20 9 29 A six-day work week will be authorized in Georgia with no premium pay. Timing (Anticipated Month or Duration) Proposed Activities TBD Initial teleconference with USAID/Georgia TBD Document review, preparation work and finalization of the evaluation design and workplan TBD Submission of the draft workplan and evaluation design to USAID TBD Team leaves for Georgia, fieldwork begins 51 TBD Submission of the final workplan/evaluation design to USAID TBD In-brief with USAID Mission TBD Fieldwork continues TBD Co-creation meeting with the USAID Mission TBD Out-brief with Mission, end of fieldwork, team departs Georgia TBD Data analysis and report writing TBD Submission of the draft evaluation report to USAID TBD Submission of the revised evaluation report to USAID Note: As needed, there may be several rounds of review of the draft evaluation report by USAID prior to finalizing/approving the report 8. PROJECTS DOCUMENTS FOR REVIEW Relevant reports and other project documentation will be provided by the Mission to the Contractor prior to travel to Georgia. The evaluation contractor shall initiate Washington-based work by reading reports and familiarizing him/herself with the program. These documents are: ● Program Description for the YES-Georgia program ● YES-Georgia Implementation Plan(s) ● YES-Georgia quarterly and annual reports ● M&E plans and performance data tables ● Initial list of in-country contacts ● Other reports and papers, as applicable. 9. DELIVERABLES The contractor will be required to provide USAID with the following deliverables: 52 a. Final Work Plan and Evaluation Design: Final Work Plan and Evaluation Design document for the evaluation shall be completed by Contractor and presented to the COR two weeks prior to the team’s arrival in country. The evaluation design will include a detailed evaluation design matrix (including the key questions, methods and data sources used to address each question and the data analysis plan for each question), draft questionnaires and other data collection instruments or their main features, known limitations to the evaluation design, and a dissemination plan. The final design requires COR approval. The work plan will include the anticipated schedule and logistical arrangements and delineate the roles and responsibilities of members of the evaluation team. b. In-brief with the mission: will be held within five days of the team’s arrival in country. This will be a maximum of 30-minute PowerPoint presentation of the plan, namely, how the questions asked in SOW will be answered. Prior to in brief, the evaluation team may have working meeting/s with Evaluation Contract COR and YES-Georgia AOR to discuss the details of the design. c. Conduct fieldwork: The in-country evaluation must expand upon the analysis in the desk review and in the facilitated discussion through methods proposed by the evaluation team that might include interviews with focus groups of sub-contractors, beneficiaries or end-users, Georgian government, private sector entities, field visits, and mini-survey, if proposed. d. Recommendations co-creation meeting: After finishing the fieldwork, the evaluation team must participate in a co-creation session with USAID Mission, including the Evaluation Contract COR and YES-Georgia AOR, to produce a set of recommendations. Prior to the co-creation session/meeting, the evaluation team must summarize and submit to USAID a matrix of preliminary evaluation findings/conclusion/recommendations. The co-creation meeting/session will serve to review the matrix and jointly formulate/refine evaluation recommendations. However, the evaluation team maintains complete editorial authority with regard to the evaluation recommendations section of the Final Evaluation Report (see deliverable (g) below). e. Mission out-brief: The evaluation team must present an outline (in bullets, possibly in power point or as a handout) of the evaluation report with general findings, conclusions, and anticipated recommendations to USAID Mission Management and other interested USAID staff at the end of their visit in Georgia. f. Draft reports: The Contractor must submit to COR a draft report within 20 working days of completing the out-briefing with USAID. This document must explicitly respond to the requirements of the SOW, answer the evaluation questions, be logically structured, and adhere to the standards of the USAID Evaluation Policy. g. Final Evaluation Report: The Contractor must incorporate USAID’s comments and submit the final report to COR within five (5) working days following receipt of the final batch of USAID’s comments on the draft report. The Contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse at http://dec.usaid.gov within 30 calendar days of final approval of the formatted report with USAID consent. In case it is determined that the full report includes sensitive information, the Contractor must produce a “sanitized” version for submission to DEC; the latter also requires COR’s clearance. 53 h. All records from the evaluation (e.g. interview transcripts and summaries, focus group transcripts, code books, etc.) must be provided to the evaluation COR as requested. All quantitative data collected by the evaluation team must be provided in an electronic file in a machine-readable format. The data should be organized and fully documented for use by those not fully familiar with the program or the evaluation. USAID will retain ownership of the survey and all datasets developed. In addition, the dataset must be submitted to the Development Data Library (DDL) as part of USAID’s Open Data Policy. Reporting Guideline The illustrative format for the final evaluation report is as follows: 1. Executive Summary—summarizes key points, concisely states the purpose, background of the project, main evaluation questions, methods, findings, conclusions, recommendations and any lessons learned; should be sufficiently detailed, yet brief, to serve as a stand-alone product (3-5 pp) 2. Introduction—state the purpose, audience, and outline of the evaluation (1 pp) 3. Background—provide a brief overview of the project and the study implemented (1–2 pp) 4. Methodology— the evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology. Greater detail can be included in the appendices (2–3 pp); 5. Findings/Conclusions/Recommendations—explicitly answer each evaluation question; the report should distinguish between findings (the facts), conclusions (interpretation of the facts), and recommendations (judgments related to possible future programming) (10–15 pp); however it should be clear what is the link between them; 6. Lessons Learned (if not covered in findings, conclusions and recommendations) (2–3 pp); 7. Annexes—annexes must include this statement of work and its modifications (if any); any “statements of differences” regarding significant unresolved difference in opinion by funders, implementers, and members of the evaluation team; a glossary of terms; sources of information, properly identified and listed; clear documentation of schedules, meetings, interviews and focus group discussions, and any tools used in conducting the evaluation, such as focus group scripts or questionnaires, checklists and discussion guides used; and signed disclosures of conflict of interest. The evaluation design should also be attached to the report. The report format should be presented in Microsoft Word and use 12-point type font throughout the body of the report, using page margins 1” top/bottom and left/right. The body of the report should ideally be within 20–25 pages, excluding the executive summary, table of contents, references and annexes. The final report must follow USAID branding and marking requirements. Per the USAID evaluation policy, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report. ● The evaluation report should represent a thoughtful, well-researched and well-organized 54 effort to objectively evaluate what worked in the projects, what did not and why. ● Evaluation reports shall address all evaluation questions included in the statement of work. ● The evaluation report should include the statement of work as an annex. ● Evaluation methodology shall be explained in detail, and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report. ● Evaluation findings will assess outcomes and impact on males and females. ● Limitations to the evaluation shall be disclosed in the report, with attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). ● Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. ● Sources of information shall be properly identified and listed in an annex. ● Recommendations shall be supported by a specific set of findings. ● Recommendations shall be action-oriented, practical and specific, with defined responsibility for the action. 10. LOGISTICAL SUPPORT USAID/Georgia and YES-Georgia Activity will provide the possible list of in-country contacts prior to the team’s arrival but will not assist in the logistics of appointing meetings. Hence, the Mission will not be responsible for arranging logistics for the evaluation team. The Contractor must suggest how they plan to arrange translation, transportation, and logistical support to the evaluation team. The COR, through USAID YES-Georgia AOR, will put the Contractor in contact with YES-Georgia implementing partner. While in Georgia, the Contractor will conduct meetings in Tbilisi. Some meetings will require travel to regions outside Tbilisi to meet with YES-Georgia grant recipients and other beneficiaries, and NGO, private sector and government stakeholders. YES-Georgia program implementing partner may assist with setting those meetings. 11. OTHER REQUIREMENTS The evaluation team must be familiar with USAID’s Human Subject Protection Policy and USAID’s Evaluation Policy (http://www.usaid.gov/evaluation). The evaluation team must provide adequate training for its survey staff on survey methodology, USAID’s survey regulations, other relevant regulations, and the data collection plan. The contractor has the responsibility to safeguard the rights and welfare of human subjects involved in the survey research supported by USAID. USAID has adopted the Common Federal Policy for the Protection of Human Subjects, Part 225 of Title 22 of the Code of Federal Regulations (http://www.usaid.gov/policy/ads/200/200mbe.pdf). Recipient organizations must familiarize themselves with the USAID policy and provide “assurance” that they will follow and abide by the procedures of the Policy. 55 All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline, need to be agreed upon in writing by the Contracting officer. INSTRUCTIONS FOR TECHNICAL AND COST PROPOSAL 1. Detailed research design and the work plan: The research design must explain in detail how the proposed methodology (mix of methods) to conduct the study generates evidence to ensure rigor and reliability of results. The evaluation design must explain in details methodologies that will be used to collect required information. Specifically, it must outline in details what methods the Contractor will use to get answers for each evaluation question. The evaluation design must include a detailed evaluation matrix (including the key questions, methods and data sources used to address each question and the data analysis plan for each question), draft questionnaires and other data collection instruments or their main features, known limitations to the evaluation design, a work plan, and a dissemination plan. The work plan must include the anticipated schedule and logistical arrangements and delineate the roles and responsibilities of members of the evaluation team. 2. Proposed evaluation team: The evaluation must be conducted by a team composed of experts. The contractor has to demonstrate that proposed team members have sufficient expertise to carry out the task at a high standard. The Contractor must justify and explain proposed team configuration and distribution of roles among team members. The Team Leader (international) must have extensive, demonstrated experience leading development assistance programs and similar evaluations or assessments of youth and women’s economic empowerment and/or private sector or financial sector development activities. Experience in private sector development in Georgia and/or in the Europe and Eurasia region will be an advantage but is not required. The team leader will be responsible for the day to day management of the team, data collection and synthesis, presentations, and drafting of the interim/final reports. Fluency in English language is required. Excellent writing skills are required and the demonstrated ability to produce well written and sound evaluation deliverables is required. Evaluation Expert must have a justifiable experience in planning and conducting evaluations using various data collection and analysis methodologies. Prior work experience evaluating economic growth activities is also required. Evaluation Expert will travel to the field, lead the team in evaluation design, data collection and synthesis, and also assist in report writing. Evaluation Expert’s role will also include document review and instrument development. Fluency in English is required. Excellent writing skills are required and the demonstrated ability to produce well written and sound evaluation deliverables is required. Locally-hired expert/consultant must have extensive, justifiable experience working in the financial sector development (banking or non-banking financial institutions). Experience of participating as a team member in conducting evaluations is preferable but not required. English language knowledge and good writing skills are required. Fluency in Georgian—both speaking and reading is required. The Contractor must provide 1–2 examples of their proposed Team Leader’s past work (i.e. past evaluation final reports) and information about the selected evaluation team members including their 56 CVs and explain how they meet the requirements set forth in the evaluation SOW. All evaluation team members must be familiar with USAID’s Evaluation Policy. USAID may request an interview with any of the proposed evaluation team member/s via conference call/Skype or any other means available. 57 ANNEX B: METHODOLOGY AND LIMITATIONS The below Methodology and Potential Limitations has been excerpted from the Evaluation Design Report, approved by USAID on November 6, 2019. For this evaluation, primary and secondary data collection will be conducted using a mixed-methods approach. Whenever possible, existing quantitative data such as YES-Georgia performance indicators will be utilized. Additional survey-based instruments will be developed to collect quantitative data to fill existing knowledge gaps. Qualitative data will be collected primarily through remotely conducted KIIs with USAID staff, including the Contracting Officer’s Representative (COR), Activity Managers, the implementing partner (IP), and mission staff points of contact (POCs). LEAP III has formed an ET with expertise in youth and women’s economic empowerment and monitoring and evaluation (M&E). It consists of an experienced international Team Leader who has experience in women’s empowerment, conducting rigorous performance evaluations and working in Georgia. Due to COV1D-19 restrictions, she will not travel to Georgia, but supervise and guide remotely an in-country team of experts, including a senior evaluator, a subject matter expert, and a logistician. LEAP III will provide additional administrative and operational support as needed. USE OF BEST PRACTICES The evaluation will use methods that generate quality data and credible evidence that correspond to the questions being asked, taking into consideration time, budget, and other practical considerations. The evaluation will use sound social science methods and include the following basic features: 1. Establish a team with the appropriate methodological and subject matter expertise to conduct an excellent mid-term performance evaluation; 2. Ensure transparency and dissemination of the evaluation design and final report, including briefings and presentations to the Mission Director in November 2020 and March 2021 and the posting of the final report through USAID-funded information dissemination websites; 3. Use data collection and analytic methods that ensure, to the maximum extent possible, that if a different, well-qualified evaluator were to undertake the same evaluation, he or she would arrive at the same or similar findings and conclusions; 4. Communicate and present separately the credible findings, conclusions and recommendations so the progression is clear and easy to follow in relation to each of the EQs included in the Evaluation SOW; and 5. Remain vigilant and flexible to the changing environment caused by the Covid-19 pandemic that impacts travel to and within different countries. The ET will consult with USAID as it determines how to safeguard the health and safety of its team members. 58 EVALUATION DESIGN The data collection methodology includes the following: a) document review; b) performance indicator assessments; c) electronic survey; and d) KIIs and group interviews. Data analysis methods will include refining the descriptive statistics and content analysis. The ET will utilize a mixed-methods approach, combining a mix of qualitative information collected from interviews, quantitative results of electronic surveys, and the collection of quantitative data from project monitoring and the verification of reported results to assess the success, challenges, and sustainability of YES-Georgia (Table 1). The ET will review performance indicators found in project documentation (e.g., contract agreement, work plans, annual reports) and incorporate as appropriate to address the EQs. Based on consultations with USAID and the ET’s desk review, the ET will select appropriate key informants for interviews and determine the optimal use of group interviews if feasible. 59 GETTING TO ANSWERS MATRIX QUESTION EVALUATION S TYPE OF ANSWER/ EVIDENCE NEEDED (CHECK ONE OR MORE, AS APPROPRIATE) METHODS FOR DATA E.G., COLLECTION, RECORDS, KIIS, SURVEYS7 SAMPLING OR SELECTION APPROACH ANALYSIS CONTENT STREND, DISTRIBUTION FREQUENCY METHODS: ANALYSIS DATA YES/NO TYPE SOURCES SPECIFIC S METHOD financing through YES trainees who did not receive increase the incomes of new businesses and/or component helped to start the entrepreneurship training EQ 1: To what extent has - approximat (and who account for Georgia’s micro equity fund ely 90 percent of beneficiaries)? Yes Description Surveys KIIs Interviews e Indicators Performanc Key Survey Online Quantitative: Interviews Qualitative: Indicators Performance Quantitative: training representatives on vocational KIIs with Ministry of Education KIIs with IPs, COR, Activity Manager Crystal Consulting beneficiaries training) Buzz Georgia participants (soft skills All training participants by sex Data disaggregated Content analysis Trend analysis Yes Comparison8 Yes Explanation9 ensure their financial viability Entrepreneurs Schools to model of the seven Youth are needed to the business EQ 2: What changes, if any, Yes Description with key Interviews collection Data collection Data Quantitative: Technology Agency, Enterprise Georgia’s KIIs with Innovation and TBC Bank, Bank of Georgia, KIIs with Crystal MFO management, Content analysis Trend analysis Cross tabulations No Comparison Yes Explanation 7 Data from evaluations are a deliverable and methods should indicate how data would be captured, i.e., for focus groups USAID requires a transcript. 8 Comparison – to baselines, plans/targets, or to other standards or norms 9 Explanation – for questions that ask “why” or about the attribution of an effect to a specific intervention (causality) 60 the program? and continuity after the life of survey Online s stakeholder Interviews Qualitative: Survey M participants Youth Entrepreneurs Schools’ Georgia entors/coaches Manager Youth Entrepreneurs School strata Nonparticipants from the same by sex Data disaggregated 61 QUESTIONS EVALUATION TYPE OF ANSWER/ EVIDENCE NEEDED METHODS FOR DATA COLLECTION SAMPLING OR SELECTION APPROACH METHODS ANALYSIS DATA YES/NO SOURCES METHOD are the two YES EQ 3: To what extent -Georgia financing facilities - the Micro Equity F facility ($2.5 million) and Crystal MFO’s loan acility ($75,500) - accessible to amount, schedule, average loan interest rates, repayment financing facilities (e.g., the aforementioned two to the design and size of changes if any are needed beneficiaries? What respective target and affordable for their etc.) to increase affordability? their accessibility and Yes Description survey Market Survey Interviews collection Data e indicators Performanc Survey : Quantitative Interviews Qualitative: collection Data e Indicators Performanc : Quantitative KIIs with Crystal Representatives International Innovation and Technology Agency, Enterprise Georgia, Georgia’s MFOs, Bank of Georgia, TBC Bank, KIIs with Credo Bank, Finca Bank, Unsuccessful loan equity applicants Recipients/beneficiaries Buzz Georgia participants donor-funded micro and SME support schemes by sex Data disaggregated Content analysis Trend analysis Yes Comparison No Explanation d have been designed discontinued in 2018) program (which was could the internship EQ 4: To what extent ifferently to better meet Yes Description Performanc Interviews KIIs e indicators Online e Indicators Performanc : Quantitative - Online KIIs with IPs, COR, AOR -survey of HRHub members Interviews with the representatives representatives and PH International Data disaggregated Trend Analysis Content Analysis Yes Comparison No Explanation 62 t goals? development/employmen toward youth and contribute resources programs in the future in similar internship encouraged to participate private sector be sector? How could the the needs of the private Survey Interviews Qualitative: Survey of business companies internship program Youth who participated in the by sex 63 DOCUMENT REVIEW The ET will conduct a focused document review of relevant resources as well as YES-Georgia activity documents. The key variables of the review will include the purpose, goals and objectives, interventions, results, and sustainability of each activity. The ET will enter this information into Excel files to serve as display tables for analysis. The team will review USAID and IP documents in an iterative process of data analysis and writing, including the following resources: USAID AND IP RESOURCES IN DOCUMENT REVIEW RESOURCES RELEVANCY ADS 205/ W-GDP Guidance for USAID’s interlinked policies and strategies to reduce gender inequality and the W-GDP initiative to improve women’s economic empowerment CDCS Specific sections to be scanned: Goal Statement & Development Hypothesis, Results Matrix, and Gender Analysis Youth in Development Policy USAID guidance to improve the capacities and enable the aspirations of youth Activity Profiles and Progress Reports Profiles provide characteristics and attributes of Activities, Monthly and Quarterly Reports show outputs, outcomes, and results Activity MEL Plan, baseline data, and Implementation Plan Allows review and comparison of performance indicators and results Activity/IP-generated websites, media, notices, tools, guides, etc. Provides information on IPs’ activities, outputs, outcomes, and key stakeholders PRIMARY DATA: SURVEYS AND INTERVIEWS The primary data collection will be conducted in the Georgian language for all respondents unless a respondent requests the interview to be conducted in English. The electronic survey and KIIs with USAID staff will be conducted in English. The survey instruments are presented in English in Annexes A and B, but will be translated into Georgian prior to dissemination. Electronic Mini-Surveys: The ET will develop and conduct three online surveys, each described in further detail below, as a pre-screening and data collection instrument before conducting the KIIs. These surveys will be structured and utilize a combination of dichotomous questions i.e. Yes/No/Don’t Know, Likert scale (using a 5-point rating scale), and open-ended responses. The online surveys will be hosted using an online platform called Survey Monkey. Respondents will be requested to complete the survey within one week of receipt, and reminder emails will be sent to those who do not complete the survey. 64 One of the online surveys will assess and compare the experiences of beneficiaries of the three training courses: Youth Enterprise Component, Buzz Georgia, and Crystal Consulting. This survey also will have two tracks of questions that are specific to beneficiaries who received micro-financing and beneficiaries of the training courses who did not receive micro-financing. As noted in the SOW, all beneficiaries are members of Facebook groups for each of the training sessions, therefore, the ET can easily disseminate the questionnaire through this network. The second and third online surveys are specific to the HRHub internship activity, which provided placements in private sector firms in Georgia to youth. The second survey is designed to assess the effectiveness and results of the internship activity from the perspective of the interns (provided that the IP has current contact information). The third survey will be disseminated to the private sector firms that sponsored the internships and gather data about how this activity could better meet the needs of the private sector and other recommendations. KIIs: The ET will conduct qualitative, in-depth individual interviews with key stakeholders, project partners, and other informants. The ET will conduct video conferencing interviews using the GoogleMeet (for all USAID staff), Zoom, Skype, and WhatsApp platforms. Key informants will include the following: ● USAID staff, including the Contracting Officer’s Representative (COR), YES-Georgia and other USAID Economic Growth staff that have been involved in the project and Program Office staff that have been involved in design and monitoring and evaluation; ● Ministry of Education, Science, Culture and Sport, Youth Agency, Ministry of Economy and Sustainable Development, and government support startup/SME initiatives working in this sector; ● Implementing partners and beneficiaries of the YES-Georgia activities: Buzz Georgia, JSO Crystal, Crystal Fund, Crystal Consulting, Youth Entrepreneurs Schools, Youth Entrepreneurs School management representatives, HRHub private sector members, and interns; ● Financing facilities and service providers including: Credo, Finca Banks and other microfinance organizations (MFOs), government programs, start-up programs and SME support schemes by the Bank of Georgia, TBC Bank, Georgia’s Innovation and Technology Agency, Enterprise Georgia, and international donors; and ● Women-owned businesses and any women-focused business organizations, given the Mission’s desire to focus on gender-disaggregated outcomes. Sampling is not intended to be random, but rather, participants will be chosen specifically for their relationship to the project, as in the case of USAID staff, partner staff, and Government of Georgia officials. DATA ANALYSIS METHODS The mid-term performance evaluation includes a comprehensive Getting to Answers matrix (Table 1) that maps the EQs to data sources and data analysis methods. Once the ET’s data plan is developed fully, the ET will use a document review instrument to guide data collection along with interview guides and 65 protocols for conducting the video conferencing interviews. The ET will then collect data from the document review and interviews, then analyze it using descriptive statistics or content analysis to develop the findings to answer the EQs. The ET will also use descriptive statistics to produce a quantitative overview of the YES-Georgia activities, including characteristics such as the number of participants, regions, and in-country partners. The ET will use standard qualitative analysis to review the data summaries and data display tables described above. Secondary priority will be given to less common themes and patterns that illustrated key characteristics relevant to the EQs. The ET will conduct semi-structured telephone interviews with USAID staff, IPs, and selected beneficiaries to gather their perspectives and additional information about the YES-Georgia process and activities. The team proposes to use a manual review process to extract key data such as keywords, quotes, or substantive information about activities from the transcripts. A core technical approach will be triangulation: the systematic, evidence-based, careful synthesis of disparate findings (from a broad variety of data sources) to discern consistent themes, trends, and patterns. Because the ET will be synthesizing data from multiple sources, it is imperative that the ET employs a broad variety of analytical technical techniques throughout the mid-term performance evaluation. These techniques will be customized to fit both the available data sources and address the EQs provided in this plan. POTENTIAL LIMITATIONS Attribution: To date, there is no common baseline or theory of change for assessing overall progress in improving youth and women’s economic empowerment. Complex Environment: The activities included in the YES-Georgia project represent a wide array of socio-economic complexities that will be viewed at a macro level. Entrepreneurial and economic empowerment issues are highly complex and often fluctuate depending upon the region, sector, and interlocutor. Youth and gender issues intersect with and sometimes exacerbate other forms of social exclusion based on religious minority status, caste, ethnicity, race, sexual identity, disability, and others. Limited Fieldwork: Working during the Covid-19 pandemic may require trade-offs in terms of the number and depth of interviews and perspectives that can be captured and the depth of analysis that can be performed. Remote fieldwork will be substituted based on the Covid-19 and both USAID and Integra’s assessment of travel risks. Instrumental Bias: Guarding against instrumental bias is a consideration because many beneficiaries and in-country partners were exposed to or participated in more than one intervention and their responses may be influenced by participation in multiple interventions. 66 ANNEX C: DATA COLLECTION INSTRUMENTS INTERVIEW TRACKING DATA This section will complete this section prior to conducting the KII. Date of Interview Location of Interview Name of Data Collector Name of Respondent Role or Position/Title of Respondent Male/Female Respondent Affiliation USAID, Implementing Partners, Training Beneficiaries, Private Sector Sponsors of Internships, Interns, Government of Georgia SCRIPT FOR START OF THE INTERVIEW Hello, __________. My name is __________and I am working with Integra to conduct an evaluation of USAID’s YES-Georgia. The purpose of this evaluation is to help you and USAID/Georgia gain a better understanding of how the Activity has worked over the last four years, what results have been achieved to date, and how it might be improved going forward. Consent to the Interview and Recording the Interview Your participation is voluntary. No one will know your responses to the questions. Let me know if you want to pause or stop the interview at any time. Would you be willing to allow the interview to be recorded? Y/N You have the right to participate in the interview without being recorded, Do you have any questions? Thank you. 67 KIIS FOR USAID DISCUSSION GUIDE 1. What was your role in the project? 2. What period of time were you engaged with the project? 3. What are some of the binding constraints hindering employment, SME, access to finance? Probe: How is YES addressing these constraints? Systemic change, focusing on key cohorts and/or geographic regions? 4. How would you describe the startup of these activities? Probe: What were early challenges and how were they addressed? Implementing Partners capacity, partnership agreements, adjusting to the cultural, social or political context, activity budget, etc. 5. How would you describe the cost-efficiency of this activity? Probe: Were some components more efficient than others? 6. To what extent have the activities achieved the contract’s specified results? What were the most significant factors that led to results? Probe: Project design, management approach, relationship with stakeholders, human resources availability, and sub-national versus national stakeholder engagement. 7. What have been the most significant achievements related to this project? 8. To what extent do you believe the three training activities (Youth Enterprise, Buzz Georgia, Crystal Consulting) prepared the beneficiaries for obtaining financing or employment? 9. Do you believe any of these three activities (Youth Enterprise, Buzz Georgia, Crystal Consulting) will be sustained after the YES funding ends? Probe: Impacts achieved to date, participants pursue SME goals, implementing partners obtain additional funding from GOG or other donors 10. Did the MEL plan contribute to adaptive management in terms of adjusting YES’s technical approaches and interventions? 11. What are some of the challenges affecting the partnerships with the implementing partners? Any suggestions to make these partnerships more effective or efficient? 12. Are there other groups or people we should talk to about this project? KIIS FOR GOVERNMENT OF GEORGIA DISCUSSION GUIDE 1. What is the country’s vision towards youth/women economic empowerment? 68 2. What is the current climate affecting employment and access to finance for micro and small medium enterprises (SMEs)? 3. What specific challenges do youth and women experience in achieving economic empowerment? 4. Which types of economic empowerment programs are currently implemented by the government at the central and local levels? Who are the targeted beneficiaries? Probe: Is the YES design and intended beneficiaries unique? How so? 5. What are the expected outcomes and strategic impact of these government-sponsored programs? Probe: Are the programs targeted geographically or towards age groups or income levels? Is it to increase employment or investment is specific sectors? 6. What are the communication or outreach strategies to attract entrepreneurs or potential investors? 7. What are some examples of effective programming to support micro and SMEs? Probe: Are these programs administered through partnerships or standalone programs? 8. Do you have an opinion on the effectiveness of grant schemes or subsidized loan schemes? What are the advantages/disadvantages of each scheme? 9. Are there other groups or people we should talk to about youth and women’s economic empowerment programs? 69 KIIS FOR IMPLEMENTING PARTNERS DISCUSSION GUIDE 1. What is your role in the activity? 2. What are some of the challenges hindering employment, SME, access to finance? Probe: How is YES addressing these challenges for women and youth? 3. How would you characterize your target group? Probe: Profile of Participants, including gender, age, rural/urban, education, etc. 4. What were some of the factors that led to success or not meeting expected goals? 5. As you made adjustments to the training venues during the COVID-19 pandemic, did the students respond similarly? Were there differences? Probe: IP Activity Reports anecdotally suggest that many female students were more comfortable with online platforms than classroom learning. 6. From your point of view, what are the underlying causes or factors that resulted in only 10 percent of the participants obtaining financing for their business plans? 7. To what extent have the activities achieved the contract’s specified results? Probe: Project design, management approach, relationship with stakeholders, human resources availability, and sub-national versus national stakeholder engagement. 8. What types of adjustments were made to make the training more efficient? Probe: What were the most significant factors that led to results? Selection criteria of participants, recruitment efforts, training curriculum, location, venue, etc. 9. What have been the most significant achievements of the activities in terms of the trainees obtaining financing or employment? 10. What percentage of participants have pursued entrepreneurial activities or investments? 11. From your point of view, which interventions/approaches worked well, and which did not? 12. How likely are these three activities (Youth Enterprise, Buzz Georgia, Crystal Consulting) to be sustainable after the YES funding ends? Probe: Impacts achieved to date, participants pursue SME goals, implementing partners obtain additional funding from GOG or other donors. 13. What will be the factors that determine whether the Youth Enterprise program will continue after the YES-Georgia funding ends? 14. What are some of the challenges affecting these partnerships with USAID and other donor-funded projects? Any suggestions to make these partnerships more effective or efficient? 15. Are there other groups or people we should talk to about this project? 70 KIIS FOR HRHUB PRIVATE SECTOR SPONSORS OF INTERNS DISCUSSION GUIDE 1. What was your role in the project? 2. Can you tell us about your firm/organization’s employee recruitment practices? 3. Do you have policies in place regarding internships? Probe: Is the YES internship unique or aligns with established practices? Are the internships paid or unpaid? 4. What are some of the challenges hindering employment, SME, access to finance? Probe: How is YES addressing these challenges for women and youth? 5. How would you characterize the profile of an ideal candidate for internships? Probe: Profile of Participants, including gender, age, rural/urban, education, etc. 6. What were some of the factors that led to interns achieving success or not meeting expected goals? 7. How did your employment practices adjust during the COVID-19 pandemic? 8. Are different support mechanisms needed to promote internships within the business community, academic, international or not-for-profit organizations? 9. To what extent has the cooperation with YES-Georgia achieved the contract’s specified results? What were the most significant factors that led to results? Probe: Project design, management approach, relationship with stakeholders, human resources availability, and sub-national versus national stakeholder engagement. 10. What types of adjustments could be made to make the internship more efficient? Probe: Selection criteria of participants, recruitment efforts, training curriculum, partnership arrangements, etc. 11. Does your company have any experience of cooperating with academic institutions or donors regarding internship programs? Probe: What types of incentives are important to foster collaboration? 12. How likely is the internship program to be continued after the YES funding ends? Probe: Sponsoring firms obtain additional funding from GOG or other donors 13. What are some of the challenges affecting these partnerships with USAID and other donor-funded projects? 14. Are there other groups or people we should talk to about this project? 71 KIIS FOR HRHUB INTERNS DISCUSSION GUIDE 1. Which internship program did you complete and when? 2. Can you tell us how you heard about YES-Georgia? 3. What was your experience in applying for an internship? Probe: YES-Georgia internship outreach, recruitment, vetting, and placement. 4. Did you select the firm or organizations for your internship or were you given several options to choose among? 5. How would you describe your internship responsibilities and expectations? Probe: Understanding of expectations, support, preparedness, level of support. 6. Did you feel prepared for the type of work? 7. What were some of the factors that made this a positive or negative experience for you? Probe: Where the expectations clearly explained or written? 8. Were you offered employment during or after the internship ended? 9. Setting aside the special circumstance of the COVID-19 pandemic, do you believe this internship prepared you for your future career? Probe: How does this internship prepare and/or contribute to the necessary skill sets in a chosen career or exploration of a career path? 10. What types of adjustments could be made to make the internship more helpful in your pursuit of a career? Probe: Did YES-Georgia check-in with the participants? Conduct exit interviews or follow-up? 11. What is your employment status? 12. Did the internship program play any role in helping you find your current job? 13. What advice would you give to future interns? Probe: What are the biggest lessons learned from the individual perspective? 72 KIIS FOR FINANCIAL INSTITUTIONS DISCUSSION GUIDE 1. What was your role in this financial institution? 2. What are some of the challenges that new business face in accessing financing? Probe: How is YES addressing these constraints? Systemic change, focus on key cohorts and/or geographic regions? 3. What is the range of loan products for micro enterprises and the corresponding annual effective interest rates and terms? 4. What is the volume of financing for micro enterprises and SMEs? What is the average size of loan and investment for micro and SMEs? 5. What is the range of collateral percentages for loan products? and what types of collateral are accepted? 6. What are the factors you take into consideration for estimating client risk? Probe: Is there written information that can be shared on client characteristics by loan product? 7. How do you reach your clients? What types of marketing do you utilize? 73 ANNEX D: STATUS OF FUNDED BUSINESS PER MUNICIPALITY 74 ANNEX E: SOURCES OF INFORMATION LIST OF INTERVIEWS # DATE TIME INTERVIEWEE FULL NAME GENDER POSITION NAME, ORGANIZATION SECTOR 1 27.11.2020 11:00 AM Tamar Samkharadze Female Acting Head of Vocational Education Department, Ministry of Education Government 2 27.11.2020 2:00 PM Revaz Kharabadze Male Owner and founder of JobGear, Former Head of HR Department of JSC MFO Crystal Implementing Partner 3 30.11.2020 4:00 PM Nino Lagvilava Female Chief of Party, Supporting Youth and Women Entrepreneurship in Georgia (YES-Georgia) Implementing Partner 4 01.12.2020 10:00 AM Giorgi Janelidze Male Executive Director, Crystal Consulting Implementing Partner 5 01.12.2020 4:00 PM Archil Bakuradze Male Chairman, Crystal Fund Implementing Partner 6 02.12.2020 11:00 AM Maya Kobalia Female Component lead/trainer, Buzz Georgia Implementing Partner Lia Salukvadze Female Co-trainer/ facilitator, Buzz Georgia Implementing Partner Nato Chakvetadze Female Trainer/Mentor Implementing Partner 7 02.12.2020 2:00 PM Giorgi Tukhashvili Male Senior Business Advisor/Mentor/Lead Trainer, Young Entrepreneur's School Implementing Partner Nikoloz Gogochuri Male Mentor/Lead Trainer, Young Entrepreneur's School Implementing Partner Medea Chukhua Female Trainer, Young Implementing 75 Entrepreneur's School Partner Monika Philishvili Female Trainer, Young Entrepreneur's School Implementing Partner 8 03.12.2020 10:00 PM Berdia Natsvlishvili Male Country Director, PH International Implementing Partner 9 03.12.2020 11:00 PM Lasha Mikiashvili Male Trainer, Young Entrepreneur's School Implementing Partner Tamta Gvelesiani Female Trainer, Young Entrepreneur's School Implementing Partner Sandro Kandelaki Male Trainer, Young Entrepreneur's School Implementing Partner 10 03.12.2020 6:00 PM Tinatin Mukhuradze Female Former HR Manager, Rustavi Steel Private Sector 11 03.12.2020 4:00 PM Ketevan Chogovadze Female Program Development Specialist, USAID USAID 12 04.12.2020 11:00 PM Ana Navdarashvili Female Consultant/ Expert/HRHub founder Private Sector 13 04.12.2020 4:00 PM Anne Chaus Female Project Management Assistant, USAID USAID 14 07.12.2020 1:00 PM Ana Totibadze Female Product owner, Business Support Squad. MSME tribe, TBC Bank Financial Institution 15 08.12.2020 10:00 AM Salome Kvakhadze Female Former Hr Manager, Natakhtari Company Private Sector 16 08.12.2020 11:30 AM Nino Samvelidze Female Programme Manager, Delegation of the European Union to Georgia EU 17 08.12.2020 12:30 PM Sophio Kharitonashvili Female HR Manager, J.S.C. Wissol Petroleum Georgia Private Sector 18 08.12.2020 4:00 PM Marika Olson Female Office Director, Office of Economic Growth, USAID USAID 76 19 09.12.2020 11:00 AM Merab Gvinianidze Male Micro Business Products Manager, Credo Financial Institution 20 09.12.2020 1:00 PM Davit Pitiurishvili Male Senior Manager of Program Management Group, Enterprise Georgia Government Avtandil Ghoghoberidze Male Head of Strategic Development and communication Department, Enterprise Georgia Government 21 10.12.2020 11:30 AM Rezo Charkviani Male Chair Person, Youth Agency Government Kristina Kapanadze Female Deputy Chair Person, Youth Agency Government 22 10.12.2020 4:00 PM Mariam Meburishvili Female Former HR Manager, Free University Georgia Academia 23 10.12.2020 2:00 PM Ana Kitiashvili Female National VET Policy Expert, PEM Industry-led Skills and workforce development EU 24 11.12.2020 2:30 PM Lela Topuria Female Head of HR Department, Finca Bank Georgia Financial Institution Nato Oboladze Female Marketing Communications Coordinator, Finca Bank Georgia Financial Institution Mariam Esebua Female Head of Marketing, PR and Customer Experience Department, Finca Bank Georgia Financial Institution 25 14.12.2020 11:00 AM Salome Khachiauri Female Head of the Department of Innovation & Commercialization, Georgia’s Innovation and Technology Agency (GITA) Government 77 26 14.12.2020 5:00 PM Nino Okhanashvili Female HR Consultant, Trainer, Coach Private Sector 27 15.12.2020 1:00 PM Eter Chachibaia Female Operation Director, MBC Financial Institution Nino Devdariani Female Risk Director, MBC Financial Institution 28 18.12.2020 4:00 PM Irakli Kutivadze Male Deputy Head of Retail Business Banking Department, Bank of Georgia Financial Institution Giorgi Kintsurashvili Male Head of Digital Banking Ecosystem Unit, Bank of Georgia Financial Institution 29 22.12.2020 3:00 PM Giorgi Kikiani Male Former Intern Intern Liza Katsiashvili Female Former Intern Intern 30 22.12.2020 7:00 PM Londa Arveladze Female Former Intern, Crystal MFO Intern Malvina Malakmadze Female Former Intern, Crystal MFO Intern Dali Kordzaze Female Former Intern, Crystal MFO Intern Nana Pkhaladze Female Former Intern, Crystal MFO Intern 31 28.12.2020 2:00 PM Archil Bakuradze Male Chairman, Crystal Fund Implementing Partner 32 20.01.2021 12:00 PM Tamar Aivazashvili Female Financial Officer, YES￾Georgia Implementing Partner 78 ANNEX F: DISCLOSURE OF CONFLICT OF INTEREST FORMS 79 80 81 82 83 84 ANNEX G: DIVERGENCE OF OPINION Pending inputs from the USAID/Georgia Mission and IP: Crystal Fund. 85 ANNEX H: EVALUATION DESIGN REPORT Please note that for the purpose of this Mid-term Evaluation Report and in an effort to reduce repetition, annexes from the EDR have not been included. November 2020 This document is made possible by the support of the American People through the United States Agency for International Development (USAID) and was prepared by Integra Government Services International LLC for the Learning, Evaluation, and Analysis Project (LEAP III) Activity. EVALUATION DESIGN REPORT SUPPORTING YOUTH AND WOMEN ENTREPRENEURSHIP (YES-GEORGIA) MID-TERM EVALUATION Contract Title: LEAP III: Learning, Evaluation, and Analysis Project Contract Number: GS-10F-083CA / 7200AA18M0004 Activity Number: 1009.1053 Submitted: Version 2 – November 4, 2020 Contractor: Integra Government Services International LLC 1156 15th Avenue NW, Suite 800 Washington, DC 20005 USAID Office: USAID/Georgia COR: Eric Hyman, ehyman@usaid.gov 86 1 CONTENTS ACRONYMS I 1. INTRODUCTION 1 2. BACKGROUND 1 3. EVALUATION PURPOSE, AUDIENCE, AND QUESTIONS 2 4. EVALUATION DESIGN AND METHODOLOGY 4 5. EVALUATION DELIVERABLES AND TIMELINE 11 ANNEX A. ONLINE SURVEY QUESTIONNAIRES 12 ANNEX B. KEY INFORMANT INTERVIEW PROTOCOLS 20 ANNEX C. DELIVERABLES AND TIMELINE 28 ANNEX D. ILLUSTRATIVE EVALUATION MATRIX 29 ANNEX E. RESOURCE DOCUMENTS 31 87 i ACRONYMS COR - Contracting Officer’s Representative DQA - Data Quality Assessment ET - Evaluation Team FDG - Focus Group Discussion LEAP - Learning, Evaluation, and Analysis Project KII - Key Informant Interview MFO - Microfinance Organization POC - Point of Contact USAID - United States Agency for International Development YES-Georgia - Youth Entrepreneurial Skills for Advancing Employability and Income Generation Program in Georgia 88 1 1. INTRODUCTION At the request of USAID/Georgia, the USAID-funded Learning, Evaluation, and Analysis Project (LEAP III) is conducting an independent mid-term performance evaluation of the Supporting Youth and Women Entrepreneurship (YES-Georgia) Program in Georgia, implemented by Crystal Fund. This Evaluation Design Report (EDR) describes the approach that the LEAP III team will take to this evaluation, along with protocols for electronic surveys and semi-structured interviews (Annex A and B, respectively), proposed timeline (Annex C), Evaluation Matrix (Annex D) and List of Resource Documents (Annex E), to implement the mid-term performance evaluation. The purpose of this mid-term performance evaluation is to determine the relevance of YES-Georgia’s programmatic approaches in achieving intended life-of-program results and to provide recommendations on corrective actions and new directions for the remaining years of program implementation. The Evaluation Team (ET) will review actual progress toward achieving key expected results and identify accomplishments, delays, challenges, and their impact on achieving YES-Georgia’s life-of-program targets. Due to the COVID-19 pandemic and the likelihood that in-person key informant interviews (KIIs) and focus group discussions (FGDs) may not be possible, this plan assumes that most primary data collection will be conducted remotely.1 Related, LEAP III is currently conducting a portfolio performance review of 48 W-GDP activities, working closely with USAID to develop data collection quality assurance and communications protocols within the COVID-19 context. The ET will consult with USAID/Georgia regarding country and local safety protocols. 2. BACKGROUND Reforms in the educational system, labor market, and the wider economy have improved the conditions for young people in recent years, but Georgia’s youth still struggle to participate in the economy. Young people experience an unemployment rate that is three times higher than that for those 30 years of age and older, and much of what counts as youth employment is rural, agricultural, and subsistence in nature. To respond to this challenge, on September 2, 2015, USAID/Georgia signed Cooperative Agreement No. AID-114-A-15-00009 with Crystal Fund to implement the “Youth Entrepreneurial Skills for Advancing Employability and Income Generation Program in Georgia” (YES-Georgia) program. The program was a joint initiative of Crystal Fund, PH International, and JSC MFO Crystal. The program is a 60-month activity with $700,000 in funding from USAID and $155,500 cost share from MFO Crystal. The goal of YES-Georgia is to create economic opportunities and empower Georgia’s young people (ages 17 – 25) by enabling youth-led enterprises, encouraging innovation, promoting entrepreneurship, and enhancing youth employability through skills training, confidence building, and mentoring. This is to be achieved through the following interventions: 1) a youth entrepreneurship training program that included “Training of Trainers” and training of beneficiaries; 2) a youth-led start-up financing program that included no-interest and semi-equity financing; 3) a youth internship program within private companies; and 4) the JobFest program, a joint initiative with Georgia’s line ministries, which brought hundreds of youth representatives and dozens of public and private companies together to offer young people internship and job opportunities. This phase of activities will be completed on January 1, 2021. 1 Although the country-based team may conduct KIIs/FDGs in person, pending COVID-related precautions/restrictions. 89 2 In 2019, the YES-Georgia project received additional funding in the amount of $840,000 from USAID’s Women’s Global Development and Prosperity Initiative (W-GDP). MFO Crystal also increased its cost share by $195,000 to support the newly named “Supporting Youth and Women Entrepreneurship Program” und the YES-Georgia. This newly expanded program focused on increasing women’s economic empowerment and access to finance will run through January 2024. 3. EVALUATION PURPOSE, AUDIENCE, AND QUESTIONS PURPOSE The objective of this mid-term performance evaluation is to provide USAID with findings and strategic recommendations based on the first five years of YES-Georgia implementation (2015 – 2020). This mid￾term performance evaluation aims to determine the relevance of YES-Georgia’s programmatic approaches in achieving intended life-of-program results and to provide recommendations on corrective actions and new directions for the remaining years of program implementation. The results of the mid-term performance evaluation will be used by USAID/Georgia for improving ongoing interventions in the areas of youth and women’s economic empowerment, enterprise development, and public-private partnerships. In addition to helping refine the implementation approaches in the remaining years of YES-Georgia, the evaluation also will inform the implementation of the Mission’s new industry-led skills development program. AUDIENCE The primary audience of the evaluation will be USAID/Georgia’s Economic Growth Office and the prime implementing partner and subcontractors. USAID/Georgia may also share the results of this evaluation with local stakeholders such as the Ministry of Education, Ministry of Economy and Sustainable Development, local not-for-profit organizations, and other partners working in this sector. EVALUATION QUESTIONS The mid-term evaluation will answer the following evaluation questions (EQs) and sub-questions: EQ1: To what extent has the entrepreneurship training component helped to start new businesses and/or increase the incomes of the trainees who did not receive financing through YES-Georgia’s Micro Equity Fund (they account for approximately 90 percent of beneficiaries)? • Is there a difference in the incomes of the trainees who received the financing and those who did not receive the financing and what are the key factors influencing these differences, if any? • To what extent has their income changed as a result of receiving the training and/or financial support (adjusted for inflation)? • To what extent did the location (rural or urban) and/or gender of the trainees affect their ability to submit applications to the Micro Equity Fund? • To what extent do the trainees believe their training and/or financial support will have a lasting impact on their ability to start new businesses? EQ2: What changes, if any, are needed to the business model of the seven Youth Entrepreneurs Schools to ensure their financial viability and continuity after the life of the program? 90 3 • To what extent do the Youth Entrepreneurs Schools’ participants see the benefits of participating in the program? • Which aspects of the training curriculum do the participants value the most (theoretical, practical, or something else)? • Are there other organizations, government services, or financial facilities offering comparable training free of charge or fee-based services? • What are the forecasted expenses if there are changes in the business model, such as fees for services or marketing expenses? • Are there identifiable resources for subsidizing the training and financial services after the YES￾Georgia funding ends? EQ3: To what extent are the two YES-Georgia financing facilities - the Crystal MFO’s loan facility ($2.5 million) and Micro Equity Facility ($75,500) - accessible to and affordable for their respective target beneficiaries? What changes, if any, are needed to the design and size of the aforementioned two financing facilities (e.g., interest rates, repayment schedule, average loan amount, etc.) to increase their accessibility and affordability?2 • What are the characteristics of the Micro Equity Facility that make it unique? • Are there any financial facilities operating in Georgia that target the same beneficiaries? • Are there financial products offered in Georgia that are comparable to Crystal MFO’s $2.5 million loan facility? • Are interest rates offered by the two loan facilities affordable and accessible to its targeted beneficiaries? • What is the percentage of associated costs, (capital, inflation, default risks, and loan administration) included in these subsidized loans? EQ4: To what extent could the internship program (which was discontinued in 2018) have been designed differently to better meet the needs of the private sector? How could the private sector be encouraged to participate in similar internship programs in the future and contribute resources toward youth development/employment goals? • To what extent was the private sector engaged in the design of the internship program? • To what extent did the implementing partner and in-country partners match the market demand for interns to the training curriculum that was offered to these interns? • What percentage of the interns gained full-time employment in the same firm after completing the internship? • Which incentives are the most important to the private sector in sponsoring internships? 2 Accessibility refers to the beneficiary's ability to obtain a loan in support of a promising business idea despite the lack of collateral, documented revenues and/or credit history. Affordability refers to the beneficiary’s ability to repay a loan using revenues generated from a startup/existing business. 91 4 4. EVALUATION DESIGN AND METHODOLOGY For this evaluation, primary and secondary data collection will be conducted using a mixed-methods approach. Whenever possible, existing quantitative data such as YES-Georgia performance indicators will be utilized. Additional survey-based instruments will be developed to collect quantitative data to fill existing knowledge gaps. Qualitative data will be collected primarily through remotely conducted KIIs with USAID staff, including the Contracting Officer’s Representative (COR), Activity Managers, the implementing partner (IP), and mission staff points of contact (POCs). LEAP III has formed an ET with expertise in youth and women’s economic empowerment and monitoring and evaluation (M&E). It consists of an experienced international Team Leader who has experience in women’s empowerment, conducting rigorous performance evaluations and working in Georgia. Due to COV1D-19 restrictions, she will not travel to Georgia, but supervise and guide remotely an in-country team of experts, including a senior evaluator, a subject matter expert, and a logistician. LEAP III will provide additional administrative and operational support as needed. USE OF BEST PRACTICES The evaluation will use methods that generate quality data and credible evidence that correspond to the questions being asked, taking into consideration time, budget, and other practical considerations. The evaluation will use sound social science methods and include the following basic features: 1. Establish a team with the appropriate methodological and subject matter expertise to conduct an excellent mid-term performance evaluation; 2. Ensure transparency and dissemination of the evaluation design and final report, including briefings and presentations to the Mission Director in November 2020 and March 2021 and the posting of the final report through USAID-funded information dissemination websites; 3. Use data collection and analytic methods that ensure, to the maximum extent possible, that if a different, well-qualified evaluator were to undertake the same evaluation, he or she would arrive at the same or similar findings and conclusions; 4. Communicate and present separately the credible findings, conclusions and recommendations so the progression is clear and easy to follow in relation to each of the evaluation questions included in the Evaluation SOW; and 5. Remain vigilant and flexible to the changing environment caused by the Covid-19 pandemic that impacts travel to and within different countries. The ET will consult with USAID as it determines how to safeguard the health and safety of its team members. EVALUATION DESIGN The data collection methodology includes the following: a) document review; b) performance indicator assessments; c) electronic survey; and d) KIIs and group interviews. Data analysis methods will include refining the descriptive statistics and content analysis. The ET will utilize a mixed-methods approach, combining a mix of qualitative information collected from interviews, quantitative results of electronic surveys, and the collection of quantitative data from project monitoring and the verification of reported results to assess the success, challenges, and sustainability of YES- Georgia (Table 1). The ET will review performance indicators found in project documentation (e.g., contract agreement, work plans, annual reports) and incorporate as appropriate to address the evaluation questions. 92 5 Based on consultations with USAID and the ET’s desk review, the ET will select appropriate key informants for interviews and determine the optimal use of group interviews if feasible. 93 6 TABLE 1. GETTING TO ANSWERS MATRIX QUESTIONS EVALUATION AS APPROPRIATE) (CHECK ONE OR MORE, EVIDENCE NEEDED TYPE OF ANSWER/ E.G., RECORDS, COLLECTION, METHODS FOR DATA KIIS, SURVEYS3 SAMPLING OR SELECTION APPROACH METHODS ANALYSIS DATA : TREND DISTRIBUTIONS FREQUENCY , ANALYSIS CONTENT YES/NO SOURCES METHOD SPECIFIC S EQ 1: To what extent has the entrepreneurship training component helped receiv of trainees who did not and/or increase the incomes to start new businesses e financing through YES-Georgia’s micro equity fund (and who account for of beneficiaries)? approximately 90 percent Yes Description Surveys KIIs Interviews Indicators Performance Key Survey Online Quantitative: Interviews Qualitative: Indicators Performance Quantitative: Buzz Georgia participants All training participants (soft skills representatives on KIIs with Ministry of Education KIIs with IPs, COR, Activity Manager Crystal Consulting beneficiaries training) vocational training by sex Data disaggregated Content analysis Trend analysis Yes Comparison4 Yes Explanation5 EQ 2: What changes, if any, the life of the viability and continuity after ensure their financial Entrepreneurs Schools to model of the seven Youth are needed to the business program? Yes Description survey Online stakeholders with key Interviews collection Data Interviews Qualitative: Survey collection Data Quantitative: KIIs with Crystal MFO management, KIIs with Innovation and TBC Bank, Bank of Georgia, Georgia’s Technology Youth Entrepreneurs Nonparticipants from the same strata Mentors/coaches participants Youth Entrepreneurs Schools’ Agency, Enterprise Georgia School Manager by sex Data disaggregated Content analysis Trend analysis Cross tabulations No Comparison Yes Explanation 3 Data from evaluations are a deliverable and methods should indicate how data would be captured, i.e., for focus groups USAID requires a transcript. 4 Comparison – to baselines, plans/targets, or to other standards or norms 5 Explanation – for questions that ask “why” or about the attribution of an effect to a specific intervention (causality) 94 7 QUESTIONS EVALUATION EVIDENCE NEEDED TYPE OF ANSWER/ COLLECTION METHODS FOR DATA SAMPLING OR SELECTION APPROACH METHODS DATA ANALYSIS YES/NO SOURCES METHOD EQ 3: To what extent are the two YES-Georgia financing facilities - the Equity Facility ($75,500) ($2.5 million) and Micro Crystal MFO’s loan facility - the to the design and size of changes if any are needed beneficiaries? What respective target affordable for their accessible to and aforementioned two affordability their accessibility and amount, etc.) to increase schedule, average loan interest rates, repayment financing facilities (e.g., ? Yes Description survey Market Survey Interviews collection Data indicators Performance Quantitative: Survey Quantitative: Interviews Qualitative: collection Data Indicators Performance MFOs, KIIs with Credo Bank, Finca Bank, Unsuccessful loan equity applicants Recipients/beneficiaries Buzz Georgia participants KIIs with Crystal Representatives Bank of Georgia, TBC Bank, International donor Innovation and Technology Agency, Enterprise Georgia, Georgia’s -funded micro and SME support schemes sex Data disaggregated by Content analysis Trend analysis Yes Comparison No Explanation EQ 4: To what extent sector? the needs of the private differently to better meet have been designed discontinued in 2018) program (which was could the internship How could the private sector be encouraged to internship programs participate in similar in the goals? development/employment resources toward youth future and contribute Yes Description Survey Interviews KIIs indicators Performance Online Indicators Performance Quantitative: - Interviews Qualitative: Survey Online KIIs with IPs, COR, AOR -survey of HRHub members Youth who participated in of business companies Interviews with the representatives and PH International representatives the internship program sex Data disaggregated by Trend Analysis Content Analysis Yes Comparison No Explanation