March 2020 This publication was produced at the request of the United States Agency for International Development (USAID). It was prepared independently by the Ethiopia Performance Monitoring and Evaluation Service (EPMES) of Social Impact, Inc. EVALUATION Final Performance Evaluation of USAID/Ethiopia’s Building the Potential of Youth Activity DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. FINAL PERFORMANCE EVALUATION OF USAID/ETHIOPIA’S BUILDING THE POTENTIAL OF YOUTH ACTIVITY Final Evaluation Report March 2020 Evaluation Mechanism Number: AID-663-C-16-00010 Ethiopia Performance Monitoring and Evaluation Service (EPMES) for USAID/Ethiopia Activity Cover photo by Yohannes Leta, Social Impact, Inc. Authors/Evaluation Team Dr. James M. Statman, Team Leader Mazengia Abera Ermias Assefa Worku Ambelu Yohannes Leta Argaw Korssa i Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity ABSTRACT USAID/Ethiopia’s Building the Potential of Youth (BPY) activity was a five-year, US$20.3 million intervention implemented by a consortium of implementing partners (IPs) led by Save the Children (SC). It aimed to increase the income and economic self-sufficiency of unemployed and underemployed youth in Ethiopia. This evaluation, conducted by the Ethiopia Performance Management and Evaluation Services (EPMES) Activity, generated evidence that would help explain the performance of the BPY activity and inform future similar programs on youth economic opportunity and employment by employing a mixed-method design combining qualitative and quantitative approaches. The focus of the evaluation questions was on employment, income status, female/male targeting, TVET capacity, public-private partnerships, YES-Hubs, evidence-based adaptation, self-sufficiency and success and challenges. Overall, the evaluation found that the activity met its objectives. It effectively targeted and recruited participants, and effectively engaged and leveraged partners in the implementation process. Areas for improvement and consideration for future programming are more active gender mainstreaming in implementation of activities, diversified approaches for rural and underserved populations, and more comprehensive support for the institutional determinants of workforce development. ii Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity ACKNOWLEDGEMENTS The evaluation team expresses its great appreciation to Awoke Tilahun, Contracting Officer’s Representative for USAID’s Ethiopia Performance Evaluation Service (EPMES) Activity, for his continued guidance and review of the documents produced for the evaluation. We also thank Tahir Garo, Agreement Officer’s Representative of the Building the Potential of Youth (BPY) Activity, and Belete Deribie, Youth Workforce Development Specialist of USAID, for their technical guidance and support in the evaluation process. Genet Lemma, Chief of Party of the BPY Activity, and Abdusemed Mussa, Senior Monitoring, Evaluation, Accountability, and Learning (MEAL) Specialist for the BPY Activity, deserve enormous thanks for their support in the coordination of the field data collection and provision of secondary data and project documents. Additionally, the BPY Activity regional managers and officers at regional and woreda levels were very supportive in scheduling key informant interviews and focus group discussions. Finally, the team thanks all the key informants in different government and non-government organizations and focus group participants who provided important information to evaluate the BPY Activity. iii Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity CONTENTS List of Tables and Figures ................................................................................................ iv Acronyms............................................................................................................................ v Executive Summary ......................................................................................................... vi 1. Introduction.................................................................................................................1 Activity Background.........................................................................................................................1 Evaluation Purpose and Questions ..............................................................................................3 2. Methodology................................................................................................................4 Evaluation Design and Methods....................................................................................................4 Desk Review and Secondary Data Analyses..............................................................................7 Data Analysis.....................................................................................................................................8 Limitations.........................................................................................................................................9 3. Findings and Conclusions .........................................................................................10 EQ1: To what extent do youth participating in the BPY have improved employment (including self-employment) status? How do these outcomes vary by participant characteristics and geographic area?..................................................................................................... 10 EQ2: To what extent did the incomes of youth participating in the BPY change? How do these incomes vary by participant characteristics and geographic area?................................ 17 EQ3: What did BPY do to equitably target male and female beneficiaries?.................... 23 EQ4: To what extent have small grants provided to TVETs increased the availability, quality, and relevance (in terms of employment/income) of training for youth? ........................ 27 EQ5: To what extent and in what manner have PPPs provided quality training and/or employment opportunities for participants in the BPY activity?..................................................... 30 EQ6: What is the operational and service delivery capacity of YES-Hubs? Will the YES￾Hubs be operationally and financially sustainable post-USAID support?...................................... 32 EQ7: To what extent has the BPY activity been evidence-based to inform USAID and its development partners on the appropriateness and effectiveness of its interventions?....... 34 EQ8: Has BPY contributed to increasing participants’ self-sufficiency? ............................ 36 EQ9: What have been the most significant successes and challenges for the design, implementation, and M&E for the BPY activity?................................................................................. 37 4. Recommendations ....................................................................................................41 Annexes ............................................................................................................................43 Annex I: Additional Tables and Figures................................................................................................ 43 Annex II: PVH Case Study ....................................................................................................................... 49 Annex III: Evaluation Statement of Work ............................................................................................ 50 Annex IV: Data Collection Instruments............................................................................................... 67 Annex V: Disclosures of Conflicts of Interest .................................................................................... 86 Annex VI: Summary of Evaluation Team Members............................................................................ 91 iv Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity LIST OF TABLES AND FIGURES Table 1. BPY Activity Key Project Documents..........................................................................................8 Table 2. Summary of Sources Regarding Employment.......................................................................... 11 Table 3. Income Levels at Enrollment....................................................................................................... 17 Table 4. Summary of Sources Regarding Income ................................................................................... 17 Table 5. Income Levels at Graduation ...................................................................................................... 18 Table 6. Top Sources of Capital for BPY Participants........................................................................... 21 Table 7. Operational Status and Sustainability of YES-Hubs Visited .................................................. 33 Table 8. KII and FGD Participants .............................................................................................................. 43 Table 9. Composition of FGD Participants by Woreda ....................................................................... 44 Table 10. Estimated Increase in Baseline to Endline Employment Rate, By Region ....................... 44 Table 11. Changes in Employment Rates for All, Males, and Females............................................... 45 Table 12. Employment and Education Levels at Baseline and Endline ............................................... 46 Table 13. Employment and Years of Age ................................................................................................. 46 Table 14. Studies, Assessments, and Monitoring Data Collected by BPY ........................................ 47 Table 15. Components of BPY Evidence Dissemination and Utilization........................................... 48 Figure 1. BPY Participants by Sex and Region............................................................................................1 Figure 2. Sample Woredas for Baseline and Endline Evaluation ............................................................5 Figure 3. KIIs by Region and Sex ...................................................................................................................6 Figure 4. FGD Participation by Region and Sex.........................................................................................6 Figure 5. Changes in Employment (All)..................................................................................................... 12 Figure 6. Changes in Employment (Males)............................................................................................... 13 Figure 7. Changes in Employment (Females)........................................................................................... 13 Figure 8. Median Monthly Income by Sex ................................................................................................ 19 Figure 9. YCS Changes in Income (2017–2018) ..................................................................................... 19 Figure 10. Median Monthly Income by Region........................................................................................ 20 Figure 11. Small Grants Allocated for Each Region ............................................................................... 28 Figure 12. Map of Woredas Visited for Primary Data Collection ...................................................... 43 Figure 13. Baseline Employment at Intake ............................................................................................... 44 Figure 14. Baseline Employment by Region ............................................................................................. 45 Figure 15. Map of Woredas’ YES-Hubs.................................................................................................... 46 v Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity ACRONYMS BPY Building the Potential of Youth Activity BYOB Be Your Own Boss CA Cooperative Agreement EDC Education Development Center, Inc. EPMES Ethiopia Performance Monitoring and Evaluation Service EQ Evaluation Question FCE Facilitator for Change Ethiopia FGD Focus Group Discussion GOE Government of Ethiopia IP Implementing Partner IR Intermediate Result KII Key Informant Interview LOA Life of Activity M&E Monitoring and Evaluation MEL Monitoring, Evaluation and Learning MFI Microfinance Institution MOU Memorandum of Understanding PADet Professional Alliance for Development in Ethiopia PDP Personal Development Plan PIRS Performance Indicator Reference Sheet PPP Public-Private Partnership REST Relief Society of Tigray SC Save the Children SNNPR Southern Nations, Nationalities, and Peoples’ Region SO Strategic Objective SOW Statement of Work TOT Training of Trainers TVET Technical/Vocational Education and Training Institution USAID United States Agency for International Development YCS Youth Cohort Study YES-Hubs Youth Economic Services Centers vi Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity EXECUTIVE SUMMARY USAID/Ethiopia’s Building the Potential of Youth (BPY) activity was a five-year, US$20.3 million intervention implemented by a consortium of implementing partners (IPs) led by Save the Children (SC). It aimed to increase the income and economic self-sufficiency of unemployed and underemployed youth in Ethiopia. This third-party evaluation, conducted under the Ethiopia Performance Management and Evaluation Services (EPMES) activity, was designed to generate evidence that would help explain the performance of the BPY activity and inform future similar programs on youth economic opportunity and employment. Overall, the evaluation found that the activity met its implementation targets. It effectively targeted and recruited participants and engaged and leveraged partners in the implementation process. Areas for improvement and consideration for future programming are more active gender mainstreaming in implementation of activities, diversified approaches for rural and underserved populations, and more comprehensive support for the institutional determinants of workforce development. Activity Background From January 2015 to June 2020, BPY worked across 30 target woredas in six regions to provide 35,984 Ethiopian youth with comprehensive workforce readiness and development support. The BPY activity targeted its intervention at unemployed and underemployed youth as well as the enabling environment in which youth receive workforce development services. The bulk of the BPY intervention was soft-skills training, employment coaching, mentoring, and access to work-based learning opportunities. The BPY activity combined these services with additional, holistic support including hard-skills training through Technical and Vocational Education and Training (TVET) institutions and Youth Economic Services Centers (YES-Hubs), to be used as a networking and employment resource center, and loans and financial support. The systems-level interventions of BPY aimed to build and strengthen a sustainable local network of private and public entities engaged in youth workforce development through strengthening institutional capacity and facilitating creation of public-private partnerships (PPPs). BPY Evaluation Report Topics Covered  Employment Status  Income status  Equitable Male/ Female Targeting  Grants for TVET Capacity-Building  PPPs  YES-Hubs  Evidence-Based Adaptation  Self-Sufficiency vii Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity Evaluation Methodology The BPY performance evaluation addressed nine evaluation questions (EQs) covering the topics outlined in the textbox to the right. The team employed a practical, mixed-method design combining qualitative and quantitative approaches to generate and triangulate evidence addressing the nine EQs. The team obtained qualitative data through primary data collection using key informant interviews (KIIs) and focus group discussions (FGDs). A total of 111 KIIs and FGDs with 184 participants were conducted. Given the programmatic component of generating an evidence base, rather than collect its own data, the team conducted quantitative analyses of secondary data, and a review of key project documents. The performance evaluation utilized purposive sampling to select study areas and respondents. All 11 woredas included in the BPY Baseline Study were selected for primary data collection (see Figure 2, page 5). Findings and Conclusions Employment: Although there is significant variation among sources in estimating employment rates, it is clear that employment among BPY beneficiaries improved. The activity was effective in targeting unemployed and underemployed youth, but eligibility criteria varied by region. Whereas the intent was to serve the most in-need beneficiaries, multiple data sources point to selection of a disproportionate share of well-educated youth (such as for hard-skills training). Improved employment status held across regions, with the highest change in Southern Nations, Nationalities, and Peoples’ Region (SNNPR) and the lowest in Oromia. Males and females also both reported increased employment; males were more likely to have higher rates of employment, but females appear to have a faster change in the rate of employment. Most of the increases come from self-employment and are largely in the agricultural sector. The majority of self-employed youth are proprietors of microenterprises, which provide low levels of income, have limited capacity for resilience, and limited potential growth (given a crowded local market). Both key informants and focus group participants credited BPY’s soft-skills training in changing youths’ attitudes toward work and their futures, providing analytical and planning skills, improving direction and motivation, and reducing idleness and negative behaviors like khat usage. Ultimately, it is not possible to estimate causal impacts of BPY on employment rates, but there was, no doubt, significant contribution to youth employment. Income: Overall, sources point toward participants increasing their income while participating in the BPY. The 2018 Outcome Survey found that incomes increased across quartiles. Both males and females reported increased incomes across the datasets, although the degree of change varied in amount and by month. Among the four regions covered by the Youth Cohort Study (YCS), there was variation, but with overall positive trends, ranging from 66 percent of respondents reporting increased income in Afar to 81 percent of respondents in Tigray. Trainings from BPY helped participants to improve their employment status and, in many cases, their income. Youth also spoke about the increase in income stemming from expanded viii Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity agricultural production, business growth, and transition into wage employment. Multiple key informants noted that BPY trainings helped youth increase the profitability of their business. Across sources, BPY participants noted access to startup or working capital as a constraint to entrepreneurship. Although BPY beneficiaries invested almost 108 million Birr (US$3.3 million), 1 this relied on one’s own access to capital from financial institutions or one’s family. This presents concerns regarding equity for youth without individual or family assets in a program that emphasized entrepreneurship, rather than other kinds of opportunities (in part because of the lack of other opportunities, like wage employment, in the target woredas). Gender equity: Women face constraints on travel, availability, access to capital, and the ability to interact with mixed gender groups. The BPY activity actively worked to address these constraints. The BPY conducted training events at times and locations convenient to young women—specifically accommodating childcare and household responsibilities and limiting travel distance for women were considered widely successful. The BPY activity also addressed cultural norms that may cause hesitation among prospective female participants and their families by adding kebele-focused gender awareness campaigns, aimed at mobilizing community-level support in rural and pastoralist communities, before trainings started. Although BPY effectively targeted and recruited women into their program, there were some areas where gender mainstreaming was not incorporated. The KIIs and FGDs showed that though women and men face different situations in terms of access to capital and perceptions of loan readiness, respondents did not show that special accommodations were made to training and support to take into account differences between men and women with regards to capital access and loan provision. Further, YES-Hub participation that focused on the social and recreational services, was predominantly male. Efforts, such as the reproductive health centers, did not effectively fill the gap in services, and women were less likely to benefit from the social and recreational elements of the activity design. Technical/vocational education and training institutions (TVET) support approach: Based on SC outcome data, the TVET component of BPY is associated with higher employment outcomes for BPY participants. The BPY activity supported the capacity-building of TVET institutions so that they could potentially provide this kind of support to a broader range of participants. The limited scope of BPY’s support for these institutions, however, limited its impact to participants themselves, rather than TVET institutions more generally. There appears to be wide variation in the quality, extensiveness, and practical utility of TVET courses; many public TVETs remain seriously under-resourced and appear to often offer theory-focused training courses of dubious quality and of limited practical or market relevance. Further, lack of partnerships between public TVETs and private employers remains an obstacle to youth workforce development and private-sector growth. Partnerships: The BPY model integrated PPPs into their approach, identifying partners throughout the implementation process. This approach was likely essential to the effective implementation of the BPY model itself because it allowed BPY to build of the work of existing 1 Using 0.031 USD to 1 Birr conversion. Oanda.com. accessed 3.9.20. ix Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity institutions, networks, and structures to effectively implement their program. That said, the PPP approach is likely limited in its sustainability. Certain functions that focus on networking and require limited funds to maintain will function, but those that require BPY inputs to maintain support will cease to exist. There are more opportunities for PPPs in urban or peri-urban areas. There are fewer opportunities to support BPY or other programs like it through PPPs in rural areas. The rural/pastoralist context of the BPY activity—with limited local business development and a dearth of large-scale manufacturing and agribusinesses—restricted opportunities for sustainable private-sector linkages, constraining BPY outcomes. In the absence of significant growth of new innovative agribusiness opportunities and technologies, PPPs in pastoralist and rural locations will likely remain limited in number and scope. YES-Hubs: The services in the YES-Hubs were widely used. The centers were popular and served a number of beneficiaries. The YES-Hubs were most effective among males and participants who lived closer to the hubs themselves. Rural users and women frequented the hubs less both because of the content of the recreational activities and the travel time needed to reach the hubs. Most YES-Hubs’ ongoing functionality appears dependent on government and/or outside donor support, particularly to sustain its full array of services. Evidence Base and Adaptations: The BPY activity established a multifaceted system for generating, reporting, and acting upon evidence, and modified its design to eliminate its basic literacy and social learning for younger youth (Aflateen) interventions. This learning approach is a success of the program. That said, the data systems themselves did have severe limitations that hindered its external usability for future decision making and an evidence base for a wider population. The database structure itself did not allow for BPY to track participants over time and identify possible relationships between beneficiary characteristics, the particular sets of services received, and BPY outcomes in terms of improved employment status and income. Self-sufficiency over the long term: Overall, it is too soon to assess whether these outcomes can be considered evidence of having achieved long-term self-sufficiency, particularly in the absence of a comparison group. Given that quantitative and qualitative (FGD) data points to generally modest gains in income or assets, the achievement of self-sufficiency appears unlikely under the current intervention. Overall Successes and Challenges The BPY activity is a pilot initiative of a holistic and innovative approach to youth workforce development. This approach is consistent with USAID2 and the Government of Ethiopia (GOE) youth development and employment policy and global best practice. Thus, the successes and challenges from BPY can inform future policy and activity design for both USAID and GOE. 2 Among the positive youth development principles BPY shares with USAID Youth Policy (2012) are: youth participation; building youth resilience; taking into account differences and similarities; supporting and engaging mentors, families, and communities; gender equity; and, to some extent, technology and innovation. x Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity SUCCESSES CHALLENGES • Soft-skills training that focused on workforce readiness and planning as well as work-based, practical learning techniques were well received. • Decentralized training and adjusted training times to ensure participation of women and rural groups. • Leveraged partnerships to expand reach and influence and promote initial levels of programmatic sustainability. • Proactive and sensitive community engagement was a driver of its success and was essential to efforts in recruitment, localized design, and gender mainstreaming. • Used evidence they generated to make programmatic adjustments during implementation. • Evidence was shared and disseminated with stakeholders. ∅ Beneficiaries identified the lack of direct job linkages and loan access and preparedness as gaps in BPY services. ∅ The activity was not able to go in depth into PPP development, YES-Hub sustainability, and TVET support. ∅ YES-Hubs and TVET institutions were difficult for rural/pastoral communities to access. ∅ PPPs were especially difficult to develop in rural/pastoral areas where the base level of private-enterprise development was either low or based on microenterprises. ∅ Prevailing gender dynamics and controls on women’s access to capital, perceptions around loan readiness, movement, and time, affected every element of the intervention. ∅ Local government and community organizations priorities did not always align with BPY priorities, specifically making it more difficult to target the most disadvantaged youth. ∅ Evidence base had data quality issues that limited its widespread use and effectiveness. Recommendations Based on these findings and conclusions, the evaluation team established the following recommendations. These are designed to be used by IPs, donors, and government entities that are designing, implementing, and reporting on workforce development activities. • Building on its demonstrated success across all regions in BPY and strong support by beneficiaries, program personnel, and local stakeholders, soft-skills training should be a core component of future youth workforce/livelihood development intervention in Ethiopia. • Design different approaches for urban, peri-urban, and rural communities. The BPYs market-driven approach works best when there are larger private-sector businesses to partner with and where wage-employment opportunities would likely provide greater value for money. In rural/pastoral contexts, entrepreneurial training and soft-skills training should focus on agricultural value-chain opportunities and creating and managing innovative microenterprises. • Develop targeting programming or supplementary programming to support the most underserved, impoverished, and marginalized segments of youth, including youth with disabilities, youth from the most economically disadvantaged communities, youth with minimal education, and youth ex-offenders. • Support further surveys, assessing the training capacity, resources, training quality, market linkages, and outcomes of all registered TVETs. Stronger data xi Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity and information would become an important tool for youth workforce program planning, management, and assessment, and would support cost-effective linkages with TVETs that have a documented success in market-relevant training delivery. • Integrate elements of BPY’s recruitment, targeting, and decentralized training, and support strategies as benchmarks for future programs. Build off of BPY effectively engaged rural communities and women in programs. • Integrate gender mainstreaming efforts into critical support on loan provision and the recreational and networking elements of the activity. Take into account the differential perceptions of men and women in terms of loans. Make specific accommodations for women’s lack of access to capital and the negative perceptions of loan repayment among young men. For recreational elements and networking, provide opportunities for women’s recreational networking. • Provide implementers tasked with developing an evidence base with third-party data quality assessments and support. This will help USAID understand the strengths and weaknesses of the data it is reviewing to make decisions. 1 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity 1. INTRODUCTION Activity Background USAID/Ethiopia’s Building the Potential of Youth (BPY) activity was a $20.3 million intervention implemented over five years between January 2015 to June 2020 (cooperative agreement AID￾663-A-15-00006). The activity was implemented by a consortium of implementing partners (IPs) led by Save the Children (SC) and including the Education Development Center, Inc. (EDC) and Ethiopian IPs HUNDEE-Oromo Grassroots Development Initiative, Professional Alliance for Development in Ethiopia (PADet), Relief Society of Tigray (REST), and Facilitator for Change Ethiopia (FCE).3 Commencing in January 2015, BPY aimed to: provide Ethiopian youth (ages 15–29) in rural areas and towns with access to workforce development/livelihood support and resources, tailored to their specific needs and market demand, so that they can (a) achieve increased income and (b) strengthen skills, knowledge, and social capital required to achieve economic self-sufficiency over the longer term. 4 FIGURE 1. BPY PARTICIPANTS BY SEX AND REGION The BPY activity was implemented in 30 woredas in six regions: Afar, Amhara, Oromia, Southern Nations, Nationalities, and Peoples’ Region (SNNPR), Somali, and Tigray (see map in Annex I, Figure 12). The activity exceeded their target (34,537 youth) to work with a total of 35,984 female and male unemployed or underemployed youth. The distribution of BPY participants by region and gender is presented in Figure 1. The BPY activity’s intervention model was designed to integrate a set of “soft-skills” group training interventions to enhance youths’ attitudes and motivation toward gaining self or wage￾employment, assisting these young people in gaining greater self-awareness and knowledge of 3 In October 2019, USAID announced that it had added funds and modified the cooperative agreement (CA) to focus on building market-driven linkages with technical/vocational education and training institutions (TVETs), local universities, and polytechnics, and adding Brandeis University to the SC-led consortium. These additional activities are not included in the Performance Evaluation SOW and are not addressed in this report. 4 USAID/Ethiopia. Statement of Work, Performance Evaluation of USAID’s BPY Activity, 2019 p. 3 2 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity the economic opportunities in their local environment, and in developing a personal development plan (PDP) enabling them to find new or better employment. The soft-skills training modules, initially created by EDC and adapted and translated for the target regions, are: • Work Ready Now!, focusing on interpersonal communication, work habits and appropriate behavior, leadership and teamwork, and creation of an individual PDP; • Work Ready Now Plus, similar to Work Ready Now! but designed for youth with minimal literacy skills; • Positive Youth Development, focusing on building positive attitudes and career ambitions, planning and saving, and analyzing the local labor market environment; • Be Your Own Boss (BYOB), addressing the opportunities, requirements, challenges, and potential rewards of creating an individual or group startup microenterprise. 5 Of the 35,272 youth who participated in BPY over the five-year life of activity (LOA), all (35,272) took the Work Ready Now! (31,225) or Work Ready Now Plus training (a similar training module designed for youth with minimal literacy skills), which enrolled 4,047 participants. Approximately two thirds of the BPY participants took the Positive Youth Development training module (23,859), and slightly more than one third received the BYOB course (13,424). The BPY monitoring data (2019) reports delivering a total of 2,823,211 person-hours of training over the LOA. 6 A small subset of BPY beneficiaries (2,668, approximately 7 percent) were also supported to enroll in a short course (four months or less) of hard-skills training at TVET institutions. Participants were further supported through a series of post-training services, including coaching from BPY staff or volunteers, mentoring services from local business people including peer mentors, and access to work-based learning opportunities, including job shadowing, internships, and apprenticeships. The BPY monitoring data indicate that about 77 percent of participants (27,858) received post-training coaching and 21 percent (7,682) mentoring services. Approximately 18 percent of activity participants (6,465) engaged in some form of work-based learning, primarily job shadowing. And reflecting the dearth of large employers and major industrial enterprises, only 0.3 percent (118) were placed in formal apprenticeships. The BPY model also included systems-level interventions aimed at building and strengthening a sustainable local network of private and public entities and institutions engaged in youth workforce development. These institutions focused on strengthening institutional capacity and facilitating the creation of public-private partnerships linking public and private education and training institutions, local government offices, lending institutions, and private-sector employers. 5 Although in practice there were regional variations, Work Ready Now! was organized as a five-week course at 15 hours per week (total 75 hours), and Positive Youth Development and BYOB training modules were five weeks at five hours per week each in duration (total of 25 hours per module). 6 With the exception of the TVET hard-skills training, the BPY database did not track employment status and income results by the particular set of BPY interventions received. 3 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity The activity also mobilized communities, creating local stakeholder councils and committees composed of community leaders, government officials, civil society representatives, and youth to select BPY participants and identify and address opportunities and challenges for youth livelihood development. Furthermore, BPY worked with local governments to renovate and revitalize dormant youth centers, transforming these into active hubs for soft-skills training, employment coaching and mentoring, and peer support. Centers also provided access to photocopying, computers and computer training, and entertainment and recreational facilities. The BPY model also included strong commitments to monitoring and evaluation (M&E), research, and evidence-based learning and adaptation; to gender equity in participation and outcomes; and to strong and genuine youth participation and leadership in the design, conduct, and evaluation of the activity. The full results framework of BPY outlining how the component interventions were designed to support the strategic objective (SO) and achieve intermediate results (IRs); see Annex IV for the full results framework. Evaluation Purpose and Questions The purpose of this final performance evaluation was to generate evidence that will help elucidate the performance of the BPY activity to inform future similar programs on youth economic opportunity/employment. Evaluation design began in August 2019, with qualitative data collection between October 2019 and November 2019. The evaluation addressed nine evaluation questions (EQs) (see Annex IV for detailed EQs). For each EQ, the team answers the question and also describes differential effects across participant characteristics and the geographic areas. • EQ1: Employment status - To what extent do youth participating in the BPY have improved employment (including self-employment) status? • EQ2: Income status - To what extent did the incomes of youth participating in the BPY change? • EQ3: Equitable targeting - What did BPY do to equitably target male and female beneficiaries? • EQ4: TVET capacity-building - To what extent have small grants provided to TVETs increased the availability, quality, and relevance of training for youth? • EQ5: Public-private partnerships (PPPs) - To what extent and in what manner have PPPs provided quality training and/or employment opportunities for BPY participants? • EQ6: YES-Hubs - What is the operational and service delivery capacity of YES-Hubs? Will the YES-Hubs be operationally and financially sustainable post-USAID support? • EQ7: Evidence-based adaptation - To what extent has the BPY activity been evidence￾based to inform USAID and its development partners on the appropriateness and effectiveness of its interventions? • EQ8: Self-sufficiency - Has BPY contributed to increasing participants’ self-sufficiency? • EQ9: Success and challenges - What have been the most significant successes and challenges for the design, implementation, and M&E for the BPY activity? 4 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity 2. METHODOLOGY Evaluation Design and Methods The BPY Performance Evaluation employed a mixed-method design combining qualitative and quantitative approaches to generate and triangulate evidence addressing the nine EQs. This approach obtained qualitative information through primary data collection using key informant interviews (KIIs) and focus group discussions (FGDs), quantitative analyses of secondary data, and a review of key activity documents. This combination of overlapping data collection and analytical approaches generated sufficient information to describe results against performance targets and programmatic objectives. Primary Data Collection The primary data collection conducted under this evaluation was qualitative and consisted of KIIs and FGDs. The respondents were primarily BPY participants, but interviews were also conducted with BPY implementer staff, partners, and other stakeholders. To maintain rigor in its qualitative research, the team established a systematic sampling approach that ensured qualitative data was pulled from a wide subsection of respondents. This sampling took place in three key areas: Woreda selection, KII selection, and FGD selection. Woreda Selection The performance evaluation utilized purposive sampling of 13 woredas to select study areas and respondents. As requested by USAID/Ethiopia, the team selected the 11 woredas included in the BPY Baseline Study and two non-baseline woredas. The additional, non-baseline woredas were included to improve the representation of the BPY sites in the Oromia region, which was underrepresented during the baseline due to insecurity at the time. Oromia (nine woredas) and Amhara (six woredas) together accounted for half of the 30 BPY woredas. All six regions in which BPY was implemented were primary data collection sites for the evaluation. The breakdown of sampling is shown in Figure 2 on the following page. 5 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity FIGURE 2. SAMPLE WOREDAS FOR BASELINE AND ENDLINE EVALUATION KII Selection The evaluation team conducted KIIs with the IP staff, their partners, and other stakeholders involved in the BPY activity. This included local government officials, TVETs, IP representatives, microfinance institutions (MFIs), and other organizations in the selected woredas. Key informants were identified based on their organizations’ involvement in the BPY activity. In consultation with the IP, a list of potential key informants or organizations was compiled. This listing enabled the evaluation team to determine how many KIIs would be run in each woreda and in the regional cities. Once the number of KIIs was determined and the organizations were selected, individuals for the KIIs from the organizations were purposefully identified by BPY based on their direct participation in BPY’s implementation process. The evaluation conducted a total of 111 KIIs with an array of stakeholders, including government officials, employers, training providers, and IPs at federal (Addis Ababa), regional, and woreda levels. The number of KIIs by region is shown in Figure 3. 6 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity FIGURE 3. KIIS BY REGION AND SEX FGD Selection The team conducted focus FDGs with BPY beneficiaries to learn more about their experience with BPY and their wider constraints and opportunities that supported their successes or challenges in the program. A total of 16 FGDs were conducted in eight of the 11 baseline woredas: Ada’a, Asayta, Azernet Berbere, Bahir Dar Zurya, Debub Achefer, Wolmera, Jijiga, and Raya Alamata. It was not possible to conduct FGDs in all woredas because of practical limitations of time, team level-of-effort, and travel logistics. The three woredas not selected for FGDs were either so remote as to require the addition of several days to the fieldwork schedule and/or appeared duplicative of woredas that were included in terms of geographic, socioeconomic, and demographic characteristics. The eight FGDs woredas were purposively selected to include all those that reported having established Youth Economic Services Centers (YES-Hubs) and/or have worked with MFIs to provide loans to BPY participants so that the evaluation could obtain beneficiary-level qualitative data on these program components, which are foci of specific EQs. FIGURE 4. FGD PARTICIPATION BY REGION AND SEX To capture possible variation within each woreda, the evaluation team conducted FDGs in two kebeles in each woreda. Whenever possible, one semi-urban and one rural kebele were 7 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity selected. In target woredas containing only rural kebeles, two were selected in different geographic areas of the woreda. The FGDs were conducted with BPY participants, who were selected with assistance from BPY staff based on beneficiary characteristics, including gender, program phase, location, age at enrollment, and services received. The evaluation team conducted FGDs in mixed male and female groups. Because of time and logistics constraints, the team determined that, given the content of many of the questions, men and women would feel comfortable discussing these topics in mixed groups. The number of FDG participants by region and sex is presented in Figure 4 and in Annex I, Table 8; FGD participation by woreda is presented in Annex I, Table 9. Desk Review and Secondary Data Analyses Through the course of the BPY activity, significant research was done on its work. This included research done externally, as well as research done by the implementer as part of the evidence component of the BPY activity itself. To promote efficiency, the evaluation team built on this existing body of work to inform this final evaluation. Key project documents were reviewed to obtain evidence on the performance of the BPY activity and provide contextual information on its implementation. Quantitative datasets were repurposed to provide quantitative data that allowed the evaluation team to compliment the qualitative data with data that is formally representative of a wider sample. These datasets were reanalyzed as warranted to provide descriptive statistics addressing specific evaluation questions and themes. A list of key documents and their characteristics are included in Table 1. The team also analyzed secondary quantitative data from sources including the BPY activity monitoring, evaluation, and learning (MEL) reports, quarterly and annual reports, enrollment data, and outcome survey data. 8 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity TABLE 1. BPY ACTIVITY KEY PROJECT DOCUMENTS DATA SOURCE ORIGIN AND YEAR GEOGRAPHIC COVERAGE SAMPLING APPROACH SCOPE SC: BPY Outcome Survey December 2017 to February 2018 All the six BPY implementation regions, 11 woredas out of the 30 Sample of all beneficiaries who participated up to June 2017. Stratified sampling to identify agrarian and pastoralist woredas, and male and female participants in woredas (2,088 total). Employment outcome, income, including productive assets, workforce readiness skills, coaching, opportunities to access resources. SC: Youth Profile Dataset January 2016 to July 2019 All BPY regions No sampling. All beneficiaries covered. Demographic data of all beneficiaries, employment status, income, loan, training before participating in BPY. SC: BPY Monitoring Dataset 2016 to 2019 (up to Y5 Q3) All regions and all BPY regions No sampling. All beneficiaries covered. Output monitoring for program: Total reach, trainings youth received, post training services, youth employment. Social Impact (Third-Party Researcher): Youth Cohort Study (YCS) 2017 to 2018 BPY participant in Tigray, Amhara, Oromiya, Afar 582 beneficiaries from 13 selected woredas who started training as a cohort. Longitudinal study over 1 year looking at income and employment status, work readiness skills, access to employment/entrepreneurial resources, functional literacy and numeracy skills. Social Impact: Youth Financial Survey June 2017 to January 2018 Subsample of the youth cohort Subsample of all beneficiaries in one kebele per woreda per region. Measures the income and expenditure of youth within the YCS time. Data Analysis The team used a combination of quantitative and qualitative methods to analyze the data and identify trends. The team acquired full datasets for intake, Outcome Survey, and monitoring data, and performed descriptive analysis to answer the EQs in Stata. In several cases, datasets required cleaning to facilitate this analysis. Cleaning and analytical protocols are reproducible and can be shared with USAID upon request. The data was disaggregated by key demographic variables, including sex, region, age, education attainment, and employment status. Qualitative data derived from document reviews, KIIs, and FGDs were coded into categories and classifications, and the team analyzed the emerging patterns and themes for each EQ. 9 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity Limitations 1. Data quality of secondary datasets: Only the FGDs, KIIs, Youth Cohort Study (YCS), and Youth Financial Survey were conducted by the independent evaluator itself. All other research was done internally by SC. The limitations here are twofold. First, the data was not independently collected or verified. Although the evaluation team can use the data, it cannot verify its quality of this data. During the data analysis period, the evaluation team identified inconsistencies with the data itself. First, cleaning performed by the team evidenced data entry errors. Though no formal assessment of the extent of this issue was performed, there was a discrepancy between the number of observations in the intake data and monitoring data – as elaborated later in the report. Further, since intake forms were administered at the local level there may have been significant heterogeneity of administration (as opposed to a centrally managed exercise) This casts some doubt on quality and/or representativeness of either data source though it is not possible to determine the full scope of this limitation. In cases where inconsistencies were significant, the evaluation team did not use the data. In cases where inconsistencies and data quality issues were more minor, the team noted the type of quality issue and its implications for the findings. Second, because the data was collected primarily for monitoring and reporting purposes, the data itself lacked key measures in the structure to allow for certain kinds of statistical analysis, blending with other studies, and disaggregation needed to follow individual participants over time. 2. Precision for financial data: The evaluation team found that many participants were either reluctant to share income data or that they could not correctly remember income data. 3. Scope of monitoring data: The BPY monitoring dataset could not support the quantitative linkage of specific arrays of services received by participants—such as particular soft-skills training received, mentoring, coaching, job shadowing, or YES-Hub participation—with subsequent employment status. Triangulation utilizing qualitative inputs from KIIs and FGDs helped ameliorate this gap. 10 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity 3. FINDINGS AND CONCLUSIONS EQ1: To what extent do youth participating in the BPY have improved employment (including self-employment) status? How do these outcomes vary by participant characteristics and geographic area? Employment Status at Enrollment The BPY activity’s youth profile data 7 indicate that at intake, 76.9 percent of all program participants were unemployed, 21.8 percent were self-employed, and 1.3 percent were wage￾employed. When asked whether employed participants considered themselves underemployed, 55.8 percent responded in the affirmative (55.3 percent of self-employed and 65.2 percent of wage-employed). Viewed another way, only 10.2 percent of beneficiaries considered themselves to be fully employed at the time of intake. Sex. Disaggregating this data by sex shows a significantly higher unemployment rate among female beneficiaries (84.3 percent for females and 70.7 percent for males), though employed females were slightly less likely to report being underemployed as compared with their male counterparts. The full employment rate among males was almost twice as large as that of females (12.9 and 6.9 percent, respectively). Region. The baseline employment situation among enrollees differed markedly across regions. The unemployment rate ranged from 90.5 percent (Afar) to 62.2 percent (Oromia). The self￾employment rate ranged from 36.8 percent (Oromia) to 5.3 percent (Afar). The wage employment rate ranged from 4.2 percent (Afar) to 0.4 percent (SNNPR). Full employment was in the single digits in all regions except Oromia and Somali (20.8 and 12.2 percent, respectively). Females were more likely to be unemployed than males in all six regions. The employment status of BPY participants at program intake, disaggregated by sex and region, is presented in Annex 1, Figures 13 and 14. Because of different beneficiary selection criteria across regions, populations differ significantly across the six regions along multiple dimensions 8 including sex, education, age, and underemployment among employed beneficiaries, which ranged from 45 percent (Oromia) to 72.3 percent (SNNPR). 7 There were significant differences between the number of respondents in the intake dataset and the figures reported in BPY’s monitoring data. Excluding the 180 respondents with missing or incorrect regional information in the intake dataset, the monitoring data had 3,295 (or 10 percent more) more observations than the intake dataset. If anything, we would expect more observations in the intake dataset because the monitoring data only tracks the number of youth completing program components (as opposed to enrollment). The discrepancy varied across regions, with Amhara having 204 more observations (3 percent difference) in the intake, while Somali had 1,637 fewer (37 percent difference) observations in the intake. This is important because we use intake data as a baseline. If missing observations differed substantively from the rest of the intake respondents, we may be misrepresenting the pre-BPY situation. 8 It is important to note that differences in baseline employment should not be interpreted to signify real differences in regional employment rates because selection of youth was not done with representativeness in mind. 11 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity Employment Status at Graduation This section present postgraduation employment. In interpreting regional changes, it is important to keep this baseline heterogeneity in mind and not attribute differences in rates of change to regional factors (as opposed to differential beneficiary composition and different selection criteria). The evaluation team used three secondary sources to assess changes in employment. Employment trends are reported separately before summarizing the three sources in Table 2. TABLE 2. SUMMARY OF SOURCES REGARDING EMPLOYMENT BPY MONITORING YCS 2018 OUTCOME SURVEY BASELINE • 23 percent employed (29 percent of males; 16 percent of females) 9 • 48 percent employed (54 percent of males; 40 percent of females) 10 • 23 percent employed (29 percent of males; 16 percent of females) 11 OVERALL CHANGES • 51.6 percentage point increase in self-reported employment rate • 7.5 percentage point increase in self-reported improvement in their employment situation after participation • 21 percentage point increase in employment rate at endline • Self-employment (41 percent) was almost six times more prevalent than wage employment (6.9 percent) at baseline and increased significantly 12 • 58 percentage point increase in employment rate at endline • Self-employment was eight times more prevalent than wage employment (66 and 8.3 percent, respectively) and explained most employment growth REGION • All six regions reported increases from baseline to endline • Employment rates ranged from 36 percent in Afar to 73 percent in Tigray • All four regions reported increases from baseline to endline • Largest increase: Tigray (192 percentage point) • Smallest increase: Oromia (13 percentage point) • All six regions reported increases from baseline to endline • Largest increase: Afar (79 percent) • Smallest increase: Oromia (17 percentage point) SEX • Female employment increased faster than male employment (360% increase as compared to 210%) • Baseline inequity was almost entirely eliminated by endline. • No substantive difference in employment rate change between the sexes. • Female employment increased faster than male employment (431% increase as compared to 269%) • Baseline inequity was almost largely eliminated by endline. Although there is significant variation among these sources in terms of estimating both baseline and endline employment rates, it is clear that employment among BPY beneficiaries improved in 9 Intake data 10 YCS baseline 11 Intake data 12 Relative to baseline, self-employment increased by 23.9 percentage points at midline and 19.5 percentage points by endline. The comparable figures for self-employment were 1.9 and 1.6 percentage points, respectively. 12 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity the aggregate. Comparing the three datasets reveals significant differences in estimates of beneficiary employment rates. The YCS estimated significantly higher baseline employment rate, more than two times that of the intake data (which were used as comparisons for both monitoring and the Outcome Survey). Estimates of endline employment were also imprecise, with a 22-percentage point spread across the three sources. The 2018 Outcome Survey estimated the highest employment rate (74 percent), followed by the YCS (69 percent), and the monitoring data (59 percent). Given the range of point estimates, the calculated differences between baseline and endline also varied significantly, ranging from a 144 percent increase (YCS) to a 322 percent increase (2019 Outcome Survey). Region. Estimated employment rates increased in all regions across all three data sources; see data in Annex 1, Table 10. The variability in rate estimates across sources discussed above held in regional analysis, as well. Furthermore, there was large variation in changes across regions, with Afar and SNNPR registering the largest increases and Oromia and Somali the smallest. In Ethiopia as a whole the unemployment rate remained consistent during the implementation period. 13 Sex. Across the three sources, males had consistently higher rates of employment at baseline and endline. Importantly, however, the rate of change in employment status was significantly greater among females in two of the three data sources (monitoring and 2017 Outcome Survey). In fact, monitoring data suggest females were approaching employment parity with males at endline. Summary graphs are presented in Figure 5, Figure 6, and Figure 7. For specific data, see Annex 1, Table 11. FIGURE 5. CHANGES IN EMPLOYMENT (ALL) 13 The World Bank Group. Data Bank. visited March 13, 2020 https://data.worldbank.org/indicator/SL.UEM.TOTL.ZS?end=2019&locations=ET&start=2015 13 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity FIGURE 6. CHANGES IN EMPLOYMENT (MALES) FIGURE 7. CHANGES IN EMPLOYMENT (FEMALES) Education. As is often the case, education was strongly associated with employment (see Annex I, Table 12). At baseline, youth with no education had the highest employment rate (34.9 percent). The largest changes at endline were measured for youth with TVET and primary education, with the former likely influenced by youth receiving TVET instruction through BPY. Conversely, the smallest change was measured among youth with no education, suggesting that outcomes may have accrued disproportionately to more educated youth. Age. Last, data from the 2019 YCS indicates a strong positive relationship between age and employment, both at baseline and endline (see Annex 1, Table 13). Employment changes were not substantively different across the four age cohorts, with the lowest rate of increase among youth aged 27+ and the largest among those in the 18–21 age bracket (increase of 245 and 298 percent, respectively). The 27+ age group was likely dampened by already high employment rates, as demonstrated in an endline rate of 92.6 percent. 14 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity Employment Type As noted across the YCS and the 2018 Outcome Survey, self-employment was the most prevalent and accounted for the greatest amount of employment increases under BPY. Given the rural and agrarian or pastoralist nature of BPY regions, the majority of employment was in the agricultural sector: 45.5 percent of 2018 Outcome Survey respondents reported working primarily in agriculture, with a further 27.2 percent engaged in trading, much of it involving agricultural inputs or outputs. The balance worked in services (17 percent) government (3.5 percent) and assorted others. Disaggregating by sex reveals that agricultural employment was by far the most prevalent among both males and females, though males were 11 percentage points more likely to be employed in the sector (61.2 and 50.3 percent, respectively). Conversely, females were almost five times more likely to be employed in services than males (e.g., hairdressing, tailoring, petty trading, driving, and mobile maintenance) in the more urban/semi-urban woredas. Across all regions, employment choices differ by sex. Men engaged in activities including animal fattening, irrigation and vegetable farming, carpentry, day-labor, construction, transport services, loading/unloading, and, for a few, government employment. Women pursued occupations including hairdressing, petty trading, retail shops, tea and coffee sales, restaurant work, tailoring, poultry, and, where available, textile factory work. BPY Interventions and Improved Employment Status Soft-skills training. Qualitative data from KIIs and FGDs showed virtual unanimity in supporting the benefits of BPY’s soft-skills training packages. Various respondents credited the training in changing youths’ attitudes toward work and their futures, providing analytical and planning skills, improving direction and motivation, and reducing idleness and negative behaviors like khat usage. Woreda and regional-level KIIs with program staff and local officials provided insight into how they understood the critical role of the soft-skills component in initiating a process of attitudinal and behavior change in the BPY beneficiaries. A local official in Somali said, “Attitude is key. Prior to BPY, youth did not believe in themselves and looked to government to support them. Now they have the freedom to take opportunities.” A regional IP representative (SNNPR) similarly stated that the soft-skills training “completely changed” the attitudes of the youth, who were previously “sitting around doing nothing [but] are now working.” In addition to changing attitudes and building, soft-skills training appears to have provided youth the concrete analytical skills and knowledge for creating microbusiness opportunities aligned with their local economies. The head of an MFI stated the soft-skills training “taught BPY youth how to make their businesses profitable,” compared with their non-BPY peers, and that youth were better able to save and repay their loans. A local representative of the Office of Women, Children, and Youth Affairs similarly credited the soft-skills training with helping youth to “build awareness” of business opportunities and to “become profitable.” Participants of FGDs echoed the positive feedback on soft-skills training, revealing that many youth found the Work Ready Now! training to be the most useful training package, followed by 15 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity BYOB. The specific soft skills these beneficiaries found most valuable focused on how to find employment, savings, customer relations and interpersonal relations, and communications skills. As one formerly unemployed graduate remarked, “Three weeks of soft-skills training were more important to my employment than all I learned in three years at university.” Other participants spoke of how they felt “awakened” or had their “eyes opened” by the training, and several spoke of changing attitudes toward migrating internationally for work opportunities. Another male participant, a formerly unemployed university graduate in Oromia, stated that he had once concluded that all he could aim to become was “the best thief” in his area, but was now happily engaged as a successful small business owner. Hard-skills training. A total of 2,668 BPY beneficiaries (7.4 percent) received hard-skills training at public or private TVETs. In a few instances, when no appropriate TVET was available, participants received training from local government offices. Although hard-skills training was envisioned to be offered to those participants who had the poorest level of skills and qualifications, quantitative and qualitative data indicate that BPY tended to target better educated youth. The 2018 Outcome Survey indicates that 24.1 percent of youth with tertiary education received hard-skill training. Comparable figures for secondary, primary, and no education were 16.9, 5.7, and 2.1 percent, respectively. An example of the rationale for this phenomenon was found by the evaluation team in Somali, where training at the polytechnic was provided to youth with the highest educational credentials because of perceptions that less-educated participants would not succeed in the academic atmosphere of the TVET. According to the 2018 Outcome Survey, hard-skills training has not translated into employment gains. In fact, recipients of this support were more than twice as likely to be unemployed as compared with nonrecipients at endline. Conclusions Targeting. According to BPY intake data, the activity did a good job targeting beneficiaries on baseline employment status: 76.9 percent of beneficiaries were unemployed, and only 10.2 percent described themselves as fully employed at baseline. However, the YCS presents a very different picture of baseline employment. According to that source, unemployment was only 52 percent. Furthermore, the eligibility criteria were sufficiently broad that there were large-scale differences in beneficiary profiles across the six regions. This flexibility could serve local programmatic needs, but it also raises questions of equity. This concern was further underscored by selection of beneficiaries into hard-skills training activities. Whereas the intent was to serve the most in-need beneficiaries, multiple data sources point to selection of a disproportionate share of well-educated youth. Employment. Estimates of endline employment differed significantly across the three sources analyzed, ranging from a 144- to 351-percent increase. Although it is difficult to estimate the population parameter precisely, we are confident in concluding that BPY youth were significantly more likely to be employed after participating in the program than before. Although both the number of employed youth and the employment rate among beneficiaries well surpassed performance targets, and qualitative data speak strongly to attitudinal changes in 16 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity beneficiaries, employment increases are not necessarily attributable to BPY. 14 Ultimately, it is not possible to estimate causal impact of BPY on employment rates, but there was, no doubt, significant contribution to youth employment. Attitudinal changes. Support through BPY was effective in creating a positive change in beneficiaries’ attitudes toward work, particularly self-employment, their ability to identify market-related opportunities, and youths’ self-confidence and optimism. In FGDs, BPY youth spoke to a wide variety of benefits stemming from their participation. By far the most commonly cited outcome was a positive attitudinal change. Although direct postgraduation employment benefits may not persist over the long term (e.g., changing jobs), these attitudinal changes have a higher likelihood of leading to long-standing, sustainable change. Whether these outcomes translate into employment and increased income is a function of local labor markets. Employment type. The most common employment among BPY beneficiaries was self￾employment in the agricultural sector, with smaller shares of youth employed in trading and service jobs (the latter especially among females). Given the agrarian and pastoral targeting, self￾employment remains the most viable option for most youth, given the lack of wage employment opportunities. Most self-employed youth are proprietors of microenterprises, which provide low levels of income, and have limited capacity for resilience, and limited potential growth (given a crowded local market). Sex. Females were less likely to be employed at baseline but had faster rates of employment increase by endline (in two of three quantitative sources). This suggests that BPY may have contributed to leveling of employment outcomes between males and females. In fact, according to the monitoring data, endline employment rates for males and females were almost at parity. Distributional effects. Although all population groups appeared to benefit from participating in BPY, there is some evidence that certain “vulnerable and disadvantaged” groups (youngest age groups, those with less education) were less likely to benefit from participation, presenting questions about equity as highlighted in the activity design. 14 It may be the case that idle youth were more likely to participate in BPY because of an abundance of free time, and although their post-training employment increased, they may have sought and attained employment without the training. This phenomenon, referred to as regression to the mean, has been well documented in several impact evaluations of workforce development programs. 17 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity EQ2: To what extent did the incomes of youth participating in the BPY change? How do these incomes vary by participant characteristics and geographic area?15 The evaluation confronted significant operational and methodological challenges in assessing change of participant income. 16 In addressing EQ2, the evaluation relied primarily on BPY intake data and YCS (including monthly financial survey data). Analysis also used primary qualitative data collected by the evaluation team, along with SC’s 2017–2018 Outcome Survey data, to describe and explain the income status of BPY participants. Income Level at Enrollment The BPY activity’s youth profile data indicate that the median self-reported income at intake was 912.6 Birr per month (equivalent to US$28.29). 17 Surprisingly, the median income was the same for males and females across the 7,530 observations. When the data are broken down by quartile, however, females had slightly higher incomes in all but the lowest quartile, as seen in Table 3. 18 Of respondents, 70.3 percent reported making less than US$2 per day. TABLE 3. INCOME LEVELS AT ENROLLMENT QUARTILE TOTAL MALE FEMALE First 151.2 152.0 149.4 Second 635.7 633.5 642.6 Third 1,514.5 1,502.8 1,542.6 Forth 39,226.9 38,708.6 40,292.6 Income Level at Graduation We report changes to income using two sources below: the YCS overall sample/subsample and SC’s 2018 Outcome Survey. 19 See Table 4 for more data. Across both sources, there is clear evidence that participants across demographic groups increased their income while participating in BPY. Across both studies, men were more likely to report a higher income than women. TABLE 4. SUMMARY OF SOURCES REGARDING INCOME YCS 2018 OUTCOME SURVEY OVERALL 15 Please note that with USAID concurrence, key evaluation themes explicitly addressed under EQ2 include an examination of sources of capital/income generation and of loan mechanisms utilized by youth for business startup and/or expansion. These areas are seen by USAID and the evaluation team to be a critical context for understanding the opportunities and challenges of possible increases in income for BPY participants. 16 As FGDs demonstrated, within rural and small-town communities and in group discussions, participants were reluctant to share specific details about their income with their peers, and the validity of such reported income data must be regarded as questionable. 17 Using US$0.031 to 1 Birr conversion, Oanda.com, accessed 3/9/20. 18 The fourth quartile is heavily skewed by a number of outliers (e.g., the highest self-reported monthly income was 1,740,000 Birr). These have not been trimmed for this analysis. 19 BPY monitoring data did not include income. 18 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity • 71 percent reported increase in income • 52 percent of respondents reported a baseline income of 0 Birr per month, and 75 percent of respondents reported a baseline income of 500 Birr per month or less • The median income increased from 0 at baseline to 1000 at endline. SEX • Males are more likely to report increased income than females (76 percent and 66 percent, respectively) • In financial diaries, females had modestly increasing and relatively consistent median incomes across months. Males reported one higher month (August), with median incomes lower than females the other seven months (Figure 8) • Male respondents had higher self-reported income at both baseline and endline (median 100 at baseline and 1,200 at endline), as compared with females (median 0 at baseline and 845 at endline). • Male income increased faster then female income Female mean income increased faster than male. Overall, sources point toward participants increasing their income while participating in BPY. The 2018 Outcome Survey also found that income increased across quartiles. Although 52 percent of survey respondents reported a baseline income of 0 Birr per month, both quartiles reported an increase over time while participating in BPY, as seen in Table 5. TABLE 5. INCOME LEVELS AT GRADUATION QUARTILE BASELINE ENDLINE DIFFERENCE First 0 231.4 231.4 Second 0 839.0 839.0 Third 291.7 1,602.1 1,310.4 Fourth 1,500.7 4,047.4 2,546.7 Sex. The data sources present different information regarding males and females. Although both sexes were likely to report increased income across datasets, the degree of change varies. In particular, the YCS financial diaries point to the importance of seasonality; while males may have higher income for a short period, females have more consistent income, potentially connected to the different frequency in which males and females participate in agricultural activities (see Figure 8). 19 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity FIGURE 8. MEDIAN MONTHLY INCOME BY SEX Region. Among the four regions covered by YCS, there was variation but with overall positive trends, ranging from 66 percent of respondents reporting increased income in Afar to 81 percent of respondents in Tigray (Figures 9 and 10). The YCS suggests that effective coordination between BPY and local and regional governments in Tigray, and easier access to loans in Tigray, may account for these differences. FIGURE 9. YCS CHANGES IN INCOME (2017–2018) 20 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity FIGURE 10. MEDIAN MONTHLY INCOME BY REGION Education. Looking at education, based on the YCS, youth with TVET and no school were the most likely to report increased incomes. Youth with college education had the lowest increases, possibly dampened by continued schooling. BPY Interventions and Income Changes As discussed in EQ1, BPY trainings helped participants to improve their employment status, and in many cases therefore their income. Youth also spoke about the increase in income stemming from expanded agricultural production, business growth, and transition into wage employment. As noted above (section 3.1.4. BPY Interventions and Improved Employment Status), multiple key informants noted that BPY trainings helped youth to increase the profitability of their business. The FGD participants in all six of the BPY implementing regions mentioned the intervention helped them to learn the procedures and requirements for starting a business using the resources around them. Participants further mentioned that BPY provided them with valuable information and insights on saving, customer relations, and the importance of persistence and patience during the inevitable times of business challenges and stress. The activity also created avenues for youth to meet with and receive information from different local offices such as MFIs, the Micro and Small-Scale Enterprise Agency (or Technical, Vocational, and Enterprise Development Office), the Office of Women, Children, and Youth Affairs (or Office of Youth and Sport), and in Tigray, with the Office of Agriculture and Land Administration. 20 Data from KIIs and FGDs revealed that some youth diversified their income sources after they began participating in the BPY activity. A FGD participant in Amhara said, “now I (also) have 15 sheep, but before I had only chickens.” 20 The specific name and purview of local government offices differ across woredas and regions and have also shifted over the course of the activity with new political leadership. 21 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity “Now I have a motorbike of my own that my brother drives and earns money.” (Female FGD Participant, SNNPR) “I started from vegetable production, then I bought a Bajaj, and I am now going to buy a truck.” (Male FGD participant, Amhara) Focus group participants and key informants spoke about benefits from increased income, including saving money and repaying loans. The YCS similarly found that, although the proportion of people saving went up modestly or not at all, the amount of savings increased dramatically. The average amount saved by YCS respondents increased over 80 percent over twelve months, with the average saving amount increasing from 3,135 Birr (US$137) at baseline to 5,757 Birr (US$251) at endline. 21 Some evidence from FGDs also suggests that even youth who did not transition to an improved employment status applied BPY lessons toward improved management of their personal or household finances. Access to Capital Evaluation FGDs and KIIs, as well as the YCS, unanimously identified the most pressing constraint for youth as startup and working capital. 22 In the YCS, survey participants identified financing as the resource needed most often for entrepreneurship (ranging from 63 percent of unemployed participants to 90 percent of wage employed participants at endline). Beneficiaries of BPY used different sources of startup and business expansion resources such as land, tools, and equipment. The BPY activity’s final annual report noted youth beneficiaries had invested 107,845,318 Birr (US$3,343,205) 23 as startup capital, collectively. Table 6 summarizes the top sources of capital. Although there is some divergence, individual or family savings, as well as accessible financial institutions, were the most common sources of capital for BPY participants. TABLE 6. TOP SOURCES OF CAPITAL FOR BPY PARTICIPANTS BPY FINAL ANNUAL REPORT 2018 OUTCOME SURVEY MFIs (40%) Families (32%) Own Savings (15%) Savings Groups (13%) Own Savings (43%) Families (gifts or loans (39%) Financial Institutions (15%) 21 Data (including exchange rates) taken from 2018 Youth Cohort Study Endline Evaluation, Final Report. 22 Participants in this evaluation FGDs and the YCS also mentioned a lack of working space, lack of infrastructure such as electricity, distance from marketplaces, lack of market linkages, seasonality of activities, especially agricultural activities, fewer business opportunities in rural areas, and too much competition from similar businesses in a local area. 23 Using 0.031 USD to 1 Birr conversion. Oanda.com. accessed 3.9.20. 22 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity In using available loan schemes, BPY participants had issues of funding or of accessibility because of collateral requirements and interest rates. The FGD participants explained the main challenges in accessing loans were collateral, the amount of savings required, a mismatch between loan amounts and business needs, and lengthy loan procedures. In all BPY regions, there were two loan schemes available to youth: regular loans from MFIs or the Government of Ethiopia (GOE) Revolving Fund (Youth Fund). Although MFI loans are still available, the high collateral requirements (10 percent) and interest rates (17–20 percent) limit access. The Revolving Fund, in contrast, provides loans through regional MFIs at a lower interest rate (8 percent). The fund has had challenges, however, with selecting qualified candidates and ensuring repayment, which ultimately undermined trust and limited further funding. Tigray had other loan schemes available. 24 There were variations in access to loans by region. The 2018 Outcome Survey found that loans from financial institutions were a source of initial capital for 49 percent of respondents from Tigray, 22 percent of respondents from SNNPR, 11 percent of participants from Amhara, and fewer than 10 percent of participants each from Afar, Oromia, and Somali. 25 Based on the 2018 Outcome Survey, participants in Tigray were more likely to have applied for and accessed capital through financial institutions, though participants in Afar and SNNPR were the most likely to receive financial assistance if they requested it. 26 From the 2018 Outcome Survey, males and females were equally likely to access loans from financial institutions (14 and 15 percent, respectively) or receive a gift from family, friends, or relatives (39 and 40 percent, respectively). Similarly, females were slightly less likely to apply for financial resources (43 percent, compared with 48 percent for males) but were slightly more likely to receive financing (31 percent approval for females, 29 percent for males). Males were more likely, however, to invest more initial capital irrespective of the source and estimated the market value of their capital at more than twice that of women (8,000 median for females and 15,000 for males). A key informant reflected this, noting that women oftentimes request lower loan amounts than men. Interviews with microfinance managers revealed that most BPY beneficiaries repaid their loans regularly. According to KIIs with MFI and woreda program coordinators, BPY beneficiaries in Amhara and SNNPR repaid their loans regularly, whereas loan repayment in Afar was reported to be poor. 27 Key informants in Afar, Tigray, and Amhara, along with other evidence on savings 24 In BPY-implementing woredas in Tigray, two additional schemes were available—the Tigray Rural Fund and the Feed the Future Livelihood for Resilience Activity Fund. Implementation of the Tigray Rural Fund was paused during fieldwork, but the maximum amount (40,000 Birr per US$1,250) and high interest rate (17 percent) limited use. Although Feed the Future Livelihood for Resilience Activity Fund loans reportedly had a high repayment rate, they are available only to Feed the Future Livelihood for Resilience Activity Fund and savings groups participants. 25 FGD group findings were heterogenous with regard to prevalence of BPY-mediated loans, ranging from zero out of 12 participants in Afar (Asayta, Wahle Fenta, and Henelle) to 58 percent prevalence in Tigray (Endemehoni, Tahtay Haya) and SNNPR (West Azernet, Lera). This mirrors the regional variation in outcome survey data. 26 Exactly 100 percent of applicants for financial assistance/loans in Afar, of the 25 percent of participants who applied in Afar, received their loan. In SNNPR, 29 percent of participants applied, and 63 percent received financing, the second highest rate receiving financing among the regions. 27 For example, out of 4,512,958.73 Birr disbursed to 10 groups with 148 members (63 female and 85 male) in Asyita woreda, 2,556,322 Birr, or 56.6 percent of the principal loan amount, was recorded as overdue. 23 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity culture in Ethiopian pastoralist communities, 28 indicate culture of loan repayment is better in agrarian regions compared with pastoralist regions, which could have implications on future loan availability and income levels. Security may have also depressed access to and repayment of loans in some BPY regions. 29 Conclusions Income. Evidence from KIIs, FGDs, and from secondary analyses all suggest that BPY beneficiaries did increase their incomes from that at project enrollment, though the increase was modest in most instances. Sources point to youth increasing their income through gaining employment as well as through increasing the profitability of existing work through both hard￾and soft-skills trainings. Participants also used this income to increase their savings, an important buffer against potential shocks or emergencies. Because of the large number of participants in the agriculture sector (directly or indirectly), however, income is seasonal and also subject to potential shocks that are becoming increasingly frequent in Ethiopia. This points to the importance of diversification and continued savings to help youth benefit from increased income in the long term. Sex. Males and females both reported increased income while participating in BPY across data sources. The degree of difference, including seasonality, varies and may be connected to the type of employment, which has important implications for the stability and resilience of male and female participants. Although females have been perceived to have less access to credit, outcome survey evidence suggests that something prior to loan application inhibits access to credit, in terms of the likelihood to apply and the loan amount. This may be the access to individual family resources for collateral, or women’s self-perceptions of resource needs or the likelihood or appropriateness of pursuing financing. Access to capital and distributional effects. Access to capital remains a major factor in the startup and expansion of small businesses, with participants—especially self-employed and wage-employed participants—most frequently identifying financing as a need. With most participants investing their own funds or family funds or going to MFIs that oftentimes have collateral requirements or high interest, this limits opportunities for less well-off youth. This presents concerns regarding equity for youth without individual or family assets in a program that emphasized entrepreneurship rather than other kinds of opportunities (in part because of the lack of other opportunities, like wage employment, in the target woredas). EQ3: What did BPY do to equitably target male and female beneficiaries? Gender Equity Vision and Beliefs Gender equality and female empowerment are formally acknowledged as cross-cutting issues incorporated into all program components, and SC states that “USAID’s Building the Potential of 28 See, for example, USAID/Ethiopia PRIME, 2015, “Financial Services.” Available at: https://www.prime￾ethiopia.org/wp-content/uploads/2015/06/Financial_Services.pdf 29 In Gode woreda of the Somali Region, unrest and violence led to disruption of the loan repayment process and subsequent disputes over the amounts actually repaid. 24 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity Youth gives equal consideration for both young men and women.” 30 Although the project agreement and BPY Performance Monitoring Plan did not specify targets for male and female participation, KIIs with SC and other IP personnel and with local government officials, and BPY reports strongly suggest that the project was seeking to achieve equal participation of males and females. EQ1 and EQ2 disaggregated intervention outcomes by gender; this EQ examines the activities BPY undertook to target males and females equitably. In multiple FGDs, respondents stated that females were prioritized over males. The inverse only held in one case. However, a female Regional M&E Coordinator stated that “equality is not equity.” In her view, given the greater challenges that young women face in the workplace compared with their male counterparts—including substantial childcare and agricultural household responsibilities, safety and travel concerns, and customary restrictions on inheritance and occupational choice—additional resources and programmatic support should be allocated to females to enable achievement of equal outcomes. Qualitative data (KIIs and FGDs) and a review of project reports clearly document that BPY made persistent efforts to ensure equitable recruitment, selection, and participation of male and female beneficiaries in the various intervention activities. • Obstacles to gender equity: KIIs and FGDs also provided helpful insight into the obstacles affecting gender equity in BPY implementation. More male beneficiaries than females utilize the youth centers or YES-Hubs services for both BPY-related services and recreation because of issues of travel distance, male-focused recreational activities, and cultural norms concerning the propriety of female participation in evening/night activities. • Transportation: In pastoralist communities, it was challenging for females to attend BPY training because of the scattered settlements, household responsibilities, and concerns for safety of women traveling far from home. This is consistent with the findings that fewer females accessed TVETs for hard-skills training, likely because of at least in part to transportation challenges. Youth intake data (2019) indicate that out of 2,668 who attended hard-skills training, 1,174 (44 percent) were female, and 1,494 (56 percent) male. • Cultural norms and ideas of propriety: o Training and support structure: Because of traditional role norms, some female participants appeared reticent to participate in activities or voice their opinions in training in some regions and woredas. In pastoralist communities especially, women were reportedly uncomfortable with male trainers, and some parents were unwilling to let girls attend training with boys. Finally, the timing, location, and scope activities also determined women’s ability to attend. Women attending late night meetings or meetings far from home was often considered inappropriate, thus limiting attendance at such events. 30 SC. BPY Year Two Annual Report, 2017. p. 68. 25 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity o Occupational choices: Although BPY sought to encourage young women to engage in nontraditional occupational choices, self-employment ventures and hard￾skills training tended to conform to traditional gender distinctions. • Loans and financial opportunities: o Access to capital: Differences in access to parents’ assets limit women’s access to capital. For instance, in Dire (pastoralist part of Oromia Region), men traditionally inherit cattle at birth and build a herd throughout childhood and adolescence. Some KII and FGD data from Amhara suggest that parents may be less willing to provide collateral for loans to daughters than to sons, viewing the sons as traditionally primary earners. o Perceptions of loan readiness: At the same time, KIIs with MFI representatives also suggest that there is a prevailing perception that women are more prudent in the amounts requested and more responsible and reliable in repayment. Informants stated that women took a relatively lower amount of loans compared with men because women tend to take only the amount needed for the business. The MFI representative believed that men were more likely to divert the loan and make use of it for unintended purposes. Recruitment and Selection The BPY activity conducted a series of community-focused interventions specifically designed to heighten and mobilize gender awareness, change community norms, and build support for female participation at all levels of the project. The majority of FGD respondents agreed that there was equity in the selection of males and females into the program. The selection of beneficiaries was conducted by kebele-level committees, which included a representative from the Office of Women, Children, and Youth Affairs, who were instructed to seek equal participation of female and male youth. In instances where male and female candidates appeared equally qualified, KIIs with SC/IPs and government stakeholders indicated that the committees were biased toward selection of the female applicant. Oversight, Staffing, and Training The BPY M&E system tracked outcomes and activities/interventions data through its regular monitoring and assessments. The data was disaggregated by sex of participants, enabling BPY to quickly identify and address any issues of inequity by sex in participation or results. For example, data showed that females utilized services at YES-Hubs less frequently than did their male peers, so BPY pilot-tested an initiative to provide reproductive health information for females in a designated corner of the youth center that proved effective in some, but not all, instances. The BPY activity also attempted to affirmatively seek females for staff positions, including the most recent BPY Chief of Party. Finally, BPY provided training on gender sensitization to people involved throughout the BPY process. For example, The SC representative in Gode (Somali Region) provided gender sensitization training for staff, facilitators, and community members and monitored gender representation in each intervention. 26 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity Implementation The BPY made efforts for equitable targeting of males and females at training sessions, YES￾Hubs, and loan services. Training sessions: The BPY provided its soft-skills training in kebeles where beneficiaries were residing, which helped to reach nearly an equal number of male and female beneficiaries. In some areas (e.g., Tigray), females with small children were accepted as BPY participants, and training schedules were organized to accommodate their household responsibilities and travel requirements. Specifically, this meant holding training in mid-mornings after women were able to complete household and farm chores. Group discussions during training were participatory and tried to encourage females engagement, as some young women appeared reticent to express their opinions, particularly at the beginning of the training. In some areas, where mixed gender classrooms were deemed inappropriate, separate training sessions were arranged for female participants. Additionally, priority for hard-skills training opportunities was given for females when males and females were at equal status in terms of meeting criteria. YES-Hub equal utilization: Across all regions, KIIs indicate that more male than female youth participate in activities at YES-Hubs. The KIIs and FGDs revealed that while females tended to utilize the hubs primarily for employment-related services, they did not generally participate equally in the social and recreational services offered. The KIIs and FGDs revealed that particularly nighttime recreational services such as watching football matches on TV were not accessed by female youth who did not feel comfortable attending. Alternatively, site visit observations suggest that tea/coffee services based in hubs were operated primarily by young women. One interesting pilot initiative was to use YES-Hubs to provide female-focused services— creating reproductive health corners where young women could obtain information and guidance on these health issues. Although this was considered somewhat successful in some sites, in most sites, they were not popular or sustainable because many female beneficiaries felt embarrassed to discuss these matters in a somewhat public setting (KIIs with Yes-Hub coordinators.) Loan service: The BPY facilitated the introduction of loan services by financial institutions at the end of the Work Ready Now! training in each of the BPY implementing regions. There was no indication in the KIIs and FGDs that gender conscious accommodations were made for the different experiences of men and women searching for loan services. Conclusions Women and men experience BPY services differently. Recognizing these challenges, BPY actively endeavored to integrate gender considerations into their program design and implementation approach. They did so with mixed success. They had better success with addressing issues of targeting, but less success addressing issues with mainstreaming gender implementation and service provision. • Women face constraints on travel, availability, access to capital, and the ability to interact with mixed gender groups. Women and men tend to follow cultural norms in 27 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity terms of business opportunity selection. Women also are perceived by loan providers as being more reliable and consistent in loan repayment. • Efforts by BPY to conduct training events at times and locations convenient to young women—specifically accommodating childcare and household responsibilities and limiting travel distance for women—were considered widely successful. • Adding kebele-focused gender awareness campaigns, aimed at mobilizing community level support in rural and pastoralist communities, before trainings started proved successful in shifting community norms and attitudes to create an enabling environment for female participation in the BPY. • Data from KIIs and FGDs showed that respondents felt that special accommodations were made to take into account differences between men and women with regard to capital access and loan provision. • YES-Hub participation that focused on the social and recreational services was predominantly male. Efforts, such as the reproductive health centers, did not effectively fill the gap in services, and women were less likely to benefit from the social and recreational elements of the activity design. EQ4: To what extent have small grants provided to TVETs increased the availability, quality, and relevance (in terms of employment/income) of training for youth?31 General Findings The BPY activity’s small grants provided scholarships that directly enabled 2,668 participants (7.4 percent of total beneficiaries) to access hard-skills technical training at participating TVETs/TTIs, thereby improving these youth’s skill set and employment outcomes. The total value of these grants was US$282,649. Although population characteristics may differ, the SC 2017 Outcome Survey data shows 82 percent employment for those who received hard-skills training as compared with 74 percent for beneficiaries as a whole, an increase of about 11 percent. The small grants program also, to a modest degree, increased the capacity of these training institutions through the funding of staff training and the provision of some equipment, materials, and supplies, strengthening their operations and service delivery, presumably benefiting their overall student populations. Small Grants Objectives, Foci, and Components The evaluation team investigated the small grant intervention and confirmed strong linkages between BPY and the TVETs and technical/vocational education and training development objectives at the federal, regional, and woreda levels. See Figure 11 for small grant allocation. 31 With USAID concurrence, key evaluation themes addressed under EQ4 are: grant criteria, grant disbursement, grant utilization, relevance of the grant to TVETs and beneficiary needs, TVET capacity-building, success of TVET graduates (employment and income), outcome from the grant initiatives (market-driven curriculum and training content), and lessons learned and challenges. 28 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity The small grants intervention had several directions and objectives focusing on tuition payment and institutional capacity-building. Small grants to TVETs served as tuition scholarships, paid to the institution thereby enabling BPY participants to access the hard-skills training offered. These grants were utilized for their intended purpose in all the TVETs visited by the evaluation team. A few TVETs used a portion of their BPY small grant funds to create sub-grants to other training institutions. For example, a private institute in Amhara Region reported that a hair salon training course was subcontracted to another private college, and the institute topped up what BPY had budgeted. FIGURE 11. SMALL GRANTS ALLOCATED FOR EACH REGION The small grants were also intended to support capacity-building of TVETs, including training TVET instructors in the BPY soft-skills modules, promoting gender diversity in trades with low female participation, establishing market linkages, and enabling the training institutions to adapt technical training to local conditions and opportunities. Although short-course tuition scholarships account for the greatest portion of the small grant utilization, under the BPY capacity-building mandate, the participating TVETs received (in aggregate) training of trainers (TOT) for their instructors; material and supplies for course development and delivery and BPY-related administration; Mobile for Career Development 32 training for program managers and instructors; instructor per diem payments; and transportation fees for students. 33 In few cases, such as the Gode Polytechnic Institute, the small grants were used both for course tuition and to cover on-site room and board for BPY 32 Mobile for Career Development is a comprehensive information system developed by BPY to provide locally relevant employment information to BPY beneficiaries, local steering committees/councils, coaches and mentors, local government offices, YES groups, and youth beneficiaries via phone, tablet or computer. Core systems components are: information and knowledge links; training manuals; practical guides; local market information and labor market data; employment listings; and an events calendar. 33 TVETs had some discretion in how the small grants resources were utilized, and most did not receive all the types of supports listed. 29 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity beneficiaries. Both government and private TVETs facilitate the Certification of Competence assessment for BPY beneficiaries. A few TVETs provided soft-skills training for their own (non￾BPY) students through instructors who took the BPY TOT course. The short courses newly developed for BPY could also be marketed by the TVETs to other populations as warranted, itself a form of capacity-building. The BPY also facilitated the process of linking and integrating public TVETs into the woreda youth workforce development system, with TVET representatives serving on YES councils and BPY steering committees. Private training institutions/businesses, however, were generally not included in this systems development and networking process. Training Institution Selection and Challenges The selection of particular TVETs for participation in BPY and receipt of the small grant was based on market opportunities as presented as beneficiaries’ preference toward specific sectors in preparing their PDPs. In some contexts, BPY opted to provide participant tuition to private institutions offering a particular training package such as male hair styling (Welmera) through a competitive bid process. If there are limited numbers of TVETs in a specific woreda, BPY established linkages with these and encouraged the creation of appropriate market-related courses if none were available. For example, utilizing the BPY small grant, a food processing hard-skills training course (and necessary training materials) was added to the course offerings at Adadely Polytechnic Institute in Afar Region, thereby increasing the institution’s capacity and providing a market-relevant training opportunity for BPY beneficiaries and other students. There appears to be wide variation in the quality, extensiveness, and practical utility of TVET courses; some institutions chronically lack supplies, infrastructure such as electricity, and functional equipment in the workshops; have a shortage of qualified instructors; and have insufficient hands-on, practical studies opportunities. This penchant for teaching theory rather than hands-on skills greatly limits the actual utility of TVET training in the marketplace. BPY small grants provided some limited equipment and supplies, which did help mitigate some of these shortages and delivered a training of trainers package in some institutions, which may serve to improve teaching and learning at these training institutions. Gender Issues Of 2,668 BPY beneficiaries who received hard-skills training, 1,174 (44 percent) were females, a slight decrease in representation of female beneficiaries, who compose 47.2 percent of the total BPY population. Females in the program were more likely to choose training in service sector skills like hairdressing, beauty salon, and petty trading, while males showed a preference for activities requiring greater physical strength like woodwork, plumbing, and metalwork. Within the rural/pastoralist context, TVETs may be situated a significant distance from scattered rural communities, posing a particular hardship for female beneficiaries. 34 34 Public TVETs are mandated to provide training both on and off sites to increase accessibility for the woreda community, but in practice many do not. In KIIs, TVET representatives raised different reasons for the lack of off￾site trainings, mostly related to budget and human resource constraints and the stationary nature of some workshop-based training machinery. 30 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity Conclusions Based on SC outcome data, the TVET component of BPY is associated with higher employment outcomes for BPY participants. BPY supported the capacity-building of TVET institutions so that they could potentially provide this kind of support to a broader range of participants. The limited scope of BPY support for these institutions, however, likely limited its impact to participants themselves, rather than TVET institutions more generally. • There appears to be wide variation in the quality, extensiveness, and practical utility of TVET courses; many public TVETs remain seriously under-resourced and appear to often offer theory-focused training courses of dubious quality and of limited practical or market relevance. • Although the BPY small grants initiative demonstrated the possibility of TVET capacity￾building, its limited scale and primary focus on providing scholarships will likely limit the sustainability of this intervention component. • Lack of partnerships between public TVETs and private employers remains an obstacle to youth workforce development and private sector growth. EQ5: To what extent and in what manner have PPPs provided quality training and/or employment opportunities for participants in the BPY activity? Definition and Findings Within the BPY model, PPPs play a structural role primarily, providing the institutional supports necessary for the delivery of training or post-training services. As such, these partnerships were essential for the organization and delivery of BPY services and employment outcomes. Under the USAID standard indicator definition in the Performance Indicator Reference Sheet (PIRS), PPPs entail a formal working relationship between a public and a private-sector entity to act together in pursuit of a common interest or activity. 35 However, in practice, and in their approved activity MEL plan, the BPY activity defined PPPs as formed when a memorandum of understanding (MOU) was formed between a private and governmental entity to work jointly in furtherance of the activities and/or sustainability of the BPY interventions. As such, BPY counted PPPs as collaboration between BPY and any other entity—public or private. Ultimately, this differs from the standard USAID definition of PPP. The evaluation team was not able to disaggregate the data to account for just partnerships between public and private entities. In accordance with the BPY definition, SC reported 157 “public-private partnerships formed as a result of BPY assistance” against a LOA performance target of 135, a 116 percent achievement (BPY Year 5, Quarter 3 Report, July 30, 2019, p. 7). 35 The PIRS for Indicator #4.5.2-12 states “A public-private partnership is considered formed when there is a clear agreement, usually written, to work together to achieve a common objective ... USAID must be one of the public partners and is almost always represented in the partnership by its implementing partner.” 31 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity For the 2,668 BPY beneficiaries (7.4 percent) who were able to avail themselves of hard-skills training—most frequently at public TVETs but also at private institutions—this technical training was conducted based on partnership MOUs between the IPs and the training institutions. 36 The YES-Hubs, often the site of post-training support services including counseling, coaching, mentoring, and employment information dissemination, were themselves a PPP between BPY and local government offices. The PPPs also structured the provision of job shadowing opportunities and the BPY apprenticeships and other practical study opportunities. In summary, partnerships were essential to creating the woreda-level institutional networks necessary for local youth workforce development within the BPY design. With the conclusion of the activity and the closure of the BPY offices at the woreda, regional, and national levels, the sustainability of the majority of formal BPY PPPs—those involving the BPY IP as an entity or party to the partnership—will not continue. However, KIIs indicate a number of examples in which community-based youth workforce development structures created by BPY, including local steering committees and YES-Councils, continue to function. In terms of sustainability, KIIs also noted that the PPP model is less likely to work in rural areas. The BPY model is built around the linkage and integration of public youth workforce development training and post-training support activities with local private-sector market￾driven opportunities. As such, it is conditioned by the availability of large businesses and industries that can provide post-training mentoring and work-based learning support, including formal apprenticeships, serves as sources of wage-employment, and provides value-chain linkage opportunities for micro- and small business entrepreneurs. Rural and pastoralist areas cannot offer the level of private-sector partnership opportunities found in more urban environments, which, according to one KII, was the implementing environment originally envisioned for this intervention model. Conclusions The BPY model integrated PPPs into their approach, identifying partners throughout the implementation process. • The PPP approach was likely essential to the effective implementation of the BPY model itself because it allowed BPY to build off of the work of existing institutions, networks, and structures to effectively implement their program. • The PPP approach is likely limited in its sustainability. Certain functions that focus on networking and require limited funds to maintain will function, but those that require BPY inputs as a party to maintain support will cease to exist. • There are more opportunities for PPPs in urban or peri-urban areas. There are fewer opportunities to support BPY or other programs like it through PPPs in rural areas. The rural/pastoralist context of BPY activity—with limited local business development and a dearth of large-scale manufacturing and agribusinesses—restricted opportunities for sustainable private-sector linkages, constraining BPY outcomes. In the absence of 36 Although the TVETs did provide short-course hard-skills training to BPY participants, the expansion of soft-skills training by TVET instructors to the greater population of TVET students was quite limited. 32 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity significant growth of new innovative agribusiness opportunities and technologies, PPPs in pastoralist and rural locations will likely remain limited in number and scope. • Variability in PPP definitions between the USAID standard definition and the BPY definition persisted through the life of the project. Although BPY did exceed their target for developing partnerships in general, the counting of partnerships with both public and private entities and BPY likely overstated the exact number of PPPs developed with U.S. Government funding. EQ6: What is the operational and service delivery capacity of YES-Hubs? Will the YES-Hubs be operationally and financially sustainable post-USAID support? Findings: Operational and Service Capacity The YES-Hubs serve as the physical site for the delivery of a variety of employment-related and other programmatic services and recreational opportunities to BPY beneficiaries and other local youth. Services are not uniformly provided by each and every YES-Hubs; however, many include computer access and training, career coaching and mentoring (including soft-skills training, access to job notices, employment-relevant education), training material, reproductive health information for females, cafeteria services, indoor recreational facilities, football club, showers, and access to a TV, often for viewing football matches. YES-Hubs were established within existing local government youth center buildings, which had become nonoperational and, in many instances, dilapidated prior to the start of BPY. Working with the local Office of Women, Children, and Youth Affairs (and earlier, with the Office of Youth and Sport), BPY provided the equipment and supplies to renovate and revitalize the youth centers and make them operational. These included the provision of computers, desks, chairs, recreational equipment, teaching and learning materials, TVs, and other items. A total of 39 YES-Hubs were established and operationalized across the six regions and in all but one of the target woredas the evaluation team visited (see Annex 1, Figure 15). BPY stopped operations in its YES-Hubs by September of 2019. The evaluation team visited sites in October and November of 2019. Service Delivery Capacity YES-Hub service delivery capacity was estimated by examining the total number of youth receiving any sort of service at a YES-Hub during the first three quarters of program year five. This was the most recent data available to the evaluation team. During this nine-month period, BPY monitoring data (2019) indicates a total of 24,068 people accessing YES-Hubs or an average of 2,674 per month across all six regions. 37 Activity monitoring data (2019) further indicates approximately 37 percent female participation. This figure is less than the overall program participation of women in the BPY activity. This figure is also consistent with the information obtained through KIIs. Observations during the data 37 The evaluation team was unable to get disaggregated data on YES-Hub attendance by region. 33 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity collection and discussions with key informants confirmed that male youth tend to make greater use of the centers as compared with their female counterparts. Household responsibilities and cultural traditions make it less likely for females to feel comfortable traveling to a hub, particularly at a time a football match is being viewed. Within the rural environment in which BPY was implemented, young people often need to travel a significant distance from their home kebele to the site of the YES-Hub, often situated in the woreda town. During FGDs, several youth expressed the need for additional youth centers to provide easier access for those residing in more distant kebeles, a sentiment also mentioned by local government officials. Operational and Service Sustainability The KIIs indicate that ongoing support by local government is the key factor in YES-Hub sustainability. Of the ten youth centers the evaluation team visited, eight remain operational post-BPY support ending in September 2019. They are now funded either by local government or from income being generated from the centers themselves. These centers continue to offer some subset of the total package of services that were available during the five years of USAID funding support. In the KIIs, local government officials in these eight woredas expressed determination to sustain some YES-Hub operations, viewing the centers as important providers of valuable services to a key population segment and political constituency. Local officials noted that they were hoping to reopen the two youth centers that had ceased operations, upon the availability of sufficient funding from local government. However, the likelihood of such financial support is uncertain. See Table 7. TABLE 7. OPERATIONAL STATUS AND SUSTAINABILITY OF YES-HUBS VISITED REGION WOREDA STATUS REASON FOR NONOPERATION Afar Asayita Operating Amhara Dera Operating Debub Achefer/South Achefer Operating Oromia Dire Not operating Local government funds pending Wolmera Not operating Local government providing guard; funding pending SNNPR Azernet Bebere Operating Endegegn Operating Somali Gode Operating Jijiga Operating Tigray Endmehoni Operating In the eight centers still operating, local government offices have provided a variety of resources to support the centers, including use of the physical structure itself (in all operational centers); salary support for some YES-Hub coordinators; provision of overnight guard services to secure the desktop computers, TVs, and other equipment; materials and supplies; and technical assistance support for computer maintenance and continued basic computer training. Further, the centers are themselves generating some limited revenue through fees for services 34 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity such as photocopying, computer usage, and coffee/tea services, but this, and a reliance on volunteer staff support, will likely not provide sufficient income over the long term to sustain the hubs as the holistic youth services entities BPY envisioned. The KIIs also showed that implementing these challenges would be even more pronounced if YES-Hubs were implementing in rural/pastoralist settings with a limited private sector and lacking a significant industrial or agricultural innovation technology base. To date, BPY has been largely unable to build the sort of true PPPs that would give the local business community a genuine stake in the success of the YES-Hubs or an incentive to utilize and support them in rural areas. Conclusions The YES-Hubs effectively provided BPY activities with a space to operate and provide services and support. It combined these services with wider services used by youth as well as recreational activities. • The services in the YES-Hubs were widely used. The centers were popular and served a number of beneficiaries. • As also reflected in EQ3, the YES-Hubs were most effective among males and participants who lived closer to the hubs themselves. Rural users and women frequented the hubs less because the content of the recreational activities and the travel time needed to reach the hubs. • Although successful during the LOA, YES-Hubs’ ongoing functionality appears, in virtually all locations, largely dependent on government and/or outside donor support, particularly to sustain its full array of services. Because the youth workforce development-related services, including soft-skills training and post-training coaching and mentoring support, are intensive in resources, labor, and administration, these core functions that are essential components of the BPY model will likely be lost, given ongoing financial constraints on local government and limited volunteer resources. Remaining services will require external support, government support, or some kind of pay-for-service model to keep the YES-Hubs open. EQ7: To what extent has the BPY activity been evidence￾based to inform USAID and its development partners on the appropriateness and effectiveness of its interventions? During KIIs, senior SC and some IP personnel noted that they regard BPY as a pilot project, in that it was an initial intervention to explore and demonstrate the utility of employing a holistic array of programmatic approaches to market-driven youth workforce development within the challenging contexts of rural/pastoralist woredas. 38 38 It should be noted that BPY was not formally designated a pilot project in the cooperative agreement between USAID and SC. 35 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity Evidence Generation As a complex activity composed of numerous intervention modalities and encouraging local adaptation and experimentation within broad programmatic criteria, BPY could not employ an impact evaluation with a rigorous counterfactual design that could attribute independent programmatic variables with result outcomes. From its inception, BPY designed and implemented a variety of MEL systems and procedures to generate and evaluate evidence concerning implementation and results, and enable the activity to learn, change, and adapt intervention modalities based on this evidence (see Annex I, Table 14). First, BPY established a formal set of MEL performance indicators and targets against which to assess implementation and results. As part of this, BPY established baseline profiles of all participants; conducted local labor market assessments and value-chain analyses; collected, aggregated, and analyzed quarterly performance data; and conducted special studies and reports. BPY’s M&E system enabled program staff to collect relevant data elements at the kebele level, aggregate this information at the woreda level, and submit it to a regional M&E officer for analysis and submission to the BPY central office at SC/Addis. Dissemination and Information Sharing The BPY utilized a broad range of mechanisms to disseminate project information and success stories, including regular and special reports to USAID; print, radio, and video distribution of success stories; and presentations at professional meetings, staff field visits, and quarterly regional and woreda-level staff meetings (see Annex I, Table 15). Of particular note is the presentation of M&E data at kebele-level BPY steering committee meetings. Here local government officials, key civil society stakeholders, community leaders, traditional and religious leaders, TVET representatives, businesspeople, and youth, including BPY beneficiaries, used the M&E reports to help them consider and address local challenges and opportunities and manage the project at the community level. Data Utilization Data from KIIs provided evidence of how BPY utilized M&E products to enable programmatic adaptations. For example, as originally designed, BPY included a basic literacy and numeracy component to address the education gaps of what was expected to be a significant number of participants. However, BPY enrollment data indicated that only a small number of participants actually required these services (3.1 percent), 39 and so this program component and its performance indicator were dropped; the few youth requiring such services were instead referred to other local education service providers. Similarly, the Aflateen social education program aimed at the youngest participants (ages 15–18) was also eliminated as unnecessary. 40 Based on trainer feedback, BPY added two additional days to the BYOB training schedule to provide sufficient time to cover all of the topics adequately. A program of volunteer youth facilitators and mentors was established based on evidence that the youth facilitators’ caseloads were becoming unmanageable, and there was a lack of sufficient adult business mentors. 39 BPY Year Three Annual Report, p. 35. 40 Aflateen focused on social education including youth roles and responsibilities and financial education for youth ages 15–18 years. Only 466 youth received this training (272 females; 194 males), and it was discontinued after program year three. 36 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity Based on KIIs and FGDs, however, no one outside of USAID and BPY identified changes in policy or approach at the time of data collection. Evidence Structure Although the BPY M&E systems and procedures were thorough and extensive, the activity did not create a centralized database that would permit analysis of individual participant outcomes based on enrollment background characteristics and particular services received, thereby constraining the performance evaluation’s ability to assess the relative contribution of particular program components, SC’s capacity to adjust the intervention model, and USAID’s learning and design of new youth livelihood and workforce development activities. We would also note that the evaluation team was not able to secure access to the dataset for the 2019 YCS and had to rely instead on the earlier 2017 dataset. Conclusions The BPY activity established a multifaceted system for generating, reporting, and acting upon evidence and, based particularly on project data and field staff feedback, modified its design to eliminate its basic literacy and social learning for younger youth (Aflateen) interventions as unnecessary. • The BPY system generated evidence that the program itself effectively used to make important changes and adaptations using data. This learning approach is a success of the program. • The data systems themselves did have severe limitations that limited its external usability for future decision making. The database structure itself did not allow for efficient use of the rich data BPY collected. This included not being able to track participants over time to identify possible relationships between beneficiary characteristics, the particular sets of services received, and BPY outcomes in terms of improved employment status and income. EQ8: Has BPY contributed to increasing participants’ self-sufficiency? In accordance with its strategic objective, the BPY activity aimed to enable youth to achieve increased income and to “strengthen skills, knowledge, and social capital required to achieve economic self-sufficiency over the long term.” According to SC, “We are ready to provide Ethiopian youth with tailored services to help them achieve increased income and gain the skills, knowledge, and social capital required to achieve economic self-sufficiency.” However, the evaluation team could not find a precise operational definition of “economic self-sufficiency over the longer term” in any of the key project documents including the activity agreement, performance management plan, and PIRS, nor was “self-sufficiency” tracked or directly assessed as part of the activity’s M&E plan and system. Although increased self-sufficiency was stated as a project purpose and is the strategic objective under which the activity was designed, BPY did not establish a PIRS definition of self-sufficiency, did not create indicators or targets for this outcome variable, and did not include measurement 37 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity of self-sufficiency in the M&E system. Further, the issue of assessing self-sufficiency is complicated by the rural/pastoralist context in which agricultural work is often seasonal. A young person’s level of self-sufficiency may, therefore, fluctuate over the course of the year. However, given its central conceptual role in the activity’s theory of change, the evaluation team worked with existing data, as well as FGD and KII respondents to define economic self￾sufficiency in their own terms. The first measure of self-sufficiency is an improvement in employment status. EQ1 provides a helpful context on self-sufficiency because 41 percent of BPY participants did not improve their employment status, and for those participants who did, evidence—quantitative and qualitative (FGD) data—points to generally modest gains in income or assets. The FGDs and KIIs did generate a number of narrative accounts of youth participants who indicated that they had become less dependent on their families than they were before joining BPY, and some youth noted that they are now supporting siblings who would otherwise be dependent on their parents. One married female FGD participant commented on her increased self-sufficiency within her own family: “I am now very satisfied because before I had to expect from my husband, but now I am engaged in petty trade and crop selling.” Conclusion Although FGDs, KIIs, and secondary data indicate that a majority of BPY youth have improved employment status, and some likely increased income, it is too soon to assess whether these outcomes can be considered evidence of having achieved long-term self-sufficiency, particularly in the absence of a comparison group. Given that 41 percent of BPY participants did not improve their employment status and that those who did improve, the evidence (the quantitative and qualitative data) points to generally modest gains in income or assets, the achievement of self-sufficiency appears unlikely. EQ9: What have been the most significant successes and challenges for the design, implementation, and M&E for the BPY activity? Many KII respondents stated that they considered BPY to be a pilot initiative—testing out different elements of this comprehensive and holistic approach. Overall, the evaluation found that BPY far exceeded many of its indicator targets such as engagement, employment, and income indicators. This section draws from the previous findings to identify the most significant successes and challenges that BPY experienced. These elements are presented in the categories of program design, gender mainstreaming, rural and pastoral targeting, PPPs, community￾engagement and youth participation, and MEL. Program Design The focus on soft-skills training was well received. The BPY soft-skills training modules— Positive Youth Development, Work Ready Now!, and BYOB—were regarded by key informants and FGD participants successes of the activity design. Participants considered the 38 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity modules to be energizing, inspiring, and equipping beneficiaries to pursue market-linked wage and self-employment opportunities. Qualitative data from FGDs and KIIs showed virtual unanimous support for the transformative power of this intervention, and many respondents listed change of attitude as the key outcome of BPY. Work-based, or practical learning techniques, were more helpful than theory￾focused training. The BPY demonstrated the utility and viability of a work-based learning approach and the willingness of many local businesspeople to support community youth through serving as mentors and enabling young people to gain insights into workplace realities and opportunities. Since many TVETs currently follow a theory-focused training model, the BPY commitment to practical, work-based learning could have implications not just for donor programming but also TVET curriculum itself. The holistic design of the activity translated to a limited scope of support in some areas. Although BPY went in depth into the direct training and support of youth, other elements such as the establishment of YES-Hubs and support to TVET institutions were limited in scope by design. This translated into limited sustainability. Beneficiaries expected greater emphasis on support in job placement and loan provision and support. Even though BPY took on a holistic approach, many FGD participants stated they had expectations for effective wage employment or loan availability for program participants. Although many respondents appreciated the trainings, they also suggested that links to available businesses and job placement would be helpful. Loan readiness and connections were also considered a gap because respondents requested better access to market-based appropriate funding through better skills development on knowledge of their funding needs and business opportunities. Targeting and Relevance for Rural and Pastoral Youth Certain programmatic elements unintentionally favor urban rather than rural youth. Although BPY actively brought training and support to rural areas, program elements such as YES-Hubs, PPPs, and investment in TVET institutions were more difficult for rural/pastoral communities to access. Programming that aims to target rural and pastoral communities would need less reliance on those programmatic elements. Specifically, the BPY model presupposes and is dependent upon the existence of a significant, thriving private sector as the central anchor and energizer of self- and wage-employment opportunities including building new businesses linked to value-chain opportunities. The rural/pastoralist context in which BPY was implemented could not provide that base level of private enterprise development, and itself posed additional challenges around the cost of goods locally and the expense of transport of products to more populated areas for sales. Gender Mainstreaming BPY effectively targeted and recruited women to participate in their program through effective community engagement. Efforts by BPY to conduct training events at times and locations convenient to young women—specifically accommodating childcare and household responsibilities and limiting travel for women were considered widely successful. 39 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity Prevailing gender dynamics and controls on women’s access to capital, perceptions around loan readiness, movement, time, and transport require active addressing throughout the programmatic process. Although intended to be for both men and women, YES-Hub participation that focused on the social and recreational services, was predominantly male. Choices for training and business support tended to follow gendered lines. Partnerships BPY effectively leveraged their partnerships to expand their reach and influence. Although clearly constrained by the limited private-sector opportunities present in the rural/pastoralist implementation environment, BPY succeeded in mobilizing, linking, and integrating kebele and woreda-level organizations and institutions to enthusiastically support youth training and business development. At activity conclusion, many of the local steering committees and YES councils appear to be continuing to function, providing locally appropriate, albeit somewhat limited, support for youth workforce development. It was difficult for BPY to effectively engage private enterprises through market￾driven benefits. This limits sustainability because programming is dependent on external funding rather than market-driven initiatives. Without market innovation and growth, and additional public and donor resources, there is a limit to the sustainability and reach of the partnerships established. Further, without properly defining PPP in their MEL plan, BPY was limited in its ability to track engagement with private enterprises from a monitoring and reporting standpoint. Community Engagement and Youth Participation The BPY’s proactive community engagement was a driver of its success. The BPY firmly rooted its interventions at the community level, and the youth facilitators and program coordinators worked with local political and civil-society leadership and structures to identify and implement community-appropriate approaches to addressing youth employment challenges and opportunities. Youth themselves played a key role in BPY, serving as staff at the kebele and woreda level, vetting training materials and modules, serving as members of YES councils and steering committees, and volunteering as peer counselors and mentors and YES-Hub staff. One BPY beneficiary, a formerly unemployed university graduate, was recently appointed Dean of the new local TVET. The participation of youth in all levels of planning and implementation brought relevance, creativity, and enthusiasm to BPY, with youth serving not as passive recipients of program inputs but as committed agents building their own economic futures. Local government and community-based organizations’ priorities do not necessarily align with BPY priorities—specifically in targeting disadvantaged youth. Although BPY’s emphasis on local decision making helped to build community support and participation, this very flexibility also may have had the unintended consequence of decreasing the participation of this most-disadvantaged segment of the youth population. The KIIs revealed, for example, that in some kebeles, unemployed university graduates and their families were most vocal in asserting their employment needs and were considered a priority for local government leadership. In this way, community engagement, though essential to the program, limits the ability of BPY to fully control programmatic targeting. This also resonated in project 40 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity results where the project did suggest that the least educated segment of beneficiaries had the smallest gains in terms of improved employment status. Monitoring, Evaluation, and Learning (MEL) The BPY effectively used the evidence generated to make programmatic adjustments to their program. The BPY conducted its own research and made important programmatic adjustments. The activity utilized a broad range of mechanisms to disseminate project information and success stories, including regular and special reports to USAID; print, radio, and video distribution of success stories; and presentations at professional meetings, staff field visits, and quarterly regional and woreda-level staff meetings. Of particular note is the presentation of M&E data at kebele-level BPY steering committee meetings where local government officials, key civil society stakeholders, traditional and religious leaders, TVET representatives, business people, and youth, including BPY beneficiaries, used the M&E reports to address local challenges and opportunities and manage the project at the community level. MEL systems limitations: The BPY M&E system was not designed to enable disaggregation of employment outcomes by the program services or combinations of service components. This limited the lessons learned regarding the utility of the various service components of the BPY model, decreasing its usefulness in designing future youth workforce development interventions in Ethiopia. Further, the system contained severe data quality limitations including missing data and database errors that affected the alignment and analysis of data. 41 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity 4. RECOMMENDATIONS There is strong evidence that soft-skills training precipitated positive attitudinal changes, and suggestive evidence that it led to improvements in employment, income, and savings. Further, BPY exceeded its targets in terms of income and employment. BPYs combination of soft-skills training and workforce-based, practical training should be a core component of future youth workforce/livelihood development intervention in Ethiopia. Given the mismatch between the activity’s market-driven approach to youth employment and the relative scarcity of larger private-sector businesses in these rural settings, USAID should consider implementing the BPY model in urban and peri-urban areas where market linkages and wage-employment opportunities would likely provide greater value for money. In rural/pastoral contexts, entrepreneurial training and soft-skills training should focus on agricultural value-chain opportunities and creating and managing innovative microenterprises. The BPY approach that is generalized and community-led limits the ability of BPY to target the most underserved, impoverished, and marginalized segments of youth. Program designers should support the design of pilot youth workforce development initiatives focused on the most marginalized segments of the youth population, including youth with disabilities, youth from the most economically disadvantaged communities, youth with minimal educations, and youth ex-offenders. As these groups often require special targeting and a specific provision of services, these groups would benefit from a specialized rather than generalized approach. This could be accomplished through specific requirements in solicitation documents and targets in performance monitoring systems. Field visits to TVETs by the evaluation team and data from FGDs and KIIs point to major disparities and gaps in the quality, extensiveness, and practical relevance of TVET training courses. In light of the proliferation of TVETs and their importance for youth workforce development in Ethiopia, USAID should support the conduct of a rigorous survey, assessing the training capacity, resources, training quality, market linkages, and placement rates of all registered TVETs; the development of a database of this information; and a reliable monitoring and reporting system tracking the integration of work-based learning and practical studies in Ethiopia’s education and training institutions. This data and information system would become an important tool for youth workforce program planning and management and assessment and support cost-effective linkages with TVETs that have a documented success in market-relevant training delivery. USAID should consider integrating elements of BPY recruitment, targeting, and decentralized training and support strategies as benchmarks for future programs. The BPY effectively engaged rural communities and women in programs, even though community members had reservations and concerns about female participation, and both women and rural participants faced challenges in transport. Integrating gender mainstreaming efforts into loan provision and networking elements of the activity is also recommended. To increase women’s participation in networking activities, make accommodations similar to those made for training activities. Schedule recreational and 42 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity networking activities in a decentralized manner and identify entertainment opportunities at a time and with content that families and communities think are appropriate for women. Loan support should take into account differential lender perceptions of loan readiness of males/females as well as differential access to capital of the two sexes. Further, gender mainstreaming support should not just be targeted at women. Specifically help young men to address gendered perceptions of their ability to repay loans. For activities in which evidence and data quality are essential, provide the implementer with third-party data quality assessments and support, or ensure that implementers have high quality research partners. This support should come in three key phases: (1) in the design of the MEL framework and data collection approaches, (2) in the design, conduct, and oversight of data collection (especially large-scale surveys), and (3) in the cleaning and analysis of data. A small investment in these efforts could significantly improve data quality, research rigor, and utility of evidence. 43 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity ANNEXES Annex I: Additional Tables and Figures FIGURE 12. MAP OF WOREDAS VISITED FOR PRIMARY DATA COLLECTION TABLE 8. KII AND FGD PARTICIPANTS LOCATION KIIS FGDS MALES FEMALES TOTAL # MALES FEMALES TOTAL Addis Ababa 4 1 5 - - - - Afar 10 1 11 2 16 8 24 Amhara 20 6 26 4 22 23 45 Oromia 15 4 19 4 34 9 43 SNNPR 16 3 19 2 14 11 25 Somali 14 1 15 2 11 12 23 Tigray 12 4 16 2 16 8 24 TOTALS 91 20 111 16 113 71 184 44 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity TABLE 9. COMPOSITION OF FGD PARTICIPANTS BY WOREDA WOREDA REGION MALES FEMALES TOTAL Ada'a Oromia 20 3 23 Asayta Afar 16 8 24 Azernet Berbere SNNPR 14 11 25 Bahir Dar Zurya Amhara 16 17 33 Debub Achefer Amhara 6 6 12 Endemehoni Tigray 16 8 24 Jijiga Somali 11 12 23 Wolmera Oromia 14 6 20 TOTALS 113 71 184 FIGURE 13. BASELINE EMPLOYMENT AT INTAKE TABLE 10. ESTIMATED INCREASE IN BASELINE TO ENDLINE EMPLOYMENT RATE, BY REGION MONITORING YCS 2018 OS Afar 316% 282% 824% Amhara 190% 128% 326% Oromia 138% 114% 140% SNNPR 458% N/A 592% Somali 178% N/A 174% Tigray 389% 292% 493% 45 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity FIGURE 14. BASELINE EMPLOYMENT BY REGION TABLE 11. CHANGES IN EMPLOYMENT RATES FOR ALL, MALES, AND FEMALES MONITORING YOUTH COHORT STUDY 2018 OUTCOME SURVEY ALL BASELINE 23% 48% 23% ENDLINE 59% 69% 74% DIFFERENCE 256% 144% 322% MALES BASELINE 29% 54% 29% ENDLINE 61% 79% 78% DIFFERENCE 210% 146% 269% FEMALES BASELINE 16% 40% 16% ENDLINE 57% 57% 69% DIFFERENCE 360% 143% 431% 46 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity TABLE 12. EMPLOYMENT AND EDUCATION LEVELS AT BASELINE AND ENDLINE NO EDUCATION (N = 191) PRIMARY (N = 992) SECONDARY (N = 920) TVET (N = 96) TERTIARY (N = 58) TOTAL (N = 2,257) BASELINE 34.9% 28.4% 28.1% 24.5% 31.0% 28.5% ENDLINE 71.7% 87.3% 77.3% 81.3% 67.2% 81.1% DIFFERENCE 205.5% 307.4% 275.0% 331.6% 216.9% 284.7% TABLE 13. EMPLOYMENT AND YEARS OF AGE 15–17 (N = 73) 18–21 (N = 805) 22–26 (N = 958) 27+ (N = 421) TOTAL (N = 2,257) BASELINE 20.0% 25.3% 31.9% 37.8% 28.4% ENDLINE 54.8% 75.3% 83.0% 92.6% 81.1% DIFFERENCE 274.0% 297.5% 260.2% 245.1% 285.7% FIGURE 15. MAP OF WOREDAS’ YES-HUBS 47 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity TABLE 14. STUDIES, ASSESSMENTS, AND MONITORING DATA COLLECTED BY BPY COMPONENT USE FREQUENCY STUDIES/ASSESSMENTS41 Baseline Survey Establish characteristics of youth at enrollment At each intake Annual Outcome Survey Track Beneficiary Employment Outcomes Annually Factors Affecting Youth Employment in Ethiopia Report Factor analysis of 2017 EPMEs Youth Cohort Study data to determine factors associated with employment for males and females 2019 Labor Market Survey Determine viable employment opportunities in local environment Project start-up Rapid Woreda Assessments Assessments on youth characteristics and challenges PYs 1-3 TVET Labor Market Assessment Determine relevance of TVET short-courses for local market PY 2 EMPLOYABILITY ASSESSMENT TOOL TVET Tracer Study Examine the employment status of TVET Graduates who completed WRN! Module January 2019 Fact sheets Brief regional level or project level timely facts Occasional Briefers Brief customized information which is distributed to targeted audiences Occasional M&E PERFORMANCE MONITORING SYSTEM TOOLS Youth intake Profile Data Collection Tool Collect participant characteristics and demographic information Ongoing at intake Youth Registration Data Collection Tool Determine enrollment in each BPY intervention Ongoing Activity Attendance Data Collection Tool Determine hours of services received Ongoing Employment/Job Placement Tool Determine beneficiary employment status Quarterly Rapid Assessment Tool Collect data for woreda Rapid Assessments PYs 1-3 41 As of September 2019, several special reports/studies were in preparation by SC for completion within the year: POTENTIAL Gender Assessment Report; Overview of the 2019 Study on the State of Unsafe Migration in Ethiopia; Assessment of POTENTIAL’s Youth Business Mentorships; and Study on the Catalytic Role of Youth Economic Social Groups Toward Improving the Livelihood of Rural Youth. 48 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity TABLE 15. COMPONENTS OF BPY EVIDENCE DISSEMINATION AND UTILIZATION MEETINGS/WORKSHOPS Woreda-Level Review Meetings Review woreda performance data, successes, challenges and adaptations Quarterly Regional Level Review Meetings Review regional performance data, successes, challenges and adaptations Quarterly Annual Experience-Sharing Visits Share best practices to regional and national level govt. officials and partner leads Four Visits of high-level government delegations and donors To showcase the successes and learnings of the Activity for policy advocacy Different Occasion M&E Capacity Workshop Build skills and best M&E practices 2016, 2017, 2018 Adaptation, Planning and Leveraging Workshop Building strategies for sustainability April 2019 Final Lessons Learned Workshop Final BPY workshop sharing lessons learned October, 2019 Planning workshops Plan future project activities Annual REGULAR REPORTS Quarterly Reports to USAID Present quarterly project activities; performance monitoring data, success stories, challenges, achievements and plans Quarterly Annual Report to USAID Present project activities; performance monitoring data, success stories, challenges, achievements and plans by project year Annually OTHER DISSEMINATION MECHANISMS Print success stories Inform communities and youth workforce development stakeholders of project achievements Ongoing Radio presentations Public information Ongoing Video Success Stories Help mobilize local communities and institutions Twice Professional Meetings Share BPY data and experiences in professional fora As available Hosting Visits On-site sharing of BPY activities, achievements, and challenges with USAID and other stakeholders As available 49 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity Annex II: PVH Case Study Company Profile PVH, one of the globe’s largest apparel manufacturers, and recipient of the 2018 Corporate Excellence Award for US companies from the U.S. Department of State, established a large clothing manufacturing facility in Hawassa Industrial Park called PVHAM, which currently employs approximately 25,000 local workers, the majority of whom are young women. Following its 2016 opening, 18 additional textile companies have also opened in Ethiopia. According to PVHAM’s Human Resources Manager, interviewed for this evaluation, SC/Hawassa Regional Office was the first to approach them for establishing a BPY partnership. He stated that “PVHAM is open to new ideas and approaches to improve the skills of employees and work with others in that area.” Key Findings Trainers vs. Teachers: PVHAM found local TVET staff ill-suited for workplace-based training. The company used “trainers” from Hawassa University and a local TVET to train its employees. However, they found them acting more like university lecturers or teachers, focusing on theoretical rather than practical skills. An interviewee stated that, “The company wasted a huge amount of money and time to train employees that way. The company did not benefit from that…We need trainers more than teachers. Good trainers are more like coaches [who can engage and motivate the workers].” Job Market Demand: TVETs training is not market linked or driven. The training that youths receive from TVETs and universities does not meet employer market demand. The specific skills companies like PVHAM need is different from the skills training they receive in those institutions. The skills training lacks a practical approach; it focuses on theory. Partnerships: The local TVET has appeared unresponsive to concrete offers of support and partnership. PVHAM has attempted to establish a functioning partnership with the TVET but found it “a one-way relationship.” The HR Manager explained that at one point, the local TVET asked PVHAM to hire their graduates. The company said that graduates could send their CVs via email, but the TVET responded that students do not have email address. Representatives from PVHAM visited the campus and found a computer room with only non-functional computers. PVHAM mobilized other companies involved in similar businesses in the industry park to donate computers and also assigned a computer trainer to the project. Despite this action, no one from the TVET was able to follow-up and the computer lab never materialized. BPY Youth: BPY beneficiaries may be better motivated than their peers. The HR Manager indicated that youths who have been through BPY and are employed at PVHAM are better prepared than other young employees. On the other hand, the Manager said that overall, the younger employees may lack motivation to work. He explained, “It is easy to motivate a hungry person to eat. But it is difficult to motivate a person who is not hungry to eat.” Corporate Social Responsibility: The synergistic relationship with BPY has generated follow￾on youth workforce development resources. In line with its Corporate Social Responsibility program. PVH granted the SC Hawassa Area Office Birr 15,000,000 to replicate the BPY model in selected woredas in SNNP Region. It provided scholarships for 450 female employees to attend skills training in their own areas of interest. It also encourages female employees to save money, providing them double the amount they save. 50 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity Annex III: Evaluation Statement of Work I. Purpose and Use of the Evaluation USAID/Ethiopia developed this Statement of Work for an independent performance evaluation of the USAID/Ethiopia’s Building the Potential of Youth. The findings of this evaluation help understand the performance of the Activity and will inform future similar programming on youth economic opportunity/employment. The primary user of the findings of this evaluation is USAID. Activity Summary Information Activity Name USAID’s Building the Potential of Youth Cooperative Agreement AID-663-A-15-00006 Start Date – End Date Jan 1, 2015 –June 30, 2020 Total Estimated Cost $ 20,326,954 Activity Funding DA, Higher Education Implementing Partner Save the Children Activity AOR Tahir Gero II. Background Information A. Problem and Theory of Change Almost 30 percent of the Ethiopian population continues to live below the poverty line, with a per capita income of $170. Changes in the national economy have been coupled with changes in the workforce demographic. With over 93 million people, it is the second most populous country in Africa, and almost half the population falls into a category of “Hungry” (20 million) and “Pastoral” (15 million). This population is beset by risky farming enterprises, variable rainfall, and accounts for half the nation’s land stock and responsible for 90 percent of meat exports. The government last conducted a nation-wide labor survey in 2005. The employment rate was 76.7 percent for individuals over the age of 10. The employment rate was higher for men overall, at 84.7 percent, than it was for women, at 69 percent; this trend was seen not only in aggregate but in each age group. The youth population aged 15-24 years recorded the highest unemployment rate (7.7 percent). Most importantly, the survey reported lower employment rates among the literate population than illiterate (68.7 percent vs. 81.4 percent). In Pastoralist areas, long term studies using up to 80 years of data have looked at trends in livelihoods. These have shown that larger and most middle income pastoralists have gained larger herds over time and become richer and more commercial, while some middle and lower income pastoralists have decreased their herd size (or lost them altogether) and become more impoverished. USAID strategy in Pastoralist areas is based on this, providing assistance to the successful livestock economy, but also assisting those “Transitioning Out of Pastoralism,” who are mostly moving to urban or peri-urban areas or seeking to migrate out of pastoralist areas. 51 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity The youth in this group have the best Building the Building the Potential of Youth of Youth and need to pick up skills, which will help them to adapt to livelihoods in their own areas or elsewhere. The youth bulge in Ethiopia presents another specific challenge to the education system. An estimated 3 million primary school-aged children and over 20 million youth are outside of the formal school system, and a large proportion of out-of- school youth have had no education (84 percent in rural areas; 33 percent in urban areas). Ethiopia has one of the highest urban youth unemployment rates at 50 percent and there is a high rate of youth under-employment in rural areas, where nearly 83 percent of the population resides. Thus, the need to address skills training for out-of-school youth is urgent. Regrettably, secondary, tertiary, and vocational institutions suffer from low quality of preparing students for the labor market and employment. Adult education programs are underfunded in a nation where there are an estimated 30 million illiterate adults. The GTP sets a target of 95 percent adult literacy over the next five years. The current literacy rate is 39 percent. This target is not likely to be achieved. A majority of Ethiopians continue not to benefit from formal education and/or training, especially marginalized populations, such as pastoralists, those living in conflict-prone areas or remote areas, and people with disabilities. The majority of Ethiopian youth are working in some capacity. Almost three-fourths of 15-24 year-olds are employed, while less than one-fifth are actively involved in education or training. Jobless rates, which take into account unemployed youth actively seeking work and youth not seeking work due to discouragement or disability, are at 18 percent for youth. However, when disaggregated by adolescents and young adults, young adults (whether employed or seeking employment) are more represented in the labor force and less represented in education or training. Youth in rural areas face specific challenges in part due to the national policy on land ownership. All land was nationalized and redistributed through the socialism movement in 1975 and land cannot be sold or exchanged; however, it is inheritable and able to be sharecropped, and the GOE allows long leases for commercial farming. This situation has resulted in shrinking areas of land for rural youth. The average rural landholding has fallen from .5 ha in the 1960’s to .21 ha in 2005; younger and female-led households have average land holdings even less than that. In addition, rural labor issues often go unmeasured and undocumented. Rural youth face additional barriers in pursuing livelihoods due to the seasonality of casual farm work. Large-scale farmers’ needs for labor are highest at times when subsistence farmers are harvesting as well, causing a mismatched labor cycle. Development of casual employment opportunities in non-farm economies such as construction, tree cutting, and driving could provide supplemental earnings to the poor and very poor; however, non-farm wage jobs are often the most difficult to enter for youth. As explained above, pastoralist communities are splitting into fewer larger herders who are commercially producing livestock, and those transitioning out of pastoralism. It is the latter group which requires assistance in developing skills and opportunities for livelihoods outside of pastoralism, in particular in small and medium￾sized towns and the expanding peri-urban areas like Gode town, which are adding non-agro￾livestock jobs and wage employment to local economies. 52 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity In supporting Ethiopia’s ambitious socioeconomic agenda, the United States Agency for International Development (USAID) launched the Building the Potential of Youth (YP) Activity in 2015 with a focus on providing unemployed and underemployed youth ages 15 to 29 with tools needed to succeed, while building the ability of national and local institutions to deliver demand-driven soft and hard skills. Designed to help youth in rural areas and towns obtain the abilities, knowledge, and social capital to increase their income and long-term economic self￾sufficiency, this support aims at providing rural youth with life skills training alongside vocational and entrepreneurship development activities through partnerships with training, small business, and microfinance institutions. The five-year (January 2015 – June 2019), US$20.3 million, YP Activity is implemented under a Cooperative Agreement (# AID-663-A-15-00006) between USAID and a group headed by Save the Children, Inc. (SC), including Education Development Center, Inc. (EDC), HUNDEE￾Oromo Grassroots Development Initiative, Professional Alliance for Development in Ethiopia (PADet), Relief Society of Tigray (REST), and Facilitator for Change Ethiopia (FCE). B. Activity Goal and Objectives The purpose of the USAID’s Building the Potential of Youth Activity is to provide Ethiopian youth (ages 15-29) in rural areas and towns with access to workforce development/livelihood support and resources, tailored to their specific needs and market demand, so that they can a) achieve increased income and b) strengthen skills, knowledge and social capital required to achieve economic self-sufficiency over the longer-term. USAID’s Building the Potential of Youth proposed strategies and approaches respond to capacity needs of unemployed and underemployed youth, leverage existing youth development activities, and contribute to improving the quality and accountability of local training and business service providers by ensuring existing entities work in concert. • By using targeted assessments, the USAID’s Building the Potential of Youth team will ensure training and service providers consider market relevance and target curricula and programs to emerging skills needs of youth participants. • By focusing on skills building, will offer tailored technical and life skills training to youth to create more viable livelihood prospects. • By expanding approaches and offerings in Work-Based Learning, including employer visits, organized job- shadowing, short-term employment, and internships, both learners and employers benefit from training and supervisory support, leading to workforce-ready and employable youth with practical experience. • By implementing “Hubs,” USAID’s Building the Potential of Youth will build Youth Service Provider Networks and provide a cost-effective approach to enhance support service access for remote communities and disadvantaged groups. The Results Framework of the Activity is included in the Annex section of this document. 53 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity III. Evaluation Questions Table 1 below outlines the evaluation questions with possible data sources. The evaluation team must present the methods of analysis for each evaluation questions when it submits the evaluation design/work plan. Questions (in order of priority) Suggested Data Sources Method (s) of Analysis Notes Geographic focus and illustrative sample/selection INDIVIDUAL-FOCUSED QUESTIONS 1 To what extent do youth participating in the Building the Potential of Youth have improved employment (including self￾employment) status? How do these outcomes vary by participant characteristics and partner? Secondary analysis of SC and partners’ data from intake and exit participant surveys [SC Youth Profile Data Collection Tool] To be completed by the evaluation team Aggregate and analyze data on all participants, or a sub-sample of participants. 2 To what extent do youth participating in the Building the Potential of Youth have increased income? How do these outcomes vary by participant characteristics and partner? Secondary analysis of SC and partners’ data from intake and exit participant surveys [SC Youth Profile Data Collection Tool] Aggregate and analyze data on all participants, or a sub-sample of participants. 3 What are the outcomes, [measured in income, productive assets, and debt level and employment quality] for youth receiving microfinance loans facilitated by the Building the Potential of Youth Activity, How do these outcomes vary by participant characteristics and partner? Primary data collection with microfinance recipients. Secondary analysis of SC and partners’ data (to the extent available). A sample of 80 recipients across 6 Regions and 18 Woredas for primary data collection and analysis. 4 To what extent did the Building the Potential of Youth Activity equitably target male and female beneficiaries, and how comparable are the results between male and female beneficiaries? Analysis of secondary and primary data. SYSTEM-FOCUSED QUESTIONS 5 To what extent have the “YES￾Hubs” implemented by the Building the Potential of Youth Activity increased access to quality employment-related services (e.g., job facilitation, business development, entrepreneurial training, connection to resources) for participating youth? Sample-based primary data collection from YES-Hub stakeholders, including youth facilitators, youth participants, partner staff. A sample of 30 Yes Hubs (stratified by type of Hub) across 6 Regions and 18 Woredas for primary data collection and analysis. 54 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity Questions (in order of priority) Suggested Data Sources Method (s) of Analysis Notes Geographic focus and illustrative sample/selection 6 To what extent have small grants provided TVET increased the availability, quality and relevance (in terms of employment/income) of training for youth? Primary data collection (including baseline) from Training Institute staff and employment partners, and youth participants. Secondary analysis of SC and Training Institute administrative records, as available. A stratified sample of 18 Training Institutions across Amhara, Tigray, Oromia, Somali, Afar, and SNNPR regions. 7 To what extent and in what manner are public-private partnerships providing quality training and/or employment opportunities for participants in the Building the Potential of Youth Activity? Primary data collection with a sample of employment partners. Secondary analysis of SC and partners’ partnership records. A purposeful stratified sample of 36 partnerships across 6 Regions and 18 Woredas for primary data collection and analysis. 8 What is the operational and service delivery capacity of YES-Hub? How sustainable (operationally and financially) is the YES-Hub likely to be post-USAID support? Primary data collection focused on YES-Hub, utilizing organizational capacity and sustainability assessment frameworks. 9 To what extent has the YP Activity been evidence-based to inform USAID and its development partners on the appropriateness and effectiveness of its interventions. Secondary data collection and analysis on evidences generated on youth workforce interventions and how it was communicated with development partners. 10 What have been the most significant successes and challenges for the design, implementation and M&E for the Building the Potential of Youth Activity? Summary analysis of secondary and primary data collected. Focus on GOE relevant sector engagement and partnership Regional ownership and capacity and status of likelihood of sustainability. 55 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity IV. Evaluation Design and Methodology A. Evaluation Design The evaluation team will be responsible for developing an evaluation strategy and methodologies that include a mix of qualitative and quantitative data collection and analysis. The team should review the evaluation questions matrix presented in the above table and present a revised one with any other relevant additions such as methods of analysis. The methodology will be presented as part of the draft work plan as outlined in the deliverables below and included in the final report. The methodology proposed must comply with USAID’s Evaluation Policy (http://www.usaid.gov/evaluation/policy) and most importantly, it should follow the methodology used during the baseline survey of the same Activity. Methodology strengths and weaknesses should be identified as well as measures taken to address those weaknesses should be identified and discussed. B. Methodology (i) Data Collection and Sources The qualitative data collection methods should include, but not limited to literature review, key informant interviews, semi-structured interviews, focus group discussions with beneficiaries, and direct observation. The quantitative data will largely come from secondary sources Activity monitoring data (ii) Data Analysis While developing the work plan, the team is also expected to provide a detailed data collection and data analysis plan; for example, what quantitative and qualitative analysis techniques will be used, how focus group responses will be documented and analyzed, etc. Very insightful or special description of interviewees and discussants should be quoted as appropriate to highlight findings. All data collected and presented in the evaluation report must be disaggregated, as appropriate, by sex, age [15-19, 20-24, and 25-29], geographic area, educational attainment and ethnicity. As baseline survey was conducted on the Activity the evaluations team’s analysis in this evaluation should exhibit changes the Activity brought up on the beneficiaries as compared to the baseline. V. Existing Information Sources Existing data and Activity information will be made available to the contractor by USAID, SC, and its partners. A list of potential documents for the contractor to review is presented below. 1. Baseline survey (Social Impact survey report) 2. Activity original and amended description (if any) 3. Activity Work Plan 4. Activity reports 5. Activity PMEP 6. Three consecutive outcome surveys 56 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity 7. Employer satisfaction survey 8. Out-of-school literacy (OLS) and numeracy assessments, implemented by SC and its partners. 9. Pre & post- workforce readiness assessments, implemented by SC and its partners. 10. Technical Training Institute (TTI) records 11. Agreement documents between Building the Potential of Youth partners and employers 12. Records of training attendance 13. Different operations research by the Activity VI. Deliverables and Reporting Requirements 1. In-briefing: Within 48 hours of the availability of the evaluation team in EPMES’s Contractor Office, the evaluation team will have an initial introductory meeting with USAID/Ethiopia’s Program Office and Education and Youth Office for the team to have an understanding of the Activity’s interventions and also to establish a common understanding of what the evaluation is all about and the expectations from it. 2. Evaluation Work Plan/Inception Report: Following this, the evaluation team shall develop evaluation work plan/evaluation design and make a presentation to USAID within seven (7) working days of the initial introductory meeting. This work plan/inception report will include: (a) the overall evaluation design, including the proposed methodology, data collection and analysis plan, and data collection instruments; (b) a list of the team members and their primary contact details while in￾country, including the e-mail address and mobile phone number for the team leader; and (c) the team’s proposed schedule for the evaluation. USAID offices shall take up to three work days to review and consolidate comments through the EPMES COR. Once the evaluation team receives the consolidated comments on the work plan/inception report, they are expected to return with a revised work plan/inception report within two work days. The revised work plan shall include the list of potential interviewees and sites to be visited. Within two work days of the receipt of the revised work plan/inception report, EPMES’s COR shall provide approve to the team to proceed with the field work. 3. Fieldwork Debrief: The Team or the Contractor is expected to provide the COR for EPMES with periodic updates of the field work. If desired or necessary, weekly briefings by phone can be arranged with the Program Office and the Education and Youth Office to provide updates on field progress and any problems encountered. Immediately after the team’s completion of the fieldwork, the team shall provide a debrief to USAID to discuss on and learn about the field level data collection experiences as well as the evaluation team members’ preliminary impression on evaluation findings. 4. Final Exit Presentation (PowerPoint Presentation) to USAID and relevant partners that will include a summary of key findings and key conclusions as these relate to the evaluation’s questions and recommendations to USAID. To be scheduled as agreed upon during the in-briefing, and prior to expats departure from Country.A copy of the PowerPoint file will be provided to the Program Office, at least three-days before the final exit presentation day. The COR for EPMES shall compiles comments 57 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity from participants in this presentation and submit it to the Contractor for consideration during the preparation of the report. 5. Draft Evaluation Report: The content of the draft evaluation report is outlined in Annex A, and all formatting shall be consistent with the USAID branding guidelines. The focus of the report is to answer the evaluation questions and may include factors the team considers to have a bearing on the objectives of the evaluation. Any such factors can be included in the report only after consultation with USAID. The Contractor will submit the draft evaluation to the Program Office within 10 working days after exit presentation and should incorporate comments made during the exit presentation. USAID’s Program Office, Education and Youth Office and other partners will have 15 working days to review and comment on the draft report and the Program Office shall submit consolidated comments to the Contractor. The Contractor will then have 10 working days to make appropriate edits and revisions to the draft and re-submit the revised final draft report to USAID. The Program Office, Education and Youth Office and other partners will have 10 working days after the submission of the second revised draft to again review and send any final comments. 6. Final Evaluation Report will incorporate final comments provided by the Program Office. The length of the final evaluation report should not be more than 45 pages, excluding Annexes and Executive Summary. The Contractor should submit the final report to the Program Office within 10 days of receipt of comments. The Final Evaluation Report submission should also include a two-pager briefer on key qualitative and quantitative findings and conclusions relative to the evaluation questions—to be given to the appropriate government counterpart(s) so that they have the opportunity to review evaluation findings and share them with the larger community. 7. All Activity data and records will be submitted in full and shall be in electronic form in easily readable format; organized and fully document for use by those not fully familiar with the Activity or evaluation; and owned by USAID and made available to the public, barring rare exceptions, on the USAID Development Experience Clearinghouse (http://dec.usaid.gov). VII. Evaluation Team Composition The evaluation team shall consist of a team leader who leads the team and three other experts with high level experience in workforce development. All experts must be fluent in English and have strong writing skills. The Ethiopian experts should have experience with youth workforce in Ethiopia and monitoring and evaluation. The Ethiopian experts should also be proficient in one or more of the local languages where this Activity is operating. USAID may propose internal staff members from USAID/Washington or other Missions to accompany the team during site visits or participate in key parts of the evaluation (specific event participation to be determined in conjunction with the contractor and the team leader), and they are expected to provide written inputs to the draft report. 58 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity A statement of potential bias or conflict of interest (or lack thereof) is required of each team member. Team Member Qualifications and Specific Responsibilities 1. Team Leader (One): The team leader (expat or local) should have at least 10 years work experience in Youth Workforce Development, in conducting evaluations on Workforce Development or similar interventions in Ethiopia or in any other countries, preferably in African countries. The evaluation team leader must have strong team management skills, and sufficient experience with evaluation standards and practices to ensure a credible product. The team leader must also be fluent in English and have strong writing skills. The team leader, in consultation with other team members, will be responsible for team coordination and performance and for ensuring the timeliness and quality of deliverables. The team leader is also responsible to ensure that the evaluation report meets the USAID Evaluation Report Standards (see Annex A.) 2. Local Experts (Team Members-Three): BA degree (MA/MSc. Preferred) in Educational Planning, Educational Psychology, Measurement and Evaluation, Sociology, Demography, Economics, Statistics or another related social science field. Local experts should have at least 5 years of relevant work experience and strong understanding of project cycle management, evaluation design and data collection and analysis techniques, survey coordination, and gender issues in Ethiopia. Survey experience in projects implementing youth, livelihood, training and employment, as well as familiarity with USAID development projects and evaluation policies, are desirable. Sound technical and evaluation experience with youth workforce development projects is expected. Facility with Microsoft Office is required and experience with statistical analysis software (such as SPSS) and qualitative analysis software are also desired. The following are the specific duties and responsibilities of the team leader. • Managing a team of three local team members, including leading team planning meetings, assigning roles and responsibilities, assessing the team performance and reporting back to COP and requesting necessary logistics for the execution of this evaluation • Developing the evaluation design with support from the local consultants and assigned EPMES staff • Lead and participate in the preparation of presentation on the evaluation design. • Developing a sampling frame for primary qualitative and/or quantitative data collection • Leading and actively participating in data collection, including travelling to different parts of Ethiopia for data collection and conducting interviews and focus group discussions with stakeholders and households. • Conducting secondary analysis of activity monitoring data, government data and relevant data from other sources. • Leading the report writing with the full involvement of the evaluation team members. • Ensuring that deadlines are met in accordance with the contractual schedule of deliverables. 59 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity • Ensuring high quality and usefulness of evaluation outputs, given technical expertise in the subject matter being evaluated. • Reporting on the progress to the EPMES Chief of party • Guiding the team on data analysis techniques and conducting analysis. • Preparing the findings, conclusions and recommendations matrix • Presenting the results to the client upon finalization of data analysis. The local experts will have the following specific duties and responsibilities. • Participate in the end line evaluation team planning meeting. • Review program background materials. • Participate in briefings with USAID/Ethiopia. • Contribute to design of data collection instruments (for example, for focus group discussions, key informant interviews, and semi-structured questionnaires). • Assist the Team Leader in recruiting data collectors as necessary. • Train and coordinate data collectors as necessary. • In consultation with the Team Leader, identify and mobilize focus group discussants and key informant interviewees; take notes of discussions and interviews. • Select respondents for the evaluation as necessary. • Check if questionnaires are properly field by data collectors. • Assist in oral interpretation of interviews (Amharic to English; English to Amharic). • Merge survey data and make data ready for analysis. • Assist team leader and other experts in data analysis and review of data quality. • Contribute written technical material (text and/or data presentations) to team internal products and the draft assessment report. • Assist in report editing. • Other activities as instructed by the Team Leader. The contractor may also hire other low-level personnel as required and appropriate with defined responsibilities. VIII. Evaluation Schedule The estimated time period for undertaking this evaluation is 85 working days including time for review of products. The ideal available time for the evaluation team is between beginning of August 2019 to December 19, however, the date will be finalized between USAID and the EPMES Contractor. The evaluation team is required to work six days a week, but with no premium payment for sixth day. The team is required to travel to selected provinces in each region where program activities are being implemented. At least 50% of the consultants’ time will be spent outside Addis Ababa to conduct interviews with Activity staff, government partners, and Activity beneficiaries. The evaluation team will prepare an exit briefing and presentation of the findings, 60 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity which it will deliver to USAID staff and other stakeholders as appropriate before expat consultants depart Ethiopia. Following from this briefing, the consultants will put together the draft report. USAID and other stakeholders such as implementing partner will have two weeks to provide comments and suggestions to the report. Comments from USAID and stakeholders will be incorporated before the submission of the final draft. Illustrative Level of Effort (LOE) in person days Activity Team Leader [1] Experts (team members) [3] Total Expat Travel to Ethiopia 2 0 2 Introductory meeting with USAID 0.5 1.5 2 Introductory meeting with implementing partner (s) 0.5 1.5 2 Work on the evaluation inception report (work plan, evaluation design, methodology, and data collection tools) 16.5 49.5 66 Work on the logistics 2 6 8 Prepare power presentation 2 6 8 Presentation to USAID on the Evaluation Design and Work Plan 0.5 1.5 2 Final Inception report submission to USAID 0.5 1.5 2 Field Work (Data collection in Addis and other areas. This includes any data collection tool test) 21 63 84 Field work experience debrief to USAID Data analysis, preliminary report and presentation preparation 15 45 60 Prepare preliminary finding (exit) PowerPoint 2 6 8 Internal presentation and addressing the comments 1 3 4 Final exit presentation to USAID and relevant partners (with PowerPoint presentation report) 1 3 4 Expat Depart County 2 0 2 Draft Evaluation Report Preparation 13 30 43 Final Evaluation Report (including a Two￾page Briefer) 6 6 12 Total 85.5 223.5 309 61 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity IX. Management Social Impact, the Contractor managing the Ethiopia Monitoring and Evaluation Service (EPMES) activity will identify and hire the evaluation team, pending the Contracting Officer’s Representatives (COR’s) and relevant technical office’s concurrence and CO approval for the rate, assist in facilitating the work plan, and arrange meetings with key stakeholders identified prior to the initiation of the fieldwork. The evaluation team will organize other meetings as identified during the course of the evaluation, in consultation with EPMES’s Contractor and USAID/Ethiopia. The EPMES Contractor is responsible for all logistical support required for the evaluation team, including arranging accommodation, security, office space, computers, Internet access, printing, communication, and transportation. The evaluation team will report to the Ethiopia Monitoring and Evaluation Service (EPMES) Contractor, Social Impact. The EPMES Contractor is responsible for all direct coordination with the USAID/Ethiopia Program Office through the EPMES COR. From a technical management perspective, the evaluation team will work closely with Tahir Gero, USAID’s Building the Potential of Youth AOR in the Basic Education Services Office. In order to maintain objectivity, all final decisions about the evaluation will be made by the Program Office. X. Logistics The contractor will be responsible for all travel and logistics associated with conducting the evaluation. USAID/Ethiopia and SC, the implementing partner, may provide the evaluation team with assistance in logistics as needed and appropriate. Annex A: USAID Evaluation Report Standards 1.Identify the evaluation as either an impact or performance evaluation per the definitions in ADS 201. 2.Include an abstract of not more than 250 words briefly describing what was evaluated, evaluation questions, methods, and key findings or conclusions. The abstract should appear on its own page immediately after the evaluation report cover. 3.Include an Executive Summary 2–5 pages in length that summarizes key points (purpose, background, evaluation questions, methods, findings, conclusions). 4. State the purpose of, audience for, and anticipated use(s) of the evaluation. 5.Describe the specific strategy, project, activity, or intervention to be evaluated including (if available) award numbers, award dates, funding levels, and implementing partners. 6.Provide brief background information. This should include country and/or sector context; specific problemor opportunity the intervention addresses; and the development hypothesis, theory of change, or simply how the intervention addresses the problem. 62 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity 7.Identify a small number of evaluation questions. 8.In an impact evaluation, identify questions about measuring the change in specific outcomes attributable to a specific USAID intervention. 9.Describe the evaluation method(s) for data collection and analysis. 10. Describe limitations of the evaluation methodology. 11. In an impact evaluation, use specific experimental or quasi-experimental methods to answer impact evaluation questions. 12. Include evaluation findings and conclusions. 13. If recommendations are included, separate them from findings and conclusions. 14. Address all evaluation questions in the SOW or document approval by USAID for not addressing an evaluation question. Include the following annexes: • Evaluation SOW. If the SOW is revised, the evaluation report should include the updated SOW as an Annex rather than the original SOW. • A description of evaluation methods (if not described in full in the main body of the evaluation report). • All data collection and analysis tools used, such as questionnaires, checklists, survey instruments, and discussion guides. • All sources of information—properly identified and listed. • Any “statements of differences” regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team. • Signed disclosures of conflicts of interest from evaluation team members. • Abridged bios of the evaluation team members, including qualifications, experience, and role on the team. 15. Include enough information on the cover of the evaluation report so that a reader can immediately understand that it is an evaluation and what was evaluated. The evaluation cover should: • Include a title block in USAID light blue background color. • Include the word “Evaluation” at the top of the title block and center the report title underneath that. The title should also include the word “evaluation.” • Include the following statement across the bottom of the cover page: “This publication was produced at the request of the United States Agency for International Development. It was prepared independently by [list authors and organizations involved in the preparation of the report].” For an internal evaluation team, use the following statement: “This publication was produced at the request of [USAID/Mission] and prepared by an internal evaluation team comprised of [list authors and affiliation].” • Feature one high-quality photograph representative of the project being evaluated and include a brief caption on the inside front cover describing the image with photographer credit. • State the month and year of the report. • State the individual authors of the report and identify evaluation team leader. 63 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity Annex B: USAID Criteria for Quality Evaluation • Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity. • Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. • The Executive Summary should present a concise and accurate statement of the most critical elements of the report. • Evaluation reports must address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. • Evaluation methodology must be explained in detail and sources properly identified. • Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. • Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. • If evaluation findings address person-level outcomes and impact, they should be assessed for both males and females. • If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. Annex C: Results Framework 64 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity Annex D: Geographic Coverage of the USAID’s BPY Intervention The following table outlines the 30 woredas that will be targeted by USAID’s Building the Potential of Youth Program. Amhara Tigray Oromia SNNP Somali Afar Woreda Woreda Woreda Woreda Woreda Woreda Danegela Ofla Limu-Bibilo Endegene Jawi Raya Alamata Kofele Merab Azernet Semin-Achefer Endamehoni Ada’a Yem Special Woreda Productive Debube-Achefer Welmera Hawassa Zuria Bahir Dar Zurya Sinana Loma Dera Agarfa Liban Chiquala Yabello Jijiga Chifra Pastoral Dire Babile Dubti Gode Asayita Kelafo 65 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity Annex E: Targets by Region and Phase Target by Phase Year 1 Year 2 Year 3 Year 4 Year 5 Total Phase 1 Woredas – 8 initial Amhara 600 600 600 700 500 3,000 Tigray 600 600 600 700 415 2,915 Somali 600 600 600 600 122 2,522 Oromia 600 600 600 700 500 3,000 Phase 1 total 2,400 2,400 2,400 2,700 1,537 11,437 Phase 2 Woredas – 11 additional Amhara 600 600 700 500 2,400 SNNPR 900 900 1050 750 3,600 Afar 600 600 700 500 2,400 Somali 300 300 350 250 1,200 Oromia 900 900 1050 750 3,600 Phase 2 total --- 3,300 3,300 3,850 2750 13,200 Phase 3 Woredas – 11 additional Amhara 600 700 500 1,800 Tigray 300 350 250 900 SNNPR 600 700 500 1,800 Afar 300 350 250 900 Somali 300 350 250 900 Oromia 1200 1400 1000 3,600 Phase 3 total 2,400 3,300 3,850 2,750 9,900 Total Target 2,400 5,700 9,000 10,400 7,037 34,537 66 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity Annex F: Final List of Target Woreda and Intake No. Region Potential Woredas Year 1 Year 2 Year 3 1 Amhara Debub Achefer X 2 Amhara Semin Achefer X 3 Amhara Bahir Dar Zurya X 4 Amhara Dangela X 5 Amhara Jawi X 6 Amhara Dera X 7 Oromia Ada’a X 8 Oromia Liben Chukala X 9 Oromia Limu-Bilbilo X 10 Oromia Kofele X 11 Oromia Sinana X 12 Oromia Agarfa X 13 Oromia Welmera X 14 Oromia Yabello X 15 Oromia Dire X 16 SNNPR Endegegn X 17 SNNPR Mirab-Azernet X 18 SNNPR Loma X 19 SNNPR Yem Special Woreda X 20 SNNPR Hawassa Zuria X 21 Tigray Raya Alamata X 22 Tigray Ofla X 23 Tigray Endemehoni X 24 Somali Jijiga X 25 Somali Babile X 26 Somali Gode X 27 Somali Kelafo X 28 Afar Chifra X 29 Afar Dubti X 30 Afar Asayita X Total 8 11 11 67 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity Annex IV: Data Collection Instruments KII Guide for Micro Finance Institution-MFI * Note for interviewer I. Establish Rapport • Greeting!!!; Introduce yourself; Tell who you work for: Social Impact • Tell purpose of visit: Conducting endline evaluation of USAID’s BPY Activity II. Obtain Written Consent III. Identify focal person for the KII together with LIP Name of Organization _______________ Interviewee Position: _______________ Date: _______________ Region: _______________ Woreda: _______________ Town: _______________ Sex: M / F Interviewer Name: _______________ Interview Code: _________________ I. General 1. How long have you been in this position? ________(years) 2. Tell me a little about your role in your organization? II. Loan (EQ4) 1. What are the sources of working/start-up capital for youths in this area? 2. What types of financial services does your organization provide to youths in this area? 3. What did your organization’s partnership with BPY implementing organizations (mention name of IP here) in providing loan to BPY youths looked like? 4. What were the procedures of your organization for screening youth applicants for loan access/service? Probes (if applicable) a) Could you tell us about the criteria used? b) What was the quality of application by BPY youth for loan? c) What are the reasons they are rejected (if any)? 5. Could you tell us roughly about the proportion of male and female BPY loan applicants and users? Was there variation in the number of male and female loan users? If yes, what are the reasons for variations? 6. Is there a mechanism that your organization uses to monitor the progress of loan users? If yes, what are they? If no, why? 7. Could you tell us about the debt level of youth who took loan from your organization? 8. Could you tell us the repayment status of BPY youths? What about their default status? 9. Could you tell us about the amount of loan per person your organization provides to BPY youth? 10. What are the challenges in providing loan to BPY youths in this area? 11. What are the lessons learned in relation to providing loans to BPY youths in this area? 12. What do you think BPY could have done to make the loan access to youths in a better way? 68 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity KII Guide for Woreda Level TVET/TTI * Note for interviewer I. Establish Rapport • Greeting!!!; Introduce facilitator and note taker; Tell who you work for: Social Impact • Tell purpose of visit: Conducting endline evaluation of USAID’s BPY Activity II. Obtain Written Consent III. Identify focal person for the KII together with LIP IV. All questions with * are taken from baseline evaluation TVET/TTI Name: _______________ Interviewee Position: _______________ Interview Date: _______________ Region: _______________ Woreda: _______________ Town: _______________ Sex: M / F Interviewer Name: _______________ General 1. How long have you been in this position? ________(years) 2. Tell me a little about your role in BPY project activities? 3. What is the role of the TVET/TTI in BPY implementation? I. Employment (EQ1) 1. What activities were undertaken by this TVET/TTI as BPY partner to improve employment status of youths? Probes (if applicable) a) Were the activities relevant to the needs/expectations of your institution and beneficiaries? How? If not, why? *1a) On a scale of 1 to 5, 5 being highest, how relevant were the training or courses for youth provided by your center/institute to the needs of the current job market? 1 2 3 4 5 *1b) Explain your answer: *1c) On a scale of 1 to 5, 5 being highest, how likely were unemployed youth that completed the training/course(s) to become employed during the project period? 1 2 3 4 5 *1d) Explain your answer: *1e) What are the most important factors that influence whether a graduate obtains improved employment or increased income (quality, relevance, access, other)? *1e) On a scale of 1 to 5, 5 being highest, how likely were employed youth that completed the course(s) to improve their employment circumstances (for example through promotion, increase in pay, or find a better job)? 1 2 3 4 5 *1f). Explain your answer: b) How did the TVET/TTI training as BPY partner contribute to access quality employment? (If yes) Which activities contributed more? How? If no, why? 69 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity 2. Are there any changes on beneficiaries’ employment status? Probes (if applicable) a) Was the change in employment increase/ decrease/ stay same? II. Gender Equity (EQ3) 3. What does the proportion of male and female trainees of this institution looks like? 4. What is the proportion of female from the BPY youths who received training in this institution? If there variation in gender, what are the reasons? 5. What are the mechanisms used to make sure gender equality is ensured? 6. Which training types are preferred by male or female? Why? *3a). Did your institution specifically reach out to female youth to access your programs and services? Yes No *3b). If yes, explain how? *3c). What are the barriers and challenges to reaching this population? *3d) Did your institution specifically help connect female youth to improved employment and/or income generating opportunities? Yes No *3e) If yes, explain how? *3f) What are the barriers and challenges to reaching this population? III. Small Grant (EQ4) 7. What kind of resources/supports were provided to your institution by BPY? (Baseline Question) *4a) If yes, (Check all that apply):  Work Ready Now (a)  “Afla” teen (b)  Be Your Own Boss (c)  Youth in Action (d)  Block Grant (e)  Labor Market Research (f) Public/Private Partnerships (incl. internships, job shadows, job fairs, etc.) (g)  Other: (i) *4b) On a scale of 1 to 5, 5 being highest, how would you rate the quality of the ToT and/or resources you received from the project so far? 1 2 3 4 5 *4c) Explain your answer (i.e, why did you rate it a 4, and not a 5 or 3?) *4d) On a scale of 1 to 5, 5 being highest, how relevant was the training and/or resources to the needs of the staff and students? 1 2 3 4 5 Explain your answer: 8. What type and amount of grants was provided to your institution? Probes (if applicable) a) For what purpose was the TVET/TTI used the grants received? b) Was the grant utilized for the intended purpose? If not why? c) Was the grant relevant to the beneficiaries skill needs? 70 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity d) Did the grant enhanced/ improved TVET’s/TTI’s capacity to deliver the intended skill training? 9. Did the grant helped TVET’s so that its graduates get better employment opportunities? How? If no, why? 10. Did the grants initiatives help the TVET’s/TTI’s to develop market driven curriculum/ training contents? If not why? 11. Were there other ways that BPY helped to develop market driven curriculum IV. Public-Private Partnership (EQ5) 12. What is the relationship your institution has with public and/or private organizations operating in this area? Did the BPY initiative play role in that? *8a) Did your institution reach out to unemployed youth to access your programs and services during the BPY project period? Yes No If yes, explain how? Did BPY help? *8b) What are the challenges to reaching this unemployed youth? *8c) Have you received any external assistance in reaching this population? *If yes, describe the assistance, from whom, if it was successful, and why or why not? *8d) Did your institution reach out to employed youth to access your programs and services? Yes No *8e) If yes, explain how? 8f) What are the barriers and challenges to reaching this population? *8g) Did your institution help connect unemployed youth to employment and/or income generating opportunities? Yes No *8h) If yes, explain how? *8i) What are the challenges to reaching this population? *8j) Did your institution help connect employed youth to improved employment and/or income generating opportunities? Yes No *8k) If yes, explain how? *8l) What are the challenges to reaching this population? V. YES-HUBs (EQ6) 13. Does your institution has any interaction with YES-HUBs established by BPY? If yes, what are they? If no, why not? VI. Evidence (EQ7) 14. Were there any practical experiences that BPY related evidences are generated, shared or utilized by your institution? If yes, what are they VII. Challenges and Lessons (EQ8) 15. What are the lessons learned during the course of BPY implementation? 16. What were the challenges encountered in relation to BPY during its implementation? (prob. For grant, training and process related challenges)? 71 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity KII Guide for YES-Hubs * Note for interviewer I. Establish Rapport • Greeting!!! • Introduce yourself • Tell who you work for: Social Impact • Tell purpose of visit: Conducting endline evaluation of USAID’s Building Potential of Youth Activities project II. Obtain Written Consent III. Identify focal person for the KII together with LIP Name of Organization _______________ Interviewee Position: _______________ Date: _______________ Region: _______________ Woreda: _______________ Town: _______________ Sex: M / F Interviewer Name: _______________ Interview Code: _________________ I. General 4. How long have you been in this position? ________(years) 5. Tell me a little about your role in your organization? II. YES-H (EQ6) 1. When was the YES-H established? How was the establishment process of the YES-H look like? How was the center selected? 2. Who manages the YES-H now? 3. What kinds of supports/resources were provided to the YES-H by BPY? Were there other support/resources they YES-H get other than BPY? 4. Is the YES-H providing service to youths now? If yes, what kinds of services are the YES￾H providing to youths in this area? 5. What is the contribution of the YES-H in improving quality (satisfying) employment for youths who uses the YES-H? 6. What is your opinion about financial and operation sustainability of YES-Hubs? (Probe: do you think the YES-H will sustainability provide service after the BPY?) 7. What is the role and importance of YES-Hubs in the overall intervention of BPY? 8. What are the provisions of YES-H for girls? Do girls have equal opportunity to access YES-H services? 9. What were the challenges to the establishment and operations of Yes-Hubs? (EQ10) 10. What lessons were learnt during the course of YES-Hub operation/implementation? (EQ10) 72 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity Focus Group Discussion Informed Consent Title: Building the Potential of Youth (BPY) Activity - Final Performance Evaluation Investigators: James Statman; Worku Ambelu; Argaw Korssa; Ermias Assefa; Mazengia Abera Sponsor: USAID/Ethiopia Introduction Hello, my name is----------. I am part of a team from Social Impact (SI) currently conducting an independent evaluation of the “Building the Potential of Youth Activity." SI is an international consulting company with its headquarters in Arlington Virginia, USA and with a Field Office in Addis Ababa, Ethiopia. SI works to improve development effectiveness around the world through evaluation, capacity building and strategic planning. The BPY Activity is a USAID-funded activity which provides youth ages 15-29 in rural areas and towns with access to workforce development/livelihood support and resources so they can achieve increased income and self-sufficiency. BPY is implemented by a consortium led by SC. This evaluation is intended to measure the achievements of this activity and to obtain opinions about how such an activity can be improved in the future. I would like to request you to read (or have read to you) this Consent Agreement. This is to make sure that you are fully informed about this evaluation. After I have introduced this evaluation to you and have gone through what is expected of you, I will ask you to sign or use your thumbprint if you agree to participate. SI Internal Review Board has approved this evaluation. We will give you a copy of this form if you would like. This consent form might contain some words that are unfamiliar to you. Please ask us to explain anything you may not understand. I want to be sure that you understand the purpose of this evaluation and your responsibilities before you decide if you want to be in it or not. Please ask me to explain any words or information that you may not understand. Information about the evaluation If you agree to be part of this evaluation, we are going to ask you and other youth who have participated in the program to be part of a group to discuss the BPY. We will also ask the group to talk about the successes and challenges BPY encountered and how the activity could be improved to achieve more significant results. The information you share will be kept confidential and will not be disclosed to anyone in a way that can be linked to you. Although we will share the opinions you and others give us in a report to other entities outside of the evaluation team, all your answers will be treated with confidentiality and will be anonymized in the report. Additionally, your decision to participate or not to participate in this evaluation will in no way affect the services you may currently receive or provide or the support you receive from BPY. You have the right to refuse to answer any questions or to stop participating in the group discussion at any time. Your participation in this evaluation will take about one hour. I will not write down your name on this form so that the answers you give cannot be linked to you. You have the right to tell whomever you choose about this evaluation. You may stop participating in this group at any point during our discussion. Again, I want you to be aware that accepting to participate or ending your participation will not affect the services you provide or the support you may be receiving from the BPY Activity. Possible risks We do not anticipate any significant risks to you or your household or your organization because of your participation in this evaluation. However, please note that should you choose to participate in this study, you will be taking time away from your regular activities, which may affect your routine tasks. I wanted you to be aware of these possible aspects of the interview that might affect your feelings before accepting to participate in the discussion. 73 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity Possible benefits The results of this evaluation are expected to inform USAID’s planning and decision-making, assess the results of BPY Activity, improve possible future programs to help Ethiopian youth better obtain job￾related skills that will enable them to become employed or self-employed and increase their income. Additionally, the results may be presented or disseminated at regional, national and international meetings to support planning aimed at mobilizing support for similar activities. The findings and recommendations from this evaluation will generate critical information that can be used by planners to determine and implement activities that support job training and support for rural youth. Your participation in this evaluation will therefore be essential to current and future similar programs. By participating in this evaluation, you will, however, get no immediate and direct personal benefit. If you decide not to participate in this evaluation You are free to decide if you want to participate in this evaluation or not. If you decide not to participate, we will accept your decision without holding anything against you. Your relationship with BPY Activity or other organizations that provide similar services or will use the evaluation results will not be affected at all. Confidentiality We will protect information about you and your involvement in this evaluation to the best of our ability. We will not record your name in our data collection tools or notes, but only in this consent form, which we will keep separately from the notes and transcripts of this interview. We will also not indicate your name in the any of the reports we prepare. We will not tell your peers, supervisors, family members, caretakers, or friends about your participation or about the information you give. Leaving the interview You may end your participation in the group discussion at any time. We will not hold anything against you should you choose to leave before the end of the discussion. Duration of interview We anticipate that this group discussion will take no more than 1.5 hours. If you have a question about the evaluation If you have any questions about this evaluation, you may contact Biruk Belayneh via his BBelayneh@socialimpact.com or phone number 0912503019. You can also contact the Social Impact Internal Review Board. The contact person is Danae Roumis Address is: 2300 Clarendon Blvd, Suite 1000, Arlington, VA 22201; phone number 703-465-1884; email address: irb@socialimpact.com. VOLUNTARY AGREEMENT If you would like to participate in this group discussion, please say ”Yes” to the following statement. I would happy to read it to you more than once if you would like. I certify that the nature and purpose, the potential benefits, and the possible risks associated with participating in this research have been explained to me. I also certify that I am at least 18 years of age at this time. Do you so certify? __________________________________________________ Yes……………………….1 No………………………..2. Date: ______________________ 74 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity Key Informant Consent Form Title: Performance Evaluation of USAID/Ethiopia’s Building the Potential of Youth Activity Investigators: Worku Ambelu; James M. Statman, Argaw Korssa, Mazengia Abera and Ermias Assefa Sponsor: USAID/Ethiopia Introduction Hello, my name is----------. I am part of a team from Social Impact (SI) currently conducting an independent evaluation of the “Building the Potential of Youth Activity." SI is an international consulting company with its headquarters in Arlington Virginia, USA and with a Field Office in Addis Ababa, Ethiopia. SI works to improve development effectiveness around the world through evaluation, capacity building and strategic planning. The BPY Activity is a USAID-funded activity which provides youth ages 15-29 in rural areas and towns with access to workforce development/livelihood support and resources so they can achieve increased income and self-sufficiency. BPY is implemented by a consortium led by Save the Children (SC). This evaluation is intended to measure the achievements of this activity and to obtain opinions about how such an activity can be improved in the future. I would like to request you to read (or have read to you) this Consent Form. This is to make sure that you are fully informed about this evaluation. After I have introduced this evaluation to you and have gone through what is expected of you, I will ask you to sign or use your thumbprint if you agree to participate. SI Internal Review Board has approved this evaluation. We will give you a copy of this form if you would like. This consent form might contain some words that are unfamiliar to you. Please ask us to explain anything you may not understand. I want to be sure that you understand the purpose of this evaluation and your responsibilities before you decide if you want to be in it or not. Please ask me to explain any words or information that you may not understand. Information about the evaluation If you agree to be part of this evaluation, we are going to ask you and other key informants about the interventions of the BPY Activity that you may know, such as capacity building, coordination and your perceptions of their results. We will also ask you about the successes and challenges BPY encountered and how the activity could be improved to achieve more significant results. The information you share will be kept confidential and will not be disclosed to anyone in a way that can be linked to you. Although we will share the opinions you give us in a report to other entities outside of the evaluation team, all your answers will be treated with confidentiality and will be anonymized in the report. Additionally, your decision to participate or not to participate in this evaluation will in no way affect the services you may currently receive or provide or the support you receive from BPY. You have the right to refuse to answer any questions or to stop the interview at any time. Your participation in this evaluation will take about one hour. I will not write down your name on this form so that the answers you give cannot be linked to you. You have the right to tell whomever you choose about this evaluation. You may stop participating in this interview at any point during our discussion. To be sure that we clearly understand your answers and opinions, we will, if you agree, record your comments. We will keep this recording confidential and erase it once we have written down your comments, if you do not want to be recorded, just tell us and we will not do so. Again, I want you to be aware that accepting to participate or ending your participation will not affect the services you provide or the support you may be receiving from the BPY Activity. Possible risks 75 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity We do not anticipate any significant risks to you or your household or your organization because of your participation in this evaluation. However, please note that should you choose to participate in this study, you will be taking time away from your regular activities, which may affect your routine tasks. I wanted you to be aware of these possible aspects of the interview that might affect your feelings before accepting to participate in the discussion. Possible benefits The results of this evaluation are expected to inform USAID’s planning and decision-making, assess the results of BPY Activity, improve possible future programs to help Ethiopian youth better obtain job￾related skills that will enable them to become employed or self-employed and increase their income. Additionally, the results may be presented or disseminated at regional, national and international meetings to support planning aimed at mobilizing support for similar activities. The findings and recommendations from this evaluation will generate critical information that can be used by planners to determine and implement activities that support job training and support for rural youth. Your participation in this evaluation will therefore be essential to current and future similar programs. By participating in this evaluation, you will, however, get no immediate and direct personal benefit. If you decide not to participate in this evaluation You are free to decide if you want to participate in this evaluation or not. If you decide not to participate, we will accept your decision without holding anything against you. Your relationship with BPY Activity or other organizations that provide similar services or will use the evaluation results will not be affected at all. Confidentiality We will protect information about you and your involvement in this evaluation to the best of our ability. We will not record your name in our data collection tools or notes, but only in this consent form, which we will keep separately from the notes and transcripts of this interview. We will also not indicate your name in the any of the reports we prepare. We will not tell your peers, supervisors, family members, caretakers, or friends about your participation or about the information you give. Leaving the interview You may end your participation in the interview at any time. We will not hold anything against you should you choose to leave before the end of the interview. Duration of interview We anticipate that this interview will take no more than 1.5 hours. If you have a question about the evaluation If you have any questions about this evaluation, you may contact Biruk Belayneh via his BBelayneh@socialimpact.com or phone number 0912503019. You can also contact the Social Impact Internal Review Board. The contact person is Danae Roumis Address is: 2300 Clarendon Blvd, Suite 1000, Arlington, VA 22201; phone number 703-465-1884; email address: irb@socialimpact.com. VOLUNTARY AGREEMENT I certify that the nature and purpose, the potential benefits, and the possible risks associated with participating in this research have been explained to me. I also certify that I am at least 18 years of age at this time. __________________________________________________ ____________________ Signature of study participant Date 76 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity KII Guide for Woreda/Region Level Public-Private Partnership * Note for interviewer I. Establish Rapport • Greeting!!! • Introduce yourself • Tell who you work for: Social Impact • Tell purpose of visit: Conducting endline evaluation of USAID’s Building Potential of Youth Activities project II. Obtain Written Consent III. Identify focal person for the KII together with LIP Name of Organization _______________ Interviewee Position: _______________ Date: _______________ Region: _______________ Woreda: _______________ Town: _______________ Sex: M / F Interviewer Name: _______________ Interview Code: _________________ III. General 1. How long have you been in this position? ________(years) 2. Tell me a little about your role in your organization? II. Public-Private Partnership (EQ5) 1. Could you tell us about the public private partnership establishment process between your organization and BPY implementing organization (mention name of the organization)? Tell us how the partnership was established? 2. What were the contributions of PPPs to BPY implementation in this area? What is its contribution in improving employment status of BPY youth in this area (if any)? What other contributions the PPP has to BPY youths other than employment status? 3. Does the partnership relationship your organization has with BPY implementing partners (mention name) still exist? 4. Is there any adaptations of PPP initiative/experience so as to increase employment opportunities for BPY youths? If yes, did they result in improved results for youths? 5. What is your opinion about sustaining the private-public partnership initiatives of BPY? 6. What are the lessons learnt from the PPP initiatives started by BPY implementing organization? 7. What are the challenges in relation to public private partnerships initiative? 77 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity KII Guide for Woreda Level SC/LIPs * Note for interviewer I. Establish Rapport • Greeting!!! • Introduce Yourself • Tell who you work for: Social Impact • Tell purpose of visit: Conducting endline evaluation of USAID’s Building Potential of Youth Activities project II. Obtain written consent III. Identify focal person for the KII together with LIP Organization Name: _______________ Position/title of Interviewee: _______________ Date: _______________ Region: _______________ Woreda: _______________ Town: _______________ Sex: M / F Name of Interviewer: _______________ I. Employment (EQ1) 1. What activities were undertaken by BPY to improve BPY youths’ employment status in this area? Probes (if applicable) a) How relevant were the activities to the needs/expectations of BPY youths in this area? b) What kinds of resources/supports were provided by BPY to improve employment status of BPY youths? Were the support/resources helpful to improve employment opportunities for youths? If yes, how? 2. Did the activities contribute to access quality employment? (If yes) Which activities contributed more? How? (EQ1) What are the employment types BPY youths engaged in in this area? Which sector/s are more preferred by youths of this area? Why? 3. Are there any changes in employment status of youths after they started participating in BPY activities in the area? Probes (if applicable) a) Was the change increase/ decrease/ stay same? b) What were the reasons for the change in employment status 4. What are the challenges and lessons learned for youths in this area to be employed? II. Income (EQ2) 5. Is there any change in income level of BPY youth in this area after they started participating in BPY? Probes (if applicable) 78 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity a) If yes, was the income level decrease/ increase/stay the same b) if yes, what are the reasons for the change in their income level? 6. What are the challenges encountered to improve BPY youths’ income level in this area? 7. What are the lessons learned to enhance income level of BPY youth in this area? III. Loan (EQ 1&2) 8. Could you tell us the role of your organization in helping youths access loan? 9. What are the main sources of start-up/working capital for youth in this area? Probe: for both formal and informal sources? 10. Could you tell us the extent to which BPY youths are accessing loan from MFIs? 11. What are the challenges for BPY youths in this area to access loan from MFI? 12. Are there changes for BPY youths in acquiring productive assets after BPY? IV. Gender (EQ3) 13. What criteria were used to select youth to BPY? Probes (if applicable) a) What procedures were used to mobilize the youth? b) Who did the selection? c) What is your opinion about the selection criteria? d) Did the selection process give equal opportunity for male and female to be included? e) Is there any variation among male and female? What are the reasons for the variations in gender? 14. What are the challenges encountered in providing equal opportunity for male and female youths? 15. What are the main lessons learned in addressing gender equity? V. Grant (EQ4) 16. Could you tell us the selection process of TVETs/TTI for small grant? What are the criteria for selecting to access grant? a) What type and amount of grants was provided to each TVET/TTI? b) For what purpose TVETs/TTIs used the grants they received? c) Was the grant utilized for the intended purpose? If not why? 17. Was the grant relevant to TVET’s capacity need? Was the grant relevant to the BPY youths skill needs? 18. Did the grants helped TVET’s so that the graduates get better employment opportunities? How? If no, why? 19. What were the lessons learnt from the small grants scheme? 20. What were the challenges related to the grants? VI. Public-Private Partnership (EQ5) 79 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity 21. Could you tell us about the PPP establishment process in this area? What were the contributions of PPPs’ in BPY and the youths? 22. What mechanism are in place to sustain the PPP initiatives? If no, why? 23. Was there any adaptations of PPP model/experience during the implementation process? 24. What are the lessons learnt from the PPP initiatives? VII. YES-HUB (EQ6) 25. Could you tell us about the YES-H established by BPY in this area? Probes (If applicable) a) Were YES-HUBs established in your intervention area? b) What is your opinion on the operational and service provision capacity of YES-HUBs in this intervention area? c) What mechanisms are in place to ensure sustainability of the YES-H initiative after the project ends? d) What opportunities are there for the financial and operational sustainability of / YES-Hs initiatives? e) What are the challenges encountered during implementing the YES-H initiative and /or at the YES-Hs? f) What lessons were learnt during the course of YES-Hub design and implementation? VIII. Evidence (EQ7) 26. Are there any practical experiences that evidences are utilized/adapted by SC/IPs/USAID? 27. Were there any challenges encountered to generate, utilize or disseminate evidences? 80 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity KII Guide for Regional /Federal Level Government Offices * Note for interviewer I. Establish Rapport • Greeting!!! • Introduce Yourself • Tell who you work for: Social Impact • Tell purpose of visit: Conducting endline evaluation of USAID’s Building Potential of Youth Activities project II. Obtain written consent III. Identify focal person for the KII together with LIP Organization Name: _______________ Position/title of Interviewee: _______________ Date: _______________ Region: _______________ Town: _______________ Sex: M / F Name of Interviewer: _______________ 1. What was your organization/Office’s role in the implementation process of BPY? In which areas were you involved? 2. What is your opinion about the employment status of BPY youths? What did BPY contribute to improved employment status of youths (if any)? 3. What do you think about the income level of youths after they participated in BPY Do you think BPY contributed in changing their income? If yes, how? If no, why? 4. What are the most significant successes during design, implementation and M&E of the project? 5. What are the challenges encountered during design, implementation and M&E of the project? 6. What were the lessons learned during the design and implementation of the project? 7. What is your opinion about sustainability of BPY initiatives (Probe: YES-H, PPP, Loan, Grant) after BPY? What mechanisms are in place to ensure their sustainability? 8. What could have been done in a better way? OR What are your recommendations for future similar projects? 9. Is there anything else you would like to tell us? 81 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity Focus Group Discussion Guide for BPY Youths * Reminders for facilitator and Note taker I. Rapport • Greeting!!! • Introduce facilitator and note taker • Tell who you work for: Social Impact • Tell purpose of visit: Conducting endline evaluation of USAID’s Building Potential of Youth Activities project II. Obtain Oral Consent III. Set discussion rules (examples bellow) • Every participant can express own opinion freely • No critique of participants because of their opinion • Keep turn when speaking • Every discussant should participate in the discussion • There will be no side talks during discussion • Switch off or put on silent mobile phones IV. Discussion Questions 1. The questions are put in order of priority (in line with EQs) 2. Use the probing questions when/if necessary V. Use the local used name for BPY (POTENTIAL, name of IP, or link it to the soft and hard skill training they used) Date _______________ Region: _______________ Woreda: _______________ Kebele/Town: _______________ Participants’ Number: M_____/ F_____ Facilitator’s Name: _______________ Note-taker Name: ________________ I. Employment (EQ1) 1. (One of the goals of BPY is to improve the employment status of youths.) In what ways did BPY try to help you improve your employment status, to go from unemployed or underemployed to employed? Probes (if applicable) a) Was there a change in employment status? If yes, was it increase/decrease/stay same? If no, why? b) Which services you received from BPY helped you the most? (can ask about specific service components)? c) Are you satisfied with the employment opportunity you had after BPY? How satisfied are you? d) What are the kinds of employment that you and youths like you who participated in BPY are engaged in in this area? e) What kinds of employments are more preferred by youths who participated in BPY? Why? 82 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity f) What does the difference in employment status between male and female who participated in BPY looks like in this area? g) What are the challenges you/youths like you who participated in BPY encounter to be employed in this area? II. Income (EQ2) 2. Similarly, (one of the goals of BPY is to improve the income level of youths.) To what extent did BPY help you to increase your income? Probes (If applicable) a) Was there a change in your income after you started participating in BPY? b) If there was a change in income level, was it a decrease/ increase/stay the same? What was the reasons for the change in income level? c) What changes have you experienced in your productive assets after you participated in BPY? d) What are the challenges you and youths like you encountered in improving income levels? e) What are the main lessons you learned in improving your income level after BPY? III. Loan (EQ1 & EQ2) 3. Now we want you to discuss youths’ access to a loan from micro-finance institutions. Did any of you try to get such a loan through the MFI; what were your experiences? Probes (if applicable) a) What are the main sources of start-up/working capital for youths in this area? b) Are you/youths like you who participate in BPY able to access loan from MFIs? c) What were the criteria to access loan? d) How many of you have accessed loan? e) What is the size of loan provided to youths per individual? f) What does the repayment situation of loan users look like? g) What does the debt level of loan users looks like? h) Are there any youths who defaulted? If yes, what are the reasons? i) If you did not get a loan from an MFI did you get able to access capital in other ways? j) What are the challenges for youths to access loan in this area? IV. Gender (EQ3) 4. Now let’s talk about gender equity. Do you think BPY treated both male and female equally during its implementation process? Probes (if applicable) a) Were there criteria used to select youths to participate in BPY activities? If yes, what were they (if they have any knowledge about) What is your opinions about those criteria? b) Were male and female given equal chance to be selected? c) Were there any intervention/activity more preferred by male or female youths? 83 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity d) What were the challenges encountered to address gender equity during selection and implementation of BPY? e) What do you think are the main lessons learned to address gender equity? V. General 5. In your opinion which activities of the BPY were more helpful to you and youths like you? 6. In what ways (if any) BPY contributing to achieve your goal for the future? 7. What do you think should be done differently to make BPY more successful or helpful to you/youths like you? 8. Is there anything else that you want to tell us about BPY? VI. YES-HUBs (EQ6) 9. Is there a YES-H in this area? (if yes) What is your opinion about the services that BPY youths receive from the YES-HUBs? Probes (if applicable) a) Was it helpful? If yes, in what ways? b) What kinds of services do you receive from the YES-HUBs? c) What is your opinion about the operational and service provision capacity of YES￾Hs? d) What opportunities are there for the operation/service of the YES-H post BPY? VII. Self-Sufficiency (EQ9) 10. Now let’s talk about self-sufficiency. (One of the goals of the BPY was to help youth become more financial independent.). Could you discuss on your status of self-sufficiency or financial independence after you started participating in BPY? Probes (if applicable) a) Do you think BPY helped you in this way? b) Before you joined BPY program, were you receiving support from your family? If yes, what kinds of support did you receive? c) After BPY, for how many months have you supported yourself without help from your family? d) Did the skill trainings helped you progress towards longer-term self- sufficiency? If so, which one is more helpful? e) Over the course of your engagement in BPY, did you exhibit sense of self￾confidence and knowledge of appropriate conduct? 84 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity KII Guide for Regional/Federal Level SC/LIPs * Note for interviewer I. Establish Rapport • Greeting!!! • Introduce Yourself • Tell who you work for: Social Impact • Tell purpose of visit: Conducting endline evaluation of USAID’s Building Potential of Youth Activities project II. Obtain written consent III. Identify focal person for the KII together with LIP Organization Name: _______________ Position/title of Interviewee: _______________ Date: _______________ Region: _______________ Town: _______________ Sex: M / F Name of Interviewer: _______________ I. Employment (EQ1) 1. What major activities were undertaken by BPY to improve BPY youths’ employment status in this area? b) How relevant were the activities to the needs/expectations of BPY youths in this area? 2. Are there any changes in employment status of youths after they started participating in BPY activities in the area? a) Was the change increase/ decrease/ stay same? b) What do you think were the reasons for the change in employment status 3. What did BPY contribute to improved employment status of youths (if any)? 4. What are the challenges and lessons learned for youths in this area to be employed? II. Income (EQ2) 5. What is your opinion about the income level of BPY youths after the project implementation? a) Is there any change in income level of BPY youth in this area after they started participating in BPY? b) If yes, did the income level decrease/ increase/? c) If yes, what are the reasons for the change in their income level? d) If no change (stay the same), why 6. What are the challenges encountered to improve BPY youths’ income level? 7. What are the lessons learned to enhance income level of BPY youth? 8. What did BPY contribute to improved income level of youths (if any)? 9. Are there changes in acquiring productive assets by BPY? (EQ3) III. Loan (EQ1 &2) 10. What are the main sources of start-up/working capital for BPY youth? (EQ3) a) What did BPY do to facilitate to access loan for youths? b) Could you tell us the extent to which BPY youths are accessing loan from MFIs? 85 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity c) What are the challenges for BPY youths to access loan from MFI? (EQ3) IV. Gender (EQ3) 11. Do you think BPY worked for female as well as male? a) What are the mechanisms BPY used to make sure male and female youths get equal opportunity? b) What criteria were used to select youth to BPY? Were male and female given equal opportunity? c) What are the outcomes of gender equity interventions of BPY? d) What are the challenges in providing equal opportunity for male and female? e) What are the main lessons learned in addressing gender equity? V. Grant (EQ4) 12. Could you tell us the selection process of TVETs/TTI for grant? What is your opinion about the grant in improving capacity TVET/TTI and /or quality of training they provide? a) What were the lessons learnt from the small grants scheme? b) What were the challenges related to the grants to TVET? TTI? VI. Public-Private Partnership (EQ5) 13. What is your opinion about PPPs contribution to youths? a) What were the contributions of PPPs’ in BPY and the youths? b) What mechanism are in place to sustain the PPP initiatives? If no, why? c) Was there any adaptations of PPP model/experience during the implementation process? d) What are the lessons learnt from the PPP initiatives? VII. YES-HUB (EQ6) 14. Could you tell us about your overall impression of YES-H established by BPY? g) What opportunities are there for the financial and operational sustainability of / YES-Hs initiatives? h) What are the challenges encountered during implementing the YES-H initiative? i) What lessons were learnt during the course of YES-Hub design and implementation? VIII. Evidence (EQ7) 15. Are there any practical experiences SC/IPs/USAID that generated and utilized evidences on BPY? a) What were the mechanisms of evidence dissemination? b) What were the challenges encountered to generate, utilize or disseminate evidences? IX. General 16. What are the challenges encountered during design, implementation and M&E of the project? 17. What were the lessons learned during the design and implementation of the project? 18. What could have been done in a better way? OR What are your recommendations for future similar projects? 19. Is there anything else you would like to tell us? 86 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity Annex V: Disclosures of Conflicts of Interest 87 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity 88 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity 89 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity 90 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity 91 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity Annex VI: Summary of Evaluation Team Members Dr. James M. Statman, Team Leader, brings over 30 years of experience in the field of international development evaluation and research to his leadership of the BPY Final Evaluation team. Dr. Statman’s professional experience spans sectors, with an emphasis on economic development and livelihoods. He has worked extensively with USAID throughout his career, most recently as a Team Leader on multiple evaluations and as Chief of Party for the Northern Nigeria Education Initiative Plus, the Zimbabwe/Works Project, and the Zimbabwe HIV and AIDS Partnership Project. As Team Leader, Dr. Statman led the BPY Final Evaluation team in evaluation design, data collection, data analysis, and report drafting. He holds a Ph.D. in Social Psychology from Yeshiva University in New York. Mr. Mazengia Abera, Local Expert, has over 15 years of experience as a research and M&E consultant, project implementer, and University instructor. Mr. Mazengia has rich experience in designing data collection tools, collecting and analyzing data, and report writing of project evaluations, studies and assessments. He has recently participated in M&E activities for a wide variety of USAID-funded projects in Ethiopia, including the evaluation of USAID’s Graduation with Resilience to Achieve Sustainable Development activity and USAID/Ethiopia’s Performance Monitoring and Evaluation Services Activity. On the BPY Final Evaluation team, Mr. Mazengia supported throughout the evaluation, including during design, data collection, primary and secondary data analysis, preliminary findings presentation to USAID, and reporting. Mr. Mazengia has Master of Public Health from Addis Continental Institute of Public Health, and an M.A in Social Psychology and B.A in Educational Psychology from Addis Ababa University. Mr. Ermias Assefa Eshetu, Local Expert, has over 10 years of experience in qualitative and quantitative research, data collection, analysis, and management for international development programs. As a member of the BPY Final Evaluation, Mr. Eshetu contributed to the development of data collection instruments, collected primary qualitative data, compiled secondary data from project monitoring and government datasets, and participated in data analysis, presentation of findings, and report writing. He was also a team member for related work of USAID/Ethiopia’s Youth Cohort Study and the baseline survey of BPY. Mr. Eshetu holds an M.A. in Measurement & Evaluation from Addis Ababa University, an M.Sc. in Economics (Specialized Agricultural Economics) from Haromaya University, and a B.Sc. in Information Systems from Addis Ababa University. Mr. Worku Ambelu, Evaluation Methods Specialist/Senior Evaluation Manager for the USAID/Ethiopia Performance Monitoring and Evaluation Services Activity implemented by Social Impact, brings over 20 years of development work expertise to the team. Mr. Ambelu possesses a wealth of knowledge in project monitoring and evaluation, planning, statistics, assessments and research activities gained from different organizations. As the team’s Evaluation Methods Specialist/Senior Evaluation Manager, he was responsible for coordinating 92 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity all the evaluation activities and supporting tool development, data analysis, and report review support. In his work with Social Impact, Mr. Ambelu has also served as Evaluation Team Member and/or Manager on a wide array of evaluations and assessments for USAID/Ethiopia, including relevant work on the Youth Cohort study and Youth Potential Baseline Survey. He holds M.Sc. and B.Sc. degrees in the field of Statistics from the Addis Ababa University. Mr. Yohannes Leta, Monitoring Evaluation and Learning Advisor for the USAID/Ethiopia Performance Monitoring and Evaluation Services Activity, brings over 10 years of experience in development as a MEL Advisor, Monitoring Evaluation & Research/MER Advisor, M&E Specialist, M&E Officer, Regional Planning Monitoring & Evaluation Coordinator for five USAID-funded international NGOs and one USAID MEL contractor. Mr. Leta has worked in Ethiopia with international development firms and agencies such as Social Impact Inc., John Snow Inc., AECOM, World Learning Inc., Pathfinder International I-TECH Ethiopia and government institute Addis Ababa University. In addition to his M&E expertise, he also specializes in the sectors of health, WASH, and education. As a Monitoring Evaluation & Learning Advisor on the team, Mr. Leta supported the BPY team throughout the evaluation, including during design, data collection, primary and secondary data analysis, presentation of preliminary finding to USAID, and draft report writing. He holds an M.Sc. degree in Population Studies (Reproductive Health) from Addis Ababa University and a B.Ed. in Mathematics from Bahir Dar University. Mr. Argaw Korssa, Team Member, brings 15 years of experience in project management, evaluations, and designing of monitoring and evaluation systems for international development organizations in Ethiopia. His work in M&E spans a wide range of sectors and has included work with prominent donors, such as USAID, DFID, EU, and UN agencies. For the BPY Final Evaluation Mr. Korssa participated in the preparation of the evaluation design and data collection tools and collection of field data. He holds an MSC in Computer Science from HiLCOE in Addis Ababa and a B.A. in Geography from Haramaya University in Ethiopia. 93 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Building the Potential of Youth Activity U.S. Agency for International Development Entoto Street PO Box 1014 Addis Ababa, Ethiopia