EVALUATION YOUTHMAP UGANDA Final Report – June 5, 2015 June 2015 This publication was produced at the request of the United States Agency for International Development. It was prepared by Tamara Duggleby, Rajan Kapoor, Chung Lai, Michael J. Midling, and Martin Opolot, under the Man￾agement Support and Technical Analysis Services Project, implemented by the Pragma Corporation and Manage￾ment Systems International. YOUTHMAP UGANDA EVALUATION JUNE 2015 Contracted Under USAID Management Support and Technical Analysis Services (MSTAS) Project, AID￾OAA-M-13-00012. DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Photo: U.S. Mission Uganda, CC – Some rights reserved. YouthMap Uganda Evaluation Report – June 5, 2015 i CONTENTS ACRONYMS ...................................................................................................................................................................... iii EXECUTIVE SUMMARY .................................................................................................................................................. 4 INTRODUCTION ............................................................................................................................................................ 9 BACKGROUND ............................................................................................................................................................... 9 EVALUATION PURPOSE AND METHODOLOGY .............................................................................................. 12 Purpose .......................................................................................................................................................................... 12 Methodology ................................................................................................................................................................ 12 Limitations ..................................................................................................................................................................... 14 FINDINGS ......................................................................................................................................................................... 16 A. Program Effectiveness and Impact ............................................................................................................. 16 1. How effective were the approaches taken for each of the four program components in achieving the goal? .................................................................................................................................................. 16 a. Orientation and skills training ................................................................................................................ 16 b. Internship placements .............................................................................................................................. 18 c. Professional development support services ....................................................................................... 19 i. Mentoring ....................................................................................................................................... 19 ii. Job placement support .............................................................................................................. 20 iii. Participation in UIP Alumni Network .................................................................................... 21 d. USAID Collaborating, Learning and Adapting (CLA) Approach, Internal Assessmentand Entrepreneurship Training ................................................................................................................................ 21 i. Facilitation of youth participation in USAID CLA-related activities ........................... 22 ii. Usefulness of an internal YouthMap Assessment for designing and or improving programs ....................................................................................................................................................... 24 iii. Build Your Business (BYB) Entrepreneurship Training .................................................... 25 2. What was the program’s overall impact on youth beneficiaries? ...................................................... 26 a. Achieving Development Hypothesis and Objectives ....................................................................... 26 b. Impact on Youth Beneficiaries ............................................................................................................... 27 c. Strength and Weakness of the Four Program Components .......................................................... 31 d. Key Factors that Enabled Interns to Secure Employment/Remain Employed ............................ 32 e. Advisory Role and Contribution to the Achievement of the Program Goal ............................. 33 3. How was the capacity of various program partners strengthened through the UIP and what YouthMap Uganda Evaluation Report – June 5, 2015 ii were the outcomes of these partnerships? ...................................................................................................... 34 4. What are the trends in intern employment data with respect to gender, sectors of interest, employer retention, and employment quality (pt/ft, salary, hours, location and promotions)? .......... 35 B. Sustainability and Scalability ........................................................................................................................ 37 5. What are the factors facilitating or inhibiting the sustainability of program activities beyond the UIP? ........................................................................................................................................................................... 37 a. Internship Placement Policy.................................................................................................................... 37 b. Institutional Capacity of Host Employers ........................................................................................... 38 c. Resources to fund the internship placement program .................................................................... 38 d. Potential for Replicability ........................................................................................................................ 39 6. What program elements should be scaled up/replaced? What entities are best placed to support the scale of the program elements? ................................................................................................................... 40 CONCLUSION ................................................................................................................................................................ 41 KEY LESSONS LEARNED ............................................................................................................................................. 45 Internship ...................................................................................................................................................................... 45 Collaborating, Learning and Adapting .................................................................................................................... 46 Build Your Business .................................................................................................................................................... 46 OVERALL RECOMMENDATIONS FOR FUTURE PROGRAMMING .............................................................. 47 Effectiveness & Impact ................................................................................................................................................ 47 Sustainability & Scalability .......................................................................................................................................... 48 Annex I: Evaluation Work Plan .................................................................................................................................... 50 Annex II: Materials received and Reviewed ............................................................................................................... 52 Annex III: List of KII and FGD Completed ................................................................................................................ 54 Annex IV: KII and FGD Questions Guides ................................................................................................................ 55 Annex V: Scope of Work .............................................................................................................................................. 78    YouthMap Uganda Evaluation Report – June 5, 2015 iii ACRONYMS BYB Build Your Business CLA Collaborating, Learning, and Adapting CV Curriculum Vitae DFCU Development Finance Corporation in Uganda EU European Union FGDs Focus Group Discussions FUE Federation of Uganda Employers HR Human Resources ICT Information and Communications Technology ILO International Labor Organization IPs Implementing Partners IRR Internal Rate of Return IYF International Youth Federation KIIs Key Informant Interviews MGLSD Ministry of Gender, Labour and Social Development MP Members of Parliament MTN Mobile Telephone Network MUBS Makerere University Business School NGOs Non-Governmental Organizations RD Restless Development SoW Scope of Work SRH Sexual and Reproductive Health TPO Transcultural Psychosocial Organization UIP Uganda Internship Program USAID U.S. Agency for International Development YouthMap Uganda Evaluation Report – June 5, 2015 4 EXECUTIVE SUMMARY The YouthMap Uganda Internship Program (UIP) is a pilot designed to strengthen the skills of young people, aged 23-30 years, to better connect them to the job market. Through provision of internship, entrepreneurship, networking and research opportunities to 100 young men and women, YouthMap aimed to enhance positive perceptions about youth among public and private sector employers. On the whole, participants reported better employment as evidenced in the higher salaries in the follow-up survey realized six to 12 months after the internship, and more than three-quarters of them were employed. More than half the participants indicated that the program helped them find a job and more than one in six reported becoming a business owner. In general, youth and employer satisfaction with the program was favorable, with youth receiving training and work experience, and employers filling gaps in human resources. The International Youth Federation (IYF) implemented the program with USAID support. Its local implementing partner, Restless Development (RD), carried out the program with IYF and USAID/Uganda oversight. Interns were matched to 37 employers, who were expected to provide cost sharing and mentoring. Participants were almost evenly divided among males (49) and fe￾males (51). Approximately one-third of the interns were placed in NGOs, and more than two￾thirds of the internships were located in Kampala, which reflects the scarcity of jobs in rural areas of Uganda. This performance evaluation is intended to assess UIP effectiveness and impact on beneficiaries as well as sustainability and scalability to inform similar internship programs that may be imple￾mented in the future. Effectiveness is defined as achievement of program objectives and assess￾ment of program quality and value. Impact in this evaluation refers to the changes (results) pro￾duced by the program. Sustainability refers to the program being able to continue after the USAID-funded pilot. Scalability looks at the program ability to geographically and over time reach more people. The evaluation is designed to provide a comprehensive analysis of the program components, including: recruitment and selection, matching, training, and placement. It responds in depth to each evaluation question: 1. How effective were the approaches taken for each of the four program components (ori￾entation and skills training; internship placements; professional development support services; and facilitation of youth participation in USAID CLA-related activities) in achiev￾ing the programs goal? 2. What was the program’s overall impact on youth beneficiaries? 3. How was the capacity of various program partners strengthened through the UIP and what were the outcomes of these partnerships? YouthMap Uganda Evaluation Report – June 5, 2015 5 4. What are the trends in intern employment data with respect to gender, sectors of inter￾est, employer retention, and employment quality? 5. What are the factors facilitating or inhibiting the sustainability of program activities be￾yond the UIP? 6. What program elements should be scaled up/replaced? What entities/organizations are best placed to support the scale up of the program elements? This evaluation used a mixed methods approach to assess effectiveness, impact, sustainability and scalability. The evaluation team collected qualitative data and made use of RD’s performance monitoring and quantitative data obtained from surveys conducted before the evaluation. Qual￾itative data was collected from FGD and KIIs. The FGDs were randomly selected from a telephone list of those who responded and were willing/able to attend the FGD. The evaluation team tried to contact all 70, but only reached 49. Of those, 20 planned to attend the FGD, and 13 actually attended one of the three FGDs. KIIs were selected purposefully to understand the program and its different components and stages. The evaluators conducted 21 KIIs, namely with RD staff (3), IYF staff (2), Board members (3), BYB consultant (1) youth participants (7) and selected host em￾ployers (5). There were two main quantitative data sources. The baseline survey and follow-up survey were provided just prior to the arrival of the evaluation team. The baseline survey was conducted near the beginning of the program, after the intern selection stage via an in-person survey. The follow-up survey was sent six months or more after the end of the internship via email as an e-survey. The main data limitations revolve around the limited field time, limited monitoring data, poor quantitative data, and limited number of follow-up survey responses for analysis. The evaluation team was in country for 1.5 weeks on two separate occasions. During that time the team was able to collect only a limited number of FGDs and KIIs of intern participants, employers and other stakeholders. To address this issue, additional KIIs were conducted over the phone. While RD provided the evaluation team with all of its program data and information, the useful M&E data was sparse and incomplete. There were program indicators but incomplete corresponding mon￾itoring data. Quantitative data were subject to interpretation as the survey questions were am￾biguous and different questions were used in the baseline and the follow-up survey. In addition, because the follow-up survey was sent out as an e-survey and a second-round survey was not completed, the number of responses was about 50. Nevertheless, this is about 50 percent of the total population, which is a good sample size. Several aspects of the program were particularly successful. Regular workplace visits by RD staff during the course of the internship for support and monitoring helped interns to ensure the suc￾cessful implementation of their individual work plans. These visits also allowed for more regular YouthMap Uganda Evaluation Report – June 5, 2015 6 dialogue and additional feedback opportunities for host employers and interns. Moreover, sup￾porting interns to set and achieve clear goals during the foundational training allowed them to focus on job activities aligned with their goals, increasing job satisfaction and performance. The overall employment rate for participants – 78 percent - compares favorably with the official em￾ployment rate of approximately 65 percent for all labor force participants with tertiary degrees. Because of the limited number of host locations (37 for 100 interns), available internships did not always match up with applicants’ education and skills. However, the overall outcome was viewed positively by both interns and employers. For example, in one FGD, 4 out 6 interns said even though the internship did not match their educational background, the placement had a positive effect on guiding their career into a different sector. On the employers’ side, all ten cohort one employers interviewed ranked the quality of interns highly. If the program is to be scaled up and sustained, the unit cost of the program – almost $25,000 per participant including costs incurred by IYF, RD and direct leveraged contributions – would have to be carefully scrutinized, and likely pared down. Given job placement costs which typically range in the $1,000 - $10,000 range for donor-supported programs internationally, this does not appear to be a cost-effective intervention approach as currently implemented. For program sus￾tainability, internship policies, institutional capacity of the host employers and resources to fund the internship program are issues that need to be addressed. A scale-up plan for future program￾ming would help address future directions that might take advantage of the YouthMap pilot. Despite what may appear to be high costs of the program, the benefits may be substantial. Of those participants who reported a salary at the end of the program, they experienced an average salary increase of $126 per month. However, as important as this increase may be, the costs are too high to make the program viable from a cost-benefit perspective and the results are not sta￾tistically significant. The lack of statistical significance reflects the fact that the data exhibited a large variance, which could be due to the small overall population size, the fact that two-thirds of the respondents did not work or did not report a salary prior to participation in the program, as well as the fact that the beneficiary population is highly heterogeneous. To increase the overall effectiveness and sustainability of the program, the evaluation team recommends that IYF and RD focus on the following areas: Overall: Although this program was a pilot, a way needs to be found to substantially reduce program costs or increase program benefits for it to be financially and economically viable. For interns: Introduce a targeted orientation and training focus by sector of employment, in ad￾dition to the current orientation and foundational training. This will provide interns with skills tailored to their individual placements. Use of specialized trainers that have sector experience YouthMap Uganda Evaluation Report – June 5, 2015 7 and knowledge, rather than RD program staff, would be a more effective approach for providing such trainings. For employers: Invest more time in preparing host employers to receive interns, including edu￾cating the prospective supervisors on the mentoring and professional development responsibili￾ties of the program. Building employer institutional capacity to host interns through partnership development will strengthen the relationship between the implementing organization and the employer. The more employers with suitable capacity (in the form of organizational structures, HR guidelines, and an internship policy) to host interns, the higher the final placement rate will be. For the Board: Make the Advisory Board a formal, functional body with a clearly defined scope of work, systems, procedures, and expected deliverables. The Board should support UIP in devel￾oping plans for sustainability and scaling up. For example, the Board should guide continued engagement with the Government of Uganda to realize a national policy on youth internship programs and related funding program for internship and job placement activities. For supplementary program components: Separate the auxiliary program components from the internship component. Develop the Build Your Business (BYB) and Collaborating, Learning, and Adapting (CLA) entrepreneurship and research agendas as separate programs that are stra￾tegically linked to the internship without interfering with the experience or timeframe for the in￾ternship (ie. optional activities after the internship). Improving communications between IYF, USAID, and RD could be useful in this regard, as delays in feedback affected the timing and effec￾tiveness of the program. For monitoring and evaluation: Develop a monitoring and evaluation plan at the beginning of the program. There was a lack of an M&E framework and performance indicators tracking table, and survey instruments were inconsistent from baseline to follow-up. A relational database is better to track intern placement, skills achieved and contact info than an Excel spreadsheet. And there needs to be more intensive efforts to ensure a higher (i.e. close to 100 percent) response rate to the follow-up surveys. To better collect quality and consistent data, there needs to be a clear and complete monitoring and evaluation plan that includes an evaluation plan articulating the evaluation methodology, sampling frame and size, and data analysis. For gender equality and sensitivity training: Conduct a gender analysis on youth in the work￾force to better inform programming as findings found men and women indicating marked differ￾ences in experiences regarding orientation and training. Also include gender equality in the life skills foundational training. One female youth participant mentioned she was sexually harassed at the workplace. In fact, six of the non-participants in FGDs also agree that sexual harassment is YouthMap Uganda Evaluation Report – June 5, 2015 8 an obstacle in obtaining a job. Future programing should include awareness training to address sexual harassment in the workplace both for interns and host employers. For employment sectors. Consider targeting the internship placement in specific sectors and region to increase the employability within that sector, region and youth income potential. Dif￾ferent sectors have distinct productivity enhancement and growth potential. Different regions vary considerably in terms of potential job opportunities because of rural-urban differences. Bet￾ter targeting internship activities in accordance with sectoral/regional growth trends/potential can help optimize the sustainable job placement and wage growth results of the program, and also contribute to enhanced economic competitiveness. Table 1: Socio-Demographic Profile of Youth Participants Gender Male (46) Female (52) Total (98) Average age (mean ± sd) 25.2 ± 1.8 24.2 ± 1.5 24.7 ± 1.7 Marital status Single 97.8 98.0 97.9 Highest level of education University/tertiary 89.1 94.2 91.8 Technical college/Institute 6.5 3.9 5.1 Postgraduate education 4.4 1.9 3.1 Average household size 5.9 ± 3.8 7.5 ±4.5 6.8 ± 4.2 Area of residence Urban 78.3 86.5 82.6 Rural 21.7 13.5 17.3 Economic status of the neighborhood Extremely Poor 2.2 3.9 3.1 Poor 56.5 48.1 52.0 Not Poor 41.3 48.1 44.9 Table 1 presents the profile of the youth participants recruited into the program. Eligible interns were “men and women” age 23 years and above who had a minimum of a technical college cer￾tificate or diploma level. Females (52) were over represented in the program compared to the men (46). The mean age was 25 years, the male interns were slightly older compared to the female counterparts. More than 97 percent of the interns were single and about 90 percent had com￾pleted a university degree, with only 3 percent completing a degree at a post graduate level. About 83 percent live in urban area with an average household size of 7. Over 55 percent live in “poor” or “extremely poor” neighborhoods, while 45 percent live in “not poor” neighborhoods. YouthMap Uganda Evaluation Report – June 5, 2015 9 INTRODUCTION Uganda's population is the second youngest worldwide, with approximately 78 percent under 30 years. At 64 percent, the country has one of the highest rates of youth unemployment in Sub￾Saharan Africa.1 With jobs in the formal sector limited and highly competitive, most young people are engaged in informal and/or seasonal work, fueling frustration and leaving many vulnerable to exploitation. Youth face multiple institutional barriers to securing employment, including misa￾lignment of education with labor market needs, lack of access to skills training and work experi￾ence opportunities, inadequate information about job opportunities and career guidance, and lack of access to land, inputs and markets. Youth also face cultural barriers, including negative attitudes among employers, who often view them as unskilled and undisciplined. With population growth at 3.3 percent, unemployment is expected to remain a massive challenge in the years ahead. However, growth in key economic sectors and Uganda's natural assets mean that oppor￾tunities are emerging. Recent research indicates that employers are willing to support youth tran￾sition into employment but remain unsure how to do so effectively, placing an increasing need on youth sector partnerships. BACKGROUND The YouthMap Uganda Internship Program (UIP) is an Innovation Fund activity under International Youth Foundations’ (IYF) USAID-funded YouthMap initiative. The program was designed to strengthen the skills of young people so that they are better equipped to enter the job market, either as employees or entrepreneurs. In Uganda, the program was implemented by Restless Development (RD), managed by IYF, and overseen by USAID/Uganda. An Advisory Board was created to help guide the program’s design, direction, and sustainability. Its development hy￾pothesis was that “if educated youth are provided a comprehensive internship opportunity that fos￾ters skills development, mentorship and peer support, then their chances of gainful employment will significantly increase.” UIP had three objectives: (i) increased capacity of young people through training, practical expe￾rience and mentoring; (ii) expanded internship opportunities for young people; and (iii) enhanced 1 In 2012, the Uganda Bureau of Statistics revealed that the share of unemployed youth (national definition, 18-30 years) among the total unemployed persons in the country was 64 percent. Gemma Ahaibwe and Swaibu Mbowa, “Youth Unemployment Challenge in Uganda and the Role of Employment Politices in Jobs Creation.” Accessed 2/5/15, http://www.brookings.edu/blogs/africa-in-focus/posts/2014/ou/26-youth-unemplyment-uganda-ahaibwe￾mbowa. YouthMap Uganda Evaluation Report – June 5, 2015 10 learning through USAID’s Collaborating, Learning and Adapting (CLA) agenda driven by engage￾ment of young interns. The program was designed for young people aged 23-30 years for a period of 14-15 months, followed by a period of review, monitoring and evaluation. UIP targeted 100 young people for internship opportunities with USAID Implementing Partners (IPs), the private sector, NGOs, and the government. There were two cohorts of youth interns: cohort one (June 2013 to November 2013) and cohort two (February 2014 to July 2014). Interns were matched with 37 employers, who were expected to provide cost share and mentoring. Interns were placed in a variety of sectors, including agri￾culture, banking and finance, government and NGOs, hospitality and tourism, and information and communications technology within seven selected districts, namely Busia, Gulu, Jinja, Kam￾pala, Kitgum, Lira and Mbarara. Table 2: Summary of Interns and Employers by District2 District Interns Employers Busia 1 1 Gulu 9 1 Jinja 15 9 Kampala 64 20 Kitgum 2 1 Lira 4 2 Mbarara 5 3 Total 100 37 UIP had four major components:  Orientation and skills training;  Internship placements;  Professional development support services, including mentoring, job placement services, and participation in an alumni network; and  Entrepreneurship training and facilitation of youth participation in activities in support of USAID’s Collaborating, Learning and Adapting (CLA) agenda. UIP was expected to achieve the following outcomes:  Youth with increased employability skills; 2 There are a total of 32 total employers, some hosted interns across multiple districts. YouthMap Uganda Evaluation Report – June 5, 2015 11  Youth with increased practical work experience;  Increased youth research skills, and contributions to the CLA process (broader develop￾ment efforts); and  Improvements in young people’s employment situations (measurable six or more months following completion of internships). UIP intermediate outcome indicators from the M&E information are: 3  Number / percentage of graduates working at follow-up (disaggregate by salaried em￾ployment and self-employment)  Number / percentage of employers satisfied with program graduates hired  Number/percentage of employers replicating the internship programme  Number / percentage of youth reporting satisfaction with work  Degree to which youth participants feel the internship and other program services have contributed to their professional development and employability See Annex V Scope of Work for the remaining performance indicators. 3 However, the M&E framework was incomplete with some indicator data and targets missing.. YouthMap Uganda Evaluation Report – June 5, 2015 12 EVALUATION PURPOSE AND METHODOLOGY Purpose This performance evaluation is intended to assess UIP effectiveness and impact on beneficiaries as well as sustainability and scalability to inform similar internship programs that may be imple￾mented in the future. The evaluation is designed to provide a comprehensive analysis of all ele￾ments of the program, including: recruitment and selection, matching, training, and placement. It responds to the evaluation questions in depth, question by question. It also provides an assess￾ment of whether UIP objectives and outcomes have been achieved, with a particular focus on how the program has contributed to employability prospects for youth beneficiaries. The target audience of this evaluation includes key stakeholders ranging from USAID and the Government of Uganda to IYF, RD, private companies, and other youth-serving NGOs/donors interested in implementing internship programs. Methodology This evaluation used a mixed methods approach to access effectiveness, impact, sustainability and scalability. The evaluation team collected qualitative data and made use of RD’s performance monitoring and quantitative data obtained from surveys conducted before the evaluation. Data was collected from a variety of sources including: Primary data  Key informant interviews with program staff from IYF and RD, youth beneficiaries, em￾ployers, and Advisory Board members  Focus groups with the youth participants, youth non-participants and RD staff Secondary data  Baseline and follow-up surveys from youth participants of both cohorts  Formative evaluation report (internal assessment), which included a formative survey of cohort one and ten employer surveys  Program documents, including the RD contract, UIP proposal, quarterly reports, intern￾ship application documents, meeting minutes, employment tracker and M&E system in￾formation  CLA Assessment Report from cohort one YouthMap Uganda Evaluation Report – June 5, 2015 13 Baseline & Follow-up survey. The baseline was conducted at the beginning, after the intern se￾lection stage via an in-person survey. The follow-up survey was sent six months or more after the end of the internship via email as an e-survey. The follow-up survey was sent to all youth partic￾ipants though RD only received 54 responses. The evaluation team received a complete database of results from the baseline survey (n=98);4 and a database of follow-up survey results (n=54)5 . The evaluation team merged these databases, and cleaned the survey data for quality and useful￾ness in further analysis. Data quality analysis was carried out in several ways, first through a review of matching the types of information collected in each instrument, and then conducting data cleaning procedures to locate duplicates and missing data. For analysis, the sample size varied from 45 to 50 due to duplicates (2), missing baseline data (3), and several questions with no re￾sponses.6 Nevertheless, this is about 50 percent of the population, which is a good sample size. Formative Evaluation. A formative evaluation was conducted after cohort one as an internal pro￾gram assessment that was completed February 2014 by AIDTS Consultants and Data Development Group Inc., under the direction of RD and IYF. It included a formative survey with a sample of 35 youth participants from cohort one and ten employer surveys. The evaluation purpose was to assess the process of program implementation and inform later stages of the program and the second cohort training. The questions asked were different from the other survey as they were focused on program improvement for cohort two so the data were used to a limited degree. Program documents. The evaluation team reviewed the program documents, which included the RD contract, UIP proposal, quarterly reports, internship application documents, board meeting minutes, employment tracker and M&E system information. Of note, the M&E framework pro￾vided the development hypothesis, objectives, outcomes and output and outcome indicators. However, the indicator monitoring data was incomplete with missing data and targets. Focus Group Discussions. FGD participants were selected from a random-generated list. Three youth participant FGDs were conducted in Kampala. RD provided the team contact information for 70 participants. The evaluation team tried to contact all 70, but only reached 49. Of those, 20 planned to attend the FGD, and 13 actually attended. In addition, three youth non-participating FGDs were conducted in Kampala in the same manner with the aim of forming the basis for a quasi-experimental design. However, it was subsequently determined that conducting of this type 4 There are important data anomalies in that there are 93 matched names from the employment tracker in the base￾line survey and 5 mismatches. This may have been due to dropouts and substitutions from the period of the baseline through the inception of the employment tracker. 5 As of January 6, 2015, RD indicated that they had completed formative surveys for 54 individuals, and the evaluators have since received a database with these 54 cases. 6 The process also found a number of outliers that were removed. YouthMap Uganda Evaluation Report – June 5, 2015 14 of analysis was not feasible given data availability and quality constraints. Overall there were 6 FGDs of a total of 27 youths (13 youth participants and 15 youth non-participants). Key informant interviews. KIIs were selected purposefully to understand the program and its dif￾ferent components and stages. The evaluators conducted 21 KIIs, namely with RD staff (3), IYF staff (2), Board members (3), BYB consultant (1) youth participants (7) and selected host employers (5). Limitations Consistent with the original SoW, the evaluation team was in country in early December for 1.5 weeks for purposes of initial planning and inception report preparation. The team then returned to Uganda to finish the evaluation in early January for 1.5 weeks. The main data limitations were:  Limited monitoring data due to the fragmented nature of the M&E framework developed to document program achievements. While RD developed performance indicators, there were no complete monitoring data to assess their achievement.  FGD method may have limitations associated with recall, especially for the interns belong￾ing to cohort one. While FGDs were selected using a randomly-generated list, there may be some element of self-selection involved since the attendees had to be willing and avail￾able to attend FGDs.  While KIIs are purposively selected, time and travel constraints limited the number of in￾dividuals that the team was be able to visit. Although the team tried to schedule meetings with employers that have both favorable and critical views, there is the potential that par￾ticularly among employers, those that have expressed the greatest willingness and availa￾bility might potentially have views that differ in important ways from other employers who were not available.  The baseline and follow-up data provided by RD was gathered by Synapse Center in col￾laboration with IYF, and was provided to the evaluation team for cleaning and analysis. As of January 6, 2015, RD had only received 54 responses from the follow-up survey sent out to the 100 participants. During the cleaning process, the team identified several anoma￾lies: two duplicates, three with no baseline data, and several questions without any re￾sponses. Therefore, the sample size used for data analysis was between 45 and 50. Nev￾ertheless, the data are about half of the population, which is a good sample size.  Surveys have a non-respondent bias that may have shifted the mean and variance. Non￾response tends to shift the mean and variance as they are generally analyzed as zero. It is not known why participants chose to not respond. YouthMap Uganda Evaluation Report – June 5, 2015 15  Self-selection bias was introduced because the follow-up survey data was sent out as an e-survey, which reduces the response rate, as it is up to the participant to respond to the survey.  The wages earned data is not statistically significant due to: (i) the inclusion of unemployed persons and/or unpaid work along with those who earned much higher average salaries; (ii) a highly heterogeneous population reflecting: (a) inequality between men’s and women’s salaries, which in turn generates high variance, (b) differences in salaries and standard of living between rural and urban regions, and (c) major differences in average wages across sectors; and (iii) a small sample size which helps generate major variance around the average (even with a finite population correction, the dispersion around the mean is still high). Nonetheless, it should be noted that the data is still a sample of about half of the population.  It appears that some of the survey questions were inadequately worded, which could in￾troduce interpretation error. The quantitative data that was provided to the team had discrepancies that may reflect the wording of the questions, how they were interpreted by the participants. For example: (i) Youth W said he wasn’t currently working but the current work type indicated that he was an intern; (ii) Youth X said he is currently working, contin￾ued to work with the host employer, and was highly satisfied with his job, but didn’t list any work type for the current job or related salary information; (iii) Youth Y said he works on a contractual basis but no salary was provided; and (iv) Youth Z said he works as a volunteer and indicated that he received a salary.  The baseline and follow-up survey instrument used were inconsistent. Some relevant questions from the baseline and follow-up survey were not worded the same and hence reduce the comparability of the data and create ambiguities regarding interpretation. This is specifically the case, for example, for the questions regarding income and employment status. In the baseline survey, the question “What was your last work type?” could have meant the last work the youth had before the internship program but was not necessarily referring to the job the participant held right before entering the program. There was no question asked regarding whether or not the participant was currently employed, and un￾employment status was not a screening criterion for participating in the program. How￾ever, the follow-up survey did ask if the participant was currently working and what was his/her last work experience. These wording inconsistencies in the baseline and follow-up surveys appear to introduce considerable uncertainty in effectively interpreting responses and analyzing changes in employment status and quality. YouthMap Uganda Evaluation Report – June 5, 2015 16 FINDINGS The findings presented in this evaluation report are arranged in response to the six evaluation questions. The evaluation questions as contained in the SOW were mapped around two broad areas: program impact and effectiveness and program sustainability and scalability. Questions one through four address the program effectiveness and impact, and questions five and six ad￾dress sustainability and scalability issues. Effectiveness is defined as achievement of program objectives and assessment of program quality and value. Impact in this evaluation refers to the changes (results) produced by the program. Sustainability refers to the program being able to continue after the YouthMap pilot. Scalability will look at the program ability to geographically and over time reach more people. Additional information was also included to respond to other inquires about the program. A. Program Effectiveness and Impact 1. How effective were the approaches taken for each of the four program com￾ponents in achieving the goal? The evaluation team looked at how the four program components helped achieved the program development hypothesis: if educated youth are provided a comprehensive internship opportunity that fosters skills development, mentorship and peer support, then their chances of gainful employ￾ment will significantly increase. To better understand the approaches employed by the imple￾menting partner in the implementation of the four components, the evaluators reviewed the pro￾gram contract and proposal, quarterly performance reports, existing surveys and reports, other program documents. The evaluators also conducted KIIs with RD, IYF, Board members, youth participants and selected host employers and FGDs with youth who participated in the program. The following are the summary findings for each component. a. Orientation and skills training The orientation and foundational skills training was co-facilitated by RD staff and external facili￾tators from Mobile Telephone Network (MTN) Uganda, Development Finance Corporation in Uganda (DFCU) Bank and the Federation of Uganda Employers (FUE). The training was guided by the curriculum designed by KPMG. After successfully recruiting the interns, RD used an activity sequencing approach for intern orientation and skills training. The orientation included the foun￾dational skills training that were meant to prepare the interns for the formal work environment. Both the orientation and the skills trainings provided the interns with an overview of the intern￾ship, internship administration and logistics, and necessary skills to work in a formal work envi￾ronment. Specifically, the foundational skills training focused on life skills and workplace technical YouthMap Uganda Evaluation Report – June 5, 2015 17 skills. Life skills training covered issues such as problem-solving and conflict resolution, critical thinking and decision making, interpersonal relationships and communication skills, coping skills and flexibility, and community service. The workplace technical skills covered goal-setting, time management, work ethics and professionalism, Information and Communications Technology (ICT), effective written communication, financial literacy, team-building and leadership. Both the life skills and technical skills were meant to prepare the interns for employability. From the FGDs, the interns lauded the foundational skills training. In their opinion, the skills train￾ing prepared them for their internships as it covered areas such as professional and ethical be￾havior, networking and ICT. The workplace technical skills training enabled them to fit well in the placement throughout the internship period. According to the interns, the networking has be￾come very useful among the group. One of the interns said, “We connect with one another using social media (WhatsApp and Facebook) and the networking has continued even beyond the in￾ternship period.” The interns praised the ICT skills training they received at Makerere University Business School (MUBS). The MUBS computer lab is reported to be well equipped and every intern had a laptop during the training. This training was facilitated by the IT lecturers from MUBS. The training focused on basic computer skills such as: MS Word and Excel, as well as Internet skills (how to use the Internet for communication). They affirmed that the ICT training was important, especially as some of the interns had no previous exposure to computers and did not know how to use them. Training on life skills was also considered helpful by FGD youth participants. They found the training helpful to deal with different types of clients, manage stress during the internship, and negotiate salary. Several female participants indicated that the communication, leadership and time management training improved their confidence, which allowed them to communicate their thoughts and feelings more effectively while at the job. One female participant, who was placed in the hotel sector, noted that she was able to communicate her grievance to superiors when her supervisor did not provide the internship stipend. She reported, “I became assertive and de￾manded for what belonged to me. I went to the higher people.” Similarly, host employers praised the UIP orientation for familiarizing the managers with the in￾ternship. While most host supervisors were involved in the orientation, in some cases, the host employer contact person was not the intern’s supervisor so they were not fully aware of the in￾ternship details and intern’s role, and hence did not provide the necessary support and mentor￾ship. For example, one supervisor did not understand the concept of an “intern” and that the “interns have university degrees and are employable for available positions.”7 7 Transcript of RD Staff interview, January 7, 2015. YouthMap Uganda Evaluation Report – June 5, 2015 18 b. Internship placements From the review of the program description document, the team understood that the intern place￾ment was a guided process. The interns were matched to employers’ intern requests. The interns were placed in various sectors, ranging from Government, Non-Governmental Organizations (NGOs), Hotel & Tourism, Information and Communication Technology, Banking & Finance and Agriculture. In one youth participant FGD, 4 out of 6 interns said the internship did not specifically “match” their educational background, but helped them take their career interests to another level, e.g., the bank intern who moved from accounting to ICT administration and an NGO intern who developed a business strategy. Two employers noted that they thought the skill set of interns matched the skills they sought. One employer elaborated that the “prospective interns come from different backgrounds and it is not easy to get exactly the skill set required by the employer.” RD staff interviews also felt that some interns “really embraced the placement and some of them have gone the extra mile in their placements.” In the Employer survey, all ten host employers interviewed ranked the quality of interns highly. However, the RD staff did mention that some interns were not well matched to their intern place￾ment. “The problem we had was with two host employers where we had an intern who got placed in their Human Resource department, but the work load was not so high like the way she expected, but it was later addressed so there are just those very few cases.” In addition, there were two participants that felt the in￾ternship didn’t specifically “match” their educational background but helped them take their career interests to another sector, i.e. from accounting to IT services. Then there were two youth participants in FGDs who informed the team that they were working in the same company and position as before the internship. After the internship, they continued to work in their respec￾tive positions. It should also be noted that one participant in Gulu indicated that there were fewer employers and, as a result, fewer opportunities for placements in regions outside of Kampala. Two Board members noted interns are recent graduates that do not have the skill set to work in a formal workplace. Hence there was a high demand for the YouthMap internship program. “The skills gap between university and what the employer requires for employment is huge.” This mir￾rors the sentiment reported by the managers in the Employer Survey. Half of the managers noted Internship Eligibility Requirements 1. A Ugandan national; 2. A resident in Kampala or one of the target Districts; 3. Aged 23-30 years old on 29th March 2013; 4. A holder of a diploma or under￾graduate degree; and 5. Fluent in both written and spo￾ken English. YouthMap Uganda Evaluation Report – June 5, 2015 19 that young staff fail to meet their employment requirements because they lack certain skills, such as computer, communications and other related skills. In terms of stipends, the evaluation team was unable to determine the process or who and how stipends were calculated. However, one Board member and an IYF staff person said that they were too high and unsustainable. The IYF staff also reported that on one occasion, a youth par￾ticipant was paid more than his/her colleagues, which was a potential factor for tension in the office. c. Professional development support services Professional development services include mentoring, job placement support and participation in RD alumni network. The following are the summary findings: i. Mentoring The mentoring of interns was a primary responsibility of RD. However, some employers also took it upon themselves to provide on-the-job mentoring. The evaluation team interaction with RD management and program staff revealed that RD provided necessary guidance and mentorship support to interns in both cohorts on a regular basis for the duration of the internship placement and beyond. The overarching purpose of the close follow-up and mentorship support was to: (i) monitor the progress of intern activities as spelled out in their work & career development plans; (ii) solicit feedback from interns and host employers on areas of the interns’ progress requiring improvement. The interns that participated in the FGDs highlighted that the professional support services like mentoring by RD were helpful to gain knowledge and skills specific to the workplace, for example two youth participants said their supervisor assisted in their career planning and two mentioned that their supervisor built critical work skills on the job. According to some interns, “the RD officers used to come regularly to provide mentoring, monitor their progress and provide feedback on their monthly reports.” The RD staff also said that interns “received regular monthly feedback from employers on how they were doing on their work plan, and assistance in improving skills in one or more areas where needed.” In an interview, one employer discussed offering additional support (beyond what was required in the program) to the interns during their placement. The employer hosted a total of three interns (two from cohort one and one from cohort two). While they had different supervisors, the head of the Finance and Administration Department directly provided the overall mentoring to all the interns. He said, “We ensured that the interns were given the space and delegated more tasks so that they learn as much as possible within the six-month period of the internship. There was also close follow-up monitoring from RD, especially from the program officers. There was close follow up and monitoring to assess the progress the interns were making as well as to share the areas YouthMap Uganda Evaluation Report – June 5, 2015 20 requiring improvement. I also had an opportunity to meet and share intern placement experi￾ences with other employers after the placement period.” ii. Job placement support In general, the job placement support activities were reasonably successful, with about 78 percent (39/50) of the youth participants employed after the program. This compares quite favorably with an official rate of unemployment for labor force participants with tertiary education in Uganda of approximately 35 percent. During the training, interns received career planning and job search skills training with a focus on self-awareness, job search tools and methods, CV development and application writing and interview techniques. During the job placement, interns were supposed to receive regular support from their immediate supervisors, including through regular meetings. The supervisors were required to regularly monitor intern performance, offering advice and pro￾fessional guidance, and identifying areas needing improvement. Interviews with host employers confirmed that the interns were provided with the necessary on-the job professional guidance and support. One employer had this to say about his/her intern, “The intern hosted by us is a graduate of Fine Arts. We had him work in our Monitoring and Evaluation Unit. He was determined and willing to learn. He was a quick learner who got very much interested in handling data. He impressed his imme￾diate supervisor who helped him acquire requisite skills related to data management. The intern has what it takes, that is why he is now working as our Data Manager. When the intern’s supervisor left, we did not even advertise for this position. We gave this op￾portunity to him. He has fit very well and he has not disappointed us.” The interns also expressed their satisfaction with the supervision by the employer and regular feedback on areas for improvement. They emphasized that close supervision and guidance pro￾vided by their immediate supervisors enabled them to learn on-the-job skills faster. The interns from the previous employer’s story said, “We were placed in the departments which were in line with our educational backgrounds. This helped us to learn faster since it was easy to follow through the instruction for completing tasks.” However, some supervisors, according to RD staff interviews, were too busy or travelled fre￾quently, and so were not able to regularly meet with interns and offer advice and professional Internship Application Scoring Dimensions 1. Completed personal details (NA pts) 2. Background assessment based on four crite￾ria: Disabled; living in a household of 7+ peo￾ple; living where head of household is not working; and orphan. (3 pts max) 3. Education type (3 pts max) 4. Additional qualifications/courses (NA pts) 5. Language skills ( 3pt max) 6. Self-assessment (3 pts max) 7. Leadership abilities (3 pts max) 8. Motivation (3pts max) YouthMap Uganda Evaluation Report – June 5, 2015 21 guidance. It was also noted that there were some issues of supervisor turnover, especially in the banking sector. “Some interns felt their supervisors did not offer enough support, or did not have enough time to help interns with career development.”8 The formative evaluation noted that all interviewed interns “indicated that they receive a stipend from RD on time through their bank accounts, indicating good management of stipends by the program team. Interns with USAID implementing partners and employers in the banking sector got employer cost share contributions on time and with ease. However, eight interns placed with employers in the private sector and the Parliamentary Forum on Youth Affairs had persistent dif￾ficulties in accessing host employer cost-share contributions.” The formative evaluation team rec￾ommended better definition of in-kind cost share for employers. It was also noted in that evalu￾ation report that the government had a ban on public service recruitment, which then limits the chance that those interns with government placements gain employment with the government. iii. Participation in UIP Alumni Network Throughout the program, RD organized networking events and invited interns to them. “Five interns got opportunities through getting contacts from such a networking event.” RD also held a career fair after the internship to support and assist interns in obtaining a job and network with employers and other youths. For cohort one, they invited employers (like the Uganda Revenue Authority) to come speak about existing opportunities and entry points to access these jobs. One intern followed up on her contact with a bank and she is currently working with the bank as a full￾time employee. However, one participant who is based in Gulu noted that most of the career events were in Kampala and that it was difficult for him to travel to them. He elaborated that he felt “neglected” as he was unable to participate. d. USAID Collaborating, Learning and Adapting (CLA) Approach, Internal Assess￾ment and Entrepreneurship Training9 CLA is a research component of the YouthMap Uganda program, where YouthMap interns con￾tribute to youth perspectives on USAID’s priority sectors. The first cohort, involving 12 youths who were attached to USAID implementing partners (IPs), conducted research on access to and perception of sexual and reproductive health (SRH) facilities in partnership with key USAID IPs. The SRH assessment report was completed in May 2014. For cohort two, the Mission selected Youth in Agriculture as the research focus to inform partners who are working on the Feed the 8 Mike Midling, transcripts of RD staff KIIs, 1/7/15. 9 Most of the information related to CLA came from: YouthMap Uganda interns, IYF and RD. “Youth-Led Research: Access to Sexual Reproductive Health Packages for Young People.” To contribute to USAID CLA agenda. May 2014, pages 6-7. YouthMap Uganda Evaluation Report – June 5, 2015 22 Future initiative. The nine youth researchers were able to develop a questionnaire and research plan but did not have time to complete the research. An internal YouthMap assessment, the formative evaluation, was completed in February 2014. The formative evaluation was conducted by AIDTS Consultants and Data Development Group Inc., under the direction of RD and IYF. The purpose of the evaluation was to assess at the mid-point implementation stage whether the program was implemented according to its original plan and design, review the program progress to date, access delivery processes and participant satisfac￾tion, and provide early indication of the results. Build Your Business (BYB) entrepreneurship training was a component that required youth partic￾ipants to form business groups after the foundational training. Both cohorts received introductory training on BYB, but since there was no further training or guidance during cohort one, only cohort two had BYB groups remaining. Groups were made up of between four and six youth participants who developed a business proposal in the hopes that they would be provided seed money to start their business venture. There were nine BYB groups formed under RD guidance, however, due to delay in entrepreneurship follow-up (six months), three dropped out. A BYB consultant was hired post-internship to support the five groups with outstanding business proposals. i. Facilitation of youth participation in USAID CLA-related activities Both CLA research topics were selected by the USAID/Uganda Mission. The Mission and IPs gave input into the survey tool and methodology. The CLA has a learning focus, whereby the youth interns learn through the research pro￾cess by providing insights into percep￾tions commonly held by young people who can inform further discussion and learning on the topic for USAID and the IPs. The research was a youth-led consul￾tative process with interns, RD and IP or￾ganizations. This approach was taken to ensure that the issues of youth were able to really be explored by fellow youth, to provide an opportunity to explore and share peer experiences, and to collect in￾formation as effectively as possible to inform USAID and its IP partners about the topics. “Before conducting the research, I had never par￾ticipated in any research where you are involved in all levels of the research process, that is, question￾naire design, identification of [the] sample group, design methodology, data collection, data analysis and report writing. I had never participated in such a research besides a youth led research…It has been really good learning experience and I was able to gain research experience in specific research areas.” John Dhikonte Act4africa, Jinja District YouthMap Uganda Evaluation Report – June 5, 2015 23 The CLA research process for cohort one was thorough. The intern researchers were initially trained on the CLA agenda and research methodologies by RD and the initial research questions put forward by USAID and IYF were redrafted by the intern research team to be more user-friendly and understood by target youth. The draft tool that the intern researchers produced was shared with USAID and implementing partners for feedback and collective agreement on the way for￾ward. RD led consultative meetings with implementing partners to finalize the tool, and sampling framework was finalized with the intern researchers who also pretested the tool. The research was implemented by the 12 intern researchers and data entry and cleaning of the collected data carried out by a youth alumni team supervised by RD in consultation with IYF. Validation and analysis of the data was conducted by the intern research team during a two-day validation work￾shop. The intern researchers presented a summary of their research to USAID and contributed and reviewed the final report. The intern research team participated in all stages of the research from design, imple￾mentation, analysis, validation and dissemi￾nation of the research. Throughout all stages IYF and RD worked in partnership with the interns. The intern researchers felt that they received an in-depth overview of the CLA research process. The experience expanded their knowledge of conducting primary research and the sup￾port from RD was adequate for the research to be implemented. Intern researchers indicated that the research was successful and that they had ownership over the process because they led it. However, youth researchers during the CLA validation workshop provided the following areas for improvement: “There was no clear work plan for the CLA collection (overlapping of activities), the questionnaire was too long and the time allocated for data collection was too short, facilitation was inadequate in terms of logistics (i.e. printing, long distances in some locations), interns were not involved in data analysis as promised, expectations from interviewees on stipends differed, and the intern researchers were not able to be involved in all data processing stages.” Based on feedback from cohort one CLA research, it was determined that a more concise tool should be developed to allow for more streamlined development, less confusion among respond￾ents in data collection, and more accurate analysis for cohort two implementation. Therefore, the RD team proposed one tool with quantitative and qualitative questions in one document, which required interns to spend more time with each interviewee. Unfortunately, cohort two nine interns did not have enough time to finish the research. RD staff noted that perhaps more efficient plan- “Before conducting this CLA research, I had been involved in other research projects where I was mainly doing data collection but CLA research has really been so detailed where we had to design the questionnaire ourselves, collect data, validate it and finally contribute to report writing…I have gained knowledge and experience in basic re￾search and data management which has contrib￾uted a lot towards my career.” Clare Atwongireho Strengthening Decentrali￾sation for Sustainability, YouthMap Uganda Evaluation Report – June 5, 2015 24 ning could have yielded more time-effective results. In this regard, cohort one succeeded in per￾forming their research during their intern placement. By contrast, cohort two had to wait for feedback on the survey tool for a month, truncating the research timeframe. Moreover, getting interns on board after they have already completed their internship placement was difficult. They also voiced frustration with the timeframe in regards to the training. “Some of the interns dropped off and we needed to train new ones. Yet after being trained, they are no longer available to collect the data. Delays affected our timeline.” There also appeared to be a problem with delayed feedback from the USAID Mission. As of January 2015, RD and the interns have reportedly not received feedback for the cohort one CLA final report from the Mission. ii. Usefulness of an internal YouthMap Assessment for designing and or im￾proving programs There was one internal YouthMap assessment completed during the program. The formative evaluation was conducted after cohort one and provided findings that helped the RD team better understand the program. One IYF staff person said that he was “disappointed” about the report. However, he did say that despite the quality of the report, they did gain some important knowledge. The major lesson was making sure employers were engaged early and that roles and expectations were understood by all parties, which they tried to do in cohort two. Nonetheless, it is not clear to the final evaluation team if any other adjustments based on the formative evalu￾ation were applied. This warrants additional clarification. YouthMap Uganda Evaluation Report – June 5, 2015 25 iii. Build Your Business (BYB) Entrepreneurship Training10 RD designed BYB as an income-generation alternative to employment. RD provided the BYB con￾sultant who supported five BYB groups: magic crafts, fresh groceries delivery, mushroom growing, live￾stock fattening, and savannah goat breeding. The consultant connected the groups to banks but the latter proved reluctant to lend money to the groups because of lack of collat￾eral and the fact that the group members did not have saving ac￾counts at the referenced banks. The consultant identified leaders among the groups to guide group activities. He also provided them with exposure to an ILO and EU exhibition and in￾troduced them to his network of en￾trepreneurs so the youth could learn and ask questions. While entrepreneurial support was valuable to the youth, it was pro￾vided during the latter stages of the program and may not have been as useful to many as it could have been, since promised seed funding was not available to undertake investment in the planned business. There was a lack of leadership and guidance until after the internship period. The participating youths were promised seed funding but the funding was not available because RD seemed to experience difficulties in determining how to manage the funds, which disappointed the youth since they had undergone training, formed groups, vet￾ted business ideas and saved jointly (thanks to the internship stipend). Some members dropped 10 Information for BYB was mainly from two sources: 1) Mike Midling, transcript of key informant interview of Otim Gerald, 1/9/15; 2) two youth participants in FGD. Cohort Two BYB Groups  Magic Crafts Group. This group was connected to guest entrepreneur who also deals in craft exports to the USA. Through close collaboration with the guest entrepreneur, the group was able to generate initial ex￾port sales in November 2014.  Fresh Delivery group (dealing in vegetables and fruits). With guidance from the consultant, the group refined their business idea and focused on house –to￾house deliveries. They source their supplies locally through fresh vegetable and fruit farmers. However, the group lacked support staff to help them do house-to￾house deliveries as all the group members have full time employment.  The Mushroom Growing Group. The group com￾prised of five women (but two dropped out later). The group managed to raise 1.5milliom Uganda Shillings part of which they used to hire an expert trainer in mushroom farming and for establishing a mushroom growing structure. The group started their mushroom farming in October 2014 and had their initial mushroom sale in November 2014.  Livestock Group. This comprised 4 men. They have agreed to contribute 100,000 Uganda Shillings per month. They accumulated 2 Million shillings. They plan to buy and fatten livestock for re-sale. They will be making their initial purchase in the months of January and February 2015.  The Savannah Goat breading Group. This group is based in Jinja. They are still at the business idea stage. Their core concept is to source savannah goats and cross bread with local goats. YouthMap Uganda Evaluation Report – June 5, 2015 26 out of the BYB groups. Participants from two FGDs indicated that the support and entrepreneur￾ship training from the consultant was useful as they actually followed through and started their business (three groups). Regardless of the disappointment which arose from the lack of promised access to start-up finan￾cial resources, the entrepreneurial skills taught interns how to develop and manage a business, and opened a window of opportunity for the interns into entrepreneurial activity as an income generation alternative. In fact, 18 youth participants reported having a business or being a con￾sultant/contractor at follow-up survey. The consultant provided these recommendations for future programing: “The person who trains the BYB group members should also be the same person to provide follow-up support and men￾toring for purposes of continuity and relationship building. The follow-up and mentoring should be implemented immediately after the groups have been formed and should continue for the entire period of internship. The BYB group should be allowed to form naturally and should be based on the shared vision of the group members to ensure commitment and group cohesion. The program should encourage the internship implementing partner to retain a percentage of the interns’ stipend to constitute a savings pool from which the BYB groups can borrow from at a minimal interest rate.” 2. What was the program’s overall impact on youth beneficiaries? a. Achieving Development Hypothesis and Objectives The extent to which the program achieved its development hypothesis and objectives is compli￾cated by the fragmented nature of the M&E framework developed to document program achieve￾ments. Thus, while RD did develop performance indicators, there are incomplete monitoring data against which to assess their achievements. RD did track participants to analyze their location, job sector, employment status and employment details. Baseline and follow-up surveys for both cohorts provided another way to assess goals and objective achievements. Given the program’s development hypothesis and three objectives, it appears from the quantitative data that they were largely achieved as documented by:  78 percent (39/50) of youth participants are currently employed. This compares favorably with an official employment rate of approximately. 65 percent for labor force participants with a tertiary degree;  56 percent (28/50) of youth participants indicated that the internship helped them obtain a job;  92 percent (46/50) of youth participants indicated that the internship improved their em￾ployment options; YouthMap Uganda Evaluation Report – June 5, 2015 27  18 youth participants reported becoming business owners or consultant/contractor at the time of the follow-up survey; and  21 youth participants in the CLA training and research process. The youth participants conducted research on Sexual Reproductive Health and Youth in Agriculture based on the CLA training provided by RD. Cohort one completed the CLA research process. Cohort two developed a questionnaire and research plan for the Agriculture assessment but did not complete the analysis and report. b. Impact on Youth Beneficiaries From KIIs conducted, it seems like most of the youth participants were recent graduates from a university and have little or no formal work experience. The youth participants in FGDs and KII cited the following program interventions as particularly impactful: mentors, foundational life and technical skills, placement/hands-on training and networking. On mentorship, they felt it was an asset to have a professional mentor who they could turn to with questions, get support, get feed￾back and career development advice. Regarding training, many indicated that the training RD offered increased their knowledge in essential professional skills, including ICT, CV writing, com￾munication, leadership, financial management, entrepreneurial skills, and project management. The acquisition of these skills increased participant self-confidence and were utilized when com￾municating with others, team building, problem solving in the workplace and with others, building their own business, teaching other youth, getting a job and even being placed into a better job. One male youth said, “YouthMap helped me in several significant ways to develop the personal and professional skills to succeed in today’s market. These included placement in a professional sector where I wanted to work and had the educational background to do so, acquisition of en￾trepreneurial and business skills which enabled me to learn how to manage finances, start and sustain a business as a fall back economic opportunity, and career guidance from RD during mid￾term and end of internship visits, which encouraged me to continue to develop myself and my professional skills and overcome obstacles in my way as I seek to enter the workplace.” Several female youths noted during KIIs that the training improved their ability to communicate and ex￾press themselves, which resulted in increased confidence at job interviews and with supervisors while on the job. One female youth elaborated during an FGD that, “The foundation training was the best thing that happened and gave me confidence at placement as I met challenges of sexual harassment at my placement workplace. The training gave us life skills and professional behavior ethics.” From the formative survey of cohort one, 86 percent (30/35) said the program met their expecta￾tions. Of the 14 percent (5/35) that said no, respondents indicated that they expected more tech￾nical training (2 interns), thought they were going to learn more (2 interns), and/or thought they YouthMap Uganda Evaluation Report – June 5, 2015 28 would get a better job (1 intern). Of the 35 respondents, 77 percent indicated that it would be very easy or easy for them to find a job. From the baseline and follow-up surveys, the team was able to ascertain findings to address this question. The following two graphs and table, generated from these surveys, illustrate the program impact of the UIP on the interns, as perceived by the same interns. Graph 1: Aspect of program Most Useful to What You Are Doing Now? Graph 1 indicates that the top four most useful aspects of the program were: networking, work￾place (technical) foundational training, life skills foundational training, and on-the job training, which corresponds to the FGD and KIIs responses regarding program impact on the youth bene￾ficiaries. It is interesting to note that more women ranked networking higher than men, and life skills training higher than men, while more men ranked workplace foundational training higher than women. Women seem to have different needs/priorities in terms of what is most useful for them. For example, eight women found entrepreneurship/financial skills training most useful. The team advises in this regard that a gender analysis be conducted on youth in the workforce to better inform programming. Another interesting result is that mentorship during the internship period was ranked of limited usefulness. This is contrary to the fact that RD and employers seemed to be in close and frequent contact with the interns in general. Interns may have inter￾preted mentorship as on-the-job training and/or job placement support. To better address this issue, survey questions need to be worded carefully with input from youths, and the questionnaire should be tested. Female 7 3 6 4 3 1 Male 4 7 2 4 2 2 1 Networking Workplace Foundational training Life skills foudnational training On-the-job training Job placement support Entrepreneurship/ financial skills training Mentorship during internship Blank 11 10 8 8 3 4 5 1 Most useful aspect of the program Follow-up n=50 YouthMap Uganda Evaluation Report – June 5, 2015 29 Graph 2: Level of Satisfaction Graph 2 indicates that 70 percent of interns (20 women, 14 men/50) were either highly satisfied or somewhat satisfied with their job or primary income-generating activity. Looking at the gender breakdown, there are no striking differences in the numbers of men and women who were either somewhat dissatisfied with their job/work or did not respond. The finding that ten interns re￾ported being somewhat dissatisfied and six did not respond may, at least in part, reflect the fact that eleven participants were not employed at the time of the follow-up survey. Table 3: Wages Earned of all Interns Reporting a Wage Group Sample Size U.S. Dollar Average of Baseline Monthly Earn￾ings Average of Endline Monthly Earn￾ings Average of Dif￾ference in Earn￾ings StdDev of Dif￾ference in Earn￾ings Female 25 38.50 182.64 144.14 192.80 Male 21 69.67 174.13 104.47 183.80 Grand Total 46 52.73 178.75 126.03 157.33 Of the 54 who responded to the baseline and follow-up survey, 46 reported a wage either at baseline or in the follow-up survey. However of the 46, 30 reported no wages at baseline, which we assumed means they were unemployed, and 16 reported last month wages at baseline, which we are utilizing for purposes of establishing their pre-program monthly earnings for this analysis. Accordingly, a comparison of the baseline and follow-up surveys yields several findings. Breaking down the salary increases noted above by the gender of the participants indicates that the average monthly increase for women was $144 and $104 for men. Of those participants who reported a salary prior to the program, the average increase was $11 monthly ($8 for women and $13 for men). Of those who reported an income after the internship, the increase averaged $181 monthly Female 4 16 4 4 Male 4 10 6 2 Highly satisfied Somewhat satisfied Somewhat dissatisfied Blank Satisfaction with current job/income generating activity Follow-up n=50 YouthMap Uganda Evaluation Report – June 5, 2015 30 ($218 for women and $144 for men). In addition, those who reported salaries before and after the internship program received relatively small increases in average earnings ($44 average total monthly increase of which women averaged $31 and men $53). The data also shows that 46 of these participants earned an average of $126 in additional monthly income (or an average annual the increase of $1,512) after completion of the program.11 It should again be noted that this includes both those who were unemployed prior to the program, as well as those who held some type of job when the program started. As might be expected, the largest average salary increases occurred with those who were unemployed at the start of the program. This implies an internal rate of return (IRR) in financial prices of only 5.5 percent, based on the following:  This includes all costs of the program, including direct and indirect costs and one-half the headquarters/global cost allocated to IYF12.  Calculation allowed continuous increments in real net earnings of a constant amount an￾nually, and assumed that these constant increments reflected benefits of the program.  Calculation assumes there was one (and only one) increase in earnings for the recipients as a result of the program and this increase affected each year of their lifetime stream of net earnings.  The calculation assumes no salary increases after the initial $126 monthly increase (pro￾jected out over a 30 year period). An average annual increase of 5 percent this would raise the IRR to 10.2 percent. With or without the assumption of a five percent annual average salary increase, the rates of return are too low to justify the program – as it was implemented -- on cost-benefit criteria13. However, it is important to keep in mind that this is a pilot whose costs may be decreased or the number and salaries of the beneficiaries increased as the program is expanded. But even more important, it is important to underscore that none of the dollar amounts reported are statistically significant. This is because the variances of all the samples cited above are high. In fact, all the standard deviations exceed the average. This means that confidence intervals drawn around the average increases in income would include zero at any of the common levels of significance (99 percent, 95 percent or 90 percent). Three factors contribute to the high dispersion around the average: (i) 11 Eight of the 54 respondents were eliminated from the calculation because two left the question blank and six indi￾cated that they were employed but did not report their monthly earnings. The remaining 46 included 34 participants with earnings and 12 who reported no earnings at follow-up. Presumably the latter were either unemployed or not in the labor force. 12 The other half is allocated to the Senegal YouthMap program. 13 A rate of return of at least 12% is commonly required for projects financed by international development institu￾tions. YouthMap Uganda Evaluation Report – June 5, 2015 31 the inclusion of unemployed persons and/or unpaid work (30/46) along with those who (by defi￾nition) earned much higher average salaries; (ii) a highly heterogeneous population reflecting: (a) inequality between men’s and women’s salaries, which in turn generates high variance, (b) differ￾ences in salaries and standard of living between rural and urban regions, and (c) major differences in average wages across sectors; and (iii) a small sample size which helps generate major variance around the average (even with a finite population correction, the variance is still high). While increasing the response rate to the follow-up survey might improve statistical significance of the data, the highly heterogeneous nature of the beneficiary population would still likely limit the statistical rigor of the results. c. Strength and Weakness of the Four Program Components Specific observed strengths of the four program components are:  Participatory approach used in program design kept all the key actors engaged during the implementation.  Orientation and foundational training prepared interns for the workplace. Interns espe￾cially mentioned the life skills training and ICT training.  Internship placement matched most interns well with their host employers.  Regular site visits by RD staff to interns in their place of work for support and monitoring ensured the successful implementation of their work plans.  Interns who were able to set clear goals during the foundation training seemed to do well in their placements, as they focused on job activities aligned with their goals.  RD organized and encouraged networking through events and job fairs.  RD and host employer mentoring on career development and individual work plan devel￾opment helped interns improve their skills in needed areas.  Interns received on-the job training where they learn specific skills in their sector.  Intern researchers for CLA activities learned how to conduct research, which is an im￾portant skill to have in the workplace.  Youth participants in BYB groups learned business skills and some was able to apply their training in setting up their businesses. Specific observed weaknesses of the four program components are:  Orientation and training was basic, and not necessarily targeted toward the specific in￾ternship sector. Interns lack the specific skills for their respective sectors.  Orientation activities were not carried out with the specific employer manager or supervi￾sor. Sometimes, the contact person that received the orientation was not the intern su￾pervisor. YouthMap Uganda Evaluation Report – June 5, 2015 32  Internship placements were based on employer intern requests, which may not have ex￾actly matched the HR needs/interests of participants.  There was no relationship with the HR department. The relationship was with the super￾visor, and in some cases, the intern was left without a supervisor when s/he travelled or were no longer employed.  There were fewer placement opportunities in regions outside of Kampala.  There were no networking events in other regions except in Kampala.  CLA research organizers had high expectations of youth participants. Feedback from USAID on process outcomes and reports delayed CLA implementation (more so for cohort two).  BYB was as an add-on activity. The groups had no leadership, support or instructions for six months after the groups were formed.  Seed funding for the BYB did not materialize, which caused many youth participants to drop out of the groups. d. Key Factors that Enabled Interns to Secure Employment/Remain Employed From the current information, we can provide the following broad perspective about what factors enable interns to secure employment or remain employed (it should be noted that neither the quantitative nor qualitative data touched directly on this topic).  Combination of workplace technical and life skills training before placement—especially on career development and life skills—better prepared youth participants for the work￾place.  Partnership development with the host employers strengthened the relationship between RD and employer.  Regular site visits by RD staff to interns in their place of work for support and monitoring ensured the successful implementation of their work plans.  Close and regular mentorship and support which develops trust among interns and em￾ployers.  Support interns to set and achieve clear goals. Interns who were able to set clear goals and know what they want in a career during the foundation training seemed to do well in their placements, as they focused on job activities aligned with their goals.  Willingness to work hard and learn/adapt to the specifics of the workplace is essential. Even though some interns’ background did not match his/her placement, the intern could be successful when s/he displayed willingness to learn and worked hard, which increased the interest of the employer in supporting and mentoring the intern. YouthMap Uganda Evaluation Report – June 5, 2015 33  Appropriate capacity of employers to retain interns for full time jobs. Some youths indi￾cated that some organizations did not have the ability to hire interns when the program concluded.  For CLA activities, it was clear that having youth-led research made a difference in the outcome. The intern researchers felt they owned the entire process and hence they were more invested in getting it done and getting feedback.  For BYB activities, what motivated the youth participants was the potential to start their own business, and to generate income from their own initiative. Hence when the seed funds didn’t materialize, the interns were disappointed and some dropped out of the group venture. Those that succeed in starting their business did so due to their own con￾viction, savings and the training and mentoring they received from the BYB consultant. e. Advisory Role and Contribution to the Achievement of the Program Goal The Advisory Board role is to help guide the program’s design, direction, and sustainability. The Board met three to four times per year with USAID, IYF, and RD staff. It provided mostly strategic guidance and direction. One Advisory Board member said that “Our role as Board members is mainly advisory. We give advice to RD on how to implement the YouthMap internship program. We are also involved in identifying employers to take in interns and host them together with RD. As Board members we provided feedback in meetings to RD.” Given the information provided through the three Board member interviews and from Board minutes, it seems that the operation of the Board has contributed to the achievement of the pro￾gram goal in three ways: (1) availability to RD staff for consultation through emails and phone calls regarding programming; (2) advocated, recruited and accepted interns in their company; (3) exploring sustainability ideas, such as canvassing members of USAID regarding seed funding strategy and related proposal writing, and working closely with the Ministry of Gender, Labour and Social Development (MGLSD) to discuss an internship/volunteer policy with Parliament. One Board member interviewed underscored in this regard the importance of work being done in this area by Hon. Gerald Karuhanga, Youth MP in Western Uganda, who he reported is currently leading development of an internship/volunteer policy document. Unfortunately, the evaluation team was unable to get an in-depth interview with Hon. Gerald Karuhanga. Another Board member noted that the Board should also include youth as members and the program should focus on micro-enterprises as most of the employment in Uganda is in the infor￾mal sector. As a result, there are many more micro-enterprises than formal companies. If the program is able to place interns with micro-enterprises, there will be many more opportunities for placement. YouthMap Uganda Evaluation Report – June 5, 2015 34 3. How was the capacity of various program partners strengthened through the UIP and what were the outcomes of these partnerships? This is an important question, though it appears to be outside the formal scope of the program. The program development hypothesis and objectives did not reference strengthening internship partners. The SoW does not specifically define any activities which would be designed to result in capacity strengthening for program partners. In a fundamental sense the capacity strengthen￾ing impact of the program on partners can best be viewed from the vantage point of the success with which the intern placement program filled important human resource gaps of the host em￾ployers. Indeed the key informant interviews with a number of host employers provided evidence that employers were involved in the intern placement process. According to an HR manager, “We rec￾ognize the fact that there is a lot of graduate youth unemployment in the country and we are open for partnership with organizations implementing internship programs. We are an organiza￾tion that has an internship policy in place. It is therefore our role to always be on a look out for deserving graduates for placement. We closely worked with RD and their program. We also have a succession strategy for filling these positions. This strategy links well with internship placement programs being implemented by RD. We have certainly benefitted from the placement of the intern as he has now transitioned from intern to full-time employment. We also benefitted from the foundational skills provided to the interns by RD. These skills prepared the intern better for placement, which he also used very well as an opportunity to gain work experience. When he worked as an intern for six months, he was very useful and we treated him as one of our team members. The partnership has specifically benefitted us in terms of talent identification, human resource development and has equally helped us address our HR gaps.” One Finance and Administration Manager shared that “the partnership is good because we have challenges with human resources. The cost associated with recruitment is high. It is good to work with partners like RD who are implementing internship programs. We declare our HR needs and skills to them. We take on interns, place them in the right departments, provide appropriate induction and build their skills on the job. The outcome of this process is that the interns are likely to transition into full-time employment. This is what has happened to interns we have retained in full-time employment.” A Chief Operations Officer noted that “as an organization, we require human resources to manage our huge scope of operations. RD is providing that human resource in the form of interns. They have the HR supply which we have the demand for. RD provided a pool of interns from which we could identify the interns who are trainable. RD provided appropriate pre-placement training YouthMap Uganda Evaluation Report – June 5, 2015 35 which enabled the interns to easily integrate in the organization work culture. It helped the or￾ganization to fill their human resources gap. At the end of the internship period, the organization management team met and assessed the individual competencies of interns. This was the process used for selecting the interns who were to be retained and given full-time employment offers.” 4. What are the trends in intern employment data with respect to gender, sec￾tors of interest, employer retention, and employment quality (pt/ft, salary, hours, location and promotions)? From the baseline and follow-up surveys, the team was able to ascertain findings to address this question. Graphs 3 through 5 below were generated from these surveys and illustrate the intern employment trends with respect to gender, sectors of employment, employer retention and em￾ployment quality. Graph 3: Interns by Sector Graph 3 illustrates the number of women and men interning in the various sectors. The largest proportion of interns (nearly one-third) was placed at NGOs. This may be because this is a donor￾funded program and the CLA research process also involved the USAID IPs. There are more women than men overall, and especially in three sectors: Agriculture, government, and infor￾mation and communication. There were considerably more men than women in Banking and Finance. 5 7 6 10 8 Female 15 4 3 7 6 12 Male 17 Agriculture Government Hospitality & Tourism Information & communication Banking & Finance NGO Internships by Sector n=100 YouthMap Uganda Evaluation Report – June 5, 2015 36 Graph 4: Interns by District Graph 4 illustrates that the majority of the interns (approximately two thirds) were placed in Kam￾pala, the capitol city, and there was general gender balance in the placements across the three regions (central, eastern and western), except for the northern region (9 women, 6 men). By re￾gion, there were 13 interns in northern, 16 interns in eastern, 5 interns in western and 64 in the central region. Given the scarcity of formal companies and job opportunities in rural areas, and the resulting migration of job-seeking youths towards urban centers, it is not surprising that there were significantly more internships in Kampala than elsewhere. In terms of professional develop￾ment support to interns in other regions, one youth participant reported that he felt neglected because all the networking events were in Kampala, and he was not able to attend them as a result. Graph 5: Intern Most Recent Work Experience Comparison14 Graph 5 compares the baseline and follow-up data in regards to the last job held by participants prior to their involvement in the internship program. The team is assuming that this refers to the last form of employment the youth participant had prior to the program, and does not necessary refer to a position held just prior to entering the internship program. In other words, the data 14 It is important to note that because some questions in the baseline and follow-up survey were not worded in the same manner and were not necessarily interpreted in the same manner by respondents, this reduces the comparabil￾ity of the data. In the baseline survey, the question “What was your last work type?” is not clear. It could have meant the last job the youth had held before the internship program was initiated, but did not necessary refer to whether or not the person held a job just before entering the program. No question was asked regarding whether or not the participant was currently employed and unemployment was not a criterion for entering the program. However, the follow-up survey did ask if the participant was currently working and what was their last work experience. 1 1 2 2 6 6 Female 33 1 2 3 3 9 Male 31 Busia Kitgum Lira Mbarara Gulu Jinja Kampala Participants by Region n=100 13 (Baseline) 3 7 15 0 12 4 (Follow-up) 9 2 24 1 10 Internship Permanent Staff Seasonal/Occasional by hours or days Temporary Contractual Other Blank Baseline n=50, Follow-up n=50 YouthMap Uganda Evaluation Report – June 5, 2015 37 could be referencing a job held two weeks or 2 years prior to entering the program – the way the question has been asked does not allow us to establish the timeframe within which the referenced position was held. Taken at face value, at baseline (before the internship), the majority of the participants were working either in an Internship (26 percent) or in Temporary Contractual work (30 percent). At follow-up (after the internship), the youths mostly work in Temporary Contractual work (48 percent). Permanent Staff tripled from 3 to 9 participants after the Internship. In fact, while the number of youths with permanent jobs showed a large increase (in percent), the largest gain was in Temporary Contractual work. Internship (26 percent to 8 percent) and Seasonal Work (14 percent to 4 percent) was also reduced dramatically, respectively. In general, there is a shift to better employment from baseline to follow-up survey. Overall 78% of participants were em￾ployed when the program ended. B. Sustainability and Scalability 5. What are the factors facilitating or inhibiting the sustainability of program activities beyond the UIP? In answering this question, the evaluators looked at three issues: internship placement policy, instructional capacity of the host employers to host interns, and the resources to fund the intern￾ship placement program. a. Internship Placement Policy The possibility of sustaining an internship program is not clear based on the information available to the evaluation team. There is currently no policy framework on youth internship and placement in Uganda. The implication of the absence of such a policy is that the government has no basis for appropriating the requisite finances for supporting the internship programs. As a start, the RD developed a “network mechanism to convene employers to share lessons learned with RD, such as generate discussion on how to influence policy and contribute to the building youth workforce with employers.”15 RD has also been involved in the National Volunteer Youth Forum, where they were part of discussions about the process for formulating policies supporting youth volun￾teership and internship.16 In this effort, the KIIs revealed that Board members and members of the Parliamentary Forum for Youth Affairs have asked Board members who are Members of Parlia￾ment (MP) to discuss with Parliament an internship/volunteer policy. In this regard, ultimate sus￾tainability of the internship program will depend on the development of a sound public policy framework and a related funding program for internship and job placement activities, which is 15 Mike Midling, transcripts of FGD RD staff, 1/7/15. 16 Ibid. YouthMap Uganda Evaluation Report – June 5, 2015 38 designed in a cost-effective and incentive-based manner. It should facilitate the provision of sti￾pends that temporarily subsidize training costs for participating firms. It would also benefit from targeted efforts to develop a network of Career Development Centers at the university level which could better advise/guide prospective college graduates regarding both internship and job place￾ment opportunities with the types of companies who are likely to participate in internship and pre-placement programs. b. Institutional Capacity of Host Employers Some of the FGD participants noted that they felt that their placements were carried out in a somewhat haphazard manner and that it is important for host employers to have a more struc￾tured and well-ordered set of human resources policies for interns. Some employers have human resource structures that provide for internship placements. For example, Sanlam Life Insurance Limited has a human resources structure which recognizes the role of interns and developed a corresponding HR approach to internship training. Transcultural Psychosocial Organization (TPO) has an internal internship policy in place and has promised to continue supporting internship placement efforts. At the time of interview with the TPO, on December 9, 2014, the evaluators learned that TPO was in touch with RD and had already communicated their need for additional interns after cohort two. For example, TPO had a need for an agribusiness graduate for placement before the end of 2014. The representative also highlighted that in 2015, TPO will need at least two graduates of clinical psychology for placement in their projects currently handling refugees in the West Nile region of Uganda. However, this does not discount the fact that some employers do not have a strong intern HR policy. Internship programs would be more sustainable if em￾ployers have the structures and policies in place to host and maintain interns. RD can work with employers to build their institutional capacity, ensuring that appropriate internship structures and policies are in place before hosting the interns. c. Resources to fund the internship placement program No sustainable external resources to fund an internship program currently exist. The cost-share (stipend) imposed on host employers may be probative to pay (ie. government offices). However, suggestions from RD staff, host employers and Board members include the following: enhanced community fund-raising efforts to support a few youths who show strong potential; grant pro￾posal development targeted towards major donors and corporate sponsors; and building part￾nerships with key private sector associations, such as with Uganda Enterprise and Federation of Uganda Employers which has the potential to host interns within their large employer member￾ship and a large pool of experienced members who can facilitate training. Actually, near the end of the program, RD secured a grant of 50,000 USD from Segal Family Foundation to offer thirty additional internships over the coming year and strengthen their partnerships with universities. YouthMap Uganda Evaluation Report – June 5, 2015 39 With regard to cost sustainability, it should be noted that the unit cost of the program was almost $25,000 per participant (including all costs). This is extremely high from a job-creation perspec￾tive; taking into account that costs/job under donor workforce development programs are gen￾erally expected to range from about $1,000 to $10,000, depending on the depth of the interven￾tion strategy and the initial skill gap between program participant skill profiles and potential em￾ployment opportunities. Two additional activities, namely CLA and BYB, which are reflected in those cost figures, in addition to the internship program itself. However, it is difficult to tease out the costs per components as that is not how the budget was developed. The budget was organized in terms of personnel, benefits, travel & transportation, equipment, supplies, grants, consultants & contracts and other direct costs. Only CLA was listed under, travel & transportation. While the CLA activities add to 21 youth participant research employability, it may be something that is done through the partic￾ipating IPs after the internship program. One IYF staff person noted that the CLA was time con￾suming and complex. “In order to do what USAID wanted, the program would have to be a sep￾arate program with its own funding because it took a lot of time for interns to complete and was a lot of complex work.” In terms of the BYB costs, a consultant was hired to advise the participating interns. While business skills (through BYB) are useful, they do not add to youth participant em￾ployability. While these additional costs may be minimal, they may overestimate the cost of the internship. Depending on the goal of the program, there is potential for adjustment to reduce the unit cost if activities are more focused and linked to other existing projects and/or institutions. d. Potential for Replicability It does not seem at first glance that it would be cost effective to replicate this program, due to the relatively high cost per job created, as well as the lack of readily available sources of public or donor financing available to scale up the program at present. As mentioned, adjustments in pro￾gram activities would reduce the unit cost, though by how much is unknown at this point. It should be added in this regard that the referenced cost figures do not, at this point, include IYF and RD overhead costs, but only the direct costs of program implementation. Financial considerations aside, the program is potentially replicable in the private sector. Imple￾menting organizations in the private sector can improve and streamline UIP to better address the needs of both youth and employers, while charging a reasonable service fee. The services can be designed as packages that have varying levels of services and support. Without a youth internship and volunteer policy from the government, it would appear to be extremely difficult to have a broad-based public internship program in the near future. The policy would designate funds for such a program. However, if a local government/city can garner the YouthMap Uganda Evaluation Report – June 5, 2015 40 funds, and motivate and get support from the right stakeholders (employers, implementing or￾ganizations, and training institutions), they may be able to replicate the internship program. A complementary initiative would be to work at the national level (MGLSD and the Ministry of Edu￾cation) on a pilot internship program which could be linked to efforts to strengthen the dialogue between employers and both the government and training/academic institutions on priority skill gaps in high growth sectors. This could be focused initially on progressive stakeholder groups at the local level, as referenced above. At a later stage, it could be tied into/rolled out through a national internship policy framework and funding program.. 6. What program elements should be scaled up/replaced? What entities are best placed to support the scale of the program elements? According to Cooley and Linn, scaling up is part of a broader process of “expanding, replicating, adapting and sustaining successful policies, programs or projects in a geographic space and over time to reach a greater number of people.”17 A new idea, model or approach is typically embodied in a program project with limited impact. By learning from this experience with monitoring and evaluation, organization-internal knowledge is created and organization-external knowledge is disseminated. Internal and external knowledge, in turn, can be used to scale-up the model through expansion, replication and adaptation with multiple impacts. The experience from scal￾ing-up feeds back into new ideas and learning. Outside knowledge can also feed scaling-up ef￾forts, if an organization picks up on the program experience and learning of another organiza￾tion.18 Given that this internship program was a program of 100 youth over a diverse set of sectors, there may not be enough data to assess if the program can be scaled-up in every sector. It is also not clear that the program was designed as part of a scale-up plan.19 Scaling-up is a process that includes planning and strategic management of the process. The team recommends that several factors be put in place to establish the preconditions for scaling-up20: legitimize the change (hav￾ing a common understanding among stakeholders or a national policy); build a constituency (to support needed changes); realign and mobilize resources; modify organizational structures; coor￾dinate action; track performance and maintain momentum. These preconditions ensure the de￾cisions and resources needed for scaling-up are approved and in place.21 These steps are part of a larger scale-up framework that can be further investigated if it is appropriate to scale-up. Similar 17 Larry Cooley and Johannes F. Linn. “Taking Innovations to Scale: Methods, Applications and Lessons.” Results for Development Institute. September 2014, 2. 18 Ibid. 19 See also Cooley and Linn’s Annex 2 on page 17 for a scalability assessment tool. 20 Ibid 9. 21 Ibid. YouthMap Uganda Evaluation Report – June 5, 2015 41 to the sustainability factors discussion above, it is important to secure a national policy, intern￾focused institutional capacity and guaranteed funding resources before scaling-up can happen. The FGDs and KIIs cited examples of a few entities best placed to support the scale of the program elements. RD staff indicated that they feel RD would be best placed and experienced to support in the scale-up of an internship program. RD can assist in building up the host employer organi￾zational capacity to have an intern. Others noted that tertiary and/or vocational training institu￾tions, such as Makerere University Business School and Business Technical Vocational Education and Training Institute are potential institutions to build graduate foundational training needs and make use of their facilities for training. One KII noted that the Ministry of Gender, Labour and Social Development (MGLSD) is the best place to help the scale-up at the policy implementation level. RD staff interview mentioned that the ILO has a research component that can be tapped to continue CLA type of research among youth. There was a strong emphasis placed by some re￾spondents on the focused application of networking resources to build a system that would facil￾itate effective scale-up, i.e. building employer networks and private sector association-based ini￾tiatives. In addition, one KII indicated that the program should target micro-enterprises for intern placement. Since micro-enterprises make up the majority of the economy, there would be greater opportunity for intern placement. Such a move would allow more interns to be placed and re￾tained and have a far greater impact on the economy as a whole. It would also allow interns to develop entrepreneurial skills. He went on to note that the program should focus on intern place￾ments in the agriculture and agribusiness sectors, as they are the largest part of the economy. CONCLUSION In this evaluation the team looked at effectiveness, impact, sustainability and scalability of the program through the six evaluation questions. 1. How effective were the approaches taken for each of the four program components (ori￾entation and skills training; internship placements; professional development support services; and facilitation of youth participation in USAID CLA-related activities) in achiev￾ing the programs goal? Overall, youth participants found the orientation and skills training; internship placements; pro￾fessional development support services; and facilitation of youth participation in USAID CLA-re￾lated activities components relevant and effective in achieving program development hypothesis. From the FGDs, the interns lauded the foundational skills training. In their opinion, the skills train￾ing prepared them for their internships as it covered areas such as professional and ethical be￾havior, networking and ICT. The workplace technical skills training enabled them to fit well in the YouthMap Uganda Evaluation Report – June 5, 2015 42 placement. Training on life skills was also considered helpful by FGD youth participants. They found the training helpful to deal with different types of clients, manage stress during the intern￾ship, and negotiate salary. Even though in some cases internship requirements did not match up with the applicants’ background and skill, overall both employers and youth participants were satisfied with internship placement. There were 37 host locations for 100 interns, which meant more than one intern, sometimes eight, were hosted by a given host employer. This appears to be due to the limited number of host employers participating in the program. Interns were matched to employers’ intern requirements as a first priority then matched with intern requests. This approach did not always result in a good match from the intern perspective. In one FGD, 2 of 5 participants said they were not well matched in their placement because they were placed in the same employer and position they held before the internship. However, in another FGD, 4 of 6 youth participants said that while the internship did not match their educational background, the placement was helpful in directing their career into a different sector. In the Employer survey, all ten cohort one host employers interviewed ranked the quality of interns highly. The mentoring of interns was a primary responsibility of RD. However, some employers also took it upon themselves to provide on-the-job mentoring. The evaluation team interaction with RD management and program staff revealed that RD provided necessary guidance and mentorship support to interns in both cohorts on a regular basis for the duration of the internship placement and beyond. The interns that participated in the FGDs highlighted that the professional support services like mentoring by RD were helpful to gain knowledge and skills specific to the workplace. In general, the job placement support activities were reasonably successful, with about 78 percent (39/50) of the youth participants employed after the program. This compares quite favorably with an official rate of unemployment for labor force participants with tertiary education in Uganda of approximately 35 percent. During the training, interns received career planning and job search skills training with a focus on self-awareness, job search tools and methods, CV development and application writing and interview techniques. Interviews with host employers, RD staff and youth participants confirmed that the interns were provided with on-the job professional guidance and support. Regarding CLA activities, training was limited to 21 youth participants (12 in cohort one and 9 in cohort two). While the research skills were valuable and well received by the youth participants, it was limited to those 21 interns only. Participation in the CLA developed research skills among the participating youth and was well-received by communities because it was youth-led. However, cohort two did not finish the CLA report and failed to reap the full benefits of the activity. Regarding BYB activities, the evaluation team felt it was an add-on activity. The development hypothesis referred to gainful employment, with no mention of self-employment. These activities YouthMap Uganda Evaluation Report – June 5, 2015 43 were implemented too late in the program for more youth participants to fully benefit from the consultancy. There were originally nine groups but due to inactivity over six months, three dropped out. Also the group size was reduced due to inactivity and the unrealized promise of potential funding. Finally, the BYB consultant worked only with five groups that had outstanding business proposals. However, as a result of the saved stipend money, the BYB training and their entrepreneurship, 18 youth participants reported they are business owners or consultant/contrac￾tors in the follow-up survey. 2. What was the program’s overall impact on youth beneficiaries? The overall impact on youth participants was positive, and as anticipated, the internship compo￾nents increased their chances of gainful employment status and employability. In fact, a number of the FGD participants indicated that six month internship is too short, they wanted a longer internship period such as one year. The most useful aspect of the program was networking, which was closely followed by workplace foundational training, life skills foundational training and on￾the-job training. In other words, networking and training (both technical and practical) had the most effect on the youth participants. 70 percent of youth participants in the follow-up survey reported they were highly satisfied or somewhat satisfied with their current job or income gener￾ating activity. 46 of the follow-up survey respondents earned an average of $126 in additional monthly income after completing the program. This increased income plus the savings they earned from the internship benefits their large households. Notably, there are 18 self-employed youth participants, which is a strong multiplier effect for the households’ future income potential. There were also negative effects and unexpected issues that arose during the program. Some interns were not matched well because the work was not as challenging or they were placed in the same employer and position as before. Some supervisors traveled frequently or were too busy to provide mentoring and job placement support. Some interns did not receive their stipend or received it later than expected. Sometimes the supervisor was not the contact person with RD, which caused some communication issues and meant limited mentoring for the youth participant. Interns in rural areas were at a disadvantage from better employment opportunities as well as alumni networking events. The Advisory Board was helpful in positively affecting the youth participants. The Board helped guide the program’s design, direction, and sustainability. It provided strategic guidance and op￾erational advice on how to implement the YouthMap internship program and how to set direction for the program. They also identified employers for intern placement. In addition, the Board provided ideas for program improvement, such as having youth be included as board members and focusing on micro-enterprises as most of the employment in Uganda is in the informal sector. However, it was also found that although many board members were active and provided the YouthMap Uganda Evaluation Report – June 5, 2015 44 assistance described above, a number of the board members were not active and contributed little. 3. How was the capacity of various program partners strengthened through the UIP and what were the outcomes of these partnerships? Since strengthening the capacity of program partners was not part of the program development hypothesis or objectives, it was not a particular focus of the program. However, one may infer host employers were strengthened by intern placement program as it filled important human resource gaps and gave the employers experience and improved institutional and supervisory capacity to host interns. In fact, RD recently published an Employer Handbook to support em￾ployers to establish their own internship programs (funded by Forward Foundation). This hand￾book will increase employers’ capacity to host interns and at the same time help the employers increase the quality of the human resources they do hire. 4. What are the trends in intern employment data with respect to gender, sectors of inter￾est, employer retention, and employment quality? In terms UIP employment data, the program aimed to have equal male and female participation. They generally achieved this with 51 women and 49 men in this program. In terms of sector employment, nearly one-third of the interns were placed in NGOs. This may be because many of NGOs are also USAID-funded and participated in the CLA activities. The wide-ranging sectors of intern placement caused a certain level of heterogeneity among the data that affected the statis￾tical significance of the employment data. In addition, the rural and urban divide also caused some level of heterogeneity as rural positions are paid less than urban positions. As expected, there were more interns (64) placed in urban areas than in rural areas. Intern employment quality improved from the baseline survey to the follow-up survey through having fewer youth partici￾pants reporting internships and seasonal/occasional work and more reporting permanent work although there was also an increase in temporary contractual work. 5. What are the factors facilitating or inhibiting the sustainability of program activities be￾yond the UIP? There are three main sustainability aspects the evaluation team consider important: internship placement policy, institutional capacity of the host employers and resources to fund the program. There is currently no national internship policy that would allocate government funds towards developing youth internship programs. However, several board members who are also members of parliament are working towards this end by introducing discussions on internship/volunteer policy within the Parliament. The capacity of host employers is also important to the sustainability of the program. RD and host employers need to work together to develop a human resources YouthMap Uganda Evaluation Report – June 5, 2015 45 structure and policy before hosting interns. Lastly, reliable funding from a reliable and consistent source is needed to sustain the program. The program as it is now is not sustainable at $25,000 per youth participant. This is a highly expensive pilot program. Given many job placement costs range in the $1,000-$10,000 range for donor-supported programs internationally, this does not appear to be a cost-effective program as it is currently implemented. In addition, the stipend was considered too high, 25% (i.e. 85-100 USD in cash or in kind), for some host employers to pay. However, those that did demonstrated their commitment to the program. Near the end of the program, RD was able to secure a grant of 50,000 USD from Segal Family Foundation which will allow them to offer thirty additional internships over the coming year and strengthen their part￾nerships with universities. Developing more public private partnership may be a viable approach to sustaining the core aspect of the program. 6. What program elements should be scaled up/replaced? What entities/organizations are best placed to support the scale up of the program elements? Currently there is no scale up plan. However, there was a discussion which components could be scaled up. RD is willing to continue developing the program and to both increase the number of host employers as well as their capacity to host interns. Other KIIs mentioned partnering with tertiary and/or vocational schools/institutions as the best place to build graduate foundational skills. Another possibility is to consider involving the MGLSD to help with the scale up at the policy implementation level. As mentioned above, it is important to develop a national intern￾ship/volunteer policy, have a network base of intern-focused host employers and develop guar￾anteed funding resources before scaling up can be successful. KEY LESSONS LEARNED Internship  The highly participatory approach used in program design kept all the key actors engaged during the implementation. Having a participatory approach allows all key actors, em￾ployers, Board members, and RD staff, to contribute to the program outcomes.  Regular site visits by RD staff to interns in their place of work for support and monitoring are important to help ensure the successful implementation of their work plans. In partic￾ular, the following tasks were mentioned: o More regular dialogue with the mentors of interns; YouthMap Uganda Evaluation Report – June 5, 2015 46 o Better provision of rights and responsibility guidelines to the supervisors, which include coaching and mentoring responsibilities on top of their normal busy work schedule; o Share visit reports with the intern and host employer for follow-up on action steps and recommendations.  Supporting interns to set and achieve clear goals are critical to success. Interns who were able to set clear goals during the foundation training seemed to do well in their place￾ments, as they focused on job activities aligned with their goals.  Employer HR needs (in terms of skills requirement) identified at the outset before intern￾ship placement were critical for purposes of effectively matching the interns’ skills to the HR needs (in terms of skills set). RD should use the employer skills requirements as a basis for identifying and recommending the prospective interns for placement with respective employers.  Targeted orientation and training by sector - in addition to the current orientation and foundational training - can provide interns with more relevant skills in their placements. Use of specialist trainers rather than program staff can help provide effectively targeted orientation and trainings. These can be done by employer representatives. Collaborating, Learning and Adapting  Intensive participation in the CLA developed research skills among the participating youth. It was well received by the youth because it was youth-led. They implemented the research and designed the questionnaire and owned the entire process. Build Your Business  BYB is an income-generation alternative to employment, but should be better managed and structured. There was a lack of leadership and guidance until near the end of the program. The participating youths were promised seed funding but it did not materialize because RD could not determine how best to allocate/manage the funds, resulting in dis￾appointment among program participants. Some of these dropped out of the BYB groups. Some continued the business with support from a BYB consultant. The entrepreneurial skills taught interns how to develop and manage a business, and opened a window of opportunity for the interns into business as an economic alternative.  Build trust among youth. Spending quality time mentoring and introducing them to intri￾cacies of the workforce development/job placement process can build trust among in￾terns. And it is critical that financial support that is committed to is actually provided. In this regard, some youth were upset with their employers when they didn’t receive their YouthMap Uganda Evaluation Report – June 5, 2015 47 stipends which would go toward their savings, and many were disappointed because there was no seed funding to fund their BYB group for those youths who felt self-employment was a better way to secure their economic futures than through wage employment. OVERALL RECOMMENDATIONS FOR FUTURE PROGRAMMING Effectiveness & Impact  Work closely with human resource departments of the partner organizations so as to en￾sure that the graduates who qualify for placement are matched with the skills needed by the potential employers. This will enhance the workplace environment and increase chances of interns transitioning to full-time employment after their internship placement period.  Allow the intern to determine his/her placement while taking into consideration the matching educational background (skills) with the needs of the employers. This increases the possibility that after the placement period, the interns will transition into fulltime em￾ployment with the host employer. The interns will also be placed in an area which is in line with their training (educational background), thus enabling them to acquire relevant skills and experience.  Invest more time in preparing host employers to receive interns. o This might involve developing a clear understanding of the “internship” concept, engaging the employer to interview the selected interns to be placed in/her or￾ganization, involving the HR department to incorporate human resource-related internship management guidelines, (e.g. leave, stipend, job description, and sexual harassment). o Employability prospects should be explored beforehand by discussing the man￾power plan of the host employer over the coming six months. For example, those with government placements should be aware of the ban on public service recruit￾ment.  Develop a monitoring and evaluation plan at the beginning of the program. There was a lack of an M&E framework and performance indicators tracking table, and survey instru￾ments were inconsistent from baseline to follow-up. A relational database is better to track intern placement, skills achieved and contact info than an Excel spreadsheet. And there need to be more intensive efforts to ensure a higher (i.e. close to 100 percent) re￾sponse rate to the follow-up surveys. To better collect quality and consistent data there needs to be a clear and complete monitoring and evaluation plan that includes an evalu￾ation plan articulating the evaluation methodology, sampling frame and size, and data analysis. YouthMap Uganda Evaluation Report – June 5, 2015 48  Include a section on appropriate workplace behavior in the life skills foundational training. One female youth participant mentioned she was sexually harassed at the workplace. In fact, six of the non-participants in FGDs also agree that sexual harassment is an obstacle in obtaining a job. The team advises that future programing include guidelines to address sexual harassment in the workplace both for interns and host employers.  Improve general communication between the key stakeholders. Activities would be most effectively implemented with feedback from required entities. Communication gaps be￾tween USAID, IYF and RD could affect intern activity as they await consensus-based deci￾sions. Delays in getting feedback affect the timeframe and effectiveness of the program.  Make the Advisory Board a functional Board with a clearly defined scope of work, systems and procedures and expected deliverables. With an explicit scope of work and deliverables over a defined timeframe, an active Board can make significant strides in putting together a scale-up plan, among other needs.  Separate the other program components from the internship component. Have full free￾standing components of the BYB (entrepreneurship) and CLA (research) that are strategi￾cally linked to the internship without interfering with the internship timeframe and expe￾rience.  Consider targeting the internship placement in specific sectors and regions to increase the employability within that sector/region and maximize youth income potential. Different sectors have distinct productivity enhancement and growth potential. Different regions have more opportunities than others due to rural-urban differences. Better targeting in￾ternship activities in accordance with sectoral/regional growth trends/potential can help optimize the sustainable job placement and wage growth results of the program, and also contribute to enhanced economic competitiveness. In addition, before and after wage data for participants can be more effectively compared when there is less dispersion in the range of sectors incorporated into the program. Sustainability & Scalability  Continue to contribute to policy development relating to the national volunteer programs and Youth Venture Capital Fund. While UIP has encouraged government engagement in the program, continued engagement with the government is essential to realizing a na￾tional policy.  Alternative and diverse sources of funding, besides donor funding, need to be sourced to sustain the internship program. Looking to local sources of funding, private and public, will increase the chances of securing funds for the program.  Build employer institutional capacity to host interns through partnership development, which will strengthen the relationship between implementing organization and employer. YouthMap Uganda Evaluation Report – June 5, 2015 49 The more employers with the appropriate capacity (structures, HR guidelines and intern policy) to host interns are utilized, the higher the final placement rates will be.  Develop a scale-up plan for future programing. Assess whether the pilot is scalable using the Scalability Assessment Tool. YouthMap Uganda Evaluation Report – June 5, 2015 50 ANNEX I: EVALUATION WORK PLAN Evaluation stages Main activities Timeframe Selection of document for review and prepa￾ration of inception re￾port.  Identification and review of the final set of secondary sources;  Initial consultations with USAID/Uganda  Initial consultation with Restless Develop￾ment and contacting of evaluation sources;  Preliminary visits to one of the employers.  Preliminary Focus group discussion with UIP beneficiaries.  Preparation of inception report. 7 days in country team activities. (8th to 15th December, 2014) In-country data collec￾tion and preliminary data analysis activities  Finalization of the inception report, includ￾ing evaluation questions, design and meth￾odology, data collection and testing and fi￾nalization of data collection tools;  Planning meeting with USAID/Uganda. Weekly abbreviated reporting of progress to USAID.  FGD/KII data collection in Kampala and Jinja. Abbreviated survey of comparison group non-participants.  Data integration and preliminary analysis.  Drafted of preliminary materials for inclu￾sion in a final report on two site visit coun￾tries (Uganda/Senegal).  Presentation of preliminary findings from Uganda research to USAID/ Uganda. 10 days in country team activities. (5th to 15th January, 2015) Evaluation report writ￾ing at Pragma HQ of￾fices  Located, organized, collated and synthe￾sized collected data  Cleaned baseline and follow-up quantitative data  Reviewed existing program materials  Analyzed qualitative data  Analyzed quantitative data into tables and graphs  Drafted report and annexes for final sub￾mission January 31, 2014- February 23, 2015 Additional KIIs via phone at Pragma HQ  Contacted other board members, youth participants and employers for additional KIIs February 9-20, 2015 YouthMap Uganda Evaluation Report – June 5, 2015 51 offices  Revised KII question guides to reflect cur￾rent needs  Conducted round 2 KIIs YouthMap Uganda Evaluation Report – June 5, 2015 52 ANNEX II: MATERIALS RECEIVED AND REVIEWED SN Documents 1 Program document 2 Work plan for years 1&2 3 Budget/expenditures 4 Annual performance report 5 Success stories 6 Advisory board materials 7 List of employers (partners) 8 List of board members including any changes since implementation 9 List of youth who benefited from the program including internship placements 10 List of youth interviewed but could not be selected to benefit from the program. 11 Data tracking tables and database 12 Baseline survey (cohorts 1, 2) 13 Follow-up survey and data (cohorts 1, 2) 14 Formative evaluation report 15 Formative survey and data 16 Employer survey and data 17 Curricular materials (various manuals, handbooks) 18 Quarterly reports 19 CLA cohort one report and cohort two questionnaire and research plan YouthMap Uganda Evaluation Report – June 5, 2015 53 20 Application process materials 21 Assessment documents 22 Handbook for employers YouthMap Uganda Evaluation Report – June 5, 2015 54 ANNEX III: LIST OF KII AND FGD COMPLETED Focus Group Discussions Date # participants FGD with YouthMap beneficiaries, treatment, Co￾hort 2, RD offices in Bukoto December 10 2 FGD with RD staff January 7 6 FGD morning session, control, RD office January 8 6 FGD afternoon session, control, RD office January 8 6 FGD evening session, control, RD office January 8 2 FGD evening session, treatment, RD office January 9 5 FGD evening session, treatment, Kabira Country Club January 9 6 Total 33 Key Informant Interviews Date # participants KII with Transcultural Psychosocial Org (em￾ployer) December 9 1 KII with RD staff, RD office January 7 3 KII with Farm Input Care Center (FICA) (employer) January 8 1 KII with Build your Business (BYB) consultant January 9 1 KII with Sanlam Life Insurance Unlimited (em￾ployer) January 12 1 KII youth participant, Cohort 2 (Jinja) January 12 1 KII with Board Member, Hotel Africana January 13 1 KII with Board Member, Uganda Parliamentary Forum on Youth Affairs January 14 1 KII with John Snow International (JSI) (employer) January 13 1 KII with IYF January 15 1 KII with Board Member, DCFU Bank February 9 1 KII with youth participant, Cohort 1 February 10 - 13 5 KII with youth participant, Cohort 2 February 10 1 KII with DFCU Bank (employer) February 19 1 KII with IYF staff February 20 1 Total 21 YouthMap Uganda Evaluation Report – June 5, 2015 55 Annex IV: KII AND FGD QUESTIONS GUIDES Round 1 of KIIs and FGDs (December 9, 2014 – January 15, 2015) Key Informant Interview Guide - Employer 1) Please describe your experience with YouthMap. (how many interns, cohort, where, when cycle ended, type of jobs interns placed in, whether employed intern). 2) How did the placement of the applicant match the skills required by your organization? What could have been done better and how? 3) What kind of employee support did you provide the intern(s)? What could have been done better and how? 4) What was your cost stipend? 5) How specifically did your company/organization benefit from this internship? 6) How did the intern(s) benefit? 7) Did you have interns before? If yes, how did the experience compare with YouthMap? 8) What lessons have you learned? 9) Now that the program has ended, how would you sustain an internship? 10) Are there other programs like this operating here? Is it a good idea? 11) Assuming another program followed, what would you do differently? 12) Is there an employer association group to take advantage of this model and cost share? Key Informant Interview Guide - Advisory Board Member 1) What was your role in the implementation of the YouthMap internship program? (How was the board structured, what was the frequency of your involvement or role how was the board operating) YouthMap Uganda Evaluation Report – June 5, 2015 56 2) What lessons have you learned from your role in the YouthMap internship that can inform future internship program? Explain. 3) In your opinion and experience what mechanisms are in place to ensure the sustainability of the program activities? 4) In your opinion, which entities/organisations are best places to support the scale up of the program elements? Key Informant Interview and Focus Group Discussion Guide – RD Staff 1) How was the recruitment, orientation, skills training and interns placement conducted? (What could have done better and how?) 2) How did the placement of interns match the skills required by the employer? Explain. 3) What professional development services did the program offer to the interns? 4) How was this assistance provided? (one on one counseling, assistance with job search, role playing for employment interviewing, other?) 5) What could have been done better and how? 6) How often did RD staff meet with the interns to provide internship and career support? 7) What (if any) linkages were provided to participants with recruiting firms? How often was this done? 8) Did RD develop and operate an internet site on participant engagement in the program, in the CLA? 9) In your assessment, what elements of the YouthMap Progam had the greatest impact in terms of improving youth participant employability, in terms of effective youth integration into the job market? Explain. 10) What lesson have you learned from the program that could inform future youth internship programming? YouthMap Uganda Evaluation Report – June 5, 2015 57 11) In what ways did the public-private partnership contribute most to improving youth skills for employment? In mentoring or preparing youth for entry into job market? Did this vary by sector of the job market? 12) In what ways did the PPP facilitate in intern screening and placement, in on the job training, in career mentoring and support? 13) What mechanisms are in place to ensure the sustainability of the program activities? (Explain in terms of policies, institutional capacity and resources) 14) Looking at cost effectiveness (affordability), efficiency and relevance, what program elements can be scaled up/replicated? Explain. 15) What other entities/organizations are best placed to support the scale up of the program el￾ements? Explain. YouthMap Uganda Evaluation Report – June 5, 2015 58 Key Informant Interview Guide – IYF Staff 1) Has IYF designed and implemented with a local partner internship based models like the UIP, which have focused on work force development PLUS components like entrepreneur￾ship training, BYB, group savings mobilization? 2) If so, has IYF working with the local partner conducted impact analysis on those add on components as well? 3) Does IYF envision implementing and scaling up the UIP during a second phase (based upon findings and lessons learned from the present evaluation and a potential impact study? 4) If so, what type of implementing partner organization is likely to be identified? 5) What is the current thinking on potential target rates for scaling up? 6) If so, what is IYF’s initial thinking about strategies for scaling up? Vehicles for doing it within the commercial business sector, the NGO sector, other? 7) What are the key lessons learned by IYF as US implementing organization for YouthMap? (orientation, technical training, intern placement, job and career support, BYB, other) 8) In what ways did IYF benefit from implementing this program with RD? (above components) 9) What could have been done better (above components)? YouthMap Uganda Evaluation Report – June 5, 2015 59 Round 2 of KIIs and FGDs (February 9-20, 2015) Key Informant Interview Guide - Employer 1) Where is your company located? a) What type of business is it (i.e. NGO, for-profit, etc.)? b) What sector/industry does your company work in? 2) How many people are employed in your company? 3) How did your organization get involved with the program? a) Did your organization host interns prior to joining UIP? b) Does your organization have in an intern policy or similar structure? 4) How many interns from the YouthMap Uganda internship program did your organization host? a) Were you satisfied or dissatisfied with the interns from UIP that worked in your organiza￾tion? i) How satisfied or dissatisfied with the UIP matching process? b) Did you attend any employer orientation with Restless Development? i) How effective was the employer orientation? c) How many interns were retained for full-time work after the program? i) If not, were there particular reasons why they weren’t retained (i.e. budget, unhappy with intern quality, etc.)? 5) Did your organization offer mentorship and/or professional development training? 6) Did your organization offer any additional benefits (i.e. healthcare, transportation costs, etc.) to the UIP interns? 7) What were the strengths that you witnessed of the program? 8) What were the weaknesses? 9) Would your organization be interested in continued participation and hosting of interns? 10) Why or why not? 11) What was your cost share (stipend) to the program? YouthMap Uganda Evaluation Report – June 5, 2015 60 12) Are there any aspects of the program that you would change? Key Informant Interview Guide - Advisory Board Member 1) What was your role in the implementation of the YouthMap internship program? (How was the board structured, what was the frequency of your involvement, how does the board op￾erate?) 2) What do you think were the strengths and weaknesses of the YouthMap program? 3) What do you think are the key factors that enabled different interns to secure and/or remain employed following the program? 4) Do you think the YouthMap program achieved its goals? a) Yes or no b) What do you think was the board’s contribution to the achievement of the program goal? Probe for whether board provided guidance, strategic direction, linkages and net￾works to enable the achievement of the program?) 5) How did RD use the formative evaluation results? Were the suggested recommendations made in the report adapted/applied to the program (cohort 2)? 6) In your opinion and experience what is the potential for replicability of the program or simi￾lar services provided under the internship program within the public and private sector? 7) In your opinion, which entities/organizations are best places to support the scale up of the program elements? YouthMap Uganda Evaluation Report – June 5, 2015 61 Key Informant Interview Guide – IYF Staff 1) What was your role in the implementation of the program? 2) Who did you have the most contact with among RD staff, advisory board and USAID staff? a) Were roles clearly defined? b) What benefits if any do you think the RD staff and advisory board got from the relation￾ship? c) In terms of these relationships, what improvements would you make between you/IYF and the other stakeholders? (probe: communication, planning, and support between the stakeholders) 3) What kind of relationship did you have with the host employers, if any? a) Were employers properly assessed in terms of capacity? b) Were expectations adequately communicated and realized? 4) Do you feel that YouthMap UIP achieved its goals? a) Why or why not? b) What do you think was IYF’s main contribution to the achievement of the program goal? 5) What do you think are the key factors that enabled different interns to secure and/or remain employed following the program? 6) Do you feel that UIP was effective, as they relate to the interns, in: a) Orientation and training b) Job placement c) Mentorship d) CLA/BYB activities 7) Did IYF collaborate with RD to develop and implement the monitoring and evaluation plan? a) In your opinion, is there anything that could have been done to improve the process? 8) What were the major lessons learned by IYF? a) Specific strengths b) Specific weaknesses 9) Did you participate in the formative evaluation? a) What did you think of the report? b) Was it useful? YouthMap Uganda Evaluation Report – June 5, 2015 62 c) Did IYF/RD adopt the recommendations? Which ones? 10) How would you improve the program if you had to do it all over? 11) In your opinion, what are the possibilities for replicability? 12) Does IYF envision scaling up UIP? a) If so, is there a potential scaling up model? Key Informant Interview Guide – Youth Participant 1) How old are you? 2) What is your current living situation? a) Do you live by yourself b) Do you live with a spouse c) Do you live with parents/relatives 3) What is your highest level of education? 4) Were you employed when you entered the program? a) If yes, was it temporary or permanent? b) Did you receive a salary? If yes, how much? 5) Please describe the type of work and company you were placed in for your internship. 6) Are you currently working? a) If yes, does your current job pay you a salary? How much is it? b) What kind of job is it? c) Is it permanent or temporary? 7) What were the key factors that enabled you to obtain this job? a) If no, how long have you been without work? 8) What type of job are you looking for? 9) Have you thought about starting a business? a) If so, what kind of business is it? b) What are the challenges you face? 10) In your opinion, how effective was the program components in terms of getting you: a) Oriented and trained? b) Matching and intern placement support YouthMap Uganda Evaluation Report – June 5, 2015 63 c) Professional development, such as mentoring, job placement services and participant in alumni network events d) BYB Entrepreneurship training and CLA activities. Ask if participated in the activity. 11) What do you think were the strengths and weaknesses of the YouthMap program in terms of: a) Orientation and training b) Internship placement c) Professional development, such as mentoring, job placement services and participant in alumni network events d) BYB Entrepreneurship training and CLA activities. Ask if participated in the activity. 12) From your perspective, what can be done to improve the program? YouthMap Uganda Evaluation Report – June 5, 2015 64 Round 1 of FGD Youth Participants (December 10, 2014-January 9, 2015) Date: Focus Group Conducted: Location: Time: Name of Focus Group Discussion Facilitator(s): Number of Participants: Breakdown by Gender: males females INTRODUCTION Hello my name is _______________ and this is (second interviewer name)_________________. We are conducting an independent evaluation of the Uganda Internship Program for the donor agency, USAID, and are external to the program implementer, Restless Development. We had your names because you were accepted to this program and carried out an internship. We would like to ask you some questions about your current living and employment situation, as well as some questions about your experience in being an intern, how this met your needs, what could be done better and the outcomes achieved from doing the internship. All of your responses are confidential, and your names will not be used in our reports. Focus group discussion (1 hour 30 min) Items in italics filled in by focus group facilitator Background (10 minutes) First of all, we would like to ask you some background questions. 1. How old are you right now? 18 to 24 ________ (number responding) 25 to 29 ________ (number responding YouthMap Uganda Evaluation Report – June 5, 2015 65 30 or older ______(number responding) 2. What did you study (tertiary education level)? 3. What kind of degree did you get? Employment (20 minutes) 4 . Are you currently working? Yes ____________(number of respondents) No ________ If there are participants who are not, skip to question 23 for non-employed. 5. What kind of work do you do? (Record responses by sector/type of work in which participants are working) For example: Commerce business Services industry Government NGO Sector program/organization Agriculture Other (specify) 6. Did the internship you completed help you to get this job? Yes ________ (number responding) No _________ (number responding) 7. If response is YES, did you get hired by the company/organization where you did the in￾ternship? Yes ______________ No _______________ (number responding) 8. Do you consider your employment situation more or less temporary or permanent? Temporary ________ (number of respondents) Permanent ________ (number of respondents) YouthMap Uganda Evaluation Report – June 5, 2015 66 9. How many hours per week do you typically work? Full-time _______ Part-time_______ Program “Process” Questions (40 min.) 10. How was the orientation (life skills) and the technical skills training conducted? (Probe for how each of these types of services was delivered) Orientation (life skills) ____________________________________________________ Skills Training (technical skills)_________________________________________________ 11. What was in your orientation that was most helpful? (Record responses given by type of skill covered by orientation). Life Skills Interpersonal skills Leadership Teamwork Communication Positive attitude Critical thinking Problem solving ___________________________________________________ YouthMap Uganda Evaluation Report – June 5, 2015 67 12. What was in your skills training that was most helpful? IT skills Financial literacy and management Entrepreneurial skills Project management ________________________________________________ 13. Did you have a mentor assigned to you? If so, was that person different than your super￾visor? 14. How frequently did your supervisor and/or mentor sit down with you to discuss your in￾ternship and how you were doing under your Work Plan (skills learning and application, quality of work, areas needing improvement)? 15. What kind of support and advice was given to you? 16. When did this occur and how? (Probe for stage in internship) 17. How was it specifically helpful to you? (Probe for specific examples of how support given helped them) 18. Was the advice and guidance you received on the job sufficient? Yes ___________________ (number responding) No ___________________ (number responding) 19. What could have been done differently, and how? (Probe for specific examples) Orientation (life skills) ______________________________________________________________ YouthMap Uganda Evaluation Report – June 5, 2015 68 Technical Skills______________________________________________________________________ Internship placement _____________________________________________________________ 20. How often did you meet with an RD staff person? 21. What professional support services did you receive from RD and how were they deliv￾ered? (Probe for specific professional support services received by a number of participants) 22. How has the Youth Map Program specifically helped you? (Probe for specific examples in terms of orientation (life skills), technical skills, placement, mentoring, employer support, financial training and savings clubs, entrepreneurship and business) For those program participants not employed (10: min.) You said earlier that you are unemployed. 23. .How many months have you been without work? Less than a month________ (number responding) 1-3 months _________ 4-6 months _________ 7-12 months ___________ More than one year __________ 24. What are you doing currently? (Probe for one or more responses regarding what uses most of these program participants’ time) YouthMap Uganda Evaluation Report – June 5, 2015 69 Studying ________ (number responding) Housework/Chores _____ Other Activity (specify ____________ Looking for Work ________ Other_______________________ 25. What did you study (tertiary education level)? 26. What kind of degree did you get? 27. Have you thought about starting a business that generates enough income to support you? 28. For those responding “yes”, what would be the most ideal business to support your live￾lihood needs? (Probe for several specific examples) 29. Have you set up the business? Yes ______ (number responding) No ______ (number responding) Aspirations (10 minutes) For all respondents 30. How would you rate the quality of your life now, and why? Good _________ (number responding) Fair __________ (number responding) Not very good _________ (number responding) YouthMap Uganda Evaluation Report – June 5, 2015 70 (Probe for reasons why response was given) 31. How do you think your future will be? Reasons why? Better _________ Worse _________ About the Same ________ (Probe for several responses and reasons why) 32. How hard do you think it will be to find a job in the future? Very hard ________________ (number of respondents) Somewhat hard ________(number of respondents) Not very hard ____________(number of respondents) 33. Do you currently have access to resources that can help you to find employment? Yes ______ (number responding) No ______ (number responding) 34. If so, what kind of resources? (Probe for examples, such as recruiting agency, public em￾ployment service, NGO, etc. and record types of resources given.) Conclusion (5 minutes) Thanks to all for your participation. Your answers have been very helpful to us in understanding questions related to youth and employment. After we have checked your survey for completeness, we will be providing you compensation for travel. Thank you! YouthMap Uganda Evaluation Report – June 5, 2015 71 Questions for Participants in CLA 35. Did you participate in a CLA research activity? Yes ________ No ______ (number respond￾ing) 36. If so, WHAT KIND of CLA activity did you participate in? (Probe for specific types of re￾search activities in which Focus Group participants were involved). 37. How has the CLA research activity impacted you? (Probe for specific examples among several Focus Group participants who took part in a CLA activity) YouthMap Uganda Evaluation Report – June 5, 2015 72 Focus Group Discussions Youth Non-Participants Date: Focus Group Conducted: (number l, 2, 3, etc.) Location: Time: Name of Focus Group Discussion Facilitator(s): Number of Participants: _______ Breakdown by Gender: ______ males _______ females INTRODUCTION Hello my name is _______________ and this is (second interviewer name)_________________. We are conducting an independent evaluation of the Uganda Internship Program for the donor agency, USAID, and are external to the program implementer, Restless Development. We had your names because you applied to this program at some time in the past, but were not enrolled in the program. We would like to ask you some questions about your current living and employ￾ment situation to understand whether there are any differences in outcomes for program partic￾ipants and non-participants. All of your responses are confidential, and your names will not be used in our reports. We will begin by requesting that you complete a survey before beginning our discussion. Survey completion (30 minutes) The survey is very important for this study, and should have priority over FGD participation. Facilitators should check each survey for completion—for persons that are late, as possible, find a separate space for them to complete the survey and check for completion, before they participate in the focus group. YouthMap Uganda Evaluation Report – June 5, 2015 73 Focus group discussion (45 minutes) Items in italics filled in by focus group facilitator Background (10 minutes) First of all, we would like to ask you some background questions. 1. How old are you right now? 18 to 24 __ (number responding) 25 to 29 __ (number responding 30 or older ___ (number responding 2. Could you briefly describe your current living situation and household?  Do you live in your own place or with others? If with others, who? Own place or with spouse/partner ______ (number responding) With parents/relatives_______ (number responding) Other, specify ______________________________________________________  Do you have children? Female, yes_________(number responding) Female, no _________(number responding) Male, yes________ (number responding) Male, no _______ (number responding) YouthMap Uganda Evaluation Report – June 5, 2015 74 Employment (20 minutes) 1 . Are you currently working? Yes _________ (number of respondents) No ________ (number responding) For those currently working:  What kind of work do you do? Responses given:  Do you consider your employment situation more or less temporary or permanent? Temporary ________ (number of respondents) Permanent ________  How many hours per week do you typically work? Full Time _________ (number of respondents) Part time __________ (number of respondents) 2. Are there any other things you do apart from your work? YouthMap Uganda Evaluation Report – June 5, 2015 75 (Probe for specific activities done from several participants) For those not working: 1. How many months have you been without any work? Less than a month____ (number responding) 1-3 months _________ (number responding) 4-6 months ___________ 7-12 months __________ More than one year ___ 2. What are you doing currently? (Probe for single response based on where participant uses most of his/her time) Studying _________ (number responding) Housework/Chores _______ Looking for Work ________ Volunteering ______________ Other ______________________ 3. Have you thought about starting a business that generates enough income to support you? Yes ________ (number responding) No __________ (number responding) YouthMap Uganda Evaluation Report – June 5, 2015 76 For those responding “Yes” to the question above 4. What would be the most ideal business to support your livelihoods? (Probe for specific examples from participants) 5. Did you set up the business? Yes ______ (number responding) No ______(number responding) 6. Have you ever applied for credit or a loan from a bank, corporation or cooperative, in or￾der to start a business or microenterprise? Yes _________ No _________  If so, did you get a loan? Yes ___________ Aspirations (10 minutes) For all respondents 1. How would you rate the quality of your life now, and why? Good _________ (number responding) Fair __________ (number responding) Poor_________ (number responding) YouthMap Uganda Evaluation Report – June 5, 2015 77 (Probe for specific reasons why participant gave response) 2. Where do you see yourself in terms of employment in the future? (Probe for specific examples of type of employment that the participant envisions achieving) 3. Do you currently have access to resources that can help you to find employment? Yes _____ (number responding) No ______ (number responding)  If so, what kind of resources? (Probe for specific responses, for example: recruiting agency, public employment service, NGO, etc.) Responses given: Conclusion (5 minutes) Thanks to all for your participation. Your answers have been very helpful to us in understanding questions related to youth and employment. After we have checked your survey for completeness, we will be providing you compensation for travel. Thank you! YouthMap Uganda Evaluation Report – June 5, 2015 78 ANNEX V: SCOPE OF WORK YouthMap: Uganda22 Final Evaluation Scope of Work These terms of reference are for an evaluation of pilot YouthMap programs in Uganda and Senegal. The programs aim to strengthen the skills of young people so that they are better equipped to enter the job market, either as entrepreneurs or employees. I. UGANDA A. Background The YouthMap Uganda Internship Program (UIP) is a pilot Innovation Fund activity under IYF’s USAID￾funded YouthMap initiative. The program is underpinned by the development hypothesis that if edu￾cated youth are provided a comprehensive internship opportunity that fosters skills development, mentorship and peer support, then their chances of gainful employment will significantly increase. Four major program components are:  Orientation and skills training;  Internship placements;  Professional development support services, including mentoring, job placement services, and participation in an alumni network;  Facilitation of youth participation in USAID CLA-related activities. The program was designed following a 2011 cross-sectoral youth assessment conducted in Uganda, which revealed the need to link graduates with the job market. The UIP is implemented by partner Restless Development, managed by IYF, and overseen by USAID/Uganda. An advisory board was created to help guide the program’s design, direction, and sustainability. The program has trained and placed a total of 100 young people divided into two 50-person cohorts. Each young person was matched with an employer, who provided cost share and mentoring. Over 30 different employers from public, private, and civil society participated (see Annex B for more in￾formation on intern and employer placements). The UIP consists of two additional components: entrepreneurship training and support and research in support of USAID’s Collaborating, Learning, and Adapting (CLA) agenda. Intended program outcomes include:  Youth with increased employability skills;  Youth with increased practical work experience; 22 References to Senegal evaluation were removed for the purpose of the final evaluation report. YouthMap Uganda Evaluation Report – June 5, 2015 79  Increased youth research skills, and contributions to the CLA process (broader devel￾opment efforts);  Improvements in young people’s employment situation (measurable six or more months following completion of internships23). Program indicators for the UIP are presented in Annex A. B. Evaluation Purpose and Use The part of the evaluation related to UIP is intended to assess its impact on beneficiaries and capture program learning and good practices for future internship programs. A summative analysis of all elements of the program (e.g., recruitment and selection, matching, training, and placement as out￾lined below) will inform the degree to which program objectives were met. It will assess the degree to which program outcomes and impacts have been achieved, and how it has impacted employabil￾ity prospects for young people. Additionally, the evaluation will examine the UIP’s supporting elements such as financial and entre￾preneurship training and the program’s contributions to the CLA agenda, to sustain and further pro￾mote youth participation in Mission and CLA-related activities. CLA is the youth-led research com￾ponent that is providing USAID/Uganda with added youth perspectives on health and agriculture￾related information. The evaluation is intended to capture overall program learning and best practices for dissemination and future scale-up of the internship program in Uganda, East Africa, and the continent. The evalu￾ation will analyze successful aspects of the UIP that can be practically applied by other relevant insti￾tutions (companies, governments, NGOs, universities), with recommendations to inform future Government and other prospective employer HR policies to address youth un/underemployment for the design of future internship programs. Specifically, the UIP-focused part of the evaluation will assess overall program performance and impact, as well as sustainability and scalability. The target audience of this evaluation includes key stakeholders ranging from USAID and the Government of Uganda to IYF, Restless Development, and other youth-serving NGOs/donors interested in imple￾menting internship programs. C. Project Implementation and M&E Approach Each of the two 50-person cohorts received 2-3 weeks training covering employability, entrepreneur￾ial life and IT skills, followed by an internship lasting about 5.5 months. Participants received in￾ternship and job placement services. An alumni database was created. In addition two Collaborat￾ing, Learning, Adapting (CLA) feedback/reflection meetings were held (one per cohort). Some youth were expected to employment offers prior to completing their internships. 23 Even though each internship was expected to last about 5.5 months, some youth may have received employment offers before the end of their internships. Therefore, follow-up surveys were to be conducted approximately six months after each cohort’s targeted completion of internships – which may be longer than six months for some youth. YouthMap Uganda Evaluation Report – June 5, 2015 80 The UIP M&E system includes the following components: A tracking system was developed and used to track progress on key indicators. Program monitoring focused on the program’s ability to deliver services as planned, tracking output and outcome and in￾dicators (detailed below). IYF developed the template in consultation with YouthMap implement￾ing partner Restless Development Uganda (RD). RD was responsible for tracking and submitting information to IYF as part of quarterly reporting. Baseline, formative, and follow-up surveys of youth participants were developed by IYF in collab￾oration with RD and administered to gather more specific information about youth participants and program performance. RD administered a baseline survey at program outset. Upon completion of internships, RD administered an exit survey with youth. A follow-up survey was administered six months after youth completed their internships. Information is available on a group of youth who applied but were not selected. This may provide the basis for a better understanding of the impact of this program. Performance monitoring of interns was conducted by RD and placement partners—to monitor pro￾gress throughout the internship period and to identify areas requiring further professional develop￾ment and mentorship. Necessary information was gathered through course evaluation forms, intern￾ship monitoring forms and progress reports (developed by IYF and RD), and site visits to facilities and employer workplaces. Outcomes of performance monitoring feed into the program and planned formative evaluation (detailed below), which may have resulted in modification of training design and other services provided by the program. Formative evaluation was conducted to assess the process of program implementation and provide the management with information necessary for continuous program improvement. RD was re￾sponsible for formative evaluation (potentially with support from a consultant) and worked closely with employers and relevant stakeholders to collect necessary data early on and on a regular basis to inform later stages of the program and the second cohort training. Employer surveys were developed and administered among internship providers and employers to solicit their input, monitor progress of interns, measure their satisfaction with interns or graduates hired, and measure employer change in perception of youth as employees and interest in supporting youth transition into employment. YouthMap administered the following surveys with a sample of internship providers and employers: a. Pre-service survey with internship providers and employers before training begins; b. End-of-service survey with internship providers after youth complete internships; c. Six-month follow-up survey with employers to measure their satisfaction with gradu￾ates, among others. Key elements of the program to examine include:  Program design  Intern recruitment  Intern training (foundational and mid-term) YouthMap Uganda Evaluation Report – June 5, 2015 81  Intern placement  Intern support/mentoring  Employer recruitment  Employer support  Intern stipends and employer cost share  Intern employment support  Public-private partnerships in support of youth employability  Advisory Board role and effectiveness in providing technical guidance, and supporting part￾nership development and policy linkages  USAID’s role  Entrepreneurship training and enterprise development support  CLA research  Intern financial training and savings clubs  M&E D. Evaluation Questions This evaluation will gather data related to UIP achievements and their impact on the intended bene￾ficiaries in order to answer the questions below. Whenever applicable, data should be disaggregated by gender, and attention paid to differences in treatments or outcomes between genders. Program Effectiveness and Impact 1. How effective were the approaches taken for each of the four program components in achieving the program’s goals? o Orientation and skills training o Internship placements o Professional development support services, including mentoring, job placement ser￾vices, and participation in an alumni network o Facilitation of youth participation in USAID CLA-related activities o Usefulness of an internal YouthMap assessment for designing andor improving pro￾grams How useful have internal assessments been for designing and/or improving training/em￾ployment programs? 2. What was the program’s overall impact on youth beneficiaries? 3. How was the capacity of various program partners strengthened through the UIP and what were the outcomes of these partnerships? 4. What are the trends in intern employment data with respect to gender, sectors of interest, employer retention, and employment quality (e.g., part-time/fulltime, salary, hours, loca￾tion, and promotions)? Sustainability and Scalability 5. What are the factors facilitating or inhibiting the sustainability of program activities beyond the UIP (e.g. UIP board, etc.)? YouthMap Uganda Evaluation Report – June 5, 2015 82 6. Considering the financial and institutional context, which program elements can be scaled, and which entities are best placed to support and are actively interested in scaling-up (e.g., government, private sector, university system, international donors, etc.)? E. Evaluation Design and Data Collection and Analysis Methodology 1. Evaluation design and data collection methods This performance evaluation will primarily be a mixed methods, non-experimental, or quasi￾experimental design (see below). Data will be collected from a variety of sources including:  Primary data collection o Key informant interviews with program staff from IYF and Restless Develop￾ment, youth beneficiaries, employers, and advisory board members o Focus groups with the program participants o E-survey of youth participants o E-survey of business participants  Secondary data o Existing project data such as pre and post quantitative data previously col￾lected by IYF and Restless Development through the program’s M&E system including number of UIP applications, number of participants in each phase, and number of host employers by sector; o Program performance data including number of youth in employment or self￾employment, number of youth applying their learned skills, satisfaction of youth/employers, and quality of internships and jobs; o Document review In order to address bias, the focus group participants will be randomly selected as is feasible, and should proportionally match the number of participants and gender distribution between the regions. The survey will be sent to all program participants. In addition to the data collection methods above, the evaluation will have contact infor￾mation of youth that applied but were not accepted into the program. The evaluation team should also collect data from this group either through interviews or an e-survey to use as a comparison to program participants. Quasi-experimental impact analysis through matching or regression discontinuity techniques should be conducted if possible. 2. Data Analysis Methods As part of the inception report developed before the start of data collection, the evaluation team will include a data analysis plan that details how the key informant interviews and focus group data will be analyzed; what procedures will be used to analyze qualitative data from key informant and other stakeholder interviews; and how the evaluation will weigh and integrate qualitative data from these sources with quantitative data from the participant survey. The evaluation team will analyze the qualitative data from the interviews and focus groups for trends. YouthMap Uganda Evaluation Report – June 5, 2015 83 The evaluation team will also analyze program data including service statistics, beneficiary demo￾graphic data, attrition and retention information, field trip reports, intern reports, employer reports, baseline data, formative surveys, and other project data to support a triangulated, critical analysis of the program in order to answer the evaluation questions above. Gender will be disaggregated as a key component of the analysis (available in Key Survey). 3. Limitations to proposed evaluation design and methods While the mixed methods design aims to arrive at unbiased findings, conclusions, and recommenda￾tions by bringing together a variety of sources, there is always the possibility of introducing bias through interviews or focus groups. Additionally, an e-survey will always have the possibility of a respondent bias based on who chooses to complete the questionnaire. The evaluation team will be mindful of these potential issues and should mitigate this bias as much as possible. F. Logistics The evaluation team will focus their data collection in Kampala, since the majority of youth, employers, and other stakeholders are located in Uganda’s capital city (see Annex B). Data collection will also take place in at least one of the other six districts. The first cohort gradu￾ated in December 2013 and the second cohort in August 2014. YouthMap Uganda Evaluation Report – June 5, 2015 84 The following is the level of effort table covering the Uganda evaluation: Position Desk Review Travel Methodology, Design, and Tool Develop￾ment (Field) Surveys/Fo￾cus Groups/In￾ter￾views/Analy￾sis/Initial Drafting Presentation to USAID/W, Draft and Final Report Total Dr. Michael J. Midling (Team Leader) 1 7 8 14 0 30 Ms. Tamara Duggleby (Vocational Edu￾cation Specialist) 1 4 8 0 13 Mr. Martin Opolot (Local Expert; Econo￾mist) 6 14 20 II. Expected Deliverables The evaluation team will be expected to produce the following deliverables: • Inception Report for each country, including final evaluation questions, data collection and data analysis plan, and a work plan to guide the overall evaluation • Data collection tools for each country, including: o Key informant interview guide o Youth focus group guide o Youth survey and employer survey to measure their satisfaction with the quality of graduates and better understand the quality of jobs • Debrief for Mission staff and program stakeholders on evaluation findings • Draft evaluation report, including all questions listed above, for review by IYF and USAID • Final evaluation report incorporating all edits/comments provided by IYF and USAID The Evaluation Report, which will review results for each country and draw overall conclusions and lessons learned, should be no longer than 35 pages, excluding annexes, structured as follows: 1. Executive Summary—including concise summary of findings and recommendations (2–3 pp); 2. Introduction—purpose, audience, and synopsis of task (1 pp); 3. Background—brief overview of the activity and purpose of the evaluation (3–4 pp); YouthMap Uganda Evaluation Report – June 5, 2015 85 4. Methodologies—describe evaluation methods, including constraints and gaps (1–2 pp); 5. Findings/Conclusions/Recommendations—for each key question (18–22 pp); 6. Key Lessons Learned—provide a list of key technical and/or administrative lessons learned, if any (2–3 pp); 7. Overall recommendations for future programming (2–3 pp); 8. Annexes—annexes that document the evaluation methods (attach final evaluation SOW), schedules, interview lists, and referenced documents. 9. Raw data collected through surveys, focus groups and interviews should be submit￾ted electronically in a separate file The Evaluation Report must also meet the following criteria as outlined in the USAID Evaluation Policy. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT (USAID Eval￾uation Policy, Annex 1)  The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why.  Evaluation reports shall address all evaluation questions included in the scope of work.  The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the AOR.  Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report.  Evaluation findings will assess outcomes and impact on males and females.  Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, un￾observable differences between comparator groups, etc.).  Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence.  Sources of information need to be properly identified and listed in an annex.  Recommendations need to be supported by a specific set of findings.  Recommendations should be action-oriented, practical and specific, with defined responsi￾bility for the action. III. Evaluation Management Pragma and the evaluation team will be responsible for arranging logistical support for this assign￾ment. However, MSTAS and IYF project staff and partners in Senegal and Uganda will assist in scheduling meetings. YouthMap Uganda Evaluation Report – June 5, 2015 86 The IYF home office personnel and local partners in Senegal and Uganda will be available to an￾swer inquiries concerning the design and implementation of the program. The evaluation team will report to and be under the supervision of the MSTAS Chief of Party. He will inform AFR/SD concerning the progress of the Evaluation Team. The evaluation will be managed by AFR/SD Senior Education Management Analyst, who is the point of contact in USAID/Washington. She will review and approve the evaluation work delivera￾bles. She will also maintain the AFR/DP Monitoring and Evaluation Advisor advised of the Evalu￾ation Team’s progress. The M&E advisor will be available for consultations concerning technical issues and questions that may arise during the course of the evaluation. The Evaluation Team will work closely with USAID/Uganda and USAID/Senegal, respectively. At the start of the field work in each country the Evaluation Team will meet with each mission and seek the mission’s guidance for best carrying out its responsibilities. In addition, Mission staff, in coordination with AFR/SD Management Analyst and Bureau M&E Advisor, will review and pro￾vide feedback on the evaluation team’s inception report, including final evaluation design and meth￾odology and data collection tools/instruments. The field work in each country will close with a presentation to each mission and in-country stakeholders of preliminary findings and conclusions. USAID/Washington and IYF may participate in this presentation via VTC if feasible, otherwise a separate presentation for USAID/Washington and IYF staff will be conducted soon after the evalua￾tion team completes field work. The evaluation team will submit the final evaluation report to AFR/SD and upload it on the USAID Development Experience Clearing House (DEC). The evaluation team will provide weekly debriefings to the AFR/SD Senior Management Analyst on progress made on implementation of the evaluation, including any challenges encountered in order to find solutions to those challenges. YouthMap Uganda Evaluation Report – June 5, 2015 87 Annex A: Program Indicators, Uganda Output Indicators:  Program effectiveness in attracting and enrolling target population o Number of youth who enroll in the internship program versus target o Total number of person hours of youth successfully trained  Program effectiveness in retaining youth in skills training and overall internship program o Number and percentage of youth who complete training and internships  Program effectiveness in placing youth in quality internship placements o Number of USAID implementing partners (IPs) / government agencies / private sector employers delivering quality internships to youth  Program effectiveness in facilitating youth to drive the CLA agenda o Number of youth participating in CLA initiatives/number of USAID staff engaged in CLA initiatives o One alumni database successfully created and functioning Short-term Outcome Indicators:  Number and percentage of youth who report satisfaction with employability and life skills training courses delivered  Number and percentage of youth who report self-efficacy at program completion  Number / percentage of youth with greater hope, optimism and aspirations about their future as a result of program participation (to be measured by comparing Synder hope scores before and after the program)  Number / percentage of youth with favorable ratings as interns  Number / percentage of youth reporting satisfaction with internships  Number / percentage of interns who have secured employment upon completion of their intern￾ships (disaggregate by type of employment)  Number / percentage of youth who feel they have contributed to the CLA agenda and that youth voices are heard (at follow-up) Intermediate Outcome Indicators:  Number / percentage of graduates working at follow-up (disaggregate by salaried employment and self-employment)  Number / percentage of employers satisfied with program graduates hired  Number/percentage of employers replicating the internship programme  Number / percentage of youth reporting satisfaction with work  Degree to which youth participants feel the internship and other program services have contrib￾uted to their professional development and employability YouthMap Uganda Evaluation Report – June 5, 2015 88 Annex B: Intern Placements, Uganda Table 1: Number of Interns and Employers by District District Interns Employers Busia 1 1 Gulu 9 1 Jinja 15 9 Kampala 64 20 Kitgum 2 1 Lira 4 2 Mbarara 5 3 Total 100 37* *There are a total of 32 total employers, some hosted interns across multiple districts Table 2: Number of Interns by Sector Sector Number of Interns Agriculture 6 Banking and Finance 20 Government 11 Hospitality and Tourism 13 Information and Communication 16 NGO 34 Total 100