EVALUATION FINAL PERFORMANCE EVALUATION OF THE YOUTH BUSINESS LEADERSHIP PROJECT (YBLP) IN AZERBAIJAN [August 2014] This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Mendez England & Associates. Final Performance Evaluation of the Youth Business Leadership Project (YBLP) in Azerbaijan Final Report August 2014 Prepared under RFTOP No. SOL-518-13-000009 Submitted to: USAID/Azerbaijan Submitted by: James Statman, Team Leader Bahruz Babayev, Local Specialist Contractor: Mendez England & Associates 4300 Montgomery Avenue, Suite 103 Bethesda, MD 20814 Tel: 301- 652 -4334 www.mendezengland.com DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. TABLE OF CONTENTS EXECUTIVE SUMMARY................................................................................................................... i Evaluation Purpose...................................................................................................................................................... i Project Background .................................................................................................................................................... i Evaluation Design, Methodology and Limitations...............................................................................................ii Evaluation Findings and Conclusions....................................................................................................................iii Recommendations and Lessons Learned ...........................................................................................................vii 1.0 EVALUATION PURPOSE & EVALUATION QUESTIONS.............................................. 1 1.1 Evaluation Purpose.........................................................................................................................................1 1.2 Evaluation Questions.....................................................................................................................................1 2.0 PROJECT BACKGROUND.................................................................................................... 2 2.1 The Development Context.........................................................................................................................2 2.2 YBLP Summary ...............................................................................................................................................3 3.0 EVALUATION METHODS AND LIMITATIONS............................................................... 4 3.1 Evaluation Methodology ...............................................................................................................................4 3.2 Evaluation Limitations...................................................................................................................................6 4.0 FINDINGS AND CONCLUSIONS ....................................................................................... 7 4.1 Question 1: To what extent was YBLP implemented as planned and able to achieve its stated objectives?............................................................................................................................................7 4.1.1 Findings...............................................................................................................................................7 4.1.2 Conclusions .................................................................................................................................... 14 4.2 Question 2: What progress in career development was achieved by students who participated in YBLP? Were there any gender differences in career progress or obstacles encountered? ................................................................................................................................................ 15 4.2.1 Findings............................................................................................................................................ 15 4.2.2 Conclusions .................................................................................................................................... 16 4.3 Question 3: How effective was the project in coordinating and collaborating with stakeholders (USAID, BP, AmCham, JAA and participating companies)?...................................... 16 4.3.1 Findings............................................................................................................................................ 16 4.3.2 Conclusions .................................................................................................................................... 17 4.4 Question 4: What are the various factors that have enhanced or limited the effectiveness, results and potential for sustainability of YBLP in Azerbaijan? ......................................................... 17 4.4.1 Findings............................................................................................................................................ 18 4.4.2 Conclusions .................................................................................................................................... 25 5.0 RECOMMENDATIONS ....................................................................................................... 26 5.1 Program Objective/Target Beneficiaries................................................................................................ 26 5.2 Student Recruitment, Testing and Selection.............................................................................. 26 5.3 Program Administration and Management ............................................................................... 26 5.4 Professional Development Workshops.................................................................................... 27 5.5 Internships and Mentorships..................................................................................................... 28 5.6 End-of-Program Enhancements................................................................................................. 28 6.0 LESSONS LEARNED............................................................................................................ 28 ANNEXES Annex 1: Statement of Work Annex 2: Evaluation Schedule Annex 3: Evaluation Design Matrix Annex 4: Interview Guides / Data Collection Instruments Annex 5: List of Documents Reviewed Annex 6: YBLP Participating Universities Annex 7: YBLP Evaluation Participants FIGURES Figure 1: YBLP evaluation student participation by gender, location, and method Figure 2: YBLP mentors by location and method of involvement Figure 3: YBLP participants by location and gender Figure 4: Student confidence in ability to have successful business career following participation in workshop Figure 5: Mentor organization by type Figure 6: Topics discussed between Mentors and Mentees Figure 7: Student average rating of the importance of mentoring experience, by gender Figure 8: Student average rating of the importance of mentoring experience, by location Figure 9: Student average rating of comfort in talking to mentor, by gender Figure 10: Student average rating of comfort in talking to mentor, by location Figure 11: Mentors’ average rating of the value of the mentoring process, by location Figure 12: Student opinions of YBLP benefits Figure 13: Student average rating of the overall value of YBLP, by location Figure 14: Student average rating of the professional development workshops, by gender Figure 15: Student average rating of internship effectiveness, by gender Figure 16: Student average rating of internship effectiveness, by location ACRONYMS AmCham American Chamber of Commerce Azerbaijan BP BP Exploration (Caspian Sea) Limited CEO Chief Executive Officer CSO Civil Society Organization E3 Economic Growth, Education and the Environment FGD Focus Group Discussion FY 2014 Fiscal Year 2014 GAA Ganja Agrobusiness Association GoA Government of Azerbaijan HR Human Resources ICT Information and Communications Technology JAA Junior Achievement Azerbaijan KII Key Informant Interview ME&A Mendez England and Associates NGO Non-Governmental Organization RA Rapid Appraisal PMP Performance Monitoring Plan SOCAR State Oil Company of the Azerbaijan Republic USAID United States Agency for International Development YBLP Youth Business Leadership Project EXECUTIVE SUMMARY EVALUATION PURPOSE This is a report on the Final Performance Evaluation of the Youth Business Leadership Project (YBLP) funded by the United States Agency for International Development (USAID) Mission in Azerbaijan. YBLP is being implemented by Junior Achievement Azerbaijan (JAA) during the period of October 1, 2012 – September 30, 2014, for a total cost of $200,000. The evaluation of YBLP was conducted during the period of May – July, 2014, by a team assembled by Mendez England & Associates (ME&A) with headquarters in Bethesda, Maryland. The purpose of the evaluation was to: 1) determine the results of YBLP interventions; 2) evaluate the progress made by YBLP under each component based on established targets; and 3) provide lessons learned and recommendations from YBLP’s design and implementation to inform the design of the follow-on activity. The evaluation covered the period of October 1, 2012 – May 1, 2014. The information uncovered by this evaluation will guide the USAID/Azerbaijan’s future project designs and implementation and may be shared with other donors, other implementing partners, and other United States Government agencies operating in Azerbaijan. PROJECT BACKGROUND Commencing in October 2012, YBLP has sought to empower the next generation of business leaders in Azerbaijan by providing undergraduate business students with hands-on professional development workshops to enhance business skills, the opportunity to gain real world experience through internships at various private companies, mentorship with successful businessmen and businesswomen, and networking opportunities with like-minded peers. The YBLP’s objectives were to: • Provide real world employment opportunities for 120 young people. • Develop their business skills beyond the classroom. • Provide role models whom will offer the students career advice and guidance. YBLP has been organized, funded, and implemented as a public-private partnership through the efforts of four key stakeholder organizations: 1) USAID, which conceptualized and designed the project; identified and recruited implementing partners; co-funded, administered and supervised program implementation; and initiated and supervised the final performance evaluation; 2) BP Exploration (Caspian Sea) Limited (BP), which on behalf of co-ventures co-funded and supervised project implementation under BP’s Sustainable Development Initiative Framework and provided mentorships under the program; 3) American Chamber of Commerce Azerbaijan (AmCham), which provided access to its 284-member network of businesses and organizations through which YLBP was able to create ‘real world’ internships and mentorships with 32 companies and organizations; and 4) JAA, which, under agreements with USAID and BP, served as YBLP’s implementing partner responsible for outreach and recruiting 120 third and fourth year business students (with a target of 60% female participation) from eight universities; creating and conducting objective student assessment, screening and selection procedures; organizing and i delivering YBLP professional development workshops taught by qualified and experienced instructors; working with AmCham to match participating students with internship and mentorship opportunities relevant to their career objectives; developing and implementing a Performance Monitoring Plan (PMP) through which to track project implementation; and providing a stipend to each student who has completed the program. As of May 1, 2014, YBLP had conducted four (of five) student intakes or ‘streams,’ engaging 105 participating students from eight participating Azerbaijani universities, and 84 mentors from 32 companies, associations, and non-profit organizations. EVALUATION DESIGN, METHODOLOGY AND LIMITATIONS To conduct the YBLP evaluation, the Evaluation Team collected quantitative and qualitative data from a broad range of stakeholders and beneficiaries to ensure independence of the evaluation process as well as accuracy and completeness of the subsequent conclusions, recommendations, and lessons learned. The team utilized rapid appraisal (RA) techniques that balance each other, such as: quantitative and qualitative data; individual and group responses; semi-structured interviews and analysis of existing information; and data sets. Data was collected using the following methods: • Critical desktop review of materials related to YBLP, such as quarterly reports, annual work plans, USAID-JAA Agreement, and PMP. • Interviews with USAID, BP, AmCham, and JAA. • Semi-structured interviews with students, participating companies, mentors, professional development workshop instructors, and representatives of participating universities. • Focus group discussions (FGDs) with students in Baku and Ganja. • E-questionnaire survey of students, participating companies, and mentors. • Field visits to Baku and Ganja. • Direct observation to cross-check information. During the implementation, the Evaluation Team encountered a number of limitations inherent to the design of this evaluation. Some of the more relevant are listed below: • RA methodological constraints. USAID’s (2010) Performance Monitoring and Evaluation TIPS (#5) Using Rapid Appraisal Methods (2ond ed.) notes that while RAs are appropriate and can generate valuable insights for formative or performance evaluations, their constraints in sample size and selection criteria limit the generalizability of their findings. In accordance with accepted RA practices, the Evaluation Team utilized multiple methods to follow up and triangulate on key issues to improve the validity of its findings and decrease possible bias. • Selection Bias. As some informants were unavailable, unreachable, declined to be interviewed or complete the evaluation questionnaire – particularly female participants in Baku – there is a possibility of selection bias: those respondents who were interviewed might differ from those who were not in terms of their perspectives, experiences, and opinions. Bias due to small sample size and self-selection were, to the extent possible, offset through the use of multiple data collection methods to secure overlapping inputs and perspectives on the evaluation questions and to provide contextual information through which to interpret results. ii • Halo Bias. There is a tendency for respondents to under-report socially undesirable answers and alter their responses to approximate what they perceive as the social norm (halo bias) or the response desired by the interviewer. To mitigate this limitation, the Evaluation Team stressed respondent confidentiality and anonymity, conducted interviews in the settings where respondents were expected to feel comfortable, made sure that respondents clearly understood the purpose of the evaluation and felt free to ask any questions and clarifications, and took the time necessary to establish appropriate rapport between the interviewer and the respondent prior to commencing with the interview or FGDs. • Time Constraint. The limited time allocated for the evaluation did not allow the Evaluation Team to identify a matched control or comparison group of similarly talented non-YBLP students as would be done in a formal, long-term outcome or summative evaluation. FINDINGS AND CONCLUSIONS Question 1: To what extent was YBLP implemented as planned and able to achieve its stated objectives? Findings 1. YBLP succeeded in achieving its target level of student participation. During the period under review, YBLP was attended by 105 third- and fourth-year undergraduates from eight Azerbaijani universities based in Baku (84 students) and Ganja (21 students). This number covers the four initial intakes or ‘streams’ of the project. The YBLP population was comprised of 48% male and 52% female students. 2. Based on qualitative input from students, stakeholders, university personnel, and professional development workshop instructors, the recruitment and selection process for YBLP was fair and transparent. 3. Although JAA listed fluency in spoken English as a selection criterion, many participants in Ganja and a few in Baku could not fluently converse in English with the evaluators. 4. In conformance with targets, all 105 students participated in the internships in 32 local and foreign companies, government agencies, and civil society organizations (CSOs). Eighty-four mentors were involved in supervising students during their internships. 5. Although YBLP was generally implemented as it was designed, the process somewhat diverged from expectations along several dimensions due to ambiguity in the language of the design and agreement. Actual hands-on internships lasted three months, not the four months stated in the design. Although all of the internships appear to have been relevant to gaining an introduction to the professional working environment, the match of internship and student career trajectory – particularly for the first intake – was often attenuated, with some placements made not with businesses or other for-profit enterprises but rather with non￾profits and non-governmental organizations (NGOs). There were also some instances in which students received mentorships without an actual hands-on internship and some others where students were placed in existing internship programs but did not receive the guidance and personal relationship of a mentor. 6. Evaluation data shows strong support for the value of the project’s three components – professional development workshops, internships, and mentorships – in enhancing students’ careers. For example, 92% of the students reported that the workshops increased their confidence in pursuing their careers. On a 5-point scale, students rated the effectiveness of internships in enhancing their careers at 3.8, where 5 represents “extremely effective,” and iii the importance of the mentoring experience at 4.1, where 5 represents “extremely important.” Students, mentors and instructors all rated YBLP as a whole to be a highly effective and useful intervention. Students rated the overall value of YBLP in assisting them in pursuing their careers at 3.6 on a 4-point scale, where 4 represents “extremely valuable.” Such a rating is a strong endorsement of the program. Conclusions 1. The overall body of quantitative and qualitative findings from all project stakeholders concur that YBLP achieved its stated objectives of selecting appropriate student participants, delivering useful and worthwhile professional development workshops, and providing internships and mentorships. All believe that the project will enhance students’ career paths. 2. The student selection process was fair and efficient. However, more should have been done in selecting students that were fluent in written and spoken English, as well as selecting students from professional career paths that are associated with business studies. 3. Qualitative and quantitative data collected during individual interviews with students, universities, and companies, as well as FGDs with students, show that YBLP benefited both students and host organizations. As a tool for career planning and development, the project enabled the participants to gain their first work experience, affirm career interests, and increase their business knowledge. The workshops and internship program improved the business, communication, and leadership skills of students in general, and taught them how to perform in a professional environment. Furthermore, the project increased students’ confidence in pursing their chosen careers. 4. There are few cases in which students’ interests and ambitions were not matched properly with the internship organization where they were placed. In addition, some of the internships were with NGOs or non-profit associations rather than businesses. 5. Although in general the program was implemented as designed, there were a few cases when internships were not associated with the required mentorship and vice versa. Question 2: What progress in career development was achieved by students who participated in YBLP? Were there any gender differences in career progress or obstacles encountered? Findings 1. Quantitative and qualitative data obtained from 40 students and 21 mentors indicated that students praised YBLP for exposing them to the professional work environment, a context of which they previously had little or no direct knowledge. They reported that the competition for entry-level professional jobs is stiff and that access to professional networks through YBLP to facilitate the job search has greatly improved their chances for success. Project stakeholders agree that after having participated in a selective program, such as YBLP, that includes actual workplace experience, students will have the means to distinguish themselves and gain a competitive edge. Some students also reported that the internship experience helped confirm their career choice, giving them more confidence as they seek their initial employment. Conclusions 1. YBLP succeeded in enhancing students’ career development along the specific dimensions it was designed to affect. Students and mentors reported that participants acquired useful, iv employment-relevant skills and knowledge. They were able to gain genuine work experience within a supportive context that exposed them to the realities, norms, and expectations of the professional work environment. In addition, the YBLP experience helped build their confidence, reaffirm career choices, and became an item on their CVs that should provide an asset and competitive advantage in the job search. Question 3: How effective was the project in coordinating and collaborating with stakeholders (USAID, BP, AmCham, JAA and participating companies)? Findings 1. Key informant interviews suggest that, particularly during project start-up, recruiting and placing the first intake (or stream) of students, project communication and coordination was challenging. During the first stream, AmCham also had difficulty in recruiting appropriate internships and mentorships from amongst its members. As the project matured, these difficulties and tensions subsided. 2. Some mentors were not fully briefed or trained on their roles and responsibilities, beyond the requirement to complete the daily narrative reports. In one instance in Ganja, students were placed into an existing internship program but company employees supervising the interns were not informed of the project’s mentoring expectations. 3. While communication with stakeholders has been good, there have been instances in which the lack of clear and updated information (such as leadership changes at JAA earlier in the process) has caused some concerns. 4. Under time pressure, particularly during stream one, and responsible to simultaneously conduct student recruitment and selection, organize the professional development workshops, and establish a working performance management plan (PMP), JAA did a remarkable job of program management and administration. However, given the complexities and time constraints of YBLP’s design, some breakdowns in communications did occur. 5. One area where JAA put in a great deal of effort with little useful return was in the creation of the project’s PMP, a cumbersome paper and narrative-based system that generated huge amounts of text but no timely management information to support program operations. Conclusions 1. The rapid start-up, difficulty of initially finding and matching students to appropriate internships, number of participating partners, restrictions of the academic year, multiple overlapping administrative and technical tasks required of JAA over a short implementation time-span, and the involvement of four key stakeholder organizations in a program jointly funded under a public-private partnership model, all contributed to generating communications and operational challenges for the project. Given this level of complexity in what was generally regarded as a rather straightforward program design, it was inevitable that some communication lapses and strains would occur. 2. To its credit, JAA appears to have met all of the USAID reporting requirements under its agreement, and to have successfully implemented the program in accordance with its schedule and budget. 3. Stakeholder coordination and information sharing could be strengthened through brief monthly meetings that would review program activities, accomplishments and challenges. In this way, YBLP and AmCham could also benefit from further input from BP and its partners v around additional private sector opportunities and resources, and gain insights into the current job market in Azerbaijan. 4. The YBLP PMP was fully implemented as planned but proved too cumbersome to provide timely management information. Project reliance on written narrative reports, particularly daily logs prepared by both mentors and students, imposed a huge administrative burden on JAA and did not offer useful management information during the course of the project itself. Question 4: What are the various factors that have enhanced or limited the effectiveness, results and potential for sustainability of YBLP in Azerbaijan? Findings 1. Students, mentors, and stakeholders report that the success of YBLP is directly tied to the quality of its three primary activities: professional development workshops, internships, and mentoring relationships. Students and mentors rated the mentorship experience highly on scales assessing: 1) students’ views of the importance of the mentoring experience and their comfort in talking with their mentors; and 2) the mentors’ assessment of the value of the mentoring experience to the students’ career development. 2. Project success stories show the importance of the fit between students’ interests and career ambitions, and the type of organization in which they were placed. In successful internships, students were given significant professional responsibilities and made contributions that directly assisted the host company/organization in pursuing its mission. Similarly, the host organization provided strong mentoring and guidance, clear direction, gave students challenging assignments that contributed to the organization, and treated them as members of the team. 3. Mentor and student accounts suggest that the three-month period was sufficient to produce meaningful internship and mentorship experiences that clearly benefitted the students and host organizations. 4. The most significant reported challenges to YBLP outcomes were the result of logistical issues related to difficulties of program implementation under time constraints and at a distance in Ganja. Another challenge found in larger organizations is that internships are often coordinated and managed by the company’s central human resources (HR) department that will then identify internships and mentors. In a few instances, the HR department recruited internship supervisors but these individuals had no direct contact with JAA and were not aware of the mentorship requirements of the project. Conclusion 1. Key factors that enhanced YBLP’s results and potential for sustainability were the quality of its three main components, the careful match of students’ interests with the organization in which they were placed, hands-on internships during which students were able to make significant contributions, and strong mentoring and guidance provided by experienced businessmen or businesswomen. 2. Over time, and with each new stream, key factors enhancing project effectiveness were strengthened through the establishment of implementation networks, and administrative and technical lessons learned. As additional internships and mentorships were identified and brought into the program, there was increasing likelihood of more appropriate matching of students’ career interests and the focus of the host organization. Mentors became more experienced in their role so they could identify techniques and approaches to make the vi mentorships even more effective. As JAA recruited each new stream, they could build upon their base of experience to improve the recruitment and selection process. 3. Among the factors that might have had an effect on the program’s effectiveness are the logistical issues (especially for implementation in Ganja) as well as the fact that in many large organizations internships are managed and coordinated by their HR departments which, in some cases, did not communicate with JAA about the mentorship program under YBLP. RECOMMENDATIONS AND LESSONS LEARNED Recommendations 1. A decision must be made as to whether a follow-on program will be targeted at entrepreneurial, business skills, and career development for undergraduate business students, or if it will be a more general career and professional development program for undergraduate students pursuing diverse professional areas that are not specifically related to business or entrepreneurship. 2. If a follow-on program is focused on the business sector, it should be restricted to student participants who are pursuing business studies and who intend to have careers in business. 3. If fluency in spoken English is deemed an essential selection criterion, the program should either ensure that this standard is actually followed in student selection, or initiate a remedial English language improvement course that would be mandatory for those students who have English language reading and writing skills but do not meet the verbal fluency standard. 4. All YBLP internships should be in an organization that is directly relevant to the students’ field of study and career goals. 5. All students should participate in both internship and mentorship experiences at the host institution. 6. The PMP data collection and analysis system should be greatly simplified to reduce the amount of unnecessary and largely unhelpful paperwork and, instead, generate timely and useful management information relevant to program administration. An automated data entry and analysis system can be built on a mobile phone platform (or other device) that entails the weekly input of data through completion of a small number of rating scales assessing key dimensions of project performance rather than requiring daily hand-written inputs into log￾books that remain un-read. In addition, a new, more agile and useful IT-based system, could easily compile and generate useful project data, reducing the JAA’s level of effort and assisting in program operations, and reporting and communications amongst key stakeholders, project beneficiaries and other role-players. 7. More time for up-front planning and working with universities and companies would enhance the process of student selection and the identification of appropriate internship/mentoring opportunities. It would also likely decrease the time students wait for the start of the internships, possibly reducing the drop-out rate. 8. Monthly project review meetings of key stakeholder organizations would facilitate program communication, coordination and planning. 9. A follow-on program should consider including two university career centers as stakeholders/implementing partners. Participation of career centers would enhance their vii capacities, increase the number of potential beneficiaries, and increase program sustainability by embedding the intervention within institutions of higher education. Lessons Learned 1. Quantitative and qualitative input from key stakeholders, company representatives, mentors, workshop instructors and students support the projection that short-term, intensive, career focused programs such as YBLP will remain of value for at least the next five years. 2. Sufficient time for project start-up in each location in which the program is implemented will facilitate better student recruitment and selection, internships, mentor training, and delivery and matching of internships with students’ career interests. 3. Programs that seek to address gender-related issues and challenges confronting university￾trained young women and men in contemporary Azerbaijan need to be built around a nuanced understanding of the complex and evolving social, cultural and economic dynamics underlying career choices, development and success. 4. Although presenting its own challenges around focus and communications, a public-private model that enlists a major international corporation such as BP in the design, implementation and funding of a program such as YBLP offers extraordinary benefits in terms of relevance to actual workplace concerns and credibility in the business community. 5. Based on YBLP’s experience, key stakeholders should make firm policy decisions around the level of specific business focus for any follow-on programs and adhere to these standards in student selection, and internship recruitment and placement. viii 1.0 EVALUATION PURPOSE & EVALUATION QUESTIONS 1.1 EVALUATION PURPOSE This is a report on the final performance evaluation of the Youth Business Leadership Project (YBLP) funded by the United States Agency for International Development (USAID) Mission in Azerbaijan. YBLP is being implemented by Junior Achievement Azerbaijan (JAA) between October 1, 2012 and September 30, 2014, for a total estimated cost of $200,000. The evaluation was conducted by a technical team assembled by Mendez England and Associates (ME&A), including: Dr. James M. Statman (Team Leader) and Mr. Bahruz Babayev (Local Technical Adviser). The main goal of the evaluation was to: 1) determine the results of YBLP interventions; 2) evaluate the progress made by YBLP under each component based on established targets; and 3) provide lessons learned and recommendations from YBLP’s design and implementation to inform the design of the follow-on activity. The evaluation covered the period of October1, 2012 – May1, 2014 and, therefore, did not include the final stream of students who will complete their participation in the program during the fourth quarter of fiscal year (FY) 2014. The results of the evaluation will be used by USAID/Azerbaijan to guide the Mission’s future project designs and implementation. The Mission will be the primary audience for this evaluation. Other audiences include: USAID/Washington (Bureau for Europe and Eurasia, Bureau for Economic Growth, Education and the Environment (E3), and Bureau for Policy, Planning, and Learning), JAA, BP, AmCham, and other USAID implementing partners, businesses that hosted interns, and beneficiaries of YBLP activities (i.e., the students). 1.2 EVALUATION QUESTIONS As per the scope of work (SOW), the Evaluation Team was asked to answer a set of evaluation questions, outlined below: 1. To what extent did YBLP achieve its stated project objectives? 2. What progress in career development was achieved by students that participated in YBLP? Were there any particular obstacles encountered by the male vs. the female students? 3. How effective was the project in coordinating and collaborating with stakeholders [USAID, BP Exploration (Caspian Sea) Limited (BP), the American Chamber of Commerce (AmCham) in Azerbaijan, and participating companies]? 4. What are the various factors that have enhanced or limited the effectiveness and results of YBLP in Azerbaijan? As presented in the Evaluation Design Matrix (Annex 3), each major question was further elaborated and operationalized through a series of sub-questions around which evaluation findings are presented. 1 2.0 PROJECT BACKGROUND 2.1 THE DEVELOPMENT CONTEXT With the agreement to commence the transport of natural gas from the Shah Deniz II field to Western Europe in 2017, the extraordinary decade-long expansion of the Azerbaijani economy, triggered by the opening of the Baku-Tbilisi-Ceyhan gas pipeline in 2005, is likely to continue and even increase. Despite a recent downturn in oil production, international and domestic energy￾sector companies continue to eagerly invest in Azerbaijani ventures, broadening their local profiles and generating sufficient revenues to enable the Government of Azerbaijan (GoA), the State Oil Company of the Azerbaijan Republic (SOCAR), and others, to channel funds towards the development, strengthening and expansion of other sectors of the economy, including construction, agriculture, banking, infrastructure, tourism, and industry. Led by private investment and encouraged by GoA’s policy, which identified ‘diversification’ as its top economic priority, Azerbaijan’s non-oil sector grew 10% in 2013, and almost 9% in the first quarter of 20141 . The impending relocation of the industrial complex in Sumgait to Garadagh, outside of Baku, is expected to generate job opportunities and serve as an impetus for the further modernization of industrial production. With the growth and expansion of these pillars of the Azerbaijani economy has come the concomitant development of enterprises and sectors necessary for its operation, elaboration and support, including transportation, communication, housing and hotels. Building upon this momentum, the GoA has actively sought to position the country, and particularly its capital city Baku, as a magnet for international popular cultural, scientific, academic, artistic, and sports events. From the wildly popular Eurovision competition in 2011, through the European Rhythmic Gymnastics Championships in June 2014, to the first European Games to take place in Baku in June 2015, Azerbaijan is rapidly creating the infrastructure, architecture, technical expertise, and human resources to become a popular global destination for commerce and tourism. And with the launch of major international hotels in Baku, including the Hilton, Fairmount, Four Seasons, and Marriott, as well as related businesses, has come the expansion of norms of international business standards and practices and the need for well trained professional personnel. Large Azerbaijani companies are professionalizing their management and recruitment systems in order to meet international standards and enhance their competitiveness. Local companies are often also required to have staff complete training programs in order to become certified to compete for international and regional tenders, conduct business with international companies operating in Azerbaijan as approved providers of goods and services, and attract international foreign investment. The use of social media, online job recruitment, and other IT innovations are also rapidly transforming the local business culture and creating new employment opportunities. Sustained economic growth has generated a substantial decrease in Azerbaijan’s unemployment rate from a high of 11.8% in 2000 to 5.2% in 2012. Since 2003, almost a million new jobs have been added. 2 However, despite an impressive rate of job creation, youth unemployment (ages 15- 24) remains high: 12.2% in 20123 , with female youth unemployment at 16.3%. Although the education sector is receiving increased resources, higher education institutions have largely 1 N. Quliyeva (2014). Goal Achieved. Region plus. No. 15, April, p. 4. 2 “Decent Work Country Profile: AZERBAIJAN;” International Labor Office, 2012. 3 The World Bank Data Indicators; http://data.worldbank.org/indicator/SL.UEM.TOTL.ZS/countries. 2 ignored the challenges young people face in the transition from university to the workplace. Slowly transforming from Soviet-era practices, undergraduate programs tend to be technically and theoretically focused with relatively few opportunities for ‘hands-on’ or practical, job-relevant experiences. University-level career classes remain rare exceptions and most university career centers do not appear sufficiently resourced or focused to creatively and effectively address the demands and realities of the local labor market. New graduates of local universities also face competition from peers trained abroad under a multi-year government-sponsored scholarship program; SOCAR, BP and other corporate overseas scholarship opportunities; and through other donor programs or private resources. Recent graduates often lack the knowledge, practical skills and access to professional networks necessary for a successful job search and, despite the fact that businesses are seeking to fill professional positions, finding a suitable first job without having the requisite ‘one to three years’ experience,’ remains for many a daunting and discouraging task. 2.2 YBLP SUMMARY Commencing in October 2012, YBLP has sought to empower the next generation of business leaders in Azerbaijan by providing undergraduate business students with hands-on business skills professional development workshops, the opportunity to gain real world experience through internships at various private companies, mentorship opportunities with successful businessmen and businesswomen, and with networking opportunities with like-minded peers. The YBLP objectives are to: • Provide real world employment opportunities for 120 young people. • Develop their business skills beyond the classroom. • Provide role models whom will offer the students career advice and guidance. As specified in the Agreement between USAID and JAA, the YBLP model is organized around three interrelated core program components: 1. Internships: YBLP works with Azerbaijan-based companies and non-profit organizations to identify practical and relevant 4-month internships for third and fourth year undergraduate students. Through the internships, aspiring business leaders are expected to acquire practical skills, receive exposure to business practices of the highest standards, and interact with and learn from successful role models. 2. Professional Development Workshops: YBLP’s design requires that students participate in two professional development workshops addressing a number of topics, including: self￾assessment, success skills for finding and keeping a job, resume development, business communication, proper interview conduct, project design, conflict resolution, and workplace decision-making. These workshops are expected to play an important role in the professional development of students by instilling confidence and providing them with practical skills needed to compete in the market. 3. Mentorships: The YBLP model envisions a mentoring relationship in which the student and mentor openly share their expectations, meet regularly over the 4-month period, and come prepared to each meeting with questions, advice and topics of discussion. YBLP has been organized, funded and implemented as a public-private partnership through the efforts of four key stakeholder organizations: 3 • USAID: Conceptualized and designed the project; identified and recruited implementing partners; co-funded, administered and supervised program implementation; and initiated and supervised the YBLP’s final performance evaluation. • BP: On behalf of co-ventures, co-funded and supervised project implementation under BP’s Sustainable Development Initiative Framework and provided mentorships under the program. • AmCham: Provided access to its 284-member network of businesses and organizations through which YLBP was able to create ‘real world’ internships and mentorships with 32 companies/organizations (including AmCham and JAA). • JAA: Under agreements with USAID and BP, served as YBLP’s implementing partner responsible for: a) recruiting 120 third and fourth year business students (with a target of 60% female participation) from nine participating universities; b) creating and conducting objective student assessment, screening and selection procedures; c) organizing and delivering the YBLP professional development workshops taught by qualified, experienced instructors; d) working with AmCham to appropriately match participating students with internship and mentorship opportunities relevant to their career objectives; e) developing and implementing a Performance Monitoring Plan (PMP) through which to track project implementation; and f) providing a stipend to each student who completes the program. As of May 1, 2014, YBLP had conducted four (of five) student intakes or ‘streams,’ engaging 105 participating students from eight (of nine4 ) participating Azerbaijani universities, and 84 mentors from 32 companies, associations and non-profit organizations (NGOs). 3.0 EVALUATION METHODS AND LIMITATIONS 3.1 EVALUATION METHODOLOGY The evaluation methodology was carefully designed to respond to the evaluation questions outlined in the SOW, as well as to document the results of project interventions, assess project delivery against established targets, and present program narratives, lessons-learned and recommendations to inform future programming. Further, the methodology describes the extent and ways in which the various components and interventions comprising YBLP achieved their objectives; assesses the career development progress of the participating students with a particular focus on female participants; documents the effectiveness of coordination and collaboration amongst the key stakeholders and participating private sector companies; and identifies the principal factors enhancing or limiting results. The evaluation was designed to be: 1) participatory, significantly engaging and giving voice to the perspectives, ideas, and experiences of the key stakeholders, participating students, mentors, university personnel, and other YBLP role-players; 5 2) multi-dimensional, creatively utilizing a variety of overlapping evaluation techniques to document, refine, and bring nuance to findings and recommendations; 3) evidenced-based, rooted in documented quantitative and qualitative results; 4 Students from the Azerbaijan Diplomatic Academy are included in the fifth YBLP “stream.” 5 An article in the most recent (June 2014) volume of the American Journal of Evaluation asserts that “…the extent to which evaluators interact with their stakeholders is the most important contributor to the utility of evaluations” (p. 173). 4 and 4) useful, generating policy and program-relevant data, results, narrative impact-accounts, and analyses. Per the evaluation SOW, a variety of quantitative and qualitative rapid appraisal (RA) methodologies were used during the evaluation. Data was collected from a broad range of stakeholders and beneficiaries to ensure independence of the evaluation process, as well as accuracy and completeness of the subsequent conclusions, recommendations, and lessons learned. The Evaluation Team used a number of techniques that balance each other, including: quantitative and qualitative data, individual and group responses, and semi-structured interviews and analysis of existing information. The evaluation obtained primary data from: • All four key stakeholders – USAID, BP, AmCham, and JAA – through semi-structured interviews and a final de-briefing with each. • Forty (40) students, of which responses were collected from 17 via individual interviews, 14 via focus group discussions (FGDs), and 9 via e-questionnaires. • Participating companies, through individual interviews, small group discussions, and e￾questionnaires. • Twenty one (21) mentors, through individual interviews, group discussions, and e￾questionnaires. • Four (4) professional development workshop instructors, through group interviews. • Two (2) representatives of participating universities, through individual interviews. During the first week of field-work, the Evaluation Team contacted student participants in YBLP using the phone numbers and email addresses provided by JAA. Starting first with those in Baku, two attempts were made to contact each student via telephone to solicit their participation in either a FGD or an individual interview. Any student not reached by phone was then sent an e￾questionnaire and was asked to return it to the Evaluation Team e-mail address. The same process was followed in Ganja. Invalid contact details were provided for approximately 25 of the 105 students, likely attributable to participants having graduated and relocated, and changed their e-mail addresses and phone numbers in the process. The Evaluation Team made the initial round of phone calls in random order and restricted its follow-up calls to female participants because a lower rate of females had agreed to participate in individual interviews and FGDs. Figure 1, below, indicates that 40 of the 105 YBLP student participants (38%) provided input into the evaluation by participating in individual interviews, FGDs, or the e-questionnaire. Figure 1: YBLP Evaluation Student Participation by Gender, Location and Method YBLP Participants Total YBLP Students Individual Interviews Focus Group Participants E-Questionnaires Males Baku 41 12 2 5 Females Baku 43 1 4 4 Males Ganja 9 4 2 - Females Ganja 12 - 6 - Total 105 17 14 9 5 Of the 50 male YBLP participants (41 Baku; 9 Ganja), 25 (50%) provided input into the evaluation. However, of 55 female students (43 Baku; 12 Ganja), 15 (27%) participated. No e-questionnaires were returned from any students in Ganja. Figure 2, below, presents, 21 of 84 YBLP mentors (25%) participated in the evaluation; 10 in Baku and 11 in Ganja. Mentoring organizations in Baku were selected for participation with the purpose of insuring that local and multinational businesses, NGOs and associations were sampled. In light of traffic and transport challenges in Baku, preference was given to businesses and organizations closer to the central business district, whereas in Ganja all mentoring organizations were contacted. Figure 2: YBLP Mentors by Location and Method of Involvement YBLP Mentors Mentors Interviews E-questionnaires Baku 67 5 5 Ganja 17 10 1 Total 84 15 6 Total Participating in the Evaluation: 21 (25% of total mentors) 3.2 EVALUATION LIMITATIONS The Evaluation Team encountered some limitations inherent to the design of this evaluation and during its fieldwork in Azerbaijan. Some of the more relevant limitations are listed below: RA methodological constraints. USAID’s (2010) Performance Monitoring and Evaluation TIPS (#5) Using Rapid Appraisal Methods (2ond ed.) notes that while RAs are appropriate and can generate valuable insights for formative or performance evaluations, their constraints in sample size and selection criteria limit the generalizability of their findings. In accordance with accepted RA practices, the Evaluation Team utilized multiple methods to follow-up on and triangulate key issues to improve the validity of its findings and decrease possible bias. Still, there were a number of limitations inherent in design of the evaluation, which included: Limited student, especially female, participation. Given the challenges in reaching and recruiting students and the time limitations for data collection – and in accordance with the RA approach requested by USAID – random sampling procedures were not practicable. Selection Bias. As some informants were unavailable, unreachable, declined to be interviewed or complete an evaluation questionnaire – particularly female participants in Baku – there is a possibility of selection bias: those respondents who were interviewed might differ from those who were not in terms of their perspectives, experiences, and opinions. Bias due to small sample size and self-selection were, to the extent possible, offset through the use of multiple data collection methods to secure overlapping inputs and perspectives on the evaluation questions and to provide contextual information through which to interpret results. Female participation in FGDs in Baku and Ganja, and mentor input for example, augmented questionnaire and individual interview data for women participants. Halo Bias. There is a tendency for respondents to under-report socially undesirable answers and alter their responses to approximate what they perceive as the social norm (halo bias) or the 6 response desired by the interviewer. To mitigate this limitation, the Evaluation Team stressed respondent confidentiality and anonymity, conducted interviews in settings where respondents were expected to feel comfortable, made sure that respondents clearly understood the purpose of the evaluation and felt free to ask any questions and clarifications, and took the time necessary to establish appropriate rapport between the interviewer and the respondent prior to commencing the interview or FGDs. Time Constraint. The limited time allocated for the evaluation did not allow the Evaluation Team to identify a matched control or comparison group of similarly talented non-YBLP students as would be done in a formal, long-term outcome or summative evaluation. 4.0 FINDINGS AND CONCLUSIONS 4.1 QUESTION 1: TO WHAT EXTENT WAS YBLP IMPLEMENTED AS PLANNED AND ABLE TO ACHIEVE ITS STATED OBJECTIVES? 4.1.1 Findings How many third and fourth-year undergraduate business students from Azerbaijani universities participated in YBLP? What was the gender breakdown? During the period under review, YBLP was attended by 105 third- and fourth-year undergraduate students from eight Azerbaijani universities based in Baku and Ganja. This number covers the four major ‘intakes,’ or streams, of the project, which accounted for 105 of 120 total students. Figure 3 presents the participants of YBLP disaggregated by location and gender. The attendance consisted of 48% male and 52% female students. Increased female selection for the fifth stream, the final intake of 15 students which is not covered in this evaluation, will enable the YBLP to achieve its target of 60% female students. Figure 3: YBLP Participants by Location and Gender Participants # of participants % Males - Baku 41 39 Females - Baku 43 41 Males – Ganja 9 9 Females – Ganja 12 11 Total 105 100 Was the recruitment and selection process for YBLP participants fair, transparent and technically sound? Based on quantitative input from students, stakeholders, university personnel and Professional Development Workshop instructors, the recruitment and selection process for YBLP was fair and transparent. Participants did not raise any complains concerning the fairness or soundness of selection. Students were selected on the basis of knowledge assessment tests and interviews. However, one issue in the selection process should be mentioned. As highlighted by students in the Ganja FGD, JAA’s pre-selection promotional activities about the project were limited in Ganja, leading to a somewhat confused process in which students were hastily informed about YBLP and then immediately sat for the screening examinations. This limited the potential applicant 7 population and left the successful students somewhat bewildered about the details of the program to which they had been admitted. ‘Most students did not receive enough information about the project in advance. If all students had been informed, more students would have participated in the project’ (Student in Ganja) Through partnerships with Azerbaijani-based companies, did the YBLP provide relevant, ‘real world’ four-month internships for 120 participating students? Almost all students from the first four streams included in this evaluation (the remaining students in the fifth stream are not included in this evaluation, as previously mentioned) participated in internships in 32 local and foreign companies, government agencies, and civil society organizations (CSOs). Eighty-four mentors were involved in supervising students during their internships. Although YBLP was implemented as it was designed, the actual process somewhat diverged from expectations along several dimensions due to ambiguity in the language of the project design and agreement. Actual hands-on internships lasted three months, not the four months stated in the design. JAA considered the one-month process of engaging and interviewing the students to match them with appropriate internships – what they have called ‘job shadowing’ - as the fourth month. It should be noted, however, that in practice the three-month internship seemed efficacious. Interviewed students and mentors uniformly agreed that this was the optimal amount of time to allow the students to gain useful on-the-job experience without interfering with the universities’ examination requirements. Although all of the internships appear to have been relevant to gaining an introduction to the ‘real world’ of the professional working environment, the match of internship and student career trajectory – particularly for stream one – was often attenuated, with some placements made not with businesses or other for-profit enterprises but rather with non-profit associations and NGOs. Therefore, a few participants found themselves placed in an internship organization hardly related to their career ambition; for example, an oil geology student was placed in a hotel management internship. There were also a few instances in which students received mentorships without an actual hands￾on internship. For example, four petroleum engineering students, who did not have a hands-on internship, were mentored by experienced and extremely caring and committed BP mentors, and following the standard recruitment procedures, received job offers from BP. There were also instances in which students were placed in existing and quite effective corporate internship programs but did not receive the guidance and personal relationship of a mentor. ‘I learned much about banking and finance during my internship at Unibank. I experienced management of conflicts in a professional setting and learned about business communication with clients’ (Student in Ganja). Did each group of interns participate in two relevant professional development workshops? What were the topics/business-related skills presented in the workshops? All students participated in professional development workshops before starting their internships/mentorships. Quantitative and qualitative data collected by the Evaluation Team from students, mentors, and workshop instructors, strongly indicate that this was a highly valued and successful YBLP component. The workshop topics included business leadership, business ethics, 8 personal finance, management, communication skills, goal setting, skills for success, and performance in a professional environment. Each workshop was taught by one highly experienced trainer, with a total of six trainers used, who were given some latitude to employ various exercises and other participatory learning activities as they judged most appropriate to the group. As Figure 4, below, illustrates, students overwhelmingly viewed the workshops as having positively affected their level of confidence in their ability to have a successful business career, which is one of the YBLP program objectives. 8% Figure 4: Student confidence in ability to have successful bu participation in workshop Response to: "did the workshop affect your confidence in your ability t career?" Data from key informant interviews (KIIs) and FGDs suggest that students found the participatory and experiential character of the workshops to be an exciting and positive change from the usual university theory-centered lecture format. The Ethical Dilemmas of the Workplace Module was frequently mentioned as valuable and engaging. During interviews and FGDs, several students also spontaneously mentioned moments during their internship experience where they found themselves reflecting on something learned in the workshops that they then usefully applied in the internship setting. Were YBLP participants provided with structured, mentoring opportunities within the companies in which they interned? Did the mentors provide the students with useful advice and career guidance? Except in the few cases mentioned above, YBLP participants were provided with structured mentoring opportunities within the organizations in which they interned. Figure 5, next page, displays the business focus of the host organizations, showing an array of areas including 30% placement with NGOs. As expected, interview and questionnaire data reveal that during their internships, mentors and students discussed a variety of relevant topics related to professional career development. Figure 6, below, shows the frequency with which various topics were discussed, with student’s career goals being the most common. 9 0 1 2 3 4 5 6 7 Figure 5: Mentor organization by type 0 2 4 6 8 10 12 Figure 6: Topics discussed between Mentors and Mentees Overall, male and female students in Baku and in Ganja endorsed the importance of the mentoring experience and agreed that it positively affected their career development. YBLP’s design posited a close mentoring relationship in which student and supervisor, mentee and mentor, felt able to discuss issues of career development and internship performance. This dimension was assessed in individual interviews and questionnaires by asking students to rate on a 4-point scale their level of comfort in talking with their mentors. Students reported that they were comfortable in discussing career and internship matters with their mentors. Students in both Baku and Ganja rated the overall value of the mentoring process as a valuable experience in enhancing their career development. For a more detailed quantitative analysis on mentorship, see page 20, in which the mentorship component is discussed in more depth. 10 Qualitative data collected by the Evaluation Team further supports mentoring as a key component in program delivery, and many mentors expressed their satisfaction in working with young people and commitment to assisting them in successfully pursuing their career ambitions. A few comments from mentors and students provide some texture to these ratings: ‘My aim was to make interns independent in a workplace. I trained the interns on leadership and communication skills, and empowered them to become independent decision-makers in a professional environment’ (Mentor) ‘You have an instructor and he is like a ‘Dean’ that enlightens your way throughout internship period’ (Student) ‘My mentor helped me to cope with challenges during the internship’ (Student) ‘Mentorship is very important for students. It helps students develop their communication skills’ (Mentor) Were any significant changes in project activities, expected outcomes, products or deliverables instituted during the period of performance? The Evaluation Team’s review of project documents and discussions with key stakeholders, students, company representatives, and mentors suggests that during the implementation there were a number of apparent deviations from the specifics of the project design: a) many participants were not pursuing academic subjects usually considered indicative of a business career; b) students were not uniformly placed in internships that ‘matched’ their career goals; c) internship placements were not restricted to businesses but also included non-profit associations, NGOs, and government-supported or para-statal institutions, and actual on-site internships were of three rather than four months’ duration; d) some students received internships without the benefit of a mentoring relationship, or a mentoring experience without a ‘hands-on’ internship; and e) additional universities were included in the program. None of these changes in program implementation resulted in formal modifications of the agreement nor were project objectives or targets changed. Did YBLP activities enable participants to develop/improve their business skills beyond the classroom? There is a broad agreement amongst students, mentors, company and university representatives, and workshop instructors that YBLP succeeded in helping participants gain valuable ‘real-world’ experience, which will enable them to find employment and advance their careers in business or otherwise. Data collected from interviews and questionnaires sent to students show that the most frequently listed benefit was gaining actual work experience, followed by networking with fellow students (see Figure 7, next page). 11 0 1 2 3 4 5 6 7 Working with others Stipend Networking Real work experience Figure 7: Student Opinions of YBLP Benefits Interviews with students also affirm the project’s success in strengthening their business skills. Illustrative quotes from students convey this belief: ‘The project taught me to be patient and confident in a professional setting.’ ‘I have gained valuable knowledge and skills….’ ‘I gained knowledge about ethical issues which are important in any workplace.” ‘I got lots information about business, management, business ethics and so on which I had not heard before’ ‘The internship definitely affected my confidence positively, and it helped me feel more adapted to the workplace’ ‘I was involved in a very interesting marketing project that can innovate the current marketing strategies of P&G’ ‘My internship at Unibank was excellent. I learned how to assess business plans and communicate with clients. It also let me know much about assessment of applications for micro loans’ Students rated the overall value of YBLP in assisting them in pursuing their career goals an average of 3.6 on a 4-point scale, in which 4 represents ‘extremely valuable,’ a strong endorsement of the program (see Figure 8 below). Figure 8: Student average rating of the overall value of YBLP, by location (1 = “not valuable;” 4 = “extremely valuable”) Student Location Number of Students Average Rating Baku 22 3.7 Ganja 4 3.3 Total 26 3.6 12 Discussions with YBLP company representatives, mentors, students and stakeholders generated numerous narrative accounts of program success: stories not focusing on participants finding work – which was not a program objective – but rather, on the effective performance of the program components, the workshops, and particularly the hands-on internships and mentorships. The internship/mentorship success stories seem to share common qualities: students are treated as full members of the organization, given genuine work responsibilities that benefit the organization and teach the student important work skills, knowledge and attitudes; are allowed an appropriate amount of independence while receiving professional supervision; and receive mentoring support and attention. The successful internships/mentorships generate ‘win-win’ outcomes for the students and host-organizations alike. Presented below are two such narrative accounts. Success Story 1: Seabak – Baku Seabak is a local private company which has been providing IT management and support services to small and medium sized businesses since 2002. Four YBLP students completed their internship program in the Marketing and Sales departments of the company. Why a success? The mentor was a very enthusiastic person who worked closely with the YBLP interns during their internship program. He was happy and motivated by the program and loved working with the interns. The company’s Chief Executive Officer (CEO) also highlighted the importance of the program and expressed the company’s willingness to host interns with diverse backgrounds in the future. The interns were treated as genuine members of the company, and were encouraged to fully use the company’s office and kitchen facilities. During the first day of the internship program, Seabak introduced its office, employees, departments and facilities to the interns and created a friendly professional environment for them. Throughout the program, the interns were treated as members of the ‘family’ and their birthdays were celebrated as were those of Seabak staff. The interns were very happy about the professional atmosphere at Seabak and they believed that working in this context helped them understand the realities and expectations of professional businesses. The interns were given challenging tasks associated with market research and development that were valuable for the company. These tasks served to increase the students’ marketing skills, increased their confidence and determination to successfully pursue their career paths. As a result, the success of interns brought new opportunities and even a new client to the company. The interns remain in contact with their mentor – who has recently moved to a new organization – and call upon him whenever they need professional advice. Success Story 2: Ganja Agrobusiness Association – Ganja Founded in 1999, Ganja Agrobusiness Association (GAA) is a local non-profit organization that promotes sustainable development of the country’s agrarian sector through human potential development, knowledge and resource transfer. One of the YBLP interns completed her internship in supporting the communications and public relations functions of the Association. Why a success? Based on her skill-level, interests and commitment, the YBLP intern was given substantial responsibilities within GAA, including translating documents, editing their magazine ‘Ecology,’ and coordinating and conducting external relations for the organization. She took the lead in developing the magazine’s ‘Organic Farming in the World’ section; researching organic farming in Finland; and translating, editing materials, and writing an article about the lessons learned from Finland’s organic farming experience and its relevance for agriculture in Azerbaijan. The YBLP intern also assisted the GAA in its distribution of materials and general communications, and, as she is fluent in spoken English, helped communicate with 13 foreign partners. The internship placement at GAA was a successful match of student and organization needs, values and interests. The organization was quite pleased with the contributions of the YBLP intern, and the intern was pleased with the responsibilities, guidance and encouragement she was given. The internship enabled the student to practice and advance her professional skills within a context that she found meaningful and important and assisted the organization in expanding service delivery to its members. The GAA mentor stated that the Association would be happy to have more YBLP interns in the future. The intern often visits the organization, and continues to receive professional advice from her mentor. 4.1.2 Conclusions • The overall body of quantitative and qualitative findings from all project stakeholders and role-players concur that YBLP achieved its stated objectives of selecting appropriate student participants, delivering useful, worthwhile professional development workshops, and providing internships and mentorships that all believe will enhance the students’ career paths. • Changes to elements of project design were implemented as JAA deemed necessary to enable them to move forward in accordance with the schedule and deliver the program to its beneficiaries. However, JAA’s changes were generally the results of the time pressure to deliver the benefits of the project to the full complement of students – itself a significant accomplishment in what amounts to a pilot project – and was never done arbitrarily. The project was beneficial and it should be noted that in no instance did the Evaluation Team find a student complaining that they were assigned inappropriate or menial tasks as the basis of their internship, a common complaint in poorly organized internship programs. • JAA conducted all components of the project effectively. Qualitative and quantitative data collected during individual interviews with students, universities and companies, as well as FGDs with students, show that YBLP benefited both students and host organizations. As a tool for career planning and development, the project enabled the participants to gain their first work experience, affirm career interests and increase their business knowledge. The workshops and internship program improved business, communication and leadership skills of students in general, and taught them how to perform in a professional environment. Furthermore, the project increased students’ confidence in pursing their chosen careers. • The student selection process to participate in the YBLP program was fair and efficient. However, more should have been done in selecting students that were fluent in written and spoken English. • There are a few cases in which student interests and ambitions were not matched properly with the internship organization where they were placed. In addition, some of the internships were with NGOs or non-profit associations rather than businesses. • Although, in general, the program was implemented as designed, there were a few cases when internships were not associated with the required mentorship and vice versa. 14 4.2 QUESTION 2: WHAT PROGRESS IN CAREER DEVELOPMENT WAS ACHIEVED BY STUDENTS WHO PARTICIPATED IN YBLP? WERE THERE ANY GENDER DIFFERENCES IN CAREER PROGRESS OR OBSTACLES ENCOUNTERED? 4.2.1 Findings From the perspectives of student participants and key stakeholders, what specific career development steps were achieved by YBLP participants? As noted above, student participants enthusiastically endorsed YBLP for exposing them to the professional work environment, a context of which they previously had little or no direct knowledge and, for which, their theory-laden academic studies had not prepared them. Students know that although the local economy is still expanding, the competition for entry-level professional jobs is stiff and likely favours foreign-educated graduates. Without access to professional networks to facilitate the job search or a means to distinguish themselves, students face a daunting task. Project stakeholders agree that having participated in a selective program such as YBLP that includes actual workplace experience, students will have the means to distinguish themselves and gain a competitive edge. Some students also reported that the internship experience helped confirm their career choice, giving them more confidence as they seek their initial employment. Two quotes convey the flavor of students’ assessments: ‘The project helped me become determined in my career objectives’ (Student) ‘The project helped me become responsible and discover new skills in myself’ (Student) In what ways do YBLP program completers differ from their peers in career development￾relevant knowledge, attitudes, planning and networking? Within the time constraints of the evaluation, there was no opportunity to identify a matched control or comparison group of similarly talented non-YBLP students as would be done in a formal, long-term outcome or summative evaluation. Project mentors, company representatives, and workshop instructors often commented on the traits that seemed to distinguish the program participants: intelligence, academic success, ambition and seriousness of purpose, and a determination to get the most out of the mentoring experience. Two quotes from project mentors convey these assessments: ‘She was hard working, serious, ambitious, very positive about education, and she used to ask many questions’ (Mentor) ‘The program was great, they are the best students, the best of the best’ (Mentor) Clearly, students that completed YBLP gained practical knowledge and experience that further distinguished them from their peers, and several received job offers from distinguished companies, including Baker Hughes, BP, and Procter & Gamble. In addition, YBLP participants mentioned networking as a principal benefit of the program and discussions of career goals as the most frequent topic of conversation with their mentors. Are there gender differences in the experience of participation as students in the YBLP? Are there gender differences in career development achievement and challenges/obstacles encountered? 15 From its inception, YBLP took a particular focus on examining possible gender issues and opportunity differences amongst female and male participants. Despite the challenges of sample size, the Evaluation Team directly and explicitly engaged male and female students, key stakeholders, company representatives, university personnel, mentors and workshop instructors on these issues. Qualitative data revealed essential unanimity that there are no significant barriers to female advancement and success in the professional workplace. Quantitative data also did not reveal any consistent, significant gender differences in confidence or expectations of professional career success. So, for the most part, the evaluation did not find gender differences in career development achievement. Male YBLP graduates face their own gendered challenge in career development: for most, one year of obligatory military service follows immediately upon university graduation. While the male participants appeared quite accepting of this policy, the break in career trajectory and momentum does require additional steps to maintain or reinforce relevant career networks, advisors and access to potential opportunities. 4.2.2 Conclusions • Quantitative and qualitative data collected by the Evaluation Team strongly support the conclusion that YBLP succeeded in enhancing students’ career development along the specific dimensions it was designed to affect. Students acquired useful, employment-relevant skills and knowledge; they were able to gain genuine work-experience within a supportive context that exposed them to the realities, norms and expectations of the professional work environment. In addition, the YBLP experience helped build their confidence and reaffirm career choices, and became an item on their CV that should provide an asset and competitive advantage in their job search. While gender expectations and challenges remain in the rapidly transforming Azerbaijani economy and society, there is strong support for the conclusion that female students benefitted equally from the YBLP interventions and, in general, do not face particular barriers to achieving professional career success. • Qualitative data revealed essential unanimity in the belief that there are no significant obstacles or barriers to female advancement and success in the professional workplace. Such data also did not reveal any consistent, significant gender differences in confidence or expectations of professional career success. Some stakeholder discussions suggested that this may be due to the selective character of the population of female YBLP participants – as university students, they are possibly from families who are particularly supportive of female career achievement, and, even amongst their university peers, are outstanding academically. Despite the overall belief of professional opportunity equality for young women and men, some gender-related issues were mentioned by students and mentors, including the reluctance of some families to allow young women to study abroad or away from their home city, and the possible disruption in the career development path of young men due to compulsory military service. 4.3 QUESTION 3: HOW EFFECTIVE WAS THE PROJECT IN COORDINATING AND COLLABORATING WITH STAKEHOLDERS (USAID, BP, AMCHAM, JAA AND PARTICIPATING COMPANIES)? 4.3.1 Findings What specific activities/mechanisms did the YBLP employ to coordinate and collaborate with key stakeholders? 16 As YBLP implementing partner, JAA maintained regular communications with USAID, delivering quarterly progress reports, annual reports, and other documents as required by their Agreement. AmCham and JAA frequently communicated via telephone and in-person around the identification of companies and other organizations for internships. In general, JAA’s communication with BP was adequate but BP was anxious to receive timely notice of major changes in JAA leadership. JAA also established linkages with participating universities through outreach visits. From the perspectives of YBLP program managers and key stakeholders, how useful and effective were these activities/mechanisms? How can these coordinating mechanisms be broadened, adapted, and institutionalized to promote effectiveness, sustainability, and local ownership? Although seemingly a simple straightforward program, the requirements to quickly communicate and establish working professional relationships with what became 84 mentors/supervisors in 32 different internship organizations, eight universities, six instructors, three other key stakeholders, and more than 120 students, in a 24-month implementation period, generated a significant administrative burden for JAA. This was particularly true at program inception when AmCham initially struggled to find sufficient appropriate internships to match student career goals and, later, as the project moved into Ganja. Under time pressure, particularly during stream one, and responsible to simultaneously conduct student recruitment and selection, reach out to the companies recruited by AmCham, organize the professional development workshops and establish a working PMP system, JAA did a remarkable job of program management and administration but inevitably, given the complexities and time constraints of the YBLP design, some breakdowns in communications did occur. Some mentors were not fully briefed or trained on their roles and responsibilities, beyond the requirement to complete daily narrative reports. In Ganja, students reported being suddenly asked by their professor to hear the YBLP announcement in an auditorium, then immediately to write the screening examination, without really having the opportunity to fully understand the program or consider their decision. An inadvertent error in the listing of stipend payment milestones resulted in one stream of students not receiving their stipend until two months after the completion of their internships, which created significant worry and mistrust on the part of the students. Further, as noted above, as a YBLP funder, BP was alarmed to learn – quite suddenly from their perspective – of the imminent departure of both the Director of the JAA and the Director of the YBLP program itself. Once BP had the opportunity to meet the new JAA leadership they were reassured as to the viability of the program and their participation in it, but had they been informed of possible leadership changes at JAA earlier in the process, there would have been less distress and concern on the part of this key partner. 4.3.2 Conclusions • The rapid start-up, difficulty of initially finding and matching students to appropriate internships, number of participating partners, restrictions of the academic year, multiple overlapping administrative and technical tasks required of JAA over a short implementation time-span, and the involvement of four key stakeholder organizations in a program jointly funded under a public-private partnership model, all contributed to generating communications and operational challenges for the project. Given this level of complexity in what was generally regarded as a rather straightforward program design, it was inevitable that communication lapses and strains would occur. 17 • To its credit, JAA appears to have met all of the USAID reporting requirements under its Agreement, and to have successfully implemented the program in accordance with its schedule and budget. • Stakeholder coordination and information sharing could be strengthened through brief monthly meetings that would review program activities, accomplishments and challenges. In this way, YBLP and AmCham could also benefit from further input from BP and its partners around additional private sector opportunities and resources, and gain insights into the current job market in Azerbaijan. • YBLP’s PMP was fully implemented as planned but proved too cumbersome to provide timely management information. Project reliance on written narrative reports, particularly daily logs prepared by both mentors and students, imposed a huge administrative burden on the JAA and could not offer useful management information during the course of the project itself. • The somewhat divergent perspectives on YBLP’s objectives created some differences about the expected outcomes of the project. 4.4 QUESTION 4: WHAT ARE THE VARIOUS FACTORS THAT HAVE ENHANCED OR LIMITED THE EFFECTIVENESS, RESULTS AND POTENTIAL FOR SUSTAINABILITY OF YBLP IN AZERBAIJAN? 4.4.1 Findings To what extent do YBLP participants, program staff and stakeholders attribute project success to variables related to: a) the professional/career development workshops; b) the internship experience; c) career-related networking, or other exogenous factors? What evidence supports these assessments? Professional Development Workshops As mentioned in Figure 9, below, students rated the effectiveness of the professional development workshops an average of 4.0 (or ‘quite effective’) on a 5-point scale, where 1 was ‘not effective’ and 5 ‘extremely effective.’ While there may be some gender difference, with female students rating the workshops 3.5 and males 4.1, the small number of female respondents makes it impossible to determine if this is a valid difference or a statistical artefact. Scores for respondents in both Baku and Ganja averaged 4 or ‘quite effective.’ Figure 9: Student average rating of the professional development workshops, by gender Student Gender Number of Students Average Rating Male 21 4.1 Female 5 3.5 Total 26 4.0 Student, mentor, and trainer comments on the workshops were consistently positive, and 92% of the students indicated that the workshops increased their confidence in their ability to have a successful career: 18 ‘I have gained valuable knowledge and skills during the workshops’ (Student) ‘I gained knowledge about ethical issues which are important in any workplace’ (Student) ‘Workshops helped me make a right decision in my career’ (Student) ‘Students were very passionate about the workshop, and the course on ethical dilemma especially caused interest of students’ (Trainer) ‘I got lots information about business, management, business ethics and so on which I had not heard before’ (Student) ‘Good program to prepare student for professional life’ (NGO Director) Internships The internship component of the project was highly rated, with an average score for effectiveness of 4.1 on a 5-point scale, in which 1 was ‘not effective’ and 5 ‘extremely effective.’ Disaggregated by gender, the four female respondents again gave somewhat lower, although still positive, ratings, averaging 3.6. Figure 10: Student average rating of internship effectiveness, by gender Student Gender Number of Students Average Rating Male 18 4.2 Female 4 3.6 Total 22 4.1 Disaggregated by location, students in Ganja rated the internships extremely positively – a sentiment echoed in their FGD. Figure 11: Student average rating of internship effectiveness, by location Student Location Number of Students Average Rating Baku 18 4.0 Ganja 4 4.8 Total 22 4.1 The comments below are illustrative of the students’ positive assessment of the internships: ‘The internship definitely affected my confidence positively, and it helped me feel more adapted to the workplace’ ‘I was involved in a very interesting marketing project that can innovate the current marketing strategies of P&G’ 19 ‘My internship at Unibank was excellent. I learned how to assess business plans and communicate with clients. It also let me know much about assessment of applications for micro loans. Mentorships The final key program component, mentorships, provided by 84 mentors from 32 host companies or organizations, were regarded as effective and valuable by students and mentors alike. During site visits, the Evaluation Team found many mentors to be enthusiastic about their role and committed to working with young people. YBLP students gave the mentoring experience an average rating of 4.1 on a scale of 5, where 1 represents ‘not important’ and 5 ‘extremely important’, with males’ ratings averaging 4.3 and females, 3.0 (‘important’) (see Figure 12, below). Given the small sample size, we cannot interpret the meaning of this difference. Qualitative results suggest that, like their male counterparts, female participants found the mentoring experience to be positive. When data is disaggregated by location (Figure 13, below), it shows a possible regional difference, with Baku students rating the importance of mentoring 3.9 (‘quite important’), and the small sample from Ganja, a unanimous score of 5.0 (‘extremely important’). Figure 12: Average Student Rating of the Importance of the Mentoring Experience Participants Number Average Rating Male 21 4.3 Female 5 3.0 Total 26 4.1 Figure 13: Average Student Rating of the Importance of the Mentoring Experience by Location Participants Number Average Rating Baku 22 3.9 Ganja 4 5.0 Total 26 4.1 Students were also requested to rate their degree of ‘comfort’ in talking with their mentor, an indicator of the degree to which the mentoring experience was judged to be open and supportive. As shown in Figure 14, below, students rated this dimension of the mentor/mentee relationship 3.4 on a four-point scale, where 1 represents ‘not comfortable’ and 4 ‘extremely comfortable’. Again female average scores (3.0) while positive, are somewhat lower than those of males (3.5) Disaggregated by location, the average Ganja student ratings (4.0) are higher than those of their Baku peers (3.3) (see Figure 15, below). Figure 14: Student Average Rating of Comfort in Talking to Mentors by Gender Participants Number Average Rating Male 21 3.5 Female 5 3.0 Total 26 3.4 Figure 15: Average Rating of Comfort in Talking to Mentors by Location 20 Participants Number Average Rating Baku 22 3.3 Ganja 4 4.0 Total 26 3.4 The mentors’ ratings of the value of the mentoring process to the students’ career development were also quite positive. On a four-point scale, in which 1 represents ‘not valuable’ and 4 ‘extremely valuable,’ the 21 mentors offered an average rating of 3.5 with essentially no difference in ratings between Baku (3.6) and Ganja (3.5). The following comments by mentors illustrate their positive assessment of this key component of the YBLP: ‘My aim was to make interns independent in a workplace. I trained the interns on leadership and communication skills, and empowered them to become independent decision-makers’ ‘We had excellent relations between me and my mentee.’ ‘She was hard working, serious, ambitious, very positive about education, and she used to ask many questions.’ ‘The program was great, and they are the best students, the best of the best.’ ‘Mentorship is very important for students. It helps students develop their communication skills.’ While the Evaluation Team found broad support for the mentoring component among all of the mentors, several, particularly in Ganja, were concerned that they had not received enough information about the mentoring role and JAA’s expectations (other than completing the daily log) prior to the start of the program: ‘The mentoring process benefits students, but JAA needs to provide more information about the project in advance.’ Although several mentors expressed the opinion that a longer internship period would enable students to develop greater professional skills and gain increased understanding of the realities of the work environment, the great majority supported the three-month internship as ideal. Students also noted that a longer internship period could overlap with their university examination period. What do narrative accounts of project achievement (success stories) suggest as critical variables for enhancing intervention outcomes? Was the intensity/length of the intervention sufficient/optimal to generate the expected results? Two project success stories were presented in Section 4.1.7 (above). Each shows an excellent fit between students’ interests and career ambitions, and the type of organization in which they were placed. In both, the students were given significant professional responsibilities and made contributions that directly assisted the host company/organization in pursuing its mission. In both cases, the host organization provided strong mentoring and guidance, clear direction, gave the students challenging assignments that contributed to the organization, and treated them as members of the team. These accounts suggest that the three-month internship period was 21 sufficient to produce meaningful internship and mentorship experiences that clearly benefitted the students and host organizations. What specific obstacles or challenges appear to have limited project outcomes? What strategies/approaches if any have mitigated these factors? The most significant reported challenges to YBLP outcomes were the result of logistical issues related to difficulties of program implementation under time constraints and at a distance in Ganja. Interviews with key stakeholders, company representatives and mentors suggest that time pressures related to rapid project start-up created difficulties around finding appropriate internships that matched students’ career interests. As mentioned above, faced with difficulties in quickly securing appropriate placements within private sector companies, particularly for stream one, several students were placed in NGOs, including stakeholder organizations JAA and AmCham. Time pressures also resulted in internships in fields quite different from those pursued by some students; an oil geology student being placed at a hotel for example. As noted earlier, under the imperative of time, students were also selected from a variety of professional career paths not traditionally associated with business studies; petroleum engineering for example. This is not to say that at some points in their careers such engineers might not become managers in an oil company or decide to pursue an entrepreneurial opportunity. However, if the goal of the project is to empower future business leaders, a policy decision must be made as to whether participation should be limited to students actively pursuing a business career or opened to a broader array of career fields. A second significant time-related challenge was that of coordinating project implementation with the constraints of the students’ academic calendar. The recruitment and selection process, delivery of the professional development workshops and three-month internships were all timed to coincide with the university schedule and not interfere with the examination period as students preferred internships to conclude prior to exams. As currently designed, the YBLP implementation model is extremely labour-intensive in its student selection, internship-matching and internship/mentoring monitoring requirements, and any additional time for planning and recruitment of students and internships would be helpful. Having successfully administered five YBLP student ‘streams,’ the Evaluation Team expects future implementation to be somewhat less pressurized. The second major logistical challenge that affected project delivery and outcome related to the obstacles posed by implementing the project at a distance in Ganja. Interviews with students, university personnel, and mentors indicate a rather hurried and somewhat confused process of student recruitment and selection, delays in student internship assignment, gaps in timely communication between JAA and students, and some instances in which internship supervisors were not informed of the project’s mentorship responsibilities beyond daily narrative reporting. There is general agreement – among participating students, instructors, mentors and companies/organizations – that YBLP offers students a valuable and relatively scarce career￾related benefit and, as such, is highly desirable. Interview and FGD comments suggest that this was particularly true in Ganja where such interventions are rare in comparison to Baku. But clearly, implementing YBLP in Ganja presented operational challenges that would only be further exacerbated if the model was more broadly rolled out in the regions. 22 What factors outside the control of the YBLP may have facilitated or constrained project success? Did the session in which the student participated affect outcomes or student, company or mentor satisfaction? With the continuing growth and diversification of the Azerbaijani economy, there remains a concomitant need for well-educated, trained, professionally competent personnel to assume positions of technical and management responsibility in international and local enterprises and opportunities for entrepreneurial success. YBLP’s participants report two pervasive beliefs within the business community that they fear may constrain the launch of their careers: a preference for graduates of foreign universities and the desire of companies to employ someone who already has ‘three-to-five years’ experience.’ Unfortunately, employers’ preference for graduates of foreign universities and for experienced personnel is beyond the project’s control. However, although these preferences or biases exist to some degree, it can be expected that the YBLP graduates’ internship and mentorship experience, as well as networking and hands-on experience in the workplace, will offer a genuine advantage as they enter the job market. As reported above, stream one placements were somewhat constrained by the challenge of quickly mobilizing and recruiting initial internship and mentorship opportunities amongst AmCham member organizations, but as the project proceeded, securing appropriate placements – particularly in Baku – became easier. The Evaluation Team does not have qualitative or quantitative data to suggest that program outcomes differed by streams, except to the extent that some streams were concentrated in either Ganja or Baku. Do differences in participating companies (size, sector, culture) appear related to project outcomes? Data collected from interviews and questionnaire does not suggest that particular structural characteristics of a host company were related to outcomes and the Evaluation Team found examples of outstanding success in large and smaller organizations, in international and local companies, and NGOs. One challenge found in larger organizations is that internships are often coordinated and managed by the company’s central human resources (HR) department that identifies internships and mentors. In a few instances, the HR department recruited internship supervisors but these individuals had no direct contact with JAA and were not aware of the mentorship requirements of the project. In working with large organizations that rely on a formal HR department, it is important that JAA coordinate with the implementing partner to ensure that they have direct contact with mentors, particularly in instances where the HR function is in Baku and the internship occurring in the region. Are there any differences within the participating student population (e.g. university, location, gender, selection variables, career interests, social/economic status) that appear related to project outcomes? This evaluation did not identify any characteristic or variable within the student population that appeared consistently or significantly related to program outcomes or success. As discussed in 4.2.3 above, the quantitative data – limited by sample size – did not reveal consistent, systematic gender differences in project satisfaction or achievement, and qualitative input from participants, mentors, instructors, and university or company personnel, overwhelmingly argued against a gender bias effect. This may be related to the YBLP female student sample itself: third and fourth year university students who were selected through a rigorous and highly competitive selection 23 procedure and who may well have come from more liberal family backgrounds that encouraged female academic and professional achievement. Consideration of quantitative data related to location was somewhat constrained by the fact that only 21 of the 105 participants in the first four streams were studying in Ganja and the sample size for individual interviews was small. FGD input however, offers some suggestion that the Ganja students were particularly supportive and enthusiastic of the benefits of the program, finding it to be a unique resource in that part of the country. However, as six of eight Ganja FGD participants were female, gender and location may be conflated in this instance. Are there any areas of divergence of opinion, concerning project outcomes and utility between relevant YBLP constituencies (young people; JAA leadership, key project stakeholders: USAID, BP, AmCham, participating companies, GoA, participating universities, other role-players)? If so, how are these differing assessments to be understood? There were no significant differences of opinion found between key constituencies involved in the program. As a key stakeholder and YBLP co-funder, BP’s particular focus on successful job placement should be considered in planning future projects. What changes to the project design, structure and implementation plan/schedule would be recommended to enhance project outcomes? Do project results suggest elements of “best practice” for similar interventions? What are the key “lessons learned”? As noted above, allowance of greater time for planning and start-up would enhance implementation. This is particularly true in implementation outside of Baku. Closer attention to matching student career interests with internships would likely enhance the value and utility of this experience. The Professional Development Workshops appear to constitute a best practice particularly, in terms of real-world focused, engaging participatory learning in contrast to lecture-based university courses that are focused on theory and are often regarded as quite dated. Lessons learned are presented in Section 6 of this report. What specific mechanisms, structures and procedures would enhance project institutionalization and sustainability? How could private sector and university participation and “ownership” be enhanced? Project sustainability would be further enhanced by including one or more university career centers in any future YBLP model. Strengthening the capacity of university career centers, the structures that should ultimately be delivering programs like YBLP, would reinforce the necessity of transformation of these institutions, and provide potential decentralized sites for the program. As a public/private partnership, YBLP benefited greatly from the involvement of BP as a stakeholder and co-funder. Not only did BP provide financial resources necessary for project implementation but, as a major highly respected member of the Azerbaijani and international business community, its support and engagement offered further legitimacy and leadership for YBLP. Continued involvement of BP and recruitment of other international and local businesses to offer financial resources as well as internships and mentorships, would further institutionalize and 24 sustain the program and ensure the continued availability of such talented, well-educated and trained Azerbaijanis in the economy. Were the Project’s PMP and M&E operations sufficient to provide stakeholders and project managers the information necessary to support effective policy and program operations and planning? What enhancements are indicated? The YBLP PMP was fully implemented as planned but proved too cumbersome to provide timely management information. Project reliance on written narrative reports, particularly daily logs prepared by both mentors and students, imposed a huge administrative burden on the JAA and could not offer useful management information during the course of the project itself. A simplified PMP built on an automated platform with inputs provided through computer, cell phone or other mobile device and which primarily utilizes a limited number of critical rating scales (rather than narrative paragraphs) would generate timely and useful information for future projects. 4.4.2 Conclusions • Key factors that enhanced YBLP’s results and potential for sustainability were the quality of its three main components, the careful match of students’ interests with the organization in which they were placed, hands-on internships during which students were able to make significant contributions, and strong mentoring and guidance provided by experienced businessmen or businesswomen. • Over time, and with each new stream, key factors enhancing project effectiveness were strengthened through the establishment of implementation networks and administrative and technical lessons learned. As additional internships and mentorships were identified and brought into the program, there was an increasing likelihood of more appropriate matching of students’ career interests with the focus of the host organization. Mentors became more experienced in their role so they could identify techniques and approaches to make the mentorships even more effective. As JAA recruited each new stream, they could build upon their base of experience to improve the recruitment and selection process. • Among the factors that might have had an effect on the program’s effectiveness are the logistical issues (especially for implementation in Ganja) as well as the fact that in many large organizations, internships are managed and coordinated by HR departments, which in some cases did not communicate with JAA about the mentorship program under YBLP. • While gender was found to be a less critical factor than believed during initial program design, the operational and technical challenges of operating YBLP in Ganja and the Regions became more important. As a scarce and valued resource, YBLP could easily and productively expand solely within Baku, offering an important resource for students and potential employers. But of course it is in Baku where the bulk of resources and opportunities already exist. The possibilities for increased program operations in Ganja and regional sites – places with far fewer resources and opportunities – offer the potential to reach beneficiaries otherwise reactively marginalized but at a likely increased cost and level of effort. 25 5.0 RECOMMENDATIONS Based on the quantitative and qualitative findings and the conclusions described above, the Evaluation Team has the following recommendations: 5.1 PROGRAM OBJECTIVE/TARGET BENEFICIARIES 1. A decision must be made as to whether a follow-on program will be targeted at entrepreneurial and business skills and career development for undergraduate business students, or if it will be a more general career and professional development program for undergraduate students pursuing diverse professional areas that are not specifically related to business or entrepreneurship. 2. If a follow-on program is focused on the business sector, it should be restricted to student participants who are pursuing business studies and who intend to have careers in business. 3. Thought should be given to whether university students enrolled in universities’ ‘Special Talent Groups’ and similar programs, or who are studying highly in-demand professions such as petroleum engineers – students who may have a relatively easier time in securing employment than their peers – should be included, or if other students will benefit more from the program. 5.2 STUDENT RECRUITMENT, TESTING AND SELECTION 1. Program and recruitment procedures should be publicized broadly at universities well in advance of student testing. 2. Specific targeted fields of study should be clearly articulated prior to recruitment and strictly adhered to. 3. If fluency in spoken English is deemed an essential selection criterion, the program should either ensure that this standard is actually followed in student selection, or initiate a remedial English language improvement course that would be mandatory for those students who have English language reading and writing skills but do not meet the verbal fluency standard. 5.3 PROGRAM ADMINISTRATION AND MANAGEMENT 1. The implementing organization must adhere to the specific design and procedures in the agreement. Any changes or modifications must be discussed with key stakeholders and be formalized in writing with USAID. 2. The PMP data collection and analysis system should be greatly simplified to reduce the amount of unnecessary and largely unhelpful paperwork and, instead, generate timely and useful management information relevant to program administration. An automated data entry and analysis system can be built on a mobile-phone platform (or other device) that entails the weekly input of data through completion of a small number of rating scales assessing key dimensions of project performance rather than requiring daily hand-written inputs into log￾books that remain un-read. 3. The program should strengthen and expand its use of social media to build program identity, enhance student engagement and morale, and facilitate student networking and information￾26 sharing from the moment they are admitted into YBLP and include active outreach and engagement with YBLP graduates. 4. Additional time and personnel resources should be allocated for program implementation outside of Baku. 5. More time for up-front planning and working with universities and companies would enhance the process of student selection and the identification of appropriate internship/mentoring opportunities. It would also likely decrease the time students wait for the start of the internships, possibly reducing the drop-out rate. 6. Students need to be kept up to date as to their program status and the timing of project events. This is particularly true at the start of each stream. 7. The date for delivery of the stipends should be clearly stated in advance of student participation in the program and strictly followed. 8. Monthly project review meetings of key stakeholder organizations would facilitate program communication, coordination and planning. 9. A follow-on program should consider including two university career centers as stakeholders/implementing partners. Participation of career centers would enhance their capacities, increase the potential number of potential beneficiaries, and increase program sustainability by embedding the intervention within institutions of higher education. 10. Efforts should be undertaken to work with the media to expand public awareness of the program’s success and the importance of internships and mentorships for university students. 5.4 PROFESSIONAL DEVELOPMENT WORKSHOPS 1. Professional development workshops should remain a core component of the program. Consideration should be given to including additional modules on public speaking and professional presentations, and on basic computer packages for professionals. 2. Workshop materials translated from English to Azerbaijani should be further reviewed to improve comprehension and utility. 3. Students should be allowed to keep workbooks and other training materials following the professional development workshops. 4. Consistent and timely student evaluation feedback should be provided to the workshop instructors. 5. Instructors should meet as a group with JAA prior to the workshops to discuss and finalize curriculum, exercises and procedures. 27 5.5 INTERNSHIPS AND MENTORSHIPS 1. All YBLP internships should be in an organization that is directly relevant to the students’ field of study and career goals. Business students should receive internships with private sector businesses not with NGOs, CSOs or public/state-owned institutions. 2. All students should participate in both internship and mentorship experiences at the host institution. Student participants should not simply be slotted into an existing internship program if this does not include a mentoring component that meets the YBLP requirements. Students should not be slotted into an existing mentoring program if no actual hands-on internship experience is included. 3. Although recruitment of partner organizations may be initiated through senior management, HR or other departments, YBLP staff must establish contact and meet directly with all potential mentors to explain and secure agreement on program goals and procedures and the specific internship and mentoring requirements. The training/induction of new mentors could be conducted by YBLP staff on-site at participating companies/organizations, or at workshops that include mentors from several participating companies/organizations. 4. Organization should be encouraged to provide students letters of recommendation following completion of successful internships. 5.6 END-OF-PROGRAM ENHANCEMENTS 1. Funds permitting, YBLP should consider developing a final workshop for program graduates, mentors, university liaisons and stakeholders to assess the program, facilitate networking, and present training on topics such as finding the first job, maintaining career momentum for participants entering military service, and small business entrepreneurship. 2. YBLP should remain engaged with program graduates by providing online links to job placement agencies, major employers and post-graduate educational opportunities. 3. Planning and design of any follow-on programs should include input from intended beneficiaries, YBLP graduates, workshop instructors and mentors. 6.0 LESSONS LEARNED 1. In view of the challenging realities of securing entry into the professional workplace in contemporary Azerbaijan, the lack of practical and hands-on training at the university-level and the inability of university career centers to offer effective supplementary programs to assist their graduates, short-term, intensive, career focused programs such as YBLP will remain of importance for at least the next five to ten years. 2. Sufficient time for project start-up in each location in which the program is implemented will facilitate better student recruitment and selection, internships, mentor training and delivery and matching of internship with students’ career interests. 3. Although presenting its own challenges around focus and communications, a public-private model that enlists a major international corporation such as BP in the design, implementation 28 and funding of a program such as YBLP offers extraordinary benefits in terms of relevance to actual workplace concerns and credibility in the business community. 4. Based on YBLP’s experience, key stakeholders should make firm policy decisions around the level of specific business focus for any follow-on programs and adhere to these standards in student selection, internship recruitment, and placement. 29 ANNEXES 30 ANNEX 1: STATEMENT OF WORK 31 SECTION C -DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK FINALPERFORMANCE EVALUATIONOFTHEYOUTHBUSINESS LEADERSHIP PROJECT I. GENERAL This Statement of Work (SOW) is to provide USAID/Azerbaijan with a final performance evaluation of the following Project: Project Title: Youth Business Leadership Project Cooperative Agreement Number: AID-112-A-12-00004 Period of Performance: October1, 2012-September30, 2014 Total Estimated Cost: $200,000.00 Implementing Organization: Junior Achievement Azerbaijan (JAA) II.PURPOSE The purpose of the final performance evaluation of the Youth Business Leadership (YBL) Project is to: 1. Determine the results of YBL Project interventions; 2. Evaluate the progress made by the YBL Project under each component based on established targets; and 3. Provide lessons learned and recommendations from YBL Project’s design and implementation to inform the design of the follow on activity. The evaluation must cover the project implementation period of October1, 2012–May1, 2014; The primary audience for the YBL Project performance evaluation is the USAID/Azerbaijan Mission. Other audiences include USAID/Washington (Bureau for Europe and Eurasia, Bureau for Economic Growth, Education and the Environment(E3), and Bureau for Policy, Planning, and Learning),Junior Achievement Azerbaijan (JAA), BP Exploration (Caspian Sea) Limited(BP), American Chamber of Commerce in Azerbaijan, other USAID implementing partners, businesses that hosted interns, and beneficiaries of YBL Project activities (i.e., the students).The Mission may share the final evaluation report with other donors, other implementing partners, and other United States Government (USG) agencies operating in Azerbaijan. The Mission will use evaluation findings to guide the Mission’s future project designs and implementation 32 III.BACKGROUND The goal of the Youth Business Leadership Project is to provide internships, mentorships, and business skills development training to youth. The primary purpose of the YBL Project is to empower the next generation of business leaders in Azerbaijan by providing undergraduate business students the opportunity to gain real world experience through internships at various private companies while providing them with mentorship opportunities with successful businessmen and businesswomen and with networking opportunities with like-minded peers. The YBL Project objectives are to: -Provide real world employment opportunities for 120 young people; develop their business skills beyond the classroom; and -Provide them with role models who can offer advice and guidance in developing their careers. A. ACTIVITIES The activities, which are listed below, help meet these objectives through the provision of internships, mentorships, and professional development opportunities for 120 undergraduate students throughout the life of the YBL Project. 1. Provide Internship Opportunities for University Students The Recipient works with Azerbaijan-based companies to identify practical and relevant four￾month internships for third and fourth year undergraduate students. Through the internships, aspiring business leaders will acquire practical skills, receive exposure to business practices of the highest standards, and interact with and learn from successful role models. 2. Provide Professional Development Workshops The Recipient conducts two professional development workshops for each group of interns on self- assessment, "16 Success Skills" for finding and being able to keep a job, resume development, business communication, proper interview conduct, project design, conflict resolution, and workplace decision- making. These workshops play an important role in the professional development of students and provide the interns with practical skills needed to compete in the market and in still confidence in them. 3. Provide Mentorship Opportunities to Participating Interns A successful mentoring relationship requires the commitment from both the mentor from the hosting company and the intern. Both need to openly share their expectations at the outset with each other, commit to meeting regularly over the four month period, and come prepared to each meeting with questions, advice, and topics of discussion. 33 B. THE PROJECT’S RECENT ACHIEVEMENTS-PROGRESS TO-DATE Junior Achievement Azerbaijan prepared and issued the project information brochure with Azerbaijani and English contents. Also, JAA prepared and printed out application forms for students and compiled tests that will be used during selection. Application form includes tests on English language, logics, economics and projected personal action plan for post-program activity. JAA developed a set of grading guidelines to justify the student selection procedures. JAA has revised training materials including: Business Ethics, Work-readiness, and Career with a Purpose, Personal Finance and Job Shadow Program. Each program component contains a workbook for student, pre-and post-program tests, and supplementary materials that will help the participants to better acquire new skills and knowledge. The Youth Business Leadership (YBL) Project involved students from the following universities: Azerbaijan State Economic University (ASEU), Baku State University (BSU), Azerbaijan State Oil Academy, Qafqaz University, Khazar University, Ganja State University, Azerbaijan State Agrarian University, and Azerbaijan Diplomatic Academy. Junior Achievement Azerbaijan has conducted trainings on Success Skills, Business Ethics, Career with a Purpose and Personal Finance and Job Shadow Induction Course for 99 participants of1st, 2nd, 3rd, and 4th semester. IV. EVALUATIONQUESTIONS The following questions are provided as a guide in formulating findings and recommendations of this evaluation. 1. To what extent did YBL Project achieve its stated project objectives? 2. What progress in career development was achieved by students that participated in YBL Project? Were there any particular obstacles encountered by the male vs. the female students? 3. How effective was the project in coordinating and collaborating with stakeholders (USAID, BP Exploration (Caspian Sea) Limited (BP), American Chamber of Commerce (AMCHAM) in Azerbaijan, and participating companies)? . To what extent did YBL Project achieve its stated project objectives? 2. What progress in career development was achieved by students that participated in YBL Project? Were there any particular obstacles encountered by the male vs. the female students? 3. How effective was the project in coordinating and collaborating with stakeholders (USAID, BP Exploration (Caspian Sea) Limited (BP), American Chamber of Commerce (AMCHAM) in Azerbaijan, and participating companies)? 34 4. What are the various factors that have enhanced or limited the effectiveness and results of YBL Project in Azerbaijan? 35 V. EVALUATION METHODOLOGY USAID recommends using both quantitative and qualitative methods (e.g., data, observations and anecdotes, participant surveys) in assessing progress, results and impacts. The Contractor must review the Cooperative Agreement, annual Work Plans, quarterly performance and financial reports, baseline data on established indicators, results/outcomes and annual targets, quality and relevance of data collected to track and monitor progress and accomplishments. The Mission recommends that the evaluators use rapid appraisal methods, face-to-face interviews with key informants, focus groups, group discussions, and community discussions as appropriate to the question to the issues being evaluated and also to gauge satisfaction by the government and key stakeholders about project performance. The Contractor is encouraged to employ the methods as long as they do not add to the duration or cost of the evaluation. The Contractor must ensure that all relevant issues are covered in the evaluation in instruments that may be used e.g., questionnaires and or list of topics. VI. TASKS A. Review of Key Documents: The Contractor must review key documents to develop a Work Plan prior to any field work. All available documentation describing YBL Project activities carried out in Azerbaijan must be reviewed. Documents for review include but are not limited to those listed in the reference section. The Contractor must contact the designated Contracting Officer’s Representative (COR) for YBL Project evaluation for access to relevant documents. B. In-briefing: Upon arrival in Azerbaijan, the Evaluation Team must provide an entrance briefing to the designated USAID officials, introduce the team, discuss logistics and scheduling, discuss submission of the Work Plan, and any other issues. USAID will assist with identification of the relevant stakeholders to meet with and provide additional suggestions for interviews. C. Work Plan: The Work Plan must be in accordance with the USAID prepared timeline for all work to be concluded and the dates for submission of the draft and final reports. The Work Plan must include the following elements: i. Schedule of contacts and site visits (regions, beneficiaries and collaborators); ii. Arrangements for local logistics; iii. Schedule of briefings and submission of deliverables; and 36 iv. Delineate the roles and responsibilities of the other members of the Evaluation Team to ensure coverage of all elements of the Statement of Work. USAID will provide comments within two days. D. Evaluation Design: The Contractor must prepare and submit to USAID for approval a final evaluation plan and schedule with the following major elements: Schedule, methodology for conducting the evaluation (data and information collection, field interviews, participant surveys), beneficiary groups to be contacted and regions to be visited. The Contractor must ensure that its findings and conclusions about the effectiveness of the YBL Project activities are based on available data that is both accurate and reliable, and that information gathered is representative of and reasonably reflects results actually achieved. The Contractor must submit a final detailed evaluation design, which must consist of the following: i. List of topics and relevant questions, methods and data sources for data gathering; ii. A matrix of regions and beneficiaries to be contacted; iii. Data analyses for each question and presentation plan; iv. Data collection instruments and v. Limitations of the evaluation design, if any. The Evaluation Team must share the evaluation design with the Implementing Partner for comment, but, in the interest of objectivity and independence, the Implementing Partner will not participate in the design, implementation, analysis, or presentation of the evaluation. E. Field Work: The Contractor must begin field work after finalization of the Work Plan and Evaluation Design and its approval by USAID. F. In-country USAID Debrief: The Contractor must provide an oral debriefing to USAID upon completion of the evaluation and prior to departing from Azerbaijan. Evaluation findings must include facts, evidence and data. Recommendations must be specific, concise and supported by evidence. Recommendations must be action- oriented and implementable. G. In-country Stakeholders/Implementing Partner Debriefs: The Contractor must provide debriefings to: 1) BP Exploration (Caspian Sea) Limited (BP) and American Chamber of Commerce (AMCHAM);and 2)Junior Achievement Azerbaijan. 37 H. Evaluation Report: i. The Contractor must provide with a detailed Outline of the Evaluation Report, main findings and recommendations to USAID prior to departing Azerbaijan. ii. The Contractor must submit a draft report of its findings within five working days after departing Azerbaijan for review and comments. The first draft must address comments and recommendations made by USAID and stakeholders during the out-briefing. iii. The Contractor must submit the final evaluation report to USAID five days after receipt of comments. The length of the final report is not predetermined, but the report must be concise, well written, and comprehensive. Recommendations must be action-oriented, practical, and specific; define responsibilities and timelines for the action; and identify mile stones and deliverables. Unresolved issues that highlight what remains to be done must also be included in the final report. The final report format must be presented in Microsoft Word and use 12-point type font throughout the body of the report, using page margins 1”top/bottom and left/right. The body of the report must not exceed 25 pages, excluding the executive summary, table of contents, references and annexes. The final report must follow USAID branding and marking requirements. The final report must include an executive summary, introduction, the development context and the background of the project being evaluated, evaluation questions, and explanation of evaluation methodology, the limitations of the evaluation, findings, conclusions and lessons learned, and recommendations for the sustainability of YBL Project activities. The executive summary must summarize the purpose, background of the project being evaluated, evaluation questions, evaluation methodology, major findings, lessons learned, conclusions, and recommendations. The evaluation methodology must be explained in the report in detail. Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) The annex to the report must include: 1. The Evaluation Statement of Work 2. Schedule of Evaluation 3. Evaluation design/methodology employed questionnaire and list of questions by topic etc. 38 4. Names and contact information of key respondents, sites visited and other sources of Sources of information, properly identified and listed 5. Information statements as appropriate regarding significant unresolved issues, difference of opinions (among members of the evaluation team, the Implementing Partner, BP Exploration (Caspian Sea) Limited (BP), American Chamber of Commerce (AMCHAM), and other relevant stakeholders) and availability of data and its quality 6. The Evaluation Design Disclosure of conflicts of interest forms of all evaluation team members, either attesting to a lack of conflict of interest or describing any existing conflicts of interest. VI.REPORTING RELATIONSHIP USAID will provide overall direction to the Evaluation Team, identify key documents. Primary contacts for the Contractor will be the designated COR for the Award. The Agreement Officer’s Representative (AOR) of the YBL Project and Economic Growth (EG) Office Director for other technical issues. VII.REPORTING REQUIREMENTS TO ENSURE THEQUALTIY OFTHE EVALUATION REPORT Per the USAID evaluation policy, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report. 1. The evaluation report should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate what worked in the project, what did not, and why. 2. Evaluation reports must address all evaluation questions included in the Statement of Work. 3. The evaluation report must include the Statement of Work as an annex. 4. The evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists, and discussion guides must be included in an Annex in the final report. 5. The evaluation findings must ensure that, where relevant, data analysis takes gender into consideration and must be sex disaggregated. 6. Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.). 39 7. The evaluation findings must be presented as analyzed facts, evidence, and data. Findings must be specific, concise and supported by strong evidence. 8. Sources of information need to be properly identified and listed in an annex. 9. Recommendations need to be supported by a specific set of endings. 10. Recommendations must be action-oriented, practical, and specific, with defined responsibility and timelines for the action. - END OF SECTION C – 40 ANNEX 2: EVALUATION SCHEDULE 41 ◄April 2014 ~ May - August 2014 ~ May 2014 ► Sun Mon Tue Wed Thu Fri Sat May 11 12 13 Award Made 14 15 Conference call w/USAID 16 17 18 19 Visa Issued/Pre￾Design Activities 20 Visa Issued/Pre￾Design Activities 21 Visa Issued/Pre-Design Activities 22 Visa Issued/Pre￾Design Activities 23 Visa Issued/Pre￾Design Activities 24 25 26 Visa Issued/Pre￾Design Activities Review Materials Home US: Memorial Day 27 Visa Issued/Pre￾Design Activities Review Materials Home 28 Visa Issued/Pre-Design Activities Review Materials Home Az:Republic Day 29 Visa Issued/Pre￾Design Activities Work Plan/Evaluation Design Preparation Home 30 Visa Issued/Pre￾Design Activities Work Plan/Evaluation Design Preparation Home 31 June 1 2 Submit Work Plan and Evaluation Design Home 3 Travel to Baku 4 Travel to Baku 5 In-Briefing w/USAID & USAID Comments on Work Plan and Evaluation Design Baku 6 Finalize Data Collection Instruments Baku 7 Submit Final Work Plan & EvalDesign Finalize Data Collection Instruments Baku 8 Baku 9 Interviews with AOR Baku 10 Key Informant Interviews Baku 11 Key Informant Interviews Baku 12 Meetings & Interviews with Stakeholders Baku 13 Meetings & Interviews with Stakeholders Baku 14 Meetings & Interviews with Stakeholders Baku 15 Travel to Ganja Az: Day of National Salvation 16 Meetings & Interviews with Stakeholders Ganja Az: Day of Nat. Salvation 17 Meetings & Interviews with Stakeholders Ganja 18 Meetings & Interviews with Stakeholders Ganja 19 Meetings & Interviews with Stakeholders Ganja 20 Meetings & Interviews with Stakeholders Ganja 21 Meetings & Interviews with Stakeholders Ganja Az: June Solstice 22 Travel to Baku 23 Analyze Data Baku 24 Analyze Data & Finalize Presentation Baku 25 Submit Presentation to USAID Baku 26 De-briefing w/USAID Baku Az: Arm. ForcesDay 27 De-briefing w/BP, AMCHAM and JAA Baku 28 Depart Azerbaijan 29 30 Write Draft Report Home July 1 Write Draft Report Home 2 Write Draft Report Home 3 Write Draft Report Home 4 Write Draft Report Home US: 4th of July 5 42 ◄April 2014 ~ May - August 2014 ~ May 2014 ► Sun Mon Tue Wed Thu Fri Sat 6 7 8 Submit Draft Report Home 9 10 11 12 13 14 15 Az: Ramadan 16 Az: Ramadan 17 18 19 20 21 22 Receive Comments Home 23 Integrate Comments Home 24 Integrate Comments Home 25 Integrate Comments Home 26 27 28 Integrate Comments Home 29 Submit Final Report Home 30 31 August 1 2 43 ANNEX 3: EVALUATION DESIGN MATRIX 44 YBLP Evaluation Matrix No. Evaluation Questions and Sub-Questions Data Source Methodology Data Analysis Question 1: Was the YBLP implemented as designed in accordance with the SOW and project schedule? 1.1 How man third and fourth-year undergraduate business students from the 8 identified Azerbaijani universities participated in YBLP? What was the gender breakdown? • Quarterly reports • Project records • Document review • Enumeration/quantitative analysis 1.2 Was the recruitment and selection process for YBLP participants fair, transparent and technically sound? • Staff interviews • Participant individual/group interviews • Project records/reports • Document review • Interview scale item ratings • Content analysis • Average scores, disaggregated by sub-population (compare mean rating scale-scores for sub￾populations of participants, for example by gender, or other variables for participants and stakeholders) • Qualitative content analysis 1.3 Through partnerships with Azerbaijani-based companies did the YBLP provide relevant four￾month internships for participating students? • Project records • Document review • Enumeration/quantitative analysis 1.4 Did each group of interns participate in two relevant professional development workshops? What were the topics/business-related skills presented in the workshops? • Project records • Document review • Enumeration/quantitative analysis 1.5 Were YBLP participants provided with structured, mentoring opportunities within the companies in which they interned? • Project documents • Staff interviews • Participant interviews • Company interviews • Document review • Content analysis • Enumeration/quantitative analysis • Qualitative content analysis 1.6 Were any significant changes in project activities, expected outcomes, products or deliverables instituted during the period of performance? • Project documents • Staff interviews • USAID interviews • Document review • Listing of any changes Question 2: To what extent did YBLP achieve its stated objectives? 2.1 Was the YBLP successful in providing “real-world employment opportunities for 120 young People? • Project documents • Staff interviews • Participant interviews • Document review • Interview ratings • Enumeration/quantitative analysis • Average ratings by interview populations (compare mean rating 45 • Company interviews scale-scores populations of participants) 2.2 Did YBLP activities enable participants to develop/improve their business skills beyond the classroom? • Staff interviews • Participant individual/group interviews • Company interviews • University interviews • Content analysis • Interview ratings • Illustrative narratives • Average ratings by interview populations (compare mean rating scale-scores populations of participants) • Narrative content analysis 2.3 Did the YBLP link participants with role￾models/mentors who provided useful advice and career guidance? • Project documents • Participant individual/group interviews • Mentor interviews/focus groups • University interviews • Content analysis • Interview ratings • Illustrative narratives • Average ratings by interview populations • Narrative content analysis Question 3: What progress in career development was achieved by students who participated in YBLP? Were there any gender differences in career progress or obstacles encountered? 3.1 From the perspectives of student participants and key stakeholders, what specific career development steps were achieved by YBLP participants? • Participant interviews • Project staff interviews • Company interviews • Other stakeholder interviews/focus group (AmCham, BP, USAID, university personnel) • Content analysis • Interview ratings • Illustrative narratives • Enumeration/quantitative analysis • Narratives content analysis 3.2 In what ways do YBLP program completers differ from their peers in career development-relevant knowledge, attitudes, planning and networking? • Participant individual and group interviews • Stakeholders’ interviews/focus group • Possible interviews with small control sample of non￾participating business students • Interview scale ratings • Interview content analysis • Illustrative narratives • Enumeration/quantitative analysis • Average ratings differences • Qualitative analysis • Short illustrative narratives 3.3 Are there gender differences in the experience of participation as students in the YBLP? Are there gender differences in career development achievement and challenges/obstacles encountered? • Participant interviews/gender focus group • Staff interviews • Company interviews • Stakeholders’ interviews/focus groups • Scale ratings • Content analyses • Illustrative narratives • Enumeration/quantitative analysis • Rating scale differences by gender • Qualitative analyses • Illustrative narratives 46 Question 4: How effective was the project in coordinating and collaborating with stakeholders (USAID, BP, AmCham in Azerbaijan, and participating companies)? 4.1 What specific activities/mechanisms did the YBLP employ to coordinate and collaborate with key stakeholders? • Project documents/records • Staff interviews • Stakeholders’ interviews/focus group • Records, minutes, reports review/analysis • Interview content analysis • Enumeration/quantitative analysis • Illustrative narratives 4.2 From the perspectives of YBLP program managers and key stakeholders, how useful and effective were these activities/mechanisms? • Staff Interviews • Stakeholders’ interviews and focus group • Interview ratings • Focus group content analysis • Average ratings by staff and stakeholders Question 5: What are the various factors that have enhanced or limited the effectiveness and results of YBLP in Azerbaijan? 5.1 To what extent do YLDP participants, program staff and stakeholders attribute project success to variables related to (a) the professional/career development workshops, (b) the internship experience, (c) career-related networking, or other exogenous factors? What evidence supports these assessments? • Participants interviews; questionnaire scale ratings • Stakeholders’ focus group/interviews • Staff interviews • Interview content analysis • Participant questionnaire scale ratings • Stakeholders’ focus group and interview content analysis • Staff interview content analysis • Comparison of average scale ratings for program components • Qualitative analyses of participants, staff and stakeholders interviews • Illustrative narratives 5.2 What do narrative accounts of project achievement (success stories) suggest as critical variables for enhancing intervention outcomes? Was the intensity/length of the intervention sufficient/optimal to generate the expected results? • Participants, staff and stakeholders interviews and focus groups • Content analyses • Narratives analysis 5.3 What specific obstacles or challenges appear to have limited project outcomes? What strategies/approaches if any have mitigated these factors? • Key stakeholders (USAID, AmCham, BP); participants, companies and other (university/expert input) interviews/focus groups • Content analyses • Comparison of attribution by various role players 5.4 What factors outside the control of the YBLP may have facilitated or constrained project success? • Project documents/reports • Staff interviews • Stakeholders’ focus group • USAID interviews • Content analysis • Document review • Narrative analysis 47 5.6 Do differences in participating companies (size, sector, culture) appear related to project outcomes? • Staff and participants’ interviews • Company interviews • Stakeholders’ focus group/interviews • Content analysis • Rating scale comparisons • Narrative/discourse analysis • Comparison of average scale ratings by company characteristics 5.7 Are there any differences within the participating student population (e.g. university, location, gender, selection variables, career interests, social/economic status) that appear related to project outcomes? • Project documents • Participant interviews • Participant focus groups • Participant telephonic/e￾interviews • Company interviews • Document analysis • Individual and group interview data analysis • Cross-tabs/disaggregation by population variables 5.8 Are there any areas of divergence of opinion, concerning project outcomes and utility between relevant YBLP constituencies (young people; JAA leadership, key project stakeholders: USAID, BP, AmCham, participating companies, GoA, participating universities, other role-players)? If so, how are these differing assessments to be understood? • Participants individual and group interviews • Company interviews/focus groups • Staff interviews • Key stakeholder interviews • Content analysis • Cross-tabs/disaggregation by role-players • Illustrative narratives 5.9 What changes to the project design, structure and implementation plan/schedule would be recommended to enhance project outcomes? Do project results suggest elements of “best practice” for similar interventions? • Role-player recommendations from initial individual/group interviews and evaluation workshop • Content analysis • Listing of recommendations • Recommendations/best practices appraisals from Evaluation Workshop. 5.10 Is there any evidence relevant to estimates of project sustainability? • Role-player individual and group interviews • Scale ratings • Content Analysis • Scale data disaggregated by population • Illustrative comments 48 ANNEX 4: INTERVIEW GUIDES / DATA COLLECTION INSTRUMENTS 49 YBLP Final Performance Evaluation Key Stakeholders’ Individual Interview Guide Stakeholder Name:________________________________ Organization: _____________________________________ Title: ____________________________________________ E-Mail: ___________________________________________ Phone #:__________________________________________ Date: _____________________________________________ Introduction: The Youth Business Leadership Project (YBLP) has been implemented through the combined leadership, expertise and resources of four key stakeholders: the United States Agency for International Development (USAID); BP Exploration (Caspian Sea) Limited (BP); the American Chamber of Commerce Azerbaijan (AmCham) and Junior Achievement Azerbaijan (JAA). As a representative an YBLP key stakeholder organization, we are interested in learning your perspectives, ideas and assessment on the project’s approach and roll-out, implementation, results and implications for further such ventures. Questionnaire Items: 1. Would you please briefly describe your organization’s role in the creation and Implementation of the YBLP? 2. What has been your personal role in the organization and implementation of the YBLP? 3. Why do you think that (name of organization) became a key partner in the YBLP? What is it about the project that inspired (name of organization) to take on such a central role? 50 4. The YBLP is a public/private partnership requiring a high degree of cooperation between the four key stakeholder organizations and with the numerous businesses that are providing the project’s internships and mentorships. How well do you think this process of coordination has gone? From your perspective, what have been its achievements and successes? Are there ways in which collaboration and coordination between the partners could be improved? 5. Overall, how would you describe the job prospects facing current graduating students seeking to pursue a business career in Azerbaijan? What do you see as the major opportunities and challenges that these students face? 6. In what ways does this situation differ between male and female business career students? 7. What particular features of the YBLP program design do you think have worked best in assisting the students in advancing into the business workplace? In the long run, what program components will prove the most valuable? 8. Have you been involved in any of the YBLP internships and mentoring at your organization? If so, how do you assess its value and effectiveness for the students and for your organization? 51 9. If there was one change they you could make to improve the project, to make it more successful and effective, what would that be? 10. What one thing about the YBLP experience did you find personally most interesting, surprising or important? 11. Do you think that there would be value in continuing a youth business leadership development program such as YBLP over the next three-five years? Would (name of organization) likely be interested in participating in such a program? 12. Is there anything else about the YBLP that we haven’t covered that you would like us to know? Conclude: On behalf of the Evaluation Team I would like to thanks you for taking the time to help us better understand the YBLP. Later in the process we will be providing a de-briefing to your organization that will present all of our key findings, analyses and recommendations. In the meantime, if you have further ideas, comments or suggestions please contact me at: yblpeval@gmail.com or at (local cell phone number). 52 YBLP Final Performance Evaluation Mentors’ Individual Interview Guide Mentor Name: __________________________________ Mentor Gender: _________________________________ Business Name: _________________________________ Title: __________________________________________ E-Mail: _________________________________________ Phone #:________________________________________ Date: _________________________ Introduction: One off the primary features of the Youth Business Development Project (YBLP) is its use of mentorships for participating undergraduate business students. As a mentor you have had the opportunity to work closely with one or more YBLP students who have been interning with your business or organization. Your views, ideas and suggestions on this aspect of the project are essential to the YBLP Evaluation and we greatly appreciate you taking the time to talk with us about them. 1. First, can you please tell us briefly about your company: what does it do; what is its size; how many people work at the office in which the YBLP internship was conducted? 2. What is your role or position in the company? 3. How did your company become aware of the YBLP? Why do you think (name of company) decided to participate? 4. How was it decided that you would become an YBLP Mentor for your company? How did you feel about this at that time? 53 5. Did the YBLP or your company provide ay training or other preparation for you prior to the start of the mentoring process? If so, please describe? 6. Do you know anything about the process of selecting a particular student for whom you would mentor? Were you part of that process? 7. Please tell us about your initial meeting with the YBLP student (s) whom you mentored. 8. What were your goals for the mentorship (s)? What did you want it to achieve? 9. How often did you meet with the intern? Did you have specific agenda or items to be covered at each particular meeting? 10. What were the three most important topics or items that you discussed with the YBLP intern whom you were mentoring? 11. Did you and the intern confront any particular challenges or difficulties in conducting the mentor-mentee relationship? If so, how did you deal with them? 12. How effective do you think the mentoring process was in assisting the intern in pursuing a business career? 54 13. If there was one change that you could make to improve the YBLP mentoring process what would that be? 14. Do you think that (name of company) would be willing to participate in further student internship programs? Would you like to continue being a mentor? Please explain. 15. Do you anticipate maintaining any contact with the intern following the conclusion of the mentorship? 55 Youth Business Leadership Project (YBLP) Evaluation Dear YBLP Mentor: We are writing to request your help in conducting an evaluation of the Youth Business Leadership Project (YBLP) under the sponsorship of the United States Agency for International Development (USAID), in conjunction with Junior Achievement Azerbaijan (JAA), the American Chamber of Commerce (AmCham) and BP Exploration (Caspian Sea) Limited (BP). The aim of the evaluation is to document the project’s strengths and successes, identify any areas for improvement and formulate key lessons learned. As an YBLP mentor, your views and experience in the project are particularly valuable and important to the evaluation. Unfortunately we do not have sufficient time to interview each YBLP mentor in person, but hope that you will take a few minutes to complete the brief questionnaire below and return it to us. It should take you about 15 minutes. Thank You James M. Statman, Ph.D. Evaluation Team Leader Mentor E Questionnaire Mentor Name:__________________________________ Business Name: _________________________________ Title: __________________________________________ 1. Prior to the YBLP has your company utilized undergraduate student interns? Yes______ No______ 2. Do you have prior experience in supervising or mentoring students within your business? Yes______ No______ 3. Do you receive any training prior to assuming your role as an YBLP mentor? No__________ Yes_________ (please describe below): 4. Was there any effort to match you with a particular student with whom you would be compatible? Do not know_________ No________ Yes_________ (if so, on what dimensions – check all that apply): (a) Gender____ (b) Business interest or specialty______ 56 (c) Other:________________________________________________ ______ 5. Did you meet with your mentee on a regular or as needed basis? As Needed_______ Regular Basis______________ 6. How frequently did you meet over the four month internship period? (a) Never met________ (b) Met once_________ (c) Met 2-4 times______ (d) Met 5 times or more______ 7. Do you think that a four month internship is: (a) Too short a period to actually benefit the student__________ (b) About the right length of time___________ (c) More time than is necessary______________ 8. What topics did you discuss with your mentee (check all that apply): (a) His/her career goals______ (b) Steps to pursue in achieving these goals_____ (c) Obstacles or challenges he/she may face______) (d) Work culture, business norms, dress and standards______ (e) Specific work challenges in the YBLP internship________ (f) Gender issues_______ (g) Business ethics and problems of corruption______ (h) Networking and business success________ (i) Additional formal education or training_________ (j) Self confidence________ (k) Other(s), Please describe:__________________________ 9. In a mentor, how often did you: (a) Provide specific career guidance or advice? 1. Frequently______2.Once or twice_______3. Never ________ (b) Teach a particular skill or technique? 1. Frequently______2.Once or twice_______3. Never ________ (c) Help the mentee deal with a gender-related problem? 1. Frequently______2.Once or twice________3. Never ________ (d) Help the mentee address a work-related issue? 1. Frequently_____2. Once or twice________3. Never________ (e) Help the mentee with issues of networking and job search? 57 1. Frequently______2. Once or twice________3. Never________ (f) Discuss work culture and/or business norms? 1. Frequently______2. Once or twice________3. Never________ (g) Provide advice on a personal matter affecting the mentee? 1. Frequently_______2. Once or twice_______3. Never_______ (h) Discuss business ethics or corruption? 1. Frequently________2. Once or twice_______3. Never_______ 10. Overall, how valuable do you think the mentoring process has been in assisting the students in advancing their business careers? (a) Extremely valuable______ (b) Somewhat valuable_______ (c) Not very valuable_____ (d) Of no value______ 11. Having served as an YBLP mentor, what do you believe to be the benefits for the students in participating in such a program (check all that apply): (a) Provides the opportunity to learn new business skills________ (b) Provides the opportunity to apply and practice business skills in a real-life setting____ (c) Gives the student a chance to display my abilities and competencies______ (d) Provides to the student the opportunity to work with others in a real business setting_______ (e) Helps in building a network of contacts in the business community_______ (f) Helps the students affirm their decision to pursue a business career________ (g) Increases student confidence in succeeding in a business career________ (h) Provides students with a better understanding of business culture_____ (i) Provides an experience that increases a student’s employability in the business sector_____ (j) Other(s) – please describe: 12. Do you anticipate remaining in contact with your mentee following the conclusion of the internship period? (a) No_____ (b) Possibly_______ (c) Most likely______ (d) Definitely______ Thank you so much for completing this questionnaire. If you have any additional thoughts to share or questions about the evaluation, please contact us at yblpeval@gmail.com. If you wish we would also be pleased to send you a summary of the evaluation findings and recommendations. 58 Youth Business Leadership Project (YBLP) – Final Performance Evaluation Professional Development Workshop Instructor Individual/Group Interview Guide Introduction: We are conducting an evaluation of the Youth Business Leadership Project (YBLP) under the sponsorship of the United States Agency for International Development (USAID), in conjunction with Junior Achievement Azerbaijan (JAA), the American Chamber of Commerce (AmCham) and BP Exploration (Caspian Sea) Limited (BP). The aim of the evaluation is to document the project’s strengths and successes, identify any areas for improvement and formulate key lessons learned. As an instructor for one or more of the JAA Professional Development Workshops your views and experience in the project are important to the evaluation and we appreciate your willingness to discuss them with us today. 1. Can you please tell us for which YBLP Workshops you have served as an Instructor: 2. Have you served as an Instructor or facilitator for other similar workshops prior to YBLP? 3. What were the principal objectives of the workshops; what were they designed to accomplish? 4. Please briefly describe the process of design and development of the workshop agenda and curriculum: 5. What role did JAA play in the workshop design and development process? 6. In your experience, what are the key gaps in practical knowledge and practice with which undergraduate business students in Azerbaijan must content? 59 7. In what ways does gender play a role in business career development for undergraduate students and new entrants into the business job market? 8. How do the YBLP students generally compare to their university peers? Do they seem to differ in any way? 9. What is your assessment of the Professional Development Workshop(s) for which you served as an Instructor? 10. To what extent can relatively short practically-focused Professional Development Workshops contribute to the career development of undergraduate business students? What in your view are their contribution and limitations? 11. How would you assess the YBLP model as a whole, including the Professional Development Workshops, internships and mentorships? Do you think that it constitutes a viable approach? In what ways can it be further strengthened? 12. As you think about your work with the YBLP students, is there one particular young person whose story you found particularly interesting, instructive or compelling? Closing: Thanks you again for participating in the YBLP Evaluation. Your insights and suggestions will certainly help inform our work and strengthen such programs in the future. Should you have further input or feedback to provide or wish to receive a summary of the evaluation report, please contact us at yblpeval@gmail.com. 60 Youth Business Leadership Project (YBLP) Performance Evaluation Dear YBLP Participant: We are writing to request your help in conducting an evaluation of the Youth Business Leadership Project (YBLP) under the sponsorship of the United States Agency for International Development (USAID), in conjunction with Junior Achievement Azerbaijan (JAA), the American Chamber of Commerce (AmCham) and BP Exploration (Caspian Sea) Limited (BP). The aim of the evaluation is to document the project’s strengths and successes, identify any areas for improvement and formulate key lessons learned. As an YBLP participant, your views and experience in the project can make a valuable contribution and we hope that you will take a few minutes to complete the brief questionnaire below and return it to us. It should take you about 15 minutes. Your participation in the evaluation is entirely voluntary and there will no consequences for you if you decline. If you do participate, please note that your name will not appear in any evaluation report or presentation nor will any quote be attributed to you as an individual. Thank You. James M. Statman, Ph.D. Evaluation Team Leader Student E Questionnaire 1. (a) What university do/did you attend?:________________________________and (b) what year of study are you pursuing?__________ 2. What is your major area of study?______________________________________ 3. How did you first learn about the YBLP? (i). Announcement by a professor or other university personnel________ (ii) Poster/flyer______________ (iii) On line notice____________ (iv) Through participation in Junior Achievement (JAA)____________ (v) Through a fellow student or friend:______________ (vi) Other (please explain)_________________________________________________________ 4. Did you participate in any YBLP professional development workshops? (i) No______ Yes______ 61 5. Using a five-point scale in which 1 is “Not Effective” and 5, “Extremely Effective”, how would you rate the overall effectiveness of the workshops in providing business skills and information that will help you in pursuing your career? (1)Not Effective___(2) Somewhat Effective___(3)Effective____(4)Quite Effective _____(5)Extremely Effective_____ 6. Do you think that the workshops increased your confidence that you have a successful business career? No_____ Yes_________ Please comment: 7. Did you participate in an YBLP internship? (i) No____ Yes______ (ii) In what type of business (sector) did you intern?_____________ (iii)What is its size: (a) local small or medium enterprise (up to 100 employees); (b) local large business (100+ employees); (3) large multinational___________. (iv)How many hours/week did you work on the internship________ (v) Did you receive any training or instruction from the company as part of the internship? No_____ Yes______ If yes, please describe: (vi)Did you work?: (a) primarily on your own_______ (b) primarily as part of a work team__________ (c) a mixture of individual and group work_____ (vii) Were there other interns also working at the company while you were there? Yes___ No___ (viii) Has this company utilized interns prior to its participation in the YBLP? Yes____ No____ Do Not Know_____ (ix) In what ways, if any, did the internship experience enhance your business career development (check all that apply): (k) Gave me the opportunity to learn new business skills________ (l) Gave me the opportunity to apply and practice business skills in a real-life setting____ (m)Gave me a chance to display my abilities and competencies______ (n) Gave me the opportunity to work with others in a real business setting_______ (o) Helped me in building a network of contacts in the business community_______ (p) Helped me affirm my decision to pursue a business career________ (q) Increased my confidence in succeeding in a business career________ (r) Provided me with a better understanding of business culture_____ (s) Gave me an experience that will increase my employability in the business sector_____ (t) Other(s) – please describe: 62 12. Did you experience any significant problems or difficulties in your internship experience? No_____ Yes____ If Yes: (I) Please briefly describe the nature of the problem(s): (ii) Did you seek advice or help from anyone in attempting to resolve the problem(s)?: No____ If Yes (Check all that apply): (a) Internship Supervisor_____ (b) Mentor____(c) Other company personnel______ (d) JAA/YBLP staff_____(e) Other YBLP students____(f) Friends/family members_____(g) University instructors/professors_____(h)Others (please describe)____ (iii)Were you able to resolve the problem(s)? Please describe: 13. Using a five-point scale in which 1 is “Not Effective” and 5, “Extremely Effective”, how would you rate your internship in enhancing your business career development? (1)Not Effective___(2) Somewhat Effective___(3)Effective____(4)Quite Effective_____(5)Extremely Effective____ 15. Did the company assign a particular individual to serve as your mentor for the internship: No______ Yes______ If yes, (i) Was your Mentor also your immediate supervisor for your work at the company? No____ Yes___ (ii) How often did you meet with your internship Mentor? (a)Daily_____ (b)Weekly_____ (c)Twice Monthly______ (d) Monthly_____(e) Less than monthly_____ (f) Occasionally_____ (iii)On a four-point scale in which 1 is “Not comfortable” and 4 is “Extremely Comfortable” please rate how free or comfortable you felt in openly raising with your mentor any questions or concerns you had and in discussing your goals and aspirations? (1)Not Comfortable_____(2) Somewhat Comfortable______(3) Quite Comfortable_______(4) Extremely Comfortable_______ (iv) What topics did you discuss with your mentor (check all that apply): (l) Your career goals______ (m) Steps to pursue in achieving your goals_____ (n) Obstacles or challenges you may face______) (o) Work culture, business norms, dress and standards______ (p) Specific work challenges in your internship________ (q) Gender issues_______ (r) Business ethics and problems of corruption______ (s) Networking and business success________ (t) Additional formal education or training_________ (u) Self confidence________ 63 (v) Other(s), Please describe (iv)On a five point scale in which 1 is “Not Important” and 5 is “Extremely Important”, please rate how important the mentoring experience has been for you in developing your business career? (1)Not important____(2)Slightly important______(3)Important_____(4)Very Important_____(5) Extremely important_____ 16. Were there any ways in which you were disappointed with the YBLP: 17. On a four-point scale in which 1 is “Not Valuable” and 4 is “Extremely Valuable” please rate the overall value of the YBLP in helping you pursue a business career: (1)Not Valuable_____ (2) Of Limited Value_____ (3) Somewhat Valuable_____ (4) Extremely Valuable___________ Thank you so much for completing this questionnaire. If you have any additional thoughts to share or questions about the evaluation, please contact us at yblpeval@gmail.com. If you wish we would also be pleased to send you a summary of the evaluation findings and recommendations. 64 YBLP Focus Group Discussion Guide - Mentors Introduction: Since October 2012 the Youth Business Leadership Project (YBLP) has sought to “empower the next generation of business leaders in Azerbaijan” by providing undergraduate business students with professional development seminars and internships and mentorships with Azerbaijani-based companies. Each of you are now or had been serving as an YBLP mentors- for your company and have first-hand knowledge of this component of the project. The aim of this focus group discussion is to enable us to learn as much as we can about your experiences and views. Ground Rules: First, here are a few “ground rules” to help produce a productive discussion: 1. Only one person should speak at a time; 2. Please no side conversations with those sitting near you; 3. Let’s avoid having one or two people dominate the conversation; and we need to 4. Be sure to hear from everyone; we want to hear as many different voices, stories and perspectives as possible. Opening Prompt: To get started, we will briefly go around the room to have everyone respond to the following question: Please take a moment to think about and write down one experience as an YBLP mentor to share that you personally found most interesting, memorable, surprising or funny. Follow On Prompts: 1. What were your hopes and expectations when you first agreed to be an YBLP mentor? 2. Did you feel fully prepared to be an effective mentor? What training or support were you given as you began this process? 3. What makes for an effective mentoring process; what are the keys to success? 4. What challenges or difficulties have to be addressed to make the mentoring process work best? 5. To what extent do you think that gender matters in the mentoring relationship? 65 6. Looking ahead five years, how successful do you think your mentee will be in terms of their business career? Do you think that the mentorship will have mattered in this process? 7. Are there any changes in the internship and mentoring process that you would like to propose that would increase its effectiveness in empowering young people pursuing business careers? Concluding Statement: Thank you for participating in this focus group discussion. Your contributions have been quite helpful. Should you find that you have other inputs to share or other comments or suggestions or would like to receive a copy of the Summary YBLP Performance Evaluation Report, please contact us at: yblpeval@gmail.com 66 YBLP Focus Group Discussion Guide – Students Introduction: Since October 2012 the Youth Business Leadership Project (YBLP) has sought to “empower the next generation of business leaders in Azerbaijan” by providing undergraduate business students with professional development seminars and internships and mentorships with Azerbaijani-based companies. The YBLP has been organized and implemented through the efforts of key stakeholder organizations: the United States Agency for International Development (USAID), Junior Achievement Azerbaijan (JAA), the American Chamber of Commerce Azerbaijan (AmCham) and BP Exploration (Caspian Sea) Limited (BP); seven Azerbaijani universities and numerous participating companies. As participants in the YBLP you will have many valuable insights into its design, operations and outcomes. The aim of this focus group discussion is to enable us to learn as much as we can about your experiences and views. Ground Rules: First, here are a few “ground rules” to help produce a productive discussion: 5. Only one person should speak at a time; 6. Please no side conversations with those sitting near you; 7. Let’s avoid having one or two people dominate the conversation; and we need to 8. Be sure to hear from everyone; we want to hear as many different voices, stories and perspectives as possible. Opening Prompt: To get started, we will briefly go around the room to have everyone respond to the following question: Please take a moment to think about and write down one YBLP experience to share that you personally found most interesting, memorable, surprising or funny. Follow On Prompts: 8. What were your hopes and expectations when you first joined the YBLP? 9. We would like to learn about your experience and assessment of the professional development seminars that you took. 10. Please tell us about your internship work and your mentoring relationship. 11. In what ways – large or small – could the YBLP be further strengthened or improved? 67 12. What do you see as the biggest challenges as you begin your business career? 13. Looking ahead five years, if your business career goes as you hope, what will you be doing? 14. How do you think the YLDP has affected your life and career? Concluding Statement: Thank you so much for participating in this focus group discussion. Your contributions have been quite helpful. Should you find that you have other inputs to share or other comments or suggestions or would like to receive a copy of the Summary YBLP Performance Evaluation Report, please contact us at: yblpeval@gmail.com. 68 Youth Business Leadership Project (YBLP) Performance Evaluation Individual Student Interview Schedule Date: / / Interviewer:___________________ Location:____________________ Participant Number:__________ Participant Gender: M___. F.____YLDP Class:________ Good Morning/Afternoon. My name is _______________. I am a member of the research team selected by the United States Agency for International Development (USAID) to evaluate the Youth Business Leadership Project (YBLP). The aim of the evaluation is to document the effectiveness of the YBLP in meeting its program goals. The evaluation will document the project’s strengths and successes, identify any areas for improvement and formulate key lessons learned. As an YBLP participant, you’re your views and experience in the project can make a valuable contribution to the evaluation and we hope that you will agree to be interviewed for this purpose. Your participation in the evaluation is entirely voluntary and there will no consequences for you if you decide to decline to be interviewed. If you do participate please note that your name will not appear in any evaluation report or presentation nor will any quote be attributed to you as an individual. The interview should take no more than 45 minutes. Do you have any questions or need any clarification about the evaluation goals or process? Do you agree to participate in this evaluation interview? Yes______ No______ 8. (a) What university do/did you attend?:________________________________and (b) what year of study are you pursuing?__________ 9. What is your major area of study?______________________________________ 10. How did you first learn about the YBLP? (i). Announcement by a professor or other university personnel________ (ii) Poster/flyer______________ (iii) On line notice____________ (iv) Through participation in Junior Achievement (JAA)____________ (v) Through a fellow student or friend:______________ (vi) Other (please explain)_________________________________________________________ 11. Why did you decide to apply to participate in the project?: 12. Prior to joining the YBLP which component or project activity listed below seemed valuable or important to you? (i) The professional development workshops_________________ (ii) The internship opportunities_________________________ (iii) The mentoring relationship with a member of the host company___________ 69 13. Did you participate in any YBLP professional development workshops? (ii) No______ Yes______ If yes please list name of workshop(s) and date(s): (iii) Name of wkshp:_________________________________ Date taken___________ (iv) Name of wkshp:_________________________________ Date taken___________ (v) Name of wkshp:_________________________________ Date taken___________ 14. For each workshop, ask participant: what did you like most about this workshop? What were the workshop’s greatest strengths? (i) Name of workshop______________________________ (ii) Name of workshop__________________________________ (iii) Name of workshop____________________________________ 15. For each workshop, ask participant: do you have any suggestions for improving this workshop, for making it more valuable and effective? (i) Name of workshop______________________________ (ii) Name of workshop__________________________________ 70 (iii) Name of workshop____________________________________ 16. Using a five-point scale in which 1 is “Not Effective” and 5, “Extremely Effective”, how would you rate the overall effectiveness of the workshops in providing business skills and information that will help you in pursuing your career? (1)Not Effective___(2) Somewhat Effective___(3)Effective____(4)Quite Effective _____(5)Extremely Effective_____ 17. Do you think that the workshops affected your confidence in your ability to have a successful business career? No_____ Yes_________ Please comment: 18. Did you participate in an YBLP internship? (x) No____ Yes______ (xi) If yes, can you please briefly describe the tasks and work activities that you conducted in the internships: (xii) In what type of business (sector) did you intern?_____________ (xiii) What is its size: (a) local small or medium enterprise (up to 100 employees); (b) local large business (100+ employees); (3) large multinational___________. (xiv) How many hours/week did you work on the internship________ (xv) Did you receive any training or instruction from the company as part of the internship? No_____ Yes______ If yes, please describe: (xvi) Did you work?: (a) primarily on your own_______ (b) primarily as part of a work team__________ (c) a mixture of individual and group work_____ (xvii) Were there other interns also working at the company while you were there? Yes___ No___ (xviii) Has this company utilized interns prior to its participation in the YBLP? Yes____ No____ Do Not Know_____ (xix) In what ways, if any, did the internship experience enhance your business career development (check all that apply): (u) Gave me the opportunity to learn new business skills________ 71 (v) Gave me the opportunity to apply and practice business skills in a real-life setting____ (w)Gave me a chance to display my abilities and competencies______ (x) Gave me the opportunity to work with others in a real business setting_______ (y) Helped me in building a network of contacts in the business community_______ (z) Helped me affirm my decision to pursue a business career________ (aa) Increased my confidence in succeeding in a business career________ (bb) Provided me with a better understanding of business culture_____ (cc) Gave me an experience that will increase my employability in the business sector_____ (dd) Other(s) – please describe: 12.Did you experience any significant problems or difficulties in your internship experience? No_____ Yes_____If Yes: (i)Please describe the nature of the problem(s): (ii) Did you seek advice or help from anyone in attempting to resolve the problem(s)?: No____ If Yes (Check all that apply): (a) Internship Supervisor_____ (b) Mentor____(c) Other company personnel______ (d) JAA/YBLP staff_____(e) Other YBLP students____(f) Friends/family members_____(g) University instructors/professors_____(h)Others (please describe)____ (iii)Were you able to resolve the problem(s)? Please describe: 13. Using a five-point scale in which 1 is “Not Effective” and 5, “Extremely Effective”, how would you rate your internship in enhancing your business career development? (1)Not Effective___(2) Somewhat Effective___(3)Effective____(4)Quite Effective_____(5)Extremely Effective____ 14.What suggestions would you make to enhance the quality or usefulness of the YBLP internship experience?: 15. Did the company assign a particular individual to serve as your mentor for the internship: No______ Yes______ If yes, (j) Was your Mentor also your immediate supervisor for your work at the company? No____ Yes___ 72 (v) How often did you meet with your internship Mentor? (a)Daily_____ (b)Weekly_____ (c)Twice Monthly______ (d) Monthly_____(e) Less than monthly_____ (f) Occasionally_____ (vi)On a four-point scale in which 1 is “Not comfortable” and 4 is “Extremely Comfortable” please rate how free or comfortable you felt in openly raising with your mentor any questions or concerns you had and in discussing your goals and aspirations? (1)Not Comfortable_____(2) Somewhat Comfortable______(3) Quite Comfortable_______(4) Extremely Comfortable_______ (iv) What topics did you discuss with your mentor (check all that apply): (w) Your career goals______ (x) Steps to pursue in achieving your goals_____ (y) Obstacles or challenges you may face______) (z) Work culture, business norms, dress and standards______ (aa) Specific work challenges in your internship________ (bb) Gender issues_______ (cc)Business ethics and problems of corruption______ (dd) Networking and business success________ (ee)Additional formal education or training_________ (ff) Self confidence________ (gg)Other(s), Please describe: (vii) Do you (or do you anticipate) remaining in contact with your mentor after completing your internship? No____ Yes_______ If you are still in contact with your mentor, please describe what this has been like: (viii) On a five point scale in which 1 is “Not Important” and 5 is “Extremely Important”, please rate how important the mentoring experience has been for you in developing your business career? (1)Not important____(2)Slightly important______(3)Important_____(4)Very Important_____(5) Extremely important_____ (ix) What did you like best about the mentoring process: (x) In what ways could the mentoring component of the YBLP be strengthened or improved: 73 16. As the implementing organization for the YBLP, Junior Achievement Azerbaijan (JAA) was responsible for the overall project planning, organization, implementation and coordination. Prior to joining the project had you been aware of JAA? No_____Yes_____ If Yes please describe: 17. In what YBLP roles/contexts did you interact with JAA personnel? (check all that apply): (a) Recruitment____ (b) Professional Development Workshops______ (c) Internship________ (d) Mentorship_________ (e) Other (please describe)__________ 18. Since completing your internship, have you had or do you expect to have, any further contact with JAA? No____ Yes_____ Please describe: 19. Imagine that it is July 2019. Assuming that your career path proceeds as you would like, please describe what your work life will be like at this time: 20. Overall, what were the three most important benefits you gained from your participation in the YBLP. 1. Most important: 2. Second: 3. Third: 21. Please describe in what ways the project met your expectations as a means for advancing your business career: 22. What aspects of your YBLP experience did you find most interesting or surprising? Please describe: 74 23. Is there a particular illustrative incident or event concerning your engagement in the project that you would like to share: 24. Were there any ways in which you were disappointed with the YBLP: 25. On a four-point scale in which 1 is “Not Valuable” and 4 is “Extremely Valuable” please rate the overall value of the YBLP in helping you pursue a business career: (1)Not Valuable______(2)Of Limited Value_____(3) Somewhat Valuable_____(4) Extremely Valuable___________ 26. What are your ideas for improving the YBLP: 27. Is there anything else about the project or your participation that you would like to tell us: Thank you so much for your participation in this interview. Your input is a valuable resource for our assessment. If you have any additional thoughts to share or questions about the interview or the evaluation, please feel free to contact me at yblpeval@gmail.com. If you wish we would also be pleased to send you a summary of the evaluation findings and recommendations. Thanks again, 75 ANNEX 5: LIST OF DOCUMENTS RECEIVED AND REVIEWED 76 Final Performance Evaluation of the Youth Business Leadership Project (YBLP) in Azerbaijan Documents Reviewed Youth Business Leadership Project Annual Report September 2012 – September 2013 United States Agency for International Development Cooperative Agreement Number: AID-112-A 12-00004 Youth Business Leadership Project (YBLP) Performance Monitoring Plan Youth Business Leadership Project (YBLP) Project Work Plan September 2012 – September 2013 Youth Business Leadership Project (YBLP) Project Work Plan September 2013 – September 2014 Youth Business Leadership Project (YBLP) Youth Business Leadership Project (YBLP) Program Report Narrative September-December 2012 Youth Business Leadership Project (YBLP) Youth Business Leadership Project (YBLP) Program Report Narrative January-March 2013 Youth Business Leadership Project (YBLP) Youth Business Leadership Project (YBLP) Program Report Narrative April-June 2013 Youth Business Leadership Project (YBLP) Youth Business Leadership Project (YBLP) Program Report Narrative July-September 2013 Youth Business Leadership Project (YBLP) Youth Business Leadership Project (YBLP) Program Report Narrative October-December 2013 Youth Business Leadership Project (YBLP) Youth Business Leadership Project (YBLP) Program Report Narrative January-March 2014 77 ANNEX 6: UNIVERSITIES PARTICIPATING IN YBLP 78 YBLP Participating Universities 1. Azerbaijan State Economic University (ASEU) Established in 1930, Azerbaijan State Economic University is one of the biggest higher education institutions in the South Caucasus. It is located in Baku, the capital of Azerbaijan. The mission of the university is to train highly qualified economists, professional business leaders and managers and contribute to the development of knowledge-based economy and social welfare in Azerbaijan. There are 11 faculties at the university, and more than 16 thousand undergraduate and graduate students study at ASEU. The university trained about one hundred thousand economists until today. More than 850 undergraduate and graduate international students study at ASEU. The university developed and offer MBA program in cooperation with Moscow State University of Economics, Statistics and Informatics (MESI). 2. Azerbaijan State Agrarian University Established in 1929, Azerbaijan State Agrarian University is a public university located in Ganja that is the second biggest city in Azerbaijan. The university offers Bachelor, Master and PhD programs, and more than 3,800 students study at Azerbaijan State Agrarian University. Azerbaijan State Agrarian University is the only public university in Azerbaijan which offers research and study programs for the agricultural sector. Many new academic programs including agriculture, civil engineering, economics and oil are offered at the university. 3. Azerbaijan State Oil Academy (ASOA) Founded in 1920, ASOA is the first oil and gas profile technical higher education institution in Europe. It is located in Baku, Azerbaijan. The university specialized in offering academic programs in oil mechanics, economics, international economic relations and management, geological exploration, and technical fields. It has been an institution associated with the growing oil and gas industry in Azerbaijan. Furthermore, the university has participated in technical training of students from foreign countries. 4. Azerbaijan Technological University (ATU) ATU is a public university located in Ganja. The university plays a crucial role in the training of highly qualified personnel for leading fields of Azerbaijan’s economy. The university offers 17 specialties, and it has four faculties, 17 departments and one department of doctoral and master’s degrees. More than 2000 students currently continue their education at the university. Scientific-research laboratories of the university play significant role in the improvement of education process and effectiveness of scientific-research work. The four faculties operating within the university are Technology of Consumer Goods and Examination, Economy and Management, Technology of Food Productions and Tourism, Standardization and Technological machines. 5. Baku State University (BSU) Established in 1919, BSU is a public teaching and research university located in Baku, the capital of Azerbaijan. BSU offers Bachelor, Master and PhD programs on a wide range of areas including international relations, journalism, law, physics, applied mathematics, library studies, chemistry, geology and others. It is considered a leader of education institutions in Azerbaijan 79 due to a rich history and educational traditions. BSU currently offers students 16 majors with 55 bachelor and 153 master programs. It is the biggest university in Azerbaijan in accordance with the number of its students and education programs. 6. Ganja State University Founded in 1938, Ganja State University is a public university located in Ganja, Azerbaijan. Its aims to train qualified teachers and contribute to the development of culture and arts in Azerbaijan. It offers Bachelor, Master and PhD programs. There are eight faculty departments within the university. The university offers academic programs in foreign languages, philology, history, geography, mathematics and physics. 7. Khazar University Khazar University is a private university in Baku, Azerbaijan. It was established in 1991 recognizing a great necessity of the country for English speaking professionals. The language of instruction at the university is English. It is a respected university among young people, businesses, embassies and government institutions due to the quality of education. Khazar University offers a diverse range of majors in social sciences, economics and management, business administration, engineering and applied sciences. The university is popular for its MBA in Azerbaijan, and it is also popular for its academic programs in social sciences, regional studies, business administration, economics, and management. 8. Qafqaz University Qafqaz University is a private foreign university established in 1993. It is located in Baku, Azerbaijan. The language of instruction is English at the university. The university also creates opportunities for its students to participate in the internship and exchange programs in abroad, and continue their education at foreign education institutions. It has created a large campus outside of Baku. The university offers majors in social sciences, engineering, economics, foreign languages, business administration and others. 80 ANNEX 7: YBLP EVALUATION PARTICIPANTS 81 Category of Informant: interviews Institution/organization Informant Name and Position Key Stakeholders Junior Achievement Hokuma Babayeva, Executive Director USAID Ulker Akbarova, Project Management Assistant BP Nigar Tagiyeva, Performance and Reporting Coordinator AmCham Jeyhun Attayev, Member Services Officer Companies Seabak – Baku Agateyyub Jafarov, Business Operations & Development Ganja Agrobusiness Association – Ganja Habib Abbasov, Public Relations Manager Ganja Business Group– Ganja Arif Cahangirov, Deputy-director Universities Azerbaijan State Agrarian University Professor Zakir, Dean of Economics Faculty Khazar University Ulviyya Hasanli, Head of Career Planning Centre Course Instructors SAPPHIRE PLAZA Toghrul Alakbarov, General Manager School N 6 Zarema Almazova, Teacher Azerbaijan Coca-Cola Bottlers Tofig Sadikhov, Mechanical and Electrical Engineer USAID-funded project office based in Ganja Gunay Zeynalova, Local Specialist YBLP Participants/ Qafqaz University Umid Hasanov, student 82 Students Azerbaijan State Oil Academy Rahim Abdullayev, Nihat Farajov, Janbulat Allahverdiyev, Faig Huseynov, Emil Huseynov and Zaur Mammadov (students) Azerbaijan State Economic University Lachin Kamilov, Elbrus Ismayilli and Isa Abdullayev (students) Baku State University Elvin Asadov, Azer Rustamli and Rufana Fataliyeva (students) Azerbaijan Technological University Gunduz Gasimov and Azer Huseynli (students) Azerbaijan State Agrarian University Farid Rahimov and Afqan Jafarov (students) Mentors SAZZ (Baku) Jahan Aliyeva, PR Manager TEMIZ SHEHER (Baku) Faig Mammadov, Head of Project Management Sector BP (Baku) Elvin Samadov, Engineer Junior Achievement (Baku) Nilufer Damirzade, Project Manager FESCO (Baku) Ilyas Efendi, Director Ganja Agrobusiness Association (Ganja) Habib Abbasov, Bridge to the Future (Ganja) Sevinj Mammadova, Unibank (Ganja) Elvin Isgandarov, Unibank (Ganja) Vugar Zeynalov, Unibank (Ganja) Khadija Melikova, 83 Asan Xidmet (Ganja) Ilaha Abbasova, Asan Xidmet (Ganja) Almaz Messimova, Ganja Regional Advice Centre (Ganja) Fakhriyya Huseynova, Ganja Regional Advice Centre (Ganja) Rafiq Mammadov, Ganja Regional Advice Centre (Ganja) Sadagat Gahramanova, Category of Informant: e￾questionnaires Institution/organization Informant Name and Position YBLP Participants/ Students Qafqaz University Gulnara Imranli and Ulkar Karimbayli (students) Baku State University Gunel Musayeva and Natavan Hasanova (students) Azerbaijan State Economic University Ilham Balakishiyev, student Azerbaijan State Oil Academy Nihat Guliyev, Shirkhan Qarazada, Samir Israyilov and Suleyman Sadikhov (students) Mentors Fairmont Sonya Flood, HR Manager Norwegian Humanitarian Enterprise Sevinj Bagirova, Accountant AmCham Jeyhun Attayev, Member Services Officer Certus Pharma LLC Seymur Shukurzade, PA to CEO P&G Rashad Hasanzade, Retail Account Manager 84 Junior Achievement Konul Huseynova, Outreach Coordinator Category of Informant: Participants involved in focus group Institution/organization Informant Name and Position Students Azerbaijan State Oil Academy Ravana Karimova, student Khazar University Turana Mustafayeva, student Azerbaijan State Economic University Orkhan Ismayilov and Gizkhanim Jafarova (students) Baku State University Panah Bannayev and Naila Samiqulina, student Azerbaijan Technological University Javahir Valiyeva, Sabina Islamova, Lamiya Khasiyeva, Ayshan Safarova and Ramin Aliyev (students) Ganja State University Mustafa Jafarov and Kamala Guliyeva (students) Azerbaijan State Agrarian University Alya Alizada, student 85