USAID/KOSOVO TLP Performance and Impact Evaluation IMPACT EVALUATION OF THE TLP SCHOLARSHIP PROGRAM Endline Study and Final Report February 2020 This document is made possible by the support of the American People through the United States Agency for International Development (USAID). The contents of this report were prepared by IBI under contract number AID-167-TO-15-00002. The views expressed herein are the sole responsibility of IBI and do not necessarily reflect the views of USAID or the United States Government. USAID/KOSOVO TLP Performance and Impact Evaluation IMPACT EVALUATION OF THE TLP SCHOLARSHIP PROGRAM Endline Study and Final Report January 2020 This document is made possible by the support of the American People through the United States Agency for International Development (USAID). The contents of this report were prepared by IBI under contract number AID-167-TO-15-00002. The views expressed herein are the sole responsibility of IBI and do not necessarily reflect the views of USAID or the United States Government. USAID Impact Evaluation of the TLP Scholarships Program– Final Report i Table of Contents Table of Contents....................................................................................................................................................... i Acronyms.................................................................................................................................................................. ii EXECUTIVE SUMMARY ........................................................................................................................................................III 1 INTRODUCTION..................................................................................................................................................... 1 1.1 KOSOVO COUNTRY CONTEXT ................................................................................................................................... 1 1.2 DESCRIPTION OF THE TLP PROGRAM AND THE TLP-SP PROJECT..................................................................................... 2 1.2.1 THE TLP PROGRAM ........................................................................................................................................... 2 1.2.2 THE TLP-SP PROJECT......................................................................................................................................... 3 1.2.3 THE TLP SCHOLARSHIPS COMPONENT ................................................................................................................... 3 1.3 ORGANIZATION OF THE REPORT ................................................................................................................................ 4 2 STUDY OBJECTIVES AND APPROACH..................................................................................................................... 5 2.1 EVALUATION OBJECTIVE.......................................................................................................................................... 5 2.2 EVALUATION QUESTIONS ........................................................................................................................................ 5 2.3 STUDY APPROACH.................................................................................................................................................. 6 2.4 LIMITATIONS OF THE STUDY ..................................................................................................................................... 9 3 EVALUATION QUESTION 1: USE OF SKILLS AND TRAINING.................................................................................. 10 3.1 CURRENT STATUS OF SCHOLARSHIP RECIPIENTS.......................................................................................................... 10 3.1.1 EMPLOYMENT ................................................................................................................................................. 11 3.1.2 VOLUNTEER WORK ........................................................................................................................................... 12 3.2 TRAINING AND SKILLS DEVELOPMENT ....................................................................................................................... 13 3.3 UTILIZATION OF TRAINING AND SKILLS ...................................................................................................................... 16 3.4 CONSTRAINTS OR BARRIERS TO UTILIZING KNOWLEDGE/SKILLS ...................................................................................... 19 4 EVALUATION QUESTION 2: CONTRIBUTION TO KOSOVO ................................................................................... 21 5 EVALUATION QUESTION 3: PERCEPTION OF TLP SCHOLARSHIP RECIPIENTS AS LEADERS .................................. 24 5.1 SELF-PERCEPTION OF SCHOLARSHIP RECIPIENTS .......................................................................................................... 24 5.2 PERCEPTION OF EMPLOYERS AND STAKEHOLDERS ....................................................................................................... 26 6 EVALUATION QUESTION 4: COMPARISON WITH NON-RECIPIENT PEERS ........................................................... 28 6.1 CURRENT STATUS................................................................................................................................................. 28 6.2 TRAINING AND SKILLS DEVELOPMENT ....................................................................................................................... 28 6.3 EMPLOYMENT ..................................................................................................................................................... 29 6.4 VOLUNTEER ACTIVITIES ......................................................................................................................................... 30 6.5 UTILIZATION OF TRAINING AND SKILLS ...................................................................................................................... 30 6.6 CHANGE IN SELF-PERCEPTIONS OF LEADERSHIP POTENTIAL............................................................................................ 32 6.7 PERCEPTION OF EMPLOYERS AND STAKEHOLDERS ....................................................................................................... 33 6.8 PLANS FOR THE FUTURE ........................................................................................................................................ 33 7 CONCLUSIONS AND RECOMMENDATIONS ......................................................................................................... 34 8 ANNEXES............................................................................................................................................................. 39 ANNEX 1. SCOPE OF WORK FOR THE EVALUATION .......................................................................................................... 39 ANNEX 2. INFORMATIONAL MEETINGS AND INTERVIEWS CONDUCTED ................................................................................. 48 ANNEX 3. LIST OF DOCUMENTS REVIEWED ..................................................................................................................... 49 ANNEX 4. SURVEY INSTRUMENTS.................................................................................................................................. 50 4.1. SCHOLARSHIP RECIPIENTS ENDLINE SURVEY .............................................................................................................. 50 4.2. TLP COMPARISON ENDLINE SURVEY........................................................................................................................ 66 ANNEX 5. DATA TABLES .............................................................................................................................................. 84 ANNEX 6. THE COMPONENTS OF TRANSFORMATIONAL, TRANSACTIONAL AND PASSIVE-AVOIDANT LEADERSHIP........................... 89 USAID Impact Evaluation of the TLP Scholarships Program– Final Report ii Acronyms AC Advisory Committee ASU Arizona State University COR Contracting Officer Representative DC Dartmouth College GOK Government of Kosovo EU European Union IBI International Business Initiatives IU Indiana University LGBT Lesbian, Gay, Bi-Sexual and Transgender M&E Monitoring and Evaluation MA Master of Arts MBA Master of Business Administration MEST Ministry of Education, Science and Technology MLQ Multifactor Leadership Questionnaire MPA Master of Public Administration PhD Doctor of Philosophy TLP Transformational Leadership Program TLP-SP Transformational Leadership Program – Scholarships and Partnerships Project UMN University of Minnesota UN United Nations UP University of Prishtina USAID United States Agency for International Development WL World Learning USAID Impact Evaluation of the TLP Scholarships Program– Final Report iii Executive Summary United States Agency for International Development/Kosovo (USAID/Kosovo) has contracted with IBI International (IBI) to conduct an impact evaluation of the Transformational Leadership Program’s (TLP) U.S. Master’s degree Scholarships component as well as a performance evaluation of the TLP’s University Partnerships component. This report presents the overall findings of the impact evaluation (including the endline study) of the Scholarships component. USAID and the Government of Kosovo (GOK) have jointly funded TLP, which seeks to enhance human capital in Kosovo, build a cadre of skilled professionals capable of addressing Kosovo’s economic and democratic development priorities and address some of the gaps in the education system. Two of TLP’s five components – the Scholarships component and the University Partnerships component, collectively referred to as TLP Scholarship and University Partnerships (henceforth TLP-SP project) – were implemented under a cooperative agreement between USAID and World Learning Inc. (WL) and its partners and are the subject of this evaluation. The objectives of the TLP-SP were to support selected citizens of Kosovo with obtaining U.S. post-graduate degrees (Master’s degrees) and professional certifications for (Scholarships component) and support improved university management and pedagogy in selected faculties of the University of Prishtina (UP) and/or other universities through partnerships and exchanges with U.S. Universities (University Partnerships component). The Scholarships component of the TLP program was designed to create a cadre of talented professionals that could rapidly bring critical leadership and innovation skills to the largest number of Kosovars across all segments of the society and help Kosovo address the demands of a modern market economy and bring transformational change to the country. This evaluation examines the impact of the Master’s degree programs in U.S. universities and colleges on the recipients of the TLP scholarships (for Master’s program only). The evaluation addresses the following broad questions: • How have the participants utilized their training since the completion of their studies? (If not, why not?) What significant constraints or barriers, if any, do the returned participants face in fully utilizing their new knowledge/skills? • In what ways are U.S.-returned beneficiaries contributing to Kosovo’s development and how have the skills acquired in the U.S. added to that contribution, particularly at the organizational level? • In what ways does the perception of TLP scholarship recipients as leaders change over the course of the program? • Given a-c above, how and to what extent have the experiences of the returned scholarship recipients differed from their non-recipient peers? This evaluation provides stakeholders, USAID and the Government of Kosovo with an understanding of how various investments in the scholarships for Master’s have increased the knowledge and skills of the individual participants, and determines the effectiveness of the program in training participants to act as change agents. USAID Impact Evaluation of the TLP Scholarships Program– Final Report iv Study Approach The evaluation followed the progress of scholarship recipients over a period of four years from the time they were awarded a scholarship to two years after their return to Kosovo. In order to estimate the impact of the Scholarships program, a control group was selected from among the applicants to the TLP Scholarships program who met the TLP-SP project criteria for applicants and qualified for the interview but were not selected for the scholarship or were selected but did not join for any reason. This control group was followed over the same time period as the scholarship recipients to estimate the counterfactual. All (185) of the scholarship recipients from the three cohorts were included in the sample for the evaluation. Employers of the TLP scholarship recipients who had at least three scholarship recipients on their staff were interviewed to get a broader perspective on the effects of the program. Quantitative data were collected through surveys of the scholarship recipients and their comparison group, which had an overall response rate of 85 percent for scholarship recipients and 63 percent for the comparison group. These data were supplemented with qualitative data collected from review of project documents, through focus groups with scholarship recipients, and through interviews and informational meetings with a variety of stakeholders, including TLP-SP staff, MEST, USAID, senior management at the University of Prishtina (UP), implementers of the TLP Citizen Corp program and select employers of the scholarship recipients. Each cohort was evaluated thrice: (i) at baseline – upon start of the Master’s program; (ii) at midterm – upon completion of Master’s program and return to Kosovo; and (iii) at endline at a point two years after completion of the Master’s program. Data collected at endline are compared with data from baseline and midline as well as the comparison group to respond to each of the study questions and assess the contributions of the scholarship program, changes in status of participants, and their contribution to Kosovo. Limitations of the Study A few caveats and limitations must be kept in mind as we review the results from this study. First, the assessment of the contribution of the scholarship recipients to Kosovo is based on self-reports and information provided by the respondents themselves. Even though this was triangulated with information from other stakeholders, the bias due to self-reporting remains. Second, since most of the scholarship recipients are employed, it was very difficult at endline to get them to participate in focus groups. While the issue was resolved by offering multiple opportunities for focus groups as well as accommodating requests for individual in-person or phone interviews at times convenient to the respondents, there is some concern of self-selection in the sample and the possibility of a bias. Incentives were offered to both the scholarship recipients (15 Euros per completed survey) and the comparison group (35 Euros), but the concern remains. Main Findings Overall, the TLP scholarships program has been well received and has had a positive impact in the short￾run. Of the 185 scholarships that were awarded for the Master’s degree programs (three cohorts: 2014, 2015 and 2016), 181 recipients have completed a Master’s program from a U.S. University or college. Many participants studied in inter-disciplinary programs not available in Kosovo. Participants in the program from all three cohorts have by and large been satisfied with the study experience in the U.S. and speak highly of the quality of program and the skills developed because of that experience. A majority of the participants report being satisfied with the support, including the stipend, provided by the program to help them get settled as well as in ensuring their academic and physical well-being. USAID Impact Evaluation of the TLP Scholarships Program– Final Report v Most TLP scholarship recipients have completed the mandatory 400 hours of volunteer work upon return to Kosovo. TLP scholarship recipients volunteered their time with public and private universities, think tanks, and NGOs working on climate change, domestic violence, gender issues, and on teaching coding to girls. Several reported working as teaching assistants and research assistants, consultants and legal advisors and mentoring those planning start-ups. Overall, the TLP Scholarship recipients are doing as well, if not better, than their peers. Out of 154 scholarship recipients that completed the endline survey, only eight reported being unemployed. Most others reported being employed in mid-to-senior level positions within the private sector, NGOs, government and public universities in Kosovo. Most of these positions are better paid and come with greater responsibility within the organization than the participants’ positions prior to their joining the scholarships program. And in most cases, the positions are related to the field of study that the participants selected for the Master’s program in the U.S. Conversations with employers suggest that the scholarship recipients have indeed developed the skills and leadership potential that was expected from their participation in the program and have already begun to contribute substantially to the achievement of the overall purpose of TLP objectives. However, the program had its challenges as well. To complete the Master’s program, the scholarship recipients are required to stay in the U.S. between 12 and 24 months, as a result of which many experienced a reverse culture shock and a problem settling back. They also felt that they had lost their social and professional networks and needed to start afresh. Many also reported issues in transitioning from volunteering to motivating themselves into applying and interviewing for employment opportunities. Participants reported being disappointed in the limited employment opportunities available to them upon their return to Kosovo, realizing quickly that they needed work experience in addition to their degrees to find a suitable job. Use of Skills and Contribution to Kosovo’s Development Since most scholarship recipients have been back in Kosovo for only two years, it is rather early to assess the impact that they are having in their workplace and their contribution to Kosovo’s development. However, self-reports by scholarship recipients and assessments by their employers and supervisors suggest early indications of positive contributions and transformational leadership. • The scholarship recipients are enhancing the stock of skills in Kosovo by returning with specializations gained in disciplines not hitherto available in Kosovo, such as Aviation Management, Sustainable Urban Planning, Cyber Security, Environmental Policy, and Social Enterprise. • Scholarship recipients are bringing advanced skills in communication, including fluency in the English language, writing and conducting research to the country and to their work place. • Scholarship recipients are demonstrating readiness to lead multi-disciplinary teams, including teams with international experts, and are demonstrating capacity in knowledge-building within the country. • Scholarship recipients – by working as teaching staff at various private universities/colleges in USAID Impact Evaluation of the TLP Scholarships Program– Final Report vi Kosovo and volunteering their time as Teaching Assistants in the University of Prishtina (UP) -- have begun contributing to reforming the teaching and learning culture at UP. • Scholarship recipients, as researchers at the university, in governments and in NGOs, are demonstrating the use of ethical ways of conducting research and using research data to support discussions/advocacy. • TLP scholars are not only engaged in jobs that are directly related to their fields; they are also using their expertise in related fields, introducing the concept that skills learned in one subject area can also be applied to another field thereby expanding the employment opportunities. • TLP scholars are creating new job opportunities in Kosovo. Several scholarship recipients have started their own businesses and are working with international partners, bringing new opportunities to Kosovo. Others are working on supporting companies in Europe while living in Kosovo, bringing in revenue for the country. • And finally, scholarship recipients are introducing a culture of volunteerism in Kosovo. They are helping support their communities and grass roots NGOs by helping write research grants and by conducting research to support advocacy for policy change. Some are also leading and helping with providing training to disadvantaged populations within their communities. All these initiatives and changes in approach have a good potential to sustain themselves and positively contribute to culture change within Kosovo. Indeed, participants from Cohorts 1 and 2, especially those who have had time to settle down, are rapidly climbing to senior and responsible levels within their organizations and are increasingly getting into positions from where they can model and push for more ethical ways of doing things. Comparison with their peers shows that they are doing things better than most of their peers and are more likely to be leaders within their organizations, willing to take the time to build trust, model ethical behaviors and coach and mentor their teams. However, there are some challenges here as well, as several scholarship recipients are showing signs of frustration as they meet deeply entrenched ideas in the Kosovar fabric of life that resist new ways of thinking and doing things and limiting the impact that returning participants thought they could have. The paucity of resources and organizational support for sharing new ideas also poses barriers to the use of newly acquired skills. The employers also voiced a note of caution, saying that since the TLP scholars were well trained and highly marketable, they were unlikely to stay in one position within the same organization for any length of time – or for that matter, within Kosovo, as there were so many opportunities for highly trained and skilled personnel in other countries in Europe, especially Germany. The employers noted that while the mobility of TLP scholars across organizations was generally good for commerce and industry, it also made it risky for employers to invest significantly in recruiting and training new employees. Recommendations Several recommendations emerge from the evaluation of the TLP Scholarships program: USAID Impact Evaluation of the TLP Scholarships Program– Final Report vii • The premise underlying the TLP-Scholarships program was that it would create a cadre of talented professionals with the leadership and innovation skills necessary to address the demands of a modern market economy and bring transformational change to Kosovo. This would happen through improved job performance and improvements in the public and private organizations where the TLP scholars get employment. Early evidence suggests that while the TLP scholars working in the private sector are already beginning to have an impact, no discernible effects are being seen in the public sector. First, very few TLP scholars have found positions within the government. Bringing about meaningful advancements in capacity development and demonstrating different ways of doing things and directly influencing policy, would require a larger number of TLP scholars to be placed in the government. Second, bringing about reforms in the government system typically takes more time than in the private sector. We therefore recommend that future programs connect the beneficiaries of such efforts and the government, with the latter committing to utilizing the services of the graduates for two to three years. • In addition to increased presence within the public sector, we recommend setting up an institutionalized process by which TLP scholars can routinely meet with officers in government and public institutions and share their skills and experience. Creation of an advisory committee of scholarship recipients, academia and policy experts to support officials in various Ministries would go a long way in realizing this objective. • The public universities within Kosovo support the majority of the students for higher education. There is a shortage of skilled staff for several disciplines. But they have not been able to absorb many of the TLP scholars due to the current rules and regulations regarding hiring teaching staff. We recommend that in future programs there is some coordination with the University of Prishtina (the single largest public university in Kosovo) to see if there is a way to support the University to fill the skills gap. • Volunteer work has helped both the TP scholars upon their return as well as their communities. The TLP scholars had something meaningful to do upon their return while they waited to hear back from job interviews, and the communities got some world-class assistance in grant writing and advocacy materials or in the cause they support. We recommend that volunteering continues to be included in future programs as well, preferably with a small stipend to help them tide over the period during which they are seeking employment. A higher stipend could be offered to those willing to work in rural areas, which would incentivize at least some of the graduates to temporarily move there and help some far-flung communities. • One downside that many TLP scholars have experienced of being away from Kosovo for two years is that of losing their social and professional networks. Strengthening support to the Alumni Association established by the TLP program would help TLP scholars enhance networking opportunities and share experiences and knowledge and exchange ideas. • In order to mitigate the reverse-culture shock that many TLP scholars have reported facing, we recommend that future programs include training and support services to facilitate their reentry to Kosovo. These support services could cover topics such as culture in the workplace, optimal use of skills in a culturally-appropriate manner, taking initiative in the workplace, etc. A small subsistence allowance to cover the first few weeks of return to Kosovo would help the recipients to avoid rushing to take the first job offered. USAID Impact Evaluation of the TLP Scholarships Program– Final Report viii • We further recommend that prospective employers be offered guidance on the training received by the TLP scholars in the Master’s program in the U.S. This would help ensure that employers have reasonable expectations of the capabilities and interest of the returning scholarship recipients, which would mitigate frustration on both sides. • Discussions with a sample of employers suggest that TLP scholars who had some work experience prior to going to the U.S. for higher studies are more mature in their expectations and have been better at adjusting upon their return to Kosovo. We recommend that a two-year work experience be built in as a criterion for obtaining the scholarship. • The beneficiaries of the program have had to reactivate their social and professional networks. They have found that the networks of TLP graduates are more useful for them due to a shared understanding of their training and challenges upon return to Kosovo. We recommend continued support for the TLP Alumni Association. • Finally, we recommend that scholarship recipients in future programs be invited to a mandatory refresher course on leadership conducted in Kosovo. This would serve to remind the scholarship recipients of the training provided to them while in the U.S., and would offer practical suggestions on the practice of leadership within the context of the workplace and prevailing culture in Kosovo. USAID Impact Evaluation of the TLP Scholarships Program– Final Report 1 1 INTRODUCTION1 USAID/Kosovo has contracted with IBI International (IBI) to conduct a concurrent evaluation of two components of the Transformational Leadership Program (TLP) – an impact evaluation of the U.S. Master’s degree scholarships component and a performance evaluation of the University Partnerships component (contract scope of work is found in Annex 1). Even though the evaluation of the two components has been completed concurrently, the findings are reported separately. This report presents the overall findings of the impact evaluation (including the endline study) of the Scholarships component, including the results of the endline study for all cohorts of scholarship recipients. The rest of this section contains a brief description of the country context, the TLP program and the TLP scholarships and partnerships project and components evaluated, followed by a section that describes how the rest of this evaluation report is organized. 1.1 Kosovo Country Context Kosovo declared independence from Serbia on February 17, 2008 after years of strained relations between its Albanian and Serb populations. It is recognized as an independent country by 114 out of 193 United Nations (UN) members and by 23 out of 28 EU members.2 Kosovo is a potential candidate for EU membership and in recent years has accelerated its EU integration process. Kosovo’s population of about 1.8 million is the youngest in Europe, with an average age of 26 years.3 A lower-middle-income country that has experienced solid economic growth over the last decade, Kosovo is one of only four countries in Europe to experience growth in every year since the onset of the global financial crisis in 2008. Despite this, Kosovo remains one of the poorest countries in Europe, and its average per capita income of approximately U.S. $3,877 (October 11, 2018) is about one-tenth that of EU levels.4 More than 80 percent of the population lives under the World Bank standardized poverty rate, defined by a threshold of U.S. $5 per day at purchasing power parity, and about 30 percent of the population lives below the domestic poverty line, defined by the Kosovo Agency of Statistics as Euro 1.72 per day (2011 prices). Kosovo has a relatively low Gini index and flat consumption distribution, with poverty rates declining gradually. No significant differences exist between urban and rural poverty, but regional differences are notable. Extreme poverty is disproportionately high among children, the elderly, households with disabled members, female-headed households, and certain ethnic minority households (especially Roma, Ashkali, and Egyptian communities). As in many other countries, there is a strong negative correlation between education and poverty in Kosovo.5 2 The World Bank group (2018). Country Context. https://www.worldbank.org/en/country/kosovo/overview. There have been some reports recently of some countries withdrawing support. See, for instance, B92, News article, TANJUG, December 3, 2018 found in Wikipedia at https://en.wikipedia.org/wiki/Kosovo Accessed on Sep 1, 2019. 3 The World Bank Group (2017): FY17–-21 Country Partnership Framework for Republic of Kosovo. Report No. 112337-XK. http://documents.worldbank.org/curated/en/297951496160148830/Kosovo-Country-partnership-framework-for-the-period-FY17- FY21 accessed on August 25, 2019. 4 See the World Bank Group (2017): Kosovo – Country Snapshot. Accessed on January 5, 2018 at http://pubdocs.worldbank. org/en/912841507748392027/Kosovo-Snapshot-Fall2017.pdf 5 See the World Bank Group (2016): Kosovo – Country Program Snapshot. Accessed on February 27, 2017 at http://pubdocs.worldbank.org/en/419461462386476530/World-Bank-Kosovo-Program-Snapshot-April-2016.pdf USAID Impact Evaluation of the TLP Scholarships Program– Final Report 2 Although Kosovo’s economic growth has outperformed its neighbors and been largely inclusive, it has not been enough to significantly reduce the high rates of unemployment; provide formal jobs, particularly for women and youth; or reverse the trend of large-scale outmigration. In the past two decades, since its declaration of independence, Kosovo’s economy has changed substantially. The current growth model relies heavily on remittances to fuel domestic consumption but has recently shifted toward more investment- and export-driven growth.6 Education is an essential part of Kosovo’s efforts to boost economic growth, increase productivity and wages, and reduce high unemployment. The Kosovo Education Strategic Plans of 2011-16 and of 2017-2021, developed by the Ministry of Education, Science, and Technology (MEST), tackle a host of issues related to inclusion, access, quality of education, accountability and improved management, as well as implementation of standards at all levels of education to ensure that its citizens possess the knowledge and skills that make them employable, both within the country and abroad.7 1.2 Description of the TLP Program and the TLP-SP Project 1.2.1 The TLP Program As a step towards enhancing human capital in Kosovo, building a cadre of skilled professionals capable of addressing Kosovo’s economic and democratic development priorities and addressing some of the gaps in the education system, USAID and the Government of Kosovo (GOK) have jointly funded the multi-pronged Transformational Leadership Program (TLP). The TLP program seeks to upgrade the knowledge and skills of talented professionals through short- and long-term educational opportunities and increase the capacity of select faculties at institutions of higher learning in Kosovo through a comprehensive, multi-faceted approach comprised of five separate but interrelated activities (components).8 These components, executed by a mix of implementing partners, are: 1. U.S. Scholarships (Master’s degree and professional certificates) – competitively awarded and open to all citizens of Kosovo. 2. Immersion Program to strengthen the skills of potential graduate students (especially those from marginalized groups). 3. Citizens Corps – a program to help and ensure that returned Master’s graduates fulfill their two￾year “give back” commitment through either by working in the public sector and/or through approved community service. 4. Short-term public service training in Kosovo to upgrade skills of current public service staff in key Kosovar government institutions 5. University Partnerships – partnerships between U.S. and Kosovar universities (most notably the University of Prishtina) to upgrade university management and pedagogy in selected faculties. 6 The World Bank Group (2018). The World Bank in Kosovo – Overview. https://www.worldbank.org/en/country/kosovo/ overview accessed on August 25, 2019 7 Kosovo Ministry of Education, Science and Technology (2016). Kosovo Education Strategic Plan 2017-2021 accessed at http://www.kryeministriks.net/repository/docs/Kosovo_Education_Strategic_Plan. Pdf on August 25, 2019 8 Description from Contract document: AID-167-TO-15-00002– TLP Impact and Program Evaluation USAID Impact Evaluation of the TLP Scholarships Program– Final Report 3 1.2.2 The TLP-SP Project Two of the five TLP components – the Scholarships component and the University Partnerships component, collectively referred to as TLP Scholarship and University Partnerships (henceforth TLP-SP project) – were implemented under a cooperative agreement between USAID and World Learning Inc. (WL) and its partners, Arizona State University (ASU), University of Minnesota (UMN), Dartmouth College (DC) and Indiana University (IU).9 Funded by USAID and GOK, in particular by Ministry of Education, Science, and Technology (MEST), the TLP-SP project was implemented over a period of five years from February 26, 2014 to June 30, 2019, and had the following three objectives: ● Objective 1: Coordinated and collaborative oversight of the transformational leadership initiative through an Advisory Committee (AC) ● Objective 2: U.S. post-graduate degrees (Master’s degrees) and professional certifications for selected citizens of Kosovo (Scholarships component) ● Objective 3. Improved university management and pedagogy in selected faculties of the University of Prishtina (UP) and/or other universities through partnerships and exchanges with U.S. Universities (University Partnerships component) The Scholarships and the Partnerships components are the subject of this evaluation. This report focuses on the TLP Scholarships (for Master’s studies) component, briefly described below. 1.2.3 The TLP Scholarships Component With the overall goal of creating a cadre of talented professionals, through U.S. Master’s degree programs as well as shorter-term targeted courses, who can more quickly bring critical leadership and innovation skills to the largest number of Kosovars across all segments of the society, thereby helping Kosovo address the demands of a modern market economy and bring transformational change to the country, the Scholarships component of TLP-SP project (henceforth called the TLP Scholarships program) was designed to provide: • 160-190 scholarships to citizens and residents of Kosovo, selected over a period of three years from 2014 to 2016, to obtain Master’s degrees at U.S. universities and colleges. At least 50 percent of the scholarships are expected to be provided to women and other underserved populations; (185 such scholarships have been awarded in three cohorts) and • Scholarships for shorter-term targeted programs that award professional certifications to help create an environment open to reform. The Scholarships program has awarded scholarships to 185 Kosovars since 2014 for a Master’s degree program in the U.S. The scholarships were awarded in three rounds: Cohort 1 (2014) had 46 recipients, Cohort 2 (2015) 84 recipients, and Cohort 3 (2016) 55 recipients.10 While the proportions varied by cohort, overall female recipients made up 51 percent of the scholarship recipients and when combined with those from ethnic minorities, living in rural areas, LGBT and with physical disabilities, 66 percent of the scholarship recipients in this cohort came from under-represented communities (women, ethnic minorities, living in rural areas, LGBTQ and physical disability). 9 To help the TLP-SP project better respond to the realities on the ground, some of its planned activities were modified during implementation to: (i) limit the activities to focus primarily on UP; and (ii) revisit the planned support for the implementation of select priorities from the UP Action Plan of 2013 to make it more relevant (see Annex 6 for details). 10 Please note these are the final numbers as provided by the TLP-SP project. USAID Impact Evaluation of the TLP Scholarships Program– Final Report 4 Table 1. Demographic description of scholarship recipients, by Cohorts Demographic variables Cohort 1 Cohort 2 Cohort 3 Total Total Awards N 46 84 55 185 Percent Gender Female 54 42 64 51 Male 46 58 36 49 Ethnicity Albanian 91 95 91 93 Serbian 4 2 5 4 Other (Gorani, Roma, Turkish, Bosnian) 4 2 4 3 Disadvantaged/ underrepresented (LGBT, physical disability; rural, ethnic minority) 22 23 29 24 Above + female 70 55 80 66 Note: Data from analysis of TLP-SP database on Scholarship recipients. The program assisted the recipients with placement in U.S. universities and colleges and the scholarships covered all expenses for the recipients for a maximum of two years with the stipulation that upon completion of their Master’s program, the recipients would return to Kosovo for a period of two years and volunteer/work in the government or private sector.11 The scholarship recipients were placed in 62 U.S. universities and colleges. Depending on the field of study, the Master’s programs ranged from 9 months to 2 years in duration, as a result of which the participants have returned to Kosovo at various points. All scholarship recipients were required to complete 400 hours of volunteer work, with an exception made for those employed in the government. They were supported by Citizen Corp, a program under the TLP umbrella, to identify these volunteer opportunities, a support that was well received by all participants. 1.3 Organization of the report The rest of the report is organized as follows. Section 2 presents the objectives of the evaluation and outlines the approach used to conduct the study. Sections 3 to 6 discuss the four evaluation questions that guide the overall evaluation of the Scholarships component. Section 7 presents the conclusions and recommendations for future programs. The report has five annexes, which contain: (i) scope of work for this evaluation; (ii) evaluation methodology; (iii) list of people interviewed and focus groups conducted for the endline study; (iv) sources of data and a list of documents reviewed; and (v) select data tables expanding upon data included in the body of the report. 11 There have been a few exceptions to this stipulation with a small number of scholarship recipients permitted to stay on in the US for the completion of further studies, USAID Impact Evaluation of the TLP Scholarships Program– Final Report 5 2 STUDY OBJECTIVES AND APPROACH As stated in the previous section, USAID/Kosovo has contracted with IBI International (IBI) to conduct a concurrent evaluation of two components of TLP-SP project – an impact evaluation of the Scholarships component and a performance evaluation of the University Partnerships component – implemented by WL and its partners. Annex 1 contains the scope of work.12 The TLP scholarships program, described in section 1.2, provided scholarships for Masters studies and for professional certificate programs from US universities and colleges. This impact evaluation, as per the scope of work, is focused only on the recipients of scholarships awarded to Kosovars for the completion of Master’s degree programs in U.S. universities and colleges and followed the three cohorts of scholarship recipients from the time they were accepted into the program until a time two years after their return to Kosovo. It also followed a comparison group over the same time period to provide the counterfactual. 2.1 Evaluation Objective The evaluation, comprising a series of studies to assess the overall impact of the Scholarship Program, is designed to: (i) provide stakeholders with an understanding of how various investments in scholarships (for Master’s level studies in US Universities and colleges) under TLP-SP increased the knowledge and skills of the individual participants; and (ii) determine the effectiveness of the program in training participants to act as change agents who can affect social and economic development upon their return to Kosovo and transition to work. The results of this impact evaluation and the lessons learned will constitute a potential input to inform future decision-making by the stakeholders, including the GOK (Ministry of Education, and other Ministries), University of Prishtina (UP), USAID and other donors. 2.2 Evaluation Questions The evaluation of TLP scholarship program is guided by the following sets of questions as posed by USAID in the evaluation scope of work (See Annex 1): a. How have the participants utilized their training since the completion of their studies? (If not, why not?) What significant constraints or barriers, if any, do the returned participants face in fully utilizing their new knowledge/skills? b. In what ways are U.S.-returned beneficiaries contributing to Kosovo’s development and how have the skills acquired in the U.S. added to that contribution, particularly at the organizational level? c. In what ways does the perception of TLP scholarship recipients as leaders change over the course of the program? d. Given a-c above, how and to what extent have the experiences of the returned scholarship recipients differed from their non-recipient peers? 12 The evaluation focuses only on the recipients of the Master’s scholarship and does not include the beneficiaries of the scholarships for professional certificates, per TLP Impact and Program Evaluation Contract #: AID-167-TO-15-00002 USAID Impact Evaluation of the TLP Scholarships Program– Final Report 6 2.3 Study Approach This study attempts to answer each of the evaluation questions listed above by following the progress for scholarship recipients over a period of four years, from the time they were awarded a scholarship to a point two years after their return to Kosovo, using a quasi-experimental design and a variety of qualitative and quantitative methods.13 The theoretical construct underlying the TLP Scholarships program is that as a result of their participation in the program and the experience gained through the Master’s level studies in U.S. universities and colleges, participants will learn new skills that will contribute to the leadership and critical thinking needed to foster innovation and economic growth in Kosovo. This principle has also shaped the design of this evaluation. A distinctive feature of this study is the implied focus on causality and attribution. In order to estimate the causal effect or impact of the Scholarships program on this expected outcome, it is necessary to estimate the counterfactual, i.e., what the outcome would have been for program participants if they had not participated in the program. This was done by identifying a control group to estimate what would have happened to the scholarship recipients without the program and following this group over the same time period as the scholarship recipients. Sample Selection All (185) of the scholarship recipients from the three cohorts were included in the sample for the evaluation. A purposive comparison/control group for all three cohorts of scholarship recipients was selected from among the applicants to the TLP Scholarships program. The sample was selected from among those who met the TLP-SP project criteria for applicants and qualified for the interview stage of the selection process but were not selected for the scholarship or were selected but did not join for any reason. The comparison group was oversampled at baseline to account for anticipated attrition over time and ensure there would be at least half as many respondents from the comparison group as for the scholarship recipients’ group at endline. The midline and endline data for the comparison group were collected from all those that responded to the baseline survey. Employers of the TLP scholarship recipients who had at least three scholarship recipients on their staff were selected for interviews and informational meetings to get a broader perspective on the effects of the program rather than impressions formed on the basis of a single employee. Non-TLP employees of the organization were used as a proxy for the comparison group. Data Collection The assessment methodology employed follows the logic of the scholarships program and examines all aspects of this component to determine progress and impact. A mixture of qualitative and quantitative methods was employed for the collection of primary and secondary data. • Primary data for the evaluation were collected from: (i) web-based surveys of scholarship recipients and their comparison group; (ii) focus groups with scholarship recipients; and (iii) interviews and informational meetings with a variety of stakeholders, including TLP-SP staff, MEST, USAID, senior 13 See USAID (2013): Conducting Mixed-Method Evaluations. Accessed on November 29, 2015 at https://www.usaid.gov/sites/default/files/documents/1870/Mixed_Methods_Evaluations_Technical_Note.pdf USAID Impact Evaluation of the TLP Scholarships Program– Final Report 7 management at the University of Prishtina (UP), implementers of the TLP Citizen Corp program and select employers of the scholarship recipients (see above). • Secondary data was collected from a review of project records and databases already in place for tracking TLP scholarship applications and project monitoring and evaluation (M&E). Documents such as the TLP-SP quarterly and annual reports, work plans and other technical reports were also reviewed to get a sense of the program. Data on activities undertaken by the scholarship recipients during their Master’s program as well as their volunteer work upon returning to Kosovo was obtained from the participants themselves and through a review of the TLP-SP website and records maintained by the TLP Citizen Corp program. Using similar but separate web-based surveys for scholarship recipients and their comparison group, data were collected on the following: general demographics; participants’ level of education; current location and employment status; self-perception of skills developed and utilization as well as any barriers to use of skills; self-perception as leaders and their contribution to Kosovo. Additional questions were included for each group to explore factors not common between them, such as experience with the Scholarships program or return to Kosovo after their Master’s program for scholarship recipients. Survey data To assess the impact of the Scholarships program, each cohort was evaluated thrice: (i) baseline – upon start of the Master’s program; (ii) midterm – upon completion of Master’s program and return to Kosovo; and (iii) endline – at a point two years after completion of the Master’s program. Baseline data was collected on beneficiaries as well as control groups to assess the effectiveness of the program, and by defining measures of the success of the Scholarships program, help focus the program on intended results.14 Baseline studies were conducted in 2015 for cohorts 1 and 2 and in 2016 for cohort 3. Midline studies were conducted between 2016 and 2018. All scholarship recipients that had completed their program and returned, regardless of cohort were included in the study. Similarly, endline studies were conducted between 2018 and 2019, and included everyone at a point two years post completion of Master’s program. As described above, data from the comparison group were also collected at the same points. Incentives, in the form of phone credits, were provided to the comparison group to encourage completion of surveys at all three points (baseline, midline and endline), and to the scholarship group at endline. To allow for comparability of data from baseline to midline and again to endline, the survey instruments and protocols used at baseline were used at midline and endline. They were revised and updated to include any additional questions needed to gather information about the impact of the program over time. Table 2 presents the survey response rates by year for the baseline, midline and endline studies. 14 The baseline study for Cohorts 1(2014) and 2 (2015) was conducted in 2015, and for Cohort 3 (2016) in 2016. Midterm studies for Cohorts 1, 2 and 3 were conducted in 2016, 2017 and 2018 respectively, after majority of the cohort had completed their program and returned to Kosovo. This report is focused on the endline studies for cohorts 1,2, and 3 and comparison of the findings to baseline and midline as needed. USAID Impact Evaluation of the TLP Scholarships Program– Final Report 8 Table 2. Sample size and response rates by cohort and point of data collection Scholarship Recipients Comparison Group Baseline Midline*** Endline*** Baseline Midline Endline N Response rate N Response rate N Response rate N Response rate N Response rate N Response rate 2014 46 96% 60 90% 60 82% 40 40% 40 50% 32 66% 2015 91* 79% 76 86% 76 86% 70 69% 68 59% 55 58% 2016 55 89% 45 80% 45 91% 95 83% 90 59% 82 65% Total 192 86% 181 86% 181** 85% 205 70% 198 57% 169 63% *some scholarship recipients from cohort 2 dropped out of the program **four scholarship recipients were unable to complete the program for various reasons ***data at midline was collected from all scholarship recipients who had completed their program irrespective of cohort, as such cohort one included some early completes from cohort 2 and so on In addition to surveys, focus groups were conducted with scholarship recipients and interviews were conducted with a select group of employers and other stakeholders. Semi-structured focus group and interview guides were developed as appropriate to guide focus groups with scholarship recipients and interviews and informational meetings held with other stakeholders listed above. The focus groups, which provided an insight into the program from the participants’ perspective, and helped clarify the challenges participants faced in using skills learned in the program and how they believed they were exercising leadership. The interviews and informational meetings with various stakeholders also elicited the stakeholders’ perception of how the program had helped the scholarship recipients improve their skills. The interviews and focus groups were also an opportunity for the participants to share their thoughts on how the program could be improved and made more relevant. The endline data for all three cohorts of scholarship recipients who had completed their Master’s program and returned to Kosovo as per the terms of their scholarship was collected during two time periods: October 2018 to March 2019 for Cohort 1 and August 2019 to November 2019 for Cohorts 2 and 3, using the same methodology as employed in the baseline and midline studies for all three cohorts.15 Following the protocols established for the collection of data at baseline, all 181 scholarship recipients from the three cohorts that completed the Master’s program were invited to complete the endline survey, with an overall response rate of 85 percent (49 percent female). For the comparison group, the endline sample of 169 was based on the midline sample after excluding those who had asked to be removed from the sample or for whom valid contact information was not available. The comparison group had a response rate of 63 percent (64 percent female).16 In addition, focus groups and individual interviews were conducted for the endline study in Prishtina with 47 scholarship recipients from all three cohorts (25 percent of the total; 28 males and 19 females), which yielded valuable data on the participants’ experience during their Master’s degree program and re-entry into Kosovo as well as on their experience with working in Kosovo and contributing 15 See the following reports submitted to USAID by IBI International under the Transformational Leadership Impact and Performance Evaluation Project: Impact Evaluation of the Scholarship Program. Baseline Study Cohort 3. January 2016; Impact Evaluation of the Scholarship Program. Midterm Study Cohort 1 Scholarship Recipients. March 2017; and Impact Evaluation of the Scholarship Program. Midterm Study Cohort 2 Scholarship Recipients. March 2018. 16 Follow-up to survey invites included multiple reminders and phone calls. Both the comparison and the scholarship recipients’ group were provided token incentives in the form of phone charge as a way to encourage them to complete the survey at endline. The incentives for the comparison group were larger than those for the scholarship recipients’ group. USAID Impact Evaluation of the TLP Scholarships Program– Final Report 9 to its development.17 Data were also collected through interviews with 17 employers of scholarship recipients (each with at least 3 TLP scholars on their pay roll, for a total of over 50 scholars), and other stakeholders including TLP-SP project staff. Data Analysis Quantitative data from web-based surveys were cleaned and analyzed using univariate and multivariate analysis. Data collected at endline are compared with data from baseline and midline as well as the comparison group to respond to each of the study questions and assess the contributions of the scholarship program, the changes in status of participants and their contribution to Kosovo. To the extent possible all data have been disaggregated by gender and cohort. Qualitative data were analyzed for themes and patterns and incorporated in the analysis. Annexes 2 and 3 contain the list of key people interviewed and documents reviewed, respectively. Annex 4 contains the survey questionnaires. 2.4 Limitations of the study Even though they do not materially affect the outcome of the evaluation, there are a few caveats and limitations that must be kept in mind as we review the results from this study. 1. The assessment of the contribution of the scholarship recipients to Kosovo is based on self-reports and information provided by the respondents themselves. Even though this was triangulated with information from other stakeholders, the bias due to self-reporting remains. 2. Given that most of the scholarship recipients are employed, at endline it was very difficult to get them to participate in focus groups. While the issue was resolved by offering multiple opportunities for focus groups as well as accommodating requests for individual in-person or phone interviews at times convenient to the respondents, there is some concern of self-selection in the sample and the possibility of a bias. 3. There are concerns regarding: (i) low response rates for the comparison group in the follow-on survey; and (ii) the possibility of survey fatigue in scholarship recipients as a result of surveys from various TLP partners. In order to overcome these potential challenges, incentives were offered to both the scholarship recipients (15 Euros per completed survey) and the comparison group (35 Euros). But the concern remains. 17 The criteria for selection included cohort, duration of Master’s program and Serbian participants living in the North. USAID Impact Evaluation of the TLP Scholarships Program– Final Report 10 3 EVALUATION QUESTION 1: USE OF SKILLS AND TRAINING This section answers the first set of evaluation questions as posed in the scope of work: How have the participants utilized their training since the completion of their studies? (If not, why not?) What significant constraints or barriers, if any, do the returned participants face in fully utilizing their new knowledge/skills? One of the theoretical constructs underlying the Scholarships program was that as a result of completing their Master’s program in a U.S. University or college, the scholarship recipients would have the opportunity for exposure to new skills and new ways of doing things and upon their return would bring these skills with them to Kosovo. It was expected that this would be a quick way of filling the gap in skills within the country. As stated previously, 181 of the 185 scholarship recipients completed their Master’s degree program (four dropped out for various reasons). As per the terms of the scholarship, all scholarship recipients have since returned to Kosovo. This section provides a brief description of what the scholarship recipients are doing at endline and discusses how they are utilizing their training and the skills developed. 3.1 Current status of Scholarship recipients Most scholarship recipients returned to Kosovo between 2016 and 2018 and at the time of the endline studies had been in the country for anywhere from 12 months (for Cohort 3) to a little over two years (for Cohort 1). Table 3. Employment status of scholarship recipients at endline Total (N-154) Cohort Gender Employment status Cohort 1 (N=40) Cohort 2 (N=68) Cohort 3 (N=46) Female (N=75) Male (N=79) Percent* Percent* Percent* Employed (includes self-employed) 88 88 76 85 80 84 Self-employed** 15 15 13 9 9 15 Unemployed and looking for a job 5 5 12 2 9 6 Taking a break, not looking for a job 0 0 0 2 0 1 Volunteering 5 5 7 9 0 0 Enrolled in an academic program 8 8 6 2 8 3 *Note: Respondents could select more than one category; **included as a separate category to show extent of self-employment Survey data show that 136 of the 154 scholarship recipients18 (88 percent) who completed the survey are employed (including self-employed) and the rest are enrolled in academic programs (8 percent) or volunteering (8 percent) to complete their scholarship requirements (respondents could select more than one category).19 Only a small proportion (5 percent) are unemployed and looking for a job. (Table 3). Survey respondents also indicated that those who are just returning (Cohort 3) are beginning to reintegrate to life in Kosovo, completing volunteer activities (to complete their program obligations) and looking for jobs. Participants reported that they were living in Prishtina or other towns in Kosovo (73 percent and 27 percent respectively). Data from focus groups supported these findings with participants reporting that since several 18 All 181 of the scholarship recipients were invited to take the survey, 154 completed it (85 percent response rate). 19 All TLP scholarship recipients were required to complete 400 hours of volunteer work as a part of a mandatory give back to the community. USAID Impact Evaluation of the TLP Scholarships Program– Final Report 11 of them were still looking for a position or, in case of those employed, a better position, living in cities and towns provided them with a larger number of employment opportunities. Several also reported that they were living in other towns but commuting to Prishtina for work. 3.1.1 Employment Most of those employed are engaged in full time positions (91 percent), and over half are working in the private sector (49 percent) or with NGOs (23 percent). A few individuals are working with the government (11 percent) or with multinational organizations (12 percent). A small proportion are working for public sector organizations such as universities (4 percent) or are free-lancing as consultants (7 percent). Table 4 has the details. Many TLP scholarship recipients working in the private sector are with smaller organizations and start-ups (including some with their own). Table 4. Primary place of employment for scholarship recipients at endline Employment status Cohort 1 (N=38) Cohort 2 (N=56) Cohort 3 (N=42) Total (N-136) Percent (col. ppercent) Government 8 9 17 11 Public organization including Universities 11 4 0 4 Private organization including Universities 39 57 48 49 NGO 24 23 21 23 Multinational/donor organization 18 7 12 12 Other 0 0 2 1 Of those employed, 41 percent reported that they head their organization or are in a senior/leadership position and 35 percent have a mid-level supervisory position. Only 11 percent of all respondents said they were in an entry level position (4 male and 11 female) in their organization. Endline survey data indicate that of those working, only a handful (5 percent) of the scholarship recipients are in the same job as before joining their Master’s program. Most have a position that is very different from what they had prior to the Master’s program (80 percent) in a new organization, and about 40 percent are in a position that is senior to their previous position. Even though many of them had gone back to their previous job upon return to Kosovo (see discussion below), at endline, two thirds of the survey respondents had changed jobs within the past year or so for a senior position or a different position with another organization (32 and 52 percent respectively) with a higher salary and/or additional responsibilities. Scholarship recipients from Cohorts 2 and 3 appear to be more mobile than those from Cohort 1. A majority of survey respondents (92 percent, midline data) reported that they had found their first position using their own networks while a small proportion (4 percent) reported they had received help from Citizen Corps. Focus group discussions with the scholarship recipients and survey responses indicate that majority of the scholarship recipients had found a job, though not necessarily the best one, within about three months of returning to Kosovo. Most others found a job within about six months. Several participants reported that either due to lack of better opportunities or for reasons of expediency they had gone back to working for the organizations they were working with prior to accepting the TLP scholarship. Participants, especially those from Cohorts 2 and 3, stated that they had learned from the experience of their colleagues from USAID Impact Evaluation of the TLP Scholarships Program– Final Report 12 Cohort 1 who had difficulty with finding a job, and had accepted whatever position was available to them upon return to Kosovo. Over time, they had moved to a better position more in line with their skills and interests. Regardless of whether they were in the same or in a new organization, most respondents agreed (strongly agreed or agreed) that their current position was better paid and included greater responsibility (82 and 73 percent respectively) than positions they held prior to completing the Master’s program. Almost two thirds of the respondents (62 percent) reported that they were very satisfied with their current positions, while a quarter (24 percent) said that this was an interim position to tide them over while they were looking for a better position. While about a third of the scholarship recipients are working in fields directly related to their field of study (for example graduates with a business management major working in marketing or law graduates working for a law firm), the rest are using their skills to support other fields (for example law graduates working in construction or in the field of finance, graduates with degrees in economics working for development companies, etc.). The positions held by the scholarship recipients, especially those at the smaller organizations, require them to wear multiple hats and take on multiple tasks in the organization. A little less than half of those employed indicated their responsibilities included mentoring others (48 percent), while over forty percent were engaged in conducting research and working on multi-disciplinary teams (42 and 40 percent respectively). Others reported their responsibilities include, inter alia, program administration (36 percent), product development (29 percent), strategy development (33 percent), new business development and grant writing (33 and 25 percent respectively). Interviews with employers and focus groups with the scholarship recipients supported these findings. 3.1.2 Volunteer work At endline, a majority (95 percent ) of the TLP scholarship recipients had completed their mandatory 400 hours of volunteer work upon return to Kosovo.20 With a few exceptions, in which the scholarship recipients decided to complete their volunteer hours at a later date, most scholarship recipients tried to complete their volunteer hours immediately upon return, and prior to starting full-time work. They found the volunteer positions through their own networks as well as with help from TLP Citizen Corps program which partnered with 178 public and private institutions, including NGOs, private sector, and public and private universities for this purpose. 21 While many were able to complete their required hours at one place, others did so by combining anywhere from 1-4 shorter term assignments (minimum of 50 hours). Data from a survey of TLP scholarship recipients showed that a majority had volunteered with NGOs and multinational donor organizations (48 percent), while the rest had volunteered time with public and private universities (12 and 16 percent respectively) and with private organizations (15 percent). Their responsibilities included providing hands-on services and working in entry level positions (46 percent) and supporting research activities, writing grants or background papers for research papers/policy/strategy development (31 20 The TLP scholarship agreements mandated that recipients must complete 400 hours of volunteer activities upon their return, preferably in the government or the public sector. For those working in the government, this requirement could be waived. 21 USAID has contracted with Leadership and Growth Council (LGC) for the establishment of the TLP Citizen Corps program designed to support returning graduates with placement in volunteer opportunities in Kosovo. The program has active partnerships with 178 organizations including: media agencies, public and private sector organizations including universities, and NGOs and multinational organizations where it can place the returned scholarship recipients. The placements can range from 50- 400 hours in duration and the hours are cumulative. USAID Impact Evaluation of the TLP Scholarships Program– Final Report 13 percent). Some were also engaged in providing administrative support for programs. A large proportion of the Scholarship recipients reported that their volunteer activities were related to their field of study (86 percent) and that they could utilize the skills learned as a result of the TLP program (92 percent) and share skills with their colleagues (76 percent). Thirty percent of the survey respondents hoped that the volunteer position would be a stepping-stone to future employment . Focus group discussions supported these findings. The participants reported that they had volunteered with public and private universities, think tanks, and NGOs working on climate change, domestic violence, gender issues, and on teaching coding to girls. Several were working as teaching assistants and research assistants, consultants and legal advisors and mentoring those planning start-ups. Depending on where they were placed, their responsibilities included writing papers, preparing reports, doing data analysis, supporting development of business plans, and other hands-on services. Several reported working as project coordinators. A few of them also reported that while they had officially completed their volunteer requirements, they were continuing with the work started as they found it to be very interesting and valuable for the community/organization. For several, particularly from Cohort 1, these volunteer opportunities also turned into full time positions. 3.2 Training and skills development Data from the survey respondents as well as a review of the TLP-SP project reports indicate that the TLP Scholarship recipients completed their Master’s Degrees from U.S. universities and colleges in variety of subjects, the most popular being Business Management (24 percent), Public Administration (23 percent), Economics (12 percent), Education (11 percent) and Engineering (10 percent). Table 5 shows the details. While the programs in Law were less than a year in duration, most Master’s programs were 18 months (33 percent) to 24 months (58 percent) in duration, and about a third (33 percent) included an internship component. Table 5. Master’s Degree pursued, Scholarship Recipients Cohorts 1, 2 and 3 Major of Study Cohort1 Cohort 2 Cohort 3 Total N 45 83 53 181* Agriculture Including Agribusiness and management 2 1 0 1 Business Management Majors in Business Administration; Business Enterprise and Business Analytics, Aviation management 20 27 23 24 Economics and Finance Including majors in Economics and Accounting 18 11 21 16 Education Majors in Education, Educational Policy, Linguistics and Teaching English 13 10 11 11 Engineering Including Majors in Software Engineering, Computer Science, Architecture and Sustainable Urban Planning, 4 13 11 10 Gender and Interdisciplinary Studies Including majors in Gender and interdisciplinary studies and Global and international studies 2 1 6 3 Information Systems 2 6 2 4 USAID Impact Evaluation of the TLP Scholarships Program– Final Report 14 Major of Study Cohort1 Cohort 2 Cohort 3 Total Majors in Information Systems, Interactive Entertainment, Cyber Forensics and Security, Information Technology and Management Public Administration Including Majors in Public Administration, International Studies, Environmental Policy, and Social enterprise 31 20 19 23 Rule of Law 7 11 8 9 *As noted previously, even though 185 scholarships were awarded, for various reasons four dropped out and 181 completed their Master’s degrees. The broad areas of study in themselves offer a wide range of added skills to Kosovo. In addition, an examination of the data revealed that many of programs were multi-disciplinary and included studies on topics not available in Kosovo, thereby bringing in new expertise to the country. For example, the broad areas of Business Management included specialization in areas such as Social Enterprise, Agribusiness, Business Enterprise and Business Analytics, and Aviation Management; Engineering included the more traditional areas of Software Engineering, Computer Science, Information systems, as well as studies in new areas such as Interactive Entertainment, Cyber Forensics and Security, Information Technology and Management; Education included Education policy, Linguistics and Teaching English, Theater Education, etc. while Public Administration included specialization in areas such as Environmental Policy, International Security, International Peace and Conflict Resolution, and International Relations (Table 5). While only about a third (31 percent) of the survey respondents indicated that their Master’s program included a practicum or internship component, a majority (82 percent) reported participating in some form of volunteer activities. This was especially true for most scholarship recipients who had enrolled in the longer programs (18 months or more). Several participants in focus groups also reported being placed as teaching assistants to professors or as administrative assistants in the library or administrative offices of their school (university), which offered them hands-on experience with teaching, grading, or doing office work, even though it could not be categorized as either a formal internship/practicum or community service. At midline, soon after completing the Master’s program, scholarship recipients reported that the practice￾oriented programs helped them develop critical thinking, analytical and writing skills as well as leadership skills that they could use in the field of work and day-to-day life. Almost all participants (96 percent agree or strongly agree) reported that they had an in-depth understanding and methodological knowledge in their field of study and felt empowered to be able to share it with others (91 percent). Over 90 percent (agree or strongly agree) reported that because of their participation in the Master’s program, they could see the “big picture”, work in multi-disciplinary teams and were able to communicate effectively. They also reported that they could see themselves in a leadership role or taking on greater responsibility (89 and 88 percent respectively) and leading teams that include international experts in their field and beyond (81 percent). They saw themselves as the ones building consensus around areas of importance for their organization (88 percent). Participants also reported having developed analytical skills, as well as skills in problem solving, conducting research in an ethical manner, organizing ideas and finding information, and writing efficiently and clearly – all skills that are invaluable in everyday life. Tables 5 and 6 have the details. The survey data were supported by focus groups with participants. They indicated that the Master’s programs were of “high quality,” useful and flexible, in that very few of the courses were compulsory and they could choose from several others based on their interests. The participants found the programs to be USAID Impact Evaluation of the TLP Scholarships Program– Final Report 15 very intense, but the coursework provided them with exposure to new fields and skills. They valued the fact that the Master’s program provided them with new ways of thinking about things and doing analysis, an opportunity to learn writing skills as well as practice in making presentations. Several participants reported they received exposure to analytical software other than excel (SPSS, R Studies, etc.). Exposure to the case study method, talking to people at events, making elevator speeches and working in groups were all well￾received by the participants as useful skills for future use. Several participants reported being able to attend conferences and seminars organized at their university and elsewhere and being exposed to cutting-edge research and knowhow. (See Tables 6 and 7). Table 6. Skills developed by Scholarship Recipients during their Master’s program N Strongly Disagree Disagree Neutral Agree Strongly Agree Total Percent (row percent) Organizing ideas 124 2 0 6 48 44 100 Finding information 124 2 1 4 39 55 100 Writing skills: writing productively, efficiently and clearly 124 1 1 4 40 55 100 Reading skills 124 1 1 7 36 55 100 Creativity 124 1 3 17 37 42 100 Ability to work with numbers and graphs 124 2 7 12 40 40 100 Ability to conduct research 124 1 1 6 31 60 100 Ability to ask questions 124 2 2 10 32 54 100 Methodological knowledge 124 1 1 6 42 51 100 Analytical skills 124 1 0 5 40 54 100 Problem solving skills 124 1 1 6 39 54 100 Time management 123 1 1 15 39 45 100 Using digital and multimedia tools 124 1 6 18 37 38 100 Ethical research 123 1 2 8 37 52 100 Ref: Midline survey of scholarship recipients USAID Impact Evaluation of the TLP Scholarships Program– Final Report 16 Table 7. Skills developed by Scholarship Recipients during their Master’s program N Strongly Disagree Disagree Neutral Agree Strongly Agree Total Percent (row percent) I have developed critical thinking skills 138 1 0 6 36 57 100 I have developed an in depth understanding in my area of studies 138 1 1 2 33 63 100 I am encouraged to take more leadership roles in any institution and context 137 1 1 9 33 55 100 I now see myself as a person with potential to take leadership positions 139 1 1 9 34 55 100 I am now encouraged to take initiatives in any work environment 138 1 2 8 36 53 100 I feel empowered to transfer the knowledge in my field of study to others 138 3 1 5 34 57 100 I feel equipped with the sufficient level of knowledge in the area of study to be able to transfer skills to others 137 1 1 6 42 50 100 I feel prepared to be able to work in an institution under the leadership of others 138 1 1 9 46 43 100 I am able to communicate effectively with others inside and outside my field 138 1 1 3 36 59 100 I can work effectively in multi-disciplinary teams 138 1 1 4 32 63 100 I can lead teams that include international scientists/think tanks inside/ outside my field 138 1 1 17 36 46 100 I am able to build consensus around issues important to my organization 136 2 1 9 40 48 100 I am able to see the “big picture” 138 1 1 6 30 62 100 Ref: Midline survey of scholarship recipients 3.3 Utilization of training and skills Scholarship recipients reported using the skills developed as a result of their Master’s program – in critical thinking, in-depth knowledge of their subject, communication, leading interdisciplinary teams, research and analytical skills, organizing ideas, creativity, problem solving, etc. (see Section 3.2 - Training and Skills Development) – both in their place of employment as well as in their volunteer work. Survey data for those employed indicated that about three-quarters of the respondents had taken on a leadership role (80 percent, agree and strongly agree), and had introduced new ways of thinking and problem solving especially at the organizational level (83 percent). They reported using participatory methods for consensus building (75 percent) and discussion as ways to resolve problems (79 percent). They also reported using ethical ways to conduct research (86 percent), and the findings from research and data analysis to support advocacy (80 percent) and policy making (76 percent). About half reported leading interdisciplinary teams and using digital and multimedia tools to support advocacy and policy making (54 and 52 percent respectively). Almost all respondents reported using their training in organizing ideas (84 percent) and preparing concise well-written reports (90 percent). Tables 8, 9 and 10 have the details. In addition, most respondents (between 80-90 percent) said they were able to use critical thinking and research skills in their work and were proficient at finding information and problem solving. They also USAID Impact Evaluation of the TLP Scholarships Program– Final Report 17 reported that the ability to be creative in their work, to ask questions (without fear of reprisal) and time management, learned as a part of their Master’s program, were useful in their day-to-day work. (Tables 8, 9 and 10). Approximately three quarters of all survey respondents agreed that as a result of their participation in the Master’s program, they could see the “big picture” (86 percent) and are able to take the initiative and a leadership role in any work situation (87 and 81 percent respectively). They see themselves as team players (81 percent) who are able to work effectively in multidisciplinary teams and lead teams that include international experts in their field and beyond (86 and 71 percent). Over three quarters of the scholarship recipients indicated that they were able to build consensus around areas of importance for their organization (77 percent) and communicate effectively with others in their field or from other fields (81 percent). (Tables 8, 9 and 10). Qualitative data from focus groups with scholarship recipients and interviews with select employers of TLP graduates supported the survey data. Several focus group participants said that as a result of the exposure provided by the Master’s program they were able to use new ways of thinking about things and doing analysis, which was helping them get ahead at the workplace. They reported that unlike the education system in Kosovo, in the US system the students were encouraged to ask questions in class and penalized for not asking questions. This had helped them to get out of their shell and feel comfortable in asking questions. They went on to say that as a result of this exposure, they had also learned that the “professor did not always have all the answers, and it was okay for a leader to not know everything.” Most of the focus group participants said that the soft skills developed during their Master’s program were serving them well. Their skills in working in teams, building consensus, time management and ability to deliver on time had helped them get ahead within their organizations. While some reported feeling awkward that they were now managing people older than themselves, most said that now that they were in a senior or leadership position within their organization and were able to introduce new and ethical ways of thinking and doing things. Interviews with select employers confirmed students’ self-reports. Several employers said that the TLP scholars stood apart within the organization and quickly made themselves indispensable and were only now (two years after completing their Master’s program) beginning to reach their full potential. They went on to say that even though the TLP scholars needed training (like all other new hires) to learn the ways of the organization they were working with, the TLP scholars stood apart due to their ability to learn what they were required to do very quickly. The employers also suggested that the TLP scholars had made rapid progress in their careers because they had a better work ethic and “were likely to complete things to the end, no matter what it takes,” were more likely to take initiative and to volunteer for activities, and they attributed these qualities to the TLP Master’s training. In addition, employers also suggested that the TLP scholars’ ability to “think outside the box,” use soft skills, write well and fluency in English made them very valuable to the employers and rise quickly to positions of greater responsibility within the organization. Several said that the TLPs commanded higher salaries than the graduates from Kosovar Universities, and one went to the extent of saying they had doubled the salary of a TLP graduate working for them to try and keep the employee. Some of the employers who had been the supervisors of the TLP graduates before and after the TLP program reported noticing a big change in the graduates after the program. The employers said that the USAID Impact Evaluation of the TLP Scholarships Program– Final Report 18 scholarship recipients showed a lot of change in the confidence and ability to conduct analysis and present findings, as well as in the communication and writing skills. They were more open, better able to ask questions, and in general were quicker at whatever they did. They were better able to organize their thoughts, manage deadlines and turn things around rapidly. Due to their ability to manage time, the employers felt comfortable in leaving the TLP scholars in charge. Several employers, and not just the ones who had supervised TLP graduates previously, felt that TLP scholars who had a job before they received the scholarship were likely to do better than those who went to the U.S. directly after completing their undergraduate programs. The employers said that those with prior work experience had gained more from the scholarship program and were better able to apply their learning to their work. They also felt that the TLP graduates with prior work experience were more well-grounded, understood the realities of the job market and set more realistic expectations in terms of work conditions and salary packages, health insurance, etc., for themselves. Table 8. Scholarship recipients’ use of skills developed during their Master’s program No response Strongly Disagree Disagree Neutral Agree Strongly Agree Total N Percent (row percent) Critical thinking 154 3 1 3 10 36 47 100 Organizing ideas 154 3 1 1 11 32 52 100 Finding information 154 3 1 1 7 36 53 100 Writing skills – writing productively, efficiently and clearly 154 3 1 0 6 31 59 100 Reading skills 154 3 1 1 12 31 53 100 Creativity 154 3 1 1 21 29 45 100 Ability to work with numbers and graphs 150 0 1 3 19 35 42 100 Ability to conduct research 150 0 1 1 10 33 55 100 Ability to ask questions 150 0 1 2 9 36 52 100 Methodological knowledge 150 0 1 2 11 39 47 100 Analytical skills 150 0 1 1 8 32 58 100 Problem solving skills 148 0 1 2 8 31 57 100 Time management 147 0 1 4 14 39 41 100 Table 9. Utilization of skills developed by Scholarship recipients during their Master’s program No response Strongly Disagree Disagree Neutral Agree Strongly Agree Total N Percent (row percent) Share and demonstrate in-depth understanding in my area of studies 116 5 1 2 4 33 55 100 Take initiative in any work environment 154 3 3 0 8 32 55 100 Transfer the knowledge in my field of study to others 154 3 2 1 12 31 53 100 Work with and under the leadership of others 154 3 1 1 14 36 45 100 USAID Impact Evaluation of the TLP Scholarships Program– Final Report 19 No response Strongly Disagree Disagree Neutral Agree Strongly Agree Total N Percent (row percent) Communicate effectively with others inside and outside my field 154 3 1 1 14 32 49 100 Work effectively in multidisciplinary teams 154 3 1 1 9 32 53 100 Lead teams that include international scientists/think tanks inside and outside my field 154 3 2 0 23 36 36 100 Build consensus around issues important to my organization 154 3 3 2 16 46 31 100 See the “big picture” 154 3 1 0 10 36 49 100 I have taken on a leadership role and am introducing new ideas to my organization and community 154 3 1 3 13 32 48 100 Table 10. Use of skills by scholarship recipients at the organizational level No response Strongly Disagree Disagree Neutral Agree Strongly Agree Total N Percent (row percent) I have introduced new ways of thinking and problem solving, by sharing skills with colleagues 149 0 1 4 12 45 38 100 I am using ethical ways to conduct research 154 3 2 2 7 44 42 100 I use data and analysis for policy making 154 4 4 4 12 35 41 100 I use discussion as a way to resolve problems at the organizational level 154 4 3 1 13 39 40 100 I use more participatory methods for consensus building, especially at the organizational level 154 3 2 1 19 39 35 100 I use research to support discussion and advocacy 154 3 2 1 14 36 44 100 I am leading interdisciplinary teams to work with national and international partners 154 3 4 7 32 27 27 100 I am using digital and multimedia tools to support advocacy and policy making 154 5 5 12 26 34 18 100 3.4 Constraints or barriers to utilizing knowledge/skills While most survey participants reported being able to use their skills, they also cited some limitations. They felt that despite being senior enough in their organization to make a change (at endline only 25 percent of the respondents said they were not senior enough), they were unable to do so as there was no organizational support for sharing new ideas (46 percent) or there was a lack of sufficient resources to introduce new ideas (59 percent). They went on to say that there was an unwillingness of the community USAID Impact Evaluation of the TLP Scholarships Program– Final Report 20 and or organizations to change (40 percent) as there were no incentives to change (41 percent). Many (55 percent) also said that changing deeply-entrenched ideas took time and that they were working on introducing change through demonstrating new ways of doing things and mentoring others. Table 11. Table 11. Barriers to utilizing skills at the organizational level N Not Applicable Strongly Disagree Disagree Neutral Agree Strongly Agree Total Percent (row percent) Unwillingness of the organization/community to change 149 2 23 15 19 26 14 100 Lack of sufficient resources to engage people and introduce new skills and concepts 154 3 5 12 19 44 15 100 There is no organizational support for sharing of new ideas 154 2 8 16 25 33 13 100 I am not senior enough within the organization to make a change 154 1 18 31 22 16 9 100 Deeply entrenched ideas take a while to change 154 2 4 15 20 36 19 100 There are no incentives for change 154 1 8 15 31 26 15 100 Focus group discussions with participants confirmed the findings from the survey data with regard to the use of skills in their organizations. Several of the participants said that now that they were in senior positions and management roles, they had started introducing new ideas and ways of working. However, it was difficult to bring about a change when people had very set habits. Despite change and modernization, the culture in Kosovo is still one of “the boss is always right,” and until the TLP graduates have proven themselves, there is a sense amongst colleagues that the TLP graduates are showing off just because they have returned from the US, making it difficult to share new skills. Given the tight job market, the employers in Kosovo are very protective of their own positions and some see the TLP graduates as a threat. This is especially true in the public institutions, which makes it difficult for TLPs to get hired. At the same time, in the words of a TLP graduate in a senior position with the government, “the change [in the government] has to come from within, there is no way someone can come in as a consultant, hold weekly meetings and hope to change the systems in place.” Interviews with employers revealed that a majority of the TLP graduates expect a better package in terms of working conditions, salary, health benefits, etc., more than UP graduates, often making it difficult for the private companies to hire them. Further, according to the employers, since the TLP scholars were highly marketable, it was difficult to keep them for long. While this was good for Kosovo in that more organizations got exposure to new ways of thinking and doing things, it also gave private organizations pause as they weighed the cost of hiring the TLP graduates and then having to replace them, with the benefits of new ways of doing things. Finally, there is a sense that the TLP graduates are a little more qualified than the Kosovar market can absorb at this point. The Scholarship recipients have completed their Master’s in new majors, which may or may not have application within Kosovo. For example, a TLP graduate in Artificial Intelligence is supporting a German company while living in Kosovo, as there are no opportunities for him at this point in Kosovo. USAID Impact Evaluation of the TLP Scholarships Program– Final Report 21 4 EVALUATION QUESTION 2: CONTRIBUTION TO KOSOVO This section responds to the second of the four evaluation questions addressed by this study: In what ways are U.S.-returned beneficiaries contributing to Kosovo’s development and how have the skills acquired in the U.S. added to that contribution, particularly at the organizational level? Most scholarship recipients have returned between the summer of 2016 to 2018 and by the endline survey had been in Kosovo for only 12- 24 months. While it is thus rather premature to assess their contribution to Kosovo’s development, there are early indications of positive contributions based on self-reports by scholarship recipients (focus groups and survey, see Table 12), corroborated by interviews with their employers as well as other stakeholders including the senior management at University of Prishtina.22 A brief discussion is below. New Subject Matter Expertise The TLP scholars have completed their Master’s degrees in nine broad areas (see Table 5). As stated earlier, several have completed Master’s programs in multi-disciplinary fields and with majors not available in Kosovo, thereby increasing the subject matter expertise available within the country. This is especially true of newer academic areas such as architecture, sustainable urban development, aviation management, gaming, cyber security, conflict resolution, interactive entertainment, cyber forensics and security, information technology and management, environmental policy, international security, and international peace and conflict resolution. Enhanced Capacity at the Organizational Level In addition to academic knowledge, scholarship recipients have also learned new ways of thinking, problem solving and work ethic, soft skills such as team work, participatory learning, and technical skills such as writing and communication, data analysis and management. All these skills are supporting change and development of Kosovo in many ways. Communication and writing There is enhanced capacity within the country for preparing proposals for grants and contracts as well as for advocacy and marketing pieces that meet international standards. Scholarship recipients report developing skills in communication; writing productively, efficiently and clearly; and conducting research using international standards. They are using these skills in their current jobs as well as in any volunteer work that they are/were engaged in to: • Prepare grant applications and write proposals for new business and community development projects, a skill much appreciated by the various NGOs and private sector organizations in Kosovo. • Support the development of white papers, background documents and in the conduct of research studies for strategy development and to support advocacy. Several of the scholarship recipients reported, and the management of University of Prishtina confirmed, that the support provided by the TLP scholars in developing the current (2016) UP strategy was invaluable and helped speed up 22 The scholarship recipients volunteered their time at the University of Prishtina to support the various centers at the University of Prishtina; they were also working as Teaching Assistants and Research Assistants at the University. Some of the recipients were also volunteering their time with the TLP-SP Advisory committee where they were supporting the ongoing research for developing white papers on current policy issues. Others have volunteered at various NGOs and multinational organizations. USAID Impact Evaluation of the TLP Scholarships Program– Final Report 22 the process considerably. Several focus group participants reported that they have helped clarify the mission statement and the strategy for the organization they are working for. • Support the development of brochures for marketing, tourism, etc. • Support new initiatives, start-ups, etc., and build capacity of the individuals, communities and organizations they have worked with, bringing about change and greater awareness of new ways of doing things. New ways of teaching and learning The TLP graduates have been exposed to active teaching and participatory learning methods in their Master’s programs, in an environment where students can ask questions without fear of repercussions. Now that many of them are working as teaching assistants at UP and as teaching staff in private colleges in Kosovo, this provides them with the right forum to bring their experience from the Master’s program to their current placements, introduce more interactive ways of teaching and contribute to changing the teaching and learning culture at these institutions of higher education. Data management and analysis The TLP graduates have also brought with them expertise and skills in data management and analysis which together with the time management skills are helping support better quality and timely reports. This is also helping them make better and more persuasive presentations of their work. As researchers at the university, at NGOs or in the government, the TLP scholars are demonstrating the use of ethical ways of conducting research and using research data to support discussions/advocacy. Work ethic and time management The TLP scholars have brought with them a new way of thinking and working. Most employers praise the work ethic of the TLP graduates and the energy that they bring to their work. They also praise their time management skills. Since their return two years ago, many TLP scholarship recipients have been promoted to senior management positions in their organizations. They are beginning to introduce new ways of thinking and doing things in their organizations. By working in teams they are being able to demonstrate the use of these ideas to their organizations and set an example for the rest of the employees. Economic Development TLP scholars are not only engaged in jobs that are directly related to their fields but are also using their expertise in related fields, introducing the concept that skills learned in one subject area can also be applied to another field thereby expanding the employment opportunities. Several TLP scholars have started their own businesses and are partnering with international partners, creating new job opportunities in Kosovo. Others are working on supporting companies in Europe while living in Kosovo, bringing in revenue for the country. They have been successful in creating linkages for companies within Kosovo with others in Europe and beyond for possible partnerships. (Table 12) TLP scholars are taking on new initiatives in their workplace and working in and leading multi-disciplinary teams, including teams with international experts, helping bring and utilize new knowledge, funding and expertise. They TLP scholars are also building the capacity of other start-ups in Kosovo by helping them develop their business plans, accounting spreadsheets, acting as legal advisors and even providing mentoring to the owners and employees to increase productivity and professionalism in their way of working. USAID Impact Evaluation of the TLP Scholarships Program– Final Report 23 Volunteerism Scholarship recipients are introducing a culture of volunteerism in Kosovo. Over half of the survey respondents said that they were continuing to volunteer time, beyond the mandatory required hours, to help support their community and to share skills with new businesses. Focus group participants reported working with various NGOs to help them bring the message to far-flung communities and supporting the training offered. Others reported that they were supporting efforts to bring about a change in their community by conducting research to support advocacy for policy change and by helping write research grants. Table 12. Ways in which TLP scholarship recipients are supporting the development of Kosovo N Not Applicable Strongly Disagree Disagree Neutral Agree Strongly Agree Total Percent (row percent) I am building capacity of my colleagues at work 154 3 3 4 25 39 27 100 I have developed/clarified the strategy for my organization to attain its vision 154 3 3 8 29 33 24 100 I am contributing to the economic development of Kosovo through my start-up and entrepreneurial efforts 154 4 12 18 27 21 18 100 I have brought new business to Kosovo through international partnerships 154 4 19 22 30 8 16 100 I support organizations within Kosovo through knowledge sharing 150 0 11 19 18 30 23 100 I support organizations by helping them establish linkages/network with other organizations inside and outside Kosovo 149 0 8 12 21 33 26 100 I volunteer my time (beyond the mandatory hours) to support and build capacity of organizations in my community 154 3 5 11 24 31 27 100 I provide support with research and writing grants that bring about change in my community 148 0 7 12 24 31 26 100 USAID Impact Evaluation of the TLP Scholarships Program– Final Report 24 5 EVALUATION QUESTION 3: PERCEPTION OF TLP SCHOLARSHIP RECIPIENTS AS LEADERS This section responds to the third of the four evaluation questions addressed by this study: In what ways does the perception of TLP scholarship recipients as leaders change over the course of the program? All scholarship recipients attended a Leadership Seminar organized by the TLP-SP project in Washington D.C., which imparted practical guidance on the concept of leadership and the skills that described a leader in addition to observing the skills demonstrated by those leading the Master’s program.. This section discusses the changes in the self-perception of the TLP scholars of themselves as leaders as well as perceptions of employers and other stakeholders. 5.1 Self-perception of scholarship recipients At endline, 81 percent of the scholarship recipients reported that they were working in a leadership role and introducing new ideas to the community. Over two-thirds of the survey respondents believe that their colleagues, employees and community members look up to them as a leader. (Table 13). Table 13. Leadership – Self-Perception of TLP scholarship recipients N Not Applicable Not at all Once in a while Some times Fairly often Frequently, if not always Total Percent (row percent) I am a leader in my organization and community 154 7 1 8 18 32 34 100 My colleagues think of me as a leader 154 3 0 5 21 36 36 100 My employers look upon me as a leader 154 8 0 7 12 37 36 100 My community thinks of me a leader 154 9 0 10 18 32 32 100 IBI administered the Multifactor Leadership Questionnaire (MLQ), a self-assessment tool to measure leadership style and to designate behavior, to scholarship recipients at baseline, midline and again at endline to understand how their self-perceptions of their own leadership style changed over time.23,24,25 The questionnaire includes 45 items meant to measure the whole spectrum of leadership style (Transformational Leadership, Transactional Leadership, Passive Avoidant/Laissez Faire leadership) as well as perception of the desirable Outcomes of Leadership (motivates others to extra effort; productive or effective in their interactions at different levels of the organization; and feeling of satisfaction in working with others). A brief description of the underlying constructs and a description of the various dimensions of leadership as defined by MLQ is included in Box 1and also in Annex 6.26 23 The same tool was used for collection of baseline data and was also administered to a carefully selected comparisons group of respondents who qualified for but did not receive the scholarship. 24 See Bass, B. M., & Avolio, B. J. (2000). MLQ: Multifactor questionnaire: Third edition manual and sampler set. Redwood City, CA: Mind Garden, 25 IBI holds a remote license to reproduce and administer the survey for the purpose of this evaluation. 26 The questionnaire employed in this survey is included as a part of the survey instrument placed in Annex 4. Details of various dimensions of multi-factor leadership styles and associated behaviors are placed in Annex 6. USAID Impact Evaluation of the TLP Scholarships Program– Final Report 25 Box 1: Leadership Styles and Outcomes of Leadership Leadership Styles Transformational leadership is a process of influencing by which leaders change their associates’ awareness of what is important, and move them to see themselves and the opportunities and challenges of their environment in a new way. Transformational leaders are proactive: they seek to optimize individual, group and organizational development and innovation, not just achieve performance “at expectations.” On the other hand, transactional leaders display behaviors associated with constructive and corrective transactions. The constructive style is labeled contingent reward and the corrective style is labeled management-by-exception. Transactional leadership defines expectations and promotes performance to achieve these levels. Contingent reward and management-by-exception are two core behaviors associated with ‘management’ functions in organizations. The passive avoidant leadership style is passive and ‘reactive’: it does not respond to situations and problems systematically. Passive leaders avoid specifying agreements, clarifying expectations, and providing goals and standards to be achieved by followers. This style has a negative effect on desired outcomes – opposite to what is intended by the leader-manager. In this regard it is similar to laissez-faire styles – or ‘no leadership’. (Bass et al., 1990) Outcomes of Leadership Extra Effort: This 3-item scale measures the frequency in which leaders are perceived to be able to influence their followers to do more than they are expected to do or motivate them to extra effort. Effectiveness: This 4-item scale measures the frequency in which leaders are perceived as being effective when interacting at different levels of the organization. Satisfaction with the Leadership: This 2-item scale measures the frequency in which raters are satisfied with their leader's methods of working with others. MLQ questionnaire was structured on a five-point Likert scale while response choices ranged from “Not at all” to “Frequently, if not always”. Transformational leaders are expected to exhibit high scores in the scales of Idealized Influence (attributed), Idealized Influence (behavior), Inspirational Motivation, Intellectual Stimulation, and Individualized Consideration. (Jones et al., 2007) Source (Bass et al 1990, Jones et al 2007) . Table 14 presents a summary of the responses to the MLQ questionnaire. Responses are presented in the form of mean values for scholarship recipients and compared with similar data from the baseline study. In both cases, respondents were asked to respond to specific dimensions of leadership characteristics on a five-point scale ranging from “not at all” through to “frequently, if not always”, with the options “once in a while”, “sometimes” and “fairly often” falling in between. Means were obtained by assigning values 0 (not at all) through 4 (frequently if not always) to each of these options and aggregating the values according to the relevant grouping of responses. Annex 5 has the detailed frequency tables. Table 14. Scholarship recipients change in MLQ scores from Baseline to Endline Scholarship Recipients Leadership Style and Associated Dimensions (summary scores) Baseline Midline Study Endline BL-EL N Mean Standard Deviation N Mean Standard Deviation N Mean Standard Deviation t values Transformational Idealized Influence – Attributes (builds trust) 116 2.9605 .0583 126 2.8214 .7415 140 2.9304 .5976 2.014† Idealized Influence – Behaviors (acts with integrity) 116 3.0955 .0534 127 3.0276 .6372 138 3.0996 .5149 -0.085 Inspirational Motivation (encourages others) 116 3.4001 .0481 124 3.3044 .6412 138 3.2047 .5269 3.9796† Intellectual Stimulation (encourages innovative thinking) 116 3.3477 .0469 130 3.2346 .6378 138 3.2192 .5103 2.7016† Individual Consideration (Coaches & Develops People) 116 3.1918 .0511 126 3.0079 .6826 139 3.1942 .4740 -0.054 Transactional Contingent Reward (rewards achievement) 116 3.2091 .0511 125 3.0040 .7128 137 3.0748 .5228 2.755† USAID Impact Evaluation of the TLP Scholarships Program– Final Report 26 Scholarship Recipients Leadership Style and Associated Dimensions (summary scores) Baseline Midline Study Endline BL-EL N Mean Standard Deviation N Mean Standard Deviation N Mean Standard Deviation t values Management-by-Exception: Active (monitors deviations & mistakes) 116 2.1250 .0734 126 1.9187 .9401 139 2.5036 .6876 -5.901† Passive Avoidant/Laissez faire Management-by-Exception: Passive (fights fires) 116 .6624 .0523 125 .6840 .7406 138 2.6504 .6504 -26.501† Laissez Faire (avoids involvement) 116 .3994 .0492 129 .5310 .6732 141 2.6117 .5782 -41.073† Outcomes of Leadership Generates Extra Effort 116 3.4024 .0494 147 2.7982 1.1824 153 2.9739 .9960 4.6272† Is Productive and Effectiveness 116 3.3844 .0401 130 3.2212 .6415 145 3.1397 .6528 4.0295† Generates Satisfaction 116 3.4397 .0446 147 2.9354 1.2040 153 3.0359 .9819 4.4240† †difference is statistically significant; ‡ difference is statistically weakly significant The data suggest that the scholarship recipients, at endline relative to baseline, are significantly more focused on the transformational and less on the transactional end of the leadership continuum, are less laissez faire about their work and are more likely to carry these traits with them as they grow in their organizations. The scholarship recipients continue to view themselves as proactive transformational leaders who take pride in their work and encourage others to do the same. They act with integrity and consider the moral and ethical consequences of decisions. They build trust among their teams and encourage innovative thinking. In addition, as at baseline, they display a sense of confidence, and believe in coaching and mentoring others. While continuing to emphasize the importance of rewarding achievement, at endline they are also beginning to see themselves as more proactive leaders rather than passive transactional leaders who focus their attention on irregularities, mistakes, exceptions, and deviations from standards or believe in not getting involved. As leaders they continue to be focused on pushing themselves and their teams to try to work harder and beyond their current scope of work, effectively meeting the organizational requirements and using a leadership style that is effective and works with the team. (See Table 14). Focus group discussions with scholarship recipients supported these findings. Several focus group participants reported that they felt more comfortable working in and leading teams, recognized the value of team accountability and felt confident enough of their own abilities to share knowledge and provide mentoring and guidance to others. As they get placed in senior and management roles in their organizations these qualities will help them become stronger leaders ready to bring in new and innovative thinking to their organizations, necessary for Kosovo to grow as a country and compete on the international stage. Some focus group participants also voiced concern that since they worked hard themselves and encouraged others to do the same, they may begin micro-managing the teams and making them less creative. This self-reflection is a positive quality in a leader who is constantly evolving and reflecting on her own abilities to meet new challenges in the workplace. 5.2 Perception of employers and stakeholders There is a significant positive change in the opinion of employers over the two years between midline and endline data collection. At endline, the employers overall are much more appreciative of the TLP graduates and their skills than they were at baseline. The employers said that even though the TLP scholars needed USAID Impact Evaluation of the TLP Scholarships Program– Final Report 27 training to learn the ways of the organization, they stood out due to their ability to take on leadership roles and learn what they were required to do very quickly. One employer reported that he had just doubled the salary of a TLP scholarship recipient, just to be able to keep him with the organization. This is a very different sentiment from that expressed at midline by some of the employers, who had felt that the TLP graduates came with an attitude and were not willing to learn, and that they were acting like senior managers rather than as new hires. Several TLP scholarship recipients, after completing their Master’s programs, have gone back to work for their previous organizations either in a senior role or in the same position with additional responsibility and been promoted since. Employers who have had the experience of working with the TLP scholars before and after they completed the Master’s degree said that there was a very noticeable difference in the self￾confidence levels of the scholarship recipients after they returned from the U.S. The scholars were able to communicate better, their English fluency had improved, and they had significantly better writing skills. Most TLP scholarship recipients demonstrated a strong work ethic and were prepared to do “whatever it took to get the job done.” These skills are well-appreciated by the organizations that they are working for and have paved the way for promotions to senior positions for the scholarship recipients. The employers feel that the TLP scholars are able to learn the basic skills and understand what was expected of them more quickly than others. They are also more likely to take initiative and volunteer for tasks. The employers went on to say that from their perspective, TLP scholarship recipients who had work experience prior to receiving the scholarship had benefitted more from the program and due to their previous work experience, are better-grounded and able to re-adjust to working and living in Kosovo. The employers also voiced a note of caution, worrying that since the TLP scholars were well-trained and highly marketable, they were unlikely to stay in one position within the same organization, or even within Kosovo, for any length of time. The employers were concerned that this may make it difficult for companies to significantly invest in recruiting and training new employees. USAID Impact Evaluation of the TLP Scholarships Program– Final Report 28 6 EVALUATION QUESTION 4: COMPARISON WITH NON-RECIPIENT PEERS This section responds to the last of the four evaluation questions addressed by this study: Given a-c above, how and to what extent have the experiences of the returned scholarship recipients differed from their non-recipient peers? In order to study the impact of the TLP Scholarships program on its recipients, the beneficiaries of the scholarship program were compared with a comparison group of peers that applied for but did not get into the Scholarships program. As at baseline and midline, endline data were collected using a very similar survey instrument for both scholarship recipients and the comparison group. For the comparison group, 106 respondents (29 percent male and 71 percent female) completed the survey, and like the scholarship recipients, the vast majority were Albanian (97 percent). This section compares scholarship recipients with their peers on various dimensions. 6.1 Current status Of the comparison group, 75 percent of those that completed the survey are currently based in Kosovo, and the rest are in Europe (23 percent) and U.S. (2 percent). Over four-fifth of this group (82 percent) are employed, including those self–employed (8 percent), and about 7 percent are enrolled in academic programs. About 15 percent of the comparison group reported being unemployed and looking for a new or better employment, a number significantly higher than for the scholarship group (p<.05). (Table 15). Table 15. Employment status of scholarship recipients and comparison at endline Scholarship recipients Comparison t-value Employment status (N-154) (N-106) Percent* Employed (includes self-employed) 88 82 3.3927† Self-employed** 15 9 0.3483 Unemployed and looking for a job/taking a break and not looking for a job 5 14 -3.0658† Volunteering 5 3 0.1902 Enrolled in an academic program*** 8 13 -2.0705† Note: *Respondents could select more than one category; **included as a separate category to show extent of self-employment; ***respondents currently in Masters programs are studying along with work; †difference is statistically significant (p<.01) 6.2 Training and skills development A bachelor’s degree was a minimum requirement to apply for the TLP scholarships program. Of the comparison group, at endline most (68 percent) had a Master’s degree, a handful (5 percent) reported a doctoral degree, while twenty-one percent reported a bachelor’s degree as their highest degree. Three￾quarters of those with a Master’s degree had enrolled in and completed their Master’s program after unsuccessfully competing for the TLP scholarship program. While most of the comparison group had completed their Master’s and PhD programs from University of Prishtina, about a third had done so from a university in U.S. or Europe. Like the scholarship recipients, members of the comparison group also reported that they had developed multiple skills. Almost all respondents from the comparison group agreed (between 81 and 94 percent, USAID Impact Evaluation of the TLP Scholarships Program– Final Report 29 agreed or strongly agreed) that they had learned to see the “big picture” and developed an in-depth knowledge of their area of studies and felt able to transfer knowledge to others. They reported having developed critical thinking and analytical skills as well as skills in time management, organizing ideas, problem solving and conducting research. Tables 15 and 16 present the mean scores for the two groups, calculated by assigning a value of one to strongly disagree and 5 to strongly agree on a five-point Likert scale. Table 16. Skills developed by scholarship recipients and comparison group Scholarship Recipients Comparison Skills N Mean Standard Deviation N Mean Standard Deviation t-values Critical thinking 154 4.19 1.071 98 4.53 .645 -2.8502† Organizing ideas 154 4.24 1.079 97 4.55 .646 -2.5231† Finding information 154 4.29 1.060 97 4.28 .813 0.1100 Writing skills- writing productively, efficiently and clearly 154 4.40 .987 96 4.44 .765 -0.2955 Reading skills 154 4.27 1.043 97 4.29 .889 -0.1246 Creativity 154 4.10 1.089 97 4.45 .778 -2.8015† Ability to work with numbers and graphs 150 4.13 .922 95 4.41 .676 -2.5914† Ability to conduct research 150 4.41 .778 97 4.45 .722 -0.4762 Ability to ask questions 150 4.37 .789 97 4.43 .776 -0.6492 Methodological knowledge 150 4.30 .801 95 3.93 .959 3.2932† Analytical skills 150 4.45 .756 95 4.11 .805 3.4232† Problem solving skills 148 4.41 .832 95 4.55 .665 -1.3333 Time management 147 4.16 .904 94 4.09 1.012 0.5700 †difference is statistically significant (p<.01); ‡ difference is statistically weakly significant (p<.05) Compared to the reference group, TLP scholars assessed themselves lower than the comparison group on critical thinking, organizing ideas, creativity, and ability to work with numbers (difference statistically significant in all cases). The comparison group assessed themselves lower than TLP scholars in methodological knowledge and analytical skills (both differences statistically significant). 6.3 Employment Two-thirds of those in the comparison group reported themselves to be employed, 78 percent reported working in different positions than when they had applied for the Scholarships program, while 19 percent reported they were in the same position at endline as at the time of applying for the TLP scholarships program. Among the employed, 43 percent are working with private organizations, less than the scholarship recipient group (difference statistically significant; p=.0394), 18 percent are working in government jobs compared to scholarship recipients (11 percent, difference statistically significant, p=.0056), 23 percent are working with NGOs and 11 percent with multinationals, not much different from the scholarship recipient group (22 and 12 percent respectively; difference not statistically significant). Table 17 has the details. Twenty-nine percent reported they were in a senior position than at the time of applying for the scholarship, with greater responsibility (49 percent) and better pay (43 percent). Thinking back to when they had applied for a job, the comparison group reported that it had taken them between one to six months to find a job. USAID Impact Evaluation of the TLP Scholarships Program– Final Report 30 Table 17. Primary place of employment – recipients and comparison group Primary place of employment Recipients (N=136) Comparison (N=86) p value (*=p<.05) Percent Government 11 18 -2.7079† Public organization – University, institution 4 1 0.8717 Private organization – University, institution 49 43 2.0723† NGO 23 23 0.0784 Multinational/donor organization 12 11 0.9421 Other 1 1 0.3304 *respondents allowed to select multiple responses; ** respondents currently in Masters programs are studying along with work; †difference is statistically significant (p<.01) TLP scholars employed in the private sector are engaged in software development, research, policy development and advocacy, providing consulting services and supporting other groups with international contracts, etc. and work in organizations that have more international clients, more use of English language skills, and a fast-paced work environment. In addition, the managers in these organizations are themselves graduates of other American scholarship programs, such as Fulbright, Kosovo American Education Fund), etc. TLP scholars, with the training and exposure that they have received in their studies in U.S. universities, seem to be preferring the distinctive characteristics of such environments. 6.4 Volunteer activities Almost all TLP scholars had either completed 400 hours of volunteer work since returning from the U.S. or were working on it, but only about 16 percent of the survey respondents from the comparison group reported volunteering as one of their activities and 36 percent reported having completed some volunteer activity since they had applied to the scholarship program. Forty-three percent reported that they had neither engaged in any kind of volunteer work nor were they planning to. Those who were volunteering were doing so through NGOs doing community level work or advocacy in areas of education, gender and human rights. However, since there is no verifiable record of volunteer work, unlike for the scholarship recipients,27 it is difficult to draw meaningful comparisons. 6.5 Utilization of training and skills Over 80 percent of scholarship recipients had taken on a leadership role and were introducing new ways of thinking and problem-solving to others in their place of work. Almost all (86 percent) reported using ethical ways to conduct research, using research to support advocacy and using discussion as a way of problem￾solving at the organizational level. Self-reports by the comparison group shows very similar patterns, with over 85 percent reporting taking on a leadership role, introducing new ways of thinking and problem-solving with colleagues as well as using data for decision making. It is interesting to note that while the patterns are similar, more of the comparison group report using these skills than the scholarship recipients. Table 18 has the details. 27 Citizen Corp, a USAID funded sister project to the TLP-SP project, helped organize volunteer placements for the TLP scholarship recipients and kept track of the hours worked. USAID Impact Evaluation of the TLP Scholarships Program– Final Report 31 Table 18. Use of skills within organizations – recipients and comparison group Scholarship Recipients Comparison Use of Skills by those employed (based on a 5 point scale) N Mean Standard Deviation N Mean Standard Deviation t-values (*p<.05) I have taken on a leadership role and am introducing new ideas to my organization and community 154 4.16 1.109 97 4.41 .718 -1.9741‡ I have introduced new ways of thinking and problem solving, by sharing skills with colleagues 149 4.13 .875 94 4.27 .792 -1.2594 I am using ethical ways to conduct research 154 4.12 1.120 94 4.45 .682 -2.5786† I use data and analysis for policy making 154 3.94 1.292 95 4.24 .942 -1.9634‡ I use discussion to resolve problems at the organizational level 154 4.01 1.204 94 4.50 .684 -3.6049† I use more participatory methods for consensus building, especially at the organizational level 154 3.94 1.139 94 4.22 .735 -2.1274† I use research to support discussion and advocacy 154 4.10 1.142 94 4.35 .786 -1.8687‡ I am leading interdisciplinary teams to work with national and international partners 154 3.55 1.247 94 3.72 1.062 -1.1002 I am using digital and multimedia tools to support advocacy and policy making 154 3.34 1.305 93 3.81 1.066 -2.9317† †difference is statistically significant (p<.01); ‡ difference is statistically weakly significant (p<.05) However, when it comes to the use of skills developed for capacity-building and economic development of Kosovo, the scholarship recipients are significantly more engaged. Even though members of the comparison group see themselves as being ready to take on leadership positions, lead multidisciplinary teams and communicate effectively, more scholarship recipients report using their skills to contribute to the economic development of Kosovo through their start-ups and entrepreneurial efforts, supporting their organizations by helping them establish linkages/network with other organizations inside and outside Kosovo and conduct research to bring about change in the community (all differences weakly or strongly significant). See Tables 18 and 19. Table 19. Use of skills to support Kosovo – recipients and comparison group Scholarship Recipients Comparison Use of Skills by those employed (based on a 5 point scale) N Mean Standard Deviation N Mean Standard Deviation t-values I am building capacity of my colleagues at work 154 3.75 1.12785 95 3.72 1.117 0.2555 I have developed/clarified the strategy for my organization to attain its vision 154 3.58 1.19749 94 3.44 1.113 0.9713 I am contributing to the economic development of Kosovo through my start-up and entrepreneurial efforts 154 3.04 1.40439 93 2.70 1.232 1.9292‡ I have brought new business to Kosovo through international partnerships 154 2.68 1.41284 92 2.55 1.171 0.7286 I support organizations within Kosovo through knowledge sharing 150 3.35 1.30634 96 3.58 1.167 -1.4034 I support organizations by helping them establish linkages/network with other organizations inside and outside Kosovo 149 3.57 1.22615 95 3.23 1.115 2.1795† I volunteer my time (beyond the mandatory hours) to support and build capacity of organizations in my community 154 3.55 1.2787 96 3.36 1.224 1.1057 I provide support with research and writing grants that bring about change in community 148 3.57 1.1904 94 3.21 1.235 2.2274† †difference is statistically significant; ‡ difference is statistically weakly significant USAID Impact Evaluation of the TLP Scholarships Program– Final Report 32 6.6 Change in self-perceptions of leadership potential The MLQ questionnaire was also administered to the comparison group as a part of their survey. Table 19 presents a summary of the MLQ section of the survey for the comparison group and compares data for the endline study with the baseline study (for the comparison group) to assess changes in this group over the years. (Annex 5 has the detailed frequency tables). And similarly, endline scores for the scholarship recipients were compared with those for the comparison group (Table 20). Table 20. Change in Leadership Style – Comparing Comparison Group to the Scholarship Recipients Scholarship Recipients (SR) Comparison (C) Comparison (C) C to C SR to C Endline Baseline Endline BL-EL EL-EL Leadership Style and Associated Dimensions N Mean Standard Deviation N Mean Standard Deviation N Mean Standard Deviation t values t values Transformational Idealized Influence￾Attributes (builds trust) 140 2.930 0.598 60 2.845 0.096 76 2.740 0.691 1.1604 2.1130† Idealized Influence – Behaviors (acts with integrity) 138 3.100 0.515 64 2.766 0.118 76 2.964 0.600 -2.601† 1.7404 Inspirational Motivation (encourages others) 138 3.205 0.527 64 3.138 0.123 75 3.167 0.636 -0.353 0.4673 Intellectual Stimulation (encourages innovative thinking) 138 3.219 0.510 64 2.918 0.118 76 3.112 0.557 -2.7303† 1.4249 Individual Consideration (Coaches, Develops People) 139 3.194 0.474 64 2.727 0.123 76 2.980 0.653 -3.0614† 2.7585† Transactional Contingent Reward (rewards achievement) 137 3.075 0.523 64 2.938 0.123 74 3.047 0.576 -1.4952 0.3520 Management-by-Exception: Active (monitors deviations and mistakes) 139 2.504 0.688 64 2.277 0.132 76 2.010 0.860 2.4633† 4.597† Passive Avoidant/Laissez faire Management-by-Exception: Passive (Fights fires) 138 2.650 0.650 64 0.813 0.080 106 0.538 0.761 2.8747† 23.3472† Laissez Faire (avoids involvement) 141 2.612 0.578 64 0.480 0.074 76 0.559 0.756 -0.8295 22.3348 Outcomes of Leadership Generates Extra Effort 153 2.974 0.996 64 3.000 0.117 106 2.267 1.511 3.8701† 4.5362† Is Productive and Effectiveness 145 3.140 0.653 64 3.138 0.118 106 2.325 1.530 4.236† 5.7367† Generates Satisfaction 153 3.036 0.982 60 3.450 0.068 106 2.354 1.561 5.428† 4.3133† †difference is statistically significant An examination of the mean scores on various MLQ traits for the comparison group shows that between baseline and endline, the comparison group, like the scholarship recipients’ group, has changed in leadership style. At endline, more respondents from the comparison group reported acting with integrity, encouraging innovative thinking, and coaching and mentoring others. At the same time, more respondents saw extra effort to achieve goals, being effective and productive in communication and creating an overall feeling of satisfaction among the team as the outcomes of their leadership. (See Table 20). USAID Impact Evaluation of the TLP Scholarships Program– Final Report 33 A comparison of the MLQ scores at endline for scholarship recipients and members of the comparison group show that significantly more scholarship recipients view the outcomes of leadership as helping others develop a heightened desire to succeed and to try harder (extra effort), working in a way that is productive as well as working with others in a way that is satisfying for all (satisfaction). Relative to their non-recipient peers, more scholarship recipients are likely to be transformational leaders willing to make the effort to build trust and act as a mentor or coach to their colleagues, while helping their teams meet expectations by defining upfront what is expected of them. In addition, the scholarship recipients are less likely (highly significant) to have a laissez faire attitude towards work or a passive attitude to work where they wait for things to go wrong before they “put out the fires.” 6.7 Perception of employers and stakeholders Data from interviews with employers corroborated the above findings. Comparing scholarship recipients with their non-scholarship recipient employees (as a proxy for the comparison group), most employers felt that the scholarship recipients were stronger and more effective leaders. They were better able to manage their work and were receiving quicker promotions. The employers reported encouraging the TLP scholars to apply for more senior roles within the organization as a way of retaining them in the organization. However, they reported being cautious about the investment in the scholarship recipients as they felt they were very mobile and were likely to leave not just their organization, but also the country. Some employers noted that while the scholarship recipients were good at their work and at managing teams, they were a little out of touch with the local networks. Two years is a long time in a country like Kosovo and the TLP scholarship recipients, by being away for the Master’s program, could likely have lost some of their social and professional networks and are thus seeing the need to start afresh. 6.8 Plans for the future All survey respondents, scholarship recipients and their peers were asked about their satisfaction with their current job and plans for the future. Almost two-thirds of the scholarship recipients (62 percent) and the comparison group (66 percent) reported being very satisfied with their current jobs, while about a quarter (21 percent comparison and 24 percent scholarship recipients) reported being somewhat satisfied and looking for a change. A small proportion (4 percent) from both groups reported not being satisfied with their jobs and actively seeking change. While a small proportion of the two groups was planning to stay in their current jobs and grow there (19 percent of the comparison and 25 percent of the scholarship group), about a quarter from each group reported planning to change jobs for a better option (28 percent of the comparison and 20 percent of the scholarship group, difference significant p<.05). A small proportion of both groups was also planning to join a PhD program in the US or elsewhere (28 percent of the comparison group and 30 percent scholarship recipients). The data for the scholarship recipients were also supported by the data from focus groups, which showed that the TLP scholars were looking to get a higher degree in their field at some point in the future, preferably from somewhere in Europe as “the US was too far away and it made it difficult to visit home frequently.” USAID Impact Evaluation of the TLP Scholarships Program– Final Report 34 7 CONCLUSIONS AND RECOMMENDATIONS Guided by a set of questions posed by USAID, this evaluation has been conducted to assess the overall impact of the TLP Scholarship Program on its recipients, provide stakeholders with an understanding of how various investments in scholarships (for Master’s level studies in US Universities and colleges) under TLP￾SP increased the knowledge and skills of the individual participants, and determine the effectiveness of the program in training participants to act as change agents who can affect social and economic development upon their return to Kosovo and transition to work. The evaluation has assessed the progress of the TLP Scholarships program since its inception and evaluated the impact on the recipients. Qualitative and quantitative data for the evaluation were collected from the recipients of the TLP scholarships for a Master’s program at US universities and colleges, and a carefully selected comparison sample of their peers who had not received the award. Data were collected from each cohort of scholarship recipients and their peers at three points – baseline conducted at the time of award, midline upon return from the US after completing the Master’s program, and endline conducted two years after the completion of the program and return to Kosovo – to understand the final impact of the TLP scholarships program. The first round data of data provided a baseline of attitudes and skills, against which the midline and endline studies data were compared. Specifically, the midline study focused on experience with the academic program as well as interim changes if any in attitudes and thinking, and the endline data provided the final comparator to evaluate the changes since the inception of the program. Overall, the TLP scholarships program has been well received and has had a positive impact in the short￾run. Against 185 scholarships that were awarded over three cohorts (2014, 2015 and 2016), 181 candidates completed the Master’s programs from US Universities and colleges. Participants in the program from all three cohorts have by and large been satisfied with the study experience in the U.S., and speak highly of the quality of program and the skills developed because of that experience. A large majority of the participants report being satisfied with the support, including the stipend, provided by the program to help them get settled as well as in ensuring their academic and physical well-being. The program, which has afforded opportunities for participants to study in the U.S. and take courses in subjects not offered in Kosovo, has greatly expanded the horizons for participants and increased their exposure to new ways of studying and new skills. Many participants studied in inter-disciplinary programs not available in Kosovo. The participants report having developed skills in communication, writing and conducting research – skills that they have already begun to use in their jobs. While the participants were not all that satisfied with the support from the TLP program in helping them identify volunteer/community service opportunities and internship opportunities while in the U.S., they are very satisfied with the support provided upon return to Kosovo. Many TLP scholars have continued to participate in volunteer opportunities, preparing grant applications, writing proposals, and supporting development of background documents and research studies for strategy development – contributing in areas that they probably would not have had it not been for the exposure and opportunities they received as part of the program. All in all, the Scholarships program has contributed extensively to increasing the level of confidence and bringing about a positive change in the way that the participants articulate their ideas and expound upon them. The program was not without its challenges, however. Many participants, upon completion of their course work in the U.S., reported experiencing a reverse culture shock and a problem with settling back. Many also USAID Impact Evaluation of the TLP Scholarships Program– Final Report 35 reported issues transitioning from volunteering to motivating themselves into applying and interviewing for employment opportunities. Participants reported being disappointed in the limited employment opportunities available to them upon their return to Kosovo, realizing quickly that they needed work experience in addition to their degrees to find a suitable job. In addition to the general paucity of employment opportunities in Kosovo, competition from graduates from similar programs from Kosovar and European universities further constrained the market and associated opportunities. A comparison of the scholarship recipients with the peer group of those who were unable to secure participation in the Scholarships program suggests that the TLP scholars are doing as well, if not better, than their peers. Out of 154 scholarship recipients that completed the endline survey, only eight reported being unemployed. Most others reported being employed in mid-to-senior level positions within the private sector, NGOs, government and public universities in Kosovo. Most of these positions are better paid and come with greater responsibility within the organization than the participants’ positions prior to their joining the scholarships program. In most cases, the positions are related to the field of study that the participants selected for the Master’s program in the U.S. Conversations with employers suggests that the scholarship recipients have indeed developed the skills and leadership potential that was expected from their participation in the program, and have already begun to contribute substantially to the achievement of the overall purpose of TLP objectives. Use of Skills and Contribution to Kosovo’s Development Since most scholarship recipients have been back in Kosovo for only two years on average. it is rather premature to assess their contribution to Kosovo’s development. However, there are early indications of positive contributions and transformational leadership based on self-reports by scholarship recipients and their employers and supervisors in their work and volunteer placements. First, many scholarship recipients have elected to specialize in majors not available in Kosovo (such as a Master’s in Aviation Management, Sustainable Urban Planning, Cyber Security, developing games, Environmental Policy, and Social Enterprise) during their study in the U.S, and at least two recipients from this cohort have continued to doctoral programs. By itself, this is a contribution to Kosovo’s development, in that it increases the level of specialized skills in the country. Second, scholarship recipients report developing skills in communication; writing productively, efficiently and clearly; and conducting research. In addition, they report gaining fluency in the English language. The TLP scholars are using these skills in their current jobs as well as in volunteer opportunities to prepare grant applications, write proposals, and support development of white papers, background documents and in the conduct of research studies for strategy development. They have helped their organizations to develop and clarify strategy, claim that is also supported by the organizations where they work or volunteer. The support provided by the TLP scholars in developing the current (2016) UP strategy helped speed up the process considerably and was well appreciated. Third, scholarship recipients report being prepared to work in and lead multi-disciplinary teams, including teams with international experts. They also report being ready to take on new initiatives in their workplace, and transfer knowledge-building capacity within the country. Fourth, several scholarship recipients are working as teaching staff at various private universities/colleges in Kosovo and have volunteered their time as Teaching Assistants in the University of Prishtina, which provides them with the right forum to bring their USAID Impact Evaluation of the TLP Scholarships Program– Final Report 36 experience from the Master’s program to their current placements and contribute to changing the teaching and learning culture at the university. Fifth, scholarship recipients, as researchers at the university, at NGOs or in the government, are demonstrating the use of ethical ways of conducting research and using research data to support discussions/advocacy. Sixth, TLP scholars are not only engaged in jobs that are directly related to their fields; they are also using their expertise in related fields, introducing the concept that skills learned in one subject area can also be applied to another field thereby expanding the employment opportunities. Seventh, TLP scholars are creating new job opportunities in Kosovo. Several scholarship recipients have started their own businesses and are working with international partners, bringing new opportunities to Kosovo. Others are working on supporting companies in Europe while living in Kosovo, bringing in revenue for the country. TLP scholars are building capacity in other start-ups in Kosovo by helping them develop their business plans, accounting spreadsheets, acting as legal advisors and even providing mentoring to the owners and employees to increase productivity and professionalism in their way of working. TLP recipients have generally been successful in creating linkages for companies within Kosovo with others in Europe and beyond for possible partnerships. And finally, scholarship recipients are introducing a culture of volunteerism in Kosovo. They are helping support their communities and grass roots NGOs by helping write research grants and by conducting research to support advocacy for policy change. Some are also leading and helping with providing training to disadvantaged populations within their communities. All these initiatives and changes in approach have a good potential to sustain themselves and positively contribute to culture change within Kosovo. Indeed, the participants from Cohorts 1 and 2, especially those who have had time to settle down, are rapidly climbing to senior and responsible levels within their organizations and are increasingly getting into positions from where they can model and push for more ethical ways of doing things. Comparison with their peers shows that they are doing things better than most of their peers and are more likely to be leaders within their organizations, willing to take the time to build trust, model ethical behaviors and coach and mentor their teams. However, there are some challenges here as well, as several scholarship recipients are showing signs of frustration as they come up to deeply entrenched ideas in the Kosovar fabric of life that resist new ways of thinking and doing things and limiting the impact that returning participants thought they could have. The paucity of resources and organizational support for sharing new ideas also poses barriers to the use of newly acquired skills. The employers also voiced a note of caution, saying that since the TLP scholars were well trained and highly marketable, they were unlikely to stay in one position within the same organization for any length of time – or for that matter, within Kosovo, as there were so many opportunities for highly trained and skilled personnel in other countries in Europe, especially Germany. The employers noted that while the mobility of TLP scholars across organizations was generally good for commerce and industry, it also made it risky for employers to invest significantly in recruiting and training new employees. USAID Impact Evaluation of the TLP Scholarships Program– Final Report 37 A final point to be noted is that several TLP graduates have had a tough time with re-entry to Kosovo and have suffered from severe reverse-culture shock, especially those that had limited prior work experience and limited association with professional networks. Recommendations Several recommendations emerge from the evaluation of the TLP Scholarships program.: • The premise underlying the TLP-Scholarships program was that it would create a cadre of talented professionals with the leadership and innovation skills necessary to address the demands of a modern market economy and bring transformational change to Kosovo. This would happen through improved job performance and improvements in the public and private organizations where the TLP scholars get employment. Early evidence suggests that while the TLP scholars working in the private sector are already beginning to have an impact, no discernible effects are being seen in the public sector. First, very few TLP scholars have found positions within the government. Bringing about meaningful advancements in capacity development, and demonstrating different ways of doing things and directly influencing policy, would require a larger number of TLP scholars to be placed in the government. Second, bringing about reforms in the government system typically takes more time than in the private sector. In the words of a GOK Minister, a former TLP scholar, “the only way to reform the government is from within”. We therefore recommend that in future programs a connection is established between the beneficiaries of such efforts and the government, with the latter committing to utilizing the services of these graduates for two to three years to benefit from their skills. • In addition to increased presence within the public sector, we recommend setting up an institutionalized process by which TLP scholars can routinely meet with officers in government and public institutions and share their skills and experience. Creation of an advisory committee of scholarship recipients, academia and policy experts to support officials in various Ministries would go a long way in realizing this objective. • The public universities within Kosovo support the majority of the students for higher education. There is a shortage of skilled staff for several disciplines. But they have not been able to absorb many of the TLP scholars due to the current rules and regulations. We recommend that in future programs there is some coordination with the University of Prishtina (the single largest public university in Kosovo) to see if there is a way to support the University to fill the skills gap. • Volunteer work has helped both the TP scholars upon their return as well as their communities. The TLP scholars had something meaningful to do upon their return while they waited to hear back from job interviews, and the communities got some world-class assistance in grant writing and advocacy materials or in the cause they support. We recommend that volunteering continues to be included in future programs as well, preferably with a small stipend to help the volunteers tide over the period during which they are seeking employment. A higher stipend could be offered to those willing to work in rural areas, which would incentivize at least some of the graduates to temporarily move there and help some far-flung communities. • One downside that many TLP scholars have experienced of being away from Kosovo for two years is that of losing their social and professional networks. Strengthening support to the Alumni Association established by the TLP program would help TLP scholars enhance networking opportunities and share experiences and knowledge and exchange ideas. • In order to mitigate the reverse-culture shock that many TLP scholars have reported facing, we USAID Impact Evaluation of the TLP Scholarships Program– Final Report 38 recommend that future programs include training and support services to facilitate their reentry to Kosovo. These support services could cover topics such as culture in the workplace, optimal use of skills in a culturally-appropriate manner, taking initiative in the workplace, etc. A small subsistence allowance to cover the first few weeks of return to Kosovo would help the recipients to avoid rushing to take the first job offered. • We further recommend that prospective employers be offered guidance on the training received by the TLP scholars in the Master’s program in the U.S. This would help ensure that employers have reasonable expectations of the capabilities and interest of the returning scholarship recipients, which would mitigate frustration on both sides. • Discussions with a sample of employers suggest that TLP scholars who had some work experience prior to going to the U.S. for higher studies are more mature in their expectations and have been better at adjusting upon their return to Kosovo. We recommend that a two-year work experience be built in as a criterion for obtaining the scholarship. • The beneficiaries of the program have had to reactivate their social and professional networks. They have found that the networks of TLP graduates are more useful for them due to a shared understanding of their training and challenges upon return to Kosovo. We recommend continued support for the Alumni Association • Finally, we recommend that in future programs, all scholarship recipients be invited to a mandatory refresher course on leadership conducted in Kosovo. This course would serve to remind the scholarship recipients of the training provided to them while in the U.S., their own vision of leadership (as expressed in their essays), as well as offer practical suggestions on how leadership could be practiced within the context of the workplace and prevailing culture in Kosovo. USAID Impact Evaluation of the TLP Scholarships Program– Final Report 39 8 ANNEXES Annex 1. Scope Of Work For The Evaluation USAID Impact Evaluation of the TLP Scholarships Program– Final Report 40 USAID Impact Evaluation of the TLP Scholarships Program– Final Report 41 USAID Impact Evaluation of the TLP Scholarships Program– Final Report 42 USAID Impact Evaluation of the TLP Scholarships Program– Final Report 43 USAID Impact Evaluation of the TLP Scholarships Program– Final Report 44 USAID Impact Evaluation of the TLP Scholarships Program– Final Report 45 USAID Impact Evaluation of the TLP Scholarships Program– Final Report 46 USAID Impact Evaluation of the TLP Scholarships Program– Final Report 47 USAID Impact Evaluation of the TLP Scholarships Program– Final Report 48 Annex 2. Informational Meetings and Interviews Conducted (List includes everyone consulted at endline studies for Cohorts 1, 2 and 3) USAID staff Mrs. Melita Cacaj, Monitoring and Evaluation Specialist, COR TLP-SP evaluation project, PPO, E-mail: mcacaj@usaid.gov Mrs. Merita Teliqi, Project Specialist USAID,, AOR TLP-SP, E-mail mteliqi@usaid.gov TLP-SP Project Team Mr. Carl Hammerdorfer, Chief of Party, TLP-SP, E-mail: carl.hammerdorfer@worldlearning.org Ms. Rrita Limaj, Monitoring and Evaluation Specialist, TLP-SP, E-mail: rrita.limaj@worldlearning.org Ministry of Education, Science and Technology Mrs. Drita Kadriu, Head of Higher Education Department, E-mail: drita.kadriu@rks-gov.net Mrs. Qamile Sinanaj, Higher Education Department, E-mail: qamile.sinanaj@rks-gov.net Rectorate - University of Prishtina 1. Mr. Marjan Dema, Rector of the University of Prishtina, E-mail: marjan.dema@uni-pr.edu 2. Mrs. Myrvete Badivuku, Vice-Rector on Finance, E-mail: myrvete.badivuku@uni-pr.edu Office for Sponsored Research and Projects, University of Prishtina Mr. Gentrit Berisha, Head/Coordinator of Office for Sponsored Research and Projects, E-mail: gentrit.berisha@uni-pr.edu Members of UP Board 1. Mrs. Laura Kryeziu, Executive Director, Leadership and Growth Council, Member Advisory Committee, Email: laura.kryeziu@lgc-ks.org 2. Ardiana Bunjaku – ex member of the UP Board Citizen Corps Program 1. Mrs. Laura Kryeziu, Executive Director, Leadership and Growth Council, implementer TLP Citizen Corps Program, Email: laura.kryeziu@lgc-ks.org Focus groups and Interviews conducted 1. Individual interviews conducted with 17 employers 2. Focus groups and interviews with scholarship recipients (n=31) from Cohort 1, 2 and 3 USAID Impact Evaluation of the TLP Scholarships Program– Final Report 49 Annex 3. List of Documents Reviewed 1) TLP-SP Project Reports and Documents (2014-2019): • TLP-SP Technical Proposal • TLP-SP Work Plan Project Years 1, 2, 3, 4 and 5 • TLP-SP M&E Plan • TLP-SP Annual and Quarterly reports Project Years 1, 2, 3, 4 • TLP-SP Quarterly reports Project Year 5 Quarters 1, 2 and 3 • TLP-SP databases for Scholarship applicants and recipients databases • Other Technical Reports and Documents (2014-2019) • TLP SP website – newsroom accessed online at http://www.usaid-tlp-sp.org/En/lajme/ • TLP-SP success stories – accessed online at https://www.usaid.gov/results-data/success-stories/kosovo-scholar-promotes-startups￾through-alma-mater https://www.usaid.gov/results-data/success-stories/kosovo-scholar-connects-kosovo-its￾diaspora • https://www.usaid.gov/news-information/frontlines/september-october-2016 2) Citizen Corps reports and documents • List of partner organizations • Volunteer placements database for returned scholarship (Master’s degree)recipients 3) Other reports as referenced within the document USAID Impact Evaluation of the TLP Scholarships Program– Final Report 50 Annex 4. Survey Instruments 4.1.Scholarship Recipients Endline Survey Introduction Background Information 1) Please tell us your name?* First name: _________________________________________________ Last name: _________________________________________________ 2) What is your gender? ( ) Female ( ) Male ( ) Other - Please Write In: _________________________________________________ 3) Disability status? ( ) No Disability ( ) Hearing ( ) Vision ( ) Physical Impairment ( ) Other - Please Write In: _________________________________________________ ( ) Prefer not to say 4) What is your ethnicity? ( ) Albanian ( ) Serbian ( ) Bosniak ( ) Turk ( ) Roma ( ) Ashkali ( ) Egyptian ( ) Prefer not to say ( ) Other - Please Write In: _________________________________________________ Master's Program 5) In which year did you receive the TLP scholarship? ( ) September 2014 ( ) September 2015 ( ) September 2016 6) What was your field of study in the Master's program? ( ) Agriculture ( ) Business Management ( ) Economics ( ) Education ( ) Engineering ( ) Finance ( ) Gender and Interdisciplinary Studies ( ) Information Systems ( ) Law USAID Impact Evaluation of the TLP Scholarships Program– Final Report 51 ( ) Public Administration ( ) Other - Please Write In: _________________________________________________ 7) When did you complete your TLP Master's program? * Month and Year - Please Write In: _________________________________________________ Your current status 8) Where are you currently based?* ( ) In Kosovo ( ) In United States ( ) In another country - write in name of country: _________________________________________________ 9) When did you return to Kosovo? Month and Year - Please write In: _________________________________________________ 10) Where are you currently based within Kosovo? ( ) In Prishtina ( ) In another city ( ) In a rural area ( ) Other (Write In): _________________________________________________ 11) What are you doing now? (Check all that apply)* [ ] Employed [ ] Self-employed [ ] Unemployed and looking for a job [ ] Taking a break and not looking for a job [ ] Doing an internship [ ] Volunteering [ ] Enrolled in an academic program [ ] Other - Please Write In (Required): _________________________________________________* 12) Please tell us about your current academic program. ( ) PhD program in Kosovo in the same field as my Masters ( ) PhD program in Kosovo in a different field from my Masters ( ) Masters level program of study in Kosovo in a different field from my Master’s ( ) PhD program in the US in the same field as my Masters ( ) PhD program in the US in a different field from my Masters ( ) Masters level program of study in the US in a different field ( ) PhD program in a foreign university (not in US) in the same field as my Masters ( ) PhD program in a foreign university (not in US) in a different field from my Masters ( ) Masters level program of study in a foreign university (not in US) in a different field ( ) Other - Please Write In: _________________________________________________ Details of Employment 13) Please tell us about your primary place of employment. (Please select one)* ( ) Government or a public agency ( ) Public university ( ) Private university ( ) Private organization ( ) NGO ( ) Multinational/donor organization USAID Impact Evaluation of the TLP Scholarships Program– Final Report 52 ( ) Running own business ( ) Other - Please Write In: _________________________________________________ 14) Please tell us about the sector in which you are working? * ( ) Agriculture ( ) Business Management ( ) Economics ( ) Education ( ) Engineering ( ) Finance ( ) Gender and Interdisciplinary Studies ( ) Information Systems ( ) International Development ( ) Law ( ) Public Administration ( ) Other - Please Write In: _________________________________________________ 15) Is this position: ( ) Part-time ( ) Full-time 16) What is your role? (Please select one) ( ) I am an intern ( ) I am in an entry level position ( ) I am in a mid-level supervisory position ( ) I am in a mid-level technical non-supervisory position ( ) I am in a senior/leadership position ( ) I head the organization ( ) I am an independent consultant ( ) Other - Please write In (Required): _________________________________________________* 17) What is your title/position in the organization? ( ) Title/Position (Please write in): _________________________________________________ 18) What is the name of the organization where you work?* ( ) Name of organization - Please write In: _________________________________________________ Employment Cont'd 19) Is this your first job after you completed your TLP Master's program? ( ) Yes ( ) No 20) How long ago did you take up your current position? ( ) 1-3 months ( ) 4-6 months ( ) 7 months to 1 year ( ) 1 to 2 years ago ( ) Immediately upon my return to Kosovo 21) We would like to learn more about your current position as compared with your position prior to the TLP Master’s program. USAID Impact Evaluation of the TLP Scholarships Program– Final Report 53 ( ) This is the same job I had prior to attending the Master’s program ( ) This is a senior position within my previous organization ( ) This is a lower position than I had previously ( ) This is a senior position with a different organization ( ) This is a very different position with a new organization 22) To what extent do you agree with the following statements about your current job as compared with your position prior to the TLP Master’s program? (Please note the column headings) Strongly Disagree Disagree Neutral Agree Strongly Agree This position is better paid than my previous position ( ) ( ) ( ) ( ) ( ) This position includes greater responsibility than my previous position ( ) ( ) ( ) ( ) ( ) 23) How satisfied are you with your current position? ( ) Very satisfied with my position ( ) Somewhat satisfied, this is a stop gap position ( ) Neither satisfied nor dissatisfied ( ) Not at all satisfied, I am looking to change 24) Still thinking of your current position, are any of the following included in your job responsibilities? This is my primary responsibility I sometimes help colleagues with these tasks Not at all Research ( ) ( ) ( ) Product development ( ) ( ) ( ) Grant writing ( ) ( ) ( ) Support policy and program development ( ) ( ) ( ) Program administration ( ) ( ) ( ) USAID Impact Evaluation of the TLP Scholarships Program– Final Report 54 Laboratory management ( ) ( ) ( ) Mentoring others ( ) ( ) ( ) Working on multi-disciplinary teams ( ) ( ) ( ) Communicating effectively with others inside and outside my field ( ) ( ) ( ) Building consensus around issues ( ) ( ) ( ) Strategy development ( ) ( ) ( ) Identifying new business and direction for my group ( ) ( ) ( ) Internship 25) Since you completed the TLP master's program, have you completed an internship or are currently doing one? * ( ) Yes, I am currently doing an internship ( ) Yes, I have completed an internship since I completed my TLP Master's program ( ) No, I am neither doing an internship nor have completed one since completing my Masters Internship - details 26) Please tell us about the place of internship. (Please select one) ( ) Government or a public agency ( ) Public University ( ) Private university ( ) Private organization ( ) NGO ( ) Multinational/donor organization ( ) Other - Please Write In: _________________________________________________ 27) Please tell us about the sector in which you did/are doing your internship? ( ) Agriculture ( ) Business Management ( ) Economics ( ) Education ( ) Engineering ( ) Finance ( ) Gender and Interdisciplinary Studies ( ) Information Systems ( ) Law USAID Impact Evaluation of the TLP Scholarships Program– Final Report 55 ( ) Public Administration ( ) Other - Please Write In: _________________________________________________ 28) What is/was your role as an intern? (Please select one) ( ) Entry level position ( ) Mid-level supervisory position ( ) Senior/leadership position ( ) Other - Please Write In 29) Is/was this position: ( ) Part-time ( ) Full-time 30) How long ago did you take up this internship? ( ) 1-3 months ( ) 4-6 months ( ) 7 months to 1 year ( ) More than 1 year ago ( ) Immediately upon my return to Kosovo 31) What has been your contribution as an intern? Strongly Disagree Disagree Neutral Agree Strongly Agree Supporting policy and program development ( ) ( ) ( ) ( ) ( ) Conducting research to support white papers ( ) ( ) ( ) ( ) ( ) Doing entry-level administrative work ( ) ( ) ( ) ( ) ( ) Organizing databases ( ) ( ) ( ) ( ) ( ) Knowledge management ( ) ( ) ( ) ( ) ( ) Building websites ( ) ( ) ( ) ( ) ( ) Supporting the university/organization’s communications team (writing short stories, PR articles, developing videos, etc.) ( ) ( ) ( ) ( ) ( ) Supporting grant and proposal writing ( ) ( ) ( ) ( ) ( ) USAID Impact Evaluation of the TLP Scholarships Program– Final Report 56 Seeking Employment 32) Has it been easy to find a job after completing your Master's program? ( ) Yes ( ) No 33) Are you currently looking for a job? ( ) Yes, I am actively seeking a new position ( ) I am not looking for a job ( ) I am enrolled in an academic program so not looking for a position ( ) I am not looking for a position as I am doing an internship/volunteer assignment ( ) Other - please specify: _________________________________________________ 34) What factors (if any) have contributed to it being difficult to find a job? (Check all that apply.) [ ] Lack of attractive positions [ ] Mismatch between my training and available positions [ ] Poor economy [ ] No/few jobs available in the area where I live [ ] Lack of networking opportunities [ ] Sense of bias against foreign graduates [ ] Long process of validation of foreign degrees and diplomas [ ] Unfair selection processes and standards [ ] Low salaries offered [ ] Other - please specify: _________________________________________________ Skills development and utilization 35) Please tell us the extent to which you agree with the following statements. As a result of the skills developed during my TLP Masters program, I am able to: Strongly Disagree Disagree Neutral Agree Strongly Agree Share and demonstrate in-depth understanding in my area of studies ( ) ( ) ( ) ( ) ( ) Take initiative in any work environment ( ) ( ) ( ) ( ) ( ) Transfer the knowledge in my field of study to others ( ) ( ) ( ) ( ) ( ) Work with and under the leadership of others ( ) ( ) ( ) ( ) ( ) Communicate effectively with others inside and outside my field ( ) ( ) ( ) ( ) ( ) USAID Impact Evaluation of the TLP Scholarships Program– Final Report 57 Work effectively in multidisciplinary teams ( ) ( ) ( ) ( ) ( ) Lead teams that include international scientists/think tanks inside and outside my field ( ) ( ) ( ) ( ) ( ) Build consensus around issues important to my organization ( ) ( ) ( ) ( ) ( ) See the “big picture” ( ) ( ) ( ) ( ) ( ) I have taken on a leadership role and am introducing new ideas to my organization and community ( ) ( ) ( ) ( ) ( ) 36) I am able to use the following skills developed as a result of my Master's studies program: Strongly Disagree Disagree Neutral Agree Strongly Agree Critical thinking ( ) ( ) ( ) ( ) ( ) Organizing ideas ( ) ( ) ( ) ( ) ( ) Finding information ( ) ( ) ( ) ( ) ( ) Writing skills- writing productively, efficiently and clearly ( ) ( ) ( ) ( ) ( ) Reading skills ( ) ( ) ( ) ( ) ( ) Creativity ( ) ( ) ( ) ( ) ( ) Ability to work with numbers and graphs ( ) ( ) ( ) ( ) ( ) Ability to conduct research ( ) ( ) ( ) ( ) ( ) Ability to ask questions ( ) ( ) ( ) ( ) ( ) Methodological knowledge ( ) ( ) ( ) ( ) ( ) Analytical skills ( ) ( ) ( ) ( ) ( ) Problem solving skills ( ) ( ) ( ) ( ) ( ) USAID Impact Evaluation of the TLP Scholarships Program– Final Report 58 Time management ( ) ( ) ( ) ( ) ( ) 37) To what extent do you agree with the following statements about your use of skills (developed as a result of the Master’s program) at the organizational or community levels: Strongly Disagree Disagree Neutral Agree Strongly Agree I have introduced new ways of thinking and problem solving, by sharing skills with colleagues ( ) ( ) ( ) ( ) ( ) I am using ethical ways to conduct research ( ) ( ) ( ) ( ) ( ) I use data and analysis for policy making ( ) ( ) ( ) ( ) ( ) I use discussion as a way to resolve problems at the organizational level ( ) ( ) ( ) ( ) ( ) I use more participatory methods for consensus building, especially at the organizational level ( ) ( ) ( ) ( ) ( ) I use research to support discussion and advocacy ( ) ( ) ( ) ( ) ( ) I am leading interdisciplinary teams to work with national and international partners ( ) ( ) ( ) ( ) ( ) I am using digital and multimedia tools to support advocacy and policy making ( ) ( ) ( ) ( ) ( ) 38) To what extent do you agree with the following statements about your use of skills (developed as a result of the Master’s program) to support the development of Kosovo: Strongly Disagree Disagree Neutral Agree Strongly Agree I am building capacity of my colleagues at work ( ) ( ) ( ) ( ) ( ) I have developed/clarified the strategy for my organization to attain its vision ( ) ( ) ( ) ( ) ( ) I am contributing to the economic ( ) ( ) ( ) ( ) ( ) USAID Impact Evaluation of the TLP Scholarships Program– Final Report 59 development of Kosovo through my start￾up and entrepreneurial efforts I have brought new business to Kosovo through international partnerships ( ) ( ) ( ) ( ) ( ) I support organizations within Kosovo through knowledge sharing ( ) ( ) ( ) ( ) ( ) I support organizations by helping them establish linkages/network with other organizations inside and outside Kosovo ( ) ( ) ( ) ( ) ( ) I volunteer my time (beyond the mandatory hours) to support and build capacity of organizations in my community ( ) ( ) ( ) ( ) ( ) I provide support with research and writing grants that bring about change in my community ( ) ( ) ( ) ( ) ( ) I do not see any need to support my community ( ) ( ) ( ) ( ) ( ) 39) From your perspective, what are some of the barriers to your use of skills (developed as a result of the Master’s program) in your work at the organizational or community levels Strongly Disagree Disagree Neutral Agree Strongly Agree Unwillingness of the organization/community to change ( ) ( ) ( ) ( ) ( ) Lack of sufficient resources to engage people and introduce new skills and concepts ( ) ( ) ( ) ( ) ( ) There is no organizational support for sharing of new ideas ( ) ( ) ( ) ( ) ( ) I am not senior enough within the organization to make a change ( ) ( ) ( ) ( ) ( ) Deeply entrenched ideas take a while to change ( ) ( ) ( ) ( ) ( ) There are no incentives for change ( ) ( ) ( ) ( ) ( ) USAID Impact Evaluation of the TLP Scholarships Program– Final Report 60 Plans 40) What are your plans for the future (next 3-5 years)? ( ) Stay and grow in my current job ( ) Change jobs to one with better salary or greater responsibility ( ) Go abroad to work ( ) Start my own company ( ) Join a PhD/ post-graduate program in Kosovo ( ) Join a PhD/ post-graduate program in the US ( ) Join a PhD/ post-graduate program abroad (not US) Leadership 41) To what extent do you agree with the following statements? In my opinion, Not at all Once in a while Sometimes Fairly often Frequently, if not always Does not apply I am a leader in my organization and community ( ) ( ) ( ) ( ) ( ) ( ) My colleagues think of me as a leader ( ) ( ) ( ) ( ) ( ) ( ) My employers look upon me as a leader ( ) ( ) ( ) ( ) ( ) ( ) My community thinks of me a leader ( ) ( ) ( ) ( ) ( ) ( ) Leadership Style 42) Leadership Style - Section 1 Not at all Once in a while Sometimes Fairly often Frequently, if not always I am effective in meeting others’ job-related needs ( ) ( ) ( ) ( ) ( ) I use methods of leadership that are satisfying ( ) ( ) ( ) ( ) ( ) I get others to do more than ( ) ( ) ( ) ( ) ( ) USAID Impact Evaluation of the TLP Scholarships Program– Final Report 61 they expected to do I am effective in representing others to higher authority ( ) ( ) ( ) ( ) ( ) I work with others in a satisfactory way ( ) ( ) ( ) ( ) ( ) I heighten others’ desire to succeed ( ) ( ) ( ) ( ) ( ) I am effective in meeting organizational requirements ( ) ( ) ( ) ( ) ( ) I increase others’ willingness to try harder ( ) ( ) ( ) ( ) ( ) I lead a group that is effective ( ) ( ) ( ) ( ) ( ) 43) Leadership Style - Section 2 Not at all Once in a while Sometimes Fairly often Frequently, if not always I provide others with assistance in exchange for their efforts ( ) ( ) ( ) ( ) ( ) I re-examine critical assumptions to question whether they are appropriate ( ) ( ) ( ) ( ) ( ) I fail to interfere until problems become serious ( ) ( ) ( ) ( ) ( ) I focus attention on irregularities, mistakes, exceptions, and deviations from standards ( ) ( ) ( ) ( ) ( ) I avoid getting involved when important issues arise ( ) ( ) ( ) ( ) ( ) I talk about my most important values and beliefs ( ) ( ) ( ) ( ) ( ) I am absent when needed ( ) ( ) ( ) ( ) ( ) USAID Impact Evaluation of the TLP Scholarships Program– Final Report 62 I seek differing perspectives when solving problems ( ) ( ) ( ) ( ) ( ) I talk optimistically about the future ( ) ( ) ( ) ( ) ( ) 44) Leadership Style - Section 3. Not at all Once in a while Sometimes Fairly often Frequently, if not always I instill pride in others for being associated with me ( ) ( ) ( ) ( ) ( ) I discuss in specific terms who is responsible for achieving performance targets ( ) ( ) ( ) ( ) ( ) I wait for things to go wrong before taking action ( ) ( ) ( ) ( ) ( ) I talk enthusiastically about what needs to be accomplished ( ) ( ) ( ) ( ) ( ) I specify the importance of having a strong sense of purpose ( ) ( ) ( ) ( ) ( ) I spend time teaching and coaching ( ) ( ) ( ) ( ) ( ) I make clear what one can expect to receive when performance goals are achieved ( ) ( ) ( ) ( ) ( ) I show that I am a firm believer in “If it ain’t broke, don’t fix it.” ( ) ( ) ( ) ( ) ( ) I go beyond self￾interest for the good of the group ( ) ( ) ( ) ( ) ( ) USAID Impact Evaluation of the TLP Scholarships Program– Final Report 63 45) Leadership Style - Section 4. Not at all Once in a while Sometimes Fairly often Frequently, if not always I treat others as individuals rather than just as a member of a group ( ) ( ) ( ) ( ) ( ) I demonstrate that problems must become chronic before I take action ( ) ( ) ( ) ( ) ( ) I act in ways that build others’ respect for me ( ) ( ) ( ) ( ) ( ) I concentrate my full attention on dealing with mistakes, complaints, and failures ( ) ( ) ( ) ( ) ( ) I consider the moral and ethical consequences of decisions ( ) ( ) ( ) ( ) ( ) I keep track of all mistakes ( ) ( ) ( ) ( ) ( ) I display a sense of power and confidence ( ) ( ) ( ) ( ) ( ) I articulate a compelling vision of the future ( ) ( ) ( ) ( ) ( ) I direct my attention toward failures to meet standards ( ) ( ) ( ) ( ) ( ) 46) Leadership Style - Section 5. Not Once in Sometimes Fairly Frequently, if not USAID Impact Evaluation of the TLP Scholarships Program– Final Report 64 at all a while often always I avoid making decisions ( ) ( ) ( ) ( ) ( ) I consider an individual as having different needs, abilities, and aspirations from others ( ) ( ) ( ) ( ) ( ) I get others to look at problems from many different angles ( ) ( ) ( ) ( ) ( ) I help others to develop their strengths ( ) ( ) ( ) ( ) ( ) I suggest new ways of looking at how to complete assignments ( ) ( ) ( ) ( ) ( ) I delay responding to urgent questions ( ) ( ) ( ) ( ) ( ) I emphasize the importance of having a collective sense of mission ( ) ( ) ( ) ( ) ( ) I express satisfaction when others meet expectations ( ) ( ) ( ) ( ) ( ) I express confidence that goals will be achieved ( ) ( ) ( ) ( ) ( ) Confirmation page 47) As a token of our appreciation for taking the time to complete this survey, we would like to provide you with a phone charge in the value of 15 Euros.* ( ) I do not want the incentive ( ) I want the incentive (please provide your phone number below) 48) Please provide us with your phone number where we should send the phone charge. Please make sure to confirm the phone number so we have the correct information. (Please allow for two weeks to receive the incentives from the time you complete the survey).* [ ] Phone Number - Please Write In (Required): _________________________________________________* [ ] Re-enter phone number - - Please Write In (Required): ___________________________________________* 49) Congratulations on completing the survey. This is the last page for this survey. If you would like to review your response please do so now using the Back button below. If you are satisfied with your responses, press "Submit" * USAID Impact Evaluation of the TLP Scholarships Program– Final Report 65 ( ) Submit Thank You! USAID Impact Evaluation of the TLP Scholarships Program– Final Report 66 4.2.TLP Comparison Endline Survey TLP applicants endline survey Introduction Background Information 1) Please tell us your name?* First name: _________________________________________________ Last name: _________________________________________________ 2) What is your gender? ( ) Female ( ) Male 3) Disability status? ( ) No Disability ( ) Hearing ( ) Vision ( ) Physical Impairment ( ) Other - Write In: _________________________________________________ ( ) Prefer not to say 4) What is your ethnicity?* ( ) Albanian ( ) Serbian ( ) Bosniak ( ) Turk ( ) Roma ( ) Ashkali ( ) Egyptian ( ) Other - Write In: _________________________________________________ ( ) Prefer not to say Academic Program 5) What is your highest academic degree?* ( ) Bachelor's ( ) Master's ( ) Professional degree/certificate ( ) Ph.D ( ) Other - Write In: _________________________________________________ 6) What was your field of study in the program? * ( ) Agriculture ( ) Business Management ( ) Economics ( ) Education USAID Impact Evaluation of the TLP Scholarships Program– Final Report 67 ( ) Engineering ( ) Finance ( ) Gender and Interdisciplinary Studies ( ) Information Systems ( ) Law ( ) Public Administration ( ) Other - Write In: _________________________________________________ 7) When did you start your highest degree program? ( ) Year enrolled - Write In (format yyyy): _________________________________________________ 8) When did you complete your highest degree program? ( ) Did not complete the program ( ) Program is ongoing ( ) Year completed - Write In (format yyyy): _________________________________________________ 9) Please tell us the name and location of the institution that awarded you your highest degree? Name of institution: _________________________________________________ Country where this institution is located: _________________________________________________ Your current status 10) Where are you currently based?* ( ) In Kosovo ( ) In United States ( ) In another country - write in name of country: _________________________________________________ 11) Where are you currently based within Kosovo? ( ) In Prishtina ( ) In another city ( ) In a rural area ( ) Other - Write In: _________________________________________________ 12) What are you doing now? (Check all that apply)* [ ] Employed [ ] Self-employed [ ] Unemployed and looking for a job [ ] Taking a break and not looking for a job [ ] Doing an internship [ ] Volunteering [ ] Enrolled in an academic program [ ] Other - Write In: _________________________________________________ 13) Please tell us about your current academic program. ( ) A second Bachelor's program ( ) Masters program ( ) PhD program ( ) Other - Write In: _________________________________________________ 14) What is your field of study in the program? ( ) Agriculture ( ) Business Management USAID Impact Evaluation of the TLP Scholarships Program– Final Report 68 ( ) Economics ( ) Education ( ) Engineering ( ) Finance ( ) Gender and Interdisciplinary Studies ( ) Information Systems ( ) Law ( ) Public Administration ( ) Other - Write In: _________________________________________________ Details of Employment 15) Please tell us about your primary place of employment. (Please select one) ( ) Government or a public agency ( ) Public university ( ) Private university ( ) Private organization ( ) NGO ( ) Multinational/donor organization ( ) Running own business ( ) Other - Write In: _________________________________________________ 16) Please tell us about the sector in which you are working? ( ) Agriculture ( ) Business Management ( ) Economics ( ) Education ( ) Engineering ( ) Finance ( ) Gender and Interdisciplinary Studies ( ) Information Systems ( ) International Development ( ) Law ( ) Public Administration ( ) Other - Write In: _________________________________________________ 17) Is this position: ( ) Part-time ( ) Full-time 18) What is your role? (Please select one) ( ) I am an intern ( ) I am in an entry level position ( ) I am in a mid-level supervisory position ( ) I am in a mid-level technical non-supervisory position ( ) I am in a senior/leadership position ( ) I head the organization ( ) I am an independent consultant ( ) Other - Write In: _________________________________________________ 19) What is your title/position in the organization? ( ) Title/Position - Write In (Required): _________________________________________________* USAID Impact Evaluation of the TLP Scholarships Program– Final Report 69 20) What is the name of the organization? ( ) Name of organization - Write In (Required): _________________________________________________* 21) Is this your first job after you completed your highest degree? ( ) Yes ( ) No 22) How long ago did you take up your current position? ( ) 1-3 months ( ) 4-6 months ( ) 7 months to 1 year ( ) 1 to 2 years ago 23) We would like to learn more about your current position as compared with your position prior to applying for the TLP Master’s program. ( ) This is the same job I had prior to applying for the TLP program ( ) This is a senior position within my previous organization ( ) This is a lower position than I had previously ( ) This is a senior position with a different organization ( ) This is a very different position with a new organization 24) To what extent do you agree with the following statements about your current job as compared with your position prior to applying for the TLP Master’s program? (Please note the column headings) Strongly Disagree Disagree Neutral Agree Strongly Agree This position is better paid than my previous position ( ) ( ) ( ) ( ) ( ) This position includes greater responsibility than my previous position ( ) ( ) ( ) ( ) ( ) 25) Still thinking of your current position, are any of the following included in your job responsibilities? This is my primary responsibility I sometimes help colleagues with these tasks Not at all Research ( ) ( ) ( ) Product development ( ) ( ) ( ) Grant writing ( ) ( ) ( ) USAID Impact Evaluation of the TLP Scholarships Program– Final Report 70 Support policy and program development ( ) ( ) ( ) Program administration ( ) ( ) ( ) Laboratory management ( ) ( ) ( ) Mentoring others ( ) ( ) ( ) Working on multi-disciplinary teams ( ) ( ) ( ) Communicating effectively with others inside and outside my field ( ) ( ) ( ) Building consensus around issues ( ) ( ) ( ) Strategy development ( ) ( ) ( ) Identifying new business and direction for my group ( ) ( ) ( ) 26) How satisfied are you with your current position? ( ) Very satisfied with my position ( ) Somewhat satisfied, this is a stop gap position ( ) Neither satisfied nor dissatisfied ( ) Not at all satisfied, I am looking to change Volunteer activity 27) Have you engaged in any volunteer activities, since your TLP application? (select one) * ( ) Yes, I am currently engaged in a volunteer activity ( ) Yes, I have completed a volunteer activity since I applied ( ) No, I am neither doing doing a volunteer activity nor have I completed one Volunteering - details 28) Approximately how many total hours have you contributed to volunteer activities (Please select one) ( ) None at all ( ) 1-25 hours ( ) 26-50 ( ) 51-100 ( ) 100-200 ( ) 200-400 ( ) Over 400 hours USAID Impact Evaluation of the TLP Scholarships Program– Final Report 71 29) Please tell us about the place of volunteer activity. (Please select one) ( ) Government org a public agency ( ) Public University ( ) Private university ( ) Private organization ( ) NGO ( ) Multinational/donor organization ( ) Community level work ( ) Other - Write In: _________________________________________________ 30) Please tell us about the sector in which you are/were volunteering? ( ) Agriculture ( ) Business Management ( ) Economics ( ) Education ( ) Engineering ( ) Finance ( ) Gender and Interdisciplinary Studies ( ) Information Systems ( ) Law ( ) Public Administration ( ) Other - Write In: _________________________________________________ 31) What is/was your role as a volunteer? (Please select one) ( ) Entry level position ( ) Mid-level supervisory position ( ) Senior/leadership position ( ) Other - Write In: _________________________________________________ 32) Is/was this position: ( ) Part-time ( ) Full-time 33) How long ago did you take up this volunteer position? ( ) 1-3 months ( ) 4-6 months ( ) 7 months to 1 year ( ) More than 1 year ago 34) What has been your contribution as a volunteer? Strongly Disagree Disagree Neutral Agree Strongly Agree Supporting policy and program development ( ) ( ) ( ) ( ) ( ) Conducting research to support white papers ( ) ( ) ( ) ( ) ( ) USAID Impact Evaluation of the TLP Scholarships Program– Final Report 72 Doing entry-level administrative work ( ) ( ) ( ) ( ) ( ) Organizing databases ( ) ( ) ( ) ( ) ( ) Knowledge management ( ) ( ) ( ) ( ) ( ) Building websites ( ) ( ) ( ) ( ) ( ) Supporting the university/ organization’s communications team (writing short stories, PR articles, developing videos, etc.) ( ) ( ) ( ) ( ) ( ) Supporting grant and proposal writing ( ) ( ) ( ) ( ) ( ) Providing training on new methods and skills ( ) ( ) ( ) ( ) ( ) Fundraising ( ) ( ) ( ) ( ) ( ) Internship 35) Have you completed an internship since your TLP application, or are currently doing one? * ( ) Yes, I am currently doing an internship ( ) Yes, I have completed an internship ( ) No, I am neither doing an internship nor have I completed one Internship - details 36) Please tell us about the place of internship. (Please select one) ( ) Government or a public agency ( ) Public University ( ) Private university ( ) Private organization ( ) NGO ( ) Multinational/donor organization ( ) Other - Write In: _________________________________________________ 37) Please tell us about the sector in which you did/are doing your internship? ( ) Agriculture ( ) Business Management ( ) Economics ( ) Education ( ) Engineering ( ) Finance USAID Impact Evaluation of the TLP Scholarships Program– Final Report 73 ( ) Gender and Interdisciplinary Studies ( ) Information Systems ( ) Law ( ) Public Administration ( ) Other - Write In: _________________________________________________ 38) What is/was your role as an intern? (Please select one) ( ) Entry level position ( ) Mid-level supervisory position ( ) Senior/leadership position ( ) Other - Write In: _________________________________________________ 39) Is/was this position: ( ) Part-time ( ) Full-time 40) How long ago did you take up this internship? ( ) 1-3 months ( ) 4-6 months ( ) 7 months to 1 year ( ) More than 1 year ago 41) What has been your contribution as an intern? Strongly Disagree Disagree Neutral Agree Strongly Agree Supporting policy and program development ( ) ( ) ( ) ( ) ( ) Conducting research to support white papers ( ) ( ) ( ) ( ) ( ) Doing entry-level administrative work ( ) ( ) ( ) ( ) ( ) Organizing databases ( ) ( ) ( ) ( ) ( ) Knowledge management ( ) ( ) ( ) ( ) ( ) Building websites ( ) ( ) ( ) ( ) ( ) Supporting the university/ organization’s communications team (writing short stories, PR articles, developing videos, etc.) ( ) ( ) ( ) ( ) ( ) Supporting grant and proposal writing ( ) ( ) ( ) ( ) ( ) USAID Impact Evaluation of the TLP Scholarships Program– Final Report 74 Seeking positions 42) Has it been easy to find a job after completing your highest degree? ( ) Yes ( ) No 43) Are you currently looking for a job?* ( ) Yes, I am actively seeking a new position ( ) I am not looking for a job ( ) I am enrolled in an academic program so not looking for a position ( ) I am not looking for a position as I am doing an internship/volunteer assignment ( ) Other - please specify: _________________________________________________ 44) What factors (if any) have contributed to it being difficult to find a job? (Check all that apply)* ( ) Lack of attractive positions ( ) Mismatch between my training and available positions ( ) Poor economy ( ) No/few jobs available in the area where I live ( ) Lack of networking opportunities ( ) Sense of bias against foreign graduates ( ) Long process of validation of foreign degrees and diplomas ( ) Unfair selection processes and standards ( ) Low salaries offered ( ) Other - please specify: _________________________________________________ ( ) Not applicable Skills development and utilization 45) I am able to use the following skills developed as a result of my academic program: Strongly Disagree Disagree Neutral Agree Strongly Agree Critical thinking ( ) ( ) ( ) ( ) ( ) Organizing ideas ( ) ( ) ( ) ( ) ( ) Finding information ( ) ( ) ( ) ( ) ( ) Writing skills- writing productively, efficiently and clearly ( ) ( ) ( ) ( ) ( ) Reading skills ( ) ( ) ( ) ( ) ( ) Creativity ( ) ( ) ( ) ( ) ( ) Ability to work with ( ) ( ) ( ) ( ) ( ) USAID Impact Evaluation of the TLP Scholarships Program– Final Report 75 numbers and graphs Ability to conduct research ( ) ( ) ( ) ( ) ( ) Ability to ask questions ( ) ( ) ( ) ( ) ( ) Methodological knowledge ( ) ( ) ( ) ( ) ( ) Analytical skills ( ) ( ) ( ) ( ) ( ) Problem solving skills ( ) ( ) ( ) ( ) ( ) Time management ( ) ( ) ( ) ( ) ( ) 46) Please tell us the extent to which you agree with the following statements. As a result of the skills developed during my academic program, I am able to: Strongly Disagree Disagree Neutral Agree Strongly Agree Share and demonstrate in￾depth understanding in my area of studies ( ) ( ) ( ) ( ) ( ) Take initiative in any work environment ( ) ( ) ( ) ( ) ( ) Transfer the knowledge in my field of study to others ( ) ( ) ( ) ( ) ( ) Work with and under the leadership of others ( ) ( ) ( ) ( ) ( ) Communicate effectively with others inside and outside my field ( ) ( ) ( ) ( ) ( ) Work effectively in multidisciplinary teams ( ) ( ) ( ) ( ) ( ) Lead teams that include international scientists/think ( ) ( ) ( ) ( ) ( ) USAID Impact Evaluation of the TLP Scholarships Program– Final Report 76 tanks inside and outside my field Build consensus around issues important to my organization ( ) ( ) ( ) ( ) ( ) See the “big picture” ( ) ( ) ( ) ( ) ( ) I have taken on a leadership role and am introducing new ideas to my organization and community ( ) ( ) ( ) ( ) ( ) 47) To what extent do you agree with the following statements about your use of skills at the organizational or community levels: Strongly Disagree Disagree Neutral Agree Strongly Agree I have introduced new ways of thinking and problem solving, by sharing skills with colleagues ( ) ( ) ( ) ( ) ( ) I am using ethical ways to conduct research ( ) ( ) ( ) ( ) ( ) I use data and analysis for policy making ( ) ( ) ( ) ( ) ( ) I use discussion as a way to resolve problems at the organizational level ( ) ( ) ( ) ( ) ( ) I use more participatory methods for consensus building, especially at the organizational level ( ) ( ) ( ) ( ) ( ) I use research to support discussion and advocacy ( ) ( ) ( ) ( ) ( ) I am leading interdisciplinary teams to work with national and international partners ( ) ( ) ( ) ( ) ( ) I am using digital and multimedia tools to support advocacy and policy making ( ) ( ) ( ) ( ) ( ) 48) To what extent do you agree with the following statements about the ways in which you are supporting the USAID Impact Evaluation of the TLP Scholarships Program– Final Report 77 development of Kosovo: Strongly Disagree Disagree Neutral Agree Strongly Agree I am building capacity of my colleagues at work ( ) ( ) ( ) ( ) ( ) I have developed/clarified the strategy for my organization to attain its vision ( ) ( ) ( ) ( ) ( ) I am contributing to the economic development of Kosovo through my start-up and entrepreneurial efforts ( ) ( ) ( ) ( ) ( ) I have brought new business to Kosovo through international partnerships ( ) ( ) ( ) ( ) ( ) I support organizations within Kosovo through knowledge sharing ( ) ( ) ( ) ( ) ( ) I support organizations by helping them establish linkages/network with other organizations inside and outside Kosovo ( ) ( ) ( ) ( ) ( ) I volunteer my time to support and build capacity of organizations in my community ( ) ( ) ( ) ( ) ( ) I provide support with research and writing grants that bring about change in my community ( ) ( ) ( ) ( ) ( ) I do not see any need to support my community ( ) ( ) ( ) ( ) ( ) 49) From your perspective, what are some of the barriers to your use of skills: Strongly Disagree Disagree Neutral Agree Strongly Agree Not Applicable Unwillingness of the organization/community to change ( ) ( ) ( ) ( ) ( ) ( ) USAID Impact Evaluation of the TLP Scholarships Program– Final Report 78 Lack of sufficient resources to engage people and introduce new skills and concepts ( ) ( ) ( ) ( ) ( ) ( ) There is no organizational support for sharing of new ideas ( ) ( ) ( ) ( ) ( ) ( ) I am not senior enough within the organization to make a change ( ) ( ) ( ) ( ) ( ) ( ) Deeply entrenched ideas take a while to change ( ) ( ) ( ) ( ) ( ) ( ) There are no incentives for change ( ) ( ) ( ) ( ) ( ) ( ) Plans 50) What are your plans for the future (next 3-5 years)? * ( ) Stay and grow in my current job ( ) Change jobs to one with better salary or greater responsibility ( ) Go abroad to work ( ) Start my own company ( ) Join a PhD/ post-graduate program in Kosovo ( ) Join a PhD/ post-graduate program in the US ( ) Join a PhD/ post-graduate program abroad (not US) ( ) Other - Write In: _________________________________________________ To what extent do you agree with the following statements? In my opinion, * Not at all Once in a while Sometimes Fairly often Frequently, if not always Does not apply I am a leader in my organization and community ( ) ( ) ( ) ( ) ( ) ( ) My colleagues think of me ( ) ( ) ( ) ( ) ( ) ( ) USAID Impact Evaluation of the TLP Scholarships Program– Final Report 79 as a leader My employers look upon me as a leader ( ) ( ) ( ) ( ) ( ) ( ) My community thinks of me a leader ( ) ( ) ( ) ( ) ( ) ( ) Leadership Style Leadership Style - Section 1* Not at all Once in a while Sometimes Fairly often Frequently, if not always I am effective in meeting others’ job￾related needs ( ) ( ) ( ) ( ) ( ) I use methods of leadership that are satisfying ( ) ( ) ( ) ( ) ( ) I get others to do more than they expected to do ( ) ( ) ( ) ( ) ( ) I am effective in representing others to higher authority ( ) ( ) ( ) ( ) ( ) I work with others in a satisfactory way ( ) ( ) ( ) ( ) ( ) I heighten others’ desire to succeed ( ) ( ) ( ) ( ) ( ) I am effective in meeting organizational requirements ( ) ( ) ( ) ( ) ( ) USAID Impact Evaluation of the TLP Scholarships Program– Final Report 80 I increase others’ willingness to try harder ( ) ( ) ( ) ( ) ( ) I lead a group that is effective ( ) ( ) ( ) ( ) ( ) Leadership Style - Section 2. Not at all Once in a while Sometimes Fairly often Frequently, if not always I provide others with assistance in exchange for their efforts ( ) ( ) ( ) ( ) ( ) I re-examine critical assumptions to question whether they are appropriate ( ) ( ) ( ) ( ) ( ) I fail to interfere until problems become serious ( ) ( ) ( ) ( ) ( ) I focus attention on irregularities, mistakes, exceptions, and deviations from standards ( ) ( ) ( ) ( ) ( ) I avoid getting involved when important issues arise ( ) ( ) ( ) ( ) ( ) I talk about my most important values and beliefs ( ) ( ) ( ) ( ) ( ) I am absent when needed ( ) ( ) ( ) ( ) ( ) I seek differing perspectives when solving problems ( ) ( ) ( ) ( ) ( ) I talk optimistically about the future ( ) ( ) ( ) ( ) ( ) Leadership Style - Section 3. Not at all Once in a while Sometimes Fairly often Frequently, if not always USAID Impact Evaluation of the TLP Scholarships Program– Final Report 81 I instill pride in others for being associated with me ( ) ( ) ( ) ( ) ( ) I discuss in specific terms who is responsible for achieving performance targets ( ) ( ) ( ) ( ) ( ) I wait for things to go wrong before taking action ( ) ( ) ( ) ( ) ( ) I talk enthusiastically about what needs to be accomplished ( ) ( ) ( ) ( ) ( ) I specify the importance of having a strong sense of purpose ( ) ( ) ( ) ( ) ( ) I spend time teaching and coaching ( ) ( ) ( ) ( ) ( ) I make clear what one can expect to receive when performance goals are achieved ( ) ( ) ( ) ( ) ( ) I show that I am a firm believer in “If it ain’t broke, don’t fix it.” ( ) ( ) ( ) ( ) ( ) I go beyond self-interest for the good of the group ( ) ( ) ( ) ( ) ( ) Leadership Style - Section 4. Not at all Once in a while Sometimes Fairly often Frequently, if not always I treat others as individuals rather than just as a member of a group ( ) ( ) ( ) ( ) ( ) I demonstrate that problems must become chronic before I take action ( ) ( ) ( ) ( ) ( ) I act in ways that build others’ respect for me ( ) ( ) ( ) ( ) ( ) I concentrate my full attention on dealing with mistakes, complaints, and failures ( ) ( ) ( ) ( ) ( ) USAID Impact Evaluation of the TLP Scholarships Program– Final Report 82 I consider the moral and ethical consequences of decisions ( ) ( ) ( ) ( ) ( ) I keep track of all mistakes ( ) ( ) ( ) ( ) ( ) I display a sense of power and confidence ( ) ( ) ( ) ( ) ( ) I articulate a compelling vision of the future ( ) ( ) ( ) ( ) ( ) I direct my attention toward failures to meet standards ( ) ( ) ( ) ( ) ( ) Leadership Style - Section 5. Not at all Once in a while Sometimes Fairly often Frequently, if not always I avoid making decisions ( ) ( ) ( ) ( ) ( ) I consider an individual as having different needs, abilities, and aspirations from others ( ) ( ) ( ) ( ) ( ) I get others to look at problems from many different angles ( ) ( ) ( ) ( ) ( ) I help others to develop their strengths ( ) ( ) ( ) ( ) ( ) I suggest new ways of looking at how to complete assignments ( ) ( ) ( ) ( ) ( ) I delay responding to urgent questions ( ) ( ) ( ) ( ) ( ) I emphasize the importance of ( ) ( ) ( ) ( ) ( ) USAID Impact Evaluation of the TLP Scholarships Program– Final Report 83 having a collective sense of mission I express satisfaction when others meet expectations ( ) ( ) ( ) ( ) ( ) I express confidence that goals will be achieved ( ) ( ) ( ) ( ) ( ) Confirmation page 51) As a token of our appreciation for taking the time to complete this survey, we would like to provide you with a phone charge in the value of 35 Euros. ( ) I do not want the incentive ( ) I want the incentive (please provide your phone number below) 52) Please provide us with your phone number where we should send the phone charge. Please make sure to confirm the phone number so we have the correct information. (Please allow for two weeks to receive the incentives from the time you complete the survey). Phone Number - Write In (Required): _________________________________________________ Re-enter phone number - - Write In (Required): _________________________________________________ 53) Congratulations on completing the survey. This is the last page for this survey. If you would like to review your response please do so now using the Back button below. If you are satisfied with your responses, press "Submit" * ( ) Submit Thank You! USAID Impact Evaluation of the TLP Scholarships Program– Final Report 84 Annex 5. Data Tables The tables below present the (i) The University where the scholarship recipients completed their Master’s programs; and (ii) Responses to MLQ Survey for both the scholarship recipients and the comparison group. MLQ data – scholarship recipients and comparison group Table 2a: Responses to MLQ Survey, Q1-9, scholarship recipients (row percent) N Not at all Once in a while Sometimes Fairly often Frequently , if not always Total I am effective in meeting others job￾related needs 145 3% 0% 14% 43% 40% 100% I use methods of leadership that are satisfying 153 8% 1% 12% 46% 33% 100% I get others to do more than they expected to do 153 7% 5% 24% 42% 22% 100% I am effective in representing others to higher authority 153 7% 7% 18% 38% 31% 100% I work with others in a satisfactory way 153 7% 1% 8% 42% 42% 100% I heighten others' desire to succeed 153 7% 1% 7% 38% 48% 100% I am effective in meeting organizational requirements 153 7% 0% 4% 41% 48% 100% I increase others' willingness to try harder 153 8% 2% 8% 37% 44% 100% I lead a group that is effective 153 8% 3% 14% 43% 33% 100% Table 2b: Responses to MLQ Survey, Q1-9, Comparison (row percent) N Not at all Once in a while Sometime s Fairly often Frequently, if not always Total I am effective in meeting others job-related needs 106 0% 23% 14% 60% 3% 100% I use methods of leadership that are satisfying 106 8% 7% 8% 50% 27% 100% I get others to do more than they expected to do 106 0% 0% 15% 47% 38% 100% I am effective in representing others to higher authority 106 0% 0% 12% 39% 49% 100% I work with others in a satisfactory way 106 0% 1% 11% 43% 44% 100% I heighten others' desire to succeed 106 0% 14% 11% 58% 16% 100% I am effective in meeting organizational requirements 106 0% 4% 12% 54% 30% 100% I increase others' willingness to try harder 106 0% 0% 8% 35% 57% 100% I lead a group that is effective 106 0% 0% 9% 31% 60% 100% USAID Impact Evaluation of the TLP Scholarships Program– Final Report 85 Table 3a: Responses to MLQ Survey, Q10-18, recipients (row percent) N Not at all Once in a while Sometime s Fairly often Frequently, if not always Total I provide others with assistance in exchange for their efforts 153 7% 1% 16% 44% 32% 100% I re-examine critical assumptions to question whether they are appropriate 153 7% 1% 19% 39% 35% 100% I fail to interfere until problems become serious 153 7% 26% 13% 40% 14% 100% I focus attention on irregularities, mistakes, exceptions, and deviations from standards 153 7% 25% 24% 37% 7% 100% I avoid getting involved when important issues arise 153 8% 19% 12% 55% 7% 100% I talk about my most important values and beliefs 153 7% 10% 18% 50% 15% 100% I am absent when needed 153 7% 9% 13% 60% 10% 100% I seek differing perspectives when solving problems 141 0% 2% 15% 48% 35% 100% I talk optimistically about the future 142 0% 1% 16% 44% 39% 100% Table 3b: Responses to MLQ Survey, Q10-18, comparison (row percent) N Not at all Once in a while Sometime s Fairly often Frequently, if not always Total I provide others with assistance in exchange for their efforts 106 30% 4% 10% 29% 26% 100 I re-examine critical assumptions to question whether they are appropriate 106 32% 3% 18% 33% 14% 100 I fail to interfere until problems become serious 106 65% 17% 8% 8% 2% 100 I focus attention on irregularities, mistakes, exceptions, and deviations from standards 76 11% 17% 29% 28% 16% 100 I avoid getting involved when important issues arise 76 71% 12% 5% 11% 1% 100 I talk about my most important values and beliefs 76 13% 16% 28% 32% 12% 100 I am absent when needed 76 79% 4% 9% 5% 3% 100 I seek differing perspectives when solving problems 76 3% 0% 12% 47% 38% 100 I talk optimistically about the future 76 1% 4% 21% 38% 36% 100 USAID Impact Evaluation of the TLP Scholarships Program– Final Report 86 Table 4a: Responses to MLQ Survey, Q19-27, recipients (row percent) N Not at all Once in a while Sometime s Fairly often Frequent ly, if not always Total I instill pride in others for being associated with me 140 0% 9% 17% 51% 24% 100% I discuss in specific terms who is responsible for achieving performance targets 138 0% 7% 30% 45% 19% 100% I wait for things to go wrong before taking action 138 0% 18% 12% 63% 7% 100% I talk enthusiastically about what needs to be accomplished 140 0% 6% 9% 54% 31% 100% I specify the importance of having a strong sense of purpose 139 0% 0% 10% 44% 46% 100% I spend time teaching and coaching 140 0% 5% 21% 39% 36% 100% I make clear what one can expect to receive when performance goals are achieved 153 8% 6% 24% 35% 27% 100% I show that I am a firm believer in "If it ain't broke, don't fix it." 153 10% 12% 20% 47% 11% 100% I go beyond self-interest for the good of the group 153 9% 8% 12% 41% 31% 100% Table 4b: Responses to MLQ Survey, Q19-27, comparison (row percent) N Not at all Once in a while Sometime s Fairly often Frequent ly if not always Total I instill pride in others for being associated with me 76 18% 12% 28% 32% 11% 18% I discuss in specific terms who is responsible for achieving performance targets 76 3% 11% 36% 39% 12% 3% I wait for things to go wrong before taking action 106 86% 5% 6% 3% 1% 86% I talk enthusiastically about what needs to be accomplished 106 31% 0% 8% 31% 29% 31% I specify the importance of having a strong sense of purpose 106 29% 2% 9% 25% 34% 29% I spend time teaching and coaching 106 29% 5% 18% 28% 20% 29% I make clear what one can expect to receive when performance goals are achieved 106 28% 2% 10% 33% 26% 28% I show that I am a firm believer in "If it ain't broke, don't fix it." 106 58% 14% 17% 7% 4% 58% I go beyond self-interest for the good of the group 106 31% 2% 7% 22% 39% 31% USAID Impact Evaluation of the TLP Scholarships Program– Final Report 87 Table 5a: Responses to MLQ Survey, Q28-35, recipients (row percent) N Not at all Once in a while Sometim es Fairly often Frequently , if not always Total I treat others as individuals rather than just as a member of a group 153 9% 3% 9% 36% 43% 100% I demonstrate that problems must become chronic before I take action 153 8% 8% 5% 59% 20% 100% I act in ways that build others respect for me 153 8% 2% 12% 50% 27% 100% I concentrate my full attention on dealing with mistakes, complaints, and failures 153 8% 18% 21% 40% 13% 100% I consider the moral and ethical consequences of decisions 153 8% 4% 13% 36% 39% 100% I keep track of all mistakes 153 8% 14% 25% 30% 22% 100% I display a sense of power and confidence 153 9% 8% 24% 36% 22% 100% I articulate a compelling vision of the future 153 8% 4% 19% 41% 28% 100% I direct my attention toward failures to meet standards 139 0% 18% 22% 42% 17% 100% Table 5b: Responses to MLQ Survey, Q28-35, comparison (row percent) N Not at all Once in a while Sometim es Fairly often Frequently , if not always Total I treat others as individuals rather than just as a member of a group 106 38% 2% 9% 25% 26% 38% I demonstrate that problems must become chronic before I take action 106 81% 7% 6% 5% 2% 81% I act in ways that build others respect for me 106 32% 5% 9% 30% 24% 32% I concentrate my full attention on dealing with mistakes, complaints, and failures 106 42% 13% 25% 14% 6% 42% I consider the moral and ethical consequences of decisions 106 28% 2% 10% 25% 34% 28% I keep track of all mistakes 76 9% 13% 26% 37% 14% 9% I display a sense of power and confidence 76 3% 7% 28% 41% 22% 3% I articulate a compelling vision of the future 106 28% 3% 14% 36% 19% 28% I direct my attention toward failures to meet standards 76 26% 13% 33% 18% 9% 26% USAID Impact Evaluation of the TLP Scholarships Program– Final Report 88 Table 6a: Responses to MLQ Survey, Q36-45, recipients (row percent) N Not at all Once in a while Sometimes Fairly often Frequently , if not always Total I avoid making decisions 141 0% 23% 14% 60% 3% 100% I consider an individual as having different needs, abilities, and aspirations from others 153 8% 7% 8% 50% 27% 100% I get others to look at problems from many different angles 141 0% 0% 15% 47% 38% 100% I help others to develop their strengths 141 0% 0% 12% 39% 49% 100% I suggest new ways of looking at how to complete assignments 140 0% 1% 11% 43% 44% 100% I delay responding to urgent questions 141 0% 14% 11% 58% 16% 100% I emphasize the importance of a collective sense of mission 141 0% 4% 12% 54% 30% 100% I express satisfaction when others meet expectations 141 0% 0% 8% 35% 57% 100% I express confidence that goals will be achieved 140 0% 0% 9% 31% 60% 100% Table 6b: Responses to MLQ Survey, Q36-45, comparison (row percent) N Not at all Once in a while Sometim es Fairly often Frequently , if not always Total I avoid making decisions 76 58% 22% 14% 5% 0% 58% I consider an individual as having different needs, abilities, and aspirations from others 76 3% 11% 13% 42% 32% 3% I get others to look at problems from many different angles 76 0% 1% 14% 45% 39% 0% I help others to develop their strengths 76 0% 0% 8% 46% 46% 0% I suggest new ways of looking at how to complete assignments 76 0% 0% 13% 41% 46% 0% I delay responding to urgent questions 76 76% 9% 7% 5% 3% 76% I emphasize the importance of a collective sense of mission 76 0% 1% 18% 39% 41% 0% I express satisfaction when others meet expectations 74 0% 0% 11% 28% 61% 0% I express confidence that goals will be achieved 75 0% 0% 9% 39% 52% 0% USAID Impact Evaluation of the TLP Scholarships Program– Final Report 89 Annex 6. The Components of Transformational, Transactional and Passive-Avoidant Leadership The self-assessment survey used to measure leadership potential of the scholarship recipients and track changes therein is based on the Multifactor Leadership Questionnaire (MLQ survey), developed by Bruce J. Avolio and Bernard M. Bass and published by Mind Garden Inc. (www.mindgarden.com). IBI holds a remote license to reproduce and administer the survey for the purpose of this evaluation. The same survey instrument was used for the control group as well. MLQ defines leadership features as being “transformational”, “transactional” or “passive avoidant” (see Box 1).28 The MLQ survey identifies these distinctive styles by posing questions that relate to the specific characteristics of these leadership styles.29 Transformational leaders reflect five different dimensions, each of which is identified by specific measurable characteristics. These dimensions and associated behaviors are summarized in Table 1. Table 1: Transformational Leadership: Dimensions and Associated Behaviors Dimension Associated Behaviors Attributes of Idealized Influence -Instill pride in others for being associated with self -Go beyond self-interest for the good of the group -Act in ways that build others’ respect for self -Display a sense of power and confidence Behaviors of Idealized Influence -Talk about most important values and beliefs -Specify importance of having a strong sense of purpose -Consider the moral and ethical consequences of decisions -Emphasize the importance of having a collective sense of mission Inspirational Motivation -Talk optimistically about the future -Talk enthusiastically about what needs to be accomplished -Articulate a compelling vision of the future -Express confidence that goals will be achieved Intellectual Stimulation -Re-examine critical assumptions to question whether they are appropriate -Seek differing perspectives when solving problems -Get others to look at problems from many different angles -Suggest new ways of looking at how to complete assignments Individualized Consideration -Spend time teaching and coaching -Treat others as individuals rather than just as a member of the group -Consider each individual as having different needs, abilities and aspirations -Help others to develop their strengths Likewise, transactional leaders reflect two dimensions, each of which is identified by specific measurable characteristics. These dimensions and associated behaviors are summarized in Table 2. 28Source: “The Manual for Multifactor Leadership Questionnaire (MLQ survey)”, developed by Bruce J. Avolio and Bernard M. Bass and published by Mind Garden Inc. (www.mindgarden.com). 29Also see AvoHo, Bruce, Bernard M. Bass, and Dong I. Jung (1999): “Re-examining the components of transformational and transactional leadership using the Multifactor Leadership Questionnaire”, Journal of Occupational and Organizational Psychology (1999), 72, 441-462 Great Britain USAID Impact Evaluation of the TLP Scholarships Program– Final Report 90 Table 2: Transactional Leadership: Dimensions and Associated Behaviors Dimension Associated Behaviors Contingency Rewards -Provide others with assistance in exchange for their efforts -Discuss in specific terms who is responsible for achieving performance targets -Make clear what one can expect to receive when performance goals are achieved -Express satisfaction when others meet expectations Active Management by Exception -Focus attention on irregularities, mistakes, exceptions, and deviations from standards -Concentrate my full attention on dealing with mistakes, complaints and failures -Keep track of all mistakes -Direct my attention toward failures to meet standards. Passive avoidant leaders also reflect two dimensions, each of which is identified by specific measurable characteristics. These dimensions and associated behaviors are summarized in Table 3. Table 3: Passive-Avoidant Leadership: Dimensions and Associated Behaviors Dimension Associated Behaviors Passive Management by Exception -Fail to interfere until problems become serious -Wait for things to go wrong before taking action -Show a firm belief in "if it ain’t broke, don’t fix it" -Demonstrate that problems must become chronic before taking action Laissez faire -Avoid getting involved when important issues arise -Absent when needed -Avoid making decisions -Delay responding to urgent questions In addition to the leadership styles and dimensions discussed above, MLQ also identifies “leadership outcomes”, and associated behaviors. These outcomes and associated behaviors, which can be used to rate leadership of others and s of one’s own self, are placed in Table 4. Table 4: Leadership Outcomes and Associated Behaviors Dimension Associated Behaviors Extra Effort -Get others to do more than they expected to do -Heighten others’ desire to succeed -Increase others’ willingness to try harder Effectiveness -Am effective in meeting others’ job-related needs -Am effective in representing their group to higher authority -Am effective in meeting organizational requirements -Lead a group that is effective Satisfactory Leadership -Use methods of leadership that are satisfying -Work with others in a satisfactory way The different leadership styles and dimensions can be captured by posing specific questions to capture associated behaviors. Thus, the 45 behaviors listed above are captured by 45 questions in the MLQ Survey, each of which is answered on a 5-point Likert scale ranging from “not at all” to “frequently, if not always.”30 Table 5 contains descriptive statistics derived from testing of this survey with large samples of leaders in US and Europe countries. These descriptive statistics provide a reference benchmark against which to compare results. 30 For leadership style dimensions and associated questions, please contact Mind Garden: http://www.mindgarden.com/ USAID Impact Evaluation of the TLP Scholarships Program– Final Report 91 Table 5: Descriptive Statistics for MLQ 5X 2004 Normative Sample US- Total sample (N 27,285) US – Self Evaluations (N 3,375) EU- Total sample (N 8,025) EU – Self Evaluations (N 1,143) MLQ5X US Mean TS US SDTS US Mean SE US SDSE EU Mean TS EU SDTS EU Mean SE EU SDSE Idealized Influence Attributed 2.94 0.76 2.95 0.53 2.73 0.7 2.83 0.55 Idealized Influence Behaviors 2.77 0.72 2.99 0.59 2.72 0.65 3 0.55 Inspirational Motivation 2.92 0.76 3.04 0.59 2.77 0.71 3 0.6 Intellectual Stimulation 2.78 0.71 2.96 0.52 2.81 0.64 3.02 0.48 Individualized Consideration 2.85 0.78 3.16 0.52 2.74 0.7 3.1 0.5 Contingent Reward 2.87 0.7 2.99 0.53 2.83 0.63 3.02 0.52 Management by Exception: Active 1.67 0.88 1.58 0.79 2.3 0.78 2.2 0.79 Management by Exception: Passive 1.03 0.75 1.07 0.62 1.1 0.71 0.96 0.6 Laissez Faire 0.65 0.67 0.61 0.52 0.8 0.68 0.62 0.51 Extra Effort 2.74 0.86 2.79 0.61 2.69 0.76 2.85 0.5 Effectiveness 3.07 0.72 3.14 0.51 2.97 0.65 3.06 0.57 Satisfaction 3.08 0.83 3.09 0.55 2.92 0.76 2.96 0.5 Source:www.mindgarden.com/docs/MLQinternationalnorms.pdf For more information contact www.ibi-usa.com USAID/Kosovo Transformational Leadership Impact and Performance Evaluation IBI 4040 North Fairfax Drive Suite 650 Arlington, Virginia 22203 Tel: 1-703-525-2277 www.ibi-usa.com