USAID/KOSOVO TRANSFORMATIONAL LEADERSHIP IMPACT AND PERFORMANCE EVALUATION PROJECT TLP UNIVERSITY PARTNERSHIPS PROGRAM PERFORMANCE EVALUATION FINAL REPORT February 2020 This document is made possible by the support of the American People through the United States Agency for International Development (USAID). The contents of this report were prepared by IBI under contract number AID￾167-TO-15-00002. The views expressed herein are the sole responsibility of IBI and do not necessarily reflect the views of USAID or the United States Government. USAID TLP-SP Partnership Program – Performance Evaluation Final Report 2 USAID/KOSOVO TRANSFORMATIONAL LEADERSHIP IMPACT AND PERFORMANCE EVALUATION PROJECT TLP UNIVERSITY PARTNERSHIPS PROGRAM PERFORMANCE EVALUATION FINAL REPORT January 2020 USAID TLP Partnership Program Performance Evaluation – Final Report 4 Table of Contents Table of Contents Acronyms ......................................................................................................................................................6 Executive Summary.......................................................................................................................................7 1 Introduction ..........................................................................................................................................9 1.1 Kosovo Country Context.............................................................................................................9 1.2 Description of the TLP Program and the TLP-SP Project..........................................................10 1.2.1 The TLP Program....................................................................................................................10 1.2.2 The TLP-SP Project.................................................................................................................11 1.2.3 The TLP University Partnerships Component .......................................................................11 1.3 Organization of the report .........................................................................................................14 2 Study Objectives and Approach..........................................................................................................15 2.1 Evaluation Objective ...................................................................................................................15 2.2 Evaluation Questions ..................................................................................................................15 2.3 Study Approach...........................................................................................................................16 2.4 Limitations of the Study ..............................................................................................................17 3 Evaluation Question 1: Implementation of The Action Plan, Change Management and Impact on Performance ...............................................................................................................................................18 3.1 Implementation of UP’s Action Plan...........................................................................................18 3.1.1 Improving internal and external communications ..................................................................19 3.1.2 Strengthening financial viability ...............................................................................................19 3.1.3 Improving the quality of research and teaching......................................................................21 3.2 University’s commitment to change management......................................................................22 3.3 Change in UP’s administrative performance...............................................................................23 3.3.1 Standardization of administrative processes...........................................................................23 3.3.2 Fiscal accountability and financial management.......................................................................24 3.3.3 Improved data systems ...........................................................................................................25 3.3.4 Improved student services......................................................................................................26 4 Evaluation Question 2: Effect on Academic Performance and Implementation of “Grand Challenges” 27 4.1 Improving academic performance in targeted faculties..............................................................27 4.1.1 Improvement in quality of Teaching .......................................................................................27 4.1.2 Improvement in quality of Research.......................................................................................32 4.1.3 Centers to support improvement in teaching and research (University-wide Activities).....33 4.2 Progress towards meeting the Grand Challenges......................................................................35 USAID TLP Partnership Program Performance Evaluation – Final Report 5 5 Evaluation Question 3: Utilization and sustainability of new technology, systems, knowledge and skills38 5.1 Utilization of new technology, systems, knowledge and skills ...................................................38 5.2 Sustainability................................................................................................................................40 6 Conclusions and Recommendations...................................................................................................43 Recommendations ..................................................................................................................................47 7 Annexes...............................................................................................................................................50 Annex1: Scope of Work for the Evaluation ...........................................................................................50 Annex 2: Evaluation Methodology ..........................................................................................................59 Annex 3: Informational Meetings and Interviews with Key Stakeholders..............................................65 Annex 4: Sources of Data and List of Documents Reviewed ................................................................67 Annex 5: Description of University Partnerships ..................................................................................69 Annex 6: Tracking TLP Partnerships ......................................................................................................77 USAID TLP Partnership Program Performance Evaluation – Final Report 6 Acronyms AC Advisory Committee ASU Arizona State University BIRN Balkan Investigative Research Network CEO Chief Executive Officer CES Center for Renewable Energy and Sustainability CTE Center for Teaching Excellence COP Chief of Party DC Dartmouth College DCOP Deputy Chief of Party EU European Union FA Faculty of Agriculture & Veterinary GOK Government of Kosovo HE Higher Education HICD Human and Institutional Capacity Development IBI International Business Initiatives Corp. (doing business as IBI) ICT Information and Communications Technology IU Indiana University KHERT Kosovo Higher Education Roundtable M&E Monitoring and Evaluation MBC Morgan Borszcz Consulting MEST Ministry of Education, Science and Technology NGO Non-Government Organization ORSP Office for Research and Sponsored Projects TLP Transformational Leadership Project TLP-SP Transformational Leadership Program – Scholarships and Partnerships Project USAID United States Agency for International Development UMN University of Minnesota UP University of Prishtina U.S. United States WL World Learning USAID TLP Partnership Program Performance Evaluation – Final Report 7 Executive Summary United States Agency for International Development (USAID) has contracted with IBI to conduct an evaluation of the Transformational Leadership Program – Scholarships and University Partnerships (TLP￾SP) components of the Transformational Leadership Project (TLP) program. The evaluation was comprised of an impact evaluation of the TLP Scholarships component (henceforth Scholarships program) and a performance evaluation of the TLP Partnerships component (henceforth Partnerships program). The Scholarships and Partnerships programs were implemented under the TLP-SP project by World Learning Inc. (WL). This report covers the activities and progress of the University Partnerships component of TLP-SP and its effect on the University of Prishtina and four of its faculties at the end of the project. The Scholarships component will be addressed in a separate report. The University Partnerships component of the TLP_SP project was designed to improve the quality of education and management of University of Prishtina (UP). The performance evaluation of this component is designed to assess progress under the Partnerships component, draw conclusions on how the selected University faculties have changed as a result of TLP assistance; and examine the impact of this change on academic outcomes and change management. The baseline study was carried out in 2015 and the midline study was completed in January 2018. The endline study, upon which this report is based, was conducted at the end of the project, data was collected over a two-month period from May to June 2019. This report includes the findings from the endline study and the changes noted over time – from baseline to endline. Project Activities The TLP Partnerships program has provided strategic advice and capacity building to UP management in financial reform, communications, and leadership. It has worked with the University and an Advisory Committee (consisting of members from the government, public universities, civil society groups, private sector, youth and disadvantaged populations, the TLP-SP project and USAID) on a series of university￾wide activities to address broad challenges and institutional reforms, including student services, teaching and assessment, university-sponsored research, online courses and support for incubation centers. It has also facilitated partnerships between selected UP faculties and United States universities that specialize in those fields. These partnerships have included training for teaching staff with activities in curricular development, teaching methodology, and research enhancement. Results Although there were some initial delays, the TLP-SP project made good progress and had a positive effect on many aspects of the University. WL and its partners successfully adhered to the timeline and maintained focus on the essential steps in the sequenced approach to implementation and achievement of the development objectives of the project. The project overcame many challenges in its short life and implementation progressed well on all fronts. As a result of the partnerships, many administrative changes were put in place, including the development of a new strategy and accompanying rule books, for more efficient management of the University and to ensure its stability. Initiatives were also launched to collect better and more reliable data to support the university management with more efficient management and data-based policy making, improve communication and the image of the university; as well as improve the quality of teaching through better staff training and new teaching resources and research. New centers were opened and/or revived including one that has introduced the concept of multi-disciplinary programs in Kosovo. The program has resulted in access of the select faculties to USAID TLP Partnership Program Performance Evaluation – Final Report 8 world class training for the teaching staff, availability of text books, resource books, new syllabi and course materials. However, in the faculties there was more individual learning than faculty-wide learning. Change across the University is slow and there is some evidence of impact that the university is moving towards better quality of education to meet the demands of the labor market. The TLP-SP University Partnerships program was rated by a member of the senior management team as having “been more effective than all other donor programs before them”, a sentiment echoed by the University Rector as well as several Deans and Vice-Deans. TLP-SP has embedded itself very successfully within UP to the point where the senior management was asking for an extension to the program. Even though a sustainability plan was developed, there continue to be questions about sustainability of the various initiatives after the end of the project. Recommendations Despite overall progress, there are a few issues that need to be addressed as soon as possible to ensure continuity of the ongoing initiatives and future programming support from donor agencies. This report makes some specific recommendations to the University of Prishtina and the donor agencies. Specific recommendations can be found on page 47 of this report. USAID TLP Partnership Program Performance Evaluation – Final Report 9 1 Introduction1 USAID/Kosovo has contracted with IBI International (IBI) to conduct a concurrent evaluation of two components of the Transformational Leadership Program (TLP) – an impact evaluation of the U.S. Master’s degree scholarships component and a performance evaluation of the University Partnerships component (Annex 1). The TLP program is a multifaceted program designed to help Kosovo address the demands of a modern market economy and bring transformational change to the country. A brief description of the country context, the TLP program and the components being evaluated is below. 1.1 Kosovo Country Context Kosovo declared independence from Serbia on February 17, 2008 after years of strained relations between its Albanian and Serb populations. It is recognized as an independent country by 114 out of 193 United Nations (UN) members and by 23 out of 28 EU members.2 Kosovo is a potential candidate for EU membership and in recent years, it has accelerated its EU integration process. Kosovo’s population of about 1.8 million is the youngest in Europe, with an average age of 26 years. 3 A lower-middle-income country that has experienced solid economic growth over the last decade, Kosovo is one of only four countries in Europe to experience growth in every year since the onset of the global financial crisis in 2008. Despite this, Kosovo remains one of the poorest countries in Europe, and its average per capita income of approximately U.S. $3,877 (October 11, 2018) is about one-tenth that of EU levels.4 More than 80 percent of the population lives under the World Bank standardized poverty rate, defined by a threshold of U.S. $5 per day at purchasing power parity, and about 30 percent of the population lives below the domestic poverty line, defined by the Kosovo Agency of Statistics as Euro 1.72 per day (2011 prices). Kosovo has a relatively low Gini index and flat consumption distribution, with poverty rates declining gradually. No significant differences exist between urban and rural poverty, but regional differences are notable. Extreme poverty is disproportionately high among children, the elderly, households with disabled members, female-headed households, and certain ethnic minority households (especially Roma, Ashkali, and Egyptian communities). As in many other countries, there is a strong negative correlation between education and poverty in Kosovo. 5 1 This report is a final report on the performance evaluation of the University Partnerships component of the TLP-SP project. It combines the findings and conclusions from the baseline and midline report. The contents of Section I: Introduction, and Section II: Objectives and Approach, are very similar to the baseline report for the performance evaluation of the University Partnerships Component of the TLP-SP project that was prepared in December 2015. In order to publish and make available the current report as a stand-alone document, these sections are repeated and updated as necessary. In the compilation of the full report in a single volume, these sections will be appropriately edited to avoid duplication. 2 The World Bank group (2018). Country Context. https://www.worldbank.org/en/country/kosovo/overview. There have been some reports recently of some countries withdrawing support. See, for instance, B92, News article, TANJUG, December 3, 2018 found in Wikipedia at https://en.wikipedia.org/wiki/Kosovo Accessed on Sep 1, 2019. 3 The World Bank Group (2017): FY17–-21 Country Partnership Framework for Republic of Kosovo. Report No. 112337-XK. http://documents.worldbank.org/curated/en/297951496160148830/Kosovo-Country-partnership-framework-for-the-period￾FY17-FY21 accessed on August 25, 2019. 4 See The World Bank Group (2017): Kosovo – Country Snapshot. Accessed on January 5, 2018 at http://pubdocs.worldbank.org/en/912841507748392027/Kosovo-Snapshot-Fall2017.pdf 5 See The World Bank Group (2016): Kosovo – Country Program Snapshot. Accessed on February 27, 2017 at http://pubdocs.worldbank.org/en/419461462386476530/World-Bank-Kosovo-Program-Snapshot-April-2016.pdf USAID TLP Partnership Program Performance Evaluation – Final Report 10 Although Kosovo’s economic growth has outperformed its neighbors and been largely inclusive, it has not been sufficient to significantly reduce the high rates of unemployment; provide formal jobs, particularly for women and youth; or reverse the trend of large-scale outmigration. In the past two decades, since its declaration of independence, Kosovo’s economy has changed substantially. The current growth model relies heavily on remittances to fuel domestic consumption but has recently shifted toward more investment- and export-driven growth.6 Education is an essential part of Kosovo’s efforts to boost economic growth, increase productivity and wages, and reduce high unemployment. The Kosovo Education Strategic Plans of 2011-16 and of 2017-2021, developed by the Ministry of Education, Science, and Technology (MEST) tackle a host of issues related to inclusion, access, quality of education, accountability and improved management, as well as implementation of standards at all levels of education to ensure that its citizens possess the knowledge and skills that make them employable, both within the country and abroad. 7 1.2 Description of the TLP Program and the TLP-SP Project 1.2.1 The TLP Program As a step towards enhancing the human capital in Kosovo, building a cadre of skilled professionals capable of addressing Kosovo’s economic and democratic development priorities and addressing some of the gaps in the education system, USAID and the Government of Kosovo (GOK) have jointly funded the multi-pronged Transformational Leadership Program (TLP). The TLP program seeks to upgrade the knowledge and skills of talented professionals through short- and long-term educational opportunities and increase the capacity of select faculties at institutions of higher learning in Kosovo through a comprehensive, multi-faceted approach comprised of five separate but interrelated activities (components).8 These components, executed by a mix of implementing partners, are: 1. U.S. Scholarships (Master’s degree and professional certificates) – competitively awarded and open to all citizens of Kosovo. 2. Immersion Program to strengthen the skills of potential graduate students (especially those from marginalized groups). 3. Citizens Corps – a program to help and ensure that returned Master’s graduates fulfill their two-year “give back” commitment through either by working in the public sector and/or through approved community service. 4. Short-term public service training in Kosovo to upgrade skills of current public service staff in key Kosovar government institutions 5. University Partnerships – partnerships between U.S. and Kosovar universities (most notably the University of Prishtina) to upgrade university management and pedagogy in selected faculties. 6 The World Bank Group (2018). The World Bank in Kosovo – Overview. https://www.worldbank.org/en/country/kosovo/ overview accessed on August 25, 2019 7 Kosovo Ministry of Education, Science and Technology (2016). Kosovo Education Strategic Plan 2017-2021 accessed at http://www.kryeministriks.net/repository/docs/Kosovo_Education_Strategic_Plan. Pdf on August 25, 2019 8 Description from Contract document: AID-167-TO-15-00002– TLP Impact and Program Evaluation USAID TLP Partnership Program Performance Evaluation – Final Report 11 1.2.2 The TLP-SP Project Two of the five TLP components – the Scholarships component and the University Partnerships component, collectively referred to as TLP Scholarship and University Partnerships (henceforth TLP-SP project) – were implemented under a cooperative agreement between USAID and World Learning Inc. (WL) and its partners, Arizona State University (ASU), University of Minnesota (UMN), Dartmouth College (DC) and Indiana University (IU).9 Funded by USAID and GOK, in particular by Ministry of Education, Science, and Technology (MEST), the TLP-SP project was implemented over a period of five years from February 26, 2014 to June 30, 2019, and had the following three objectives: ● Objective 1: Coordinated and collaborative oversight of the transformational leadership initiative through an Advisory Committee (AC) ● Objective 2: U.S. post-graduate degrees (Master’s degrees) and professional certifications for selected citizens of Kosovo (Scholarships component) ● Objective 3. Improved university management and pedagogy in selected faculties of the University of Prishtina (UP) and/or other universities through partnerships and exchanges with U.S. Universities (University Partnerships component) Established in 2014 in consultation with USAID and the Prime Minister’s office, the AC was designed to provide “coordinated, collaborative oversight of the Transformational Leadership Initiative” (Objective 1.2.1 of the TLP-SP project). The AC, which included representatives from GOK, public universities, civil society groups, private sector, and youth and disadvantaged populations, was expected to guide/steer the various components of the TL initiative, help ensure coordination across initiatives, as well as keep the GOK engaged and informed. Upon establishment, the AC developed its own mission statement, by￾laws, work plan and action plan to support its strategy. Despite this, its purpose and role are still evolving, and the AC has yet to establish a niche for itself. Over the life of the project, the AC was restructured several times to strengthen its effectiveness and to lay the groundwork for future sustainability. In its final incarnation at the end of the project, the AC was established as the Kosovo Higher Education Roundtable (KHERT) and was housed in the office of the Minister of Education to advise the MEST on higher education reform. However, it seems unlikely that the AC would be sustainable after the closure of the TLP-SP project. The Scholarships and the Partnerships components are the subject of this evaluation (though the findings from the evaluation studies for each component will be reported separately). This report focuses on the TLP University Partnerships component, briefly described below. 1.2.3 The TLP University Partnerships Component The TLP University Partnerships component (henceforth called the Partnerships component) was designed to support partnerships between universities in the U.S. and Kosovo to upgrade select faculties and position them so that they could provide training, knowledge and skills that met the demands of modern day and competitive markets. Since the University of Prishtina was the largest public institution 9 To help the TLP-SP project better respond to the realities on the ground, some of its planned activities were modified during implementation to: (i) limit the activities to focus primarily on UP; and (ii) revisit the planned support for the implementation of select priorities from the UP Action Plan of 2013 to make it more relevant (see Annex 6 for details). USAID TLP Partnership Program Performance Evaluation – Final Report 12 of higher education in Kosovo and fed all other institutions in the country, TLP-SP, in agreement with USAID, focused the Partnerships program only on UP. Planned support under this component initially covered the two-year period from February 15, 2015 to February 14, 2017 (Phase 1) but was later extended by a little over two years to June 2019 (Phase 2). In order to support and enhance capacity of management and academic staff within select UP faculties to meet the challenges of 21st century student-centered teaching and applied research, four selected faculties of UP – (i) Agriculture and Veterinary, (ii) Economy, (iii) Education, and (iv) Computer and Electrical Engineering – were each paired with a U.S. university selected on the basis of its resources and capacity to provide support to the faculty. The U.S. partner universities received sub-grants from TLP-SP project to implement the designated activities, which they supplemented with own resources. The initial sub-grants were for a two-year period (Phase 1 – 2015-17), and were then extended in three cases for another two years (Phase 2, through end of project). Table 1 lists the partnerships: Table 1: University Partnerships University of Prishtina Faculty U.S. Partner University* Current Status** Agriculture and Veterinary Includes Department of Veterinary Medicine, Department of Food Technology and Department of Plant Protection University of Minnesota Phases 1 (2015-17) and 2 (2017-end of project) completed Economy (Includes Faculty of Economy and academic staff from Faculty of Philosophy associated with the UPs Center for Gender Studies and Research Dartmouth College Phase 1 completed with six month no-cost extension Phase 2 Not extended Education (Includes all departments in the faculty) Indiana University Phases 1 and 2 completed Computer and Electrical Engineering (Services extended to include Department of Physics and other natural science departments at UP Arizona State University (includes support for the Faculty and for several university-wide Centers) Phases 1 and 2 completed *Note: The U.S. Partners are comprised of teams from various schools and departments within the University. See Annex 5 for details. ** Phase 1 and Phase 2 – from Feb 28, 2015 to November 30, 2018. This was further extended to February 2019 as the TLP-SP project itself was extended to June 2019. All departments within the faculties of Electrical and Computer Engineering (including the Department of Physics and other natural science departments), Education, and Economy (including academic staff associated with the Center for Gender Studies and Research especially from the Department of Philosophy) were included in the Partnership. The Faculty of Agriculture & Veterinary included three departments –Veterinary Medicine, Department of Food Technology with Biotechnology and Department of Plant Protection, enabling a larger pool of participants to benefit from the exchange and mentoring activities of the University Partnerships component. A brief description of these partnerships USAID TLP Partnership Program Performance Evaluation – Final Report 13 as planned at the time of inception of the project, extracted from the Baseline report (2016), 10 is included in Annex 5 for easy reference. Activities under the Partnerships component can be grouped into three categories: 1) Those providing support to the Rector and the management team in development and implementation of the University strategy and Action Plan; 2) Activities designed to strengthen the academic capacity and research development in four faculties selected in collaboration with the Rector and the management team: (i) Agriculture and Veterinary; (ii) Economy; (iii) Education; and (iv) Computer and Electrical Engineering. Each of these faculties was partnered with a U.S.-based university, namely University of Minnesota (UMN), Dartmouth College (DC), Indiana University (IU) and Arizona State University (ASU) respectively; and 3) Cross-cutting activities that were expected to benefit more than just the four faculties included in the partnerships and hence were based at the university level. Review of project records and reports and interviews with TLP-SP project staff, USAID, and administrators from UP suggest that the project has, with a few changes, completed most of the activities planned (as per the annual workplans) for the Partnerships component.11 While the initial progress was slow, three of the partnerships made good progress towards their targets in Phase 1 and were extended through the end of the project (Phase 2). The Partnership for the Faculty of Economy was not extended into Phase 2 as the Dean of Economy and the University partner could not agree on a meaningful program that would be useful for the faculty. It is also important to note that while the partnership between ECEN and ASU continued into Phase 2, the faculty received very little support in this term as most of the resources were diverted by ASU/TLP-SP to supporting the Center for Energy and Sustainability (established at the Rectorate level). A summary of planned and completed activities is placed in Annex 6. Notwithstanding the turnover in TLP-SP project leadership and multiple administration-level changes at UP during the first two years of the project, most activities targeted towards the four UP faculties participating in the Partnerships program continued in accordance with the work plan. Even though several cross-cutting, university-wide activities as well as activities supporting the implementation of the priorities from the UP Action Plan were delayed for a period of time because of the changes in the Rectorate and UP administration,12 the stability provided by the UP management ensured that most were successfully completed in the last two years of the project or were completed to a level where UP 10 IBI International. Performance Evaluation of the University Partnerships Program: Baseline Study. January 2016 11 Since implementation of the TLP-SP University Partnerships component was started in 2015, which is the second year of the TLP-SP project as planned, year one of the University Partnerships component is actually the second year of the TLP-SP project overall. 12 There have been four rounds of changes in the Rectorate since the start of the TLP-SP project in 2014. The most recent was in Spring 2016. Each change in the Rectorate meant a change in the full management team at UP, and also a change in priorities for the University. As expected, each of these changes has had an impact on the performance of the project and, based on the priorities of the new management team, also resulted in changes in the planned activities themselves. The current Rector has been able to create a more stable environment at the University and consequently a more conducive environment for implementation of TLP-SP activities. USAID TLP Partnership Program Performance Evaluation – Final Report 14 could take over and bring them to a satisfactory conclusion. Overall, everyone interviewed by the evaluation team was unanimous in strongly appreciating the work done by the TLP-SP team. 1.3 Organization of the report The rest of the report is organized as follows. Section 2 presents the objectives of the evaluation and outlines the approach used to conduct the study. Sections 3 to 5 discuss the three evaluation questions that guide the overall evaluation of the Partnerships component. Section 6 presents the conclusions. The report has six annexes, which contain: (i) scope of work for this evaluation; (ii) evaluation methodology; (iii) list of people interviewed and participants in focus groups for the endline study; (iv) sources of data and a list of documents reviewed; (v) description of University Partnerships; and (vi) tables noting progress against the activities described for the TLP-SP project and elaborated in its annual work plans. USAID TLP Partnership Program Performance Evaluation – Final Report 15 2 Study Objectives and Approach The performance evaluation of the Partnerships component consists of a series of studies that were conducted concurrently with the 2015-2019 implementation of the component. The studies were designed to assess progress under this component against the planned support, document any changes in the planned activities within the program description, highlight any challenges to implementation that may have affected the outcomes of this component, and provide recommendations to support and strengthen the implementation of the program. 13 Data on performance of the component were collected annually, and studies were completed at the baseline (2015), midline (2017), and the end of the project (2019). This report presents the overall findings of the performance evaluation (including the endline study) of the Partnerships component. 2.1 Evaluation Objective The Partnerships component was based on the premise that to reduce unemployment rates in Kosovo, particularly among the youth, there was an urgent need to improve the quality of education to provide adequate learning opportunities for students as well as skills required by the labor markets. To do so, the premise maintained, management systems in institutions of higher education must be improved, teachers must be provided with training to improve the quality of teaching and research, and resources available to the students must be increased. The theoretical construct underlying the Partnerships component was that, because of their participation in the TLP-SP program, UP and select faculties in UP and other institutions of higher education in Kosovo would have improved university management and updated technology systems, enhanced quality of teaching methodology and curricula, and strengthened research development. These improvements, in turn, would enhance the capacity of these institutions to offer longer-term solutions to Kosovo’s education needs. The performance evaluation of the Partnerships component was designed to assess the extent to which the targeted support provided by TLP-SP to UP and select faculties at UP and other universities in Kosovo has resulted in improved university management, quality of teaching methodology and curricula, and strengthened research development. The results of this performance evaluation will inform decision-making by USAID, Government of Kosovo as well as the University of Prishtina and other partnering universities and faculties to sustain the investments and use the model to make upgrades and investments in other areas and faculties. 2.2 Evaluation Questions The performance evaluation of the University Partnerships component has been guided by the following three sets of questions posed by USAID: 14 1) Has the UP’s Action Plan been effectively implemented? Have tangible measures been adopted that show the University’s commitment to change management? How has the TLP Partnerships component changed the UP’s administrative performance? 13 Note that since this project is being implemented under a cooperative agreement, the scope of work is defined by the program description and activities planned for each year within the program description. 14 Contract document: AID-167-TO-15-00002– TLP Impact and Performance Evaluation USAID TLP Partnership Program Performance Evaluation – Final Report 16 2) How has the TLP Partnerships component changed academic performance in targeted faculties over the course of the project? Have the “Grand Challenges” identified for each partnership been effectively implemented? 3) Have the selected faculties been able to utilize, maintain and build up new technology, systems, knowledge and skills? What actions have been taken and/or mechanisms put in place to help ensure the sustainability of the partnership efforts? 2.3 Study Approach Conducted concurrently with the implementation of the TLP-SP Partnerships program, this study was designed to respond to each of the broad questions above and gather evidence on the extent to which participation in the TLP University Partnerships program had helped improve the management and quality of teaching at the participating faculties at the University of Prishtina. This is done by following the progress of the activities annually, using a variety of qualitative and quantitative methods. The study design is briefly summarized below (Annex 2 contains a more detailed description of the methodology). The assessment methodology employed follows the logic of the activity and examines all initiatives within this component to determine progress. To respond to the research questions listed above, data were collected on several qualitative and quantitative indicators and sub-indicators (see Annex 2) based on criteria drawn from the UP Strategy and Action plan (2013 and 2016), which together informed the results of the partnership activities as well as future possible modifications in the activities. To assess the changes and the contribution of the TLP University Partnerships program, the evaluation used a mixed￾methods approach, and a variety of data were drawn from different sources and systematically integrated.15 Studies were conducted periodically over the evaluation period to collect qualitative and quantitative data to assess progress on milestones and against the baseline. This study, the endline study, is the last in a series of evaluation studies. The first of these studies, the baseline study, was conducted in 2015 and data were collected annually thereafter to reduce the effects of time on recall. The midline study was conducted at the end of year 2 of the Partnerships program (2017). Interim data were also collected in 2016 and 2018, which informed the midline and endline studies respectively. Data for the endline study were collected from May to July 2019, concurrently with the end of the TLP-SP project. The baseline and midline studies established the status at the beginning of the project and progress at a point two years later respectively. These studies, together with the annual interim data collection, also served to identify major issues and problems that posed a present or imminent risk to this component and thus needed to be addressed at that stage. This endline study followed the methodology used for the baseline and midline studies. Like the interim data collection efforts, data for the endline performance evaluation were collected from a variety of primary and secondary qualitative and quantitative sources. Primary data were collected through surveys, key informant interviews, and focus groups conducted with stakeholders and beneficiaries of the program, including management of UP, management and academic staff of the participating faculties, and academic and non-academic staff from UP that participated in TLP activities, such as the exchange 15 See USAID (2013): Conducting Mixed-Method Evaluations. Accessed on November 29, 2015 at https://www.usaid.gov/sites/default/files/documents/1870/Mixed_Methods_Evaluations_Technical_Note.pdf USAID TLP Partnership Program Performance Evaluation – Final Report 17 programs, training for leadership and change management, and workshops organized by the Center for Teaching Excellence. In collaboration with WL, the implementers of TLP-SP project, secondary data were collected from a review of the quarterly and annual reports of the project, work plans, studies conducted to inform implementation of the project, databases maintained by TLP-SP and stories published by the project. A list of key people interviewed at endline is placed in Annex 3, and the list of documents reviewed is placed in Annex 4. 2.4 Limitations of the Study There are a few caveats and limitations that must be kept in mind as we review the results from this study. First, the performance evaluation was conducted over a period of four years (2015 to 2019), and even though data were collected annually to reduce the bias due to recall of activities over time, it is possible that effects of some of the initiatives undertaken were not reported fully or were missed. Second, access to the faculty administrative data, such as student enrolment, number of staff, number of publications, etc., was a challenge. The evaluation team had to go through the Deans to get access to the person-in-charge which, while not a problem per se, became an issue due to the busy schedule of the Deans and an unwillingness on their part to delegate the completion of the survey. Obtaining administrative data from faculties participating in the University Partnerships program has also been a challenge, with two of the faculties being unable/unwilling to provide complete data for the most recent year. Similarly, the survey of students was a challenge, especially since two faculties (Economics and Education) were unwilling to allow their students to take the survey at endline. And finally, the TLP-SP project was implemented under a co-operative agreement between USAID and World Learning. Cooperative agreements are designed to be agreements that enable the implementing partners to change course to accommodate changes in the field with an agreement from USAID. Changes made to specific tasks under the broad program description are not always documented through contract modifications, and to that extent may have been missed and not featured in the progress updates. USAID TLP Partnership Program Performance Evaluation – Final Report 18 3 Evaluation Question 1: Implementation of The Action Plan, Change Management and Impact on Performance This section answers the first set of evaluation questions: Has the UP’s Action Plan been effectively implemented? Have tangible measures been adopted that show the University’s commitment to change management? How has the TLP Partnerships component changed the UP’s administrative performance? One of the objectives of the Partnerships component was to support the implementation of UP Action Plan of 2013, which provided a roadmap for addressing challenges in the 5 priority areas decided by the then Rector: (i) Revenue Generation and Financial Viability; (ii) Student Admissions, Retention and Services; (iii) Research; (iv) Academic Staff/Teaching; and (v) Facilities, Infrastructure and Equipment.16 However, following a change in management at UP, the new Rector decided to start afresh with a new strategy for UP, and thus a new Action Plan was drafted in 2016. Though not a part of the original statement of work, TLP-SP supported the Rector and his team in the process of developing the new 2016 strategy and the rule books as well as the new Action plan. TLP-SP recruited an international expert to assist the Rector in the planning and design stages of the new strategy, whose contributions proved to be very valuable for the end-product. As a result of TLP’s support, the university was able to successfully draft a new strategy in a very short time and establish standards and prepare the rule-books for implementation of this strategy. The new strategy continued to prioritize strengthening of financial viability, improvements in quality of research and teaching, and improvements in internal and external communications, and since there were no major changes in the broad areas of support, TLP-SP continued to use and report against the milestones and targets developed for the project at the time of its start-up. Table 1 compares the 2013 and 2016 priorities. Table 1: Implementation of UP Action Plan – Strategic Objectives Selected for TLP-SP support Old Strategic Objectives, 2013 Revised Strategic Objectives, 2016 (i) Enhance revenue generation and financial viability (i) Improving internal and external communications (corresponding to old objective 3) (ii) Improve the quality of academic staff and teaching (ii) Strengthening financial viability (corresponding to old objective 1) (iii) Developing the Rectorate’s management, internal communications and public relations capabilities (iii) Improving quality of research and teaching (corresponding to old objective 2) 3.1 Implementation of UP’s Action Plan For purposes of this evaluation, effective implementation of UP’s Action Plan is defined to mean that the priority areas selected from those listed in the Action Plan – improving internal and external communications, strengthening financial viability, and improving quality of research and teaching – have been implemented in a manner likely to be sustainable beyond the life of the project, and that the 16 The Action Plan was an outcome of the USAID-funded Technical Assistance to the Office of the Rector, implemented by Morgan Borszcz Consulting in 2013. USAID TLP Partnership Program Performance Evaluation – Final Report 19 Rector and his team have been provided the necessary assistance by TLP. Each of these priority areas is discussed in turn. 3.1.1 Improving internal and external communications In order to improve internal and external communication within UP, TLP-SP has assisted the university in developing a new Public Relations and Communications (PR) strategy to help improve its image, enhance engagement with students and staff to make them feel a part of the university, and create a transparent and effective way of communicating internally and externally. Activities undertaken for implementing the strategy included updating the UP web site (https://www.uni-pr.edu/), developing a new branding plan for the university, publishing pictures, web stories and articles on the university web site and other social media sites, and creating an UP alumni association to connect current and past students. The PR office was established under the direction of the Vice-Rector for International Cooperation and Public Relations, and was equipped with cameras and camcorders and staffed by interns with expertise in journalism and video production for managing UP’s presence on social media sites. So far, over a hundred web-stories have been published on social media. In addition, five educational awareness-raising videos on important topics – such as academic integrity and plagiarism, and sexual harassment – were uploaded to the UP website. For current and prospective students, the web-site is a way of accessing information on important dates in the UP calendar. The Alumni Association was established at the newly revived Career Center in UP. Discussions with senior management suggest that the PR office has helped improve the way UP is perceived by the community and has helped the “university work with the community, connect with businesses.” The web-site is an easy way of communication for all internal and external stakeholders. Parts of the UP’s first ever graduation ceremony were also streamed online. Managed by UP IT staff, the website is available in Albanian and English, which helps it reach a large audience. The Alumni Association is beginning to play an active role in supporting UP in organizing orientation and graduation ceremonies for the students at UP. Better PR management, transparent rules and stability of senior management have all played important roles in changing the public perception of UP. There is less negative press about the University and its management, and a general belief that the university management is making a serious effort to end corruption. All in all, improvements in internal and external communications are being implemented effectively, and the positive momentum generated in the last couple of years bodes well that these improvements will be sustained over time and beyond the life of this project. 3.1.2 Strengthening financial viability Financial viability in the context of UP refers to better management of its revenues and expenditures, and greater fiscal responsibility to allow for autonomy and self-reliance for UP. UP is a public university, fully funded by the government. A significant proportion of its budget goes towards the payment of salaries, leaving only a small amount for other strategic interventions. Tuition fees, which is its major source of revenue, were reduced significantly by a government order in 2015 that eliminated all tuition fees for first year students and lowered what was charged to students in later years. While the government has since made up the shortfall, the University has very little by way of discretionary funds USAID TLP Partnership Program Performance Evaluation – Final Report 20 to undertake any new initiatives or to expand to new areas. For new initiatives to be sustainable, therefore, they must generate their own revenue and not depend on external funding sources. TLP-SP has implemented several initiatives designed to identify internal and external sources of revenue, improve financial practices, and make UP financially viable. First, TLP-SP established the Financial Advisory Committee to assist the Rector in designing and implementing financial reform. Working closely with the Vice-Rector for Finance and her staff, the Financial Advisory Committee developed eight recommendations for financial reform that could be implemented within the existing laws and rules governing UP. These recommendations along with the supporting draft regulations have been submitted through the Rector and Vice-Rector for Finance to the UP Steering Board for ratification. The recommendations include changing the organizational chart to maximize oversight and minimize mismanagement, adjusting statute and higher education laws to restore management role to the Rector and his senior team, disaggregating university budgets into discrete budgetary divisions and departments, tapping into support offered from outside organizations to train university staff for additional budget management responsibilities, developing comprehensive financial and administrative policies and procedures manuals, normalizing academic staff salaries and ending supplemental pay practices, using recovered funds to staff administration and teaching staff, and revising tuition and fees structure and realigning incentives for students to ensure revenue inflows needed to function as a leading public research university. The recommendation for normalization of salaries is already being implemented. As a result of this reform, UP is saving approximately one million Euros annually, a considerable amount given the tight financial situation of UP. These funds are now available for hiring new administrative and teaching staff. At the time of data collection (May 2019) a call had been put out for the hiring of 35 new administrative staff (10 percent of the current strength). The other recommendations are pending ratification by the UP Steering Board and the Ministry of Finance. Second, TLP-SP assisted in the development of policies and procedures to enable UP to connect its financial system with MEST’s financial system and enable seamless transfer of information between the two systems and provide transparency and better oversight of the budget. Third, TLP-SP brought in Baker and Tilly Kosovo LLC, a consultancy, to work with the Vice-Rector and Director of Finance to develop a financial policies and procedures manual and write the regulations to help make the procurement process work smoothly once the procurement function of the University is decentralized. The finance staff at the faculty level have also been trained on procurement rules and regulations in preparation for the decentralization. The manual for procurement procedures is already in use by the Office of Finance. In addition, after their training the finance staff have a better understanding of their roles and responsibilities and are prepared for the decentralization of procurement. However, since UP has yet to initiate the decentralization process, the effectiveness of this initiative remains uncertain. And finally, TLP-SP worked with UP’s various centers to help them generate revenue and become USAID TLP Partnership Program Performance Evaluation – Final Report 21 fiscally self-reliant.17 Each of the Centers (except for the Small Animal Clinic, which is in the Faculty of Agriculture and Veterinary) has been established at the central/rectorate level with accompanying regulations to facilitate their operation and allow for the collection of revenue. As an example, the Center for Renewable Energy and Sustainability is charging a fee for its courses and has done so successfully for the first cohort of students enrolled18. In addition, Venture UP, an incubator designed to support students and researchers in establishment of new ventures, has been established on the UP premises. Venture UP, an indirect result of the TLP Partnerships program, is self-sustaining and is expected to generate revenue for UP through licensing rights for products developed at the incubator. 19 All of these initiatives have been well received by the senior management at UP. According to the Vice￾Rector for Finance, they could not have achieved this level of progress without the help of TLP-SP. While there is some progress towards financial viability, the total extent of the impact remains to be seen. 3.1.3 Improving the quality of research and teaching UP’s strategy to remain relevant as a center of research and teaching is to improve the quality of education by enhancing the quality of faculty and scholarship and fostering research excellence. For purposes of this evaluation, improving the quality of research at the university refers to the ability of the University to offer targeted support for the academic staff to conduct quality research and attract grant funding, resulting in an increase in the research conducted under the UP banner, an increase in the number of scientific papers and publications, increased participation in internal and external conferences, etc. Similarly, indicators of improved quality of teaching include availability and use of in-service training, use of new methods of teaching and learning, use of technology in the classroom, use of technology for grading and tracking students, and availability of library and other resources. TLP-SP has supported UP in the improvement of the quality of teaching and research for the academic staff through various university-wide and faculty-specific initiatives undertaken by partnerships between UP and four of its faculties – Education, Economy, Engineering and Agriculture and Veterinary – and TLP-SP and partner U.S. Universities: Indiana university, Dartmouth College, Arizona State University and University of Minnesota. These initiatives include the University-wide initiatives which, as the name suggests, are based at the central/Rectorate level and are designed to support staff from across the various faculties at UP. These initiatives include the opening of new centers, such as the Office for Research and Sponsored programs, or reviving existing centers, such as the Center for Teaching Excellence, for improving the quality of research and provision of in-service training to the academic staff. It has also included pilot-testing a new continuous grading policy for students prior to rolling it out to all faculties. 17 These include the Office for Research and Sponsored Projects (Center for Research), Center for Teaching Excellence (CTE), Center for Energy and Sustainability (CES), the Alumni Association and Small Animal Clinic and Institute for Agriculture and Veterinary 18 Some issues have been reported with levels of enrollment and payment of fees for the following cohorts, but the provision exists. 19 Venture UP is a direct outcome of a white paper titled, “Building Venture Incubation at the Public Universities in Kosovo – Strategy, Costs, Benefits and Approach” produced by the TLP-SP Advisory Committee. USAID TLP Partnership Program Performance Evaluation – Final Report 22 Several initiatives have also been implemented at the faculty level to create a culture of team-work in production of scientific papers and research and workshops and seminars to enhance learning of new teaching and research methods and subject-specific knowledge. In addition, academic staff have also benefitted from exchange visits between UP faculties and their partner universities, which has resulted in improved curricula, teaching and reference materials and equipment, and research publications. 3.2 University’s commitment to change management The University’s commitment to change management is reflected in the several tangible measures taken by the Rector and his senior management team. First, UP’s management team started by designing a strategy and supporting rule books to increase transparency and establish standards that could be communicated to everyone to enable rule-based action for governance of UP. The premise behind this was that in a small country like Kosovo, “where everyone knows everyone else…these standards will help implementation and reduce nepotism.” Establishing open and transparent processes serves to convey a high level of intent and commitment to change. Second, UP’s senior management has openly embraced one-on-one coaching and mentoring from an expert in the field of organizational change as it thinks through the various reform processes to enhance the quality of research and teaching and improve the relevance of UP as a center of learning. This willingness to seek outside help suggests both a humility as well as an intent to learn how best to lead and manage change in ways that are sustainable. Study tours for UP’s senior management and the University Steering Board members have also helped provide exposure to best practices in management of similar universities in the U.S. and help expand their own ideas of management. Third, UP’s senior management has encouraged its mid-level administrative staff (those with 15-20 years of experience) and the Deans of various UP faculties to participate in Change Management and Leadership seminars organized for them by TLP-SP project. Attended by all the Deans and 54 management staff (50 percent female and 80 percent from UP), these change management seminars focused on the practical aspects of change management, communications and leadership including delegating to others, demonstrating communication between multiple levels and the fact that leadership can be exercised within their current job descriptions. Mid-level management reported that these seminars have been useful as they are now more comfortable with communicating with their supervisors and those that they supervise. For many of them this was a first exposure to training on leadership. Surveys and focus groups with participants of the Change Management and Leadership seminars indicated that this was the first such training for over two-thirds of the participants, and most reported that the training had given them the skills for communicating with their senior and junior peers. Most agreed that the training was relevant for them and that it took into account the higher education context, though many also reported that the training was too general and did not address issues specific to their levels. The participants felt that as a result of TLP-SP, there was now a “critical mass” at UP to manage change and there was a common perception that even though they were not fully engaged in supporting staff with their projects, the Rector, senior management and the Deans had changed as a result of the change management training and had greater appreciation and respect for the work of their USAID TLP Partnership Program Performance Evaluation – Final Report 23 staff. At the same time, there is also a feeling of discontent stemming from the fact that while the seminars successfully trained the mid-level managers on new ways of managing their work, they have not been empowered to implement the new ideas. And finally, UP has provided training to its finance staff on procurement processes to update their skills on financial management and reporting, and to other administrative staff on managing and coordinating activities of UP’s various centers, such as the Career Center, the Office of Sponsored research, the Center for Teaching Excellence, etc. Members of the University Steering Board, a select group of the mid-level management, and finance and administrative staff, were taken on study tours to expose them to best practices of administration in similar universities in the U.S. Study visits for Center coordinators focused on management of similar centers as well as on new ways of incorporating technology in the centers. All these visits have resulted in a high level of individual learning and an appreciation for use of technology to help with knowledge management. Participants also reported that the study tours provided them with a lot of exposure and novel ideas on how to problem-solve. Perhaps the most compelling evidence of commitment to change is the ownership that UP’s management has taken of the reform process. After an initial intense period of coaching and mentoring and the development of a new strategy and policy, the management team assumed the responsibility of implementing the change in its own way and at its own pace without any external assistance or pressure. UP’s senior management is now working on the reforms on its own schedule and together with UP’s Steering Board, which is suggestive of a commitment that is likely to be sustained. 3.3 Change in UP’s administrative performance A necessary part of improving the quality of education offered by UP is to improve its functioning by strengthening management and administrative performance. Indicators of change in its administrative performance include standardization of administrative processes; fiscal accountability and management; training for administrative staff; improved data systems (systems for information, communication and technology), including access to and availability of data for decision-making; and improved student services (credit-bearing internships, career support, alumni relations). While UP has made progress across several of these changes, it is still too early to assess the impact of this progress. The initiatives and resulting improvements as well as interim impacts are discussed below. 3.3.1 Standardization of administrative processes With assistance of TLP-SP, UP has undertaken several initiatives to standardize administrative processes including developing rule books as a way of communicating the process for implementing the strategy, developing an HR manual for all employees of UP; and developing a Deans framework (to be implemented) to help the Rector and senior management evaluate the Deans in a non-punitive manner against a set of criteria. The development of the strategy and associated rule books has been an inclusive effort and has involved all levels of the Rectorate staff as well as faculty Deans. The rule books, developed with assistance from TLP-SP, have provided a common means of communicating the steps to be taken in implementing the strategy as well as the standards to be followed in its implementation. The inclusive process for USAID TLP Partnership Program Performance Evaluation – Final Report 24 developing the rule books has had the benefit of establishing ownership of the strategy and the standards as well as helping UP’s senior management in its communication and negotiations with the UP Senate and Steering board. UP has developed an HR manual as a main reference document for communication of HR policies, programs and procedures for the employees of UP. The manual outlines the core performance expectations for administrators and staff, as well as the processes for hiring and promotion of academic staff across all faculties in UP. The existence of a written, consolidated source of directives and processes ensures everyone had the same information regarding policies and procedures, increases the likelihood of equitable application of these criteria and policy guidelines, and reduces nepotism and corruption. The administrators have been given the power to enforce these regulations. While some of the administrators complained that “enforcing these regulations was leaving them with no friends,” they felt that the regulations were fair and would help improve the current status at the university. The fact that the Senate, for the first time pushed, back on about 70 percent of promotion requests for academic staff because they did not meet the standards outlined in the regulations, could be considered to be an effect of the rule books and change management training.. Another attempt to professionalize UP management is the development of a Dean’s Leadership Framework outlining all responsibilities of Deans. Also called the Dean’s Score Card, the framework, developed with assistance from TLP-SP and other resources such as the Fulbright exchange program, is intended to be a non-punitive instrument that can enable the Rector to monitor a Dean’s performance each year against certain criteria and enable identification of over and underachievement. It can also serve as a tool for showcasing best practice and as a peer learning platform. The tool has full support of the Rector, but at the time of data collection, was still to be shared with the Deans whose buy-in would be necessary for it be successful. 3.3.2 Fiscal accountability and financial management As discussed earlier, UP is working on financial reforms that would enable it to have greater financial viability and autonomy. The proposals for these reforms along with the draft regulations to support them have been submitted to UP’s Steering Board for review and concurrence. This TLP program identified a perceived need need to move away from the highly centralized model of financial management currently in use at UP and give the Rector and the Deans greater flexibility in managing the University and their Faculties respectively. At this point, the UP Steering Board meets monthly (or more frequently, if needed) to approve all financial decisions for the university. These decisions range from hiring new personnel to participation of academic staff in conferences and to procurement. Also, in the course of writing the strategy and the rule books, it became more apparent to the senior management that much of the decision-making and personnel management within the university goes through the General Secretary of the University. For instance, the staff in the office of the Vice-Rector for Finance report to the General Secretary and not to the Vice-Rector, creating a complicated chain of command. The desire for more control, especially over their own budgets, was also echoed in interviews with Deans of the four UP faculties participating in the TLP University USAID TLP Partnership Program Performance Evaluation – Final Report 25 Partnerships program. 20 As a part of supporting the development of a new strategy and the rule books, TLP-SP has also helped UP draft the financial regulations for possible decentralization of some budget lines to the Faculty level (still in process of approval by the Senate and UP Board). The administrative positions of Chief Technical officer and Chief Financial Officer of UP exist but are currently(at the time of this evaluation) filled with academic staff. TLP-SP has supported the Rectorate in writing the job descriptions for these positions so that they can be filled with professionals with experience in these roles leading to more streamlined management. 3.3.3 Improved data systems Over the past few years there has been a growing recognition that the UP management’s ability to make evidence-based decisions was being hampered by incomplete information on such issues as number of students enrolled, number of students that have graduated, years in program, and so on. An example of such data issues is that prior to the establishment of the data center, UP was thought to have on its rolls approximately 51,000 students. However, following consolidation of the data, the actual number is about 18 percent lower, closer to 42,000. An effort to collect administrative data21from the four faculties engaged in the partnership by this evaluation team have resulted in very mixed outcomes, in that only one faculty could provide the data easily and consistently over the past four years. The remaining three provided baseline data with a lot of cajoling in 2015, but have since declined, indicating that it was difficult for them to generate this information. There are several data management systems at the university, but these systems don’t “talk” to each other. The SEMS data system – an administration and course management system, currently in place at UP and owned by the IT department – was supposed to be able to generate this administrative information at the University and faculty level, but the data was not available either due to an inability of the individuals managing the system to generate data from the system, or an unwillingness to do so. In addition, the University has created another system, linked with the SEMS system, to support it in the process of seeking re-accreditation22. The accreditation will be at the institutional level as well as at the program level, and 90 courses (including new courses developed as a result of the Partnerships program) will be up for accreditation. One of the requirements of this process is quality assurance. All courses are being put on SEMS and as per the quality assurance requirements of the accreditation agency, students will be able to evaluate the courses in the system. In an effort to improve the availability of administrative data, UP, with assistance of TLP-SP, has worked on identifying what administrative data are needed for effective decision-making as well as sources of this data, and has established a system for collecting the missing data. In addition, a new software has been developed in collaboration with the UP IT department that connects with and has the ability to mine and synthesize data in real time from all systems at UP, both old and new. It is expected that once the system is fully functional, it will be able to help UP management with decision-making by quickly and 20 However, there does not appear to be a consensus on the definition of decentralization, with the faculties looking at it as a procurement function in which decentralization would allow them to have their own bank accounts (rather than a subaccount within the University budget) and the ability to manage procurement at the faculty level. 21 Administrative data from the faculties includes the number of students enrolled overall and by year, number of academic staff, publications by staff, etc. 22 UP is a part of the Bologna process and is required to go through periodic re-accreditation. USAID TLP Partnership Program Performance Evaluation – Final Report 26 efficiently generating the data needed to respond to their queries. At the time of this evaluation, the center was in the process of automating reports to support the university with periodic updates on student and faculty data. 3.3.4 Improved student services UP has revived the Career Development Center to enhance services to students and help improve their readiness for the workforce. The Career Development Center will offer guidance to students on finding internships and job opportunities. At this point, it offers current students access to the Career Center database that includes information on current students at the University and information about companies that they may apply to for internships and/or work opportunities. The database enables the students to connect with potential employers, sign up for trainings, and apply for jobs and create CVs using pre-loaded templates. Career Center is in the process of working with the UP Faculties to develop a program to enhance “service learning” and “credit-bearing internships.” In addition, it has established an Alumni Association under the Office of Academic Development to increase the students’ exposure and interaction with the alumni. The Alumni Association, registered as an NGO, is located on UP premises. As it shares space with the Career center and has access to facilities for printing, internet, and allied services, it offers students an opportunity to interact with the alumni who may be there using these services. The Association has a webpage and a database linked to the Career Center database and offers current students with access to information on UP’s large and diverse alumni base to enable them to connect with alumni to seek mentorship, internships and potential work opportunities. The Alumni Association is also supporting the Office of Academic Development through organizing new-student orientation days to welcome the new students and graduation ceremonies to celebrate completion of their course. The first graduation ceremony was broadcast live on UP website. Following recommendations from the TLP-SP Advisory Committee, Venture UP, an incubator, was set up at UP to provide opportunities for students and researchers interested in starting new ventures. Founded by a TLP-SP Scholarship recipient, Venture UP is an independent NGO, based on UP premises, providing aspiring entrepreneurs, especially from UP, with workspace and services such as internet, legal advice, assistance with development of marketing materials and support on pitching ideas to the community. It is providing training to 500-700 UP graduates annually in skills to improve their employability. Even though Venture UP is not a direct outcome of the Partnerships program, it has been included here due to its tangential relationship to the University. USAID TLP Partnership Program Performance Evaluation – Final Report 27 4 Evaluation Question 2: Effect on Academic Performance and Implementation of “Grand Challenges” This section answers the second set of evaluation questions: How has the TLP Partnerships component changed academic performance in targeted faculties over the course of the project? Have the “Grand Challenges” identified for each partnership been effectively implemented? 4.1 Improving academic performance in targeted faculties Improving academic performance at UP refers to improvements in both the quality of teaching and the quality of research being conducted at UP. Improvement in quality of teaching includes use of more modern student-centered methods of active learning and teaching, improvements/changes in curricula and syllabi used by the faculty, and availability of adequate resource materials and modern equipment for teaching in the classroom. Improvement in research refers to improvements in both quality and quantity of sponsored research and the support needed for the conduct of research. The TLP-SP Partnerships Component, has helped improve the academic performance at UP through a two-pronged approach. One, it has created and supported partnerships between four select faculties at UP and four U.S. university partners on targeted initiatives, including elements of staff exchange visits (U.S. to UP and UP to U.S.), and support for the faculty in terms of equipment, materials, textbooks, etc. It has also included small research grants and support for research clusters in each of the faculties. 23 And two, it has enabled the establishment of centers that support improvements in teaching and research at UP, which support not only the four faculties in the Partnerships program but also the remaining faculties at UP. These faculty-specific partnerships and university-wide activities are discussed in turn. 4.1.1 Improvement in quality of Teaching 4.1.1.1 Faculty-specific partnerships Faculty of Education: The partnership between the Faculty of Education and Indiana University covered the full span of four years and was one of the more successful partnerships brokered by TLP-SP. The partnership included 16 exchange visits of teaching staff to the U.S. and 40 visits of staff from U.S. to UP. Exchange staff at the Faculty of Education reported being engaged in research, meetings, discussions, classroom observations, and collaboration with the faculty at Indiana university. They stated being able to observe professors teaching classes while using different ways of teaching, including active teaching and learning methods, and being encouraged to attend seminars. In addition, the faculty provided them with books and materials that they could bring back with them. One staff member reported that she had received so much material from their visit that she had not been able to review it yet. Another reported that she had learned new ways of teaching research and presentation. Others reported that it was only after a 23The U.S. partner universities were a part of the TLP-SP implementing team led by World Learning Inc. and received sub￾grants from WL. The partner universities included Arizona State University, Indiana University, University of Minnesota and Dartmouth College. The initial grants were for a two-year period (2015-17) to implement the designated activities, which they supplemented with own resources. These sub-awards were extended through 2019 and in one case, Arizona State University, expanded to include university-wide initiatives as well. USAID TLP Partnership Program Performance Evaluation – Final Report 28 lapse of almost two years that they were they able to start using and incorporating the new skills into their teaching. The visiting faculty from Indiana University have held workshops on various topics to demonstrate and share new methodologies with the staff. The workshops included qualitative and quantitative research methods, proposal writing, grounding courses and developing syllabi for student centered teaching and discussions on pedagogy. They have also helped UP’s Faculty of Education organize an annual scientific summer symposium at UP. In addition, the partnership has supported the translation and printing of a textbook on research methods, a much needed resource for the faculty. It has also been instrumental in enhancing the resource repository of the Faculty with a broad range of textbooks and other resource materials as well as access to key instructional and academic research resources through the Indiana University library. According to the Dean of Education, the joint cooperation between Faculty of Education and Indiana University has been “one of the more influential programs” at the Faculty of Education and the TLP-SP project has been an invaluable source of assistance. According to the Dean, some academic staff have changed their teaching styles and one of them is developing a manual on teaching methods. Faculty of Agriculture & Veterinary: The partnership between the Faculty of Agriculture & Veterinary and University of Minnesota was also a successful partnership, and lasted the full four years of the duration of the TLP project. The Faculty of Agriculture & Veterinary has expanded its participation from one department (Veterinary Medicine) and has added two more (Department of Food Technology with Biotechnology and Department of Plant Protection), enabling a larger pool of participants to benefit from the exchange and mentoring activities of the Partnerships component. It included 29 staff exchange visits (13 UP to U.S. and 16 U.S. to UP). At the Faculty of Agriculture & Veterinary, the exchange staff observed classes using multiple methods of teaching, attended seminars including an experiential learning course on “Focus on Food Production”, participated in discussions with faculty and gained laboratory experience in veterinary pathology. They also went on field visits to observe the use of technology in dairy and food animal farms and processing facilities, as well as in food safety and safe meat-processing techniques. The visiting faculty presented workshops on various topics, including active learning teaching strategies, mentoring students and strategies for success of graduate students, an introduction to extension services and models, and an intensive seminar – Farm to Table – focusing on the poultry system. In addition, they offered guest lectures and/or co-taught courses in their field of specialization including soils, food microbiology, infectious diseases, small animal radiography and personal safety, small animal anesthesia and emergency procedures, and followed up with the UP to U.S, exchange staff on their use of new content, discussion activities, and assessment techniques. They also used their time at UP aiding the faculty staff in understanding and setting up equipment that was lying unused and planning for new clinics for providing services. They also brought in parts for making the equipment functional and provided training to the UP staff on use of the equipment. At the Faculty of Agriculture & Veterinary, the assistance provided by the Partnerships program has been more technical rather than academic in nature, and has enabled the faculty to provide a bridge between UP and the private sector and offer veterinary and agriculture services. With technical USAID TLP Partnership Program Performance Evaluation – Final Report 29 assistance from the Partnerships program, the faculty has established a newly-furbished Small Animal Clinic ready for use and has the strategy and business plan for an institute of agriculture and veterinary. In addition, small animal radiography equipment available at the faculty has been installed and missing parts provided to make it functional, and both the veterinary faculty members and students have been provided training on small radiography and personal safety resulting in the first small animal inhalant anesthesia clinic in Kosovo. The initiative for the Small Animal Clinic was so successful that the faculty received Euros 170,000 from UP as funding for a building to house the clinic and the institute. Finally, with the assistance of University of Minnesota, the Faculty of Agriculture & Veterinary has developed the strategy and the plans for launch of an institute for provision of agriculture and veterinary services. UP has approved the plans for the institute, which is currently pending approval from the UP senate. According to the Dean of Agriculture, the partnership has resulted not only in enhanced curriculum for the Faculty, but also in exposure to new techniques that can be practiced by the Faculty staff. He felt that the exchange program had resulted in a great deal of personal learning for the participants who were able to review the syllabi and observe the way the faculty organized themselves and taught classes at the University of Minnesota. The Dean reported that to ensure sustainability of the initiative, the faculty was focusing on sending junior academic staff from UP to the semester-abroad program. These staff have returned with curricula, books, etc., and have shared their experience with their students – but there was no formal mechanism to share knowledge with other colleagues or for the faculty management to track any changes. Faculty of Economy: The partnership of the Faculty of Economy with Dartmouth College was expanded during Phase 1(2015- 17) to include teaching staff from the department of Philosophy, which increased the pool of participants in the exchange program (6 from UP to U.S. and 20 from U.S. to UP). However, the partnership lasted only for the first phase as the partners could not arrive at an agreement for a meaningful work plan for the second phase. The visiting faculty from Dartmouth worked with UP staff to help facilitate the delivery of an Experiential Learning module in an existing class at the university. The visiting professor and his team of graduate students demonstrated the use of surveys for collection of data to provide information to a client. The pilot module lasted for three weeks and had three groups of UP students working closely with three Kosovo companies on real-life business cases. The exercise was very well received. In addition, they presented a seminar on Building Grassroots Organizations, launched clubs and provided training on Active Learning techniques. Interviews with staff from the Faculty of Economy, who participated in exchange visits during that time, indicate that, while at Dartmouth, they closely shadowed the faculty who were teaching the experiential course on “Business Consulting” using the “active learning via consulting” approach so that they could present the module upon their return to UP. The UP exchange staff have successfully piloted the course at UP and have now submitted it for accreditation. The exchange visitors from the department of Philosophy attended the GRID Spring Symposium focused on the topic of Gender Matters – Feminist Ecologies and Materialism while at Dartmouth. USAID TLP Partnership Program Performance Evaluation – Final Report 30 While there were no formal means of assessing sharing or use of the learning, the Faculty of Economy was the only one to insist that the returned exchange visitors hold brown bags discussions and share their experience with their colleagues in the faculty with the aim of helping to improve their practice. Even though the partnership lasted only for the first phase, it was effective in having a lasting impact on the Faculty of Economy in terms of the use of training and course materials. Faculty of Engineering: The partnership of the Faculty of Engineering (ECEN) with Arizona State university (ASU) started out on a solid footing with exchange visits from ASU to ECEN and planning for a multidisciplinary program at ECEN that would offer advanced degrees for studies in energy and sustainability. However, the partnership did not fulfill the needs of the Faculty and eventually the university moved the Center for Energy and Sustainability to a central level at the Rectorate. During 2015-16, the first year, the partnership delivered four sets of seminars intended for a broad audience of ECEN academic staff and students, as well as members of the wider public interested in computer engineering and energy in Kosovo. Attended by over 150 faculty members, students, and interested community participants, these seminar presentations had the desired effect of building broader interest in the field of energy and related studies. At the same time, three ECEN faculty went to the U.S. for a semester-long exchange visit under the semester in residence program. Under the guidance of specially chosen ASU faculty mentors, these staff pursued syllabi intended for use and classroom delivery upon their return. It included courses and seminars on “photovoltaics, ” “introduction to sustainability,” and traditional courses in energy engineering at the ASU Fulton School of Engineering as well as from the ASU School of Sustainability. In addition to regular coursework, a special weekly seminar was arranged for the visiting scholars, using the seminar to introduce them to specialists working in the field of renewable energy and sustainability. Due to difficulties (related to visa issues and limited interest) in finding enough staff to go on the exchange visits, the program was expanded to include academic staff from the physical sciences and to increase the number of visits of U.S. partners to UP. This resulted in a total of 45 exchange visits, of which 9 were from UP to U.S. (3 academic staff, 6 management) and 36 from the U.S. partners to UP. But most of these visitors to UP were directed towards setting up the multi-disciplinary Center for Energy and Sustainability at the Rectorate level. Moving the Center from the faculty to the central level resulted in a significant reduction in the resources available for the partnership at ECEN and weakened its position. 4.1.1.2 Effect of the Faculty exchange program As described above, most of the partnership activities were facilitated through exchange visits of academic and management staff from the partner universities to UP and vice-versa. The semester-in￾residence program, a part of the exchange visits component, which allowed academic staff from UP to spend a semester in the United States participating in courses and seminars, developing course materials and building relationships with the faculty at the partner university, was modified to meet the needs of UP’s academic staff. For a multitude of reasons, including instability at the university, financial concerns, language barriers and family responsibilities, especially for those with young children, there was less￾than-anticipated interest in the program, with fewer academic staff applying to the program. All four of USAID TLP Partnership Program Performance Evaluation – Final Report 31 the participating faculties re-visited the program and revised the conditions of participation so as to encourage participation.24 On their side, the U.S. partner universities also had to make some adjustments to their plans for the exchange visits based on the experience with the first cohort of exchange visits. These changes included ensuring more hands-on planning with participants at the beginning of the program to ensure their expectations were realistic, the workload manageable and in line with the participants’ goals, and that the participants came in groups of two or more. These modifications resulted in the distinct advantage of a program custom-tailored for each faculty. The program continued in Phase 2 for three of the four participating faculties (all except the Faculty of Economy). The most successful component of the faculty-specific partnerships was the exchange visits by the faculty from the U.S. University partners to UP. This component enabled the U.S. faculty member to present seminars, lectures and workshops at UP, and have a much larger impact than that of individual academic staff from UP going to the United States. Many academic and non-academic staff from UP, graduate students and in some cases even members of the private sector (as in the case of Engineering), were able to attend the seminars and benefit from the discussion and presentations. The visiting academic staff presented on topics such as grant writing, development of syllabi and other topics for improvement of teaching. In some faculties, for example the Faculties of Agriculture & Veterinary and Engineering, the visiting faculty also co-taught courses demonstrating active learning techniques and techniques for management of large classes and presented seminars of specific interest to those faculties. Across all four partnerships, academic staff that participated in the semester-in-residence program (UP to U.S. exchange) were unanimous in their appreciation of the program and the level of exposure to various kinds of teaching methods and technologies. Approximately three quarters (between 70 and 85 percent) of the respondents to a survey of participants of the exchange program reported that during their visit to the U.S. they were able to attend and observe classes at their host university, spend time at the library and visit institutions outside the university that were relevant to their field of interest. A little over half also reported attending conferences (52 percent) and engaging in individual consultations with a U.S. professor (65 percent). Respondents also reported being able to develop new syllabi (44 percent), developing a whole new program/curriculum (30 percent) and engaging in field work related to their area of interest (35 percent). Two-thirds (65 percent) of the respondents reported bringing back books and course materials from their visit to the U.S. They agreed (strongly agreed or agreed) that they had introduced new elements into their courses based on their semester abroad experience (70 percent) and that these changes were likely to be sustainable due to support from the management (52 percent). Exchange program participants reported learning new teaching methods (87 percent), new ways of organizing the classroom (78 percent) and managing large and midsize classes (65 percent). Focus groups with participants and interviews with Faculty management also supported this data. Focus group participants also reported that they observed classes where the professors were using interactive 24 For example, Faculty of Education negotiated for a reduction in the duration of the program from a full semester (almost four months) to two months. The Faculty of Agriculture & Veterinary proposed sending 3 people for 40 days each in place of one for 120 days. The exchange staff under the partnerships at Faculty of Economy and the Faculty of Agriculture & Veterinary included academic staff from other faculties (Economy has included staff associated with Center for Gender Studies, staff came from department of Philosophy; and Agriculture & Veterinary has included Veterinary Medicine) so as to have a large enough pool for selection of candidates. Still others, like the Faculty of Engineering which was not able to send academic staff to the U.S. for various reasons, requested that the program be substituted with increased visits to UP by U.S. faculty. USAID TLP Partnership Program Performance Evaluation – Final Report 32 teaching methods, including gymnastics, circular seating arrangements, etc., and while they found them interesting, they would be hesitant to use them in Kosovo. 4.1.2 Improvement in Quality of Research The TLP program, through exchange of academic staff between UP and the U.S. Universities, has facilitated the exposure of academic staff to research methods and new methods for development of courses. In a survey of exchange staff (UP to U.S.), almost all reported that during their exchange visit they had access to the library (96 percent), and a little over three-fifth also reported access to research journals and databases (61 percent). Seventy percent of surveyed exchange staff reported developing professional relationships with the host faculty, while a little less than half reported receiving mentoring (48 percent) and support (44 percent) from the host faculty for developing their research competencies. About a third of survey respondents reported being able to work on their own research projects (30 percent) and writing papers (35 percent), while only 13 percent reported collaborating on research projects and 17 percent writing papers with U.S. colleagues. Respondents who reported working with colleagues in the U.S. partner universities also reported continuing to work with them even after their return to Kosovo. These findings were also supported by comments from exchange program participants in focus groups and by the management of the faculties. The visiting faculty presented workshops and seminars on writing and publishing scientific papers, on mentoring graduate students, and on grant-writing. They also encouraged the academic staff at UP to work in teams, demonstrating this concept by creating research clusters. As a result of these presentations, academic staff and students at the Faculty of Economy, for instance, have learned new ways of conducting market research and presenting the data. Despite their almost complete focus on developing service clinics, the Faculty of Agriculture &Veterinary has also benefited from the research training, as evidenced by co-authored research papers. Mini-research grants have also been provided through the Center for Energy and Sustainability. Administrative records from TLP-SP show five research projects being implemented under the ECEN/ASU partnership.25 The largest beneficiary of the research component appears to have been the Faculty of Education. Under guidance/mentoring of professors from Indiana University, the Faculty of Education organized itself into seven research clusters, which were provided with mentoring in-person and via email and Skype by faculty from IU. These research clusters are working collaboratively on research papers for publication and to present at the Scientific Summer Symposium: Enhancing Education Research at the Faculty of Education. The first drafts of these papers were presented by the faculty at the symposium and will be revised for submission to journals. Regardless of whether the research clusters develop into a sustainable research model, exposure to research methods will improve the quality of teaching and learning in the faculty. Over the course of the partnership, the academic staff at the Faculty of Education have come a long way. For some this has been a first introduction to using email and online materials; for others, who were more advanced to begin with, it has been a higher level of collaboration and access to scientific journals. As a result of the research clusters and expanded professional networks, two research papers have already been published and others are in process. The research clusters have 25 However, according to the Dean of ECEN, these projects are not located at ECEN and are in fact being conducted at the Center for Energy Sustainability. USAID TLP Partnership Program Performance Evaluation – Final Report 33 also had the effect of changing the culture at the Faculty with more of the academic staff working together in teams on research papers. A team of professors has jointly applied to the U.S. Embassy for grants on science teaching. Overall, the partnerships have helped each of the faculties in the program improve their curriculum as well as the instruction methods through several different ways, including sharing of curricula and syllabi by faculty at partner universities, helping UP academic staff develop new courses based on their learning, demonstrating active teaching and learning methods either by having the exchange staff observe classes during the semester-in-residence program or through co-teaching courses during visits of partner university faculty to UP, lectures on teaching methods and development of curricula, and assistance with expanded library resources. 4.1.3 Centers to support improvement in teaching and research (University￾wide Activities) In addition to the faculty-specific partnerships and exchange program, the project has supported the improvement in quality of teaching and research at UP by establishing and/or reviving various centers, including the Center for Teaching Excellence, Centre for Energy and Sustainability, and Office for Research and Sponsored Projects. 26 These are described below. Center for Teaching Excellence (CTE): With support and technical assistance from the Partnerships component, CTE has been revived and relaunched with the express purpose of providing UP academic staff with in-service training to improve teaching and research.27 Staffed by 22 trainers (12 newly certified) over the past four years, CTE has offered 12 workshops and trainings on various topics of interest to 267 academic staff at UP. Some of the earlier trainings and workshops offered with assistance from TLP-SP provided trainers (ASU and WL experts) included workshops on teaching methodology, course management, syllabi preparation, hybrid teaching methods, student evaluation/assessment, leadership and mentorship of MA and PhD candidates, research grant writing and a lecture series on renewable energy – Measuring Your Carbon Footprint: The Greenhouse Effect (delivered to ECEN Faculty and students). The center is now offering two trainings on a regular basis – a two day basic training and a shorter advanced training – that all academic faculty must complete. The basic training covers topics such as theories of learning/teaching, teaching/learning styles, classroom management, interactive techniques in teaching, learning assessments, etc. Advanced training includes modules on syllabus design, problem￾based learning, action research in classrooms, active and interactive university classroom, and designing learning outcomes. In response to demands from academic staff for more in-depth courses, the center has developed courses on differentiated instruction (how to modify instruction to reach every student in 26 Other initiatives included in the original plan of the project, such as the Center for Continuous Education and the provision of online courses, were modified and/or dropped in line with the changing priorities of the UP administration. An additional center recently approved by UP is DataUP, which follows the working principles of “Domestic Data Center” and will serve the big data analysis needs inside the university and externally. As it is a recent addition to UP and is just starting up, it has not been included in this evaluation. 27 The Center for Teaching Excellence was established at UP in 2004 with donor support. However, lack of support after the project ended resulted in the Center becoming an inactive body, that existed only on paper. USAID TLP Partnership Program Performance Evaluation – Final Report 34 the classroom) and use of technology in the classroom. It also has the capacity to offer trainings on use of online course management systems, specifically working with the “CourseSites” software. Surveys and focus groups with participants of the workshops and seminars offered by CTE reveal that over two-thirds of the respondents found the training very relevant for them. The format used for presentation was conducive to learning and easy to use and adapted to the UP context. They reported that as a result of the training they felt empowered to make a change in their practice. However, a large proportion of the respondents said that university policies did not allow them to make any changes to the way they taught. They noted that while workshops were a good beginning, the topics covered provide only a broad over-view and there was a need to include opportunities for hands-on practice for skills learned. They observed that in some cases the training was very high level while for others it was too basic, suggesting the need to ensure that the seminars and trainings offered were at the ability level of the participants, even if it meant having separate sessions for academic staff from social sciences and natural sciences (including engineering) as well as by years of teaching experience. Participants from focus groups reported that the activity of teaching students varied by discipline, because of which having common courses across the board for all academic staff was useful for introducing the concepts – but needed more discipline-specific examples to make it relevant. Finally, they asked for the training to address the training topics in greater detail than was possible in the 45 minutes, which was the maximum allowed in the current program. Our research revealed evidence of tension between the senior trainers and the junior, newly certified trainers. For the UP academic staff to be supported by the CTE, this tension must be resolved. Centre for Energy and Sustainability (CES): Launched in September 2017, CES offers a multi￾disciplinary certificate program in sustainable and renewable energy. A first of its kind at UP and in the region, CES was developed as a vehicle to generate practical solutions and policies to address problems of global warming, climate change, conservation, etc. Initially planned to be a part of the Faculty of Engineering, it was established as a stand-alone center at the central level in UP. With oversight from the Vice-Rector for Teaching, the center draws its staff from several UP faculties. CES offers a Certificate program in “Renewable Energy and Sustainability” for UP students as well as for the private sector.28 This course consolidates courses currently taught in different faculties, which were updated with assistance from TLP-SP and ASU to meet the needs of certificate participants. Offered in addition to the Bachelor’s degree awarded by UP, this certificate program is expected to lead to better employment opportunities for the students. Students pay a nominal fee of Euros 35 to enroll in the certificate program and have the flexibility of completing the coursework in one semester or over a period of two years. The first round of the certificate program was offered to two cohorts of 50 students each in 2018-19. However, the program has not been very successful, and after the initial enrollment of students, there has not been enough interest in the program. In the second semester, the students did not pay their fees and not enough students enrolled for the following semester. The program is currently being reorganized both in terms of content (to make it less geared for science students) and administration. 28 Initial discussions for the launch of this center were held at the faculty for Engineering, however, due to its multidisciplinary nature, the center was established at the University level. USAID TLP Partnership Program Performance Evaluation – Final Report 35 Office for Research and Sponsored Projects (ORSP): Following recommendations from the Advisory Committee, UP established the ORSP in 2017 to encourage and facilitate the engagement of its academic staff in sponsored research and generate revenue for UP. Located in a space provided by the University, the center is fully furnished and functional with a coordinator and two staff members. The center has access to research tools such as “COS” and “PIVOT” for finding grants opportunities. The center is helping improve the quality of research by working with staff members at the various faculties to provide training as needed on basics of current day research such as ability to use a computer, to have an email to access online journals and reference materials to hosting seminars and workshops on writing grant proposals and responding to funding opportunities. 29 The staff also send out periodic emails informing academic staff about upcoming grant and research opportunities. In addition, the center staff have worked carefully to update and correctly track research and publication information for all of the UP’s teaching staff. This improvement in data quality is said to have contributed to UP’s significant increase (several hundred points) in rank on the Webometrics platform. While ORSP is a step in the right direction for UP to be recognized as a research university, there is a need to ensure it becomes a central point in all research. Conversations with academic staff at the faculties participating in the Partnerships program suggest that not all staff interested in conducting research were even aware of ORSP’s existence (Only 60 percent of the respondents in a survey of academic staff that had participated in the Partnerships exchange program reported being aware of the program). Also, at this point, ORSP is able to offer only the very basic services such as disseminating information on grant opportunities, photocopying and printing. It is not able to, for various reasons including lack of staff, offer support for writing proposals, developing proposal budgets or post-award project management, things that can make it attractive to prospective researchers. Finally, ORSP needs to market itself to the faculties, since not all Vice-Rectors and Deans have bought into the idea of a common research center. 4.2 Progress towards meeting the Grand Challenges For purposes of this evaluation, effective implementation of the Grand Challenges for each partnership means that: (i) the Grand Challenges were identified; (ii) information about these challenges was shared with key stakeholders, including UP senior management and perhaps even externally with the public; and (iii) the Faculties, with assistance of the senior management of UP, actively worked towards developing a program that would enable them to meet the challenges. TLP-SP, with help of its Advisory Committee (AC), identified the challenges facing higher education institutions in the fields of Engineering, Agriculture, Education, and Economics as they prepare students for the workforce, so that its activities could be designed in a way to help mitigate these challenges. Called the “Grand Challenges,” these are: 29 The seminars and workshops were offered by TLP-SP partner/sub Arizona State University. USAID TLP Partnership Program Performance Evaluation – Final Report 36 Grand Challenges as identified by Advisory Committee ● Engineering: Sustainable energy, information technology and new materials ● Agriculture & Veterinary: Development of profitable agricultural production, changes to regulations and laws, innovation in production techniques, understanding the importance of environmental impact, training in contemporary modes of marketing and management, and strengthening of extension services ● Education: Workforce development, innovation and technology, and specific needs and challenges facing key industries and sectors ● Economy: Enhancing managerial skills, research capacity in institutes, collaboration with ICT, ways of developing incubators, and incentives to develop interdisciplinary work However, as was noted at the time of the baseline study and again at midline, once the Grand Challenges were identified, there was no public communication of the identified issues and they were not actively considered by the members of the Partnership program as they designed and/or planned their activities. Neither the four selected UP faculties nor their U.S. Partners seemed to have any knowledge about these Grand Challenges. However, many of the activities undertaken by the Partnerships component support these Grand Challenges. Some examples are presented below: ● The Faculty of Engineering has held seminars on topics such as ‘Electricity Regulation and its Impact’ as well as ‘Signal Processing and Machine Learning with Application to Power Systems’. and ‘Renewable Energy, Sustainability, and Measuring Carbon Footprint.’ During their semester￾abroad, the exchange staff focused on the topic of renewable energy and received a lot of exposure. One of them has designed and accredited a new course on this topic. However, the planned Master’s program at the Faculty did not materialize. Resources were instead diverted to helping UP with establishing the Center for Renewable Energy and Sustainability at the central level (under the Vice-Rector for Teaching) that will provide certificates in energy audit. In sum, these initiatives together are helping the faculty prepare students to address the Grand Challenge of sustainable energy, information technology and new materials in Kosovo, albeit not at the same level as would have been achieved by having a Master’s program focusing on Renewable Energy within the Faculty. ● The Faculty of Agriculture & Veterinary, with workshops on topics such as ‘Farm to Table: Dairy Systems in Kosovo’ and introduction of new technologies such as X-Rays for animals is addressing its own challenge of innovation in production techniques. Similarly, with the repair and upgrade of existing machines the department of Veterinary Medicine is able to offer anesthesia services, a first for the country. Setting up a small animal hospital in the University and plans for setting up an institute at the faculty (a strategy and business plan has been approved by the Rector and is pending approval by the Senate), to provide services and linkages with the private sector are also steps in the same direction. All of these initiatives together are addressing the challenge of innovation in production techniques as well as the strengthening of extension services, a part of the Grand Challenge. ● The use of “active learning via consulting” approach by the Faculty of Economy is enhancing their own research capacity as well as that of the private sector institutions they work with. Based on the use of research methods and hands on data collection and analysis this approach will enhance the decision-making capacity of the private sector institutions. Introduction of a course that includes these research methods (pending accreditation) will enhance the management skills of the graduates from the Faculty of Economy, again partially supporting the USAID TLP Partnership Program Performance Evaluation – Final Report 37 Grand Challenge of “enhancing managerial skills, research capacity in institutes, collaboration with ICT, ways of developing incubators, and incentives to develop interdisciplinary work.” ● In addition, originating from the recommendations of the Advisory Committee, the incubation center at UP – Venture UP – has been established as an NGO, but the fact that it is located on UP premises and has an MOU with the University makes it an integral resource for university graduates. To the extent that this is an incubator and offers training on skills for the workforce and provides resources for new enterprise, it is helping meet the Grand Challenge for the Faculty of Economy and support students and young academic staff from other faculties in UP and other Universities in Kosovo. ● As a way of preparing its students to be prepared for the workforce the Faculty of Education is beginning to introduce more technology in its classrooms, as well as seminars and lectures that include training on use of computers, email, English language, etc. It is also supporting research clusters as a means of broadening the exposure for the academic staff and students to the current trends and innovations. Workshops on topics, such as use of quantitative research and data analysis, qualitative research and mixed methods, and applied research methods, are designed to build the capacity of the Faculty to conduct primary and secondary research. The academic staff have also been introduced to the Qualtrics platform, a powerful online tool for collecting qualitative and quantitative data. The Faculty hosted the first ever Scientific Symposium, now an annual activity, with support from TLP-SP and the TEMPUS project. All these initiatives together are a step towards meeting the Grand Challenge of “workforce development, innovation and technology, and specific needs and challenges facing key industries and sectors” for the Faculty of Education. In addition, the various university wide cross-cutting activities under the TLP-SP Partnerships program were designed to together improve the quality of academic staff and teaching, thereby also meeting the essence of the Grand Challenges”. In sum, even though no formal connection has been made between the activities and the expected cumulative effect of finding a solution to the challenge, good progress has been made towards meeting the Grand Challenges. USAID TLP Partnership Program Performance Evaluation – Final Report 38 5 Evaluation Question 3: Utilization and sustainability of new technology, systems, knowledge and skills This chapter responds to Evaluation Question 3: Have the selected faculties been able to utilize, maintain and build up new technology, systems, knowledge and skills? What actions have been taken and/or mechanisms put in place to help ensure the sustainability of the partnership efforts? 5.1 Utilization of new technology, systems, knowledge and skills All four faculties participating in TLP-SP – Agriculture & Veterinary, Economy, Education, and Engineering – have made significant but varying degrees of progress in building up new technology, systems, knowledge and skills. Using resources from TLP-SP as well as from other ongoing donor￾funded programs at UP, the faculties have updated their technology and systems to track and monitor progress of students, provide them with access to online course materials, connect them upon graduation with alumni for internships and work opportunities, and give them access to the latest knowledge and research materials available. Across all the four faculties engaged in the Partnerships program, online courses are generating significant interest among academic staff and students, though the offerings have been limited and the uptake slow. Focus groups with the staff and interviews with the Deans indicate that the staff are ready to use online courses as a part of their teaching – or at the very least, use the online platforms for posting their materials or syllabi for their courses – but since there are two competing systems – CourseSites, which enables staff to upload up to five courses each, and SEMS, the old system in existence at UP – staff are awaiting a decision by the University which system to use so that they do not end up duplicating efforts. In terms of equipment support, the faculty of Engineering has received white boards, computers and projectors to modernize their classrooms and the faculty of Agriculture has received support with making existing equipment functional and some new equipment to support the creation of the small animal clinic. Specifically, the Faculty of Agriculture & Veterinary has, with assistance from TLP-SP, fulfilled its long-standing vision of setting up clinics and providing services to the public. The Faculty has also developed the design and strategy for establishing an Agricultural institute within the Faculty and has submitted it to the Board for approval. Existing equipment that was unusable has been repaired and put in use with support in terms of new parts and technical knowhow from the partner University. The selected faculties are beginning to use the technology, knowledge and skills offered by the Partnerships program to enhance the quality of teaching and research, which are discussed in detail below. Teaching methods Across all four faculties, the exchange program staff reported receiving exposure to active learning and teaching methods, group work and use of technology in the classrooms. The visiting staff from the partner universities also provided training on teaching methods, curriculum development and research skills to the participating faculty staff, who also received training on management of large classrooms while using group work and active teaching methods. USAID TLP Partnership Program Performance Evaluation – Final Report 39 Some of the returning staff reported using the new teaching methods in their teaching at UP, while several others reported being unable to use the new skills as they could not make the connection between the new teaching methodology and its application to the context in Prishtina as the class sizes in Prishtina were much larger than those in the U.S. and the classrooms did not have the kind of technology as in the U.S. classrooms. Still others reported feeling self-conscious in using the active learning techniques in their classrooms in UP. The experience across the four faculties is mixed. At the Faculty of Education, academic staff have been slow in adopting and incorporating new teaching methods that they learned from the visit to the partner universities in the U.S. Likewise, there has been minimal use of the new methods in the faculties of Engineering and Agriculture & Veterinary. In the former, only one of the three academic staff that went to the U.S. is currently at the faculty, and he is using new approaches in his teaching. In the Faculty of Agriculture and Veterinary, staff have been unable to use new methodologies in their classes at UP as the classrooms are not internet-enabled. Staff are trying to include quizzes and electronic pages to assess students, but the large class sizes and the lack of technology is making it difficult. At the Faculty of Economy, however, the academic staff have been faster in incorporating the new methods of teaching, grading, etc., into their course work, and the Professor as well as the two teaching assistants who went to the U.S. are all implementing active learning in their approach to their courses. The returning exchange staff was expected to share their new knowledge and skills with colleagues within the Faculty, and hence spread the knowledge through peer-to-peer learning. This has not happened, however, since there is no formal mechanism, such as brown bag seminars, to do so. The only exception is the Faculty of Economy, where the Dean requested the returned exchange staff to demonstrate the new methods to the other staff. One of the goals of the faculty-specific partnerships was to help the Faculty gain new and updated curricula and course materials. The semester abroad participants from all four faculties reported that having developed new syllabi with the help of the partner university or bringing back enough materials for whole new courses. Several exchange staff reported that the faculty in their host institutions were very generous with sharing materials in hard and soft copy. Even though it had taken time, almost two years in some cases, for the academic staff to do this, they have incorporated the new teaching methods and the course materials into their own teaching and some of them are planning to submit these new/updated courses for the next round of accreditation. The Faculty of Economy has developed a new course on Business Consulting, which was introduced following the staff visit to Tuck School of Business at Dartmouth and will shortly be submitted for accreditation. The Faculty of Electrical Engineering is planning to introduce three new courses at the next round of accreditation. No new courses have been developed at the Faculty of Education or at the Faculty of Agriculture and Veterinary at this point. Research methods In an effort to improve the research culture at UP, the Partnerships program has helped create research clusters in which academic staff could work in teams and write papers that they could present at symposia and conferences at UP and in the region. Thirteen such research projects have been implemented: 7 in the Faculty of Education, one in Agriculture and 5 that are based in the Center for Renewable Energy and Sustainability but reported (by TLP-SP) under Engineering. USAID TLP Partnership Program Performance Evaluation – Final Report 40 At the Faculty of Education, the research clusters have worked on various topics of interest and the staff participating in research clusters have published two papers in scientific journals and others are in progress. However, there is no mechanism to ensure sustainability of these clusters. Interviews with management and staff have revealed that the clusters were operating as long as the U.S. partner was involved and present on the ground to facilitate their meetings. At this point it does not appear that these clusters are sustainable and embedded in the work culture of the institution. The Faculty of Engineering had hoped that the TLP would provide opportunities for joint research projects between them and the U.S. partner. However, this has not happened at the level that the Faculty had hoped for. Five research clusters designed for the Faculty of Engineering were created at the Center for Renewable Energy and Sustainability and provided with seed grants for research, but they have not moved very far along. The other two faculties have not invested much in research methods. At the Faculty of Economy, the visiting faculty introduced a course on Onsite Global Consulting that included components of primary data collection. The Faculty of Agriculture has focused on development of clinics and provision of services as the focus of the partnership, so that there was only one research project at the Faculty. 5.2 Sustainability TLP-SP has helped UP take several actions to help ensure the sustainability of the partnership efforts. Some of them are aimed at implementation of the Action Plan, others to improve student services, and yet others to improving academic performance. These are discussed below. Initiatives to support the implementation of the Action Plan Working closely with the Rector and his team of senior and mid-level management staff, TLP-SP has helped the UP team take ownership of the change and make it sustainable over time. UP management staff, at all levels, provided input into the development of the Action Plan and the accompanying rule books. They were also engaged in the process of deciding the specific areas of TLP-SP support for implementing the Action Plan and making the changes more acceptable. The one-on-one coaching and training for the Rector and the senior management as well as the Leadership training offered to the mid-level management staff has helped create a mass of administrators with a new understanding of their roles and responsibilities and training in leadership and risk-taking. The Rector has worked hard to build a relationship with the Steering Board, and the resulting mutual cooperation has helped bring stability to the university and gather the support needed for change. As pointed out by some key stakeholders, “there has been no hung board” and the university is functioning in a more normal manner. Yet there have been some suggestions from the Rectorate to slow down the pace of change to accommodate the culture of Kosovo. The Rector and his team have done well in leading change so far; how they do after the end of the project remains to be seen. Similarly, the submission to the Steering Board of the planned reforms of the financial systems together with draft regulations is an indicator of the University’s efforts to make the reforms permanent. One of USAID TLP Partnership Program Performance Evaluation – Final Report 41 these reforms, normalization of academic staff salaries, has been approved by the Board and has already been implemented, resulting in a large saving for the university. If the remaining regulations are approved by the Board, they will enable the administrators to follow through with the changes. Similarly, the fact that there is now an HR Manual, which presents all the rules and regulations governing staff employees in one place and which can be shared internally within UP and externally, is an important step in the direction of transparency in the hiring and promotion processes. TLP-SP has helped the university develop an overarching system for data collection, which fills a gap in the way data is currently being collected. In addition, a software program has been developed to query the existing data systems and to mine the data from these systems. Templates have been developed to create periodic reports on administrative data to assist the management with decision-making. However, the accuracy and completeness of the reports generated depends on the data provided by the faculties, and to be sustainable and useful in the long-run there must be a way to ensure that the faculties provide complete and accurate data. Initiatives to improve student services To improve the level of services available to students, UP has revived the Career Center at UP and established an Alumni Association. Both are designed to help improve the employment prospects of UP students through developing networks with alumni, and information about possible opportunities for internships and employment. In addition, the Alumni Association, with funding from the University, is helping organize ‘Orientation Days’ for new students and ‘Graduation Days’ as a public relations exercise for the university. Venture UP, an NGO, has been established with tangential support from TLP-SP to support the students with soft-skills for employment and draws its funds from donor agencies. It will be sustainable to the extent it can continue to raise funds. The Career Center and the Alumni Association will be sustainable only if the University provides management leadership and financial support for them and the students are encouraged to use them. Improvement of academic performance The new/revived centers at the University – such as the Office for Sponsored Research (ORSP), the Center for Teaching Excellence (CTE), and the multidisciplinary Center for Renewable Energy and Sustainability (CES) – are fully functional and have staff and equipment. Importantly, these centers are supported by regulations and budget lines that are integrated in UP’s budget, which reinforces their sustainability over the long run. The research and teaching centers are also supported by regulations that mandate and formalize their use by the staff. For instance, all staff must provide evidence of in￾service training from the CTE or equivalent to be promoted. UP is moving towards a regulation that will ensure that all research conducted by the academic staff at UP goes through the ORSP. Each of these regulations is likely to lead to sustainability of these centers. The academic staff at UP have been exposed to new methods of teaching, including active teaching and group work. With the continued support of in-service training offered by CTE, the use of these new methods is likely to be sustainable. The staff have also been exposed to online courses and are ready to take the next step by rolling out materials online. This must be supported by UP, which must support the academic staff with selection of a single platform that is used by all at UP and by incentivizing the use of online teaching. USAID TLP Partnership Program Performance Evaluation – Final Report 42 The partnerships with the faculties have resulted in a lot of individual learning and capacity building. Regardless of what happens at the University, the knowledge and skills developed by the academic staff through semester-exchange programs and workshops will be sustained over the long-term. However, the likelihood and degree of sustainability of activities and initiatives implemented varies by faculty. Sustainability is perhaps most likely at the Faculty of Economy where, despite the partnership ending two years ago, the professors involved have maintained relationships with their partner institution and are using their newly-acquired skills and course materials. At the urging of the Dean, they have also shared these skills with other academic staff, creating a cadre of staff using active learning in research methods. In addition, the Faculty is planning to submit its new courses for accreditation, which will then become a more regular part of the faculty’s curriculum. Likewise, the degree and likelihood of sustainability is high at Faculty of Agriculture & Veterinary, where the partnerships have resulted in new equipment, refurbishment of equipment and establishment of clinics. The Faculty of Agriculture and Veterinary has also developed, and submitted to the Board, plans for establishment of an institute that will expand the service offered by the clinics. These plans are likely to be sustained with support of UP. The likelihood of sustainability is lower in the other two faculties. At the Faculty of Engineering, while the new teaching methods and course materials are likely to be well-received, they are unlikely to be sustained over the long-term. The Center for Renewable Energy and Sustainability, planned to be a part of the partnership with Engineering but implemented as a standalone center at the Rectorate, needs to rethink its curriculum and fee structure, and since it offers only a certificate rather than a degree, it is unlikely to be sustained in its current form. Likewise, sustainability is difficult at the Faculty of Education, where even though the exchange staff are beginning to use the skills learned in their own teaching, there is no formal mechanism for sharing or support except through training by CTE. Further, while the staff participating in research clusters have published two papers in scientific journals and others are in progress, there is no mechanism to ensure sustainability of these clusters. USAID TLP Partnership Program Performance Evaluation – Final Report 43 6 Conclusions and Recommendations The Partnerships component of the TLP-SP project supported partnerships between University of Prishtina (UP) and universities in the U.S. to upgrade select faculties to provide training, knowledge and skills that meet the demands of modern day and competitive markets. The four selected faculties of UP – Agriculture and Veterinary, Economy, Education, and Computer and Electrical Engineering – were each paired with a U.S. university selected for their expertise in the specific field.. Analysis of the evidence gathered from project records as well as interviews with key informants overwhelmingly indicates that almost all activities planned for the Partnerships component have been completed. 30 Notwithstanding delays caused by changes in the senior management both at UP31 and within the project itself, the stability provided by the current UP management ensured that almost all planned activities were successfully completed during the project period. Overall, everyone interviewed by the evaluation team was unanimous in appreciating the work done by the TLP-SP team. Many of the activities can be deemed to be successful, but their sustainability is likely to be limited. A summary discussion is below. Leadership and change management training as well as one-on-one coaching for senior and mid-level management staff at UP have been well received by the participants and have created leaders with a vision for transforming the university into a well-functioning university with improved academic performance. The training has helped all staff understand the theoretical concepts of leadership and the importance of taking initiative. The training has helped strengthen leadership skills among senior management more than among mid-level staff with more of the senior management, especially the Rector, reporting that the training has helped them think in a more strategic way about implementing their ideas and leading the change. However, the change has not yet affected the top-down leadership culture that exists at UP. Despite the training and a better understanding of the concepts of leadership, the mid-level staff still tend to struggle with taking initiative within their own scope of work. In addition, leadership training has had limited success in facilitating communication between the senior and mid-level management at UP. An example of this is the lack of support from senior management for projects planned by mid-level staff for improvement of UP. During training, all mid-level staff were encouraged to develop ideas for a project designed to help improve UP. Except for two or three such projects that have been implemented (such as the commencement and the orientation ceremonies), others have not received support of the senior management. While the senior management reported being open to new ideas from the mid-level management, the fact that they were not even aware of the various project ideas or the need for approvals is an indication of a gap in communication between the senior and mid-level management. At the Rectorate level, the support provided to the senior management (Rector and Vice Rectors) has 30 The project was awarded as a cooperative agreement, that allows for modification to the broad activities in response to changes on the ground. As a result even though all activities planned in the work plan for the year may have been completed, some of the original activities planned at the beginning of the project may have changed along the way. 31 There were four changes of UP Rector and his team between the start of the TLP-SP project and nomination of the current Rector in 2016 USAID TLP Partnership Program Performance Evaluation – Final Report 44 helped to develop and implement a new strategy, supported by regulations and rule books. This new strategy replaced the UP Action Plan of 2013, which became redundant when the current Rector took over the management of UP in 2016. The Rector has been able to provide stability to the University and work collaboratively with the University Board and Senate to ensure normal functioning of the University over the past three years. The senior management at UP report being empowered to lead implementation of new policies and support new initiatives introduced by TLP-SP, but at the same time caution that change must be slow and within the cultural context of Kosovo to be sustained. With help from the TLP-SP project, the University has improved its image through improved external communications and public relations. It has a new website designed to provide more information to the students and the public. The database of updated listing of publications by UP academic staff developed by the newly created Office for Research at UP has led to a significant improvement in the ranking of UP on the webometrics ranking system. The University has also improved its public relations through increased attention paid to alumni relations, as well as student centered activities. UP has introduced various initiatives that are expected to contribute to revenue generation, with support of TLP-SP Partnerships program. Several regulations, such as those aimed at streamlining salaries for teaching staff, have already been implemented and the resulting savings have been used to pay for several reforms at the university. Other regulations that will result in reform of the financial management at the university, including decentralization to give more autonomy to the faculties in management of their budgets and procurement, are awaiting approval by the UP Board. It is also expected that the multidisciplinary Center for Renewable Energy and Sustainability, the Office for Research (ORSP) and the revitalized Center for Teaching Excellence (revitalized with TLP-SP assistance) will contribute to revenue generation by charging fees for provision of services. While support for these initiatives is almost unanimous, conversations with various stakeholders, including members of the Board, indicated that the decision on decentralization needs more thinking. The centers are also facing some hiccups: the Center for Renewable Energy and Sustainability is not being able to attract new students, and the ORSP is still not at a point where it can support academic staff in writing research proposals or develop proposal budgets. The training provided by the Center for Teaching is well￾received but is still in the process of planning provision of services to external clients and generating revenue. Administrative services and services for students have improved as evidenced by the increased focus on Alumni, establishment of an Alumni Association, student orientation and graduation ceremonies, establishment of Venture UP (an NGO, on UP premises), to name a few. A new database has been designed to collect information from existing systems to assist students with on-time graduation. The Career Center is now equipped with an e-career platform to help students search for employment opportunities and an optical reader to help for faster grading of exam papers. While it is difficult to see the processes and mechanisms that have been put in place for use and sustainability of the Career Center, the increased focus on alumni, and student orientation and graduation ceremonies have been liked enough by all stakeholders to be sustained over the long run. In addition, UP has a new database that will enable it to draw data from the existing data management systems and present it in a user-friendly manner. This is expected to have the benefit of more efficient USAID TLP Partnership Program Performance Evaluation – Final Report 45 management and development of policies supported by data. It will also support students with readily available information to enable them to manage their own study programs. While four faculties – Engineering, Agriculture, Economy and Education – were included in the Partnerships program at the beginning of the project, only three continued to the second phase of the partnerships. The partnership at the Faculty of Economy was not extended as it could not develop a meaningful workplan with its partners. There is some evidence that the faculty-specific partnerships are beginning to bear fruit in terms of use of new methods of teaching and managing classes, modified syllabi and development of new courses – all likely to be sustained over the long run. The faculties that were a part of the program are starting to see academic staff using new ways of teaching and managing classes. The academic staff that were a part of the exchange program brought back new syllabi and curricula for new courses as well as textbooks in addition to the skills learned through observation of teaching by their American colleagues. Several of the academic staff that had gone to the U.S. on exchange visits reported taking almost two years (two academic cycles) to modify their existing syllabi to incorporate new techniques such as continuous assessment and active learning methods into their teaching. The exchange staff also reported developing new courses to be included in the next round of faculty accreditation process (faculties of Engineering and Economy). While none of the faculties mandated the sharing of learning and experience in a formal way, academic staff reported sharing their learning with colleagues informally. The most surprising finding was from the Faculty of Economy. The partnership at the Faculty of Economy was not extended beyond the first phase due to a lack of agreement between the partners on a meaningful scope of work, this was the only faculty where the Dean mandated the exchange academic staff to share its experiences with colleagues through brown bags. As a result, the Faculty now has a cadre of academic staff that are using active learning in their classrooms and have developed new courses based on materials obtained from the exchange visit for accreditation. Through faculty-specific partnerships, the project has been successful in supporting efforts to introduce a better research culture in the participating faculties. At the Faculty of Education, research clusters were created to support the academic staff to collaborate on writing papers. Similar clusters were also created at the Center for Renewable Energy and Sustainability. Interviews with management and staff shows that the clusters were working well with the support of the U.S. partner, but at this point the clusters are not embedded in the work culture of the institution and there are no concrete plans for their sustainability beyond the life of the project. Both the faculties of Education and Engineering received access to academic journals through the Partnerships program. This access has ended with the end of the partnerships/project and there are no plans at this point to fill the gap. The Faculty of Engineering has received some support from the partnerships but not to the extent hoped for by the Dean of the faculty. While the establishment of the multidisciplinary Center for Renewable Energy and Sustainability has been hailed as a success for UP and a first in the region, it has come at a high cost for the Faculty of Engineering. The Center has been established at the Central level, not within the faculty of Engineering. Most of the funds available for partnership activities for the faculty of Engineering were diverted to the center with the result that the activities slowed down at the faculty USAID TLP Partnership Program Performance Evaluation – Final Report 46 and the hoped for joint research projects did not materialize.3233 The Faculty of Agriculture has had a successful partnership, with faculty from the U.S. partner holding workshops on teaching methods and offering support on the repair and refurbishing of existing equipment. The workshops were attended by academic staff and graduate students and were well￾received. Existing equipment that was unusable has been repaired and put in use with support in terms of new parts and technical knowhow from the partner University. The academic staff in the exchange program saw a lot of learning from the field visits in the U.S. The faculty has, as per its plan with the TLP-SP project, established an animal clinic that is offering services to the University and the public (perhaps also earn some revenue in the future through fees for services provided). The clinic will eventually be housed within an institute at the Faculty of Agriculture, plans for which are already in place. Both the clinic and the institute have dedicated funding and full support of UP management and are likely to be sustained. Similarly, the partnership at the Faculty of Education has been successful. All of the exchange staff brought back new syllabi and curricula for new courses as well as textbooks. The academic staff are beginning to use the skills learned and incorporating new materials into their exiting syllabi. They are also developing new courses for accreditation. The faculty is now using a book on research methods that was translated into Albanian providing additional resources for the students. As a result of the partnership there have been efforts at getting the academic staff to conduct research, but these efforts have met with limited success. The revival of the Center for Teaching Excellence (CTE), supported by the regulations mandating all academic staff attend at least two professional development courses as a requirement for promotion, is helping support improvement of quality of teaching at the University. With assistance of the TLP-SP project, CTE is now functional, with a fully-furnished training room and office for the coordinator and is providing training to UP teaching staff. Over a third of the staff has already been trained. In addition to the existing trainers at UP, a cohort of 12 new trainers has been trained and certified and will begin providing training in the next academic year. The Center has a budget line in the UP budget which, given the requirement of training for promotion, is likely to be sustained over the long run. However, there is tension between the senior and junior trainers at CTE which must be resolved to ensure sustainability. Last but not the least is the introduction of CourseSites to UP. This is a free platform for the faculty to put their courses online and incorporates a lot of the technological advances. However, this conflicts with SEMS, a grading and online course system already in use by UP. Academic staff across all faculties are hesitant to use the new system until such time the university has decided which platform it will support. Interviews with academic staff suggested that the academic staff are willing to put in the time and effort to learn the new system and to create online content, but would like for the university to formally adopt one or the other system, even if it is not mandatory for all staff to use it, so that their efforts to create an online course are not wasted. 32 The funds diverted from the faculty of Engineering to The Center for Energy Sustainability was as per the terms of the subcontract. 33 A review of TLP-SP databases indicated that all research grants/ projects conducted at the Center for Energy and Sustainability have been reported under the Faculty of Engineering. USAID TLP Partnership Program Performance Evaluation – Final Report 47 Data collected through interviews and focus groups with various stakeholders and beneficiaries of the Partnerships component indicate that the TLP-SP project is beginning to have an impact on the UP and bring about a change to its landscape. There are some administrative changes in place for more efficient management of the University and ensure its stability; initiatives have been launched to collect better and more reliable data to support the university management with more efficient management and data￾based policy making, improve communication and the image of the university; as well as help improve the quality of teaching through better training for the staff, and input of new resources for teaching materials and research. The addition of three new centers – Center for Sponsored Research, Center for Energy and Sustainability and the UP Alumni Association – and the revival of the Center for Teaching Excellence have added to the resources available to the University to support university-wide improvements in the quality of teaching and learning. However, in the faculties there has been more individual learning than faculty-wide learning. Change across the University is slow and there is some evidence of impact that the university is moving towards better quality of education to meet the demands of the labor market. Many of these changes have been institutionalized or are in the process of institutionalization through proposed changes to the university rules and regulations, but questions about the sustainability of some of these initiatives remain. Recommendations Based on the discussion above, a few recommendations are offered for UP and for future donor support to UP. First, there are several concurrent university-wide activities that have been initiated under the TLP Partnerships program. Many of these activities were led by consultants and staff paid for by the TLP-SP project but are now led by UP staff. It is not clear if these staff have been sufficiently trained to carry out these activities, or if the University has the financial resources to ensure long-term sustainability of these arrangements. This is especially true for the newly formed multi-disciplinary Center for Energy and Sustainability/ We recommend that the university make a realistic assessment of the resources needed, in terms of additional training, staff or budgets, to ensure sustainability of these activities. Second, sustainability was built into the initiatives implemented under the University partnerships component through rule books and regulations. However, these regulations had not been approved at the time of this evaluation. In addition, TLP-SP has helped UP develop a sustainability plan for each of the initiatives after the end of the project. Now that the project has ended, to ensure sustainability of the initiatives, UP management must take ownership of these plans by including them in their management plan. This will ensure these plans are revisited periodically and updated to ensure their validity . . There is also a need for the leadership to pursue the approval of these regulations by the UP Board to ensure sustainability of these initiatives. Third, the leadership management training offered to mid-level staff may have been at a level higher than what was needed. It is important to develop follow-on training to continue the process of developing skills and communications. There is also a need for training that targets the communication between USAID TLP Partnership Program Performance Evaluation – Final Report 48 staff at various levels of seniority within UP management. Fourth, while the Deans of the faculty and a few senior members of the faculty management team were aware of the accomplishments of the academic staff who have benefitted from the semester-in-residence exchange program, there are no established means for beneficiaries to share their experiences. It is important that academic staff have opportunities and access to systems that facilitate the sharing of experiences with colleagues as a form of informal training. Fifth, the Faculties do not have any means of tracking whether the returned staff have met the requirements of the program and developed the required syllabi, participated in lectures, or even whether they are using their new learning. This type of tracking is needed to monitor how the TLP-SP program has benefited the Faculties. Sixth, we recommend including research as a part of the University’s mission. Research clusters have been established in participating faculties as part of the faculty-specific partnerships. These clusters must be connected with the Office for Research and Sponsored Projects, which would help both, the members of the cluster to further their research as well as the Center by getting more people using its facilities. Including research as a part of the University’s mission will ensure sustainability of the center as well as improvement in the quality of research programs at the university. Seventh, there is some continuing tension at the Center for Teaching Excellence between the senior trainers and the junior trainers on the training to be provided to academic staff and the methods of teaching and assessment. It is important to resolve this tension so that the academic staff can benefit from the expertise in new methods offered by the junior trainers and help make UP a world class university. We recommend UP provide exposure for the senior trainers to the active learning methods currently in use in the U.S., so that they are more encouraging of the use of those methods by the junior trainers. Another possibility is to pair junior trainers with senior trainers and, even though it will upset the prevalent practice at UP, have the junior trainers lead the training. Eighth, many activities under the Partnerships program support the Grand Challenges in the labor market as identified by the University and the Advisory Committee at the beginning of the program, but this has not been clearly communicated to all stakeholders. As a result, not everybody buys into the intent of this initiative to provide training, knowledge and skills to Kosovars that meet the demands of competitive markets. Failure or delay in doing so may compromise the efforts of the University to prepare its students for the workforce. Ninth, the students graduating from UP lack practical experience and as a result are not as competitive in the current labor market. This practical experience can be gained through course work that includes problem solving, or through internships and practicums. We recommend the University examine its program delivery culture to include ways and means of helping students gain practical experience. There may also be a space to bring in expert assistance to help redesign these courses to include a practical component. USAID TLP Partnership Program Performance Evaluation – Final Report 49 Tenth, the labor market requires students to be well versed in more than one discipline. This is especially true in the science fields. The newly established multi-disciplinary center for Energy and Sustainability is a first step in this direction. However more effort is needed in this regard and it would be good for UP examine/revise its programs and policies to enable students to take credit-bearing courses in other disciplines as a part of their graduation requirements. Eleven, there is a strong demand for decentralization from the faculties within UP. While TLP-SP made some small attempts at helping UP in this direction, there is a need for a larger initiative supporting governance reform at UP. Twelve, UP has several different ongoing donor-funded programs. It is important to ensure that these programs build on each other or are supportive of ongoing initiatives, so as to ensure there is no duplication of efforts. In addition, there is a need to bring examples of administration, program management and training programs at the various centers from a mix of U.S. and European universities so that UP can choose the most appropriate given the context of Kosovo. And finally, there is a need to slow down the pace of training and recommendations for change provided to allow the senior management time to absorb and internalize all the recommended activities and implement them well. USAID TLP Partnership Program Performance Evaluation – Final Report 50 7 Annexes Annex1: Scope of Work for the Evaluation USAID TLP Partnership Program Performance Evaluation – Final Report 51 USAID TLP Partnership Program Performance Evaluation – Final Report 52 USAID TLP Partnership Program Performance Evaluation – Final Report 53 USAID TLP Partnership Program Performance Evaluation – Final Report 54 USAID TLP Partnership Program Performance Evaluation – Final Report 55 USAID TLP Partnership Program Performance Evaluation – Final Report 56 USAID TLP Partnership Program Performance Evaluation – Final Report 57 USAID TLP Partnership Program Performance Evaluation – Final Report 58 USAID TLP Partnership Program Performance Evaluation – Final Report 59 Annex 2: Evaluation Methodology The Performance Evaluation of the TLP-SP University Partnerships (Partnerships) program was designed to follow and assess the progress and achievements of the Partnerships component. Data for the evaluation were collected at various points along the study on project indicators and milestones. Baseline data was collected at the start of the Partnerships component in 2015. The midline study was conducted in 2017, at the end of the first two years of activities (Phase 1, 2015-2017) and prior to the extension of activities into Phase 2 (2017-2019). This was a follow-on study, designed to assess progress on planned activities since the beginning of the component two years past the start of the component. An endline study was conducted near the end of the life of the TLP-SP project and assessed the cumulative progress on planned activities from the beginning of the component to date (May 2019). The endline study followed the same approach as that for the baseline and midline studies. The approach is described below. The premise of TLP-SP University Partnerships component is that in order to reduce unemployment rates in Kosovo, particularly among the youth, there is an urgent need to improve the quality of education to ensure a system that is able to provide adequate learning opportunities as well as skills required by the labor markets. To do so, there is a need to improve the management systems in institutions of higher education (universities), provide teachers with training to improve the quality of teaching and pedagogy, as well as enhance the resources available to the students. USAID is supporting the Ministry of Education in their efforts to reform the higher education system in Kosovo. The theoretical construct underlying the University Partnerships component of the TLP-SP project is that, as a result of their participation in the TLP-SP program, UP and select faculties in UP and at universities in Kosovo will have improved University management, pedagogy, quality of teaching methodology and curricula, and strengthened research development. They will also have updated technology systems. The performance evaluation of the TLP-SP Partnerships program follows the logic of the project. Approach The Performance evaluation of the TLP-SP Partnerships component was conducted in three stages – Baseline study conducted at the beginning of the partnerships component, followed by midterm evaluation of the component at a point two years later, and a final assessment conducted in the last year of the TLP-SP project to see how the TLP-SP Partnerships program has affected the UP. In addition, interim data was collected annually to assess progress on milestones through surveys of faculty and students, focus groups and key person interviews as needed. Each interim report was a standalone report, so there may have been some repetition from year to year. The final report consolidates and synthesizes all data to report progress on this component. Using a modified Human and Institutional Capacity Development (HICD) approach, the evaluation was conducted at the level of the organization (UP and select faculties) and at the individual level (academic staff and student levels) to provide a holistic view of the partnerships and the resultant changes in the University. Data were collected on the implementation of the UP’s Action Plan, the progress of the partnerships and the outcomes of the targeted assistance and any changes in learning outcomes. Key USAID TLP Partnership Program Performance Evaluation – Final Report 60 metrics were tracked over the life of the project to see long-term changes in all of the indicators. Data were collected using primary and secondary sources and analyzed using descriptive analysis. Each of these is described in greater detail below. The approach used for the endline study is consistent with the overall approach for the performance evaluation of the TLP-SP partnerships component and provides data on progress on activities planned under the Partnerships component in its final year. This study was conducted by a team of consultants that included two international expert and two local consultants, one of whom is a faculty member from the UP. He was included for his expertise in evaluation of higher education programs, understanding and insight into the university context and access to the university. 1. Selection of Indicators A number of qualitative and quantitative indicators were selected for monitoring the progress as well as assessing the impact of the proposed initiatives under the Partnerships program. These indicators were selected based on the criteria that they provide information that is useful in measuring progress effectiveness in the most efficient way. In addition, the indicators selected should themselves be “monitorable,” meaning they can be independently validated or verified. Evaluation questions and Sub questions Indicators Source of data Note: To the extent possible, all individual level data – on beneficiaries of the program – were disaggregated by gender, minority status, disability status, and LGBT status. ) 1. Has the UP’s Action Plan been effectively implemented? Have tangible measures been adopted that show the University’s commitment to change management? How has the TLP University Partnerships component changed the University of Prishtina’s administrative performance? Has the University of Prishtina’s Action Plan been effectively implemented? ● Implementation of each of the recommendations listed in the 2013 UP Action Plan included in the report by Morgan Borszcz Consulting (MBC). (This indicator has since changed to include the current Strategic plan which owes its origins to the plan of 2013, but is somewhat different.) Baseline data from: MBC report; Key person interviews with UP administrators, MEST and TLP-SP staff; Review of assessment of implementation of UP Action plan by TLP-SP project. Have tangible measures been adopted that show the University’s commitment to change management? ● Revenue generated, ● The degree of compliance with course registration guidelines, ● The percentage of courses accredited by the Kosovo Accreditation Agency ● Annual student course satisfaction surveys conducted ● The implementation of formula-driven admissibility ● Tracking system in place to track students while in UP ● System for tracking students post completion of baccalaureate for type of jobs obtained, time to find a position, etc. ● Faculty survey ● Student survey ● Key person interviews ● Focus groups ● Desk review of TLP-SP quarterly, annual reports and other technical reports and records ● BIRN student satisfaction survey USAID TLP Partnership Program Performance Evaluation – Final Report 61 Evaluation questions and Sub questions Indicators Source of data How has the TLP￾Partnerships component changed University of Prishtina’s administrative performance? ● Financial viability ● Percentage increase in operating capital coming from student tuition ● Improved facilities ● Improved infrastructure, equipment ● Improved outreach ● Internal communication ● Faculty survey ● Key person Interviews ● Student focus groups ● Student Satisfaction survey ● Desk review of TLP-SP quarterly, annual reports and other technical reports and records 2. How has the TLP University Partnership component changed academic performance in targeted faculties over the course of the project? Have the “Grand Challenges” identified for each partnership been effectively implemented? How has academic performance in targeted faculties changed over the course of the project Reviewed at three levels – institutional, academic staff and student levels. See below : Institutional level outcomes ●Number of students enrolled (by gender, minority status, disability status, LGBT status) ● Drop-out and retention rates ● Graduation rates ● Number of Degrees awarded ● Credits to degree ● Time to degree ● Reductions in faculty-student ratios ● Degree of satisfaction of students determined by course evaluations ● A reduction in failing grades ● Increases in faculty registering for professional development courses ● Number of students enrolled in for-credit internship programs ● Number of partnerships with external organizations offering externship placements ● New technology and other resources made available ● Faculty survey ● Student survey ● Key person interviews ● Focus groups ● Desk review of TLP-SP quarterly, annual reports, and other technical reports and record Academic staff level outcomes ● Number of courses taught ● Number of new course developed ● Number of dissertations supervised ● Number of students advised ● Professional growth ● Participation on University/department committees ● Participation in curriculum development, ● Regular office hours ● Coming to class on time ● Research ● Increases in publications by faculty in peer￾reviewed journals ● Internal communication ● Faculty survey ● Student survey ● Key person interviews ● Desk review of TLP-SP quarterly, annual reports and other technical reports and records Student level outcomes ● Number of students attending courses, ● Number of students completing course ● Number of students enrolling in for-credit ● Faculty survey ● Key person Interviews ● Student focus groups USAID TLP Partnership Program Performance Evaluation – Final Report 62 Evaluation questions and Sub questions Indicators Source of data internship programs, ● Time for students to graduate ● Credits to degree ● Number of students entering graduate programs ● Employment rates for graduates ● Type of jobs obtained ● Time to employment ● Average and median salaries ● Student survey ● Desk review of TLP-SP quarterly, annual reports and other technical reports and records Grand Challenges identified ● Grand Challenges are well publicized ● Partnership activities link back to the grand challenges ● Interviews with faculty and management ● Desk review of TLP-SP quarterly, annual reports and other technical reports and records 3. Have the selected faculties been able to utilize, maintain and build up new technology, systems, knowledge and skills? What actions have been taken and/or mechanisms put in place to help ensure the sustainability of the partnership efforts? Utilize, maintain and build up new technology, systems, knowledge and skills? ● New technology received/purchased maintained locally and in use ● Use of new Knowledge and skills ● Systems in place for dissemination of Knowledge and skills ● Collaborations formed and maintained ● Interviews with faculty and management ● Desk review of TLP-SP quarterly, annual reports and other technical reports and records Actions have been taken and/or mechanisms put in place to help ensure the sustainability of the partnership efforts? ● Systems in place for maintenance and updating of new technology ● Oversight body in place to ensure sustainability of new interventions ● Budget lines for maintenance of new technology ● Budget line for continued professional development ● Support for academic staff to try out new skills ● Interviews with faculty and management ● Desk review of TLP-SP quarterly, annual reports and other technical reports and records 2. Data Collection and Analysis Data on the above indicators was collected from a variety of primary and secondary sources. Primary data was collected over two, three-week periods in Kosovo through a set of key informant interviews, and surveys with stakeholders and beneficiaries of the program to understand the implementation of the program as well as any interim impacts. This included a survey of UP faculties participating in the program for the collection of administrative data and a survey of beneficiaries from the select faculties to assess their satisfaction with the quality of exchange programs, resources available and their use of the training. Secondary data was collected from a review of documents that informed the implementation of the project as well as TLP-SP project records and reports. These are described below. Administrative records and reports: Working in collaboration with World Learning (WL), secondary data was collected from databases already in place for TLP-SP project monitoring and evaluation (M&E) and from a review of project records. Project-generated documents and reports were reviewed to USAID TLP Partnership Program Performance Evaluation – Final Report 63 assess progress of the project towards pre-determined activities and tasks as well as to understand any changes that may have led the project to change direction. The documents reviewed include: (i) Quarterly and Annual Progress Reports submitted to USAID; (ii) annual project work plans; (iii) TLP Results Framework and Performance Monitoring Plan (PMP); (iv) reports of any studies conducted by TLP-SP to inform the project of the current status of implementation of the Rector’s action plan at the UP; (v) project contract and subcontracts; and (vi) success stories and highlights. A full list is included in Annex 4. Key informant interviews: Semi-structured interviews were held with key stakeholders and beneficiaries of the project to understand their perspective on progress of the University Partnerships component of the TLP project, any challenges to implementation, interim outcomes and impacts, as well as recommendations for the following year of the program. Interviews and informational meetings were conducted with the following groups: (i) the TLP-SP Chief of Party, Deputy Chief of Party and other members of the TLP-SP team (including the U.S. based University partners) implementing the project; (ii) senior officials from the relevant departments of Ministry of Education, Science and Technology at the central level; (iii) Rector and vice-Rectors at UP, administrators from UP Rector’s office and academic and non-academic staff from each of the four faculties participating in the University Partnerships program; and (iv) members of the Advisory Committee. Surveys of academic faculties at UP: Instruments were designed to collect administrative data from each of the four academic faculties participating in the partnership program on indices such as student enrollment, number of academic and non-academic staff, number of courses taught, research publications, etc. The purpose of these surveys was two-fold: (i) to obtain the most current data available; and (ii) assess the ease with which the faculties could collect and provide the information. The surveys were submitted to the Dean and completed by either the TLP University Partnerships Program liaison or someone assigned to this task from the Dean’s office. Surveys, focus groups and interviews with beneficiaries of the Partnerships program: Data from beneficiaries of the program were collected through email or web-based surveys, focus groups and interviews conducted with academic staff from the four faculties participating in the Partnerships program and officials from UP Rector’s Office, who have benefitted from the various activities conducted under the TLP project, to understand their perspective on the quality of the activity, use of the training received and any areas of concern. The activities under the TLP project included leadership training program for midlevel and senior administrators and management, exchange programs for the academic staff and the administrators, workshops and trainings organized by U.S. partners at the participating faculties or through the Centers for Teaching Excellence, Office for Research and Sponsored Projects, etc. The interviews and focus groups were conducted in Albanian or English, based on the composition of the group and their comfort level with speaking in English. Data collection instruments: Instruments, such as check lists and semi-structured interview and focus group guides, were developed, as appropriate, for the various sources of data and each group of respondents listed above. The interview guides were developed in English and translated into Albanian as necessary. USAID TLP Partnership Program Performance Evaluation – Final Report 64 Data analysis: All data collected was entered into electronic databases (Excel, SPSS) and analyzed using descriptive univariate analysis, which involved the computation of frequencies and cross tabulations. Data has been disaggregated, as appropriate, by gender and academic faculty. Data from focus groups and interviews was analyzed for themes and patterns. 3. Caveats and Limitations There are a few caveats and limitations that must be kept in mind as we review the results from this study. First, the performance evaluation was conducted over a period of four years (2015 to 2019), and even though data were collected annually to reduce the bias due to recall of activities over time, it is possible that effects of some of the initiatives undertaken were not reported fully or were missed. Second, access to the faculty administrative data, such as student enrolment, number of staff, number of publications, etc., was a challenge. The evaluation team had to go through the Deans to get access to the person in-charge which, while not a problem per se, became an issue due to the busy schedule of the Deans and an unwillingness on their part to delegate the completion of the survey. Obtaining administrative data from faculties participating in the University Partnerships program has also been a challenge, with two of the faculties being unable/unwilling to provide complete data for the most recent year. Similarly, the survey of students was a challenge, especially since two faculties (Economics and Education) were unwilling to allow their students to take the survey at endline. And finally, the TLP-SP project was implemented under a co-operative agreement between USAID and World Learning. Cooperative agreements are designed to be agreements that enable the implementing partners to change course to accommodate changes in the field with an agreement from USAID. Changes made to specific tasks under the broad activities described in the overall scope of work are not always documented through contract modifications, and to that extent may have been missed and not featured in the progress updates. . USAID TLP Partnership Program Performance Evaluation – Final Report 65 Annex 3: Informational Meetings and Interviews with Key Stakeholders (Informational meetings and Interviews conducted with key stakeholders in May 2019 USAID staff 1. Mrs. Melita Cacaj, Monitoring and Evaluation Specialist, COR TLP-SP evaluation project, PPO, E￾mail: mcacaj@usaid.gov 2. Mrs. Merita Teliqi, Project Specialist USAID,, AOR TLP-SP, E-mail mteliqi@usaid.gov 3. Mr. William Trigg, USAID Economic Growth, E-mail: wtrigg@usaid.gov 4. Ms. Antigona Mustafa, USAID Education, Email: amustafa@usaid.gov WL staff – TLP-SP Project 1. Mr. Carl Hammerdorfer, Chief of Party, TLP-SP, E-mail: carl.hammerdorfer@worldlearning.org 2. Mr. Fatmir Salihu, Program Officer, TLP-SP, E-mail: fatmir.salihu@worldlearning.org 3. Dr. Ed Cooper, Consultant, TLP-SP, E-mail: dredcooper@gmail.com Ministry of Education Science and Technology 1. Mrs. Drita Kadriu, Head of Higher Education Department, E-mail: drita.kadriu@rks-gov.net 2. Mrs. Qamile Sinanaj, Higher Education Department, E-mail: qamile.sinanaj@rks-gov.net 3. Ms. Shpresa Mehmeti, Officer of the Division for International Cooperation in Higher Education and Science 4. Mr. Baton Dushi, Chief of Cabinet/Political Advisor to Minister of Education Rectorate - University of Prishtina 1. Mr. Marjan Dema, Rector of the University of Prishtina, E-mail: marjan.dema@uni-pr.edu 2. Mr. Hysen Bytyqi, Vice-Rector for Teaching and Quality Assurance, E-mail: hysen.bytyqi@uni￾pr.edu 3. Mrs. Merita Kocinaj Berisha, Vice-Rector International Cooperation, E-mail:merita.berisha@uni￾pr.edu 4. Mrs. Myrvete Badivuku, Vice-Rector on Finance, E-mail: myrvete.badivuku@uni-pr.edu 5. Mr. Faton Berisha, Vice-Rector for Scientific Research, E-mail: faton.berisha@uni-pr.edu Center for Renewable Energy and Sustainability 1. Mr. Naser Sahiti, Head of Center for Renewable Energy and Sustainability, E-mail: naser.sahiti@uni￾pr.edu Center for Teaching Excellence, University of Prishtina 1. Mrs. Shykrane Germizaj, Head of Center for Teaching Excellence, E-mail: fatlume.berisha@uni￾pr.edu Office for Sponsored Research and Projects, University of Prishtina 1. Mr. Gentrit Berisha, Head/Coordinator of Office for Sponsored Research and Projects, E-mail: gentrit.berisha@uni-pr.edu UP Alumni Office & Data Management System 1. Mr. Besnik Loxha, Director of the office for Academic Development, Email: besnik.loxha@uni￾pr.edu 2. Mr. Faton – Acting IT Director Venture UP 1. Mr. Fis Malesori, CEO, Venture UP, E-mail: fismalesori@gmail.com USAID TLP Partnership Program Performance Evaluation – Final Report 66 Faculty of Agriculture 1. Mr. Skender Kaciu, Dean of Faculty of Agriculture, E-mail: skender.kaciu@uni-pr.edu 2. Head of Teaching, Faculty of Agriculture 3. Head of Technology, Faculty of Agriculture, Faculty of Electrical and Computer Engineering 1. Mr. Enver Hamiti, Dean of Faculty of Electrical and Computer Engineering, E-mail: enver.hamiti@uni-pr.edu 2. Mr. Gazmend Pula, Prof. Assoc. Dr., FECE, Faculty of Engineering, E-mail: gazmend.pula@gmail.com Faculty of Education 1. Mr. Ethem Çeku, Dean of Faculty of Education, E-mail: ethem.ceku@uni-pr.edu Citizen Corps Program 1. Mrs. Laura Kryeziu, Executive Director, Leadership and Growth Council, implementer TLP Citizen Corps Program, Email: laura.kryeziu@lgc-ks.org Members of UP Board 1. Mrs. Laura Kryeziu, Executive Director, Leadership and Growth Council, Member Advisory Committee, Email: laura.kryeziu@lgc-ks.org 2. Mr. Zeqir Veselaj, Associate Professor, Fundamentals of Natural Sciences with Methodology, Biodiversity and Botany, Faculty of Education, E-mail: zeqir.veselaj@uni-pr.edu Advisory Committee (interviews conducted in Nov 2018) 1. Mr. Xhavit Rexhaj, Member Advisory Committee, E-mail: xhavitr@hotmail.com 2. Mr. Fetah Podvorica, Member Advisory Committee, E-mail: fetah.podvorica@uni-pr.edu 3. Mrs. Sharon Hart, Member Advisory Committee and President of Rochester Institute of Technology – Kosovo (RIT Kosovo) , E-mail: shart@auk.org 4. Mrs. Merita Teliqi, Project Specialist USAID, Member Advisory Committee, E-mail mteliqi@usaid.gov 5. Mrs. Laura Kryeziu, Executive DiRector, Leadership and Growth Council, Member Advisory Committee, Email: laura.kryeziu@lgc-ks.org Focus groups conducted (list of participants not included to ensure confidentiality) 1. Focus groups with UP teaching staff – TLP-SP Academic Exchanges 2. Focus group with Leadership and Change Management Training participants 3. Focus group with new Trainers – UP Center for Teaching Excellence 4. Focus group with participants of trainings at UP Center for Teaching Excellence USAID TLP Partnership Program Performance Evaluation – Final Report 67 Annex 4: Sources of Data and List of Documents Reviewed TLP-SP Project Reports and Documents (2014-2015, 2015-2016, 2016-17, 2017-18, 2018-19): i. TLP-SP Technical Proposal ii. TLP-SP Work Plan – Years 1, 2, 3, 4 and 5 iii. TLP-SP M&E Plan iv. Sub-awards for Partner Universities v. TLP-SP Annual and Quarterly reports vi. UP University Wide Activities Plan – revised July 6 vii. Transformational Leadership Program (TLP-SP, 2014). Assessment – Implementation of UP Action Plan Milestones Final Report. Document for Internal use only. 2015 viii. Transformational Leadership Program – Scholarships and Partnerships Advisory Committee Strategy (Sep 2016) ix. Lists of participants in faculty exchanges; certifications by Center for Teaching Excellence; participants of change management programs, etc. x. Reports and white papers developed as a part of the project implementation xi. Success stories from the TLP-SP project Kosovo Ministry of Education, Science and Technology (2016). Kosovo Education Strategic Plan 2017- 2021 accessed at http://www.kryeministriks.net/repository/docs/Kosovo_Education_Strategic_Plan. Pdf on August 25, 2019 Odell, Malcolm., et al (2004). Human and Institutional Capacity Development Assessment: Kosovo/ Final Report. Submitted to USAID 2004. TANJUG, B92, News article, December 3, 2018 found in Wikipedia at https://en.wikipedia.org/wiki/ Kosovo, Accessed on Sep 1, 2019 The World Bank Group (2016): Kosovo – Country Snapshot. Accessed on February 27, 2017 at http:// pubdocs.worldbank.org/en/739371475757000674/Kosovo-Snapshot-Oct2016FINAL.pdf The World Bank Group (2016): Kosovo – Country Program Snapshot. Accessed on February 27, 2017 at http://pubdocs.worldbank.org/en/419461462386476530/World-Bank-Kosovo-Program-Snapshot￾April-2016.pdf The World Bank Group (2017): Kosovo – Country Snapshot. Accessed on January 5, 2018 at http:// pubdocs.worldbank.org/en/912841507748392027/Kosovo-Snapshot-Fall2017.pdf The World Bank Group (2017). The World Bank in Kosovo – Overview. Accessed on January 5. 2018 at http://www.worldbank.org/en/country/kosovo/overview accessed on January 5. 2018 The World Bank Group (2017): FY17–-21 Country Partnership Framework for Republic of Kosovo. Report No. 112337-XK. http://documents.worldbank.org/curated/en/297951496160148830/Kosovo￾Country-partnership-framework-for-the-period-FY17-FY21 accessed on August 25, 2019. USAID TLP Partnership Program Performance Evaluation – Final Report 68 The World Bank group (2018). Country Context. https://www.worldbank.org/en/country/kosovo/ overview University of Prishtina, (2016) Strategy and Action Plan. Prishtina December 2016 USAID (2015). Transformational Leadership Program. Needs Assessment Report – University of Prishtina Rectorate and Four Faculties Prepared for USAID Cooperative Agreement No. AID-167-A-14- 00002 February 25, 2015 USAID TLP Performance Evaluation of the University Partnerships Program: Baseline Study. January 2016. Conducted by IBI International. USAID TLP Partnership Program Performance Evaluation – Final Report 69 Annex 5: Description of University Partnerships 34 Faculty-specific partnerships under TLP-SP were launched as planned on April 23, 2015, between four UP faculties and the U.S. partner universities as listed in Table 1. In addition, university-wide cross￾cutting activities were implemented at the level of the Rectorate with the assistance of University of Arizona (one of the TLP-SP partners). Table 1: Faculty partnerships Participating Faculty and Departments from University of Prishtina U.S. Partner University and its participating schools and colleges Agriculture and Veterinary Includes Department of Veterinary Medicine, Department of Food Technology and Department of Plant Protection University of Minnesota ● The College of Veterinary Medicine, Center for Animal Health and Food Safety (CAHFS), ● College of Food, Agriculture and Natural Resource Sciences, Department of Food Science and Nutrition (FScN), ● Humphrey School of Public Affairs, and ● Extension Program at the University of Minnesota (UMN) Phases 1 and 2 completed Economy All departments US Exchange visits extended to teaching staff from Faculty of Philosophy associated with the UP Center for Gender Studies and Research Dartmouth College ● Dartmouth’s Tuck School of Business ● Dartmouth Provost’s Office, ● The Gender Research Institute at Dartmouth (GRID), and a ● Core group of faculty from other Dartmouth departments Phase 1 completed (no cost six-month extension to enable completion of Phase 1 activities) Partnership not extended to Phase 2 Education All departments Indiana University ● Center for International Education, Development and Research (CIEDR) within the Department of Educational Leadership & Policy Studies; and ● Department of Curriculum & Instruction, at Indiana University Phases 1 and 2 completed Computer and Electrical Engineering (Services extended to include Department of Physics and other natural science departments at UP) Arizona State University ● The Melikian Center at ASU (organize all logistics) ● ASU School of Electrical, Computer, and Energy Engineering and ● The Global Institute of Sustainability at ASU. Phases 1 and 2 completed University wide Cross-cutting activities Lead – Arizona State University Same as above Value of subgrant: see above Phases 1 and 2 completed Prior to the launch of the Partnerships component, TLP-SP, in collaboration with the management at UP and USAID, identified four UP faculties for inclusion in the program. The faculties – (i) Agriculture and 34 Drawn from USAID/Kosovo (2016). Transformational Leadership Impact and Performance Evaluation Project: Performance Evaluation of The University Partnerships Program. Baseline Study January 2016. Conducted by IBI International. USAID TLP Partnership Program Performance Evaluation – Final Report 70 Veterinary, (ii) Economy, (iii) Education, and (iv) Computer and Electrical Engineering – were selected on the basis of UP Needs Assessment Study35 and demonstration of willingness/commitment to participate in the Partnerships program by the Faculty.36 The goal of these Partnerships was to enhance capacity of the management and teaching staff within UP and the selected faculties to meet the challenges of 21st century student-centered teaching and applied research. As such each of the four selected faculties was partnered with a U.S. university that had the resources and the capacity to provide support to the faculty. (see Table 1) The Partnerships focused only on UP and its selected faculties. The decision to work only with UP, rather than with multiple universities in Kosovo (as planned at the time of inception of the project), was made with USAID concurrence. This was based on the thinking that (i) UP is the single largest university in the country that serves over 50 percent of the students in higher education, and (ii) the teaching staff at UP also teach at other Universities in the country, and thus would carry any new knowledge with them to these universities. All departments within the faculties of Education, Economy and Electrical and Computer Engineering were included in the Partnerships. The Faculty of Agriculture expanded its participation from one department (Veterinary Medicine) and added two more (Department of Food Technology with Biotechnology and Department of Plant Protection), enabling a larger pool of participants to benefit from the exchange and mentoring activities of the University Partnerships component. Similarly, the partnership with the faculty of Economy was expanded to include teaching staff from the Department of Philosophy (Center for Gender Studies and Research) in the exchange visit (to U.S.) program. The Partnerships focused on three principal areas: (i) curricular development; (ii) teaching methodology; and (iii) research enhancement. Based on the needs and priorities of individual faculties, each partnership selected activities from a larger menu that included: ● Development of curricula and courses; ● Mentoring relationships in joint applied and basic research between the U.S. and UP partners; ● E-learning course management and delivery; ● Faculty exchanges involving younger UP faculty and assistants; ● Co-teaching courses between U.S. and UP faculty that can employ blended learning, utilizing the e-learning course management platform; ● Internships and service learning from the faculties developed in cooperation with Career Counselling Center; and ● Provision of essential and necessary resources, e.g., computers, software, and textbooks. The activities, as prioritized by the UP faculty in the partnership, were implemented over two phases covering a period of four years; Phase 1 extending from 2015-2017 and Phase 2 from 2017-2019. All four faculties were included in Phase 1, however only three (Education, Agriculture and Veterinary, and Engineering) continued on to Phase II. The partnership between faculty of Economy and its partner U.S. university was not continued beyond Phase 1 due to an inability of the partners to agree on a meaningful 35 Ref: Transformational Leadership Program – Scholarships and Partnerships, Needs Assessment Report University of Prishtina Rectorate and Four Faculties Prepared for USAID Cooperative Agreement No. AID-167-A-14-00002 February 25, 2015 36 The Needs Assessment Study examined the relative strengths and weaknesses of the different departments, including their key challenges, quality of equipment and laboratories, and cooperation with other Faculties and government institutions. It also inquired into financial management and the implementation of the System for the Electronic Management of Students, as well as the quality of teaching, student services, and internships. USAID TLP Partnership Program Performance Evaluation – Final Report 71 workplan. Detailed list of activities by faculty planned at the time of inception of the project are presented in Table 2. The planned activities for Phase II of the partnerships were largely an extension of Phase I with two major changes: (i) the partnership for Faculty of Engineering with Arizona State University did not include exchanges for the teaching staff, as the Faculty of Engineering was unable to send the faculty to the U.S., and (ii) the scope for University of Arizona was expanded to include support for the establishment of University-wide Centers at UP. Progress under the TLP-SP partnerships component is assessed against the proposed activities at inception of project as well as the annual work plan and is placed in Annex 6. Comparison against the proposed activities and the annual workplan helps track the changes to the workplan made by TLP-SP to accommodate the fluid situation at UP and remain relevant. USAID TLP Partnership Program Performance Evaluation – Final Report 72 Table 2: Expected Deliverables from Partnerships – Summary Table Expected Results from the Partnership Program (Matrix developed on the basis of an analysis of the sub-grant documents for each partnership – for the period 2015-2017; 2017-2019) Activities for Phase 1 (first 2 years) of partnership that also extend to Phase 2 Results/ deliverables of activities Faculty of Engineering/ Arizona State University Faculty of Education/ Indiana University Faculty of Economy/ Dartmouth College Faculty of Agriculture/ University of Minnesota Details of Partnership activities, by category of activity Initial Visit Project launch Launch of partnerships and workplan development Collaborative development of a work plan Initial screening of applicants for the fall Semester-in￾Residence in U.S. program Conduct needs assessment of research capacity to identify gaps in skills Develop selection criteria to identify candidates for scholarly research learning community (SRLC), and Semester-in-Residence in U.S. program Workshop on “building and Educational and research culture” Terms of reference incorporating research into daily activity for submission to UP Faculty Senate Prepare a Research Framework and Terms of reference for working groups Roadmap for a cooperative research framework to ensure continuity and sustainability Workplan development Identify gaps and priorities Work plan with a prioritized set of activities developed A directory of teaching faculty and staff from Faculty of Agriculture (FA)-UP to identify shared areas of educational and research interests between UP and UMN; Develop criteria for competitive selection of faculty for exchange visits Document the basis of a mentoring network to be established between FA-UP and UMN; Develop a short course syllabus. /outline for use in creating follow-on faculty exchange institute in fall 2015 Faculty exchanges – short-term visits by U.S. partner to UP comprised of workshops targeting teaching Workshops and seminars held at the Faculty in UP by staff from the partner university Workshops or seminars addressing problems of research methodology Short-term scholarly visits to ASU to launch collaborative Minimum of 2 workshops per year/7 day duration (March and Nov) Workshops on topics such as: 7 events – 4 consulting projects (3 weeks and 1 week); 2 workshops by TUCK faculty (March and Nov); and 1 workshop by GRID (Dartmouth College) Minimum of 2 workshops per year/7 day duration ● Active teaching methods; ● Problem-based and / or case-based teaching and learning; USAID TLP Partnership Program Performance Evaluation – Final Report 73 Expected Results from the Partnership Program (Matrix developed on the basis of an analysis of the sub-grant documents for each partnership – for the period 2015-2017; 2017-2019) Activities for Phase 1 (first 2 years) of partnership that also extend to Phase 2 Results/ deliverables of activities Faculty of Engineering/ Arizona State University Faculty of Education/ Indiana University Faculty of Economy/ Dartmouth College Faculty of Agriculture/ University of Minnesota faculty and staff, and short courses for students research projects ● “ Building an Educational Research Culture ● “Pedagogical training in Cooperative Learning strategies”, ● Examining and assessing teacher quality”, and ● other topics Seminars for MEST focus on policy and accreditation Outcomes: development of Faculty of Economics research maps, literature reviews and work plans for research/training activity • 4 projects – 1 per semester UP faculty assigned to work with TUCK teams • 2 seminars by TUCK faculty –one learning seminar/ workshop annually onsite in Prishtina, to share best practices in experiential learning with the Faculty of Economics and other interested UP faculty and teaching staff • One seminar by participating faculty from Gender Research Institute at Dartmouth that will focus on the empowerment of young women as entrepreneurs and leaders, particularly minority women ● Communications ● Managing large class sizes; ● Experiential learning model development; ● Scholarship of teaching and learning; ● Interactive Hybridized online teaching modules; ● Leadership skills development; monitoring and evaluation methodologies; ● Policy courses (institutional and government) ● Research methods; ● Information systems; ● Risk assessment and analysis and others Faculty exchanges that involve younger UP faculty – professors and teaching assistants – UP to U.S. universities (Semester-in-Residence in U.S. program) Exchange visits to the partner university in the US for UP teaching staff – varying duration from a two-months to a full semester ● 6 scholars will go to ASU, 4 in fall of 2015, 2 in 2016 ● Minimum of two courses developed or revised (see above); ● (see above) ● Audit courses ● Attend graduate seminars and special weekly seminars organized for visiting fellows ● Semester long collaborative research practicum (min. 1 fellow per year as per PI, total of two not in sub award ) ● Semester long-teacher training collaboration (planned Fall semester) ● Faculty selected for long and short-term visits support the research framework (below) ● Audit courses ● Attend faculty meetings Host two visiting faculty members per year who will participate in the Gender Research Institute at Dartmouth (GRID) Seminar, in conjunction with activities at the Tuck School. ●2 fellows per year, total 4 - 5 month duration ●Minimum of two courses ●Developed or revised (see above); ●Trained in methods for peer-mentoring; seminar presentations to their FA￾UP colleagues sharing knowledge and experience; document and deliver a capstone or leadership project; ●Attend/participate in curricular design meetings USAID TLP Partnership Program Performance Evaluation – Final Report 74 Expected Results from the Partnership Program (Matrix developed on the basis of an analysis of the sub-grant documents for each partnership – for the period 2015-2017; 2017-2019) Activities for Phase 1 (first 2 years) of partnership that also extend to Phase 2 Results/ deliverables of activities Faculty of Engineering/ Arizona State University Faculty of Education/ Indiana University Faculty of Economy/ Dartmouth College Faculty of Agriculture/ University of Minnesota ● Develop action plan for curriculum development and teacher improvement ●Audit courses to deepen knowledge, observe teaching methods and gain ideas for course development ●Field visits to understand role of other organizations in support of the agriculture sector Faculty exchanges – short-term visits by administrators at faculties in UP to U.S. partner Administrators exposed to management and teaching systems at the Partner University Six visitors will work with: ASU online resources ASU center for teaching and Learning Visits will occur in August 2015, Jan 2016, and Jan 2017 Not listed in sub award – but team of 3 administrators visited IN (November) to experience culture of IN and get an understanding of their systems ● Summer study tour: ● (2015) for research collaboration ● (2015) for course development Not listed in sub award – but team of administrators will visit to experience culture of Dartmouth and get an understanding of their systems Not listed in sub award – but team of administrators will visit UMN to experience culture of Dartmouth and get an understanding of their systems Development of curricula and courses in selected departments in Faculties; ●Course plans, curriculum, or a plan for operations of an office or service ●Jointly developed by UP faculty and U.S. partner ●Pilot tested in UP ●Delivered via a digital learning platform ● Each fellow returns from semester in residence with ● Minimum of two courses developed or revised for their home dept. (with corresponding syllabi, teaching materials and evaluation materials in hand); ● Long term visit results in a curriculum and plan of action for implementation at UP leading to improvements to teacher training program design ● At least 2 new experiential courses and revision/updating of 4 existing courses ● Each fellow returns from semester in residence with ● Minimum of two courses ● Developed or revised for their home dept. (with corresponding syllabi, teaching materials ● And evaluation materials in hand); E-learning course management and delivery; (Hybrid courses to be developed in years 3 and 4) ● Interactive Hybridized online teaching modules; USAID TLP Partnership Program Performance Evaluation – Final Report 75 Expected Results from the Partnership Program (Matrix developed on the basis of an analysis of the sub-grant documents for each partnership – for the period 2015-2017; 2017-2019) Activities for Phase 1 (first 2 years) of partnership that also extend to Phase 2 Results/ deliverables of activities Faculty of Engineering/ Arizona State University Faculty of Education/ Indiana University Faculty of Economy/ Dartmouth College Faculty of Agriculture/ University of Minnesota Co-teaching courses between U.S. and UP faculty that can employ blended learning, utilizing the e￾learning course management platform; Two of the ASU short seminars to be delivered to UP will train UP colleagues for introduction of hybridized and online instruction; All six Semester-in￾Residence in U.S. fellows will be trained to deliver courses online or in hybridized format Train 4 faculty on the experiential methods, co￾teach workshops U.S. and UP faculty co-teach short courses in UP (see below) Workshops and short courses will be co￾developed prior to the institutes using real-time web-based communications Mentoring relationships in joint applied and basic research between the U.S. and UP partners; Includes Research Enhancement Development of data sets and articles and/or reports that can be published in a refereed academic journal ● Capacity building of UP faculty for mentoring graduate students, teaching, research and advising according to modern contemporary practice ●Workshops or seminars addressing problems of research methodology ●Short-term scholarly visits to ASU to launch collaborative research ●Access to ASU scientific digital library through ASU accounts to all faculty in partnership Develop a learning community to conduct research using action research and case study methods – activities include ● Development of a research trajectory, ● Literature reviews, and ● Defining “next steps” for research activity and position papers. ● Develop faculty-wide framework for continuity and sustainability of research ● Collaborative research symposium hosted by UP (2016) 4 UP members trained to design and deliver effective experiential, project based courses through involvement Fellows will receive mentoring from UMN faculty partners and be trained in methods for peer-mentoring in their home institution; Fellows will document at three and six month intervals after their return how they are leveraging their professional network Internships and service learning from the faculties that are developed in cooperation with career counselling center; and Building relationships with external institutions for service-learning institutions Not stated in sub-grant Not stated in sub-grant Not stated in sub-grant Not stated in sub-grant USAID TLP Partnership Program Performance Evaluation – Final Report 76 Expected Results from the Partnership Program (Matrix developed on the basis of an analysis of the sub-grant documents for each partnership – for the period 2015-2017; 2017-2019) Activities for Phase 1 (first 2 years) of partnership that also extend to Phase 2 Results/ deliverables of activities Faculty of Engineering/ Arizona State University Faculty of Education/ Indiana University Faculty of Economy/ Dartmouth College Faculty of Agriculture/ University of Minnesota Provide some essential and necessary resources, e.g., computer, software, textbooks ASU will provide Fac. Engineering with access to open source course management platform software Access to ASU scientific digital library through ASU accounts to all faculty in partnership Research repository established – with material secured and shipped by IU – for research, training and teaching as support for an improved research and training environment Access for UP faculty to resources developed by Tuck’s Experiential & Global Learning Collaborative around best practices in experiential learning. Access to Tuck’s online learning management system (CANVAS) Access to materials from Gender Institute and to member of Dartmouth faculty as needed None mentioned in Sub￾grant Per interviews: Access to University Consortium through the Center for Animal Health Access to partnerships with industry leading to better training for students USAID TLP Partnership Program Performance Evaluation – Final Report 77 Annex 6: Tracking TLP Partnerships TLP SP – Planned Activities and Status of Implementation 37 This section documents the planned activities as per the original description of the project and planned activities and the approved work plans, and tracks their status of implementation. The activities are listed by the broad objectives of the TLP-SP project and for each objective are presented in the following order: Planned broad activities and outcomes over the life of the project; followed by detailed activities as listed in the workplan by project year. Year 1 refers to project year (PY) 2014-15; Year 2 is project year 2015-16 (component year 1); Year 3 refers to project year 2016-17 (component year 2); Year 4 is project year 2017-18 (component year 3); and Year 5 refers to project year 2018-19 (component year 4). Any changes to planned activities within the original description of the project have been noted with the caption: Change. As this is a co-operative agreement, not all changes have resulted in or needed a contract modification. Planned - life of project Status of implementation Contractual Cooperative Agreement No. AID-167-A-14- 00002 Implementation 2014-2019; Funding : $30,199,763 Effective Dates: 02/26/14 – 06/30/19 Change: No cost extension provided for three months extending the end date from February 25, 2019 to June 30, 2019 Cooperative Agreement Total: $30,199,763 Change: ● Currency (Euro) devaluation has had an impact on the project in terms of total funds available ● Amendment to workplan approved on Sep 19, 2016 – includes changes to Objectives 2 and 3 of the TLP_SP program Objective3 - implementation starting in Year 2 of project initial award for two years 2015-2017; option years 2017-19 As planned – implementation began from PY2 of the TLP-SP project, some preliminary preparatory work completed in PY1 Option years approved in PY3, (2016-17); subgrants extended for university partners through November 2018 and then again to February 2019 Please note that as this is a report on the performance evaluation of the TLP-SP Partnerships component (Objective 3 of TLP SP), the tables below focus only on the progress of activities of the University Partnerships over the life of the project. The progress and the outcomes for the other two TLP-SP objectives (Objective 1: Establish Advisory Committee; and Objective 2 – Award Scholarships for Master’s and Certificate Program in the U.S.) are not addressed here. 37 It is important to note that the project year is from February to February – the Annual reports and the Workplan for TLP SP follow the project year for reporting; This is different from the Financial year which is from October to September – the monitoring and evaluation reports for TLP-SP follow this schedule. The tracker reflects and follows the Project Year for reporting achievements. USAID TLP Partnership Program Performance Evaluation – Final Report 78 Objective 3: TLP-SP University Partnerships component Also see Annex 5 for a detailed list of expected activities and outcomes for the faculty specific partnerships Planned outcomes - life of project Broad activities planned over life of project Key outcomes of planned activities Objective 3: Improved University Management and Pedagogy (in Selected Faculties of UP and/or Other Universities) through Partnerships and Exchanges with U.S. Universities Changes of Note: (i) at the time of inception of the TLP-SP project, the Partnerships program was designed to provide support to the Office of the Rector for implementation of three areas from Rector’s Action Plan of 2013. However due to a change in management, the Action Plan of 2013 became redundant, and a new strategy for UP was developed by the incoming management. However, the priority areas selected by the UP management for assistance from the TLP-SP Partnerships program aligned very closely with the priority areas selected by the previous management, hence while activities were added in to support the change, the broad activities and planned outcomes for the partnerships component outlined in the original workplan remained valid. (ii) the overall plan for TLP-SP to provide support to UP and other Universities in Kosovo was changed to focus on UP only so as to concentrate the resources on the single largest institution of higher education in Kosovo Support for Implementation of three areas from Rector’s Action Plan of 2013 changed to priorities from Rectors Strategy (marked as old and new; new priorities were used as guidelines for implementation) Old: enhancing revenue generation and financial viability; New: strengthening financial viability for UP • Rector’s Financial Advisory Committee - Strategy, operational guidelines and by￾laws developed • Center for Research and Sponsored Projects established at UP, an administrative office that will support an increase in both the quality and volume of research projects and external funding at the university. • Some policies implemented/modified to generate additional revenue Old: improving quality of research and teaching;; New: improving quality of research and teaching; See below – university wide activities Old: developing the Rectorate’s management, internal communications and public relations capabilities New: • Assistance to UP Management team on change management • Improving internal and external communications at UP • Change management program delivered to mid-level management (54 university administrative staff certified); one-on-one coaching provided for senior management • 13 administrators, including the Rector and several vice-rectors and the Deans and Finance officers from various faculties have been provided with one-on-one coaching in Prishtina and have participated in tours of US universities identified as good examples • Communication plan and PR strategy developed, and is being implemented; • New web site developed and populated • Web stories and short videos published • Alumni office revitalized and alumni and new student events planned and organized USAID TLP Partnership Program Performance Evaluation – Final Report 79 Targeted assistance to strengthen the academic capacity (teaching and assessment methodologies) and research development Assistance provide through ● University wide activities ● Faculty specific activities University-wide Activities: some examples ● Enhancing Student Services ● Expanding Life-Long Learning Programs ● Facilitating University Sponsored Research ● Improving Teaching Quality and Assessment ● Hybridized and Online Course Delivery ● Career Development Center revitalized; Model developed for a credit bearing internship program; Improved database developed to better connect students and graduates with internships, jobs, and networking resources; and ● Center for Teaching Excellence revitalized, a center coordinator appointed by UP, trained under TLP-SP; several seminars and workshops presented; cadre of new trainers developed to support the existing trainers at UP ● Center for Research and Sponsored Projects established (see above) ● Interdisciplinary Center for Renewable Energy and Sustainability established at the Rectorate level Change: Work on the Life-Long Learning program was not pursued; resources were instead diverted to the Center for Teaching Excellence, the Multi-Disciplinary Center and the Alumni Association (for collaboration with the private sector) Faculty specific activities • Four university partnerships established • 30+ academic staff, staff or students participate in exchanges between Kosovo and the U.S. 4 University Partnerships established between University of Prishtina, institutions of higher education in the United States. The partnerships were as follows: • UP Faculty of Agriculture – University of Minnesota (UMN) • UP Faculty of Education – Indiana University (IU); • UP Faculty of Economics – Dartmouth College (Dartmouth); and • UP Faculty of Electrical and Computer Engineering – Arizona State University (ASU) Three of the partnerships extended to Phase 2 (2017-19) leading to stronger relationships; partnership at Faculty of Economy not extended; Faculty of Engineering provided minimal support in Phase 2, funds redirected to Center for Energy and Sustainability instead. 156 exchange visits completed: 13 administrators and 31 teaching staff from UP supported for exchange visits to US 112 exchange visits from US to UP Faculties provided with equipment as needed including laptops, monitors, etc. Also distributed copies of a translated text￾book on research methods Change: TLP University Partnerships program focused exclusively on UP faculties, other universities were not included for direct participation USAID TLP Partnership Program Performance Evaluation – Final Report 80 Table 2: University Partnerships Exchange Visits38 Activities completed (Feb 2015 – June 2019) Arizona State University/ UP Engineering Indiana University/ UP Education Dartmouth College/ UP Economy Univ. of Minnesota/ UP Agriculture DC area and Colorado/ Management Number of Exchange visits - UP to U.S. and U/S/ to UP Partner visits (Semester-in-Residence in U.S. program and Study tours) Life of project 45 16 6 13 29 Number of UP to U.S. Partner visits (Semester-in-Residence in U.S. program and Study tours) by project year 2014-15 2 2 2015-16 1 5 0 2 2016-17 5 (3 academic staff) 4 5 8 (2 management) 2017-18 1 7 1 1 22 2018-19 0 3 2019-20 0 4 (2 DC; 2Colorado) Total 939 16 6 13 29 Number of U.S. to UP Partner visits by project year 2015-16 4 2 3 4 2016-17 10 9 9 8 2017-18 10 10 8 2018-19 7 16 2019-20 5 3 Total 36 40 20 16 0 Source: TLP-SP reports and records; interviews with stakeholders Partnerships component Project Year 1 (2014-15) – Pre-partnerships Planned activities prior to establishment of partnerships Status of Implementation Any Changes to Planned Activities Support for Implementation of UP Action Plan Activity 1: UP Action Plan Milestone Assessment Milestone Assessment conducted and approved University -wide activities To be implemented in Year 2 Cross-cutting activities identified for housing in the Rector’s office Faculty specific activities Activity 2: Identify and Assess the four UP Faculties for Partnership: will also work with other universities in Kosovo (e.g. University of Prizren and University of Peja) to assure the assessments and Action Plan goals are beneficial to their academic work and structure. Four UP faculties have been paired with four U.S. Universities University of Minnesota (UMN) – UP Faculty of Agriculture; Indiana University (IU) – UP Faculty of Education; Dartmouth College (Dartmouth) – UP Faculty of Economics; and Arizona State University (ASU) – UP Faculty of Electrical and Computer Engineering. Change: TLP University Partnerships program will only work with UP faculties, other universities will not be included for 38 Numbers reported here are based on data provided by TLP-SP project 39 Program was modified to increase U.S. to UP partner visits as the Faculty of Engineering was unable to send staff to the U.S. USAID TLP Partnership Program Performance Evaluation – Final Report 81 direct participation (even though some participants were invited to the leadership seminars from other universities) Activity 3: Develop Partnership Activities Plan Activities plan developed, and approved. Project plans and budget submitted to TLP￾SP by the university partners, along with UP and other academic institutions Draft Partnership Activities and Budget Plans summited to USAID and approved Sub-awards issued to partner universities Activity 4: USAID Approval of Activities and Budgets for Partnerships: Project Year 2 (2015-16) (Partnerships Year 1) Planned activities Status of Implementation Any Changes to Planned Activities Support for Implementation of UP Action Plan • Enhancing revenue generation and financial viability; - support formation of rector’s financial development advisory committee -finalization of operating guidelines and three year strategy • Improving the quality of academic staff and teaching; - (i) providing teaching instruction for faculty (center revitalized; workshops held; coordinator trained) - (ii) - supporting credit-bearing internships with up Career Development Center and partner faculties (develop model) • ,Developing the rectorate’s management, internal communications and public relations capabilities - (i) provide technical assistance to rectorate and across the university to build leadership and drive a comprehensive change-management program. - (ii) improve internal communications and public relations functions • Model and Operating guidelines for university financial development committee developed University -wide activities • Relaunch of the UP Center for Teaching Excellence; Seminars/workshops organized at the Center For Teaching Excellence – 54 academic staff members certified; awaiting appointment of coordinator • Single model of credit bearing internships scheme developed • Functional model of UP Career Development Center developed Change management training program offered to mid-level staff of the Rectorate • UP Public Relations and Communications Plan – strategy developed and recommendations offered Faculty specific activities Activity 1. University Partnership Launch Event ● Launch event organized in April 2015; ● All partner universities onsite for the launch activities and to hold roundtables to discuss and finalize detailed activity plans with faculty partners. ● (see Annex 5 for summary of two year workplans by partnership) ● First workshops organized at this time; selection criteria established for U.S. to UP exchange visits, recruitment initiated ● Only 8 UP to U.S. visits completed; 20 U.S. to UP visits ● TLP-SP is working closely with its partners to ensure delivery of all activities Activity 2. First American University Partners Visits to UP Activity 3. First Workshops with UP Teaching Staff Activity 4. First UP Visiting Scholars Travel to the U.S. -plan for 10scholars to travel to the U.S. in 2015 4 from Faculty of Electrical Engineering and Engineering; 2 from Education, 2 from Economics; and 2 from Agriculture to respective partners Activity 5. TLP-SP Oversight over Sub-award Activities USAID TLP Partnership Program Performance Evaluation – Final Report 82 Project Year 3 (2016-17) (Partnerships year 2) Planned activities Three of the partnerships extended to Phase 2 (2017-19); Economy not continued Status of Implementation Any Changes to Planned Activities Support for Implementation of UP Action Plan Improvement of management at UP and other universities ● Three year strategic plan for UP developed and approved by the UP Steering Board .Change: ● Rector’s Action Plan of 2013 will no longer be implemented ● New three years strategy developed for UP with one-on-one coaching and mentoring for the Rector and his management team form TLP-SP Areas of focus for project support will be same as before, so activities remain valid (i) Support from TLP-SP for Improvement of management at UP and other universities ● One-on-one coaching and mentoring to the Rector and his management team continuing to implement the strategic plan ● Training in leadership and Management for University Administrative staff: training offered total of 40 staff members from UP, University of Gjakova and University of Mitrovica attended the seminars. Four separate offerings took place in May, July, September and November 2016. (ii) Enhancing revenue generation and financial viability; - Support Formation of Rector’s Financial Development Advisory Committee (finalize operating guidelines, establish committee) Model and Operating guidelines for university financial development committee developed and shared with the Rectorate (iii) Improve Internal Communications and Public Relations Functions 1. Presentation of UP Public Relations and Communications Plan to the new UP Management 2. Hiring of two interns 3. Purchase of equipment (Camera and PC) 4. U.S. study visit for PR Officer 5. Support outreach and public relations efforts (open days, orientation days, etc.) ● PR strategy presented; new branding plan developed ● University building its own website to incorporate new PR strategy and branding ● Communication strategy developed; ● Interns hired; short videos and web stories (48) developed for publishing on social media websites University -wide activities Improving the quality of academic staff and teaching; Center for Teaching Excellence - (i) - Providing Teaching Instruction for Faculty - (Finalize ASU sub-grant for the work with CTE; complete rebranding; Carry-over from Year 2: U.S. study visit for CTE Coordinator; Additional Training for the CTE Coordinator) ASU subcontract finalized; CTE web site launched; branding and marking plan developed in three languages Albanian, Serbian and English CTE coordinator appointed and trained at ASU; seminars and workshops held at CTE USAID TLP Partnership Program Performance Evaluation – Final Report 83 Project Year 3 (2016-17) (Partnerships year 2) Planned activities Three of the partnerships extended to Phase 2 (2017-19); Economy not continued Status of Implementation Any Changes to Planned Activities UP Office for Research and Sponsored Projects) Opened with the appointment of a coordinator; Space from UP furnished by TLP-SP; Implementation plans developed for next two years development of Business plan being planned Career Development Center - (ii) - Supporting Credit-Bearing Internships with UP Career Development Center and Partner Faculties (finalize and implement model) - (Presentation of single model for internships - Work with UP Career Centre to develop alumni networks, internships fairs, career banks - Develop database and IT platform for placement of students; Placement of first students based on new internship program (pilot project with four partner faculties ● Career Development Center revitalized/supported under the Rector for Quality Management ● IT bids collected from local IT companies to help develop an alumni database; ● Draft database developed Faculty specific activities Indiana University/ Education Activity 1. Managerial Visit to Indiana University None Activity 2. Semester-in- Residence Program at IU And US to UP exchange visits 5 UP professors went to U.S. Program has been modified from a whole semester in duration to two months in duration to suit the needs of the academic staff members 8 exchange visits from IU to UP – offered seminars and workshops on research methods and worked with the research clusters (described below) Activity 3. Sustaining the FE Scholarly Learning Communities (SLCs) - collaborative research forums with SLCs at UP Activity 4. Research Forums at FE-UP Teamwork and collaborative work from academic staff members 7 research clusters formed, each with a mentor from IU; preparing research projects to present at Research symposium Organized first ever Scientific Summer Symposium: Enhancing Education Research at the Faculty of Education University of Minnesota (UMN) /Agriculture and Veterinary (FAV) Activity 1. Short-term visit by UMN faculty to UP Faculty of Agriculture and Veterinary Six academic staff/staff members from UMN visited UP and presented a four day seminar; guest lectures; and helped with setting up and refurbishing existing radiology and anesthesia equipment; conducted demonstrations in using the machine Activity 2. Short term UP faculty exchange visits University of Minnesota (UMN) 3 academic staff members visited UMN for a one month visit 2 academic staff/manager (the Dean and the vice Dean) visited UMN to observe various site visits including one to a diagnostic lab Activity 3. Faculty Development Fellowships at University of Minnesota None USAID TLP Partnership Program Performance Evaluation – Final Report 84 Project Year 3 (2016-17) (Partnerships year 2) Planned activities Three of the partnerships extended to Phase 2 (2017-19); Economy not continued Status of Implementation Any Changes to Planned Activities Dartmouth College/ Economy Activity 1. Tuck First-Year Project in Kosovo Activity 3. Second Tuck On Site Consulting Team First year project completed – experiential learning module incorporated into an existing class – three teams presented research findings Building Grass-Roots Organizations at the UP seminar completed; Five clubs opened at the Faculty of Economy and launched Demonstration on Use of Active Learning Techniques Activity 2. Five UP visiting scholars to Dartmouth College (three scholars at Tuck School and two at Gender Research Institute at Dartmouth) Completed 3 professors from the Faculty of Economy hosted at Tuck School; 2 from the Faculty of Philosophy hosted by the Gender Institute, focus primarily on the GRID Spring Symposium series. Arizona State University/ Electrical and Computer Engineering (ECEN) Activity 1. Inter-disciplinary Certificate Program “Energy and Sustainability” Certificate program developed; however it has been housed at the Center for Renewable Energy and Sustainability instead of at ECEN Activity 2. Short-term UP Scholarly Visits to ASU Professors from UP unable to go to ASU, program reconfigured ASU presented a two-week Renewable Energy and Sustainability seminar, in Kosovo. Eleven professors attended the seminar, and outcomes included individual action plans for professors and draft curricula. 10 staff from ASU to UP Project Year 4(2017-18) (Partnerships year 3) Planned activities Status of implementation (as of Nov 2017) Any Changes to Planned Activities Support for Implementation of UP Action Plan Objective1: Change management and improve PR and communication (i)Support for Improvement of Management at UP and other Universities Activity 1: Leadership and Change Management Seminars – 3 more seminars for midlevel administrative staff planned: ● One-on-one coaching and mentoring to the Rector and his management team continuing to implement the strategic plan ● Training in leadership and Management for University Administrative staff: training offered total of 14 staff members from UP, (total 54 to date) Activity 2: Senior Executive Leadership Summit – two day leadership seminar for Deans of UP faculties Completed Activity 3a and 3b: Leadership and Change Management Alumni Exchange Visits (2 weeks in duration) Leadership fellowships awarded to 4 of the certified administrative staff for a study tour in the U.S. Activity 4: UP Senior Management Exchange Visit to the United States Completed USAID TLP Partnership Program Performance Evaluation – Final Report 85 Project Year 4(2017-18) (Partnerships year 3) Planned activities Status of implementation (as of Nov 2017) Any Changes to Planned Activities (ii) Improve Internal Communications and Public Relations Functions Support outreach and public relations efforts (continue support for interns.) Support for interns work with UP on implementation of the Outreach and Communications strategy developed by the Program provided through November 2017 Activity 5: Unified Website for University of Prishtina and sub-sites for all faculties Completed Activity 6: Producing Educational and Promotional Videos for UP Ongoing Objective 2: Enhance Revenue Generation and Financial Viability Activity 1: Continue support to the Rector’s Financial Development Advisory Committee Financial Advisory Committee established; Comparative study prepared and presented to UP Committee by-laws drafted and forwarded to the vice-rector for finance Activity 2: Workshops and Training for members of the Financial Advisory Committee on strategy development Completed – strategy and recommendations developed pending approval/support from UP management Center for Teaching Excellence (i) - Providing Teaching Instruction for Faculty - Study visit for the Vice-Rector for Training and three of the trainers - Four trainings for teaching staff and trainers of CTE Completed University -wide activities (ii) – Assessment/data development to inform decision makers White paper developed to address issue of paucity of data; Work on Data systems initiated UP Office for Research and Sponsored Projects (Center for Sponsored Research) Activity 3: Support Creation of an Office for Sponsored Programs - ASU to establish 2 year plans to establish office - Establish system for tracking of donor-funded programs - Needs assessment for long term sustainability Center established, furnished by TLP-SP; Coordinator appointed and sent for study tour/training; Implementation plans developed and activities such as improving UP presence on online networks initiated Career Development Center - (i) - Supporting Credit-Bearing Internships with UP Career Development Center and Partner Faculties (finalize and implement model in three faculties) - Training on model - Implement new courses that include internships ● Career Development Center supported and revitalized under the Rector for Quality Management ● Career Center data base developed to house information on all students from UP ● Database has a portal specifically for UP Alumni ● Internship program initiated with participating faculties USAID TLP Partnership Program Performance Evaluation – Final Report 86 Project Year 4(2017-18) (Partnerships year 3) Planned activities Status of implementation (as of Nov 2017) Any Changes to Planned Activities - Develop database and IT platform for placement of students; - Data base also used to develop alumni networks – public launch for portal Other activities are planned Faculty specific activities Indiana Univ/Education Activity 1. Continue Support for research clusters to extend understanding in research methods and result in publications - Assign mentor - Support via email and video conference - Hold brown bags Research clusters in place, supported through email and videoconference support Activity 2. IU academic staff visits UP to hold workshops Completed as planned Activity 3. Planned exchange visits for Faculty of Education teaching staff and management (Dean and Vice Dean) Completed as planned Activity 4. Support for annual “Scientific Summer Symposium” Completed Univ. of Minnesota/ Agriculture and Veterinary Activity 1. Short-term visit by UMN academic staff to UP for support strategic planning and institute development Completed Strategic planning workshop facilitated as a part of the process for drafting the Strategic Plan 2018 - 2022 Activity 2. Short term assistance to build research capacity and capability at faculty of Agriculture and Veterinary - Two such visits planned Completed Activity 3. Co-mentorship of graduate students leading to publications Ongoing Activity 4. Faculty Development Fellowships at University of Minnesota – 2 fellows expected for 1 month each One completed, one planned; Activity 5. Strengthening pedagogy and curricula Activity 6. Short-term visits to assess progress Planned Dartmouth College/Economics No Activities planned as the Partnership has been suspended Two activities continuing from Years 1 and 2 Continuing support for GRID fellowship Assistance to Summer school on Gender and Sexuality Completed Arizona State University/ ECEN Assistance for Center for Teaching Excellence Activity 1: Advancing best practices in teaching through teaching methodology seminars -develop teaching guide for Coursesites – course management software -seminar on best practices in course syllabus preparation -Seminar sessions on hybrid/blended learning and on experimental online course delivery Completed USAID TLP Partnership Program Performance Evaluation – Final Report 87 Project Year 4(2017-18) (Partnerships year 3) Planned activities Status of implementation (as of Nov 2017) Any Changes to Planned Activities -Introductory Seminar on Interactive Teaching, Collaborative Learning, and New Course Development -Seminar session on best practices in preparation and administration of standardized course evaluation instruments Activity 2. Support for CTE to develop sustainability Ongoing Faculty of Engineering Partnerships Activity 3. UP CREATES – expanding the role of the Center for Renewable Energy and Sustainability Building curriculum and research capacity (also see under University-wide activities) Planned Change Interdisciplinary Center established as an academic research and teaching center at the Rectorate level (not in the Faculty of Engineering) Activity 4. A State of the Art UP Sponsored Research Management Office established Completed – see above under faculty-wide activities Activity 5. Continued Enrichment and Exchange with UP Engineering Faculties, including FECE -hold seminars -offer department heads library access Completed Project Year 5(2018-19) (component year 4) Planned activities Status of implementation (as of May 2019) Any Changes to Planned Activities Support for Implementation of UP Action Plan (i) Support for Improvement of management at UP and other universities – provide technical assistance to rectorate to manage change Activity 1: Leadership and Change Management Alumni exchange visits for participants of the senior leadership seminar – 10-14 day visits planned for September 2018 ● One-on-one coaching and mentoring to the Rector and his management team continued through end of project ● (ii) Improve Internal Communications and Public Relations Functions Activity 2: PR officer and Vice Rector for International Cooperation – visit to the US Activity 3: Support outreach and public relations efforts at UP through production of videos and promotional materials Completed - PR officer and Vice Rector for International Cooperation visited US Videos and promotional materials developed highlighting accomplishments of the university and addressing topics such as importance of academic integrity; safe employment environment, sexual and other forms of harassment, etc. PR activities organized to engage alumni in making university a better place for students USAID TLP Partnership Program Performance Evaluation – Final Report 88 Project Year 5(2018-19) (component year 4) Planned activities Status of implementation (as of May 2019) Any Changes to Planned Activities Objective 2: Enhance Revenue Generation and Financial Viability Activity 1: Continue support to the Rector’s Financial Development Advisory Committee Activity 2: UP Financial Policies and Procedures Manual Activity 3: Workshop and Training for UP financial staff in all (13) faculties, the UP management team, UP Vice Deans for Finance as well as Rector’s Financial Advisory Committee members Completed – manual developed with help from Sr. Financial advisor from World Learning and Baker and Tillie (a local financial advisory company) Completed Training provided for the financial staff Activity 4: Support creation of Office for Sponsored Programs – to address lack of revenue generating research at UP and enhance its capacity Completed – UP Office for Research and Sponsored Projects (Center for Sponsored Research) staffed and functional; coordinator has been trained and is beginning to roll out a system for application and management of grant and contract funds to provide enhanced tracking and management of donor-funded program; ORSP has helped improve the online presence of UP teaching staff, leading to an improved ranking for UP ASU provided ongoing mentoring and training for coordinator University -wide activities Objective 3: Improve quality of Academic Staff and Teaching Center for Teaching Excellence (i) - Providing Teaching Instruction for Faculty - Study visit for the Vice-Rector Training and three other trainers - Four trainings for teaching staff and trainers of CTE (also see section on Arizona State University support for CTE) Completed Center for Teaching Excellence revitalized; Coordinator appointed and trained Courses developed on Coursesites, active learning in the classroom Cadre of 12 new trainers trained and certified to support the existing trainers; instructional trainer also hired for Coursesites (ii) – Assessment/Assurances of Learning Systems/Data Development to inform decision makers Data systems developed with support of a local company to build a system to capture missing data or to pull data from multiple existing sources Career Development Center - (i) – Support ‘Service learning’ and Credit-Bearing Internships with UP Career Development Center and Partner Faculties - Change model to combine work of Career center and Alumni Association for a cohesive approach Completed USAID TLP Partnership Program Performance Evaluation – Final Report 89 Project Year 5(2018-19) (component year 4) Planned activities Status of implementation (as of May 2019) Any Changes to Planned Activities - Establish Alumni Association and community service engagement for earned credit - Community service synchronized with PR activities; - Support the Career Center Database to track employment after graduation, and provided information on internships and job opportunities Faculty specific activities Indiana Univ/Education Activity 1. Continue Support for research clusters to extend understanding in research methods and result in publications - Support via email and video conference - Completed Activity 2. IU faculty visits UP to hold workshops on topics related to research methodology and improving teaching methodology with the faculty of education Completed as planned Activity 3. Planned exchange visits for Faculty of Education management - IU Deans visit UP - FE management staff visit US to extend partnership and to understand how to expand primary research in Kosovo Completed as planned Activity 4. Support for annual “Scientific Summer Symposium” – continued through an annual research conference at UP Completed Faculty in Residence program at IU 6 teaching staff completed an eight week exchange visit (straddling over PY4 and PY5) Univ. of Minnesota/ Agriculture and Veterinary Activity 1. Strategic planning and institute development Strategic plan 2018-22 Completed Institute development in progress, contingent on move of Faculty to new space Activity 2. Short term assistance to build research capacity and capability at faculty of Agriculture and Veterinary - UMN faculty visit UP Co-mentorship of graduate students leading to publications Completed Activity 3. Building research capacity completed; support provided through supplies, training for research fellows, mentoring of students and submitting manuscripts for publication Activity 4: Faculty Development Fellowships at University of Minnesota – 2 fellows expected for 1 month each On-going from previous year; Completed Activity 5. Strengthening pedagogy and curricula Completed as planned Activity 6. Short-term visits to assess progress Completed as planned Dartmouth College/Economic No Activities planned as the Partnership has been suspended Completed USAID TLP Partnership Program Performance Evaluation – Final Report 90 Project Year 5(2018-19) (component year 4) Planned activities Status of implementation (as of May 2019) Any Changes to Planned Activities Arizona State University/ ECEN Support for Center for Teaching Excellence Activity 1: Advancing best practices in teaching through teaching methodology seminars: - Follow-on seminar for all UP teaching staff on Coursesites software Seminar on using technology and on experimental online course delivery Planned Faculty of Engineering Partnerships Activity 2 UP CREATES – expanding the role of the Center for Renewable Energy and Sustainability Building curriculum and research capacity First two cohorts enrolled in the certificate program in “Renewable energy and Sustainability” – courses offered at the center are already accredited at other faculties and departments 5 Seed grants awarded to applied research proposals All Faculties in the Partnerships program: Additional equipment and textbooks provided to the faculties as needed at the end of the project USAID TLP Partnership Program Performance Evaluation – Final Report 91 Page left Blank USAID TLP Partnership Program Performance Evaluation – Final Report 92 For more information contact www.ibi-usa.com USAID/Kosovo Transformational Leadership Impact and Performance Evaluation IBI 4040 North Fairfax Drive Suite 650 Arlington, Virginia 22203 Tel: 1-703-525-2277 www.ibi-usa.com