U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES MIDTERM EVALUATION AUGUST 3, 2018 This publication was produced for review by the United States Agency for International Development. It was prepared by Management Systems International, A Tetra Tech Company. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES MIDTERM EVALUATION Contracted under Order No. AID-391-C-15-00004 Performance Management Support Contract DISCLAIMER This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of the Management Systems International and do not necessarily reflect the views of USAID or the United States Government. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION iii CONTENTS Acronyms...........................................................................................................................vi Project Summary ............................................................................................................viii Executive Summary.......................................................................................................... 1 Key Findings.......................................................................................................................................... 2 Key Conclusions.................................................................................................................................. 3 Key Recommendations....................................................................................................................... 3 Evaluation Purpose and Questions.................................................................................. 5 Evaluation Questions.......................................................................................................................... 6 Project Background .......................................................................................................... 6 Theory of Change................................................................................................................................ 6 Evaluation Methods and Limitations............................................................................... 7 Data Collection.................................................................................................................................... 7 Data Analysis......................................................................................................................................... 7 Evaluation Limitations......................................................................................................................... 8 Findings .............................................................................................................................. 8 Findings for Evaluation Question 1: Partnership Experience .................................................... 8 Satisfaction with Partnership .............................................................................................................9 Partnership Benefits to Pakistani Universities............................................................................10 Partnership Benefits to American Universities...........................................................................11 Partnership Challenges.....................................................................................................................12 Conclusions: Partnership Experience............................................................................................13 Findings for Evaluation Question 2: Effectiveness of CAS Components..............................13 Exchange Programs...........................................................................................................................13 Joint and Applied Research.............................................................................................................15 Lab Utilization and Effectiveness..................................................................................................17 Curriculum Reform............................................................................................................................18 Public-Private Partnerships..............................................................................................................19 Graduate Training for Employment.............................................................................................. 20 Gender Equity.....................................................................................................................................22 Conclusions: Effectiveness of CAS Components........................................................................24 Findings for Evaluation Question 3: Sustainability......................................................................24 Overall Findings..................................................................................................................................25 CAS – Energy (NUST Islamabad).................................................................................................25 CAS – Energy (UETP).......................................................................................................................28 CAS – Water (MUET Jamshoro)...................................................................................................31 Conclusions: Sustainability...............................................................................................................34 Findings for Evaluation Question 4: Governance.......................................................................34 CAS – Energy (NUST Islamabad).................................................................................................35 CAS – Energy (UETP).......................................................................................................................38 CAS – Water (MUET Jamshoro)...................................................................................................41 Conclusions: Governance .................................................................................................................45 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION iv Conclusions ...................................................................................................................... 45 Recommendations .......................................................................................................... 47 Establishing Context for Recommendations...............................................................................47 Structure of CAS-Specific and General Cross-CAS Recommendations..............................48 CAS-Specific Recommendations....................................................................................................49 CAS/NUST ...........................................................................................................................................49 CAS/UETP ............................................................................................................................................52 CAS/MUET ..........................................................................................................................................55 General Cross-CAS Recommendations for USAID and HEC................................................57 Annex 1: Statement of Work......................................................................................... 61 Annex 2: Assignment Work Plan .................................................................................. 70 Annex 3: Further Analysis of Project Components ..................................................... 92 CAS/MUET..........................................................................................................................................92 CAS/NUST........................................................................................................................................115 CAS/UETP .........................................................................................................................................126 Annex 4: Methods.......................................................................................................... 136 Survey of CAS Faculty, Students, and Graduates.....................................................................136 Key Informant Interviews..............................................................................................................136 Group Interviews.............................................................................................................................137 Validation Workshops....................................................................................................................138 Secondary Review............................................................................................................................138 Annex 5: Sustainability Indicator Data ....................................................................... 139 Annex 6: CAS Midterm Evaluation Conceptual Framework ................................... 140 Annex 7: Qualitative and Quantitative Data Collection ........................................... 141 Annex 8: MAXQDA Screen Shot, Provision, and Final Code List ........................... 143 Final Code List..................................................................................................................................144 Annex 9: List of Secondary Documents...................................................................... 159 Annex 10: Data Collection Instruments ..................................................................... 160 Individual Interview Consent Form.............................................................................................160 Group Interview Consent Form..................................................................................................161 Group Interview Instrument for CAS Students.......................................................................162 Group Interview Instrument for CAS Alumni..........................................................................164 Interview Instrument for CAS Faculty........................................................................................166 Interview Instrument for CAS Administrator/USAID ............................................................170 Interview Instrument for CAS Lab Personnel...........................................................................175 Interview Instrument for CAS University Leadership ............................................................176 Interview Instrument for U.S. Partner University....................................................................180 Interview Instrument for Higher Education Commission......................................................184 Interview Instrument for Sector Organization.........................................................................187 Survey Questionnaire for Faculty ................................................................................................190 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION v Survey Questionnaire for Graduates/Alumni ...........................................................................199 Survey Questionnaire for CAS Current Students...................................................................208 List of Tables Table 1: Project Summary...............................................................................................................................viii Table 2: Female Faculty, Students, and Alumnae at CAS ........................................................................22 Table 3: Value of CAS Components as Perceived by Water Sector Organizations........................33 Table 4: Agreement with Statements About CAS/NUST Governance Structures..........................35 Table 5: Agreement with Statements About CAS/UETP Governance Structures...........................39 Table 6: Agreement with Statements About CAS/MUET Governance Structures..........................43 Table 7: Online Survey Response Rate for Current Students...............................................................92 Table 8: Female Faculty and Students at CAS Institutions....................................................................112 Table 9: Water Sector Organizations’ Responses on U.S.-PCAS Project Components...............113 Table 10: Participation Rates of Students in Exchange Programs.......................................................118 Table 11: Online Survey Response Rate ...................................................................................................136 Table 12: Key Informant Interviews...........................................................................................................137 Table 13: Distribution of Group Interviews.............................................................................................137 List of Figures Figure 1: CAS Locations................................................................................................................................... ix Figure 2: Satisfaction with Partnership........................................................................................................... 9 Figure 3: Exchange Programs Improved Research Skills (% of Respondents)....................................14 Figure 4: Alignment of Applied Research with Sector Needs................................................................16 Figure 5: Curricula Aligned with Sector Needs (% of Respondents)...................................................18 Figure 6: Teaching Balances Theory and Practical Application (% of Respondents) ........................19 Figure 7: Satisfied with Job Placement (% of Respondents)....................................................................21 Figure 8: Women Face Greater Barriers to Employment than Men (% of Respondents)..............23 Figure 9: Impact of Exchange Program on CAS/MUET Students and Alumni ...................................95 Figure 10: CAS/MUET Faculty Opinions on Whether Women Face Greater Barriers than Men to Employment.......................................................................................................112 Figure 11: CAS/MUET Faculty Perceptions of Sector Organizations’ Preference for Men in Lead Research Roles..................................................................................................114 Figure 12: Perception of CAS/NUST Current Students of Impact of Exchange Program............116 Figure 13: CAS/NUST Students’ Preparation for Exchange Program................................................117 Figure 14: CAS/NUST Faculty’s Perceptions of Sector Organizations’ Participation in Applied Research......................................................................................................................119 Figure 15: CAS/NUST Students’ Responses Regarding Operational Status of Labs......................123 Figure 16: Faculty Using CAS/NUST Laboratories for Teaching ........................................................124 Figure 17: Faculty Using CAS/NUST Laboratories for Research........................................................124 Figure 18: Improved Student Applied Research Skills, Applied Research Output, and Proposal Writing Skills at CAS/UETP .................................................................................126 Figure 19: Faculty Engagement in Applied Research Since Joining CAS/UETP ................................129 Figure 20: CAS/UETP Faculty Perceptions About Research Funding ................................................130 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION vi ACRONYMS ADS Automated Directives System AGL Attock Generation Limited AOR Agreement Officer’s Representative ASU Arizona State University BoG Board of Governors CAS Center for Advanced Studies CAS-E Center for Advanced Studies in Energy CAS-E NUST Center for Advanced Studies National University of Science and Technology CAS-E UETP Center for Advanced Studies University of Engineering and Technology, Peshawar CAS-W Center for Advanced Studies in Water CAS-W MUET Center for Advanced Studies Mehran University of Engineering and Technology, Jamshoro CLA Collaborating, Learning and Adapting CoE Center of Excellence COR Contracting Officer’s Representative DISCO Distribution Companies FATA Federally Administered Tribal Areas GIS Geographic Information Systems GoP Government of Pakistan HEC Higher Education Commission of Pakistan IBA Institute of Business Administration IWRM Integrated Water Resources Management KII Key Informant Interviews KP Khyber Pakhtunkhwa KPOGCL Khyber Pakhtunkhwa Oil & Gas Company Limited LUMS Lahore University of Management Sciences MEL Monitoring, Evaluation, and Learning MESC Materials for Energy Storage and Conversion MOU Memorandum of Understanding MSI Management Systems International MUET Mehran University of Engineering and Technology NEPRA National Electric Power Regulatory Authority U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION vii NOC No Objection Certificate NTDC National Transmission & Dispatch Company NUST National University of Sciences and Technology ORIC Office of Research, Innovation, and Commercialization PEDO Pakhtunkhwa Energy Development Organization PESCO Peshawar Electricity Supply Company PKR Pakistani Rupee PMU Project Management Unit SDG Sustainable Development Goal SIDA Sindh Irrigation and Drainage Authority SOP Standard Operating Procedure UETP University of Engineering and Technology, Peshawar USAID United States Agency for International Development USD United States Dollar U.S. United States U.S.-PCAS United States-Pakistan Centers for Advanced Studies WASH Water, Sanitation, and Hygiene XRD X-ray Diffraction U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION viii PROJECT SUMMARY Table 1 summarizes the basic information about the U.S.-Pakistan Centers for Advanced Studies (U.S.-PCAS) project. TABLE 1: PROJECT SUMMARY Title/Field Project/Activity Information Contract/agreement numbers • AID-391-A-14-00004 - National University in Science and Technology (NUST) • AID-391-A-14-00007 - University of Engineering and Technology in Peshawar (UET) • AID-391-A-15-00001 – Arizona State University (ASU) • AID-391-A-15-00003 - Mehran University of Engineering and Technology (MUET) • AID-391-A-15-00004 - University of Utah (UU) Contracting/agreement officer’s representative (COR/AOR) Start date • September 2014 - National University in Science and Technology (NUST) • October 2014 - University of Engineering and Technology in Peshawar (UET) • November 2014 - Arizona State University (ASU) • December 2014 o University of Utah (UU) o Mehran University of Engineering and Technology (MUET) Completion date • September 2019 – o National University in Science and Technology (NUST) o University of Engineering and Technology in Peshawar (UET) • November 2019 - Arizona State University (ASU) • December 2019 – o University of Utah (UU) o Mehran University of Engineering and Technology (MUET) Locations Islamabad, Peshawar, Jamshoro Implementing partner(s) Center for Advanced Studies in Energy (CAS-E) at the National University of Science and Technology (NUST); CAS-E at the University of Engineering and Technology, Peshawar (UETP); Center for Advanced Studies in Water (CAS-W) at Mehran University of Engineering and Technology (MUET), Jamshoro; Arizona State University (ASU), USA; University of Utah (Utah), USA; Higher Education Commission (HEC) USAID/Pakistan Mission Strategic Framework objectives addressed DO4 – Education Budget $102 million as per project documents • $48 million for the Centers for Advanced Studies in Energy • $25 million for the Center for Advanced Studies in Water • $20 million for modern facilities • $9 million for HEC, support and evaluation heads U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION ix Figure 1 illustrates the CAS locations. FIGURE 1: CAS LOCATIONS U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 1 EXECUTIVE SUMMARY In response to the Government of Pakistan’s (GoP’s) request, the United States Agency for International Development in Pakistan (USAID/Pakistan), in collaboration with the Higher Education Commission (HEC), funded four Centers for Advanced Studies (CASs)—two in energy, one in water, and one in agriculture—under the U.S.-Pakistan Centers for Advanced Studies (U.S.-PCAS) project. Funding for the center in agriculture was discontinued in October 2017. The project also included funding for three American universities to provide technical assistance and support to these centers. U.S.-PCAS is a five￾year project intended to improve the capacity of Pakistani higher education institutions in finding innovative solutions to some of Pakistan’s greatest development challenges through applied research; produce graduates who are better prepared to meet the needs of industry, government, and their communities; and provide leadership in policy dialogue. This midterm evaluation of the U.S.-PCAS project assesses project performance with a focus on center management and sustainability, while gathering lessons learned for each of the three remaining CASs in the project and the partnerships that support them. It focuses on three centers—Mehran University of Engineering and Technology (MUET), National University of Sciences and Technology (NUST), and University of Engineering and Technology, Peshawar (UETP)—and two partner universities—Arizona State University (ASU) and the University of Utah (Utah). The evaluation used a mixed-methods approach to answer four questions: 1. How was the partnership experience in general for both Pakistani and United States (U.S.) universities under each CAS? 2. How effective is each component of each CAS in achieving the overall project objectives? 3. How does each of the partner universities perceive/approach sustainability for the CAS activities/results? 4. How are the universities utilizing their governance structures to sustain the CAS activities/results? Notwithstanding the efforts of the evaluation team to obtain valid and reliable data, there were several limitations: 1. Response errors of interview data. Mitigation efforts included triangulation through a variety of methods and sources. 2. Attribution: The evaluation design does not rigorously attribute results to the U.S.-PCAS project. Instead, the team collected the information to provide plausible attribution. 3. Generalizability across sector organizations: The evaluation design allowed 12 industry partner interviews. Responses from this small, purposive sample are not generalizable across the whole sector. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 2 Key Findings The evaluation surfaced a number of general findings relating to all three centers, as well as several findings related to the development of each specific CAS. Partnership experience: The partnership experience has been largely positive for both Pakistani and U.S. universities. Affiliation with U.S. universities has elevated the prestige of the Pakistani universities, and the partnerships have been instrumental in building the centers’ capacities in curriculum, applied research capacity, and sector engagement. On the U.S. side, partnerships have enabled U.S. faculty and students to understand sector-related issues from a Pakistani perspective and helped U.S. universities recruit students. Challenges to the partnerships have included aligning the partners’ priorities to achieve lasting change. Effectiveness of components: The project encompasses multiple components: Foreign exchanges: International exchanges have been transformative in enhancing students’ research capacity and developing professional focus. Faculty exchanges have been less popular, initially due to poor planning, but also because prolonged travel can be difficult for faculty. Applied/joint research: Each CAS pursues applied research in a manner that best serves its needs. For NUST and UETP, this has meant awarding grants to both CAS faculty and outside experts, while MUET focuses on developing internal capacity around a defined research agenda. Potential clients/collaborators in the sector vary in their satisfaction with CAS research; MUET-associated organizations are very satisfied, NUST organizations are less so, and UETP organizations see potential but have not seen results. Lab utilization: Each CAS developed high-quality research labs specific to sector needs. Procurement challenges caused extensive delays in operationalizing labs, and now each center is developing utilization strategies to guide student and faculty use, as well as commercialization. Curricula: U.S. partners help each CAS develop high-quality curricula targeted to sector needs. Pakistani universities value this best-practice input in curricular design, though several centers confront challenges in curriculum delivery and in engaging sector stakeholders in the design process. Public-private partnerships: Though centers are making progress toward the target of establishing five partnerships by the project’s end, they focus more on signing memoranda of understanding (MOUs) than on building the robust relationships that will sustain long-term partnerships. Training for employment: Sector organizations are pleased with the caliber of CAS graduates compared to typical graduates they encounter. Students across all centers are dissatisfied with the job placement support they receive, though in some centers internship support is relatively strong. Gender: While each CAS struggles to meet targets for gender equity, the centers have had more success recruiting female students than female faculty. The exception is UETP, which struggles overall with gender targets. MUET has adopted the CAS gender policy university-wide and is recognized by sector organizations for its efforts. Sustainability: Sustainability planning at each CAS appears to focus more on sustaining physical assets and infrastructure than on center outcomes. Centers’ sustainability plans vary in how they articulate the value proposition that will draw stakeholders to pay for what they now receive courtesy of USAID funding. Each CAS expects that HEC will help sustain its operating costs, though the precise nature of these arrangements has yet to be clearly defined and confirmed. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 3 Governance: Stakeholders generally believe that governance at each CAS is effective while project funding continues. Once USAID funding ends, most stakeholders assume that centers will align their governance to existing host university structures in the belief that this will enhance chances for sustainability. Those who do not agree with this assumption maintain that the centers need to keep a degree of autonomy in order to create value for the sectors they are designed to serve. Key Conclusions CAS is effectively a combination capacity-building and change management project in the higher education sector. By articulating a vision for CAS of building the mechanisms for deeper engagement with economic sectors key to Pakistan’s long-term growth, the project has created the basis for each university to develop the capacity that will enable it to change the nature of its relationships with the sectors. The expertise provided by the U.S. partners serves as a catalyst for guiding change, while the extensive resources provided by USAID create the basis to channel this expertise into mechanisms through which each CAS can establish its reputation and build its capacity. Knowing that these resources will not exist in their current form beyond the project’s funding horizon, each institution should use these resources as short-term scaffolding upon which to consolidate its distinctive brand and grounded capacity for the longer term. Each center varies in the degree to which it leverages resources to balance short-term impact with longer-term capacity and brand development that will help it sustain outcomes. Sustainability will depend on each center’s ability to produce outcomes that are resilient and adaptive in the face of changing circumstances over time. By using the remaining time and resources of the project to shift more purposively to a longer-term focus on people, technical skills development, and governance, each center will have its best chance to build a sustainable future. Key Recommendations Recommendations for CAS/NUST Strengthen industry-academic linkages to foster a better understanding of industry problems and commercialization of CAS/NUST resources: CAS/NUST should deploy its faculty to identify the needs of specific organizations and to design solutions. Moreover, as it strategizes its outreach and relationship development, CAS/NUST should engage high-level energy-sector professionals on its corporate advisory board to advise the center. The center should also engage the University Office of Research, Innovation, and Commercialization (ORIC) 1 and HEC to help build stronger relationships with sector organizations. Develop comprehensive marketing and outreach efforts: CAS/NUST should establish a dedicated marketing and outreach unit to design and implement a comprehensive strategy to analyze market needs and to develop messaging across multiple sector segments. The strategy should address the market for CAS consulting, training, lab testing, applied research, and career placement of students. The unit should also dedicate resources to focus exclusively on a strategy for developing and procuring funding from domestic and international donors. Establish a technology center to offer short-term training courses and energy audits: CAS/NUST should work with USAID and HEC to produce a comprehensive business plan for a technology center that defines realistic market analyses, resources, revenue potential, seed funding sources, and the time horizon needed to achieve sustainability. Once the center is established, NUST can leverage faculty expertise to provide skills-based, short-term training using center resources as a means of generating 1 At NUST, ORIC has two divisions: the Directorate of Research and the Directorate of Innovation and Commercialization. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 4 revenue. However, the readiness of faculty to deliver such training is a prerequisite for such an endeavor. Recommendations for CAS/UETP Revise the research grant process to emphasize commercialization of solutions: CAS/UETP should revamp its existing research grant award process to enhance prospects that the products and services of its applied research are technologically sound and relevant to clients’ needs. The process should establish a clear connection between applied research and revenue-generating potential by identifying a clear path to outcome commercialization prior to grant approval. ASU has a critical role to play in helping CAS/UETP (and CAS/NUST) researchers develop best-practice approaches to product development and commercialization. Adapt the promotion process for faculty to emphasize applied research and commercialization: To encourage faculty to invest their time in applied research and commercialization opportunities, CAS/UETP, with the support of HEC, should amend its promotion criteria for CAS faculty. Revised criteria should expand the traditional emphasis on publishing research in academic journals to formally recognize industry-focused, applied research in promotion decisions. This shift will provide a critical, yet missing, incentive for faculty to undertake the kind of applied research and commercialization opportunities that will move the center toward full sustainability. Adapting promotion criteria will also position CAS/UETP to strengthen and consolidate its brand as a valued resource for sector organizations, both public and private. Build capacity to deliver nondegree training, online learning, and blended learning opportunities: With the increased size and capacity of CAS/UETP faculty, the center should consider developing the capacities to start offering a portfolio of nondegree training programs, online learning, and blended learning options. Online and blended learning options will allow CAS/UETP to pursue revenue sources that are less constrained by limited faculty time and classroom space. The center will need to undertake a market survey to understand demand and to design a more complete portfolio of training courses. Additionally, the center should design and implement incentive plans to secure faculty participation in course design and delivery. Commercialize CAS/UETP labs for testing: The labs have the technology needed to serve as testing platforms for outside organizations. To realize and optimize the commercial potential of the labs, CAS/UETP should collaborate with USAID and ASU to develop a comprehensive strategy including price modeling and utilization schedules. Additionally, dedicated technical staff must be hired and trained to secure certification of equipment and perform tests. Developing the commercialization strategy will require market research, pricing plans, staff development, and certifications. Recommendations for CAS/MUET Secure CAS/MUET post-activity status as an autonomous center of excellence: Although the current university leadership supports the present CAS/MUET governance model, it is essential to formally secure the structure for the longer term. Without formally establishing the governance model, the center’s ability to forge innovative sector links and solutions could be at risk if future university leaders decide to change governance structures. HEC notes that the center must demonstrate that the structure is essential to developing the CAS/MUET brand name and to generating revenue. Initiate a flagship training program: Based on the foundational capacity of the center in training, CAS/MUET should consider establishing mechanisms to ensure that it contributes meaningfully to sustainability. These include a fully developed marketing and sector outreach plan, as well as a team able to implement this plan and build relationships with sector organizations. To establish these mechanisms, U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 5 CAS/MUET will need to work with the University of Utah to ensure that the marketing and outreach planning processes are pursued rigorously during the final year of the activity. This work should be guided by the existing market needs assessment study. The center can use remaining activity funds as appropriate to build internal capacity and aggressively begin outreach efforts as quickly as possible. Launch and scale the planned drinking water initiative to generate revenue: The center has the initial resources and expertise available to launch this initiative, although over time ongoing reinvestment in capacity will help the center scale the initiative and its revenue. Implementing this initiative can also provide the experiential knowledge necessary to identify opportunities to develop and launch similar initiatives. General Cross-CAS Recommendations for USAID and HEC The following recommendations are for USAID and HEC to consider to keep the activity focused during its final year so that outcomes have the greatest chance of being sustained. • Continue to challenge centers through a clear, outcome-oriented focus on sustainability. USAID and HEC can collaborate proactively to find creative ways to promote the sustainability of each center. • Ensure that centers develop realistic and comprehensive marketing, outreach, and lab commercialization plans, including securing the resources to implement those plans. • Create a career services office at each CAS. • HEC can help facilitate discussion of the best structure for CAS governance going forward. • Promote greater linkages across the CAS network, including collaborative research, joint programming, and resource sharing. • USAID should encourage HEC to continue the CAS activity focus on gender equity once USAID funding ends. EVALUATION PURPOSE AND QUESTIONS USAID/Pakistan’s Education Office requested this midterm evaluation to assess the performance of the U.S.-PCAS project. The evaluation was meant to gather lessons learned for each CAS and identify areas for improving implementation. Specifically, the evaluation has the following learning objectives: 1. To learn about the CAS U.S. and Pakistani university partnerships and identify areas for improvement; 2. To assess overall project management and learn about the areas for improvement for each CAS; and 3. To understand the sustainability aspects of the project and identify opportunities for improving prospects for sustainability. Lessons from the evaluation are intended to improve the performance of each CAS during the remaining life of the project and beyond. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 6 Evaluation Questions The evaluation focused on assessing the effectiveness of university partnerships; overall project management; and sustainability of the results, including the role of governance structures. The evaluation also examined each component of each CAS separately to measure progress toward achieving the goals of each CAS and of the overall project. The evaluation statement of work (Annex 1) specified four evaluation questions, which the evaluation team further elaborated in the assignment work plan (Annex 2). 1. How was the partnership experience in general for both Pakistani and U.S. universities under each CAS? 2. How effective is each component of each CAS in achieving the overall project objectives? 3. How does each of the partner universities perceive/approach sustainability for the CAS activities/results? 4. How are the universities utilizing their governance structures to sustain the CAS activities/results? PROJECT BACKGROUND Pakistan faces crises in the energy, water, and agriculture/food security sectors. However, Pakistani universities have limited capacity to produce the applied research necessary to address issues in these sectors, and their graduates do not possess the technical skills necessary to fill positions in public and private sector enterprises that work on these issues. Because these sectors are a high priority in the bilateral relationship between the U.S. Government and the GoP, the GoP requested the U.S. Government’s assistance to address the institutional capacity and workforce skill gaps. In response to the GoP’s request, USAID/Pakistan, in collaboration with HEC, funded four CASs—two in energy, one in water, and one in agriculture. As part of U.S.-PCAS, three American universities provided technical assistance and support to these centers. The CAS in Agriculture and Food Security at the University of Agriculture, Faisalabad, supported by the University of California Davis, ended in October 2017 and is not included in this evaluation. Of the three centers covered by this evaluation, one Center for Advanced Studies in Energy (CAS-E) is located at NUST, and the other at UETP, and both receive support from ASU. The Center for Advanced Studies in Water (CAS-W) is based at MUET and receives support from the University of Utah. CAS is a five-year project intended to improve the capacity of Pakistani higher education institutions in finding innovative solutions to some of Pakistan’s greatest development challenges through applied research; produce graduates who are better prepared to meet the needs of industry, government, and their communities; and provide leadership in policy dialogue. Theory of Change The theory of change presented in the statement of work (Annex 1) is: 1. IF the Pakistani universities in the CAS network learn from each other’s experiences on how to assess changing market needs to adapt and improve the curriculum, better deliver applied and U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 7 policy research, and increase the quality of their faculty without relying entirely on public or donor financing, 2. THEN the results will inform other Pakistani universities as well as HEC to identify good practices and promising opportunities to best align and integrate university research with clients in the public and private sectors, both within and beyond the project’s key sectors. EVALUATION METHODS AND LIMITATIONS Data Collection The evaluation employed a mixed-methods approach that included collecting quantitative and qualitative data from multiple sources (e.g., project documents, CAS faculty and students, U.S. partner universities, partner organizations, stakeholders, and experts). The evaluation team collected primary data through online census surveys of CAS faculty, current CAS students, and CAS graduates; group interviews with current CAS students and graduates; and key informant interviews with various stakeholder groups including CAS faculty, implementing partners, and U.S. partner universities. All told, the team interviewed 72 key informants and 65 individuals in group interviews, and surveyed 671 faculty members, students, and graduates. The mixed-methods approach ensured multiple levels of triangulation to strengthen findings. A key feature of the CAS midterm evaluation was the effort to ground findings and recommendations in the realities and experiences of each CAS as a distinct entity. To this end, the evaluation team conducted findings validation workshops with stakeholders from each CAS before presenting the findings to USAID. Similarly, the team organized a recommendations workshop to provide CAS stakeholders with the chance to explore opportunities and develop actionable recommendations that would help ensure sustainability of activity outcomes once USAID funding ends. Both the validation and recommendations workshops were designed to embed collaborating, learning, and adapting (CLA) approaches into both the process and use of the evaluation as a strategic learning tool. Annex 4 provides additional information on the data collection approach. Annex 10 reproduces the survey questionnaires and interview guides. Data Analysis The surveys yielded quantitative information regarding the partnership experience, exchange programs, joint and applied research, lab utilization, curriculum reform, governance, graduate training, academia￾industry partnerships, linkages to industry, gender issues, employment of graduates, and sustainability. Qualitative data from the group and key informant interviews augmented the quantitative data from the surveys by providing additional depth of understanding, answering the questions of how and why reported changes occurred. The evaluation team used descriptive statistics (frequencies, averages, and cross-tabulations) to summarize characteristics of respondent groups and their perceptions of the project and of each CAS. The analysis disaggregated findings by CAS, university (NUST, UETP, MUET), and, when applicable, gender. The team used relevant statistical tests to examine the statistical significance of results. Additionally, it compared results on gender issues to those of the baseline. The evaluation team used MAXQDA, a qualitative data analysis software package, to manage and identify patterns in the qualitative data using deductive and inductive coding systems. Deductively, the team developed a provisional code list based upon background documents, evaluation questions, and data collection instruments. As the team coded interview transcripts against this provisional list, other U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 8 (inductive) codes emerged progressively from the data itself. Such inductive codes were empirically grounded in the data and helped identify more nuanced patterns in the data. The team used an explanatory approach to integrate quantitative and qualitative data. In this analysis, qualitative findings helped explain trends and findings in the quantitative data. That is, the quantitative data described what happened, and the qualitative data provided why and how explanations for the quantitative results. The integration was designed to triangulate the quantitative with the qualitative data and help explain how and why CAS components changed the skills of faculty, students, and alumni, or how the CAS interventions worked—or failed to work—as expected. Evaluation Limitations Notwithstanding the efforts of the evaluation team to obtain valid and reliable data, the evaluation has the following limitations. 1. Response errors: Interview data constituted the primary source of information for answering all evaluation questions. Interview data are subject to error, and perhaps bias, when either interviewers or interviewees do not accurately and consistently interpret questions or responses. The team mitigated this source of error by pretesting the instruments, training enumerators and interviewers, and systematically triangulating evidence from a variety of methods and sources to minimize potential bias and enhance the validity and reliability of findings. 2. Low response rates: Online surveys (of CAS faculty, students, and graduates) typically suffer from low response rates, which may introduce bias. To mitigate this potential source of bias, the evaluation team worked closely with CASs to access faculty members, students, and graduates. Additionally, enumerators made several attempts to contact respondents. Consequently, the surveys achieved relatively high response rates—62 percent for graduates, 74 percent for students, and 67 percent for faculty. 3. Attribution: The evaluation design does not accommodate rigorously attributing results to the U.S.-PCAS project. Instead, the team collected the information necessary to provide a level of plausible attribution, i.e., a plausible explanation for how project activities contributed to observed results. 4. Generalizability across sector organizations: The evaluation design specified interviewing 12 industry partners from the list of industry partners provided by the CASs. Though the representatives from the selected organizations had firsthand knowledge of the partnership and understanding of the CAS project, their assessment/responses are not generalizable across the whole sector. FINDINGS Findings for Evaluation Question 1: Partnership Experience How was the partnership experience in general for both Pakistani and U.S. universities under each CAS? This question examined the partnership experience from the perspective of both Pakistani and American universities. The team assessed the Pakistani universities’ level of satisfaction with the partnerships, the U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 9 various ways in which the partnerships benefited both Pakistani and American universities, the challenges of the partnerships, and opportunities for addressing those challenges. Satisfaction with Partnership Finding: Faculty, students, and alumni are overwhelmingly satisfied with the partnership experience. A substantial majority of CAS faculty, students, and alumni who responded to the surveys were either extremely satisfied or satisfied with their partnerships with the American universities (Figure 2). Similarly, all 22 key informants representing host university leadership and partner universities said they were satisfied with the partnership experience. FIGURE 2: SATISFACTION WITH PARTNERSHIP Source: PERFORM survey of CAS faculty, alumni, and CAS program students. Note: These findings are based on the following sample sizes: NUST faculty 10, alumni 57, and students 175; UETP faculty 14, alumni 6, and students 229; and MUET faculty 17, alumni 19, and students 122. Survey respondents from CAS-W were significantly more likely to be satisfied with the partnership than respondents, especially alumni, from CAS-E (either center) (the difference is statistically significant at p<0.10).2 Moreover, collectively for all CASs, alumni who graduated in 2016 were less likely (68 percent to 87 percent, p<0.05) than those who graduated in 2017 or later to be satisfied. For CAS/NUST, participants in the validation workshop suggested that these results may reflect the fact that the U.S. partner university, ASU, took some time to appoint a technical director, which may have delayed full implementation of the partnership arrangement. The data also show that junior faculty (lecturers and assistant professors) were more likely than senior faculty to be dissatisfied with the partnership. Five of 27 junior faculty members interviewed (two each from NUST and MUET and one at UETP) were dissatisfied, compared to none of the 14 senior faculty 2 The p-value represents the probability that the observed difference between the two groups is due to chance instead of true differences between the groups. In this case, the chance is less than 10 percent. This implies a chance of more than 90 percent that the observed difference reflects a real difference between the two groups. Standard thresholds for statistical significance are p=0.05 (a 95 percent chance that the difference is real) and p=0.10 (a 90 percent chance that the difference is real). Most analyses do not consider p-values greater than 0.10 to reflect suitably strong evidence of differences. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 10 members interviewed. Qualitative data suggest that junior faculty have concerns about maintaining salary levels, as well as the potential effects their focus on applied research may have on promotion prospects in a system that may not give the same weight to applied research as to theoretical research, and that they may offer pushback against being “mentored” and thus losing some autonomy in conducting their research. Female survey respondents of all types were more satisfied than male respondents—100 percent of female faculty members compared to 85 percent of male faculty members, 94 percent of female alumnae compared to 75 percent of male alumni, 3 and 92 percent of female students compared to 88 percent male students. Partnership Benefits to Pakistani Universities Finding: From the perspective of Pakistani universities, the CAS partnerships contributed to building research capacity, establishing linkages with industry, improving curricula and teaching, and changing the institutional culture of the host university. The evaluation team’s interviews indicated that the partnerships built researchers’ skills and improved their capacity to conduct research in general, and applied research specifically (20 of 35 individual interviews with representatives of the CAS faculty, CAS administration, and host-university leadership and 11 of 18 group interviews with alumni and students). When asked how the partnerships had improved research capacity, faculty and students explained that they learned how to write proposals, secure funding, conduct research collaboratively, operate various research apparatus in their field of study, and conduct client-focused research. They also mentioned that they were collaborating with researchers at other universities, both inside and outside Pakistan. Respondents across the spectrum of those involved with CAS believed that the partnerships improved curricula and teaching quality (11 of 18 faculty members, 11 of 17 representatives of university leadership and CAS administration, and 9 of 18 group interviews with alumni and students). The evaluation team did not find any significant differences in these opinions among the three centers, or between the water and energy sectors. “The partnership with ASU was excellent, and they have been providing great help in terms of designing courses for both the M.S. and Ph.D. levels.” – Representative of host university leadership Interviews with representatives of the American partner universities and HEC confirmed that the partnerships played a pivotal role in improving the research and development capacity of the centers (9 of 16 respondents). Members of the sector organizations with which the CASs collaborated on research also agreed that the partnerships improved the research skills of CAS-based researchers. Respondents, including representatives of sector organizations, explained that improved applied research skills and research capacity allowed the centers to strengthen their linkages with the sector organizations, which contributed to strengthening the water and energy sectors (22 responses) in Pakistan. “[T]here was immense need for research … in the water management sector in recent years. In the past, there had been good water research institutions, but they had been ‘washed away.’ The research done by CAS-W in the last two years has ‘broken the ice.’” – Representative of sector organization Project documents reported that the American universities worked closely with their Pakistani partners and industry stakeholders to review and reform curricula. The American universities’ professors also 3 The difference is significant at p<0.10. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 11 worked with and mentored their Pakistani counterparts on teaching methods, which in turn improved the quality of teaching. “[We provide] training to faculty when we go to Pakistan and do faculty mentoring. We also work with faculty remotely in terms of their lecture content; we have Skype calls with them; I share my teaching materials with them. I exchange my course materials as well.” – Utah faculty member Representatives of the host universities’ leadership and administrations also felt that their engagement with the U.S. universities was changing the organizational and institutional culture of their universities (50 percent of 4 respondents from NUST, 33 percent of 6 respondents from UETP, and 100 percent of 7 respondents from MUET). Because of this interaction, the institutions were adopting best international practices in institutional governance, logistics, conducting applied research, providing solutions to industry problems, curriculum development and quality of teaching, and project management. “[Due to CAS] our systems improved…[when] we started working on CAS, our logistics were also below par. We improved that facility and established a women’s hostel. Moreover, we have a day care center, which was a new concept here, but we did that. After this we had gender balance.” – Representative of host university leadership In 36 key informant and 18 group interviews with CAS representatives (faculty, administration, students, and alumni), 18 sessions or interviews (61 percent of those at MUET, 28 percent of those from UETP, and 11 percent of those from NUST) noted that their partnerships with the American universities improved the prestige of their universities and degree programs. MUET students and alumni, in particular, felt that the degree program of the MUET center was prestigious and would elevate the ranking of their university. “Previously, it [MUET] was not considered a high-ranking university for Ph.D. and M.S. scholars. … The center improved the image of MUET. … The HEC ranking of MUET will improve because of CAS-W.” – CAS-W students Partnership Benefits to American Universities Through interviews, the evaluation team found that the partnerships benefited the American universities in two ways: recruiting and improving their understanding of Pakistan. Seven of 15 interviewed representatives of ASU and Utah mentioned that the partnerships with the Pakistani universities helped them recruit graduate students and obtain their assistance in conducting research (7 responses). “ASU always tried to engage and identify the best students in China and India, they have campaigns to travel to these two countries, but after this [CAS] we have a strong relationship with Pakistan and we are planning recruitment campaigns [to get] the best talent from Pakistan.” –ASU representative Four of the 15 respondents, representing both ASU and Utah, noted that the partnership increased their interest in Pakistan and provided an opportunity to understand Pakistani culture and its people. “We have mobilized a very large faculty at Utah. This is not a faculty which has previous experience working on Pakistan. … The only reason that they are increasingly engaged is because they see it as a potentially productive partnership, as something that is going to produce significant results.” – Utah representative “The CAS partnership has benefited ASU in many ways. In the exchange program, students bring their culture, learning, and experiences to ASU, which is also a way to understand this U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 12 region and its problems. … this provides a learning opportunity to faculty.” – ASU representative Partnership Challenges Representatives of both American and Pakistani universities identified challenges in working with their respective partners. From the perspective of the host universities, these problems stemmed largely from cultural misunderstandings and communication issues. Six respondents from Pakistani centers—four of them from CAS/MUET—complained about the directive behavior of American partners. They explained that the relationship has become more a “subordinate-boss” relationship than being a “partnership.” “Instead of helping CAS do things, Utah took the lead in doing some things. It resulted in micromanagement, especially during the first two years …. [Unfortunately], at the start, a boss￾subordinate relationship was established between CAS and Utah. This [boss-subordinate relationship] has countered the good work done under the project.” – MUET representative Seven of 13 respondents from UETP complained that their American partner failed to tailor the activities to the specific needs of UETP and the region. This reflects the issue, discussed later in this report, that ASU’s “one-size-fits-all” approach ignored the different capacities and needs of NUST and UETP. “UETP and NUST are different in terms of capacity, governance, faculty, and location. … ASU has a standard formula for deploying their people, i.e., two for NUST and two for UETP, but they should have hired two extra people for UETP, as UETP had less capacity than NUST.” – UETP representative Twelve host university respondents—representing leadership and administration from all three universities—noted challenges related to communication and their American partners’ limited understanding of Pakistani issues and culture. They explained that their American partners tended to do things in their own way or make decisions that did not account for Pakistani circumstances or realities. “It was a major challenge for us to develop our understanding of research in the U.S. universities, and it was difficult to communicate with them [our U.S. partner] about the current situation, needs of industry, and market of the country.” – NUST faculty member From the perspective of the American universities, challenges included differences in academic culture and the capacity of local researchers. Seven representatives of American partner universities— representing both universities—remarked that the academic culture in Pakistani universities, which reflects public sector bureaucratic culture that resists change, is not innovative and is difficult to change. “The biggest challenge I think is [that the] academic culture in Pakistan is not very innovative. … So getting [MUET] people to take on substantive issues and try to address them more systematically [is a challenge], but …we are making progress.” – Utah representative Ten partner university respondents, representing both universities, noted difficulties in recruiting suitable candidates for positions with the centers. Furthermore, many qualified candidates were not willing to work in the remote areas of Jamshoro and Peshawar. Participants in the validation workshop mentioned that the delayed appointment of a CAS-E technical director seriously affected the partnership in the project’s initial years. They also reported that it was difficult to recruit qualified women in the traditionally male-dominated sectors of energy and water. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 13 “The biggest challenge was to attract good quality faculty members … [because of] location. I talked to many people and professors to come and join MUET; many of them rejected the offer. We lost [staff] sometimes because the locations were not attractive for top talent.” – Utah representative Moreover, the limited ability of international staff to travel to project locations, especially Peshawar and Jamshoro, due to security issues posed challenges to the partnerships. These limitations affected UETP the most because ASU teams were unable to travel to Peshawar. Working in different time zones and across geographic distance also affected the quality of the partnership experience, according to nine respondents. Conclusions: Partnership Experience Overall, the evidence suggests that both American and Pakistani universities are satisfied with the partnership. The evidence also suggests that CAS/MUET is more satisfied than the other centers. The partnership between CAS-E and ASU confronted issues, especially in the early years of the project, related to staffing. Many challenges within the partnerships stemmed from issues related to communication and academic culture. Despite the challenges, the partnerships contributed to significant improvements in applied research, research collaborations, lab utilization, curriculum reform, teaching, and linkages with industry. These improvements are attributable to the technical support the American universities provided—the centers have the potential to meet the applied research needs of their respective sectors and address Pakistan’s challenging issues in water and energy. The presence of the centers at the universities is also positively affecting the host universities as they learn from the centers’ experience and adopt best academic practices. The CAS and host university researchers are also collaborating with researchers at other Pakistani universities. These are clearly positive changes in the academic culture of Pakistan that indicate that the theory of change is working and bringing about the changes in outcomes as intended. Findings for Evaluation Question 2: Effectiveness of CAS Components How effective is each component of each CAS in achieving the overall project objectives? This section provides a project-level, cross-activity analysis of findings related to the main components of CAS—exchange programs, joint and applied research, lab utilization, curriculum reform, governance, gender, and the postgraduate training program. A separate annex (Annex 3) contains more in-depth, CAS-specific analysis of each component. Exchange Programs Finding: Students and alumni at each CAS report that participation in exchange programs had a positive impact on their research capability. In individual interviews, 15 of 20 university leaders, HEC representatives, and CAS administrators reported that the exchange program contributed to the professional development of awardees (students or faculty); 3 gave no response. Similarly, most surveyed faculty, students, and alumni who participated in exchange programs said that the program improved their applied research skills (Figure 3). Interviews conducted at each CAS support the perception that the exchange programs improved research skills. Of 11 group interviews with students across all CASs, 9 referred to the positive impact the exchange programs had on their research capability, particularly in helping them experience firsthand U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 14 the research process at a high standard. Alumni were equally positive, with five of seven alumni interviewed reporting that exchange programs improved their research abilities. “It was a great feeling for me to know what actual research is. In Pakistan you find a lack [of] research capacity. It was a great opportunity to engage directly with those faculty, to know how research [goes], how to achieve the objectives and, if we cannot achieve objectives, how to present the work in a positive way.” – NUST student FIGURE 3: EXCHANGE PROGRAMS IMPROVED RESEARCH SKILLS (% OF RESPONDENTS) Source: MSI faculty, student, and alumni surveys. Number of respondents: Faculty: NUST (n=10), UETP (n=14), MUET (n=17); Students: NUST (n=179), UETP (n=231), MUET (n=125); Alumni: NUST (n=23), UETP (n=4), MUET (n=16). “I am happy about the applied research being done [during the exchange program]. I received many opportunities after coming back from the exchange because of the skills I developed.” – MUET alumnus Finding: Faculty perceptions of the impact of exchange programs on their research skills vary by CAS. Faculty indicate that the structure of exchange programs has been problematic. Faculty perceptions of the impact of exchange programs on their own applied research are much more mixed. Interviews with faculty at each CAS help explain these results. Utah initially placed exchange faculty with students, leaving them feeling like chaperones. At ASU, the initial length of exchange programs was only six weeks, leaving insufficient time to focus on skills development. As one NUST faculty member observed, “The program was not about skill development or teaching methodologies, but introduction to various researchers.” Faculty at all centers now report that the structure of exchange programs has improved, and even faculty who have not gone on exchanges notice a difference in their colleagues who have participated in the programs. According to a MUET faculty member, “You can see a change in faculty’s dedication and commitment after the exchange program.” Nonetheless, some faculty members questioned the value of participating in exchange programs, given family obligations and the fact that a number of faculty members already have significant international experience from having obtained their degrees overseas. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 15 Finding: Unlike other centers, CAS/NUST allows faculty and students outside of CAS to participate in exchange programs. At NUST, the percentage of CAS students accepted for exchange programs after applying was significantly lower than at other institutions (28 percent at NUST versus 53 percent at MUET and 50 percent at UETP), although data from the alumni surveys do not reflect this disparity. Evidence from one alumni group interview at NUST provides a possible explanation for the lower acceptance rate among current NUST students: “NUST changed [its] policy and made the [exchange] program open for other departments. They were sending other departments’ students to exchange programs on CAS funding. Only 11 of the 20 students from our CAS batch went. Nine were from other departments, including seven from the School of Mechanical and Manufacturing Engineering.” – NUST alumni interview Discussion during the validation workshop confirmed that students and faculty of other NUST departments are permitted to participate in CAS exchange programs. Mission staff clarified that this decision was reached collaboratively between university leaders and USAID. This practice, particularly with regard to faculty exchanges, raises the question of whether opening a critical project component to non-CAS students and faculty may dilute efforts to build the center’s capacity and reputation for longer￾term sustainability. Joint and Applied Research Finding: CAS/UETP has a lower percentage of faculty engaged in applied research than do other centers. Applied research is a key focus of each CAS, and joint research projects between faculty at Pakistani universities and their U.S. partners are a way of building the capacity and research profile of each center. A majority of faculty surveyed were engaged in applied research, i.e., 9 of 10 at NUST, 9 of 14 at UETP, and 16 of 17 at MUET. Validation workshop participants explained that the percentage of UETP faculty engaged in applied research is lower because funding is open to faculty (and students) from other departments. The center maintains that this allows it to broaden its research scope. Finding: Both Pakistani and U.S. university partners benefit from joint research, but CAS faculty report some challenges in the way joint research is approached. Both U.S. and Pakistani university partners value the partnership as a way to strengthen applied research. The U.S. partners see it as an opportunity for U.S. faculty to better understand the Pakistani context. Of the 14 personnel interviewed from U.S. partners, 3 Utah and 2 ASU representatives discussed this value at length, noting that collaboration strengthens outcomes. For example, two of the Utah representatives noted that teams are now collaborating to find new funding opportunities so they can continue their work once the project ends. Nine of 18 faculty members interviewed across all centers highlighted the role that joint research plays in helping build or improve applied research capacity at the Pakistani universities. “Due to this funded project, the international engagement with another university is more useful than before. In the past there was only online collaboration going on with international partners. Now, with the increased interaction and exchange program, the quality of research has improved.” – NUST faculty member U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 16 Nonetheless, the partners have experienced challenges in joint research. A persistent theme across faculty interviews at UETP (3 interviews) and NUST (3 interviews) was the limited funding for the Pakistani side of the joint projects (which is less than the U.S. partner receives). At MUET, three of five faculty members mentioned the difficulty of collaborating across distance and time zones. One noted that it is ultimately up to the individuals involved to figure out how to collaborate: “CAS [U.S.-PCAS] provides a platform, but not in a systematic way. The main challenge is individual initiative. Once you take initiative, there is no challenge.” Finding: Faculty, students, and alumni agree that CAS applied research is aligned to the needs of sector organizations, though sector organizations vary in their opinions of CAS research quality. Faculty, student, and alumni survey respondents largely agreed (strongly agreed or agreed) that CAS applied research aligns to sector needs (Figure 4). FIGURE 4: ALIGNMENT OF APPLIED RESEARCH WITH SECTOR NEEDS Source: MSI faculty, student, and alumni surveys. Number of respondents: Faculty: NUST (n=10), UETP (n=14), MUET (n=17); Students: NUST (n=169), UETP (n=222), MUET (n=118); Alumni: NUST (n=57), UETP (n=6), MUET (n=19). The interviews with sector organizations (selected randomly from a list of sector organizations provided by each CAS) provided another perspective by asking respondents whether their organizations were satisfied with the quality of applied research from the CAS. Satisfaction varied substantially (within the limits of a small sample) across centers, with one of five organizations associated with NUST, one of three associated with UETP, and three of four associated with MUET satisfied with the center’s research quality. This was a closed-ended question asked to a small sample of organizations, and therefore these results should not be taken as the sole determinant of the state of applied research at any CAS. That said, sector organizations’ comments in the interviews provided interesting insights into the quality of applied research. One of the four organizations that was not satisfied with the research quality at CAS/NUST questioned the center’s expertise: “They are just taking cover of industry that they are involving industry [in projects] however it [applied research] is not actually happening. They [CAS faculty members] have not U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 17 approached me saying that they have any expertise to solve our [energy sector industry] problems so we do not have any confidence [in engaging them in projects].” Organizations associated with CAS/UETP were impressed with the center’s potential and resources, but they awaited concrete outcomes. “It is limited yet, and I’m trying to link them up [with our projects] because they have relevant equipment and expertise.” Finally, organizations were quite satisfied with CAS/MUET, possibly because of the way the center approached applied research. According to a sector organization representative who worked with MUET, “The research they are conducting on sea water intrusion is very scientific. They are in possession of raw data which we [a sector organization] can use.” Finding: Approaching applied research as a process to collaboratively engage organizations rather than as a product to be presented may help build confidence in research quality. The CAS baseline study questioned whether applied research was a product or a process, and this distinction appears to be the issue in sector organizations’ satisfaction with the centers’ research quality. Unlike CAS/NUST and CAS/UETP, CAS/MUET has an explicitly defined, and published, research agenda designed in collaboration with stakeholders around Sustainable Development Goal 6 (SDG6) established by the United Nations in 2015. 4 This agenda shapes not only how the center engages with others in the sector, but also how Utah works with faculty and students to build the long-term capacity of the center. Though CAS/NUST and CAS/UETP have articulated research goals, they have not explicitly linked these goals to longer-term capacity development objectives for CAS students and faculty. This is seen in the way both CAS/NUST and CAS/UETP encourage external researchers to apply for CAS research grants. Although each CAS engages with sector organizations through stakeholder meetings, organizations appear to seek deeper collaboration: “We can collaborate for research. … For university professors, they need some interesting work that they can publish. It’s strange that we are willing to provide them funds but they are not ready to research with us.” – Representative of sector organization working with NUST Lab Utilization and Effectiveness Finding: Each CAS has yet to finalize a lab utilization strategy to guide lab access by key stakeholder groups, including students. Interviews with students and sector organizations suggest that each CAS still needs to develop a lab utilization strategy to embed labs into teaching, research, and outreach (commercialization) efforts. Students at each CAS indicated their access to equipment was limited. As one MUET student explained, “We do not know how to operate some equipment. Instruments are available, but only one person knows how to operate the equipment.” Sector organizations interviewed noted that they were aware of CAS labs but had not yet received information on how they could access them. Participants in the validation workshops indicated that centers were developing lab utilization strategies. In two of the three validation workshops, faculty members noted that the labs are intended to be research labs rather than teaching labs. They explained that, although master’s students should be able to access the facilities, this should not imply that they have direct access to equipment and machinery. To maintain the precision of equipment and protect machinery from careless use, technicians should conduct tests alongside students. Such structured usage is critical for labs if they are to receive and maintain accreditation for commercial use. 4 SDG6: Ensure availability and sustainable management of water and sanitation for all. See “Sustainable Development Goal 6,” United Nations, https://sustainabledevelopment.un.org/sdg6. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 18 Curriculum Reform Finding: CAS faculty believe that curricula are aligned with sector needs and that the balance of theory and practice in teaching are strong. Students and alumni are less convinced. Survey results indicate that faculty are more likely than students, who are more likely than alumni, to say that curricula are aligned with sector needs and that teaching adequately balances theory and practice (Figure 5 and Figure 6). FIGURE 5: CURRICULA ALIGNED WITH SECTOR NEEDS (% OF RESPONDENTS) Source: MSI faculty, student, and alumni surveys. Number of respondents: Faculty: NUST (n=10), UETP (n=14), MUET (n=16); Students: NUST (n=166), UETP (n=222), MUET (n=116); Alumni: NUST (n=56), UETP (n=6), MUET (n=19). Participants in 10 of 18 group interviews with students and alumni supported the idea that curricula were aligned with sector needs, while in 6 group interviews they suggested that they were not fully aligned. With regard to appropriate balance of theory and practice in teaching, interviewees suggested that the preferred balance depends on the individual, with several student groups noting that faculty who have international experience (including through the exchange program) tended to be more hands￾on in their teaching methods. Students also noted the need for more faculty equipped to teach policy￾related issues. Finding: Each CAS is very positive regarding the curricular support it receives from U.S. partners. Sector organizations are also positive on curricula, despite some concerns about how the CASs engage them in the development process. Centers viewed the curriculum revision process as largely positive. All host universities highly valued the contributions of ASU and Utah to the revisions of their respective curricula. In an illustrative comment, a UETP faculty member captured the sentiments raised in many faculty interviews across each CAS: “The input of ASU regarding our curriculum is tremendous. I really appreciate that because they have always commented on curriculum.” U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 19 FIGURE 6: TEACHING BALANCES THEORY AND PRACTICAL APPLICATION (% OF RESPONDENTS) Source: MSI faculty, student, and alumni surveys. Number of respondents: Faculty: NUST (n=10), UETP (n=14), MUET (n=16); Students: NUST (n=166), UETP (n=222), MUET (n=116); Alumni: NUST (n=56), UETP (n=6), MUET (n=19). Sector organizations also viewed the curriculum support as relevant, despite their desire to be engaged more directly in the curriculum design process. A sector organization representative explained: “There is no mechanism or system in place to ensure the relevance of curricula to industry needs. They do note our suggestions, but I am not sure they take our feedback seriously.” Such comments seem to have more to do with the processes by which sector organizations are engaged, rather than the quality and relevance of the curricula themselves. Public-Private Partnerships Finding: Centers tend to focus on signed MOUs as the key indicator of effective partnerships rather than on cultivating robust relationships. Of 28 faculty and administrative staff members across the 3 centers, 23 discussed various examples of their center’s efforts to establish relationships with sector organizations, both public and private. Signing MOUs emerged as a central theme across these interviews. An illustrative comment from NUST captured the focus on MOUs: “We have three MOUs, and the fourth one is approved, to be signed in one or two months. We have support from our partners, and some have provided funding and support for projects.” Similarly, U.S. partners noted their focus on helping each CAS develop MOUs. Despite the effort to develop partnerships, half of the 14 U.S. partner university representatives interviewed discussed challenges the centers face in trying to sustain partnerships after MOUs are signed: “The biggest challenge is how to sustain partnerships. There is limited capacity and attitude, but they are learning, and things are changing in the way they approach partnerships.” – Utah representative U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 20 “ASU was given the initiative of leading the public-private partnership initiative. [But] it has been managed mainly by NUST. I do not think the approach is how public-private partnerships should be. All they do is sign MOUs. After that, nothing happens.” – ASU representative Finding: Both sector organizations and CAS faculty seek partnerships based on an ongoing commitment to relationship building. Interviews with sector organizations indicated a desire to build strong, collaborative relationships that evolve over time. Of 12 sector organization representatives interviewed, 7 highlighted the importance of building a partnership around the needs and strengths of each partner. One sector organization representative explained, “We feel that so far our partnership is successful. However, the results are just at lab scale, while my management wants to see results at a commercial scale.” Five of the 12 focused on the importance of collaborative relationships that evolve over time. Faculty interviewed at each CAS expressed the view that partnerships should focus on cultivating deeper relationships, though centers tend to be focused more on MOU development. “In applied research, MOUs don’t work. … They become formal and have no output. Faculty members say there is no need to sign MOUs—there is [a] need to engage the people at ground level.” – UETP faculty member Graduate Training for Employment Finding: Sector organizations are satisfied with the quality of CAS students, particularly with their research capacity and professionalism. Of 12 sector organization representatives interviewed, 7 said they were satisfied with the quality of recruited CAS graduates. Respondents suggested that the research capability and commitment of CAS students differentiates them from graduates of other programs. As a power sector respondent commented, “Students from CAS/UETP are already doing work on research. … Government organizations and power sector [organizations] need such specialized graduates. This will help to improve performance.” Similarly, a water sector respondent noted how the professionalism of CAS/MUET graduates differentiates them from other university graduates: “When Karachi University visited us, they said we will do this and that, but there was no follow￾up. CAS/MUET sent two students who remained for a whole day and even stayed till night. … They came, got samples, and went away. You have to get your clothes dirty, there is a smell, but they stayed till the end, did their work completely with ease.” Finding: Students at NUST, where a system is in place to help students secure internships, are most satisfied with their internships. Student satisfaction at MUET and UETP is lower, and students indicate that there are no formal systems in place to help them find internships. Overall, most students were satisfied (either extremely satisfied or satisfied) with their internships, although levels of satisfaction were significantly lower among UETP students than among NUST or MUET students. Eighty-seven percent of 154 NUST students, 76 percent of 29 MUET students, and 35 percent of 58 UETP students reported being satisfied. When asked about their experiences with internships, NUST students explained that there were established systems in place to help them get internships. As one NUST student commented, “We get internship opportunities at different organizations, and our teachers and industry liaison office support us.” The only evidence to elucidate the lower satisfaction of UETP students appears to suggest that support for U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 21 internships is not consistent across departments. According to one UETP student, “The students enrolled in energy management and sustainability were offered internships. However, students enrolled in renewable, electrical, and thermal engineering were not offered any internships.” Responses from MUET students suggest that the center lacks established systems for placing students in internships. One MUET student said, “CAS did not help us in getting internships—they just provided recommendation letters.” Another student said his supervisor helped him find an internship based upon his research. These comments suggest that internship support at UETP and MUET may be less formal and based upon the faculty’s personal connections. These findings suggest that established systems are important for placing interns. Some students are able to secure internships based on informal means (such as faculty connections), but this results in inconsistent experiences for students. As the next finding suggests, that a lack of established systems is an even greater issue for postgraduation job placement. Finding: Postgraduation job placement is weak at each CAS, and students often struggle to convey the value of their degrees to potential employers. Most students and alumni at each CAS were relatively dissatisfied with postgraduation job placement (Figure 7). FIGURE 7: SATISFIED WITH JOB PLACEMENT (% OF RESPONDENTS) Source: MSI faculty, student, and alumni surveys. Number of respondents: Faculty: NUST (n=10), UETP (n=14), MUET (n=16); Students: NUST (n=166), UETP (n=222), MUET (n=116); Alumni: NUST (n=56), UETP (n=6), MUET (n=19). The data in Figure 7 suggest a disconnect between institutional (i.e., faculty) perceptions of placement services offered to students, and the actual experiences of students and alumni, who are the ultimate consumers of job placement services. The higher levels of satisfaction among students than among alumni suggest that placement systems may be improving. Still, there is more work to do: “There is not a system for finding jobs. … There is no career center at CAS-W, and the Office of Research, Innovation, and Commercialization only provides some information. Jobs are obtained through the struggle of the student, often on the basis of their B.S. degree.” – MUET students U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 22 “Market placement at CAS is poor. Students do not have any support in this regard. There should be a career office here, which has links with organizations.” – UETP students “Yes, systems are in place for internships, and we know the systems to get an internship. We cannot say that about job placements though.” – NUST students This evidence suggests that the centers do not have established systems for placing graduates in jobs. The positive perceptions of sector organizations about the caliber of CAS graduates suggest that there is a ready market for graduates and that the centers have an opportunity to strengthen their brand by expanding partnerships to provide the trained workforce that sector organizations require. Gender Equity Referring to findings from the baseline, the evaluation team explored how the high targets for gender equity set by USAID at the outset of the project have helped promote the true participation of women in what are traditionally male-dominated careers in the water and energy sectors. Finding: U.S.-PCAS’s gender efforts have been more successful with respect to students than faculty. Women’s representation on the faculty of each center is still a challenge. Based on the number of female students, alumnae, and faculty members reported by the centers, it appears that the target of 50 percent female inclusion is ambitious. TABLE 2: FEMALE FACULTY, STUDENTS, AND ALUMNAE AT CAS CAS Institution Female Students Alumnae Female Faculty NUST 18% (41 / 230) 16% (16 / 99) 14% (3 / 18) UETP 11% (34 / 309) 14% (1 / 7) 0% (0 / 18) MUET 36% (57 / 159) 31% (11 / 35) 23% (7 / 30) Total 19% (132 / 698) 20% (28 / 141) 15% (10 / 66) Source: Data provided by each CAS. Note: Faculty includes all faculty members with full-time, visiting, or adjunct status. Table 2 indicates that MUET has engaged a substantially larger percentage of women as students than have the other centers. This result may be associated with female-friendly initiatives taken at MUET. “Now MUET has a gender policy because of CAS-W. A researcher has written a paper on why women are not in the engineering profession. … CAS-W has set up a good model for selecting women students and providing facilities for their convenience.” – Sector organization representative “The center is setting an example for other universities in Pakistan in terms of gender equity, as females are given free accommodation and a women’s resource center.” – MUET Student Although NUST has not been as successful as MUET at engaging female students, one sector organization representative acknowledged that the center is making an effort to increase women’s participation: “They have even done outreach to recruit female students, which NUST does not normally do because they felt they could get students on merit. However, this was the first time they went out from the university to increase female enrollment.” – Sector organization representative U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 23 Conversely, UETP representatives explained the low participation of female students and almost nonexistent female faculty as a function of the cultural environment in Khyber Pakhtunkhwa (KP): “We tried our best to take women on board, but there are many reasons [that it is difficult]. First, in KP you might not find ladies who are willing [to become students or faculty in Peshawar]. Qualified people will always go to Islamabad or Lahore. Secondly, our advertisements encourage female applicants, but no women applied for top management roles.” – CAS/UETP leadership representative Finding: Male students and alumni believe that women face fewer barriers to employment now than at the time of the baseline. Women, however, are not so positive. The baseline assessment revealed several persistent beliefs about the barriers women would face pursuing careers in the water and energy sectors. For example: “A woman as a field engineer cannot do her work. The labor will not listen to her.” – Baseline, water sector organization “Unfortunately, our university is male-dominated, and there is less involvement of women in the research process. Women’s research input will, of course, be in proportion to their number in the university.” – Baseline, UETP faculty member These findings provide an interesting context to frame newer gender-related findings in each CAS. Figure 8 illustrates the percentage of survey respondents who agreed or strongly agreed with the statement that women face greater barriers to employment than do men. Men were more positive than women about women’s prospects in the job market, and they became even more positive between the baseline and midterm. Furthermore, alumnae, who presumably have directly experienced barriers, reported little change in perceived barriers between the baseline and midterm. FIGURE 8: WOMEN FACE GREATER BARRIERS TO EMPLOYMENT THAN MEN (% OF RESPONDENTS) Source: MSI faculty, student, and alumni surveys. Number of respondents: Midterm Evaluation [Alumni: (F=17, M=64); Students: (F=109, M=395)], Baseline [Alumni: (F=64, M=323); Students: (F=184, M=339)] U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 24 Finding: There are no women in senior decision-making roles at CAS. In 23 of 50 key informant interviews and 9 of 18 group interviews, CAS faculty, leadership, students, alumni, and representatives of U.S. partner universities confirmed that CAS governance is male￾dominated and women do not have any decision-making role. Women have junior roles in administration—e.g., in project management units (PMUs)—and in academics, with the exception of CAS/UETP, which has no female faculty. “From a governance standpoint, there are no women in a decision-making role. But they are part of the faculty governance, so their voices are heard in the faculty governance process. However, the governance of CAS-W is only men.” – Utah representative “If you look at the governance structure, there are no women in higher strategic positions, and there are no women in decision-making roles.” – ASU representative Interviews suggest that CASs have made progress on gender issues, although there is still more work to do, particularly in faculty recruitment. There is a degree of optimism as well. As a participant in a CAS/MUET alumni group interview commented, “Women are doing well in professional fields, but can’t attribute it to CAS. Maybe some years down the line, there will be women in decision-making roles owing to the present intervention.” Comments from female students support the wisdom of pushing the universities to engage women, as this comment from a NUST student illustrates: “Female students are encouraged to participate in this program. There was even diversity in the exchange program; two out of the five girls were from remote areas.” CAS/MUET vocalized its pride in adopting the gender policy developed by U.S.-PCAS. Interviews with sector organizations working with CAS/MUET reflected this; 75 percent of the respondents in four group interviews highlighted the center’s gender equity initiatives as something they valued. Conclusions: Effectiveness of CAS Components Each center has the freedom to implement the components of CAS in a manner that best suits the host university. Consequently, outcomes vary from one CAS to another. Overall, student exchanges have proven very effective in building research capacity, while faculty exchanges have been less effective. Joint research has also contributed to enhancing research capacity. Although each CAS is making an effort to pursue applied research, evidence suggests that the manner in which the centers approach such research can affect sector organizations’ perceptions of research quality. Partnerships provide an opportunity for growth, particularly if the centers move from focusing on signing MOUs to more concerted efforts at establishing longer-term relationships. Sector organizations seem pleased with both CAS curricula and the quality of graduates. This provides an opportunity for each center to strengthen relationships with sector organizations, particularly through the creation of formal job placement and internship systems that will enhance prospects for graduates, while meeting the talent needs of organizations. Finally, progress in gender equity has been uneven, although better at the student than faculty level. Holding each center to high goals for gender equity has had an impact on outcomes, and the focus on gender has caught the attention of sector organizations. Findings for Evaluation Question 3: Sustainability How does each of the partner universities perceive/approach sustainability for the CAS activities/results? To assess how each CAS perceives and approaches sustainability, the evaluation team first reviewed the sustainability plan that each center developed in accordance with its cooperative agreement. These plans U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 25 contain sustainability program targets as articulated by USAID. The team’s assessment of these targets determined that they are primarily output targets for indicators such as funding from nongovernmental sources; public-private partnerships and global development alliances; deep, long-term partnerships with the U.S. partner university; and other indicators. Annex 5 contains the complete list of output targets, as well as thelatest available data for each CAS in meeting those targets.To augment these data, survey and key informant interview instruments asked about sustainability. from this analysis were incorporated into each of the validation workshops,with one breakout discussion dedicated to sustainability and governance issues. Overall Findings Centers exhibit general confidence about their own sustainability, though stakeholders mainly define sustainability in terms of maintaining physical assets and infrastructure, not outcomes. Eighty-eight percent of 40 faculty members surveyed believed that their respective CAS was likely to continue to exist after current donor funding ended—40 percent said it was likely and 48 percent felt it was highly likely. A similar percentage (87 percent—62 percent likely and 25 percent highly likely) believed it likely that their CAS would generate other financial resources after USAID funding ended. In interviews with 32 university leaders, U.S. partner university representatives, and CAS administrators, 29 believed that “CAS [will] continue to exist after the current donor funding ends.” Overall findings from interview data suggested (and validation workshops later confirmed) that the confidence about each center’s prospects for sustainability derives primarily from plans to financially sustain physical assets (mainly the CAS building) and staff (salaries). Potential sources of future funding include endowments (mentioned by representatives of each CAS); HEC (mentioned by CAS faculty and administrators and U.S. partners); and donor, including USAID, support (mentioned by university leaders from MUET and NUST). When the evaluation team used the validation workshops as an opportunity to frame sustainability in terms of each center’s ability to produce desired (and valued) results over time (USAID’s definition of sustainability from ADS 201), center stakeholders noted that it was not the way USAID had articulated expectations for sustainability. The centers understood sustainability primarily in terms of output targets set by USAID for fundraising, partnerships (MOUs), graduate employment and internships, female participation, and diversity, as well as meeting operating costs for assets such as labs and faculty salaries. When discussing external funding sources, with several exceptions, most of the projected revenue sources seemed more anticipatory than secure. The following sections provide specific details regarding each center’s sustainability efforts. CAS – Energy (NUST Islamabad) Analysis of quarterly reports submitted by CAS/NUST indicates that the center has raised PKR 112.01 million, bringing it close to the USAID target of USD$1 million in external funding for sustainability indicators. However, further analysis indicates that this amount contains a mix of public sector (PKR 44.79 million) funding in addition to nonprofit (PKR 36.6 million) and private sector (PKR 30.72 million) funding. Thus, the funding sources reported appear not to be aligned to the original USAID definition of “funding generated from non-governmental sources such as private sector, foundations, foreign organizations and alumni groups.” Finally, emerging findings U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 26 Finding: Despite somewhat high confidence at NUST regarding the sustainability of CAS after USAID funding ends, plans currently appear to be conceptual rather than grounded in concrete measures. Nine of the 10 NUST faculty members the team surveyed felt it was somewhat likely or highly likely that CAS would continue after USAID funding ended. The same percentage felt that the center would be able to generate alternate financial sources to sustain itself. NUST students were slightly less optimistic about sustainability, with 76 percent of 166 survey respondents expressing the view that it was somewhat likely or highly likely that CAS would continue after donor funding ended. NUST leadership conveyed confidence in its ability to sustain the center financially: “We will continue to get funding from the Government of Pakistan. Our desire is to reduce the burden on the Government of Pakistan as much as we can, and for that we are making a technology center—this center will provide trainings, market solutions, and commercialization.” – NUST leadership representative However, NUST leadership respondents also hoped that USAID would continue or expand funding. One respondent noted, “We will try to retain the U.S. funding once this funding ends.” Regarding the aforementioned technology center, he also commented, “We have brought a $5 million investment for this center. If USAID supports us, we [can] take the center to the next level, we will be able to reach the target more quickly.” Interviews with seven CAS faculty members also demonstrated their belief that the center would be financially sustainable. All seven mentioned the commercialization potential of center resources such as lab testing and short training courses. Four of the seven faculty members and two administrators grounded their confidence for sustainability in their belief that government or donors would continue to support the center. As one administrator explained, “We will contact different development agencies and HEC [to] provide some funds to ensure sustainability.” Most of the faculty and administrator comments about alternative funding referred to future or potential sources of funding; few mentioned established sources. Three respondents, however, mentioned current revenue sources: testing services conducted by the materials lab, research funding received from the private sector, and work commissioned by the United Nations Industrial Development Organization and the World Bank. Further investigation suggests that the center may have secured the latter work prior to CAS. The sustainability of the partnership with ASU is also questionable. University leadership was hopeful, stating, “If we can provide them good research and teaching, we can assume ASU will be willing to support the center even after the funding ends in 2019.” However, an ASU representative was clear in noting, “Where time is needed from ASU faculty, it has to be backed up by funding. If there is no funding, there is no time.” A CAS faculty member echoed this sentiment by stating, “ASU says that when there will be zero funding, there will be zero collaboration.” This may explain why 40 percent of the 10 CAS/NUST faculty members the evaluation team surveyed believed that it was somewhat unlikely that joint research will continue after project funding ends. An interview with a different ASU respondent raised questions regarding the NUST sustainability plan: “ASU assisted NUST in developing the sustainability and strategic plan. Inputs from ASU in the strategic plan were not incorporated much by NUST. … The sustainability plan is very basic, as it does not have any vision and mission. It’s not a very thorough document. It could have been better if other stakeholders were involved in the development process.” U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 27 The same respondent went on to note that though ASU is pushing NUST to develop proposals for funding, NUST faculty and administrators have not yet approached donors, adding, “Perhaps because they are not well acquainted with the development sector, they do not know that getting funding is a regular process.” Finding: University leadership recognizes that it will be challenging to sustain key components of the project, though it also appears to make some potentially erroneous assumptions about the impact of not sustaining scholarships. Four of seven faculty members interviewed reported that the exchange programs would be the most difficult activity component to sustain, while two felt that sustaining applied research would be most difficult. Though no faculty members identified student scholarships as difficult to sustain, university leadership noted that, although the number of scholarships would fall, students would continue to be attracted to the programs since the cost of degrees at NUST “is already very low.” However, in two of the four student group interviews at NUST, respondents highlighted that it was the USAID-funded scholarships (and exchange programs) that attracted them to CAS. One group was quite specific about the challenges CAS will face once that funding ends: “If scholarships end, students will not be attracted to this department and the number of students will decrease … foreign funding is easily available for M.S. programs abroad.” When asked which CAS components they valued the most, three of four sector organization representatives cited applied research/labs, while one noted the CAS curriculum. Three of seven faculty members interviewed—and USAID representatives—mentioned plans to start an undergraduate program, a strategy they believed would increase revenue and enhance prospects for sustainability. One faculty member, however, noted that faculty had “mixed feelings” about this plan. The faculty members in the validation session emphasized that creating an undergraduate program could have a negative impact on the labs, saying, “CAS labs are research labs, not teaching labs.” They explained that, although graduate students (particularly Ph.D. candidates) can be trained to use the labs, opening them to undergraduates would be a mistake, as the expensive equipment would be put at greater risk. Finding: It does not appear that NUST leadership understands the critical link between sustainability and creating an environment where faculty and staff can build strong relationships with sector organizations. NUST respondents emphasized commercialization of labs and research as a cornerstone of sustainability. Interviews with sector organization representatives, however, raised questions about this strategy. “The liaison office is not in contact with us. We would love to fund their research. … It’s strange that we are willing to provide funds to them but they are not ready to research with us.” “They invite us to stakeholder meetings, and students display posters, but we do not get the chance to know details. … [Practically] nothing has been done for industry collaboration.” “We feel that CAS people are not proactive and the staff is busy with other tasks. We are more proactive with them, and the role should be vice versa.” Interview data suggest that faculty time constraints may play a role in the negative perceptions sector organizations have of CAS relationship building. Three of seven faculty members interviewed discussed the limited time they have available to contribute to developing a high-level research culture: “The professors are overly burdened with many duties. … The environment is not very research-oriented.” Leadership acknowledged this issue by mentioning at three separate points during interviews, “We push/drive faculty U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 28 to the limits.” During the validation workshop, several faculty members also noted that their significant operational responsibilities alongside their teaching loads compromised their research. They believed that if they were freed from these administrative burdens by a more developed administrative function at CAS, they would be able to bring in revenue through research projects and by conducting industry training. They further argued that faculty retention will affect sustainability: “Sustainability is directly linked to how much value you give to faculty.” CAS – Energy (UETP) CAS/UETP quarterly reports state that the center has raised PKR 12.5 million in external funds, approximately 11 percent of the USAID target of USD$1 million. However, additional detail provided by UETP indicates that this is solely from public sector organizations and thus does not align with USAID’s indicator definition of “funding generated from non-governmental sources such as private sector, foundations, foreign organizations and alumni groups.” Additionally, the center reports that it has established four public-private partnerships. The evaluation team viewed the CAS/UETP plan as the weakest of the three it reviewed. The plan was much more conceptual than specific and offered little detail on anticipated operating costs and how targeted sources of funding/revenue are expected to offset those costs. Finding: Stakeholder views on the sustainability of CAS are mixed. Despite the weaknesses described above, all 14 CAS/UETP faculty members surveyed believed that it was somewhat likely or highly likely that CAS would continue after USAID funding ended. They also all believed that it was somewhat likely or highly likely that CAS would generate alternative financial resources to sustain its activities. CAS/UETP students surveyed were also optimistic about the likelihood of sustainability, but less so than faculty. Sixty-five percent of 222 surveyed students believed that CAS would continue after USAID funding ceased. Interviews with key stakeholders revealed much more mixed perceptions of CAS/UETP’s prospects for sustainability. The representative of university leadership expressed confidence in CAS sustainability, saying, “HEC promises it will hire people on TTS [tenure track system] after the project ends. I don’t think there will be any problem once the project is over.” Yet the same respondent also contradicted himself and noted that some faculty members may leave once USAID funding concludes. In addition to HEC funding, he emphasized that the center’s ability to generate joint research projects with sector organizations will contribute to sustainability. One of the two administrative staff members interviewed echoed this sentiment, noting that the provincial (KP) Department of Science and Technology has funded various projects that, in conjunction with HEC funding, should ensure sustainability after 2019. Others on the administrative team were less optimistic. Two of four administrators interviewed said that sustainability could go either way. One explained, “I can say yes, and I can say no. If HEC provides funds according to the needs of the center and according to what they are planning, then yes.” Alternatively, the other noted: “I believe it will require lots of funds to sustain [the center]. UETP will not be able to provide sufficient funds, but if we connect to different donors and support organizations it will be helpful. But till now we have not done anything visible on sustainability.” Faculty perceptions of sustainability were similarly divided. Three of five faculty members interviewed discussed some potential major collaborations that will provide significant revenue for the center, including opportunities with the Aviation Designing Complex Institute, the Pakistan Council for Renewable Energy Technologies, the Federally Administered Tribal Areas (FATA) Secretariat, and the U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 29 Pakistan Science Foundation. The validation session confirmed that the Aviation Designing Complex Institute opportunity has already matured and a project, titled “Condition Monitoring and Predictive Maintenance of JF-17 Thunder using Machine Learning Techniques,” has started. Finding: Sustaining some existing project components will be a challenge, though there may be some opportunities to work around these challenges. Five of six faculty interviews discussed sustainability. Three faculty members and two of four CAS administrators noted that the exchange program would be difficult to sustain. One of the administrators discussed efforts to arrange exchange opportunities with universities in countries closer to Pakistan, such as Turkey. However, funding remains an important concern. The issue of CAS sustainability emerged as a concern in two of the four student group interviews conducted at UETP. Several participants believed that scholarships and the exchange program were deciding factors for many students who enroll in CAS, and without them CAS would be no different from any other UETP department. However, other students noted a recent trend in which individuals already employed in the sector have enrolled in CAS programs because of the attractiveness of the courses: “In our class, there are two students [who work] and they are not on scholarships. When people know that there is [a] focus on research [at] the center, they will come and it will be sustainable.” These comments emphasize the importance of CAS applied research and how challenging it may be to sustain. Two of the three interviews with sector organization representatives confirmed the importance they place on research and labs, while one of the three interviewees highlighted the CAS focus on gender as a critical component. Respondents in two of the six faculty interviews highlighted applied research as the component most valued by organizations but also noted it would be a challenge to sustain. Conversely, over 85 percent of faculty surveyed felt that it was somewhat or highly likely that applied research would continue. By comparison, 21 percent of 222 students surveyed felt that partnerships for applied research were highly unlikely to be sustained, the largest percentage among all CASs. These findings suggest two conclusions. First, applied research (and the labs that support it) is the CAS component that attracts the attention of sector organizations. However, there may be some longer￾term challenges in delivering on the promise of applied research. Second, though scholarships and exchange programs have been critical in attracting students to CAS as it established itself, some evidence suggests that a strong applied research profile could draw self-funded, already-employed students to CAS. As the center builds its brand and reputation, it may find that its student profile changes. As the center evolves, maintaining scholarships and exchanges may not be as crucial to sustainability. However, the sustainability of applied research remains a concern. Finding: There are opportunities to generate revenue, but the center lacks sufficient capacity to engage sector organizations. Faculty and administrative staff expressed a degree of optimism that faculty research and CAS labs can provide sustainable revenue streams. Three of six faculty interviewed noted that there are multiple projects in various stages of development that appear promising. Faculty and administrators supported the view that the core assets of the center are strong and that sector organizations are impressed when they visit CAS and observe the potential of labs and applied research. As one faculty member noted, “Applied research is the missing partner in Pakistan. We are meeting industrialists, and [working] on their problems, and offering solutions.” A representative of ASU noted that its work to establish technology centers at both UETP and NUST can strengthen the potential of both centers to offer commercial services to sector organizations as part of their sustainability planning. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 30 Potentially strong revenue-generating prospects identified by CAS/UETP’s new industry liaison may also contribute to sustainability. An administrator noted, however, that the center’s limited capacity to develop effective research proposals represents a challenge to turning prospects into revenue￾generating programs. Two faculty members articulated similar views but focused more on the need to develop a stronger consultation capacity for the center. Additional information provided during the validation workshops confirmed that the center does indeed have some viable leads for developing revenue-generating engagements in the energy sector, but the major obstacle is the ability to deliver on the solutions these clients require. One challenge facing the center is that taking advantage of research opportunities may require faculty to act like consultants, something that they may not be able or willing to do. The ability to respond to emerging opportunities may also require teams of trained personnel equipped to implement solutions. These may represent challenges for the center because it still lacks a complete managerial infrastructure and is stretched to meet the multiple existing demands of U.S.-PCAS, including student programming, faculty needs, research, day-to-day management, and planning. Finding: Not having a dedicated ASU team embedded in Peshawar places CAS/UETP at a distinct disadvantage relative to other centers. Validation workshop participants believed that sharing a U.S. partner between CAS/UETP and CAS/NUST sometimes presents challenges to capacity building. Both CASs are energy centers, with similar offerings. Partially due to NUST’s national reputation and location in Islamabad, CAS/UETP sometimes struggles to attract students and faculty to Peshawar. Evaluation team members developed the general consensus that even though ASU’s local team travels regularly to Peshawar, having its offices embedded at NUST puts UETP at a relative disadvantage since physical proximity and NUST’s relative strength in the three-way relationship appear to give NUST greater leverage in shaping the way ASU as an organization understands, prioritizes, and addresses the needs of each institution. One CAS/UETP administrator discussed the challenges the center faces because the ASU international team and USAID cannot visit UETP when they make field visits to Pakistan: “The ASU international team cannot visit us. The same thing goes for USAID. It is purely because of NOC [No Objection Certificate] [issues]. It’s not like they do not want to visit, but it still hampers our ability to maintain frequent interaction with them.” Finding: CAS/UETP has an opportunity, and a need, to focus on building a distinct brand identity. CAS/UETP’s sustainability is also linked to its ability to further cultivate relationships and market itself in the energy sector. Interviews with a cross section of faculty, administrators, ASU representatives, and university leadership representatives highlighted efforts to market the center. ASU particularly noted that CAS/UETP has become much more focused on marketing itself over the past year. Both the university leadership and faculty noted that marketing efforts are starting to build the reputation of the center, particularly as CAS graduates begin to enter the job market. Interviews cited branding as a critical consideration in the center’s sustainability efforts, though two individuals noted that current branding strategies are grounded in the center’s affiliation with USAID (and ASU). A CAS administrator also noted that HEC has proposed to designate CAS/UETP a Center of Excellence (CoE) because of the financial benefits to the university. Participants in the validation workshop noted that CoEs, as HEC defines them, imply a level of institutional autonomy that UETP may not want to grant CAS once USAID funding ends. Workshop participants confirmed that designation as U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 31 a CoE is a real possibility, particularly as the university leadership increasingly sees the potential benefits of CAS/UETP as an autonomous brand. CAS – Water (MUET Jamshoro) According to CAS/MUET’s quarterly reports, the center has not made any progress on attaining the USAID goal of raising USD$1 million in external funding. However, further investigation revealed that, unlike CAS/NUST and CAS/UETP, CAS/MUET adheres to USAID’s definition of the target: “funding generated from non-governmental sources such as private sector, foundations, foreign organizations and alumni groups.” The center advised the evaluation team that it has raised funds from public sector sources, though it has not reported these because they do not meet the definition for the indicator. Center respondents claim that public sector funding has already surpassed the USD$1 million mark. The center also reports that it has established five public-private partnerships, thereby meeting that target. Finding: Compared to other centers, CAS/MUET has the greatest percentage of faculty who are skeptical of sustainability potential, though students are slightly more confident. Seventy-five percent of 16 CAS faculty members surveyed believed that CAS/MUET was somewhat likely or highly likely to continue to exist after current USAID support ends, a lower proportion than at the other two centers. The same proportion said that the center was somewhat likely or highly likely to generate alternate financial resources after USAID funding ends, and this was also lower than corresponding percentages at the other two centers. Conversely, 79 percent of 115 students surveyed felt that CAS/MUET was somewhat likely or highly likely to continue after USAID support ends, a higher proportion than at the other two centers. Finding: Questions regarding faculty compensation and status appear to be at the core of inconsistent perceptions of sustainability. All three university leaders and six of seven representatives of the partner university (Utah) believed the center would continue to exist after project funding ends. Respondents noted that planning for sustainability is ongoing: “I am confident that CAS will sustain. I prepared an endowment policy at the start of the project, which has good funds. We hope that after 2019, some donor agency will also continue to support the center. If it doesn’t, then MUET has the capacity to sustain CAS. We … often talk about ways to sustain CAS.” – CAS/MUET leadership representative Though faculty appear aware that leadership is discussing sustainability, they do not appear to know how the process is unfolding. In each of the five faculty interviews conducted at CAS/MUET, faculty members expressed varying degrees of concern about the process. A representative of the University of Utah described the core sustainability issue concerning faculty as uncertainty regarding their salaries, saying, “There are conversations about what to do with the faculty and how [their pay scales] will be adjusted to MUET. They are paid more at CAS. … There is brainstorming on solutions that can be adopted.” Though Utah respondents discussed plans to support faculty salaries by funding research that faculty should be well positioned to secure as a result of efforts to develop applied research and proposal-writing capacity, a comment from one CAS faculty member suggests that the issue is more complex: “MUET’s hiring is done on the Pakistan Government pay scale, and HEC has tenure track, which has higher salaries, and one becomes an associate professor after six years under this rule. HEC gives 13 months’ salary in tenure track. There is a gratuity for us, but my experience U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 32 will count for nothing after the end of this contract. The university will start from zero and I will need another six years to become an associate professor.” Three of four CAS administrators interviewed confirmed that sustainability planning is advancing and that HEC has made verbal commitments to cover the center’s recurring costs. Discussions during the validation workshop in Mehran confirmed that the center plans to submit its revised sustainability plan to HEC in August 2018. Finding: Host and partner universities are collaborating strategically to address sustainability issues. MUET leadership recognizes the financial challenges inherent to sustaining project components at their current levels. One respondent spoke of the effect the end of USAID funding may have on the center’s ability to attract quality students, who are currently drawn to Jamshoro by scholarships and exchange program opportunities. He also felt, however, that the quality of programs and opportunities for professional development will continue to attract students. MUET has some plans in place to address sustainability; a representative of university leadership noted the endowment fund the university plans to use to continue critical CAS components, including scholarships. Interviews with university leadership and with CAS administrators suggested that MUET and Utah collaborate in developing a strategic approach to sustainability. For example, a Utah representative noted that both sides are working together to try to approach the sustainability of these components strategically, saying, “Our plan is to be able [still] to attract good students. The exchange program is not an end in itself. The question is how you can scale it down without interfering with quality.” Utah appears to be strongly committed to supporting the sustainability of CAS even after the formal project ends. A Utah representative noted that many of the mechanisms already implemented are designed to be self-sustaining: “A lot of programs … such as mentoring and training, are being packaged for continuous use by MUET. A lot of documentation and procedures [are available] online. These will continue to exist after funding ends.” This individual also emphasized that even when components will require fundraising to continue, that can be approached collaboratively: “The research collaborations will sustain because that is what we do as university professors—we go out and raise funds. We will do that collaboratively with our partners.” Finally, a CAS administrator reflected on how collaborative capacity building and networking position the center well going forward: “Looking back on the last three years, we have built the capacities of the faculty, administration, and the students. … We have developed networking, which will be the key in the future to build relationships with Utah and other universities. It will assist us in achieving sustainability in the future.” Finding: Sector organizations see value in several components of CAS, including research and gender. Sector organization representatives interviewed identified research, gender, and labs as important CAS components, and internships, research, gender, and labs as those they were most likely to support (Table 3). U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 33 TABLE 3: VALUE OF CAS COMPONENTS AS PERCEIVED BY WATER SECTOR ORGANIZATIONS Sector Organization Most Important Components Components Most Likely to Support Organization 1 Gender, curriculum Internship, gender Organization 2 Research, gender, labs Internship, research, labs Organization 3 Teaching, research, exchange Internship, labs, research Organization 4 Research, gender, labs Research, internship, gender Source: MSI water sector organization interviews. Representatives of four of five organizations interviewed explained that they are impressed with the quality of CAS research and efforts to develop dynamic partnerships. Although representatives of three of the four organizations also identified research as a component they would likely support, CAS appears to struggle to find ways to translate such interest into revenue. As one administrator noted: “Engaging stakeholders is a big challenge. … They are happy about the research … but they hesitate to give funds. … Our faculty sends proposals to many agencies, but we hardly get any funds. … Their research discipline is quite weak. It is challenging to bring the research community together.” Finding: CAS/MUET is considering ways to commercialize its services, while sector organizations stress the need for CAS to focus on building its brand. All three centers focus on commercializing key assets, including labs, research capacity, and training opportunities. Three of five MUET faculty members and three of four administrators interviewed highlighted efforts to commercialize laboratory services as a revenue stream. Despite the interest, several faculty members, administrators, and Utah representatives saw potential challenges due to the small size of the labs and the cultural obstacles to universities’ commercializing their services. Nonetheless, one CAS administrator noted that efforts are underway to move commercialization forward: “We are trying to develop SOPs [standard operating procedures], which are almost ready, to get commercial projects. Recently we got a project from the National Rural Support Program which included water quality analysis. … Our main approach is to create a service-oriented institute, whose services can be commercialized. The labs are now ready and equipped.” MUET faculty members (3 of 5 interviewed), administrators (3 of 4), Utah representatives (3 of 7), and one university leader noted that MUET is interested in commercializing faculty knowledge in the form of sector training programs. A design for training developed for the Sindh Irrigation Department serves as a model that can be scaled for further commercialization. Sector organization respondents saw potential value in the research, training, and laboratory offerings of CAS/MUET, although some (3 of 5) questioned whether the center is sufficiently focused on establishing a competitive brand. “We only receive newsletters from CAS. We need research and discussion papers for public policy. They could launch a research journal. … IBA [Institute of Business Administration, Karachi] has become a brand. LUMS [Lahore University of Management Sciences] is a brand. CAS is not a brand today. When will CAS become a brand?” One sector organization representative was impressed enough with the center’s potential that it offered its resources and network to help CAS develop its brand. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 34 “We are quite satisfied. … We can work together with CAS to collect data from the field and go for joint publications. We can help CAS convert those publications into informational briefs. A couple of million people visit our network websites around the globe, and if we link CAS research to those websites, it can help the center to disseminate the information.” In short, sector organizations see significant potential in CAS resources, and wait for CAS to strategically leverage those resources to support development of the water sector. Conclusions: Sustainability Sustainability is an area that needs more rigorous definition and consideration at each CAS. The centers currently appear to view sustainability as a project deliverable rather than a strategy for providing valued outcomes to internal and external stakeholders. Specifically, centers focus on meeting sustainability output indicators, with less emphasis on strengthening the underlying systems and relationships that can make alternate revenue and funding opportunities for the centers a reality. What sustainability is—and is not—needs to be defined so that centers and stakeholders can focus on sustainability planning. Interviews and surveys left the evaluators with the perception, confirmed in the validation workshops, that centers primarily define sustainability as sustaining the physical assets of the centers once USAID funding ends. Though this must be an important part of any sustainability plan, it is not sufficient. Sustainability is also largely determined by the value critical stakeholders place on outcomes. As USAID’s definition describes, sustainability is “the ability of a local system to produce desired outcomes over time. Programs contribute to sustainability when they strengthen the system’s ability to produce valued results and to be both resilient and adaptive in the face of changing circumstances” (ADS 201). Centers’ sustainability plans vary significantly in the degree to which they articulate a strategic focus on creating and maintaining the value proposition that will draw stakeholders to pay for what they now receive courtesy of USAID funding. Centers appear to make too many assumptions about the value stakeholders will continue to ascribe to applied research and laboratory resources and give too little consideration to the rigor required to maintain the high quality of these components absent external financial support. Additionally, there is little evidence that the centers adequately consider the interdependencies between the components, such as the impact on the entire CAS system if student enrollment drops significantly due to fewer scholarship and exchange program opportunities, or whether faculty will be motivated to conduct research if funding proves more difficult to obtain. Considering sustainability through a systems lens, as the USAID definition suggests, and rigorously planning to sustain longer-term outcomes rather than physical assets will provide the best chance for each university to sustain CAS as a value-creating entity. The centers now have the core capacity to deliver sustainable outcomes. They must be realistic and rigorous in planning for the future and consider how best to be resilient and adaptive in the face of the “shock” to the system that the end of USAID funding will create. Findings for Evaluation Question 4: Governance How are the universities utilizing their governance structures to sustain the CAS activities/results? To address this question, the evaluation team examined the perceptions of stakeholders related to the effectiveness of CAS governance structures and practices, how governance structures are aligned to host university structures (institutionalization), and perceptions of the impact such alignment has on the longer-term sustainability of each center. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 35 CAS – Energy (NUST Islamabad) The CAS/NUST Monitoring, Evaluation and Learning (MEL) Plan articulates the project’s governance objective as follows: “Established governance structures for improved capacity, institutionalization of CAS into NUST’s existing structure.” Finding: Respondents’ perceptions of the effectiveness of CAS/NUST governance structures depended on their roles in the system. NUST university leadership and ASU representatives held generally positive views of the effectiveness of CAS governance. In terms of project implementation and collaboration, leadership representatives noted: “Eighty percent of the work is done … the project team will complete their task with the end of this project. We have had experience of establishing other centers from funding as well and they are sustained after the funding ends. We had a central team for management.” “All the decisions in CAS are taken in consultation with ASU and NUST directorates. We also have a university curriculum committee for the development and revision of curriculum. The governance structure will sustain. All the governance structures are aligned with NUST.” Four of seven ASU representatives interviewed felt that the overall structures in place were effective, although they noted that NUST procedures often slow decision-making on issues such as faculty hiring and curriculum approval. One respondent distinguished between the PMU and NUST when discussing governance effectiveness, saying the “PMU is being run effectively,” while “NUST is [just] ok.” Survey responses suggest that faculty, students, and alumni held moderately positive perceptions of the effectiveness of CAS governance structures (Table 4). TABLE 4: AGREEMENT WITH STATEMENTS ABOUT CAS/NUST GOVERNANCE STRUCTURES Statement Agree or Strongly Agree (%) Students (n=166) Alumni (n=56) Faculty (n=10) Mechanisms for addressing grievances (e.g., of faculty, students, etc.) are in place. 60% 36% 80% The CAS governance structures (policies, procedures, oversight committees) are open to suggestions for improvement. 62% 32% 60% The CAS governance structures (policies, procedures, oversight committees) are flexible in incorporating suggestions/feedback. 57% 32% 50% In CAS governance committees, all members have a voice in decision￾making. Not asked Not asked 70% Source: MSI student, alumni, and faculty surveys. Note: The surveys of students and alumni included a “don’t know” response. Interviews elaborated respondents’ rationale for these perceptions. Six of seven faculty members interviewed highlighted challenges with equipment procurement as a leading example of ineffectiveness and inflexibility of governance structures: U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 36 “Overall governance structures were good, but procurement procedures were too long and tedious, and these procedures are still the same. There are two reasons for the late procurement, one is the long procedures, and [the] other is rotation of procurement staff.” Three of the seven spoke more generally about the challenges presented by poor coordination between faculty and the PMU. As one faculty member explained, “Coordination between the PMU and faculty is not good. There is a need for effective implementation of project activities. We try to communicate our interests to them, but there is a lack of interest at PMU’s end.” Another faculty member indicated that faculty do not communicate directly with the PMU, noting that the dean or principal communicates on their behalf. In two of the four group interviews with students, respondents indicated that they were not part of the decision-making process, with one group strongly noting the intransigence of the CAS administration in addressing their concerns. Only one of the three group interviews with alumni touched on respondents’ perceptions of governance, and that yielded only a few isolated comments regarding bureaucratic processes governing laboratory access. Finding: Respondents within CAS held divergent opinions about what alignment of CAS governance should mean and how it is unfolding. The following narrative description from the CAS/NUST MEL plan results framework describes how NUST addresses governance structures: “The second objective focuses on establishment and operationalization of CAS-E governance structure and integration of CAS-E continuing functions into NUST existing governance structure.” This statement sets the context for understanding a major point of tension regarding alignment of CAS governance structures to NUST governance. Both interviews and the validation workshop revealed that alignment appears to mean different things to different constituencies at NUST—from autonomy on one side to absorption (integration) on the other. All three representatives of university leadership and CAS administrative staff interviewed confirmed that CAS governance structures are being aligned with existing NUST structures. The university leadership framed this in a way that implied that CAS would continue to have a degree of autonomy: “All the decisions in CAS are taken in consultation with ASU and NUST directorates. We also have a university curriculum committee for the development and revision of curriculum. Governance structure will sustain. All the governance structures are aligned with NUST.” In four of seven interviews with ASU, respondents agreed that efforts are being made to align CAS structures to existing university structures. Although one respondent suggested in a follow-on comment that there should be no alignment challenges once the project concludes, two other respondents believed that the distinctiveness of CAS may change: “The CAS governance system should be replicated to across the entire university to have smooth operations of the university. It won’t be easy for them to absorb our governance system—other departments looking to the performance of such a small center—but I think other departments should try to replicate this governance model.” – ASU representative 1 “I would say our partnership is more successful and strong with UETP as compared to NUST, because NUST has their own governance structure and prefers to follow their own regulation. When it comes to flexibility, UETP is much more flexible.” – ASU representative 2 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 37 Follow-on discussions with the CAS administrators who believed that structures are aligned gave a more direct indication of the future: “When the funding ends the PMU will leave …and the center will be a school. We have organized this center in a way that when the PMU will leave, NUST will be able to own the center and the center will be run under the NUST rules and regulations.” – CAS administrator Five of seven faculty members interviewed confirmed that CAS systems are well aligned with NUST systems, and most did not question the process. As one stated, “The governance structure of the CAS is completely aligned with NUST, and HEC is also on board, so there are no challenges in this regard.” However, others had a different interpretation: “I supposed that CAS was going to provide us with a new governance structure with different policies and procedures, but it appears that we are doing the host university’s structure. … We can say that we are being aligned but not necessarily in a positive way.” – NUST faculty member The leading HEC representative interviewed noted, “The centers have different governance structures, but I think the universities’ BoG [boards of governors] and this CAS BoG are not that different. The positive thing in this [CAS structure] is that industries have more participation.” However, each of the four sector organization representatives said that they had little participation in or knowledge of CAS/NUST governance structures. As previously noted, representatives of three of the four sector organizations felt that CAS/NUST was weak in sector engagement. As one explained, “They should have more frequent meetings of stakeholders. They should have focused meetings instead of general meetings.” The same interviewee went on to conclude his interview with the following comment: “I hope this interview will be helpful, and positives should be taken out of it to make this a real Center of Excellence. I think we should be able to do something for the university so that it becomes a real Center of Excellence.” Participants in the validation workshop discussed the issue of the alignment of governance structures at length and clarified a number of issues. • Regarding alignment, one staff member explained that a key assumption the project made about sustainability and governance is that integrating CAS systems with the host university’s systems would enhance prospects for sustainability. The result is that CAS governance systems are being absorbed into NUST systems. • Two faculty members participating in the discussion argued that instead of becoming a role model, CAS has become a NUST entity, and NUST’s bureaucratic processes and inertia are slowing CAS operations. They feel that CAS is increasingly adopting many of the negative aspects of NUST systems. Once the project ends, and the PMU is dissolved, there will no longer be a stopgap in place to counter this. • University leaders are firm about the logic of the policies and procedures being aligned to existing NUST policies and procedures. They believe that there is no reason for separate CAS systems once the project ends since the center can rely on NUST mechanisms. • There is talk about CAS/NUST becoming a CoE, but participants clarified that a CoE has a degree of autonomy and an autonomous board of governors. This is not the CAS model that is unfolding at NUST, so talking about a CoE without building the appropriate context is problematic. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 38 • Others proposed the alternative of a clear focus on being a CAS. This means that CAS would not be autonomous like a CoE. It is part of NUST, and there are resources and systems with which the center should align. The best way forward is to strike a balance between CAS systems and NUST systems in a way that optimizes the value the center provides. CAS – Energy (UETP) The CAS/UETP MEL plan articulates the project’s governance objective as: “Improved governance structures and processes of CAS-E.” Finding: There are mixed perceptions of the effectiveness of governance structures at CAS/UETP. Some feel that leadership challenges have created some negative perceptions, while others feel that the structures of CAS/UETP are adaptive and effective in meeting project objectives. The university leadership speaks highly about the effectiveness of CAS governance structures and the value they add to the university: “The CAS governance structure is vibrant and it may be replicated to govern other centers at UETP. The university has already started replicating the system.” Three of the seven interviews with ASU representatives focused on issues of governance related to UETP. Two ASU representatives noted the leadership issues that UETP faced early in the project, and the changes that have been made since. One stated: “The vice chancellor changed, we abolished the board of governors, and now we have a steering committee. Other departments unfortunately don’t have what we have—we [have direct access] to the syndicate.” 5 The other ASU representative confirmed the changes and also noted that PMU staff changes and challenges in staffing the center’s managerial team, including finding a director, have negatively affected the effectiveness of governance structures. A USAID representative also mentioned the challenges that CAS/UETP continues to face in trying to secure qualified individuals to manage the center: “They are still in the process of rebuilding the team. They are still struggling to find the right people, including the project director. The vice chancellor has been acting director for more than a year. It has been an ad hoc approach to things.” Despite these challenges, an ASU representative noted that in comparison to NUST, UETP is more adaptive: “I would say our partnership is more successful and strong with UETP as compared to NUST, because NUST has their own governance structure and prefers to follow their own regulation. When it comes to flexibility, UETP is much more flexible.” Similarly, three of four CAS administrators interviewed felt that governance structures are effective. One respondent said, “I think we are very transparent and very open,” while another noted, “We have improved the university's practices … we have improved the system … I can say [our changes] are not against university rules; [they are] an improved version of university rules.” With the exception of grievance mechanisms, faculty survey responses suggest positive agreement with the effectiveness of CAS governance structures (Table 5). 5 The syndicate is a governance and executive decision-making body that plays a role similar a governing board in private sector universities. The syndicate acts like a governing board and performs similar tasks to such a board including the management of university assets, developing budgets, and making staff appointments. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 39 TABLE 5: AGREEMENT WITH STATEMENTS ABOUT CAS/UETP GOVERNANCE STRUCTURES Statement Agree or Strongly Agree (%) Students (n=222) Alumni (n=6) Faculty (n=14) Mechanisms for addressing grievances (e.g., of faculty, students, etc.) are in place. 54% 17% 57% The CAS governance structures (policies, procedures, oversight committees) are open to suggestions for improvement. 56% 17% 78% The CAS governance structures (policies, procedures, oversight committees) are flexible in incorporating suggestions/feedback. 53% 17% 78% In CAS governance committees, all members have a voice in decision￾making. Not asked Not asked 86% Source: MSI student, alumni, and faculty surveys. Note: The surveys of students and alumni included a “don’t know” response. Interviews with faculty members gave a more mixed impression of effectiveness. Four of six interviews with faculty focused on the effectiveness issue. Two respondents mentioned confusion about CAS governance. One stated: “There is a lack of clarity about who is actually the governing body, is it USAID, ASU, or UETP? We don’t know. … Nothing is clear, sometimes we are part of UETP, sometimes we are part of USAID, sometimes we can’t do anything without ASU.” – UETP faculty member Two were more positive, noting that the structures support the achievement of project objectives well. One faculty member noted the uniqueness of the CAS system: “The governance system of CAS is a new concept in UETP. It has a director and deputy director, and there is a hierarchy in the project like monitoring and evaluation, communication, and HR [human resources]. This concept is not in UETP. In UETP, HR is in the registrar’s office. The [CAS] governance system makes it easy to achieve project objectives. Rules are everywhere, but one-to-one interaction helps solve problems.” – UETP faculty member The lack of clarity led USAID to tell UETP that it was too submissive and failed to challenge ASU to help it identify innovative solutions to the managerial challenges it faces. UETP respondents expressed the wish that ASU international leaders would visit UETP in Peshawar, rather than Islamabad, when they travel to Pakistan. ASU cited the challenges of obtaining NOCs, but USAID staff said that obtaining NOCs is not as challenging as it was previously. Responses from the alumni surveys were largely critical of CAS governance structures, while students were more moderate (Table 5). Respondents in the four student group interviews claimed that stipends were not paid to students after enrollment and that student evaluations of faculty were not considered seriously. Participants in the one group interview with alumni said only that students felt that the senior leadership was not effective. Finding: Though some are optimistic about the alignment of CAS/UETP governance structures to UETP structures, others have significant concerns that the center and its future are quite vulnerable. Two of three university leaders interviewed felt that UETP was making efforts to align CAS/UETP governance structures to the university’s structures. They explained that CAS governance structures U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 40 were helping change the way UETP operated. One indicated that the university was considering the CAS model for other centers at the university, while the second reported that UETP was adopting some of the systems developed by CAS, particularly management information systems, more broadly. However, one of these individuals also discussed tentative plans to make CAS a department: “In order to merge the center as a department of UETP after the funding [ends], we have to follow the rules of UETP. We have departments like procurement and finance, and all of these are centralized. They will be used for the center also after the PMU is dissolved.” With regard to the alignment of CAS/UETP to existing university structures, four of seven ASU representatives interviewed agreed that governance systems were aligned with UETP systems. Three of these, however, indicated that alignment was still a work in progress, and that past managerial uncertainties at UETP and CAS had taken their toll. One explained, “It’s not properly aligned at UETP. UETP went through numerous transition phases, the change of vice chancellor, and some issues in the project management unit as well.” Others were less certain about the current structure, noting that the current center steering committee is not playing its oversight role. Even though CAS is struggling, the steering committee has not requested an emergency meeting. Four of six faculty members interviewed felt that the center was vulnerable because it did not have a strong and complete managerial team in place. One faculty member argued that it was critical to build the academic and research culture of the center if it is to survive: “Sometimes it seems like they are running a factory, it doesn’t look like an academic institution. … [Staff] in academic organizations have research and much time to think about things. In a factory, you have a target and this much product you have to produce. We don’t have an academic environment.” In fact, there is a great deal of uncertainty regarding the future of CAS once USAID funding ends. Faculty are concerned about the possibility that CAS will become a department rather than remaining a center. If this happens, then existing structures at the university will dominate. One faculty member explained: “Faculty is a giant force in UETP. UETP faculty is everything—they run the departments. We are different in a positive way. If we go towards UETP’s structure, it will be like a government organization, and that will be the end of CAS. It will be a major problem after funding ends, as PMU staff will not be here, and somebody has to take responsibility, and I foresee that no one will take it and that will be the end of the center.” Three of six faculty members interviewed, as well as USAID representatives, were concerned that the center will become a UETP department once funding ends. Finding: Its lack of a complete and focused managerial infrastructure makes CAS/UETP vulnerable. However, CAS/UETP also has the potential to exploit opportunities within its sector if it can build its managerial infrastructure quickly. The challenges identified during data analysis left the evaluation team skeptical of CAS/UETP’s prospects for sustainability. However, conversations and additional evidence presented during the validation workshop suggest that the center does have a chance for a sustainable future. Key findings from the workshop include: • Labs will be complete and ready for commercialization by September 2018, as will a complete commercialization strategy. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 41 • There are a number of significant revenue-generating opportunities that the center could seize if it is able to develop a viable plan to quickly develop the capacity. One of these is a major project in FATA related to establishing electrical grids. The center has recently hired a consultant to scope the needs of this project, but the challenge will be to hire consultants with the requisite skills who can deliver the solutions identified by CAS based upon the needs assessment. • Though having the vice chancellor serve as the project director has its disadvantages, since his engagement at CAS is somewhat ad hoc, the center reports that he is increasingly responsive to the idea of the center’s becoming a CoE, an idea that HEC reportedly supports. Several members of the ASU team were present at the validation workshop, and one believed the CoE designation was a realistic possibility, noting that UETP is open to the idea of CAS’s having a degree of autonomy. • Participants in the sustainability discussion noted that if CAS ultimately becomes a department, expectations of future consulting opportunities may be unrealistic. CAS will be no different from any other part of UETP, and sector interest will wane. • Time and available resources are slipping away. However, the center is included in several proposals with other development agencies (the International Monetary Fund, the World Bank, and the United Kingdom’s Department for International Development) which can help it secure its future. It is also working with HEC regarding its sustainability. • The center needs two more years of financial support. If some of the development opportunities noted do not come through, it may require bridge funding from HEC. • When evaluators asked what skills the center would want if it could have one expert start tomorrow, participants described a marketing and fundraising expert. In fact, the center has just received approval to hire such an individual. • Participants in the validation workshop expressed the desire for critical stakeholders—HEC, ASU, and USAID—to become much more proactive and creative in helping the center overcome its challenges so it can set itself on a clear path forward. The evaluation team concluded that the center leadership has many innovative ideas for how to leverage CAS/UETP’s assets and expertise, but is challenged when it comes to operationalizing those ideas. Identifying a strong operations person who can turn prospects into actionable solutions can help secure the center’s future. As noted in the findings on sustainability (Question 3), CAS/UETP’s future may not lie in a traditional view of programs and students. Collaborative strategizing and creative resource idea generation on behalf of the key stakeholders could go a long way in securing the center’s future as a distinctive resource in the energy sector. CAS – Water (MUET Jamshoro) The CAS/MUET MEL plan articulates the project’s governance objective as follows: “Enhance vision and strengthen governance structure of the Center for Advanced Studies in Water at MUET to make it a leading Center of Excellence in water domain.” Finding: Partners recognize that CAS governance at MUET is complex and take a strategic approach to optimizing effectiveness. Two of three interviews conducted with the leadership of MUET discussed the challenges the university will confront when USAID funding for CAS ends. However, respondents also noted that the CAS BoG U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 42 is committed to setting the center on a realistic future course, though some difficult decisions may need to be made. One explained: “We have discussed how CAS will merge with MUET. It is not on paper yet, but it is always on our mind. If we don’t find another donor, it will not be possible to run as we do now, but we will do our best. Faculty is aware of these factors. Sustainability is dependent on the commitment.” Another member of the leadership team highlighted the steep learning curve the university has been on with regard to CAS and was grateful that USAID has challenged and guided them in the process: “This was the first time that the university entered in this type of partnership, and meeting USAID’s expectations was difficult. The process took time, it was achieved through the guidance of USAID’s management team, and they deputed a lady who guided us. They [USAID] told us what strengths they want and weaknesses that are to be covered.” Representatives of the University of Utah, however, painted a more pessimistic picture of effectiveness. Of seven representatives interviewed, five noted that governance structures have been effective in some ways, but less effective in others. A cultural shift is needed, but the issue goes beyond MUET itself, as one representative explained: “Overall, the governance structure is fair, sometimes it can be very good, and other times it’s poor. I think it’s in the overall culture of higher education in Pakistan, and how they conduct business in certain areas presents a challenge. This needs to change, but when you look to make changes, you meet resistance. It is just going to take some time as it is not part of the standard culture.” Another Utah respondent noted that the BoG does not (yet) play an executive function: “There is real governance and there is window dressing. We should be more concerned about the real governance issues that are confronted at university and program levels in an attempt to overcome them. Formal governance is facilitative rather than end-use effective. The purpose of the BoG to us is as a group of stakeholders who through this process acquire more in-depth understanding of what the center is doing and is therefore able to facilitate its work and also open doors and link it to other institutions. We do not really at this moment see this [the BoG] as the executive body. We see it as a kind of facilitative and oversight body, and that is the distinction.” Within this context, the same individual laid out a case for embedding the executive function directly in the center: “The executive functions have to be vested in the center itself for real governance to take place. Now we see the systems that need to be corrected not in terms of the relationship between this entity and the BoG, but in its rules of operations of the entity itself; how does it function, how does it evaluate its own performance, how does it measure quality, how does it evaluate the performance of individuals who work for the system? … If this center wants to aspire to excellence, it has to revamp itself, regardless of what the BoG asks it to do.” Conversations with Utah representatives made it clear that the center is focused on embedding effectiveness into its culture and operations: “For the purpose of sustaining CAS, they are trying to develop SOPs for various things like labs. They are developing documentation which … can tell new staff how things are done in CAS. If U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 43 you want to make sure that you have high-quality activities going on, all processes and policies should be a part of this center.” Respondents in each of the four interviews conducted with CAS/MUET staff believed that the existing governance systems are effective. One explained, “Governance structures are helping in achieving the overall project objectives, because there are visionary people involved. They have huge experience, and their suggestions are helpful.” Finding: Faculty, students, and alumni viewed governance structures as largely effective, but they also recognized that achieving change can be a slow and intensive process. Faculty at CAS/MUET were positive about governance mechanisms, although satisfaction with structures that allow individuals to provide feedback or voice grievances was somewhat lower than with other elements of governance (Table 6). TABLE 6: AGREEMENT WITH STATEMENTS ABOUT CAS/MUET GOVERNANCE STRUCTURES Statement Agree or Strongly Agree (%) Students (n=116) Alumni (n=19) Faculty (n=16) Mechanisms for addressing grievances (e.g., of faculty, students, etc.) are in place. 72% 74% 69% The CAS governance structures (policies, procedures, oversight committees) are open to suggestions for improvement. 72% 68% 69% The CAS governance structures (policies, procedures, oversight committees) are flexible in incorporating suggestions/feedback. 66% 73% 81% In CAS governance committees, all members have a voice in decision￾making. Not asked Not asked 75% Source: MSI student, alumni, and faculty surveys. Note: The surveys of students and alumni included a “don’t know” response. Interviews with faculty helped elucidate some of these findings. Two of three faculty members who spoke to the effectiveness of CAS/MUET governance structures were largely positive in their assessments, and one was more critical. On the positive side, faculty felt that progress is being made, but things move slowly. As one respondent observed, “The present governance is effective, because things are moving towards achieving objectives, although it’s slow. I think follow-up is effective. Initially, there were fewer human resources.” The more critical response indicated that governance is not as collaborative as it should be: “In a good governance structure, the whole team that is involved works on the basis of incentives and disincentives. There should be incentives for the team.” Responses from the student survey about CAS/MUET governance structures were also largely positive (Table 6). Responses from students in two of the three student interviews conducted were critical, but they demonstrated the focused effort required of everyone at CAS/MUET: “Students are scared by everyone from the very start. It feels like we are buried under a mountain, and you have to manage it somehow. We were 85 students at the start, and now we are left with 39 students, within 6 months. The students who left felt that they will not be able to perform in the program and that it was very tough for them. Students were of the view that they should get breaks, and room for extra-curricular activities.” U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 44 Despite this concern, students also noted that CAS/MUET was “a plus point for Mehran and for the country,” and they worried that it will become like any other department after USAID funding ends. The three alumni interviews conducted at MUET did not offer substantive findings regarding governance effectiveness, other than the observation of students that there were no women in visible decision￾making roles at CAS/MUET. Finding: MUET has committed to assuming responsibility for CAS after USAID funding ends, but it is not clear how much autonomy the center will have, and this matters to sector organizations. In 14 interviews with university leaders, CAS administrative staff members, and Utah staff members, 12 respondents believed that “effort is being made to align CAS governance structures with the university’s existing structures.” University leaders explained their uniformly positive responses by noting that there are other entities that exist autonomously within MUET. One noted, “CAS has different pay scales and SOPs [than the rest of the university]. However, aligning should not be a problem, this university is autonomous, and within the university, things can be autonomous. The center has to be decentralized.” They cited the MUET Center for Art and Design as an example of a center that operates autonomously. Interviews with faculty suggest that, although most people understand that efforts are being made to align CAS/MUET governance structures to those of the university, understanding of the implications of alignment is uneven, and faculty have various perceptions of the future. On the one hand, “CAS-W and MUET are well aligned, as MUET also has a management by objectives approach,” while on the other, “The governance system is different and not aligned. This is the biggest sustainability problem. Things will be known after the completion of the project. Most of the faculty here are too dynamic to move to MUET.” Administrative staff expressed a much more nuanced perspective: “More or less, all policies of CAS are aligned with university policies, but we have autonomy to develop policies and come up with ideas. The BoG will continue as a subcommittee of the university and this institute [CAS] and the whole structure of this center will be retained. This is what is being assumed at the moment. The whole governance system will be in shape.” Even within the University of Utah, the question of how the center will operate is still unclear. Six interviews with Utah representatives revealed the sentiment that MUET has committed to maintain the center after funding ends, but that the degree of autonomy CAS/MUET will actually have is not clear. One respondent noted, “[It] depends on how this center operates. If it is still autonomous, they could maintain their vigor if they wanted to. If they are not autonomous, then it could go either way.” Alternatively, another said, “Our concern is that there is a situation there which will actually pull down [the center], and we do not want that. We want to maintain a degree of protection for CAS. I think that something like that will happen.” Finally, four of five sector organization representatives also questioned the future of CAS/MUET once USAID funding ends. As described earlier, sector organizations recognize the value they derive from CAS research and programs, as well as the model for change it provides through its focus on gender issues. Sector organizations hope that decisions regarding governance will consider the need to continue optimizing the center’s contributions to industry. As one respondent explained: “There is a need to think through the relationship of CAS to MUET. It is a very important point for CAS. IBA is a good model; after getting autonomy and independence from Karachi University, IBA has become a brand. LUMS is a brand. CAS-W is not a brand today. When will CAS become a brand?” – Sector organization respondent U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 45 Ultimately, the University of Utah realizes governance of CAS/MUET over the long term is MUET’s decision: “The governance structure has been a continuous problem, and we continue to advise them. It may be more of a personal issue than governance issue, but my point of view is that we try to solve the problems which we can solve and not try to solve the problems which we cannot solve … so there continue to be problems, and we are trying to solve them in the way which [aligns] to our mandate.” Conclusions: Governance Governance appears to be an evolving issue at each CAS. Though not everyone is satisfied with the way the centers operate, the common belief is that governance structures at each CAS are effective, though there are areas for improvement. Decision hierarchies, particularly with regard to how faculty can/should engage with the PMUs, often lack clarity. Beyond this, the significant delays procurement policies caused in making facilities operational influence stakeholder views of effectiveness. By and large, however, the respondents viewed governance structures as effective in running each center while project funding continues. A different picture emerges for the longer term, and how each center will function once funding ends is uncertain. The underlying assumption is that centers will be aligned to the host universities’ existing structures. However, alignment is a word that means different things to different people. The validation workshops confirmed the project’s guiding belief that the more closely a center is aligned with existing university structures, the stronger the chances for sustainability (particularly if one takes an expansive view of sustainability that focuses on outcomes). Others do not agree with this assumption and maintain that the centers need to keep a degree of autonomy in order to create value for the sectors they are designed to serve. The more that centers look like other parts of the universities where they are housed, the more difficult it will be to provide the distinctive things they were designed to do. CONCLUSIONS The U.S.-PCAS project has achieved a great deal in a relatively short period of time. Each host university has established a center, recruited talented staff, and designed a strategy to establish the center as a resource in its respective sector. The results can be assessed through the products (graduates and research) and relationships each center has developed. Each host university has approached implementation of U.S.-PCAS differently, and the mode of engagement of the two partner universities differs as well. Consequently, each CAS has carved its own pathway to the future, each with its own distinctive set of opportunities and challenges going forward. This said, there are also some commonalities across these opportunities and challenges, giving the centers the opportunity to address their futures somewhat collaboratively, learning from each other’s experiences. This evaluation has tried to build a deeper understanding of these opportunities and challenges—both individual and common—in order to serve as a learning tool for possible ways to move forward in the U.S.-PCAS project’s remaining time. CAS is effectively a combination capacity-building and change management project in the higher education sector. By articulating a vision for CAS of building the mechanisms for deeper engagement with economic sectors key to Pakistan’s long-term growth, the project has created the basis for each university to develop the capacity that will enable it to change the nature of its relationships with the sectors. This is an innovative model for higher education institutions anywhere, and one that can help Pakistan more purposively align education with economic sector needs. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 46 To this end, the organizational learning model promoted by James March6 provides a frame through which to understand and assess each center’s progress in capacity development and organizational change. March distinguishes between two types of learning that all organizations should engage in to survive: exploitative learning and exploratory learning. In the first form—exploitative learning—emphasis is placed on deriving short-term advantage from knowledge and resources that are readily available. The organization “exploits” what is already known to create competitive advantage. The resources and expertise provided by the U.S. partners have served as the primary catalyst for such exploitative learning, while the extensive resources provided by USAID for key components create the basis to channel this knowledge into mechanisms through which each CAS can establish its reputation and build its capacity. The resources of each CAS (scholarships, exchange opportunities, labs, research grants, and affiliation with global universities) have drawn both individuals and organizations into the CAS orbit. Knowing that these resources will not exist in their current form beyond the funding horizon for the project, each institution should use these resources as a short-term scaffolding upon which to consolidate its distinctive brand and grounded capacity for the longer term. This is achieved through an explicit focus on the second form of organizational learning espoused by March—exploratory learning. It is through the ongoing adaptation of best practices from recognized leaders in the field (in this case ASU, Utah, and others) that each CAS is able to “explore” and absorb innovative models of training, research, and governance that enable it to establish a resilient reputation for the future. Exploitative learning has served each CAS’s capacity building in the short term, while exploratory learning provides the basis to leverage that capacity toward sustainability over the longer term. Different approaches to exploitative learning and exploratory learning can be seen at each CAS. At the conclusion of the baseline, the assessment team posed the question of whether applied research at the CASs should be approached as a product or as a process. As a process, sector organizations are brought into the design process up front so that research is clearly aligned to their needs. As a product, application is done collaboratively after the research is complete. Both approaches are valid, though the engagement mechanisms needed for each vary. CAS/MUET appears to approach applied research as a process: It has a defined research agenda around SDG6, and capacity-building efforts are enacted within that context. Joint and applied research, student research, exchanges, and sector engagement strategies all are anchored to that agenda. In this manner, the center appears to be trying to balance short-term exploitative learning with longer-term exploratory learning. Initial results indicate that sector organizations are pleased with the approach. CAS/NUST, conversely, appears to have approached applied research more as a product. NUST solicits research ideas broadly from across the sector and then provides the resources to turn those ideas into research solutions that can be applied to sector organizations. Similarly, it opens exchange opportunities to a range of individuals beyond the immediate population of CAS itself. Many of those initial research initiatives have not yet been completed. However, evidence developed during the evaluation indicates that sector organizations have concerns about the capacity of the center to deliver on its promises. Additional evidence gathered during the validation workshop suggests that faculty feel that their substantial administrative responsibilities limit their ability to deliver on solutions to sustain center outcomes over the longer term. The NUST approach, therefore, appears to emphasize shorter-term exploitative learning because it draws focus to the immediate opportunities the center provides. The question for consideration, however, is how this approach specifically helps consolidate the center’s capacity for exploratory growth over the longer term, particularly if NUST is not willing or able to sustain exchanges, research grants, and other financial inputs at their current levels. 6 James G. March, “Exploration and Exploitation in Organizational Learning,” Organization Science 2, no. 1 (1991): 71-87. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 47 Finally, CAS/UETP appears to combine aspects of both the CAS/NUST and CAS/MUET approaches. Its approach to research appears to be a more modest version of the CAS/NUST model: Research funding is not limited to CAS faculty and students, although it appears to be limited to faculty and students within UETP. CAS leaders maintain that this allows the center to expand its research scope, which translates into an approach that amplifies exploitative learning. For the longer term, CAS/UETP aspires to develop the center’s consulting capacity (including the faculty’s consulting capacity) to be able to provide hands-on solutions to sector organizations’ needs. This is a challenge for CAS/UETP, however, for it is not clear whether and how it can source the experienced individuals it needs to deliver on that vision. Given evidence that indicated that the center does not even have the resources to deliver on a major portion of the courses it advertises in its prospectus, CAS/UETP clearly requires a stronger bench of professionals to be able to sustain its longer-term goals. Additionally, evidence indicates that faculty often do not take advantage of existing exchange opportunities at ASU because they fear their absence will further reduce course availability for students. The validation workshop indicated that the opportunity for longer-term exploratory learning is there, but the critical input of experienced professionals who can be trained is lacking. Finally, sector organizations note that they see potential, though they have not yet seen results. Thus, the center’s capacity to seize longer-term exploratory learning opportunities appears to be weak. By using the remaining time and resources of the project to shift more purposively to a longer-term focus on people, technical skills development, and governance, each center will have its best chance to build a sustainable future. RECOMMENDATIONS The evaluation team conducted a recommendations workshop to engage CAS stakeholders in developing the key actionable recommendations. The primary objective of the recommendations workshop was to identify realistic and grounded recommendations that can help ensure the sustainability of CAS outcomes once USAID funding concludes at the end of 2019. The importance of these recommendations has now increased dramatically, for since the evaluation was undertaken, HEC has made clear that it will only support the centers financially after 2019 if they can provide concrete evidence of their continuous efforts to achieve self-sustainability through the strategic use of the capacity and relationships they have built through the U.S.-PCAS project. This presents an unanticipated challenge to the centers. Throughout the evaluation process, centers conveyed the expectation that HEC would assume a significant portion of the financial responsibility for sustaining the centers once donor funding ended. Though the more stringent criteria for support appear to surprise the host universities, they are not totally surprising if one is realistic about sustainability, which is defined as being both “resilient and adaptive in the face of changing circumstances.” The circumstances in which the centers find themselves are changing dramatically, and they must work quickly to build their resilience and adaptiveness for the future. Establishing Context for Recommendations Discussion during the National Steering Committee meeting, which preceded the recommendations workshop, established the context and expectations under which each CAS must plan for the remainder of the project under USAID funding and for after funding ends in late 2019. Specifically: 1. Although USAID will honor its commitments to approved work plans, budgets, and activities, it may not consider changes in the current environment. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 48 2. HEC is fully supportive of the U.S.-PCAS project. It will provide funds if CASs can demonstrate that they are/can be clearly sustainable. In planning for sustainability, HEC noted that each CAS must make strategic trade-offs on which aspects of its center to prioritize. Centers are expected to conduct cost-benefit analyses of the opportunities they wish to pursue and undertake those that have the highest return. Similarly, they should seek out innovative ways in which to manage and apply their assets. The U.S.-PCAS project was designed to create the assets and capacity to approach higher education differently than it has traditionally been approached in Pakistan. HEC therefore expects centers to be innovative, to model new ways of thinking, and to demonstrate sustainability. 3. HEC wants to see the CASs collaborate with the GoP, with the provincial governments of Pakistan, and with the private sector. For example, the HEC chairman noted his commitment to match any research funding CAS receives from the Ministry of Energy. 4. Revenue generated by CAS services and products should support the CAS; it should not be absorbed by host universities. 5. Building on expectations of sustainability, USAID indicated that the centers should expect to make a much more concerted effort to market themselves to organizations in their respective sectors, be they public or private. Each center should have a strategic marketing plan executed by individuals who are able to research and understand market needs; promote the center externally as a resource to meet those needs; and build relationships between sector organizations, faculty experts, and trained students so that knowledge-driven solutions can be identified and executed. 6. Each CAS should establish an outreach unit to secure funding from multiple sources (e.g., donor, public/private sector, and government). Structure of CAS-Specific and General Cross-CAS Recommendations The remainder of this document details recommendations. These fall into two general categories: 1. CAS-specific recommendations: The recommendations identified by each CAS during the afternoon breakout session are presented, along with brief analyses of the resources and constraints each center identified in trying to realize the recommended course of action. The analysis also includes insights from the evaluation team based upon its understanding of each center’s work gained during the evaluation process. The team assessed these insights through the lens of sustainability issues surrounding recommended courses of action, with critical assumptions challenged as appropriate to ensure that recommendations are realistic and strategic. In addition to the recommendations each CAS team identified during the workshop, the evaluation presents CAS-specific recommendations the evaluation team felt should be considered during the remainder of the project. Again, such recommendations are grounded in the perspective and insights developed by the evaluation team through the evaluation process and are guided by the objective of enabling sustainability. 2. Project-wide recommendations: Following presentation of the CAS-specific recommendations, the document outlines several project-wide recommendations for USAID and HEC to consider in order to enhance prospects for the sustainability of project outcomes in general. These recommendations focus on creating an enabling environment that can help centers make the transition from USAID funding to sustainability as resiliently and adaptively as U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 49 possible. This transition is challenge-ridden for the centers, for they now are expected to be sustainable post-project in a manner that they did not previously anticipate, while at the same time having limited flexibility in how they can deploy remaining funding under existing budgets and work plans to optimize that outcome. Recommendations in this section are intended to challenge USAID and HEC to consider how they may find opportunities for flexibility that can strengthen these transition efforts. Some ideas were put forth by USAID and HEC themselves in the debrief wrap-up following the CAS table discussions. The evaluation team identified others based upon its analysis of needs. CAS-Specific Recommendations CAS/NUST Self-Identified Recommendations Short Term Strengthen industry-academic linkages to foster a better understanding of industry problems and commercialization of CAS resources: NUST identifies this as a recommendation that can be achieved in the short term. In moving forward with the recommendation, it identifies the university’s existing ORIC, the CAS Industry Liaison Office, and existing marketing functions as resources it can leverage. Its vision is to deploy faculty expertise to identify needs of specific organizations and to design solutions. To realize this vision, CAS notes the need to strengthen its corporate advisory board to include proactive, high-level, energy sector professionals that can advise the center as it strategizes outreach and relationship development. NUST already has a number of high-level governmental relationships in the energy sector that the center can potentially use to catalyze creation of a proactive strategic advisory board involving industry members. However, CAS also notes that the senior leadership of the university must be willing to advocate on behalf of CAS by proactively supporting efforts to build high-level industry relationships and by directly inviting industry leaders to campus to experience CAS resources and expertise firsthand. In both the key informant interviews and the validation workshop held at NUST, faculty noted that they are already tasked with significant administrative obligations that hinder their ability to build high-level consultative relationships with industry organizations. They argue that their time must be freed so they are able to optimize their value in creating revenue-generating corporate relationships for CAS. Additionally, CAS can hire and develop professional staff who have experience working directly in the energy sector in close collaboration with industry organizations. Professional staff can fulfill the consultative selling function of opening doors and building revenue-generating opportunities based upon faculty expertise and center resources, with faculty brought in to lead delivery of the ultimate client solution. This is the best-practice industry outreach model used by many universities internationally. This is a recommendation that USAID and HEC can easily support to help strengthen center capability. NUST already has a strong name in the market place, and both USAID and HEC can leverage their influence to help focus efforts toward achievement of this objective. CAS is correct in recognizing the need to develop a team of industry professionals to provide the consultative bridge between CAS and sector organizations, and faculty should be positioned in a way that allows them to deliver high-level solutions that build the center’s reputation as a strategic partner in the sector. Though commercialization of CAS resources is noted as an objective, NUST (and all host universities) could benefit from explicit HEC support to develop a commercialization strategy that enables it to build stronger relationships with sector organizations. This support could take the form of inviting some of U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 50 the more successful university ORICs to train CASs (and their corresponding ORICs) to collaborate more effectively in building sector relationships. Develop comprehensive marketing and outreach efforts: NUST also noted the critical short-term need to rapidly enhance CAS marketing and outreach efforts in order to build awareness of the center and its capabilities. Until now, CAS has relied on the NUST marketing office and ORIC, as well as the CAS Industry Liaison office. As the challenges posed by the end of USAID funding and the very limited availability of ongoing HEC funding become apparent, CAS/NUST sees the urgency to focus resources on building brand awareness and market development opportunities through multiple channels. CAS seeks to create a dedicated marketing and outreach unit skilled at designing and implementing a strategy to analyze market needs and to develop messaging across multiple sector segments. A comprehensive strategy would address the market for CAS consulting, training, lab testing, applied research, and career placement of students. One portion of the marketing and outreach team would focus exclusively on a strategy for developing and procuring funding opportunities from domestic and international donors. CAS/NUST representatives expressed the urgency of this need given the realities of HEC funding going forward. They cited existing relationships that can be strengthened in this effort, including with donors and governmental organizations that are already working with other parts of the university. However, they recognize that CAS needs to focus serious attention on developing the capacity for high-level engagement and follow-up with external organizations, especially donors (a view that aligns with findings from the evaluation). In the short term, CAS seeks USAID and HEC support to advertise the center, activities, and services using national television and radio stations. USAID concurs with this wish, and also encourages each CAS to focus on developing a comprehensive marketing strategy and implementation plan. In conjunction with the suggestion that HEC support the development of realistic and actionable CAS commercialization strategies, HEC should also consider support focused on developing actionable marketing strategies based upon the experiences of other Pakistani universities. Medium Term Establish a technology center to offer short-term training courses and energy audits: The leadership of NUST identified this center as a strategic priority during the evaluation and continues to promote it as a key part of its plan to generate revenue for sustainability. During the evaluation, representatives of ASU also discussed technology centers as a strategic resource for both NUST and UETP. NUST argues that once the technology center is established it can leverage faculty expertise to provide skills-based, short-term training using center resources as a means of generating revenue. However, when pushed further on the readiness of faculty to deliver such training, NUST representatives noted that the center would need to invest in developing training content to an international standard, as well as faculty capacity to customize such training to the specific needs of sector organizations. Similarly, though NUST highlighted its existing relationships with governmental, academic, and private sector partners as a resource to leverage in operationalizing this center, further probing revealed that more work needs to be done to develop more resilient relationships with sector organizations. Findings from the evaluation suggest that NUST in fact has much to do in order to build the trust-based relationships needed to operationalize this recommendation realistically. NUST maintains that establishment of the technology center is something that can be achieved in the short term. However, when asked to identify strategic considerations essential to success, it cited funding as the most significant issue, raising the question of what external funding is available to finance the center. This suggests that a short-term time horizon for this initiative is not realistic. For this reason, the evaluation team feels it should be considered at best as a medium-term initiative. If NUST remains fixed on the idea of a technology center as part of its future planning, both USAID and HEC would require NUST to produce a comprehensive business plan that details realistically the market U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 51 analysis, resources, revenue potential, seed funding sources, and time horizon needed to achieve sustainability. This requirement does not imply that USAID or HEC would consider funding the center, but rather it can focus NUST on realistically developing the project mindset and internal capacity needed to approach center planning more strategically than it currently appears willing or able to do. As this evaluation will suggest in the more general recommendations presented at the end of the report, USAID and HEC may wish to consider providing some business planning support to all the centers so that they are equipped to develop such business plans as part of ongoing center management. Long Term Start undergraduate programs to generate new sources of revenue: As a longer-term strategy, CAS/NUST discussed creating an undergraduate program to generate revenue that could sustain the center financially. Such a program would be undertaken with other engineering schools at NUST so that other labs and classroom space could be used. A decision to launch an undergraduate program would require hiring additional faculty and developing an undergraduate curriculum that is targeted to the needs of sector organizations. The experience of developing the existing CAS graduate curricula would prove useful in this regard. When and if CAS/NUST decides to pursue this option, the center would consult HEC for support, input, and ultimately approval. At that time, key considerations should include what impact an undergraduate program would have on the mission and resources of the center as they currently exist. Faculty who participated in the validation workshop at NUST expressed concern that starting an undergraduate program would be a mistake. A new program at the volume anticipated would invariably increase their teaching and administrative workloads, thereby diminishing their available time to focus on applied research and other sector-focused activities. Moreover, introducing an undergraduate program into CAS could have an adverse impact on the mission of the center. For example, CAS labs are intended to be research/testing labs, not teaching labs, and use for other purposes could result in damage to the very expensive lab equipment. As CAS explores the viability of an undergraduate program, HEC should encourage NUST to approach such a decision carefully, so that the implications (from the perspectives of multiple internal and external stakeholder groups) are given due consideration. Evaluation Team Recommendations Regarding CAS/NUST USAID and HEC should make every effort to focus CAS/NUST on building the capacity needed for sustainability: NUST’s strategic location in Islamabad, as well as its recognized brand, give it advantages that the other CAS host universities do not have. Nonetheless, findings indicate that there is still significant room for improving the capacity of the center to provide valued solutions in the energy sector. This includes giving faculty the time and resources needed to design and deliver high-quality applied research that meets the needs of sector organizations and training staff to identify opportunities for ongoing sector engagement. The HEC chairman’s commitment to match applied research funding that CAS/NUST receives from the Ministry of Energy or another governmental department is an ideal way to provide NUST the incentive to work aggressively over the remaining life of the project to develop the capacity it needs. For long-term sustainability, USAID and HEC could help CAS/NUST significantly by continuously challenging it to engage sector organizations dynamically, while also challenging it to think more self-sufficiently about its strategic choices. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 52 CAS/UETP Self-Identified Recommendations Short Term Revise the research grants process to emphasize commercialization of solutions: Based upon the experiential knowledge it has developed in building capacity to deliver high-quality applied research, CAS/UETP recommends revamping its existing grant award process to prioritize the technological readiness of proposed applied research products and services. The goal is to create a more certain connection between applied research and revenue-generating potential by identifying a clear path to outcome commercialization prior to grant approval. In redesigning this process, CAS/UETP stakeholders propose drawing lessons on the grant approval processes from the National Science Foundation and Defense Advanced Research Projects Agency, the largest technology and innovation-based awarding bodies in the U.S. In addition to revising the award process, CAS/UETP seeks to develop a deeper understanding of the product development life cycle and critical processes related to it. Though CAS/UETP initially sought the support of its local ORIC to pursue innovation and commercialization of research outputs, it has realized that the ORIC at UETP itself lacks the knowledge and experience to build this capacity internally. ASU has a critical role to play in helping CAS/UETP (and CAS/NUST) researchers develop best-practice approaches to product development and commercialization. Such training should also include training in effective proposal writing, particularly in helping CAS faculty (and UETP ORIC staff) to take advantage of international grant opportunities over the medium term. USAID can help ensure that ASU delivers training (or a similar type of support) targeted to CAS/UETP’s specific needs in this area. Additionally, HEC can help UETP ORIC (and CAS/UETP) in this effort by providing opportunities for the more accomplished ORICs in the Pakistani higher education system to share their knowledge of how to pursue effective commercialization and innovation strategies. With these changes and capacity opportunities, CAS/UETP will be better positioned to understand industry problems, produce innovative research designs, and find marketable solutions. Through this effort, the center’s research will help establish new funding streams that can iteratively fund ongoing research initiatives. Adapt the promotion process for faculty to emphasize applied research and commercialization: The currently limited number of standing faculty means that faculty workloads, particularly in teaching and administration, limit their ability to take on additional, revenue-generating responsibilities. Although the university’s recent approval to hire additional CAS/UETP faculty will significantly alleviate this constraint, CAS must attend to freeing up sufficient faculty time so that faculty can focus on applied research and commercialization opportunities. One way to address the current situation is for UETP to amend faculty promotion criteria for CAS faculty, expanding the traditional emphasis on publishing in academic journals to formally recognize industry-focused, applied research in promotion decisions. Such a shift will provide a critical, yet missing, incentive for faculty to undertake the kind of applied research and commercialization opportunities that will move the center to full sustainability. HEC can advance this process significantly by encouraging CAS host universities, and in particular UETP, to amend faculty promotion criteria. During the National Steering Committee meeting, the HEC chairman was clear that he expected the centers to “put on a different hat” and not operate under traditional academic approaches. If HEC is serious about this, then it needs to use its influence to create the enabling environment in which each CAS can realize its goal of becoming a catalyst for academic U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 53 innovation to have an impact in the energy and water sectors. USAID can use its influence to encourage HEC to make the necessary changes before the project ends so that the investment it has made in CAS realizes its potential. Adapting faculty promotion criteria will position CAS/UETP to strengthen and consolidate its brand as a valued resource to sector organizations, both public and private. The center may derive additional downstream revenue-generating opportunities by adapting promotion criteria to CAS incentive structures. As the center and its faculty grow in stature in the sector, commercial organizations may finance academic chairs and professorships. LUMS is a prime local example of the potential for setting up endowed chairs and professorships. Establishing such chairs requires marketing and outreach to corporate entities that have the resources and interest to finance such a legacy. The return can be significant, not only in terms of revenue potential, but in terms of the center’s ability to attract and retain top talent. Medium Term Build capacity to deliver nondegree training, online learning, and blended learning opportunities: As CAS/UETP continues to grow the size and capacity of its faculty, the center plans to enact a strategy for offering a portfolio of nondegree training programs, online learning, and blended learning options. Online and blended learning options will allow CAS to pursue revenue that is not as constrained by limited faculty time and classroom space. The center has recently hired a specialist staff member who was trained in nondegree and online learning at LUMS, a local leader in the field, and is mobilizing an IT expert to travel to ASU to learn from its distance and e-learning experts. ASU maintains an online presence for conducting such programs through multiple “massive open online course” platforms. CAS/UETP has also examined the offerings of LUMS and IBA, both of which are considered successful local examples of market leaders in the field. Initial offerings will be modest and incremental, designed based upon the skills and topics in which local stakeholders have expressed an interest, such as energy efficiency and energy management. As part of the center’s marketing plan, it will also conduct a market survey to understand demand and to design a more complete portfolio of courses. Additionally, the center is designing incentive plans that will compensate faculty for their additional work on course design and delivery. Both HEC and USAID can support this initiative by helping CAS/UETP scale up its course offerings as they are developed. HEC can help build awareness through its existing networks and access to national media. Similarly, USAID can support the marketing of these programs by providing access to organizational partners and stakeholders in the U.S. energy portfolio, as well as through the U.S. Commercial Service in Pakistan. Offer professional certifications: In the medium term, the center plans to offer professional certifications to energy professionals as an iterative component of its nondegree training and online learning portfolio. The United Nations Industrial Development Organization and the Global Environment Facility currently offer similar certifications for energy auditors. Possible areas for certification include energy efficient technologies and renewable energy technologies. Ongoing market needs assessments will help identify more areas for potential certification. CAS/UETP already has the internal capacity to deliver such certifications through its standing faculty. However, curriculum approval, certification profiling, and documentation and promotion of programs will require support from the university and industry organizations. Development of a comprehensive marketing function is critical to this process. Additionally, as discussed previously, USAID and HEC can support this effort through their existing networks and access to national media. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 54 Commercialize CAS/UETP labs for testing: Commercializing labs can provide a reliable revenue stream. The labs have the technology needed to serve as testing platforms for outside organizations. To realize and optimize the commercial potential of the labs, the center must develop a comprehensive strategy, including price modeling and utilization schedules. Additionally, it must hire and train dedicated technical staff to secure certification of equipment and perform tests. ASU is intended to help develop the center’s lab commercialization strategy, but its focus on this has been relatively small to date. USAID can support this effort by helping ensure that ASU becomes proactive in engaging with CAS/UETP to develop a viable commercialization strategy that includes market research, pricing, staff development, and certification. Long Term Create spin-off companies for technology transfer: The revenue potential of creating spin-off companies to commercialize technology is something that CAS/UETP seeks to explore over the long term. The center notes that the cooperative agreement permits creation of such companies and plans to leverage existing funds to the extent possible to create the enabling environment to realize this objective. To optimize the revenue potential of this effort, it is important to focus both on ways to innovate new solutions and on identifying niches in existing energy products and services. CAS/UETP must undertake some activities in the short run to realize the potential long-term payoffs it may derive from this strategy: • Orienting faculty toward commercialization of technologies. This can be achieved through in￾house training and collaboration with other universities to learn from their experiences of commercializing research work. The previously described ORIC training is an example of this. • Building the CAS/UETP and ORIC staff and capacity, as previously described, utilizing to the extent possible the expertise of ASU in this area. • Leveraging the USAID network to identify contacts in its commerce portfolio. Evaluation Team Recommendation Regarding CAS/UETP Strengthen the managerial team at CAS/UETP: The evaluation team was very pleased to observe the progress that CAS/UETP has made since it visited Peshawar for the validation workshop in June 2018. Nonetheless, there is still a need—actually, an excellent opportunity—to strengthen the managerial capacity of the CAS/UETP team so that it is able to realize many of the opportunities it has to develop its own distinctiveness. It is critical that USAID, HEC, ASU, and UETP work together to address the capacity needs of the center. Evidence from the evaluation suggests that the lack of a collective focus on behalf of all stakeholders has limited the ability of the center to establish a solid managerial infrastructure upon which to anchor its future. The findings validation workshop led the evaluation team to conclude that the center has a much greater opportunity to achieve sustainability as a CoE than originally thought, although it is partially encumbered by its ability to secure key staff. For example, to balance the strong innovation-oriented approach of the current CAS/UETP leadership, the center would benefit tremendously from a strong operations manager who is able to prioritize the many opportunities for growth the center has, and to develop implementation and/or business plans that can ensure their success over time. The entrepreneurial orientation and creative insights of the current CAS/UETP leadership should not be taken for granted—they have significant potential. Properly channeled and supported by strong operational capacity, they can help place the center on very solid footing for the long term. Rather than expecting other partners to address current gaps, all partners U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 55 need to come together and take a collaborative orientation to solutions to set the center on a solid path forward. Additionally, USAID should consider measures to help CAS/UETP and CAS/MUET work together. The challenges of creating a CAS in Jamshoro are somewhat similar to those of establishing a CAS in Peshawar, and UETP can learn from the success of CAS/MUET, in particular the way the center differentiates itself and builds institutional capacity through its clear research agenda. Additionally, potential opportunities for collaboration between the energy and water sectors could give CAS/UETP further opportunity to differentiate itself. CAS/MUET Self-Identified Recommendations Short Term Secure CAS/MUET post-project status as an autonomous CoE: To ensure that CAS/MUET continues on its current trajectory as an innovative link between MUET and the water sector, the CAS/MUET team argues that the center’s current autonomy and board governance system must be preserved. This is essential if the center is to “put a different hat on” and continue to move beyond traditional academic operating models as the HEC chairman advised during the National Steering Committee meeting. Though the current university leadership supports the present model, it is essential to formally secure the structure for the longer term. Without formally establishing the governance model, the center’s ability to forge innovative sector links and solutions could be put at risk if future university leaders decide to change governance structures. HEC has a supportive role in securing the governance model for the long term; however, HEC notes that the center must demonstrate that the structure is essential to developing the CAS brand name and to generating revenue for sustainability. If CAS/MUET meets this requirement, it can approach university management to support an autonomous governance model into the future. Since CAS/MUET is still working toward sustainability, center leaders must be more strategic in how they make the case for ongoing autonomy. For this reason, it may be more realistic to make the case for continuing the existing autonomous governance model for a trial period of at least one year beyond the end of the U.S.-PCAS project, during which time the center can work aggressively to secure long-term sustainability. With concrete evidence to demonstrate that the autonomous governance structure is essential to the center’s self-sustainability, a stronger argument can be made for securing the model for the long term. In the short term, USAID and HEC have critical roles to play in this discussion. Both are clear in setting expectations for what the centers should achieve, but they must also help create the enabling environment in which those achievements can be realized. The evaluation revealed that USAID has largely left it to the host universities to determine the best governance model going forward. The outcome has been that universities largely tend to seek to absorb the CAS governance structure into existing university structures once the project ends. CAS/MUET in particular has maintained that absorption would make CAS like any other university center, with the strong risk that its ability to provide innovative solutions and cutting-edge applied research to the water sector will be significantly curtailed. Consequently, CAS representatives argue for ongoing alignment of the governance structures to university structures, as detailed above, rather than absorption. The HEC chairman’s expectation that the centers not act like traditional academic institutions, but rather provide innovative solutions, provides a strong foundational argument for re-examining how HEC and USAID consider governance going forward. Their support in securing a more autonomous model for CAS/MUET (and the other CASs) over the medium term is essential. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 56 Initiate a flagship training program: The center sees customized nondegree training courses as an important way to generate revenue for sustainability. To this end, one governmental department has already set aside funding and committed to sending its employees to the courses. To date, the center has already customized eight courses and is negotiating their anticipated delivery with various governmental departments. To grow this foundational capacity to a larger scale so that it contributes meaningfully toward sustainability, several mechanisms must be put in place. These include a fully developed marketing and sector outreach plan, as well as a team able to implement this plan and build relationships with sector organizations. To establish these mechanisms, CAS/MUET will need to work with the University of Utah to ensure that the marketing and outreach planning processes are pursued rigorously during the final year of the project. This work should align with the market needs assessment that was underway during the evaluation; however, the marketing and outreach plans must be comprehensive and action-oriented so that the center can use the remaining project funds as appropriate to build internal capacity and aggressively begin outreach efforts as quickly as possible. USAID can contribute to this process by making its expectations on marketing clear and pushing all the centers to establish milestones and timelines so that AORs can follow up regularly. Medium Term Launch and scale the planned drinking water initiative to generate revenue: The center is currently working on a drinking water project. It has developed a business plan, with forecasted annual revenue anticipated to be PKR 7 million. The center has initial resources and expertise available to launch this initiative, though over time ongoing reinvestment in capacity will help the center scale the initiative and its revenue. Implementation of this initiative can also provide the experiential knowledge necessary to identify opportunities to develop and launch similar initiatives. HEC and USAID can aid this effort through their networks. HEC can advertise the initiative nationally as an example of the cutting-edge work that CAS/MUET is undertaking. USAID can support the initiative by building awareness among its partners in other programs in Pakistan and the region. For example, at the public-private partnership workshop between the USAID Mission in Karachi and Proctor and Gamble, both organizations identified drinking water for rural communities as a collaborative opportunity warranting further exploration. CAS/MUET’s work could align well with this collaboration. Build stronger linkages with the government of Sindh to expand applied research: Now that CAS/MUET has built the capacity and reputation for producing quality applied research, it needs to promote this capacity in order to make it a source of revenue. This requires stronger linkages with provincial governmental departments and federal ministries. CAS/MUET estimates that the government of Sindh allots 1 percent of its annual budget for monitoring and evaluating its diverse projects. The applied research capacity of the center, at both the faculty and student levels, gives it the potential to tap into some of this revenue. To do so, the center needs a resource mobilization unit with the expertise to identify opportunities and to coordinate with the government. As CAS/MUET delivers quality analyses for the provincial government, it can seek to leverage its expertise to build opportunities in other provinces, with the federal government, and perhaps internationally. Evaluation Team Recommendation Regarding CAS/MUET Ensure CAS/MUET is focusing adequately on the business development side of CAS: The center has done a fine job of building its internal capacity for applied research and graduate training. Feedback from sector organizations about both has been strongly positive. It appears, however, that the center is not yet adequately focusing on the business aspects that are essential to its long-term sustainability. This means ensuring that it prioritizes the development and implementation of actionable marketing, U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 57 outreach, and lab commercialization plans. This is something that all centers must do, but it is of critical importance for CAS/MUET given its remote location in Jamshoro. One piece of feedback from the partner university that emerged during data collection was that ASU struggled with pushing CAS/MUET staff to think of CAS as a center to be sustained rather than as a project. USAID and HEC should push the center to produce marketing and outreach plans, and to work toward securing the human resources it needs to activate them. General Cross-CAS Recommendations for USAID and HEC The following recommendations are for USAID and HEC to consider to keep the project focused during its final year so that outcomes have the greatest chance of being sustained. They are informed by some of the CAS-specific issues/recommendations that emerged from the recommendations workshop. Continue to challenge centers through a clear, outcome-oriented focus on sustainability: USAID can redefine its expectations regarding sustainability to shift the focus away from existing project outputs articulated in the cooperative agreements and work plans to longer-term center outcomes based upon the evolving capacity of each center. The outputs around which the project articulates sustainability are insufficient to help ensure sustainability of outcomes once funding ends. Currently, evidence suggests that centers assess longer-term sustainability primarily in terms of hard assets and infrastructure (building, labs, and faculty compensation) and less in terms of optimizing use of the softer assets such as internal capacity and external relationships needed to generate an adequate level of revenue to support outcomes. A focus on both is critical, especially in the new reality where HEC will support only select activities of the centers once USAID funding ends. Redefinition can be grounded in the existing USAID definition of sustainability (ADS 201) so that it focuses on the perceived value that key beneficiaries—including sector organizations, faculty, students, and host university leadership—place on the center so a focus on outcomes can be strengthened. A clearer focus on the sustainability of outcomes should also consider the link between project components and sustainable capacity, so that centers are working toward a future that is indeed sustainable. In the evaluation, USAID emphasized that centers were allowed a significant amount of leeway in deciding how to use project resources to establish each center distinctively in terms of applied research, curriculum, and other project components. However, at times it is not clear that the way each component is implemented supports longer-term capacity for sustainability. For example, the NUST decision to use CAS exchange program funding to send non-CAS students and faculty on exchange to ASU, though good for building the short-term name of the center, potentially dilutes development of internal capacity in the long term. CAS/UETP recognition that its approach to applied research limits revenue generation is another example. Though each CAS should be able to pursue its own distinctive path forward, decisions should be guided by how they will support (or diminish) longer-term efforts toward sustainability. This requires thinking of each CAS not only as part of a USAID project, but also as a center that is expected to have a lasting impact in its sector long after USAID funding ends. With the future of center support looking very different than the centers anticipated, it is essential that the centers prioritize the components they can realistically sustain and be prepared to make the (sometimes difficult) strategic trade-offs necessary to continue to build each CAS into a respected brand in the face of challenges. USAID and HEC should collaborate proactively to find creative ways to promote the sustainability of each center: USAID should seize the opportunity provided by the leadership change at HEC to engage the commission as a renewed collaborative partner in the long-term success of each center. During the National Steering Committee meeting, HEC presented the centers with a tough path forward: HEC will not assume broader financial responsibility for the centers in the manner many had U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 58 anticipated once USAID funding ends. Additionally, centers cannot reallocate the remaining funds (which were considered as savings in some instances) that USAID has committed for the project and must adhere to already agreed work plans and budgets. Thus, the centers must quickly identify new sources of revenue to support their operations once the project ends, but they have limited scope to plan strategically and allocate resources adaptively in anticipation of that challenging future. Though this does not suggest that USAID or HEC reconsider the parameters they have set, it does suggest that the two institutions collaborate proactively to find creative ways to help the centers as they wrestle with the constrained situation in which they find themselves. Some excellent examples of such collaboration arose during the National Steering Committee meeting and the recommendations workshop. HEC has made clear that it can selectively support innovation initiatives that have a strong potential for financial sustainability. The chairman’s commitment to match applied research funding secured from other areas of the GoP is a good example. Similarly, USAID’s suggestion of using approved staffing budgets to secure short-term consultants who can build capacity is another. The joint suggestion to seize the opportunity to promote the centers nationally on television and radio is another. Centers and their host universities must be challenged to build their own sustainable futures, but the more that USAID and HEC can challenge themselves to find creative ways of supporting them in this process, the greater the chances will be for each center to be successful. Both institutions should also consider “in-kind” ways to support these efforts through their existing networks and programs. Ensure that centers develop realistic and comprehensive marketing, outreach, and lab commercialization plans, including securing the resources to implement those plans: To strengthen the linkage between supply and demand, each center could greatly benefit from establishing a strategic marketing function that not only cultivates deep sector relationships, but also adapts messaging to communicate research results appropriately to the constituencies the center seeks to cultivate. Planning for marketing, outreach, and lab commercialization is essential to achieving this goal. In key informant interviews, center stakeholders referred to the need to develop marketing, outreach, and lab commercialization plans. In the recommendations workshop that took place more than six months later, it appeared that the centers had made little progress in these areas. The evaluation team came away from the recommendations workshop with the sense that the centers do not appreciate the criticality of sound business planning and implementation for their future survival. Participants in the recommendations workshop discussed the need for these plans too abstractly, rather than as critical tools to their future sustainability. As a matter of urgency, USAID and HEC can set clear milestones for these plans, as well as expectations for what they should contain. For example, the HEC chairman noted that he expects to see sound cost￾benefit analyses for proposed initiatives, as well as strategic decisions for pursuing initiatives with the greatest return on investment. If development of such plans is meant to be led by the U.S. partner universities, then they should be held to the same deadlines, expectations, and focus on sustainability as the centers themselves. USAID and HEC may also wish to consider offering some high-level institutional capacity building for center and university leaders to ensure that key managerial staff develop the perspective and skills needed to steward the entrepreneurial aspects of the center. The chairman made clear his expectation that the centers not act like traditional academic institutions, but rather “put a new hat on” to be engines of higher education entrepreneurship and innovation. Communicating this vision is one thing, but building the capacity to achieve the vision is another. Create a career services office at each CAS: Students at each CAS were critical of the limited help they receive from their respective centers as they pursue employment after their studies. Though the U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 59 centers appear to expend more effort on helping students find internships than employment, even this support could be stronger. Postgraduation employment support is largely nonexistent. Multiple student groups described the struggles they face in trying to convey the value proposition of a “water” or “energy” master’s degree to potential employers, and a number of students noted that employers place more value on their undergraduate degrees than on their master’s degrees. Establishing adequately resourced career services offices at each CAS could serve the longer-term sustainability of each center, for such efforts would strengthen the perceived value that sector organizations place on one of the critical products of CAS: highly trained professionals. HEC should help facilitate discussion of the best structure for CAS governance going forward: The HEC chairman was clear about his expectations for each center, and USAID can use its influence with HEC to help it create the enabling environment for centers to meet those expectations. Governance is an area where HEC and USAID can help centers by reviewing the wisdom of aligning CAS governance structures to existing university governance structures. CAS/MUET has raised this as a critical issue for ongoing sustainability of outcomes, and recommendations put forth by CAS/UETP implicitly raise the same issue. USAID can encourage HEC to take the initiative to engage centers, host universities, and partner universities in frank discussion of the underlying assumptions regarding the impact of alignment on center sustainability, as well as define precisely what is meant by alignment. As noted in the evaluation findings, definitions should consider the critical differences between aligning CAS governance structures to existing university structures and absorbing CAS governance structures into existing university structures. Given the chairman’s vision that centers not act like traditional academic entities, HEC needs to do what it can to create the enabling environment that gives centers the best chance of attaining that vision. Governance is key. Promote greater linkages across the CAS network, including collaborative research, joint programming, and resource sharing: Not all of the challenges that confront Pakistan fit into discrete categories such as water and energy. Now that centers have built capacity on an individual institutional level, HEC and USAID should strongly encourage them to explore collaborative solutions to issues through joint research, by sharing of resources and knowledge, or in other ways. This issue arose in the recommendations workshop, and the evaluation team noted that CAS/MUET and CAS/UETP showed significant interest. For reasons described in the evaluation findings, USAID may wish to consider measures to encourage greater collaboration between CAS/UETP and CAS/MUET. Though the centers at both UETP and NUST are similar in their common focus on the energy sector, UETP and MUET have more in common as institutions than do UETP and NUST. CAS/UETP and CAS/NUST offer similar programs and capabilities, but CAS/UETP noted that it tends to be placed at a disadvantage because of its location. The challenges of creating a CAS in Jamshoro are somewhat similar to those of establishing a CAS in Peshawar, and UETP can learn from the success of CAS/MUET. The approach used by the University of Utah in building sector capacity, as well as the way the center differentiates itself through its clear research agenda, could be of use to CAS/UETP, particularly given the latter’s expressed interest in restructuring the way it approaches applied research. USAID should encourage HEC to continue the U.S.-PCAS project’s focus on gender equity once USAID funding ends: Over the remaining life the project, USAID can maintain focus on gender through stretch goals for student recruitment and staffing while also encouraging stakeholders to identify innovative approaches that support the ongoing push toward gender equity. Each CAS appears to have had greater success in gender equity in student recruitment than in recruitment for faculty or management positions. Either as part of the discussion on governance models going forward, or perhaps as a free-standing discussion, USAID could work with HEC to explore ways to continue the CASs’ current trajectory as a model for equity. This can include innovative ways of identifying and engaging female professors at each CAS. It can also include setting targets and creating the enabling conditions U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 60 where women eventually hold senior leadership roles in each center, an achievement that each CAS has yet to realize U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 61 ANNEX 1: STATEMENT OF WORK U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 62 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 63 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 64 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 65 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 66 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 67 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 68 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 69 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 70 ANNEX 2: ASSIGNMENT WORK PLAN U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 71 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 72 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 73 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 74 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 75 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 76 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 77 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 78 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 79 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 80 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 81 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 82 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 83 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 84 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 85 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 86 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 87 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 88 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 89 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 90 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 91 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 92 ANNEX 3: FURTHER ANALYSIS OF PROJECT COMPONENTS CAS/MUET Effectiveness of Exchange Programs The exchange program is a central feature of the U.S.-PCAS project, which provides opportunities to faculty and students to visit partner universities in the U.S. The objective of the student exchange program is to encourage opportunies for research and training while providing the necessary modern research knowledge, skills, and techniques. The faculty exchange program at MUET is primarily based on the alignment of faculty research interests with an available professor at the U.S. partner institution. This section explains the process and impact of the exchange program for both students and faculty at CAS/MUET. Exchange Program Process The most important activities in this exchange program process are the selection of students and faculty, their preparation, and the support provided by the host university for adjusting to the environment in the U.S. The evaluation team used both surveys and qualitative interviews to collect in-depth information about the exchange program process. For CAS/MUET, this analysis addresses the responses of recent alumni and students together; a total of 17 faculty members, 20 alumni, and 125 students participated in the survey. Transparent Exchange Program Selection Process Nine of 17 faculty members applied to the exchange program, and all confirmed the transparency of the selection process. Less than half (9 of 19) of the CAS/MUET alumni felt that the process of selection was transparent. However, 61 percent of the current students felt that the selection process was transparent, 28 percent did not know about the criteria, and only 10 percent ranked the process as not being transparent, which is a lower proportion than reported at the other CASs. Eighty-four percent of alumni who applied reported that they were selected for the exchange program, and all of the selected alumni participated in the program. In comparison, only 53 percent of the current students who applied were selected. Ninety-five percent of the selected students participated in the exchange program (Table 7). TABLE 7: ONLINE SURVEY RESPONSE RATE FOR CURRENT STUDENTS Exchange Experience CAS/MUET (n=125) Applied for exchange 70/122= 57% Accepted to exchange 37/70= 53% Participated in exchange 35/37= 95% The selection criteria were changed over time as the project evolved. One member of an alumni group said that he was told three times that he would be going the next time. However, he was not told why he was never selected. He reported that the criteria kept changing and he was very depressed when not selected after accomplishing the requirements set out each time. Two members of a four-person student group felt that merit selection was not followed and supervisors sent their favorites for the exchange. Members of another group of students complained that the University of Utah did not read U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 93 the student research proposals, which formed the basis for selection, and that students interested in technical programs should not be required to go to the entrepreneurship course. Particularly for students, these changes in the criteria for the exchange selection process caused confusion and frustration. A PMU team member shared, “It was time to fix the criteria. The fourth and fifth group was preparing for the exchange program. Still, every time, deviations [changes] were being made from the criteria published in the CAS-W handbook. It was very frustrating for the students.” One of the student groups reported, “The selection process was quite effective, but delayed. The University of Utah faculty had not read the research proposals.” A few believed they should have been selected based on a comparison of grade point averages with fellow students. One student shared “I wanted to participate, but I was not selected, despite having a 3.48 GPA. I was also not even given a reason why I was not selected.” An important role of the U. S. partner in the exchange program is guiding the local partner in the effective implementation of the program. A CAS/MUET faculty member shared that “the selection process is there but they [University of Utah] keep changing the criteria, and this is a problem. Students do not know the criteria; it is managed at the last minute. The intention is to improve the process, but my objection is on timely information.” One Utah team member shared that “every time there were new ideas for the selection process, there was confusion at the start of the exchange. However, we are moving in the right direction.” The following is the key statement of the Utah faculty member responsible for the selection process: “We were very selective for the first group of students to convey the message to work hard to be in the exchange. We assessed four different things: an essay, a personal statement, their grades, and faculty evaluation. We looked at their attendance. It was so low; 80 percent, 85 percent ... I said in this modern age you couldn’t afford 75 percent. I said I am not going to consider anybody from next year [with] less than 100 percent. When I teach, if somebody misses one class I give them a warning, if they miss two classes, I throw them out. Here you have people who are getting a full scholarship and a professor is trying to teach them, and they miss one-fifth of the classes! So in any case, today it is 100 percent. For grades, we just took students from the top. Faculty evaluation was useless because what they did was everybody was supposed to be the best person around after Einstein. I told the faculty as I said I couldn't use your evaluation because they are of no use to me unless you can distinguish between the good, bad, and ugly, it is of no use. So that was the way of educating the faculty as well. Then we told the students as well if your faculty gives you a good evaluation that you do not [deserve], it is of no use. Then finally their essays and research proposals—we had extreme difficulty with them. We set up a system so that they could revise them again and again. We actually put a little bit [a lot] of effort into this. Now the thing that has happened—and this is a very interesting phenomenon—we do not now have to repeat it. It is actually part of the students’ own knowledge, and it is getting transmitted from student to student. We now don’t have to talk about this at all; what is expected of students is in some sense apart from the academic culture of CAS-W. It is actually happening directly because of them.” Adequate Preparatory Support for Faculty with Some Issues in Students’ Preparation All of the nine CAS/MUET faculty members who applied for the exchange program agreed or strongly agreed that CAS had adequately prepared them for the exchange program, compared to 93 percent across all CASs (n=15). In particular, 44 percent of CAS/MUET faculty strongly agreed that CAS/MUET had adequately prepared them for the exchange. This level of agreement was not found in responses by the faculty of either of the other two CASs. The adequacy of logistical preparations for the visit by CAS/MUET and USAID is supported by a direct quote from a CAS/MUET professor who said, “There were no difficulties, no visa problems, and no entry problems at immigration.” U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 94 Regarding student preparation for the exchange program, overall 94 percent of the alumni and current students agreed or strongly agreed with the statement that CAS/MUET had prepared them adequately before the start of the program. For current students, 75 percent felt that the host university had adequately supported them in adjusting to the new environment. However, the situation appeared to be different in the qualitative interviews. A majority of University of Utah representatives (four of seven) shared that they were not satisfied with the CAS/MUET preparation of students for the exchange program. This was qualified by one Utah representative who shared: “They do prepare, but not to the level where we are telling them they should be prepared. I think they are improving over time, but again we need to look at why they are not doing it. My guess is that there is probably a capacity issue. They know that their students should be able to perform this, and faculty should be able to prepare well-shaped proposals, but usually … the proposals are not well prepared.” Similarly, in a key informant interview, a CAS/MUET faculty member shared: “CAS does not facilitate [students] adequately. [CAS/MUET] tells students [that] they can check the University of Utah website and find relevant professors to contact and collaborate with. [Once students reach out to a Utah faculty member,] the Utah faculty [member] rejects a student but does not tell him/her that there are six to eight other [faculty members] relevant to his/her field. There is no mechanism for that.” Eighty-nine percent of the CAS/MUET faculty members who participated in the exchange program (n=9) agreed or strongly agreed that the host university had adequately supported them in adjusting to the new environment. In particular, one-third of the CAS/MUET faculty members strongly agreed that the host university had made adequate support arrangements. This level of agreement was not found among the participating faculty members at either of the other two CASs. In regard to Utah’s efforts to support exchange scholars in adjusting to the new environment, six of seven interviewed participants at Utah showed their satisfaction with the arrangements carried out at their end. A female faculty member of CAS/MUET shared, “I was upset at leaving my family behind, so the University of Utah offered for my daughter to join me. Subsequently, the rules were changed, and a male colleague was allowed to bring his family along.” Conclusion – Exchange Program Selection Process and Preparation There is substantial evidence in both the qualitative and quantitative results that the exchange program selection and preparation processes have evolved positively, and that they have gained a gradually higher level of satisfaction among both students and faculty. One Utah faculty member said, “The University of Utah has created a very hospitable environment for CAS-W faculty and students, though there was a lot of learning by doing involved. Utah tries to make sure that visiting faculty and students are not hampered by mundane matters like housing. Utah really takes care of all these things, enabling the visitors to engage [in] the educational process and even the social life of the university.” However, the process remains challenging for all involved stakeholders. The University of Utah, being a key part of this program, has acknowledged the substantial improvement in Utah and CAS/MUET facilitation of the exchange process over time. Some voices in the interviews recognized that there are differences in student and faculty exchanges in that the later have remained less structured. One faculty member shared, “Faculty exchange has been different from the student exchange. We have had fewer, less well-designed faculty exchanges. It may be because it is difficult, as faculty members are all at different levels.” U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 95 Exchange Program Impact Overall, a total of 17 faculty members, 20 alumni, and 125 students from CAS/MUET participated in the surveys. In the context of the exchange program, the alumni are recent graduates and alumni and student qualitative responses are addressed together. This section examines the responses of the qualitative interviews of selected CAS/MUET and Utah faculties, and CAS/MUET alumni and students. The transparency of the selection process, the adequacy of preparation, and host university support in adjusting to the environment in the U.S. are addressed in the previous section. Transformative and Impactful Experience for Students and Alumni Ninety-four percent of CAS/MUET alumni reported that the exchange program had improved their research skills, and 88 percent confirmed that it had improved their research output (Figure 9). Importantly, 88 percent of alumni confirmed that their proposal writing skills also improved as a result of the exchange program. Eighty-three percent and 77 percent of current students surveyed reported that the exchange program had improved their research skills and outputs, respectively. Ninety-one percent of students and 88 percent of alumni stated that the exchange had improved their proposal writing skills. Students and alumni were also asked about the exchange program’s contribution to improving their teaching skills; however, the question was applicable only to Ph.D. students. In response to this question, 46 percent of current students acknowledged that the exchange program improved their teaching skills. Thirty-seven percent stated that the question was not applicable to them. FIGURE 9: IMPACT OF EXCHANGE PROGRAM ON CAS/MUET STUDENTS AND ALUMNI Source: PERFORM survey of CAS/MUET students (n=35) and alumni (n=16). During individual interviews conducted at CAS/MUET, the evaluation team also asked the CAS faculty, administration, and leadership about the exchange program’s impact. Overall, respondents who talked about the students’ research skills all agreed that the exchange program improved the research skills of the visiting scholars (20 responses including 11 from CAS/MUET and 9 from Utah). The center administration reported an increase in student confidence: “For example, when the first batch came back, the council general [Karachi] and USAID deputy Mission director visited and interacted with the U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 96 exchange students. They were very happy to find that students were more vocal and confident.” One faculty member was clear about what the exchange achieved for the students: “The problem-solving skills of all [have], of course, improved. Student writing has improved. They are presenting a lot and reading a little. Writing boot camps have improved writing skills, but reading is a habit that takes more effort to develop.” A female faculty member observed great changes in the exchange students’ career paths. She further shared that some have received good scholarships, and she noted that four students were offered Ph.D. courses and are studying in the U.S. Students have better applied research skills and perhaps better problem-solving skills. She felt that initially some students wasted their time, but generally, they acquired better soft skills, especially writing, because of writing boot camps and presentations. She agreed it was a challenge to instill reading habits, but observed some improvements in the students’ reading skills too. There was a consensus among the Utah faculty that students benefited from the exchange, and that the program itself was improving. One Utah faculty member said: “The first group that came, their grades were B and below, and some people got B+, nobody got As. One thing we discovered is that [students] were not used to reading … so we had to work very hard to make people read things. … So, we are actually getting to some of those questions, how to motivate students, monitor them, and how to make sure they do something that is helpful in the future. So, what we found is that from group one to group two to group three, there was a steady improvement in their performance in the courses.” The responses of alumni and students were more mixed. Alumni confirmed that the exchange program improved problem-solving skills. Alumni also confirmed that exchange motivated them to pursue a Ph.D. and to develop global linkages that will be useful when they need recommendation letters for higher education or international conferences. Moreover, alumni confirmed that the exchange program improved their communication skills, as they had interactions with American students as well as students from other countries (for example, a group from Korea that was staying at the same hostel at the University of Utah). Similarly, all three group interviews with current students confirmed the exchange program’s impact on research, communication, and problem-solving skills. Students who were not exposed to the exchange program as yet shared their perceptions about other students who had been on exchange visits to the University of Utah. Two different groups of students confirmed that the availability of a supervisor’s time and proper guidance were prerequisites for a fruitful exchange visit. One of the exchange scholars confirmed that she also did voluntary work with a cultural, diplomatic program. Faculty Reported Exchange Impact Seventeen faculty members participated in the surveys from CAS/MUET, and nine of these visited the U.S. for the exchange program. Eighty-nine percent of those who had participated in the exchange program agreed that the program had helped improve their research skills. This is compared to 33 percent and 67 percent who gave positive responses to this question at the other two CASs. The same percentage (i.e., 89 percent) also confirmed improvement in their research output, teaching, and proposal writing skills. Again, the proportion of CAS/MUET faculty who responded was higher than at the other two CASs. However, during qualitative sessions with the CAS/MUET faculty, Utah representatives, and students and alumni, the evaluation team also learned of the challenges associated with the faculty exchange program and its impact. Qualitative Findings The findings from the qualitative sessions are presented from three different perspectives: first, how the faculty and administration members perceive the changes in their own capacities and those of fellow U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 97 faculty members; second, how the Utah faculty perceives these changes; and third, how alumni and students see the changes. Perspectives of CAS/MUET Faculty The quantitative findings on the improvements in research and teaching skills are corroborated by several statements by faculty about their own experiences and observed changes in the skills of other faculty members. One respondent stated, “Now I can use a groundwater model, and have started my own research project with CSU [Colorado State University]. The University of Utah has helped me become a better teacher in preparing and motivating students to do research.” Another respondent said, “I attended a course on teaching. It helped a lot in making my teaching more effective.” She also observed that “CAS-W faculty applied research skills have improved, with room for more improvement; teaching skills have improved, but I am not so sure about changes in problem-solving skills.” Another respondent stated, “You can see a change in the faculty’s dedication and commitment after the exchange program.” CAS administration and university leadership were asked about whether the exchange program was valuable in the overall professional development of the awardees. Of the eight respondents from CAS/MUET, six confirmed that the program was valuable for the awardees. Two respondents opted not to answer the question due to their limited knowledge of the results. A few of the faculty members shared that the exchange program was not focused on improving teaching skills (three responses). As one faculty respondent said, “I do not think there is a change in faculty teaching methodology.” Another said, “Few faculty members have collaborated well with the University of Utah. Teacher skill enhancement is not a focus of the exchange program; during my time at least, it was not so.” Perspectives of Utah Faculty Some of the Utah faculty responses support the quantitative findings regarding the exchange program’s impact on the CAS/MUET faculty’s research and teaching skills. One Utah faculty member said, “We have observed changes in the applied research capacity of the faculty. We have a goal-setting program where we lay out exactly what the faculty member … wants to accomplish. That program is then monitored by a research host. We are seeing an incremental change from start to finish.” Another Utah staff member reported, “For the faculty, it was a huge change and a positive turn. After the exchange, I sat in their lectures and recorded and shared them with our colleagues. They have become diversified in terms of lessons and modules of teaching. The concept of evaluation of professors was introduced, setting up the learning objective, and doing an evaluation.” There were, however, contradictory views. One Utah faculty member said, “The teaching skills of the [CAS/MUET] faculty are not impacted through the exchange program as it is targeted on research. We have other elements which focus on the teaching.” Another said, “Faculty exchange has been different from the student exchange. We had fewer faculty exchange, maybe because it was not well designed. The faculty exchange is different because they are all at different levels.” A balanced, moderating response from Utah was, “We did our best; I do not think there was some serious failure. Some faculty members believe that they were not prepared enough in terms of lab skills and safety culture, and so on. [But] they were not exposed to these things, so it was difficult to train them.” Another moderating voice from Utah said, “With regard to the faculty, each one of these [teaching skills, research, problem-solving] is a different issue. We tried to impart teaching skills in Jamshoro itself through the effective teaching workshops, exercises, and so on. I would say we are about 50 percent successful.” Perspectives of Alumni and Students In contrast to the views of some CAS/MUET and Utah faculty members interviewed, many alumni and students believe there has been little effect on the research and teaching capacities of CAS/MUET faculty as a result of their exchange visits to the U.S. All three alumni in one group interview said that they did U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 98 not see any difference in the faculty owing to the exposure. One of them added that most of the faculty already had foreign degrees, so there was no reason to expect further changes. Two students out of four in a group interview said that they saw no improvement in the teachers who went on the exchange program, as their teaching methodologies were still the same. The other two students, who thought that the teachers were experienced and competent, did not relate their performance to the exchange program. A minority of alumni and students observed positive changes in the CAS/MUET faculty after the exchange program. One alumnus said that he noticed that faculty did not rely on the traditional methodologies and were bringing in some new ideas after the exchange, while another said that the faculty was more organized and considerate after the exchange. He added, “Before, we used to get emails at night, saying that we have a test the next day. We do not get those anymore.” Conclusion – Exchange Program Impact There is strong evidence in both the qualitative and quantitative results that the exchange program was a positive experience for students and that they developed research, communication, and problem￾solving skills. Other stakeholders (faculty, students who have not been on the exchange so far, administrators, and CAS leadership) also confirmed this view. With respect to faculty, though some report positive outcomes from faculty exchanges, others note that faculty exchanges need better planning to achieve clear outcomes. Research This section summarizes the results of the survey of faculty, alumni, and students regarding applied research, noting the salient differences between CAS/MUET and the other project CASs. In addition, it discusses the evaluation team’s interviews with selected leaders of sector organizations and higher education, CAS/MUET administrators and faculty, relevant Utah faculty, and groups of CAS/MUET alumni and students regarding applied research, joint research (with U.S. partner universities), and public-private partnerships. Quantitative Results A total of 41 CAS faculty members (17 from CAS/MUET), 88 alumni (20 from CAS/MUET), and 542 students (125 from CAS/MUET) participated in the survey. Their responses to 12 closed-ended questions provide the basis for the analysis in the following subsections. As gender is a crosscutting theme, the questions relating to gender and applied research are addressed both here and (more fully) in the discussion of gender mainstreaming (see “Gender” below). CAS/MUET Faculty Only 1 of 17 faculty members reported not being engaged in applied research. Of the 16 faculty members engaged in applied research since joining CAS/MUET, only one stated that his/her research was not relevant to the needs of sector organizations. All 41 faculty members at the 3 CASs felt that research by women scholars was acceptable to sector organizations, and all reported that women were indeed involved in research at their CAS. At CAS/MUET, all 17 faculty members insisted that women were provided opportunities to take the lead in research (while there was one dissenting opinion at each of the other CASs). However, 5 of 17 respondents (or nearly 30 percent at CAS/MUET compared to one-fourth across all the CASs) admitted that industry preferred men in lead roles. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 99 Four of 17 faculty members at CAS/MUET were dissatisfied with industry participation in the conduct of research. At 24 percent, the level of dissatisfaction was somewhat higher than the average of 17 percent across the three CASs. However, the share of those at CAS/MUET who were extremely satisfied with industry participation (4 of 17) was also higher by the same margin (24 percent to 17 percent across the CASs). Nine of 17 faculty members at CAS/MUET were extremely satisfied with the process of selection of research areas and topics, while another 6 were satisfied. The share of those at CAS/MUET who were extremely satisfied (53 percent) was more than double the average across the CASs (24 percent). Sixteen of 17 faculty members conducted applied research, and all 16 were satisfied or extremely satisfied with the financial support received from CAS/MUET to conduct the research. The share of those extremely satisfied with the financial support (44 percent) was four times that at NUST (10 percent) and twice that at UETP (21 percent). Thirteen of 16 faculty members at CAS/MUET responded that they were satisfied or extremely satisfied with the technical support (in research design, analytical tools, publication support, and guidance in the use of labs and lab equipment) from U.S. partner universities. The proportion extremely satisfied (31 percent) was three times that at NUST (10 percent) and slightly higher than at UETP (29 percent). Only 1 of 16 researchers (6 percent) at CAS/MUET was dissatisfied with the level of autonomy in spending research grants, while 10 were satisfied and 5 were extremely satisfied. The results compare favorably with NUST, where 50 percent of researchers were dissatisfied or extremely dissatisfied, and UETP, where 14 percent were extremely dissatisfied. Alumni and Students Fourteen of 19 alumni (74 percent) were engaged in applied research while at CAS/MUET, and 93 current students (79 percent) were engaged in applied research. In contrast to the improvement at CAS/MUET, across the three CASs the level of engagement in applied research dropped from 71 percent for alumni to 54 percent for current students. All 14 alumni engaged in applied research at CAS/MUET agreed that their research was relevant to the needs of the industry, while 85 current students (91 percent) agreed that their research was relevant to industry needs. The reported relevance is in line with the average levels prevailing across the CASs (86 percent of alumni and 94 percent of current students). Only one student at CAS/MUET disagreed with the statement on the relevance of the applied research, while seven said they did not know if the research was relevant. Fifteen of 19 alumni (79 percent) at CAS/MUET agreed or strongly agreed with the statement that there was a process in place to align the applied research to the explicit needs of industry, while 99 students (84 percent) also agreed or strongly agreed with the statement. The levels were higher than the average across the CASs (70 percent for alumni and 75 percent for students). Only 1 of 19 CAS/MUET alumni (5 percent) felt that research by women scholars was not acceptable to industry (another 2 reported they did not to know), while only 5 of 118 students (4 percent) felt that research by women scholars was not acceptable to industry (another 16 reported they did not know). The extent of this minority opinion (4–5 percent) was at the same level across the CASs. The next question, on industry preferences for research team leaders, set a higher standard for assessing gender bias. Again, only 1 of 19 CAS/MUET alumni (5 percent) felt that industry preferred male lead researchers to female lead researchers, while 2 reported they did not know. However, a larger proportion of students (21 of 118, or 18 percent) felt that industry preferred male to female lead U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 100 researchers, while another 20 reported they did not know. The extent of the minority view and uncertainty among students was similar across the CASs. Seventeen of 19 CAS/MUET alumni (89 percent) agreed with the statement that women were involved in applied research at their center, while 18 (95 percent) agreed that women were provided opportunities to take the lead in applied research. Similarly, 108 (92 percent) and 107 (91 percent) of 118 students agreed that women were engaged in applied research and that they were provided opportunities to take the lead in research. Thirteen of 19 alumni (68 percent) agreed with the statement that they had the freedom to choose their research topics at CAS/MUET, while 2 strongly disagreed and 4 disagreed. Sixteen of 19 alumni (82 percent) agreed with the statement that they had been adequately advised by faculty members for achieving high-quality research. One alumnus strongly disagreed, and two disagreed with the statement that CAS/MUET faculty provided adequate advice, but such disagreement was much less prevalent than at the other CASs (33 percent). One hundred of 117 CAS/MUET students (85 percent) agreed or strongly agreed with the statement that they had the freedom to choose their research topics, while 4 strongly disagreed and 13 disagreed. One hundred and four students (89 percent) agreed or strongly agreed with the statement that they had been adequately advised by faculty members for achieving high-quality research, while 5 strongly disagreed and 8 disagreed. This was lower than the level of disagreement at the other CASs. Seventeen of 19 alumni (89 percent) were satisfied or extremely satisfied with the financial support they received from CAS/MUET to conduct applied research. This return compares favorably with the 42 percent at NUST and 17 percent at UETP who were satisfied or extremely satisfied with the financial support they received for their applied research. Similarly, 86 of 118 students (73 percent) were satisfied or extremely satisfied with the financial support they received from CAS/MUET to conduct applied research. This return also compares favorably with the 55 percent of students at NUST and the 43 percent at UETP who were satisfied or extremely satisfied with the financial support they received for their applied research. Sixty-seven of 117 CAS/MUET students (57 percent) were satisfied or extremely satisfied with the level of autonomy they had in spending the research grants, while 10 were dissatisfied or extremely dissatisfied, and 40 reported they did not know. The corresponding levels of satisfaction and extreme satisfaction were 48 percent and 37 percent at NUST and UETP, respectively. Sixteen of 19 CAS/MUET alumni (84 percent) were satisfied or extremely satisfied with the technical support in research design, analytical tools, publication support, and guidance in the use of labs and lab equipment received from U.S. partner universities. This compares favorably with the 53 percent and 50 percent at NUST and UETP, respectively, who were satisfied or extremely satisfied. Similarly, 84 of 117 CAS/MUET students (72 percent) were satisfied or extremely satisfied with the technical support in terms of research design, analytical tools, publication support, and guidance in the use of labs and lab equipment received from U.S. partner universities. This compares favorably with the 52 percent and 49 percent at NUST and UETP, respectively, who were satisfied or extremely satisfied. Qualitative Responses The qualitative data presented in this section provides additional explanation of the quantitative results presented above by identifying supportive and corroborative—as well as contradictory and moderating—statements (including direct quotations) about applied research from the concerned stakeholders. In this way, it provides evidence to clarify the survey results and illuminate their U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 101 implications. This section also summarizes the additional information on joint research with U.S. partner universities and public-private partnerships that emerged through the interview process. Applied Research by CAS/MUET This subsection looks at the engagement of CAS/MUET in applied research from three different perspectives. It reports on how: 1. CAS/MUET faculty members perceive their own work, challenges, and opportunities in applied research; 2. Utah faculty members view the research work and its challenges and opportunities for CAS/MUET; and 3. Students, especially as interns, view the applied research work being undertaken at CAS/MUET. Perspectives of CAS/MUET Faculty CAS/MUET is undertaking applied research under the thematic framework of SDG 6. One faculty member mentioned a research project on canal monitoring with four partners that is relevant to issues faced by the Irrigation Department, Sindh. Another faculty member reported on her work in ecosystem services valuation and lake ecosystems management, and that of colleagues on industrial waste treatment, water treatment, and water resource management; all addressed solving water issues. A common thread in all the staff interviews was that CAS/MUET faculty members agreed that there are significant challenges and opportunities ahead in applied research. Perspectives of Utah Faculty Several Utah faculty members mentioned the CAS/MUET research program’s focus on meeting SDG 6 targets, especially the provision of clean water and sanitation for all. One interviewee gave examples of providing high-performance computing capacity to enable analysis for climate dynamics, lab work on antibiotic resistance, and nanoscale water treatment technologies, which are capacities that did not exist before at MUET or in Pakistan. Perspectives of Students One student felt that CAS/MUET and industry linkages were quite weak. The student felt that industry was not interested in research, and thus there was no project and no real work for interns at the Sindh Irrigation and Drainage Authority (SIDA), for example. Another student felt there should be a diversity of research proposals. Most students were working on the same or similar topics. The research agenda was quite narrow and excluded social conditions and linkages. A third student said that research should focus on public policy issues and on communication to policymakers. A fourth student said that the new specialization in water, sanitation, and hygiene (WASH) was a very good step for public hygiene. Joint Research This subsection reflects on the joint research of CAS/MUET and partner U.S. universities. It reflects the assessments made by CAS/MUET administrators and relevant Utah faculty, as well as the self￾assessments and peer assessments of faculty and alumni. Perspectives of CAS/MUET Administration One CAS administrator said none of the joint research projects with Utah had been completed so far, and when asked about any significant impacts, responded, “Not so far.” However, he mentioned prospects of joint research projects with other universities. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 102 Perspectives of Utah Faculty One Utah faculty member mentioned the mentoring partnerships that give each faculty member at MUET individualized attention. The arrangement enables effective joint research projects. He added that the teams that Utah brings to MUET periodically dedicate their time to collaborative research meetings, conducting field visits, and doing joint lab work to develop collaborative partnerships, in addition to meeting with external stakeholders. Perspectives of CAS/MUET Faculty and Alumni A faculty member reported that CAS/MUET has more than four joint research projects with U.S. partner universities. Another faculty member observed that joint research had been facilitated through a series of research grant writing workshops with the target of generating one idea per week. The participants were proceeding with grant writing proposals on the basis of their existing contacts at U.S. partner universities. Several alumni mentioned getting help from U.S.-based mentors, including those at Utah and the University of Nevada. They also mentioned participation in the research proposal writing workshops at CAS/MUET. They mentioned several challenges, including those of data collection and the costs of software. Some reported that CAS/MUET did not help with the data collection issues. Public-Private Partnerships This subsection assesses responses from four perspectives: 1. The view of HEC, which funds public sector universities and is a stakeholder in the U.S.-PCAS project; 2. The responses of water and water-related sector organizations (in the public, semi-public, and private spheres) with which CAS/MUET has established some degree of formal relations; 3. The views of the visiting Utah faculty; and 4. The self-appraisal of the CAS/MUET faculty. Perspective of Higher Education Commission A HEC senior advisor said it was a good thing that CAS/MUET sought to establish an executive master’s degree program on water policy. It was a most important area for empowering policymakers and bridging the policy weaknesses at the federal level. Perspectives of Partner Sector Organizations All five water-related sector partners interviewed for this evaluation were satisfied with their partnership with CAS/MUET. By itself, this may be only an acknowledgment of their professional relationships; however, how they described the benefits is much more significant. The head of a sector partner organization stated that CAS/MUET provided a valuable third-party perspective on the environmental impacts of the dam being built for Thar coal exploitation. Also, CAS/MUET has engaged the Federation of Pakistan Chamber of Commerce and Industry in water sector issues. It has sponsored discussions on water quality, on competing water demands (with agriculture), and on the need for water efficiency. Similarly, the Pakistan Business Council, comprising CEOs of leading industries, has for the first time concerned itself with water issues. The respondent added, “The CAS-W initiatives have been successful because they facilitated government departments and university professors to come out of their silos, and generated the goodwill for further action.” U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 103 Perspectives from Utah A Utah faculty member mentioned the Business-Academia Committee on Water established in the second year of the project as a focal point for meeting industry needs for ultra-pure water, wastewater treatment, and developing efficiency in manufacturing processes, and to catch the interest of the private sector. Utah and CAS/MUET have started research projects with the private sector. However, these projects are yet to be concluded, and tangible funding is yet to be received from these entities. He added that the CAS/MUET faculty’s lack of experience in identifying what the industry or nongovernmental organizations (NGOs) wanted from university researchers was an inhibiting factor for public-private partnerships. Perspectives at CAS/MUET One CAS/MUET administrator was not sure whether any tangible outputs had emerged so far. One faculty member mentioned training programs for SIDA as an example of how linkages were being developed. She also mentioned a field visit to a site with anticipated environmental impacts, and the CAS/MUET team’s meeting with local community members and NGOs regarding locally acceptable mitigation measures. She also mentioned MOUs with the World Wide Fund for Nature-Pakistan and Oxfam. It is apparent from these observations that much potentially useful applied research, some of it quite innovative, is ongoing at CAS/MUET, but very little, if any, has been completed so far. Value Chain of Producing and Promoting Applied and Policy Research Participants in the validation workshop were asked to provide definitions of applicable and applied research. The CAS/MUET faculty did not limit the definition of applied research to topics or subjects that had arisen as a result of a demand from sector or user organizations or to the topics or subjects that had emerged from a process of discussions with sector partners. The CAS-/MUET faculty members wrote down individual points in response to the challenge question: What are the weak points in the value chain of producing and promoting applied and policy research that meets the water sector’s needs? Their responses were collated and displayed for discussion by the group. Applied Research The following recurring themes emerged from the moderated discussion among the participants: • Lack of confidence of sector organizations in academic research and little input from industry; • Limited funds and limited faculty and student capacities; • Understanding market needs and developing a marketing strategy; and • Reaching out to, and strengthening linkages with, industry. The following were noted as minority opinions: • Continued focus on fundamental research; • Continued dependency on Utah; and • Lack of coordination within CAS/MUET on its research program. Policy Research The following issues emerged as important from the moderated discussion among the participants: • Lack of a multidisciplinary approach; U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 104 • Retaining and hiring good quality faculty; • Poor understanding of policy research; • Less faculty for social science research; • Existing research groups should be more independent and empowered; • Need for data and analysis of policy issues related to the water sector; • Fewer linkages with the public sector and very little outreach to legislators; • No institutional coordination; and • Hard to transform applied research into implementable policy recommendations. The following were noted as minority opinions: • [Non] selection of good quality students; • Lack of relationship with policymakers; • Lack of coordination between industry and academia; and • [Poor] quality of policy research. Lab Utilization This section summarizes the results of the survey of faculty, alumni, and students regarding laboratory utilization and effectiveness. It notes the salient differences in lab use between CAS/MUET and the other CASs. Further, it supplements these quantitative results with qualitative information obtained through interviews with key stakeholders, including selected CAS/MUET faculty and laboratory technicians, Utah faculty members who guided and supervised lab and equipment use, and groups of CAS/MUET students with some experience in the use of CAS/MUET labs. This summary also takes into account the opinions of sector partners, CAS administrators, and university leadership on the utility of the labs. Questions relating to gender and lab use are addressed here as well as (more fully) in a separate section on gender mainstreaming (see “Gender” below). Finally, this section presents inferences drawn from this review of the evidence. Quantitative Results from Survey on Lab Utilization A total of 41 CAS faculty members (17 from CAS/MUET), 88 alumni (20 from CAS/MUET), and 542 students (125 from CAS/MUET) participated in the survey. The faculty responses to eight closed-ended questions (Q24 to Q31) and alumni and student responses to five questions (Q23 to Q27 and Q26 to Q30, respectively) are the basis for the analysis in the following subsections. CAS/MUET Faculty Sixteen of 17 faculty members responded to the question on the operational status of the labs at CAS/MUET, and all of them stated that the labs were indeed in operational condition. One hundred percent of the faculty respondents at NUST also reported that their labs were operational, but only 79 percent of those at UETP did so. Fifteen of 16 CAS/MUET faculty members stated that the labs were accessible or easily accessible and that they used the labs frequently or very frequently. Fifty percent reported that CAS/MUET labs were easily accessible, while at UETP, 82 percent of faculty members gave this response, making UETP the leader in this category. Still, CAS/MUET remained the leader in the very frequent use category, with 25 percent of CAS/MUET faculty reporting that they used the labs very frequently, compared to 20 percent at NUST and 9 percent at UETP. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 105 All 16 CAS/MUET faculty members responded that they used the labs for research, while 13 of 16 (81 percent) responded that they used the labs for teaching purposes. These responses compare favorably with the 92 percent and 76 percent who responded that they used the labs for research and teaching, respectively, across all three CASs. Only one faculty member (6 percent) at CAS/MUET responded that the labs were not sufficiently equipped to meet teaching needs, compared to 30 percent and 27 percent at NUST and UETP. Twelve of 16 CAS/MUET faculty members (75 percent) responded that the CAS labs had significantly improved the quality of their applied research. This compares favorably with the 20 percent and 36 percent who gave this response at NUST and UETP, respectively. Furthermore, 8 of 16 CAS/MUET faculty members (50 percent) were extremely satisfied with the adequacy of lab equipment for conducting applied research, compared to 10 percent and 9 percent at NUST and UETP, respectively. Alumni and Students Ten of 19 alumni (53 percent) reported that the labs were operational during their studies at CAS/MUET, while 101 of 116 current students (87 percent) reported that the labs were operational, compared to 43 percent and 81 percent for alumni and current students across the three CASs. Nine of 10 CAS/MUET alumni (90 percent) reported that the labs were accessible or easily accessible, as did 87 of 101 current students (86 percent). Furthermore, 7 of 10 alumni (70 percent) said they used the labs frequently or very frequently, as did 68 of 101 students (67 percent). It appears that the proportion of students who have frequent access to labs has reached a ceiling, perhaps owing to the increase in student enrollment. The frequent or very frequent use of labs reported by alumni across the three CASs was 76 percent, while by current students it was 65 percent, indicating that CAS/MUET has improved its position in relative terms over time. All 10 CAS/MUET alumni responded that lab use had somewhat improved or significantly improved the quality of their applied research skills, while 93 of 101 current students (92 percent) responded that lab use had somewhat or significantly improved the quality of their applied research skills. This compares favorably with the 82 percent of alumni and students across the three CASs reporting somewhat or significant improvements in their applied research skills. Finally, all 10 CAS/MUET alumni were satisfied or extremely satisfied with the adequacy of lab equipment for conducting applied research, while 98 of 101 students (97 percent) were satisfied or extremely satisfied with the adequacy of lab equipment for conducting applied research. This compares favorably with the 82 percent of alumni and 89 percent of students across the three CASs reporting satisfaction or extreme satisfaction with the adequacy of lab equipment for conducting applied research. Qualitative Responses from Interviews on Lab Utilization This section explains the preceding quantitative results by identifying supportive and corroborative—as well as contradictory and moderating—statements (including direct quotations) about laboratory utilization by the concerned stakeholders. It provides qualitative evidence to clarify the survey results and illuminate their implications. This section also summarizes additional information on the responses to questions regarding women’s access to and comfort in the use of labs. The section examines lab utilization at CAS/MUET from six different points of view: 1. How CAS/MUET faculty members perceive their own lab work and the challenges and opportunities ahead; 2. The opinions of the laboratory technicians; 3. How Utah faculty members assess the lab work of CAS/MUET faculty, technicians, and students, and the challenges and opportunities for CAS/MUET; U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 106 4. What the students think of the lab work being undertaken at CAS/MUET; 7 5. The external perspective of sector partners on lab utilization; and 6. The views of the CAS/MUET administration and university leadership. Perspectives of CAS/MUET Faculty One faculty member reported that all the labs, except the hydraulic lab, were complete and functional, all the equipment was being used, and students were doing research work there. The soil lab was ready to take work for outside agencies against consultancies. Respondents reported issues related to operationalizing the labs; for example, one had to buy arsenic test kits twice himself. Women reported no problems in accessing the labs and reported they were comfortable in them. One female faculty member said that a year had passed since labs were established, but that the process of operationalizing them was “very, very slow”, in part because the vendors were not providing training at the right time. She also reported that, in the past, there had been one or two cases of junior staff not behaving appropriately when women were working late in the labs. As a gender focal person, she read them the university rules on the subject of behavior. Views of CAS/MUET Laboratory Technicians One female lab supervisor said that the water quality lab was being run effectively. Most faculty members used the lab frequently, while a few busy professors asked for her help in running their programs. Ph.D. and M.S. students, who had been trained in lab procedures, were researching a range of topics. Some had also been trained in the use of portable equipment for the field. Students from other MUET departments also came to the lab for internships. The lab was being used to test the water and wastewater samples on a commercial basis. Sustainability plans for the next 10 years included steps to maintain the expensive specialized equipment through commercialization. A website with lab policies and SOPs, such as safety procedures, and a rate list of services was available. A research focus was water quality in remote areas with many waterborne diseases. Samples from the Thar Desert were being tested free of charge as part of a social responsibility effort. Research was underway on cost-effective ways to repair the membranes of faulty reverse osmosis plants in the desert. Women, including an engineer, a biologist, and a lab technician, reported being comfortable in the lab. One male lab technician said that his lab had many projects, such as those on sea water intrusion and lake water quality. He reported that students were making good use of the lab, with frequency depending on their research topics and methodology. The permission procedure was straightforward. Faculty members were required to fill out forms for a project or for student work, specifying equipment and consumables needed. Permission was given quite readily, but the procedure was necessary as the lab needed to keep a record of chemicals used and left over. He added that the training of lab technicians was a priority for sustainability. Furthermore, it was important that lab technicians were taken to see the proposed lab equipment before it was procured to enable them to understand how exactly it would be used, and what were key accessories. Assessments by Utah Faculty One Utah faculty member said that it had taken until 2017 to get the labs established, but they were being used very effectively at the time of his reporting, with much joint research being conducted. Resources had been set up to teach environmental lab procedures. Procedures for soil and water testing were being used in coursework. Students were able to make good use of the labs, though there were 7 Two alumni groups interviewed at CAS/MUET said that the labs were not in place during their time as students and they had little direct experience to contribute to the discussion. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 107 challenges of capacity, with more than 150 students looking to accomplish various research activities. He said the facilities were being used to the maximum and there was a need for additional ones. Another Utah faculty member said the labs were being used by faculty members for their own research and for teaching students. He would like to see them used as a service to partners. Utah had developed a course to enable CAS/MUET to do so. There were, at the time of his reporting, no overt challenges to the use of the labs by women during the daytime, but a situation could arise when the labs started to be used on a 24-hour basis. Perspectives of Students One member of a student group said that all required instruments and software were available. Another said that it depended on the topics taken. A female student said she did not know how to operate instruments owing to her non-science background. She needed to learn how to use the lab equipment, but CAS/MUET did not include practical sessions in the M.S. coursework. Two members of another student group said they had been provided with written parameters and clear instructions on lab use. Another said that the geographic information system (GIS) lab was helpful in her coursework. A fourth student said that computer lab work was taught nicely, but not the lab work in biology and chemistry. The first student said no lab was available when he needed it during coursework and that procurement was the main challenge, followed by inefficiency in installing equipment. The third student suggested that the center should install upgraded software on more machines as the software she needed had not been upgraded. Perspectives of Partner Sector Organizations One sector partner mentioned that he had requested the project director of CAS/MUET to undertake soil and water tests for a farm near Jamshoro. CAS/MUET had responded cooperatively, and so its labs were accessible. He expected a gauge calibration service to be in place by December 2018, which would be the test of CAS/MUET efficiency. Another sector partner said his enterprise and the fisheries export sector required specific tests for salmonella and E. coli, and so on. He wondered whether the CAS/MUET labs planned to get the required accreditation. Point of View of CAS/MUET Administration and University Leadership One administrator said that the sustainability of the center depended on the quality of services provided, including water quality analysis, water management services, and so on. CAS/MUET labs conducted water analysis and soil analysis and were trying to develop SOPs, which were almost ready, for commercial projects. CAS/MUET had organized four or five GIS trainings for other stakeholders. The first challenge was to establish a reputation for quality results. A university leader said, “There is a need for more lab technicians and engineers at CAS; they do not have a proper lab custodian. A simple trip to the USA is effective. Cost is not an issue. I think it is worth it.” Summary Interpretation The survey revealed the generally superior performance of the CAS/MUET labs over the other CAS labs on key indicators. The interviews also identified tensions between the users and their different objectives for the labs, for example, between their educative function and their potential for commercial use and contribution to sustainability. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 108 Graduate Degree Programs This section on graduate degree programs summarizes the responses of faculty, alumni, and students to survey questions on curriculum development at CAS/MUET, noting any differences from the responses at the other project CASs. It also assesses the quantitative results using qualitative information obtained through interviews with key stakeholders, including selected CAS/MUET faculty, Utah faculty who guided and supervised the curriculum reform and development process, and several groups of alumni and students. Further, this section reports the responses of sector partners to queries on their engagement in the process of curricular reforms as well as the utility and relevance of the revised curricula; it reproduces the narratives of CAS administrators and university leadership; and it presents inferences drawn from this review of the evidence on curriculum development at CAS/MUET. Quantitative Results from Survey on Curriculum Development A total of 41 CAS faculty members (17 from CAS/MUET), 88 alumni (20 from CAS/MUET), and 542 students (125 from CAS/MUET) participated in the survey. The responses to four closed-ended questions from faculty (Q33 to Q36), alumni (Q29 to 32), and students (Q32 to Q35) are the basis for the analysis in the following subsections. CAS/MUET Faculty Sixteen of 17 faculty members at CAS/MUET responded to the questions on curriculum reforms. Fourteen of 16 faculty members (88 percent) responded positively to the question of whether CAS/MUET had a mechanism in place for periodic curriculum reforms. This is compared with 100 percent at NUST and 93 percent at UETP. Fourteen of 16 faculty members (88 percent) responded positively to the question on whether CAS/MUET curricula were aligned with the needs of industry. This result was consistent with an average across the three CASs. Fourteen of 16 faculty members (88 percent) agreed or strongly agreed with the statement that CAS/MUET curricula adequately balanced theory and practical application. This result was also consistent with the average across the three CASs. Fourteen of 16 faculty members (88 percent) agreed or strongly agreed with the statement that CAS/MUET teaching methodologies adequately balanced theory and practical applications. Two faculty members disagreed with the statement, thus the level of dissent was higher than at the other two CASs (one differing faculty member at one and none at the other). Alumni and Students Twelve of 19 alumni (63 percent) responded that there was a mechanism in place at CAS/MUET for periodic curricular reforms, while 6 said they did not know, and 1 stated that there was not. This compares favorably with the responses from the other CASs, with 45 percent of alumni at NUST and 50 percent at UETP stating that their respective CASs had such mechanisms in place. Fifty-five of 116 current students (47 percent) responded that there was a mechanism in place for periodic curricular reforms, while 47 responded that they did not know, and 14 stated that there was no such mechanism in place. Current student support for the assertion was equal to the 46 percent from NUST and better than the 36 percent from UETP. Twelve of 19 alumni (63 percent) responded that CAS/MUET curricula were aligned with the needs of sector organizations, while 4 did not know, and 3 disagreed. The proportion of positive responses was U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 109 higher than the average across the CASs (49 percent), while the proportion of dissenting responses was lower (16 percent compared to 41 percent). Seventy-five of 116 current students (65 percent) responded that CAS/MUET curricula were aligned with the needs of sector organizations, while 26 did not know, and 15 disagreed. The responses of the current students at CAS/MUET were roughly the same as the responses of students across the three CASs. Thirteen of 19 alumni (68 percent) agreed or strongly agreed that CAS/MUET curricula were adequately balanced between theory and practical application. This was a more favorable result than the 52 percent average across the three CASs. Eighty-four of 116 current students (72 percent) agreed or strongly agreed that CAS/MUET curricula were adequately balanced between theory and practical application. This was slightly more favorable than the 69 percent across the three CASs. The result in the subcategory of strong agreement was less favorable for CAS/MUET (10 percent) than the average across the three CASs (16 percent). Twelve of 19 alumni (63 percent) agreed or strongly agreed that faculty teaching methods were adequately balanced between theory and practical application. This was more favorable than the 56 percent across the three CASs. A lower proportion of CAS/MUET alumni strongly agreed with the proposition (11 percent compared to 16 percent across the CASs as a whole), but a smaller share also strongly disagreed with it (5 percent compared to 15 percent across the CASs as a whole). The response patterns of current students were similar to those of the alumni on the question of adequate balance between theory and practical application in teaching methods. Seventy-nine of 116 current students (68 percent) agreed or strongly agreed with the proposition, compared to 65 percent for current students across the three CASs. In line with the alumni responses, a lower proportion of CAS/MUET students strongly agreed with the proposition (12 percent compared to 16 percent across the CASs), and a smaller share strongly disagreed (6 percent compared to 11 percent across the CASs). Qualitative Responses from Interviews on Curriculum Development This section elucidates the quantitative results reported above by identifying supportive and corroborative—as well as contradictory and moderating—statements (including direct quotations) on curriculum development. Breaking down the results by category of concerned stakeholder, it seeks to clarify the survey results and characterize their implications. Perspectives of CAS/MUET Faculty Several CAS/MUET faculty members said that they, along with departmental experts and faculty from other universities, were involved in the curriculum development process. One gave an example of how the curriculum had been made relevant to sector needs. He said a subject had been added to the syllabus in response to a question in the civil services examination. Another mentioned his role in the design of a three-course curriculum from scratch. A number of faculty members mentioned mentoring from the University Utah in an interactive process of curriculum development. Most respondents appreciated the critical evaluation of Utah mentors and their help via weekly Skype calls for syllabus revision. One staff member mentioned in particular how the interaction helped in enhancing teaching skills by introducing the concept of learning by doing. However, at least two faculty members complained about the lack of constructive feedback. In one case, the advice was limited to the recommendation to add two more chapter heads and resolve some formatting issues. A second respondent complained that no guidelines or standards were provided by the University of Utah and that alignment and experimentation in the curriculum were ongoing. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 110 One CAS/MUET faculty member felt that some of the challenges arose from the limited skills of the faculty itself. The issue could be addressed, he felt, by establishing teaching partnerships. Reflection and Assessment by Utah Faculty Some members of the Utah faculty reported that they were not satisfied with the baseline quality of CAS/MUET students. They explained that students with master’s degrees in the U.S. were very good at research, and they gained experience and knowledge for entry into professional practice through undergraduate programs. They reported that unfortunately this was not the case in Pakistan, where students had minimal internship opportunities. Utah faculty were trying to address this transition through an internship program at the graduate level, which was not something they had expected to do. The Utah faculty members reported that the revised curricula were good to very good from a technical standpoint but only fair from a professional development standpoint. From a communication and writing standpoint, several Utah faculty members felt that the students were emerging with the best teaching and writing abilities in Pakistan. Utah supported the CAS/MUET faculty at multiple points in the project in terms of developing curricula collaboratively from scratch. Utah emphasized entrepreneurship and set CAS/MUET on a course for technology commercialization. The degree and courses such as WASH introduced by Utah had opened up opportunities in the development sector. One respondent explained that water management is a complicated issue in Pakistan owing to governance and policy issues, and Utah tried to develop curricula related to the problems, but the curricula were not relevant to Pakistan’s water issues. Some felt that CAS/MUET graduates were better prepared than the other graduates of MUET because they were being pushed, but they were still not at the level of U.S. students, and further pushing and development were needed. Views of Alumni and Students All four members of an alumni group reported they were happy with the curriculum. One stated that it had helped her secure a job. She was especially happy with the practical work during her research as it provided the skills she needed at her job. The second participant stated that the revised curriculum incorporated new techniques, such as modeling and GIS applications, that enabled CAS/MUET graduates to compete even with U.S. university graduates. The third alumnus felt that the new curriculum was useful because of the introduction of subjects such as professional development. The fourth member said that the new curriculum was more advanced and that the teaching methodologies were better and more oriented to problem-solving, as well as more relevant to his current job. In another alumni group, one respondent said that the curriculum had helped her get a job and was relevant to her work. However, another said that the subjects were general and broad in scope and that the full benefits of the curriculum reform were not obtained as the faculty was not skilled in curriculum design. He also said that no lab work was incorporated in the curriculum. In a third alumni group, two members said that government departments were quite indifferent to the new CAS/MUET curriculum. The respondents reported they had gotten government jobs on the basis of their B.S. degrees. Several students said that the curriculum and teaching methods were effective. They were asked to make presentations, which were learning opportunities that increased their confidence. However, some students felt that the content and teaching were not satisfactory. In particular, one said that the faculty was not competent to teach the new WASH course. One group mentioned that an evaluation conducted at the end of their first semester had recommended improvements in coursework and teaching. However, the changes had yet to be implemented, even as the second semester was about to end. Another mentioned the exclusion of numerical exercises and fieldwork in response to the demand of students for less demanding courses. “Now everything taught is available on Google,” said one U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 111 student. Others mentioned the repetition of particular topics in the coursework on integrated water resources management (IWRM). All four respondents in one student group complained that they had been provided with the same notes as the previous batches, and no new methodology had been adopted. Students in another group asserted that there were no teachers to supervise policy, law, and community research and that visiting teachers did not possess the requisite qualifications. Their recommendations included more optional Ph.D. courses to increase relevance for diverse employers’ needs, the inclusion of sustainable development modeling in IWRM, and more practical work on the water, energy, and food security nexus. In a contrary opinion, some students expressed the need to focus on water issues. One group felt that it was not possible to question the CAS/MUET authorities on the curriculum. For example, in response to a recommendation for a course on water governance, the project director reportedly responded by querying the need to hire new faculty to teach one or two students. Observations of Partner Sector Organizations None of the five partner organizations reported formal engagement in the process of curriculum development, though two respondents had seen the curriculum in their individual capacities. All indicated an interest in making suggestions for curricular reforms. One sector partner said, “We have not seen the CAS curricula, but the topics they study in the WASH program are quite relevant to our field of sanitation. There is no other institution which is providing a master’s degree in WASH. If we need people who are qualified in this sector, we will find them only in CAS.” Another sector partner said his organization had not been formally invited to the seminars on curricula. He had attended in his individual capacity and provided some ideas but had not checked to see if any had been incorporated. A new sector partner said he did not get the opportunity because his relationship with CAS/MUET was recent. He had not seen the curricula but would be glad to give his input as there was always room for improvement. Opinions of CAS/MUET Administration and University Leadership A CAS/MUET administrator stated that CAS/MUET did not consult the local stakeholders, communities, or industry at the start of the curriculum reform process. The suggestions mostly came from U.S. partner universities. The University of Utah asked CAS/MUET to forget about the previous syllabus and recommended a new format focusing on water sector needs, governance, and sustainability. A workshop was then held to get feedback from Sindh stakeholders on the format, their suggestions were incorporated, and the curriculum was revised. The review of the curriculum was an ongoing process with feedback from the students, while a marketing research firm was developing plans to market CAS. Utah also provided continuous training assistance. The HEC representative praised the Utah team leader for the curricular reform process. He said that the process of facilitating the development, design, maintenance, and assessment of a modern curriculum at Mehran was a model that should be imitated by the Pakistan Science Academy. Summary Interpretation Curriculum development at CAS/MUET has been appreciated from several perspectives. It was praised by most stakeholders, ranging from national education experts at HEC to first and second-semester students at CAS/MUET. However, Pakistan’s water sector issues are complex and evolving, as well as crucial in the immediate, medium, and long term. More engagement with sector organizations is called for to continuously upgrade the curricula for relevance. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 112 Gender Increased Female Enrollment Rates CAS/MUET has the highest percentage of women among current students (36 percent) compared to CAS/NUST (18 percent) and CAS/UETP (11 percent) (Table 8). Interviews with CAS administrators, faculty, and students have shown some possible reasons for this increased representation of female students in recent admissions. These include free accommodation to female students (8 responses) and free transport (5 responses). TABLE 8: FEMALE FACULTY AND STUDENTS AT CAS INSTITUTIONS CAS Institution Female Students Alumnae Female Faculty8 CAS/NUST 18% (41/230) 16% (16/99) 17% (3/18) CAS/UETP 11% (34/309) 14% (1/7) 0% (0/12) CAS/MUET 36% (57/159) 31% (11/35) 27% (6/22) Total 19% (132/698) 20% (28/141) 17% (1/6) Source: CAS provided data during the midterm evaluation. A faculty member, who is also a member of the gender and admission committee at CAS/MUET, shared that “The percentage of girls’ recruitment has improved. It was achieved by going to other universities to recruit girls. There is no quota for the exchange program. Students were recruited from Tharparkar district and Balochistan province, but without compromise on quality criteria. Two teams went to the disadvantaged areas to recruit students on merit.” Barriers to Employment Faced by Women In the faculty survey, respondents’ opinions were equally divided between agreement and disagreement with the statement: “Barriers to employment faced by women are greater.” Percentage responses are given in Figure 10. FIGURE 10: CAS/MUET FACULTY OPINIONS ON WHETHER WOMEN FACE GREATER BARRIERS THAN MEN TO EMPLOYMENT Source: PERFORM survey of CAS/MUET faculty (n=16). The student survey found that 42 percent of the respondents at CAS/MUET (n=116) disagreed with the statement that women faced greater barriers to employment than their male colleagues in water sector 8 Faculty includes all faculty members with full-time, visiting, or adjunct status. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 113 organizations, and approximately 58 percent agreed or strongly agreed with the statement. Gender disaggregation of the responses reveals that a significantly higher proportion of women (71 percent) than men (49 percent) agreed or strongly agreed with the statement. This difference shows the importance of the work needed to create greater acceptability of women as potential employees in the water sector. However, it is important to analyze the water sector organizations’ perspective as well. In the interviews with sector organizations, three of four respondents clearly identified improved gender equity as one of the important U.S.-PCAS project components. Similarly, respondents from two of the four organizations also shared in interviews that they will support sustaining a gender-based component at CAS/MUET (Table 9). TABLE 9: WATER SECTOR ORGANIZATIONS’ RESPONSES ON U.S.-PCAS PROJECT COMPONENTS Sector Organization Most Important Components Components Most Likely to Support Organization 1 Gender, Curriculum Internship, Gender Organization 2 Research, Gender, Lab Internship, Research, Lab Organization 3 Teaching, Research, Exchange Internship, Lab, Research Organization 4 Research, Gender, Lab Research, Internships, Gender Source: PERFORM interviews with water sector organizations. Some of the organizations’ responses are provided below: “SIDA is already trying to create space for women, in technical and non-technical areas. The coming women graduates will find more space in SIDA, but CAS-W needs to do more lobbying with industry, and with the Public Service Commission, to set quotas for women engineers and water managers, as it does for doctors, nurses. [The] Irrigation Department has 5 women among 32,000 staff, while SIDA has 15 percent women.” – Sector organization respondent “… [W]e work very closely with communities, so we will want that CAS will work on gender equity. I am hopeful that the management of CAS will address this to link their product with the end user. Because we will take CAS graduates not only based on their product but also on the basis of our organization’s need. Hence gender equity will be valued.” – NGO respondent “We value gender mainstreaming as a second most valued component of the project after applied research.” – Public sector organization respondent “Two [young women] came to us for internships, and they knew about their subject, which tells about the quality of CAS-W graduates.” – Sector organization respondent Decision-Making Roles CAS/MUET does not have a woman in any organization-leading or decision-making position. This was confirmed during interviews with CAS/MUET administrators as well as the partner university’s staff. A University of Utah representative shared, “No, I am not aware of any women in decision-making roles; there should be. I think there is one woman who has a Ph.D. and she does have some influence there. She is very active in research. You know we also have the same problem here in the United States in terms of women.” Other respondents reported that the center has developed a gender equity policy, which has been acknowledged and adopted by MUET, and that along with the Women Resource Center, it has helped in creating the outlook of the center as women-friendly. This view is evident in the participation rates of women in the student enrollment. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 114 “Now MUET has a gender policy because of CAS-W. A researcher has written a paper on why women are not in the engineering profession. The reasons include fieldwork, including fieldwork at night. There are often no separate washrooms for women in the offices of government engineering departments. CAS-W has set up a good model for selecting women students and providing facilities for their convenience.” – Sector organization respondent “We have been focused on providing an adequate pool of students to achieve equity when we can. We do not have to run a quota program because we have a very good fraction (35 percent) of female students right now in the program.” – Utah respondent “I do not think there is any mechanism for females in less traditional career roles, but in [the] WASH department females are preferred, but it is [the] sector’s demand in WASH.” – CAS/MUET faculty member “The center is setting an example for other universities in Pakistan in terms of gender equity as females are given free accommodation along with the establishment of a Women Resource Center, which conducts many activities.” – CAS/MUET student However, CAS/MUET is no different from other CAS institutions with respect to ensuring women in the decision-making roles. Multiple respondents highlighted that there are women who are part of the administration, but they are in junior positions and hence do not influence key decisions. Similarly, there are female faculty members in different streams as assistant professors; however, there is no female department head. Issues Regarding Women’s Taking a Lead Research Role At CAS/MUET, 90.7 percent of students agreed that women have opportunities to take a lead role in conducting research in the water sector (n=118). Similarly, 94.7 percent of alumni agreed that opportunities are available for women to take a lead role (n=19). Figure 11 below shows the response distribution among faculty members regarding the question of sector organizations’ preferring men to women in lead research roles (n=17). Thirty percent of faculty members agreed that sector organizations prefer men in lead roles. FIGURE 11: CAS/MUET FACULTY PERCEPTIONS OF SECTOR ORGANIZATIONS’ PREFERENCE FOR MEN IN LEAD RESEARCH ROLES Source: PERFORM survey of CAS/MUET faculty (n=17). 24% 47% 24% 6% 0% 20% 40% 60% Strongly disagree Disagree Agree Strongly agree Percentage of respondents U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 115 The sector organization representatives were asked the same question in the key informant interviews, and three out of four shared that they do not consider gender in executing their research work; rather, they consider only industrial needs and required results. One of the sector organization respondents shared a belief that women are more suited to WASH-related work as they are better at quality control than their male colleagues. CAS/NUST Effectiveness of Exchange Programs The exchange program, which provides opportunities for faculty and students to visit partner universities in the U.S., is a central feature of the U.S.-PCAS project. Students and faculty members from NUST have made U.S. exchange visits arranged by U.S. partner ASU. The objective of the student exchange program is to provide students with the opportunity for research and training, expose them to modern techniques in research, and equip them with the necessary knowledge and skills to conduct research. This section explores the effectiveness of the exchange program for both students and faculty at CAS/NUST. Perspective of CAS/NUST Faculty Ten faculty members responded to the online survey. All NUST faculty respondents had Ph.D.s; one earned a Ph.D. in Pakistan, and all others received their degrees from North American, European, and Asian universities. Of 10 respondents to the online survey, 5 applied for the exchange program, and 3 were selected for it. Quantitative data show that 67 percent of CAS/NUST faculty believed that the exchange program did not help in their professional development, and they did not notice any change in their research skills. Only 33 percent of NUST faculty reported that the exchange program improved their applied research skills and outputs. Moreover, 67 percent of faculty believed that the exchange program did not help in improving their proposal writing skills. However, quantitative data show that 100 percent of CAS/NUST faculty believed that there was an improvement in their teaching methodologies because of the exchange program. The quantitative data also provide evidence of improvement in teaching methodologies of NUST faculty. Four of five NUST faculty respondents found the selection process for the exchange program to be transparent. The evaluation team conducted five faculty interviews at CAS/NUST, and these provided evidence that the exchange program has not been highly beneficial in building the capacity of CAS/NUST faculty. The duration of the exchange program for faculty was initially 6 weeks, and later it was increased to 16 weeks; these different durations yielded different results. Interview data validated this finding, highlighting that the short exchange program did not help in improving research skills or teaching methodologies. Nevertheless, all faculty members who participated in the online survey believed that the exchange program improved their teaching skills. When asked about learning and improving teaching methodologies through this exchange program, one of the NUST faculty members responded, “The program was not about skill development or teaching methodologies, but regarding introduction with various researchers.” However, one of the faculty members found it very helpful and said, “I have learned new teaching methodologies and use of labs. During the exchange program, I observed the teaching methodologies of different teachers in the U.S. and learned a lot from them.” One NUST faculty member said, “My experience of the exchange program was good for developing linkages; however, it did not help in enhancing my skills regarding research.” Another NUST faculty member said, “[The] exchange program was good; however, it would have been better if the duration of the program would have been more as in this short time I could only meet scientists and build relationships; however, [I] could not do any actual work.” U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 116 Another member said, “[D]ata also shows that faculty who spent four to six weeks did not improve at all; however, they did notice a few changes in faculty who spent four months at ASU. They did notice that their research ideas, research techniques, and teaching methodologies … improved because of the exchange program.” One of the faculty respondents said, “As a teacher, I have learned new teaching methodologies and use of labs. Also, I have learned data presentation from there as data representation had never been the focus in our academia. We had a chance to use their design, and we later installed those sensors in our labs as well.” One of the faculty members who shared his views on the effectiveness of the exchange program said, “I have also observed that my colleagues who came back from the exchange program now … have better research ideas. They are clear about the usage of lab equipment. They are now familiar with more research techniques.” About preparing faculty at CAS before departing for the exchange program, 33.3 percent of NUST faculty were not satisfied with the predeparture orientation arrangements. Perspective of CAS/NUST Current Students A total of 179 students from CAS/NUST participated in the online survey; 167 were enrolled in the master’s deree program and 12 in a Ph.D. program. Of these, 96 students applied for the exchange program, 79 students did not, and 4 did not respond. More than one-third (36 percent) of respondents believed that the selection process for the exchange program was transparent, while 44 percent did not know much about the selection process, 15 percent did not find the selection process transparent, and 5 percent did not answer. Twenty-seven (28 percent) of the student respondents were selected for the exchange program, 68 (72 percent) were not, and 1 did not respond to the question. FIGURE 12: PERCEPTION OF CAS/NUST CURRENT STUDENTS OF IMPACT OF EXCHANGE PROGRAM Source: PERFORM survey of CAS/NUST students. A survey of CAS/NUST students showed that 86 percent of respondents found the exchange program helpful in improving their applied research skills, while only 14 percent did not (Figure 12). The survey also reflected that 76 percent of students found the exchange program helpful in improving their research output. Twenty-one students responded to the question on improvement in proposal writing skills; 14 students (67 percent) found the exchange program helpful in improving their proposal writing skills, and 33 percent did not. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 117 Alumni showed a similar trend in skill acquisition and research output as a result of the exchange program. Of the 23 alumni surveyed, 91 percent reported an impact on applied research skills, 83 percent reported an impact on research output, and 56 percent reported an impact on proposal writing skills. Of the 27 NUST students selected for the exchange program, 6 did not participate. Of the 21 participants, 24 percent did not agree that CAS prepared them well before the start of the exchange program; however, 86 percent believed that ASU adequately supported them in adjusting to the new environment during the exchange program. Compared with alumni satisfaction rates, student satisfaction shows a noticeable improvement in the preparation of students for the exchange program. Of 23 alumni, 35 percent disagreed with the statement that they were well prepared by CAS, compared to 24 percent of students (Figure 13). Analysis of qualitative data from the three group interviews conducted with current students of CAS/NUST show that the exchange program was helpful for improving students’ applied research skills, utilization of lab equipment, problem-solving, and communication. Data from the interviews show that students had positive experiences during the exchange program. One student said of his exchange program experiences: “I stayed for two and [a] half months. Those were [the] best time in my life … [I] experienced new things and research. It was a great feeling for me to know what actual research is. In Pakistan, you lack in research and have no idea; foreign faculty in Pakistan do not bother with such things, [not even] at NUST. It was [a] great opportunity to engage directly with those faculty, to know how research [is] going on, how to achieve the objectives, and if we cannot achieve objectives, how to present the work in a positive way.” Students also found the exchange program helpful in improving their problem-solving and research skills. Qualitative data also reflect that the exchange program helped the participants in writing more research papers and developing more research ideas. Students also found that professors at ASU were very welcoming, which was helpful. However, they reported they did not experience such support from teachers at NUST. One student reported: “During my stay of two and [a] half months there, I got published two papers. It’s one and [a] half years [later, and] I am in [the] research phase; here I am unable to write papers or to conduct good research because of deficiencies. There is no coordination between different schools at NUST. If I have [a] problem in modeling or mathematical things, I go to other schools to consult and they do not [receive] me as their own student. In [the] U.S. any professor would welcome us despite busy schedules, and it gave us motivation.” Students also expressed that easy access any time to labs also helped in their research and that they could not use labs freely and had limited access to labs at NUST. The exchange program also helped in enhancing students’ exposure to cultural diversity and entrepreneurship skills. One of the students said, FIGURE 13: CAS/NUST STUDENTS’ PREPARATION FOR EXCHANGE PROGRAM U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 118 “The cultural diversity and exposure exchange students get is [reflected] in them when they come back. The conference we talked about by Kenneth Wellington on entrepreneurship [and] the volunteers who were helping in arranging the conference were our own seniors from [the] exchange program from ASU, [who] showed good skills in entrepreneurship.” The qualitative data also show that some of the students did not have good experiences with the exchange program and believed that it did not help them. A few students raised concerns over planning of the exchange program in terms of allocating research labs and research topics. One said, “When students were in Pakistan, they had a plan for working on some research ideas; however, at ASU they had to opt for other research topics.” Students suggested that enough time should be given to interact with the research supervisor at ASU to finalize the research plan and research needs ahead of time. Participation Rates of Students in the Exchange Program Compared to other CASs, CAS/NUST has a low acceptance rate for the exchange program. Based on survey responses of 179 current students, 96 students applied and only 28 percent of the applicants were accepted for the exchange program. This is substantially lower than the acceptance rates at the other two CASs: UETP’s acceptance rate is 50 percent. and MUET’s is 53 percent (Table 10). TABLE 10: PARTICIPATION RATES OF STUDENTS IN EXCHANGE PROGRAMS Exchange Experience NUST (n=179) UETP (n=231) MUET (n=125) Applied for exchange 96/175 = 55% 121/231 = 52% 70/122 = 57% Accepted to exchange 27/95 = 28% 60/121 = 50% 37/70 = 53% Participated in exchange 21/27 = 78% 46/61 = 75% 35/37 = 95% Evidence suggests that, unlike other CASs, NUST allows for students outside of the center to compete for the exchange program. Interviews with CAS students and alumni confirmed this finding. “The program was for CAS not for NUST, but NUST changed [the] policy and made the program open for all other departments. The funding was given only for CAS students, but they were also sending other departments’ students to exchange programs on our funding. Our batch was supposed to go on [an] exchange program when this policy was announced, and from our batch some students were dropped as a result of this policy. … Only 11 students from our department had gone out of total 20. Nine were from other departments and seven were from the School of Mechanical and Manufacturing Engineering.” – CAS/NUST alumnus This strategy was further discussed in validation and debriefing sessions, and different reasons were shared for opening the program to other students, including competition among students, creating goodwill toward CAS from other departments, extending opportunities to bring more quality students into the exchange, and research work. Partner University’s Perspective ASU is the host university for the NUST exchange program, and the evaluation team interviewed ASU representatives, discussed the effectiveness of the exchange program, and asked whether the ASU team noticed any changes in NUST faculty members once they returned to their university after the exchange. ASU respondents reported that they did not notice any changes. One ASU respondent also highlighted that ASU sees that the exchange program was not as effective as anticipated for faculty, but that it was useful for CAS/NUST students. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 119 NUST faculty, CAS administration, and ASU representatives explained that the major reason that the program was not as useful for faculty as it was for students was its short duration. Respondents felt that the program for faculty should be either discontinued or revised with detailed deliverables and outcomes. Faculty time spent away from Pakistan also affects students on the NUST campus, and the center has to arrange for substitute lecturers. The ASU faculty also reported that faculty in Pakistan could engage further with students to prepare them to make the most of the exchange program. They also suggested that NUST faculty could engage in continual, weekly follow-up with exchange students to monitor the progress of their plans with respect to deliverables and timelines. Some felt that more students could be accommodated in place of faculty, as most of the students did not have an opportunity to experience the international education facilities. They thought that it would also provide additional opportunities for other students to experience the highest standards of labs, research work, and learning opportunities at ASU. Conclusions Overall, the analysis of all the stakeholders’ combined perspectives leads to the conclusion that the students’ time spent at ASU needs more planning and more structure. The program may require transcripts with credit contributing toward a degree. Students need more time to discuss their research interests and needs and their plan for the exchange program. There needs to be more follow-up so that research labs identified in students’ plans are available and provided to the students as per their research plans. Required courses and lab work should have specific credits for each deliverable. Faculty at CAS/NUST should be engaged in monitoring the progress and tracking the timelines and deliverables of the exchange students. Joint Research, Applied Research, and Public-Private Partnerships Faculty Perspective on Research and Contributing Factors Overall, NUST faculty saw strong relevance of CAS applied research to the needs of sector organizations and were satisfied with the level of sector organizations’ participation. Of 10 NUST faculty members who participated in the survey, 9 said that they are involved in applied research at CAS. With respect to sectoral relevance, 44 percent of faculty strongly agreed that their research was relevant to the needs of energy sector organizations, and 56 percent agreed. Overall, 90 percent of faculty members agreed that applied research topics being covered at CAS are aligned with the explicit needs of the energy sector. The NUST faculty agreed that energy sector organizations are participating in and supporting the applied research being conducted at NUST. Eighty percent of the faculty respondents were satisfied with the level of participation of sector organizations in conducting applied research at CAS/NUST (Figure 14). Role of ASU in Research The faculty was also asked about their level of satisfaction with different research aspects in relation to ASU team members. A substantial percentage of NUST faculty respondents (40 percent) were not FIGURE 14: CAS/NUST FACULTY’S PERCEPTIONS OF SECTOR ORGANIZATIONS’ PARTICIPATION IN APPLIED RESEARCH U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 120 satisfied with the financial support provided by ASU for applied research projects. Moreover, 20 percent of NUST faculty respondents shared their dissatisfaction with the technical support they received from ASU for applied research projects. Technical support includes areas such as research design, analytical tools, publishing support, and guidance on use of labs and lab equipment. Interviews helped in understanding the reasons of this dissatisfaction. Mainly these were related to the limited physical presence of the ASU staff for technical guidance. As one NUST faculty member stated, “While conducting joint research, our U.S. partners hesitate to come to Pakistan, and we believe that with physical presence we can do even better research.” During interviews, NUST faculty members informed the evaluation team that there were five ongoing joint research projects. Most of the projects were in their initial stages, with some of the research started within the past few months. One of the challenges faced by NUST faculty concerns data collection, and one faculty member said, “In joint research, the major challenge being faced is regarding the availability of data, and data collection is very difficult. It is not easy to deal with government institutes. This research will help in public policymaking. In order to make our partnership with the Planning Commission formal and official, we drafted [an] MOU; however, due to some issues at the Planning Commission, it is not signed yet, but there is willingness from their end to cooperate with research.” CAS/NUST faculty were also asked to share their satisfaction with CAS-specific procedures and systems (internal arrangements) regarding autonomy in utilizing research grants. Half of NUST faculty respondents were dissatisfied or extremely dissatisfied with the present situation regarding autonomy in utilizing their research grants. Although not directly related to autonomy, a faculty member raised concerns about utilization: “It is to be mentioned here that we believe that funds for the joint research are not enough—USD$30,000 is not enough to conduct joint research. We discussed this issue internally, only we have not risen [it] in any committee or in front of any leader. However, funding for the joint [research is much] less. The funding is not enough to bear the cost of researchers’ [travel] and equipment procurement.” NUST faculty respondents also raised issues over the research’s relevance and its technical or engineering features. One faculty member said, “Here are currently five joint research projects with ASU and I find only one of them close to engineering. [The] rest of them are just like studies. I have [a] very different view of engineering. Where I studied engineering, it is related to designs and solutions; here the joint research projects I found are very different, and I don’t believe in this type of work, which is just merely related to studies. We need people from sciences and engineering as well, and our management does not even understand our scope of work because they are not engineers. I believe in providing solutions by designing machines, which is not happening in our joint research.” According to the evaluation team’s assessment, applied research projects at CAS/NUST were at early stages, and no project had yet been able to reach the product development and commercialization stage. Research grants received were mostly utilized in salary and other overhead, and the remaining amount was not enough to take the project to product development. Faculty reported that students were not highly involved in applied research and joint research projects at NUST. Sector Engagements CAS/NUST has initiated some collaboration with energy sector organizations in both the government and the private sector. CAS/NUST representatives have made efforts to engage organizations including Fauji Fertilizer Company Limited, Attock Generation Limited (AGL), Sky Electric, and a few startup organizations. Qualitative data show that energy sector organizations saw great value in collaborating with CAS/NUST as they thought the center would provide solutions to their needs; however, they felt that this collaboration had not been effective. One private sector organization respondent said: U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 121 “Based on my involvement, I can’t see any significant cooperation from CAS. We provided them with one system, and [the] initial intention was that some of their students would be evaluating some research on this system, but we did not see any publication from [the] center. Even if there was any, it was not shared with us. This partnership has not been [an] effective partnership. I expect a lot more from them, especially in terms of research work they need to contribute; they need to publish and they need to make their research available to the sector organizations; however, this is not the case.” Another sector organization respondent said: “I received [an invitation to the Energy] Expo [at the] Serena [Hotel in Islamabad], but I couldn’t go there this year. I mean there is a method that they invite annually; however, there is no mechanism to meet regularly so we could be involved and provide our feedback in applied research. I am also working with NUST directly with their school of electrical and computer engineering. I am also working with some of their professors working on a certain applied research with us, but no professor is working with us from this energy center.” Sector organizations saw value in collaborating with CAS/NUST to seek solutions to their problems. One sector organization respondent said, “We started collaborating with CAS/NUST because we had two options: one was to ask [an] international firm to provide us these services or work with an institution in Pakistan. CAS/NUST suited us in that we are in Rawalpindi and we find CAS/NUST more appropriate as we can easily come to NUST rather going for [an] international company.” Another respondent from a key energy sector organization said, “We once had [an] MOU with NUST in the past for exchange of students for internships; however, we do not have any active engagement with them now, and moreover, nothing has practically been done for industry collaboration by CAS.” Another sector organization respondent, who was satisfied with the applied research at CAS/NUST, reported on an ongoing project: “We are doing one applied research project with CAS/NUST, and so far we are satisfied with the quality of work. We are working on utilizing our industry biowaste and converting that waste into high-purity methane. We are working on [a] project which will generate high purity methane, which will provide high heating value. We have designed [a] digester by ourselves, and this has been designed on international standards. We started this project in 2014; however, we signed [an] MOU in 2016. Methane is [a] major raw material for [the] fertilizer industry, so [the] idea was to produce high-purity methane, as methane is depleting day by day. The size of [the] pilot digester is 1–1.5 cubic meters, and we will upscale it later. We are doing this project to keep in mind abundant biomass in Pakistan, and if this project gets successful, we can use potato waste and use generated gas in our boilers. We feel that so far our partnership with CAS/NUST is successful; however, results are just at lab scale and our management wants to see results at commercial scale.” Another two sector organization interviewees were not satisfied with the quality of applied research done by CAS/NUST. When asked about applied research at CAS, a power generation company representative responded: “Teachers are not aggressive over there, and they are not following the program proactively. They are just taking cover of industry that they are involving industry; however, it is not actually happening. I have always been highlighting in meetings with them that industry should have confidence in academia that they can solve industry problems. If I am providing [a] platform for them to learn practical skills at our plant, then it should be two-way communication that they can also provide solutions to our problems right in our country. [The] university should be able to solve our problems rather [than] sending [them] abroad. They have not approached me to U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 122 showcase that they have any expertise to solve our problems so we have confidence in their skills.” CAS/NUST has made some efforts in initiating a few joint research projects. However, energy sector organizations’ needs have not been taken into consideration, and there is no set mechanism available to involve sector organizations and receive their input before initiating a joint or applied research project. ASU is involved in technical vetting of the projects, but energy sector organizations are not included in this process to examine the relevance of such projects in terms of their sector’s needs. Joint research projects are also at the initial stages, and it is not possible to highlight any significant achievements made by CAS/NUST through joint research interventions. Public sector organizations working in energy are not fully involved or working closely with CAS/NUST on regulatory and policy matters. There have been a few meetings and dialogues; however, there has not been a concrete contribution to public policymaking in the energy sector. There is only one joint research project, on integrated energy modeling and planning, recently started by CAS/NUST and ASU. This project aims to provide technical analysis and integrated energy modeling to the Planning Commission of Pakistan to make energy policy decisions. When asked about public-private partnerships, industry linkages, and faculty’s understanding of the needs of the industry, an ASU respondent said, “Faculty members are rarely engaged in both corporate and the industrial activities at NUST.” When asked why ASU did not try to lead the public-private partnerships initiative, the respondent said, “I think we have been pushed away, and NUST wanted to be in the driver’s seat. ASU took the back seat, as it’s not … a mandatory activity. But we have tried to engage industry by initiating meetings with NUST faculty, but the success rate of this activity has not been good. ASU tried PMU’s support in building corporate relations, but our efforts were not appreciated by the PMU.” The CAS/NUST faculty has been unable to engage government officials from the Ministry of Energy, Ministry of Industries, National Electric Power Regulatory Authority, provincial energy departments, National Transmission and Dispatch Company, distribution companies, and other relevant government offices to showcase its capacity and the role it can play in providing solutions to Pakistan’s energy sector. Interviewees from private sector organizations showed great interest in working closely with CAS/NUST, but most of them were not clear about the capacities of CAS/NUST and how they could support it. The interviews also highlighted that CAS/NUST was not very open to close cooperation with the industry. The private sector respondents also reported the belief that the faculty at CAS is overburdened and engaged in so many things that it does not have time to visit industries to understand the challenges they face. The CEO of one of the power plants said, “Our power plants have technical issues every day and we want to use [the] expertise of CAS students and faculty to solve our issues. We have to send our power generators and turbines abroad for small issues, which can be fixed locally.” One respondent felt that CAS/NUST research should be designed to provide solutions to local problems, but that the faculty was more interested in publishing research papers in international journals, and that research is not particularly relevant to the needs of the country. The evaluation team’s interviews with applied research grantees, and private sector organizations receiving grants revealed some serious concerns. There were not clear processes in place to follow up with research grantees to complete the project on the given timelines. Procurement was a serious challenge causing delays to the results of applied research projects. One research grantee said, “It has been several months since our project got selected for applied research funding; however, so far we do not have any clarity. NUST is managing the procurement of material required for device fabrication; however, we are just waiting and there is no clarity. There is only one person from NUST PMU who coordinates with us, and except [for] that, there is no follow-up mechanism. [The] coprincipal investigator is also from CAS/NUST faculty who is never in touch with us. We want to complete our project as early possible; however, nothing is happening. We U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 123 have hired the staff to work on our project; however, we do not know what will happen and it is also adding to our overhead costs.” Public-Private Partnerships Public-private partnerships set up by CAS/NUST have not been very effective, and the results are not promising. According to the feedback of sector organizations, faculty at CAS/NUST is not open to the private sector or to understanding its problems. Faculty members currently teaching have Ph.D.s from abroad, and most came into teaching right after completing their doctoral degrees and have not worked in industry. CAS/NUST is influenced by a business-as-usual academia mindset, and it would benefit from bringing in people from industry and energy policy experts who could provide real-time insights on the needs of the energy sector. The NUST industry liaison office has primarily focused on seeking internships for students (mainly in the Pakistan Council of Renewable Energy Technologies and AGL), but the office could not provide adequate support to the CAS for building relationships with industry or for expanding collaboration on research projects. Alumni did not find any support from the center for employment once they had graduated. Lab Utilization Operationalization of Labs Operationalization of labs is recognized as a positive indicator of project performance. All CAS/NUST faculty survey respondents (100 percent) reported that the labs were operational at CAS/NUST. Eighty-eight percent of students reported that the labs were operational (n=167) (Figure 15). However, 56 percent of alumni respondents reported that not all CAS labs were operational, which could be attributed to procurement delays resulting in nonoperational labs in the early years of the project. The faculty sees great utilization of labs and its importance for energy research in Pakistan. One CAS/NUST faculty respondent said: “CAS provides the infrastructure for labs and capacity building for the lab utilization. Labs are the most critical part of CAS. We will not make this lab only for experiments. Our research labs will provide real solutions for [the] industry sector. We will also develop the NUST technology center as well. CAS will be part of it. We would be providing trainings, certifications, and test beds for various energy-related equipment. We are in the process of developing test beds for testing micro-hydro turbines, testing of energy-efficient fans, testing of energy-efficient lights, and motors, etc. Most importantly, we will be testing solar panels. We will set up a solar panel reliability and testing lab. CAS will conduct reliability tests for solar PV [photovoltaic] panels.” FIGURE 15: CAS/NUST STUDENTS’ RESPONSES REGARDING OPERATIONAL STATUS OF LABS U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 124 Ease of Access and Utility With respect to access to lab facilities, 20 percent of NUST faculty survey respondents felt that lab facilities were not easily accessible; however, 50 percent believed they were accessible, and 30 percent believed the labs were very easily accessible. Twenty percent of faculty rarely used the lab facilities, 20 percent used them very frequently, and 60 percent used them frequently. Forty percent of faculty did not use labs for teaching students (Figure 16), and 90 percent used them for their research work (Figure 17). Thirty percent of faculty believed that labs were not adequate to meet their teaching needs, and 30 percent felt that the lab equipment was not adequate to conduct applied research. Among students surveyed, 29 percent believed labs were easily accessible, and 44 percent said labs were accessible. Seventeen percent of students said they never used the labs, and 31 percent rarely used the labs. Labs were used frequently by 31 percent and very frequently by 14 percent. Concerning the quality of applied research, 36 percent of students said they believed that the labs significantly improved it, and 44 percent said they improved it somewhat, but 10 percent believed the labs made no improvement to the quality of applied research at CAS/NUST. Fifteen percent of students were dissatisfied with the lab equipment, and 1 percent were extremely dissatisfied and believed that the lab equipment was not adequate to conduct applied research. Faculty shared perspectives on utilization of labs by energy sector organizations. One respondent said: “Industry partners are using our labs for material testing. They use GCMS [gas chromatography – mass spectrometry] and XRD [X-ray diffraction] for material testing. Our material labs and combined labs are more frequently used by outside researchers because of the up-to-date equipment and facilities. Moreover, other universities, like International Islamic University, Islamabad; Quaid-i-Azam University; and University of Engineering and Technology, Lahore also send their researchers to our labs. Our solar thermal lab specifically provides solutions to the market. We have different sensors in this lab and energy auditing systems are in place.” Student group interviews were also conducted to provide qualitative data. One respondent, comparing lab utilization procedures at ASU and NUST, remarked: “I think it’s much harder to get permission to use the equipment [at NUST].” The students all contended that it was hard for faculty to get access to labs, as administrative delays and procedures made it harder to use lab equipment. The students discussed the challenges they faced in accessing labs and using them for experiments beyond the limited hours of 9:00 a.m. to 5:00 p.m. A special form and a cumbersome process are required to use the labs outside these hours, and the form only allows the student to stay until 8:00 p.m. Students who went on the FIGURE 17: FACULTY USING CAS/NUST LABORATORIES FOR RESEARCH FIGURE 16: FACULTY USING CAS/NUST LABORATORIES FOR TEACHING U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 125 exchange program experienced a different culture at ASU, where labs were open all the time. Students shared that they received training on some of the lab equipment but not on all of it. Current students also shared other views on lab utilization: “We do not see much difference in [the] skills of the faculty after [the] establishment of labs. However, we do see lab engineers guiding students on how to use labs. We have not used labs much as yet except the computer lab and smart grid lab as we are just starting with our research.” One alumnus did not realize much benefit from utilization of labs and reported, “We would have a limited benefit of it because we don’t have access to the equipment. In the industry, you have to handle the equipment. One should know how to handle all the equipment used in research work; however, our expertise is limited. The problem is we were not allowed to use all the equipment.” A CAS/NUST PMU administrator believed, “Labs are being effectively utilized by students, faculty, and also other schools of NUST … [and] for commercial use [for which] we have [a] limited fee. For [lab] usage, [our] first preference is CAS students, then the other schools of NUST, and the last priority is commercial use.” Some students and alumni highlighted the bureaucratic process at CAS/NUST for getting access to labs. They believed “access to the lab is still subject to a tedious bureaucratic processes and approvals. This also causes a delay in processing research data and is also a source of discouragement among the CAS students, as access to the lab is always governed by a rigid power structure in the CAS. Students that are supervised by the principal, dean, or head of department get the opportunity to use lab equipment; however, other candidates have limited access just because they are not supervised by people with decision-making powers.” The group also pointed out that collaborative research is highly restricted, as students are not allowed to use the labs of other departments. The alumni group agreed that the issues described above still exist and that their research experiences were not improved by the CAS research-support mechanisms. An evaluation team member (energy expert) observed that the biofuel lab, smart grid lab, computer simulation lab, and materials lab were almost fully equipped except for a small amount of equipment that was in the procurement process. Lab procurement and installation had been a challenge, with delays resulting from the need to purchase specific equipment only from U.S. vendors or to procure only U.S.- made software. Conclusions Sector organizations do not make much use of the CAS/NUST labs, and there is a need to market and launch an outreach program for the labs. This would allow government and private sector organizations to learn about the capacity of CAS labs and what kind of services they can provide. Sector organizations have heard about the labs, but they are not aware of their capacity or whether the labs are accredited so they can trust their results. There have not been any significant contributions by the labs to applied research. Students should be provided easy access to the labs, and the process of expanding the hours that labs are available for experiments should be improved. Students should be provided with hands-on experience with the lab equipment, which could add to their skill set and expand their opportunities after graduation. The strengths of the state-of-the-art equipment and information on how these labs can help meet energy sector needs in Pakistan should be shared across energy sector stakeholders throughout Pakistan. The solar PV lab could be helpful in ensuring the quality of imported solar panels, which is the major concern of solar PV project developers and owners in Pakistan. CAS/NUST should also obtain the necessary laboratory accreditation so that the stakeholders will have confidence in benchmarked testing and research services provided in Pakistan. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 126 CAS/UETP Effectiveness of Exchange Programs Overall, 231 students from CAS/UETP, all of whom were enrolled in the M.S. program, participated in the online survey. One hundred and ten students did not apply for the exchange program, and 121 applied. Of the respondents, 20 percent believed that the selection process for the exchange program was not transparent, 25 percent did not know much about the selection process, 52 percent found the selection process transparent, and 3 percent did not answer. Of 121 applicants to the exchange program, 61 (50 percent) were selected and 60 (50 percent) were not. Of the 61 selected students, 15 did not participate in the exchange program. Of those participating, 22 percent did not agree that CAS prepared them well before the start of the exchange program; however, 96 percent believed that ASU adequately supported them in adjusting to the new environment during the exchange program. The percentage of students who confirmed the support of their U.S. partner university was higher at UETP than at the other two CASs (86 percent at NUST and 75 percent at MUET). Among UETP student respondents, 85 percent found the exchange program helpful in improving their applied research skills, and 15 percent did not. The survey also found that 80 percent of students found the exchange program helpful in improving their applied research output. Of the 46 students who responded to the question on improvement in proposal writing skills, 36 (78 percent) found the exchange program helpful in improving their proposal writing skills, and 22 percent did not (Figure 18). FIGURE 18: IMPROVED STUDENT APPLIED RESEARCH SKILLS, APPLIED RESEARCH OUTPUT, AND PROPOSAL WRITING SKILLS AT CAS/UETP Source: PERFORM survey of CAS/UETP students, n=46. The evaluation team conducted three group interviews with current UETP students. The data showed that the exchange program was helpful for students in improving their skills in applied research, utilizing the equipment in the labs, problem solving skills, and communication. One of the students said, “My research skills were polished. Before going to Arizona, I didn’t have any experience. I used to read research papers but was totally blank in understanding them because I was not exposed to technology at all. I didn’t have experience with machines, characterization techniques, and how the research is done in real time.” Students also found the exchange program helpful in improving their problem-solving ability along with their research skills. One student shared his views as follows: “For students, improvement has been seen in my U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 127 opinion. Their technical and critical thinking skills and their command over the subject in which they have performed their research at ASU.” The qualitative data also reflect that the exchange program helped students in writing more research papers and developing more research ideas. Students also found that professors at ASU helped them in changing their perspective about education, technology, research, and engineering. Students also experienced the use of labs for their research work at ASU and considered this to be helpful. ASU staff showed the exchange students all the relevant labs at ASU, and the students had a chance to work in the labs of their interest. One of the students had an interest in working in the policy research lab, and during the exchange program he had the opportunity to work in ASU’s policy research lab. Most of the UETP students also worked on their research related to their theses as they did not have labs set up at UETP at that time and they wanted to utilize this opportunity effectively. A few of the students studying materials for energy storage and conversion also did research work in the materials and characterizations labs at ASU. Qualitative data also show that some of the students lacked confidence and effective presentation skills. Data from interviews with students and faculty show that students from UETP improved both their confidence levels and their presentation skills. UETP faculty also noticed a change in the students who went on the exchange program, especially female students. Faculty believed their students were more confident and comfortable interacting with fellow students and also with the teachers after the exchange. Students highlighted a few challenges they faced regarding the exchange program. One challenge was the timing of the interviews. Students felt that having interviews for the exchange program while they were preparing for exams was too hectic. A few students also raised concerns over the selection process. One said, “My name was on [the] notice board; however, they said you were not selected. When I questioned [this], they said we had received email from ASU. Then I asked them to show me the mail, and they said this is confidential.” Another challenge raised by UETP students concerned research proposals. One student said, “UETP students in [the] exchange program do not have [a] research proposal ready when they go on [the] exchange program. A core reason is sending of first- and second-semester students on the exchange program, whereas NUST selects its exchange scholars from the third semester and they have the capacity to develop research proposals by the time they get to enroll in the third semester.” Overall, 14 faculty members from UETP participated in the online survey, and of these, only 4 applied for the exchange program. Among the four applicants, only one faculty member was not selected for the exchange program, and the remaining three selected faculty members participated in the exchange program. Two of these three faculty members said that the exchange program helped in improving their applied research skills and research outputs. They also felt that the exchange program improved their teaching and proposal writing skills. In-depth interviews were conducted with five faculty members. Faculty members who were on the exchange program at ASU at the time of the evaluation were interviewed on Skype. Interview data show that most of the faculty members who did not apply for the exchange program held a Ph.D. from a European university and felt that a program lasting only a few weeks would not make much difference to their professional development. One of the respondents shared his perspective of the exchange program by saying, “My master’s is from Sweden … from [one of the] top 50 universities in engineering. My Ph.D. is from Denmark, and with five and [a] half years of exposure in European universities, my research skills are well developed; therefore, [a] six-week program did not make any difference.” U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 128 Some faculty members thought it was better for them to stay in Pakistan and work with their students rather than participating in the exchange program. They felt that being away from their students for four months was a challenge and that the students needed their time. They could work virtually with the ASU team rather than going on the exchange program. Others did not opt for the exchange program due to personal and family issues. All the faculty members participating in the exchange program felt that the program improved their teaching skills and applied research output; in addition, they initiated joint research projects. Some of the faculty members who did not apply for the exchange program had a different perspective, and they believed that the exchange program was not well planned or well structured. The low ratio of exchange program applicants among the faculty at UETP indicated that the faculty did not see value in the program. They also were not provided with a clear and detailed plan for the duration of the exchange program. The exchange program for faculty is open-ended and does not require deliverables to be submitted over the course of the program or include a follow-up program with exchange faculty. The ASU local team in Pakistan also did not notice differences in exchange faculty after they returned from ASU; however, they found the program very useful for students at UETP. Conclusions The exchange program was very helpful for students, and the faculty noticed changes in the returning students’ confidence and presentation skills. There is a need to improve the exchange program for students in terms of detailed planning for the desired outcome of the program. UETP should provide more time for students to discuss their research proposals, research objectives, and the timelines of well-defined deliverables. A mechanism for following up with exchange program alumni is also missing. Ongoing and long-term professional connections developed during the exchange program should be reinforced and pursued to facilitate future collaboration on research projects. The timing of the exchange program for students causes a delay in the students’ graduation. The four months of the exchange program disrupt the students’ studies at UETP, and the time spent on the exchange is not credited toward a degree. The program should be structured in a way that allows some part of the coursework or lab work completed at ASU to be considered as official credit hours in degree transcripts. This strategy would further motivate students to have a clear and structured plan for the exchange program and help them better utilize their time on the exchange program. This approach also would not necessitate delaying the students’ graduation date. The exchange program for faculty was not as useful as the one for students, with the major reason being that the foreign-qualified faculty does not see value in an exchange program of short duration. The program for faculty should be either discontinued or revised with detailed deliverables and outcomes. Faculty time spent away from Pakistan also affects the students on the UETP campus, and the center has to arrange for substitute lectures. Faculty could engage further with students to prepare them to make the most of the exchange program. Faculty also could engage in continual, weekly follow-up with exchange students to monitor the progress of their plans with respect to deliverables and timelines. Additionally, more students could be accommodated in place of faculty. Most of the students did not have an opportunity to experience the international education facilities. Accommodating more students would provide additional opportunities for students to experience the highest standards of labs, research work, and learning opportunities at ASU. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 129 Joint Research, Applied Research, and Public-Private Partnerships Fourteen faculty members responded to an online survey regarding joint research, applied research, and public-private partnerships. The majority of faculty (64 percent) were involved in applied research at CAS/UETP (Figure 19). Nine faculty members responded to the question on the relevance of applied research to energy sector organizations’ needs; 78 percent strongly agreed that their research is relevant to the needs of energy sector organizations. Overall, 93 percent agreed that applied research topics are aligned with the explicit needs of the energy sector, and only 7 percent disagreed. UETP faculty agreed that energy sector organizations are participating in and supporting applied research being conducted at UETP. Satisfaction (i.e., satisfied or extremely satisfied) with the selection process for applied research projects was reported by 86 percent of faculty, and the same percentage was satisfied with grants for applied research projects. However, 7 percent were extremely dissatisfied, and another 7 percent were dissatisfied. Of the UETP faculty respondents, 14 percent were not satisfied with the technical support provided by ASU regarding applied research projects, and another 14 percent were extremely dissatisfied; however, 71 percent were either satisfied or extremely satisfied with the ASU support. Concerning autonomy for using research grants, 14 percent of faculty were either dissatisfied or extremely dissatisfied, 64 percent were satisfied, and 21 percent were extremely satisfied. Nearly all the responding faculty (92 percent) agreed that the labs contributed to improving applied research at UETP; 9 percent of faculty did not agree. During interviews, faculty discussed three joint research projects that UETP initiated with ASU. These projects are mostly in the initial phase of procurement, and no results have been reported to date. The projects include: 1. Integration of biomass PV and micro-hydro; 2. Energy generation from biomass; and 3. Energy policy. The faculty believes that these joint research projects will be helpful for the energy sector of Pakistan. The UETP faculty and ASU interviews indicated that both universities engaged in a consultative process with the sector organizations to prioritize the joint research projects and that the universities then discussed and finalized the joint research projects. An ASU respondent further shared that the target of five joint research projects has been achieved with two additional projects recently awarded. Three cycles of joint research project funding have been completed. A faculty member from UETP explained the review and evaluation process for joint research projects by saying, “There are two levels for reviews, one is midterm and one is that final level of review. There are technical advisors from ASU, and in both these evaluations, we have the technical advisor from UETP as [a] member of the committee. [The] committee monitors the progress of joint research projects.” FIGURE 19: FACULTY ENGAGEMENT IN APPLIED RESEARCH SINCE JOINING CAS/UETP U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 130 Since ASU works with both UETP and NUST, the three universities have encountered major challenges in working together, mostly in terms of administration and competition between UETP and NUST regarding joint research projects. A respondent from ASU shared his view that “NUST and UETP had [a] rivalry as ... [a] big brother, big sister rivalry, maybe, but it was resolved in the later cycles.” Faculty also raised issues with the distribution of funding between ASU and UETP (Figure 20). The faculty point of view was that most of the fieldwork is done in Pakistan, and fieldwork requires funding; however, a major part of the funding goes to ASU, and this has been a challenge for UETP faculty. One faculty respondent said: “The research grant that we get is $30,000, while $40,000 goes to ASU. The problem with some of the projects is that everything is done here like data collection [and] fieldwork, so [a] major chunk of the money goes to these operational expenses. I suggested [they] do something—at least they should provide some support in surveys or provide us some software, which could cover our cost. The distribution of money is not rational, and I think more … should be given to [the] Pakistani partner as we collect data, [do] fieldwork, and provide them, and they just do analysis.” Faculty members also shared that they should be equally involved in decision-making. As one said, “We should be equal in terms of decision and monetary terms. I raised this point many times.” ASU professors and researchers have not been able to visit Pakistan for joint research projects, and UETP faculty has to communicate distantly; both UETP faculty and ASU respondents highlighted that this has been a challenge. An ASU respondent said, “I think the distance is obviously one of the issues, but that is always the case, even with joint projects in the U.S., we have projects with, no matter how many Skype calls we have, face to face makes a significant difference.” UETP also highlighted that the review process from the ASU side takes considerable time and gave the example of current joint projects: “The process with ASU is too lengthy, for example the project, which I am going to do today, the process to get it, started from October. It’s like almost seven months.” The UETP CAS administration was asked about the challenges in joint research projects. An administrator explained: “[What] we were expecting is not there. It’s not physically there on the ground. I cannot see any American person coming here and working and using or deploying these technologies. This is the first thing, and [the] second thing is I cannot see the impact. If there is a project going on, nobody knows what is the scale and what might be the bigger impact. Realistically, I do not see any major impact so far. I was very keen that if [the] U.S. university can help us in technology transfer because they have the labs, they have robust mechanisms, and we were very keen if they give us some of the product. Our experts can re-fabricate those here and then [they] can be brought into the market. That was my expectation, but unfortunately we are not up to the level or up to that mark yet.” To identify applied research projects, CAS/UETP has an open call for project proposals, which are selected based on their relevance to meet the needs of the energy sector in Pakistan. There are many applied research projects in the implementation phase, but no project has yet reached the product FIGURE 20: CAS/UETP FACULTY PERCEPTIONS ABOUT RESEARCH FUNDING U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 131 development and commercialization stage. When the PMU leadership was asked about significant contributions made by CAS to applied research, a respondent replied: “I will say, ‘No.’ I can tell you the four components of applied research, one is [research and development], the second one is proof of concept, the third one is pre-commercialization, and [the] last one is commercialization. Even in [the] U.S., when I am talking about Silicon Valley, it takes almost nine years. We actually started back in 2015, and we started from … scratch, so … [the] time … is very short, and making some new materials and [bringing them] to the market is something which realistically no one expects. The second thing is that we do not have [a] supporting ecosystem in Pakistan.” There was evidence of some interesting applied research projects related to integrated energy modeling and power factor improvement. In interviews conducted with energy sector organizations, there was clear excitement and interest in working closely with UETP. Interviews conducted with Pakhtunkhwa Energy Development Organization (PEDO), Peshawar Electric Supply Company (PESCO), and Khyber Pakhtunkhwa Oil and Gas Company Limited (KPOGCL) showed that sector organizations are looking forward to working with CAS faculty and students on applied research projects. PESCO highlighted challenges in power theft and line losses and also the need for a system to identify when transformers need maintenance before they break down. PESCO wanted to work closely on applied research projects related to power distribution and net-metering as well. There has not yet been any project initiated with PEDO or PESCO. The significant results of applied research are not yet apparent. Moreover, relationships with the sector organizations have developed quite recently and are limited to only a few meetings and discussions at the junior level. The higher management of PEDO and PESCO has not yet become involved, and no formal MOU is in place. A sector organization respondent said, “There is no applied research, the only focus is on copy and paste, no field exposure. If they are given field exposure, they can give good results.” Students also responded regarding the relevance of applied research to the needs of the energy sector. One replied, “We feel there is no linkage between academia and industry.” They also felt that the unavailability of labs, limited applied research funding, and slow process of proposal acceptance have been major challenges for applied research. The students’ perspective was that the research activities undertaken at UETP are primarily for academic purposes and that these activities have little to do with research that can solve the country’s energy issues. The students felt they were involved in research that may have academic merit but does not offer anything to the energy sector of Pakistan. Energy sector organizations were consulted in the technical review of applied research projects, and a representative of the sector organizations sits on the review committee. CAS/UETP leadership and sector organizations were also satisfied with the process in place for conducting applied research that will meet the needs of the energy sector. ASU highlighted that there are set rules and policies and a template for applied research projects, which are shared with the UETP faculty and other stakeholders. ASU provided its review, feedback, and support in the absence of technical advisors. CAS alumni were also interviewed and shared their views regarding public-private partnerships. A representative said, “Honestly, on paper they have [public-private partnerships]; in reality a lot is to be done. There are MOUs they have signed but no follow-up—there was no result.” Recent MOUs have been signed with Haier and the National Transmission and Dispatch Company. There were also discussions and meetings with the Pakistan Air Force and Heavy Industries Taxila to build partnerships with the center to address their energy and materials research. The recently established industry liaison office has also been actively trying to engage organizations from the government and private sector. There have been efforts and meetings to establish working relationships with PEDO and PESCO, but these partnerships are at a very early stage of discussions. Sector organizations need the clarity and confidence provided by U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 132 the center’s showing that it can provide solutions to their problems and that it has the capacity to deliver. The outreach and linkages between sector organizations need to be enhanced to build the center’s reputation for excellence in the energy sector. KPOGCL has a strong working relationship with CAS/UETP and provides considerable support to the center. The CEO of KPOGCL is on the curriculum review committees and also teaches as a visiting faculty member in the energy management and sustainability degree program. However, UETP relies mostly on KPOGCL, and there is a need to expand the scope of partnerships and also involve other sector organizations in applied research and curriculum review committees. Lab Utilization Overall, 79 percent of UETP faculty (n=14) participating in the survey said that labs were operational, and 21 percent said labs were not fully operational at UETP. Of UETP faculty, 82 percent believed the lab facilities were easily accessible, and 18 percent believed they were accessible. Lab usage by faculty was reported as follows: 18 percent rarely used lab facilities, 72 percent used labs frequently, and 9 percent used them very frequently. Over four-fifths (82 percent) of faculty used labs for teaching, and the same percentage used them for their research work. More than one-fourth (27 percent) of faculty believed that the labs were not adequate to meet their teaching needs. More than one-third (36 percent) of faculty believed the labs significantly improved the quality of applied research, 55 percent believed they somewhat improved the quality, and 9 percent believed that there was no improvement in applied research quality because of the labs. Among the UETP students surveyed, 73 percent said the labs were operational. In terms of accessibility, 17 percent of students believed labs were not easily accessible, 1 percent said they were not accessible at all, 38 percent believed labs were easily accessible, and 44 percent said labs were accessible. In terms of lab usage, 13 percent of students said they never used labs, 31 percent rarely used them, 45 percent used them frequently, and 11 percent used them very frequently. Regarding whether the labs improved the quality of applied research, 39 percent of students believed they improved the quality of applied research, 40 percent said they improved it somewhat, and 9 percent believed the labs made no improvement to the quality of applied research. Eleven percent of students were dissatisfied with the lab equipment, and 1 percent were extremely dissatisfied and believed that the lab equipment was not adequate to conduct applied research. Qualitative data regarding the labs and their utilization was collected through interviews with students, faculty, sector organizations, HEC, and USAID. In response to questions regarding labs, students said that the labs were operational and that they were provided with training to use the lab equipment. Some of the students provided a different perspective, saying that they were not given training to use the lab equipment, but that the engineers working in the labs helped them a great deal. Students also highlighted that they could access the labs on weekdays but could not work on weekends, as lab engineers were not available on weekends. One student commenting on the adequacy of labs for applied research reported: “Labs are not here for applied research and … faculty isn’t professional. I raised [this] specifically with [the] head of department regarding the labs. … I have told them the faculty should not be like the way it is right now. The labs must be inaugurated. The equipment is lying here for the last one year and it isn’t yet opened for students. They say they have problems from HEC. I have been to [the] materials research lab [and] we had [an] XRD machine; it took 16 days coming from Germany to here. I don’t know where are those written things, like a lab should be opened for students in two years. They don’t have enough knowledge and experience to run these things.” U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 133 Another student responded and said, “That’s a very good facility; however, faculty don’t have [the] knowledge to run these facilities and how they work, and if they go out of order how to fix them. They have hired contractors from China and Germany.” All the students agreed that their research work was affected by the delays in operationalizing the labs and that they did not get a chance to work on the equipment in the labs. Another student shared an example of a student whose research work was affected: “One of my friends … graduated from GIKI [Ghulam Ishaq Khan Institute of Engineering Sciences and Technology]; he then … enrolled here in CAS. He started his research back in January 2017; now it’s May 2018 and he still has to perform two tests of XRD and Scanning Electron Microscope. He is asking … the faculty to use the lab for these tests, as materials lab installation is complete. He has been told that you have not maintained [a] specific account for the usage of [the] lab. He has been asked to submit the application to use the labs. He chooses his research topics based on the lab and equipment availability. He waited till August 2017 for [the] labs to [be] functional …. [then he was] sent to NUST and GIKI to use their labs.” Faculty believed that 70 percent of labs were operational and that students were using the labs with the supervision of lab engineers. One faculty member said: “Labs have just been established in [the] last three or four months. The equipment is interlinked, and labs are being utilized 70 percent; however, there are some issues of costing because if our students were performing the tests free, then our systems would be degrading. For students to use labs, there is budget allocated for them, which is 1 lac, so out of this money he can utilize the labs, but we cannot give a free hand to the students to use equipment because if you give free hands others will not get their turn. For thermal system engineering, we don’t have [many] facilities because this was started last year, and I tried my best because I joined the center in April, and the target was to establish labs by [the] end of [the] first semester, but due to funding issues, approval from USAID [was] delayed, and this is [the] end of the second semester, and we have yet to establish one of the labs. We don’t have any thermal energy lab, so no utilization.” ASU respondents also shared their perspective regarding labs and their utilization. ASU found the labs to be the most critical and challenging component. The ASU team believed that students were properly utilizing the labs as well as some of the equipment, which was solely for training purposes. A week-long training program is provided to the students on the use of specific equipment. One ASU respondent reported, “Most of the equipment [is] accessible to students; however, there is some sophisticated equipment which I don’t think could be left to students, like [the] electron microscopes and others. This equipment needs well-trained operators. I think that is one of the ways to better utilize the equipment.” An ASU respondent also commented on lab utilization by sector organizations: “The services of labs are being availed by the energy sector, including NEECA [the National Energy Efficiency and Conservation Authority]. We also have [an] energy auditing kit, and we are approached by the industry for energy auditing services. When we arranged [an] industry visit, the industry wanted their employee to be trained on the lab equipment that we have in the center. We are trying to facilitate the industry to use the lab equipment. The focus on weekend use, evening hours opening, etc., is already a part of the process. We’re trying to use infrastructure better.” Sector organizations were also asked if they have used the labs. A respondent from PESCO said: “We have not used it yet; however, we visited it last month. We have already submitted it to [the] chief, and after the chief’s visit something will happen. We are also having time-to-time U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 134 training of our management in [the] power sector, electronics, and electrical power lab. We will keep it for visits. Whatever is taught in our staff colleges is an old technology, and it is all shifting on [to the] new one now. We want to bring our staff here for training. The KPOGCL respondent said: “I am an electrical engineer, and I have seen the equipment lying dormant with UETP in the past, and some of the equipment used by oil and gas still lying dormant in the campus. We had a number of meetings with the professors. We proposed a number of ways to utilize them, and we have given the rates—this is their rate and you charge this rate, and this is so automatic that you don’t need to do hard work. Fortunately, things are very easy for oil and gas. … We have taken rates from HDIP [Hydrocarbon Development Institute of Pakistan], OGDCL [Oil and Gas Development Company Limited], and other departments; however, there is a kind of mental block at the university to market it.” The respondent from PESCO said: “Yes, I know about their labs, and I have visited all of them including [their] technical lab, metallurgy lab, policy lab, and computer lab. We can get a lot of help. [The] power lab will be very helpful, and they are working on smart grids because it is a basic need for the future, and we can work with them. The lab that I’ve visited is well equipped; one smart grid is being established in PESCO headquarters, and hopefully we will be able to get help from them regarding the smart grids.” One lab engineer, supported by research associates, works at each lab. The evaluation team collected qualitative data from the lab engineers. One engineer reported: “The trainings are offered on a daily basis from 8:30 to 10:30 a.m. Only two lab engineers were there for five labs, and only three labs were reported to be in students’ use: solar PV, material characterization, and material synthesis. All these are fully functioning and regularly in use. Students spend eight hours a week in [these] three labs. Moreover, students of UETP are also facilitated and they are charged for using labs. Charges that applied to students are deducted from the grants of projects they were involved in. Students are not permitted to use the lab equipment independently without mandatory trainings. I don’t think that there are any challenges. Students are allowed to use lab equipment even after office hours, however, only when the lab engineers supervise them.” Most of the labs are operational at UETP, except the thermal energy lab, and some other labs are missing equipment. Materials synthesis and materials characterization labs were set up at CAS/UETP to meet the research needs of the materials for energy storage and conversion degree program, which was later terminated, and the center started a new program on renewable energy. These labs were recently merged into the renewable energy lab. No detailed needs assessment was performed before initiating the Materials for Energy Storage and Conversion (MESC) program and setting up the materials synthesis and characterization labs. Students enrolled in the MESC program chose their research projects based on the hope that these labs would be available. However, the labs were installed late, and students had to go to other universities to work on their research projects; some of the students were told by faculty to opt for only simulation-based projects due to the unavailability of research labs. This situation has affected the research of the MESC program’s students. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 135 Conclusions The data also provided evidence that students were not allowed to use equipment or work on their research ideas independently. Within CAS/UETP, some are concerned that the equipment is very expensive and that some kind of mishandling will ruin the machines. This affects the innovation that is the central idea of CAS in the energy sector. Innovation comes with freedom in using the machines and instruments. There is also a need to focus on an outreach program to market these state-of-the-art labs, which can be utilized by energy sector organizations and other universities in KP and FATA. The center should market its strengths with this state-of-the-art equipment and show how it can be helpful in solving problems faced by the energy sector in Pakistan. UETP should allow access to the labs even after office hours, as some lab tests take a lengthy amount of time; the results of such tests cannot be obtained in a few hours and may require working late hours. Faculty also need to enhance their capacity and understanding of the lab equipment to ensure better knowledge transfer to the students. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 136 ANNEX 4: METHODS Survey of CAS Faculty, Students, and Graduates The evaluation team conducted online census surveys of CAS faculty, current CAS students, and graduates to ensure that findings are generalizable to the populations. The surveys collected data on partnerships between Pakistani and U.S. universities, exchange experiences, joint and applied research, lab utilization, curriculum reform, governance, graduate training, academia-industry partnerships, linkages to industry, gender issues, employment of graduates, and sustainability. The evaluation team implemented the structured surveys online. The team obtained lists with the email addresses and telephone numbers of 61 faculty members, 677 students, and 138 CAS graduates from each CAS. Each CAS first notified all faculty, students, and alumni by email of the upcoming evaluation and introduced MSI as a third-party evaluator. Following the introductory email, the evaluation team sent each faculty member, student, and graduate a link to an online survey and requested that he or she complete the survey within seven days. After seven days, the evaluation team sent a reminder email to those who had not responded to the survey. The team sent a couple of further reminder emails to those who had not participated in the surveys. If the reminder emails did not elicit a response, the evaluation team telephoned respondents who had not completed the survey and asked them either to complete the survey on the phone or to complete the survey online at their earliest convenience. To boost the response rate, the evaluation team also requested each CAS to encourage the faculty, students, and alumni to complete the online surveys. In total, 67 percent of faculty, 74 percent of CAS students, and 62 percent of CAS alumni responded to the surveys (Table 11). Of those surveyed, 17 percent of faculty, 22 percent of students, and 19 percent of alumni were women. TABLE 11: ONLINE SURVEY RESPONSE RATE Respondent Group CAS University Overall NUST UETP MUET Faculty 59% 64% 77% 67% CAS students 72% 77% 73% 74% CAS alumni 61% 86% 53% 62% Key Informant Interviews The evaluation team conducted 72 semi-structured individual interviews with key informants associated with the project and external sector experts who could provide feedback on the project’s design, management, implementation, and sustainability aspects. Specifically, the key informant interviews elicited qualitative data about the benefits and challenges of various components of the U.S.-PCAS project (e.g., exchange programs, joint and applied research, lab utilization, curriculum reform, and graduate training) and how these components were effective in terms of meeting industry needs, preparing graduates for employment, and conducting applied research. The key informant interviews also collected data on how the CASs perceive sustainability and what efforts CAS host universities are making to incorporate CASs into the university systems. The evaluation team purposively selected key informants to capture specific expertise, experience, or perspectives. The team selected informants who were familiar with CAS and could provide the information that answered the evaluation questions. Key informants included representatives from USAID, CAS implementing partners (NUST, UETP, MUET, U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 137 ASU, and Utah), CAS faculty, sector organizations,9 and HEC personnel (Table 12). The interviews focused on developing a comprehensive understanding of the partnership experience of Pakistani and U.S. universities, exchange experiences, joint and applied research, lab utilization, curriculum reform, CAS governance, graduate training, academia-industry partnerships, linkages to industry, gender issues, employment of graduates, and sustainability. The evaluation team designed seven separate interview guides, one for each informant type based on their expertise and understanding of various aspects of U.S.-PCAS. TABLE 12: KEY INFORMANT INTERVIEWS Respondents NUST UETP MUET Total CAS faculty 7 6 5 18 CAS lab personnel 2 2 2 6 Host university leadership 2 3 3 8 CAS admin/PMU 2 4 4 10 Sector organizations 4 3 5 12 U.S. partner universities 4 3 7 14 USAID (AORs) 1 1 - 2 HEC - 2 Total 22 22 26 72 Group Interviews The evaluation team conducted 18 group interviews with current CAS students and graduates to collect qualitative data about how various components of the U.S.-PCAS project (e.g., exchange programs, joint and applied research, lab utilization, curriculum reform, and graduate training) were effective in preparing them to meet industry needs, obtain employment, and conduct applied research. The group interviews also collected data on how CASs are addressing gender issues and improving women’s participation. The evaluation team randomly selected students for group interviews in a way that provided the perspectives of students from senior batches and current exchange students. However, the participants for group interviews with alumni were selected using convenience sampling. 10 The evaluation team conducted 11 group interviews with current CAS students and 7 with CAS graduates (Table 13). Each group interview consisted of three to four participants. In total, the group interview sample of current students included 40 participants, of which 21 were women, and the group interview sample of CAS alumni had 25 participants, of which 10 were women. TABLE 13: DISTRIBUTION OF GROUP INTERVIEWS Respondents NUST UETP MUET Total Current CAS students 4 4 3 11 CAS alumni 3 1 3 7 Total 7 5 6 18 9 These are the organizations with which CASs have signed MoUs and with which they collaborate for applied research, curriculum reform, and recruitment of graduates. 10 Convenience sampling was used for the selection of alumni because accessing alumni was a challenge. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 138 Validation Workshops Along with surveys and qualitative interviews, the validation workshops were a primary source of data. At the end of preliminary data analysis, the evaluation team conducted three validation workshops with implementing partners. In these workshops, the evaluation team presented preliminary findings to the implementing partners to stimulate dialogue around how each CAS operates as an integrated system. Such open, facilitated, and constructive dialogue embeds collaborating, learning, and adapting (CLA) principles directly into the evaluation process. These workshops provided an opportunity for CAS stakeholders to affirm (or probe) evaluators’ interpretations of the existing data set and integrate their diverse perspectives. The findings of the validation workshops also informed the conclusions and recommendations. Secondary Review In addition to collecting primary data, the evaluation team consulted the extensive secondary information that the project has produced and maintained (see Annex 9: List of Secondary Documents for a bibliography). This included project documents, the project performance data, and relevant research studies. The team used information in project documents to develop a thorough understanding of the project and measure various constructs used in the evaluation. The team also used quantitative data from the project’s databases to document project activities in each CAS and examine how trends have changed over the period. Specifically, the indicator data was useful for answering evaluation questions related to the effectiveness of various CAS components. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 139 ANNEX 5: SUSTAINABILITY INDICATOR DATA Target Areas MUET NUST UETP Fundraising No funding has been reported from the private sector. A MUET representative shared that the center has generated funds from the public sector, but these are not being reported as they do not satisfy the sustainability indicator criteria. PKR 112.01 million was raised/leveraged as of Sept. 2017 (Q4 2017), with PKR 30.72 million from the private sector, PKR 36.6 million from the nonprofit sector, and PKR 44.79 million from the public sector. PKR 12.5 million was raised from the public sector: Pakistan Air Force, KPOGCL, PEDO, several in process. Private and public partnerships Five partnerships established. Three partnerships established. Four partnerships established. Graduate employment and internships • Graduates: 35 (24 men, 11 women) • Employed: 22 (15 men, 7 women) • 25% current students confirmed internship placement • Graduates: 99 (83 men, 16 women) • Employed: 53 (48 men, 5 women) • 32.5% current students confirmed internship placement • Graduates: 7 (6 men, 1 woman) • Employed: 3 (3 men) • 26.1% current students confirmed internship placement Increased female participation • Faculty: 22 full-time (6 women) • Total student enrollment: 142 (33% women) • Scholarships given to all female students • Faculty: 18 full-time (3 women) • Total student enrollment: 301 (55% women) • Scholarships given to 34 women (62%) • Faculty: 12 full-time (no women) • Total student enrollment: 116 (14% women) • Scholarships given to all female students Enhanced diversity At MUET, all enrolled students are considered to be from disadvantaged groups. 228 Students: 76% At UETP, all enrolled students are considered to be from disadvantaged groups. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 140 ANNEX 6: CAS MIDTERM EVALUATION CONCEPTUAL FRAMEWORK U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 141 ANNEX 7: QUALITATIVE AND QUANTITATIVE DATA COLLECTION U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 142 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 143 ANNEX 8: MAXQDA SCREEN SHOT, PROVISION, AND FINAL CODE LIST U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 144 Final Code List Code System # Code System 6444 1. Partnership Experience 0 1.1 Suggestions to Enhance Partnership 4 1.2 Satisfaction with partnership 0 1.2.1 Yes 21 1.2.2 No 0 1.2.3 No Response/Unable to Answer 0 1.3 Perceived Benefits 0 1.3.1 Financial Opportunities 5 1.3.2 Improving Perceived Value/Prestige of University 16 1.3.3 Improving Curriculum/Teaching 47 1.3.4 Cumulative (General) Opportunities for improvement 19 1.3.5 Collaboration Potential 36 Relationship building with faculty 13 1.3.6 Facilities 27 1.3.7 Strengthening Sector Linkages 22 1.3.8 Research Skills Development/Strengthening 66 1.3.9 Exchange Programs 8 1.3.10 Modelling Best Practice 42 1.3.11 Capacity Building 25 1.3.12 Developing Regional Understanding 10 1.4 Anticipated Benefits 0 1.4.1 Strengthening Sector 22 1.4.2 Ongoing University Collaboration(s) 17 1.4.3 Strengthening University Prestige 6 1.4.4 Strengthening Sector Linkages 7 1.4.5 Ongoing Strengthening of University Capacity 10 1.4.6 Commercial Benefits 2 1.4.7 Sharing Knowledge with other Universities 2 1.4.8 Strengthening Center Sustainability 1 1.4.9 Securing Alternative Funding 3 1.4.10 Job Opportunities 1 1.5 Challenges and Solutions 0 1.5.1 Travel Constraints 1 1.5.2 Communication Issues 3 1.5.3 Design Issues 3 1.5.4 Planning Issues 24 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 145 Code System # 1.5.5 Difficult to Change Academic Culture 9 1.5.6 Lack of Technical/Project Management Skills 9 1.5.7 Partnership Uncertainty/Unfamiliarity 1 1.5.8 Delayed Facilities causing Constraints 6 1.5.9 Inexperienced Faculty/Difficulty Building Team Capacity 2 1.5.10 Lack of Regional Understanding 5 1.5.11 Attracting/Retaining Quality Faculty 1 1.5.12 Misunderstanding of Partner Roles 2 1.5.13 Lack of Awareness of Center 1 1.5.14 Governance/Management Issues 3 1.5.15 Outstanding Challenges not resolved 0 1.5.15.1 Weak Linkage between Host Needs and Partner U Resource 17 1.5.15.2 Solution: Brought in Other Partners 1 1.5.15.3 Limited Flexibility when Challenges Encountered 1 1.5.15.4 Planning Issues 17 1.5.15.5 Inconsiderate Behavior 1 1.5.15.6 Communication Issues 4 1.5.15.7 Directive Behavior by US Partners 11 1.5.15.8 Sustainability Questions 5 1.5.15.9 Differences between Actual and Reported Capacity 1 1.5.15.10 Not Understanding Academic Culture 1 1.5.15.11 Balancing CAS and Host University Requirements 5 1.5.15.12 Sector Linkage Issues 3 1.5.15.13 Failure to Adapt to Specific Needs of Host University 6 1.5.15.14 Security Issues 3 1.5.15.15 Failure to Understand Local Culture/Issues 1 1.5.15.16 Achieving Adequate Standards of Performance 5 1.5.15.17 Resistance/Responsiveness to Change 7 1.5.15.18 Student Prep for Sector 1 1.6 Solutions 0 Solution: Improved Marketing 1 Solution: Strong Training 1 Solution: Negotiating Changes 1 Solution: Sending Faculty, staff and students to Islamabad 1 Solution: Adapting to Exchange Student Needs 4 Solution: Building Trust Based Relationships 10 Solution: Strong Leadership/Management 7 Solution: Willingness to Learn 1 Solution: Extending Duration of Semester 1 Solution: Achieving Results Collaboratively Builds Excitement 2 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 146 Code System # Solution: Having US Partner Resident at CAS 3 Solution: Extending Duration of Exchange Program 2 Solution: Better Visa Process 1 Solution: Developing Procedures and Manuals 2 Solution: Student Involvement 1 2. Effectiveness of Project Components 0 2.1 Exchange Programs 0 2.1.1 Impact of Exchange program 0 2.1.1.1 Valuable in professional development of awardees 0 2.1.1.1.1 Yes 15 2.1.1.1.2 No 1 2.1.1.1.3 No Response/Unable to Answer 3 2.1.1.2 Research Skills 72 2.1.1.3 Teaching Skills (for faculty) 35 2.1.1.4 Problem Solving (for students) 31 2.1.1.5 Lab Use 44 2.1.1.6 Communication Skills 53 2.1.1.7 Cultural Understanding 15 2.1.1.8 Expanding Perceptions of Academic Excellence 23 2.1.1.9 Other areas 53 2.1.2 Role in preparation of exchange scholars 4 2.1.2.1 Role of Pakistani CAS University 4 2.1.2.1.1 Yes 10 2.1.2.1.2 No 4 2.1.2.1.3 No Response/Unable to Answer 0 2.1.2.2 Role of US CAS Partner University 17 2.1.2.2.1 Yes 13 2.1.2.2.2 No 0 2.1.2.2.3 No Response/Unable to Answer 1 2.1.3 Challenges and Solutions 0 2.1.3.1 Challenges 0 2.1.3.1.1 No challenges 4 2.1.3.1.2 Planning issues 26 2.1.3.1.3 Living issues 3 2.1.3.1.4 Adaption issues 8 2.1.3.1.5 Selection process 4 2.1.3.1.6 Unclear objectives 8 2.1.3.1.7 Travelling w/o family 1 2.1.3.1.8 Time/Duration 15 2.1.3.1.9 Research expertise/Labs 10 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 147 Code System # 2.1.3.1.10 Language skills/Communication skills 8 2.1.3.1 Challenges not resolved 0 Lack of support from CAS and ASU 5 Planning issues 30 Living issues 3 Adaption issues 7 Selection process 3 Unclear objectives 16 Travelling w/o family 1 Time/Duration 5 Research expertise/Labs 3 2.1.3.2 Solutions 0 2.1.3.2.1 Better planning 37 2.1.3.2.2 Providing more exchange opportunities 1 2.1.3.2.3 Support to students/Motivating students 8 2.1.3.2.4 Adjusting time duration 7 2.1.3.2.5 Managing expectations 4 2.1.3.2.6 Allowing faculty to travel with family 1 2.1.3.2.7 Building skills during program 9 2.1.3.2.8 Managing students workload 1 2.1.3.2.9 Make process more transparent 1 2.1.3.2.10 Students adapt to environment 4 2.1.3.2.11 Adapting to exchange students needs 3 2.1.4 Selection Process 1 2.1.4.1 Transparent/Effective 52 2.1.4.2 Not Transparent/Not Effective 37 2.1.4.3 Emphasis on Undergraduate Grades 14 2.1.4.4 Faculty 11 2.1.4.5 Students 96 2.1.4.6 Changing Selection Criteria 17 2.1.4.7 Solutions/How Addressed 13 2.1.5 Inclusion 1 2.1.5.1 Gender 0 2.1.5.1.1 Merit Based 34 2.1.5.1.2 Preference for Females 12 2.1.5.2 Special Needs 7 2.1.5.3 Economically Disadvantaged 20 2.1.6 Reasons for Not Participating 0 2.1.6.1 Personal Reasons 5 2.1.6.2 Family Resistance 5 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 148 Code System # 2.1.6.3 Visa Issues 2 2.1.6.4 Faculty Reasons 4 2.1.6.5 Grading Issues 2 2.1.6.6 Research Reasons 2 2.2 Joint Research (CAS University w International Partner Uni 0 2.2.1 CAS partnership Impact on Uni Joint Research Capacity 1 2.2.1.1 Too early to tell 3 2.2.1.2 Exposing US partners to Pakistani issues 8 2.2.1.3 Proposal development 6 2.2.1.4 Engaging Students in research 2 2.2.1.5 Building Research capacity/improving impact 41 2.2.2 Challenges and solutions 0 2.2.2.1 Too Early to see results 1 2.2.2.2 Taking Initiative 2 2.2.2.3 Proposal Development 1 2.2.2.4 Accessing Quality Data 2 2.2.2.5 Equipment Availability 1 2.2.2.6 Research Capacity Issues 1 2.2.2.7 Available Time 1 2.2.2.8 Balancing International and Local Issues 11 2.2.2.9 Funding 3 2.2.2.10 Challenges not resolved 0 2.2.2.10.1 Funding 5 2.2.2.10.2 Acceptance/Relevance of Findings 3 2.2.2.10.3 Matching Interests 3 2.2.2.10.4 Balancing International and Local Issues 4 2.2.2.10.5 Available Time 2 2.2.2.10.6 Research Capacity Issues 1 2.2.2.10.7 Equipment Availability 2 2.2.3 HEC Help in Promoting 12 2.2.4 Examples of Joint Research with other Universities 7 2.3 Applied Research in CAS 3 2.3.1 Definition of AR 5 2.3.2 Sector Satisfaction with Quality of Applied Research 0 2.3.2.1 Yes 8 2.3.2.2 No 1 2.3.2.3 No Response/Unable to Answer 3 2.3.3 Relevance to Needs of Sector 0 2.3.3.1 Significant contributions towards solving problems 0 2.3.3.1.1 Yes 11 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 149 Code System # 2.3.3.1.2 No 1 2.3.3.1.3 No Response/Unable to Answer 5 2.3.3.2 Process to Engage Sector Orgs in 0 2.3.3.2.1 Effective process in conducting research 0 2.3.3.2.1.1 Yes 14 2.3.3.2.1.2 No 0 2.3.3.2.1.3 No Response/Unable to Answer 3 2.3.3.2.2 Examples of Engagement 34 2.3.3.2.3 Opportunities to Engage (Not Taken) 5 2.3.3.2.4 Student Engagement in Research 13 2.3.3.2.5 Challenges in Process 19 2.3.3.2.6 Engagement through stakeholder Meetings 26 2.3.3.2.7 Engagement through Workshops/Seminars 14 2.3.3.2.8 Engagement through proposal Review 6 2.3.3.2.9 Financial Issues 3 2.3.3.3 Specific Examples of Contributions to Sector 43 2.3.3.4 General Examples of Relevance to Sector 38 2.3.3.5 Contribution or Relevance of Student Research 20 2.3.3.6 Challenges to Relevance or Contribution 13 2.3.3.7 Changes needed to strengthen relevance/ contribution 9 2.3.3.8 Perceptions on CAS capacity to contribute to sector nee 35 2.3.3.9 Perceptions on Sector ability to use Research 19 2.3.3.10 Anticipated Contributions/Relevance (Too early to tell 9 2.3.4 Role of US Partner University in CAS Applied Research 0 2.3.4.1 Data Analysis/Equipment Use 6 2.3.4.2 Lab Equipment Planning 3 2.3.4.3 Proposal Review/Writing 16 2.3.4.4 Project Development 5 2.3.4.5 Publication 2 2.3.4.6 Relationship Building/ Industry Funding 12 2.3.4.7 Training/Support 18 2.3.4.8 Little Help 5 2.3.5 Examples of Applied Research with other Universities 10 2.3.6 Challenges and Solutions 0 2.3.6.1 Faculty/Student Research Capacity 19 2.3.6.2 Engaging Sector Orgs/Data Access 44 2.3.6.3 Securing Adequate Funding 18 2.3.6.4 Research planning/Monitoring (Including cross-CAS) 13 2.3.6.5 Admin Challenges 15 2.3.6.6 Limited Research Scope 16 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 150 Code System # 2.3.6.7 Lab Resources/Procurement 22 2.3.6.8 Lack of Research Oriented Environment 17 2.3.6.9 Dissemination of Research/Information 6 2.3.6.10 Lack of Support for Students 23 2.3.6.11 Relevance of Research 11 2.4 Policy Research in CAS 0 2.4.1 Contribution to Policymaking Research 0 2.4.1.1 Yes 8 2.4.1.2 No 4 2.4.1.3 No Response/Unable to Answer 1 2.4.2 Efforts to contribute to policy research 0 2.4.2.1 Too early to tell 9 2.4.2.2 Little/No Impact 3 2.4.2.3 Holding policy forums 15 2.4.2.4 Creating Think Tank 6 2.4.2.5 Publications 7 2.4.2.6 Building capacity for policy research 2 2.4.2.7 Examples of capacity for Policy Research 20 2.4.2.8 Early Stage of Planning 13 2.4.2.9 Examples of CAS Policy Research 36 2.4.3 Challenges and Solutions 0 2.4.3.2 Challenges 0 2.4.3.2.1 Poor research findings dissemination/communication 7 2.4.3.2.2 Weak sector capacity to absorb policy research 5 2.4.3.2.3 Weak engagement of policymakers 21 2.4.3.2.4 Lacking strategy for policy research 28 2.4.3.2.5 Lack of capacity for policy research 22 2.4.3.2.6 Solutions/Suggestions 26 2.5 Public Private Partnerships 0 2.5.1 CAS Efforts to Create Partnerships with Sector Orgs 0 2.5.1.1 Need to have Focused Engagement Strategy 18 2.5.1.2 Efforts to Map /Understand Market 30 2.5.1.3 Focus on Signing MOUs 22 2.5.1.4 In Private Sector 24 2.5.1.5 With NGOs 5 2.5.1.6 With Startups/Entrepreneurial Orgs 5 2.5.1.7 With Government Organizations 46 2.5.2 Role of stakeholders (HEC, Others) in creating PPP 14 2.5.3 Factors Contributing to Successful Partnerships 0 2.5.3.1 Recognition of Strength & Needs of Each Partner 21 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 151 Code System # 2.5.3.2 Leadership Vision & Support for Partnership 6 2.5.3.3 Focus on Relationship Building 10 2.5.3.4 Collaborative Identification of Gaps and Opps 13 2.5.4 Factors Inhibiting Successful Partnerships 0 2.5.4.1 Resource issue 8 2.5.4.2 Dominant Partner 10 2.5.4.3 Weak Follow-up/ MOU Activation 15 2.5.4.4 Weak Strategic Outreach/ Relationship Building 30 2.5.4.5 Capacity to Deliver Timely Solutions 8 2.6 Lab Utilization 1 2.6.1 Lab Utilization process/ Training 73 2.6.2 How Utilized (Examples) 15 2.6.2.1 By Students 64 2.6.2.2 By Faculty 13 2.6.2.3 By Sector Organizations 56 2.6.2.4 By Non-CAS universities 6 2.6.3 Utilization Effectiveness 39 2.6.3.1 Contribution to improvement of research quality 10 2.6.3.1.1 Yes 9 2.6.3.1.2 No 0 2.6.3.1.3 No Response/Unable to Answer 4 2.6.4 Challenges and Solutions related to Lab access & use 1 2.6.4.1 Labs still not Operational 25 2.6.4.2 Labs not Integrated into Teaching 21 2.6.4.3 Lack of Training on Lab Use 31 2.6.4.4 Lab Access Issues 39 2.6.4.5 Poor Planning (Limited Space & Resources) 26 2.6.4.6 Managerial Oversight of Labs 25 2.6.4.7 Keeping Equipment/Software Up to Date 12 2.6.5 Procurement Issues 42 2.6.5.1 Solutions/How Addressed 5 2.6.6 Comfort of Females using Labs 45 2.6.6.1 Females present on Lab Staff? 13 2.6.6.1.1 Yes 12 2.6.6.1.2 No 0 2.6.6.1.3 Don’t know 0 2.6.6.2 Possible Limitations on Female Use 2 2.7 Curricular Reform/New Programs 0 2.7.1 Effectiveness of Curricula in Prep Students for Sector 0 2.7.1.1 Satisfaction of stakeholders (employer, uni) with train 3 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 152 Code System # 2.7.1.1.1 Yes 8 2.7.1.1.2 No 4 2.7.1.1.3 No Response/Unable to Answer 9 2.7.1.2 Strategies for making curriculum effective 13 2.7.1.3 Effective 27 2.7.1.4 Not Effective 14 2.7.2 Curriculum Revision Process 0 2.7.2.1 Sector Organization Involvement 0 2.7.2.1.1 Conduct Market/Industry Analysis 9 2.7.2.1.2 Presenting Curriculum at Stakeholders Meetings 22 2.7.2.1.3 Articulating established University Curriculum Process 17 2.7.2.1.4 Direct Sector Org Involvement 32 2.7.2.1.5Little/No/unclear Engagement w Sector Orgs on Curricula 41 2.7.2.2 Partner University Involvement 54 2.7.2.3 How Inputs/Feedback Addressed 31 2.7.3 Faculty Capacity to Deliver and Revise Curriculum 0 2.7.3.1 Role of US Partner in Capacity Development 45 2.7.3.2 Evaluation System 3 2.7.3.3 Role in Revising Curriculum 25 2.7.3.4 Adequate Capacity and/or Improvement Shown 42 2.7.3.5 Weak Capacity and/or Little Improvement Shown 37 2.7.4 Challenges and Solutions in Curricula 0 2.7.4.1 Challenges in coping with partner uni 5 2.7.4.2 Lack of relevant Faculty 7 2.7.4.3 Sector issues/needs 11 2.7.4.4 Missing Components/Courses in Curriculum/Not Practical 47 2.7.4.5 Lack of Hands-on-Training 4 2.7.4.5 Shifting Faculty Mindset Regarding Curricula Development 4 2.7.4.6 Solutions/Suggestions 32 2.8 Graduate Training for Employment 0 2.8.1 CAS Support to Students 0 2.8.1.1 To get Internships 0 2.8.1.1.1 Student Motivation for 6 2.8.1.1.2 Including in Sector MOUs/Linkage with sector 30 2.8.1.1.3 Linking to Learning/ Curriculum 10 2.8.1.1.4 Relying on Center/ ORIC/ University 19 2.8.1.1.5 Liaison Officer/Career Center Role 16 2.8.1.1.6 Others 4 2.8.1.2 To get Post-Graduation Employment 0 2.8.1.2.1 Linkage to Internship 6 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 153 Code System # 2.8.1.2.2 Focus on Academic/ PhD 18 2.8.1.2.3 Role of Sector Relationship in 16 2.8.1.2.4 Relying on Center/ORIC/ University 20 2.8.1.2.5 Too early to tell/ Can't comment 7 2.8.1.2.6 Entrepreneurship Initiatives 8 2.8.1.2.7 Other 9 2.8.2 Feedback from Sector Orgs on Student Skills 45 2.8.2.1 Satisfaction of employers with recruited CAS graduates 0 2.8.2.1.1 Yes 13 2.8.2.1.2 No 1 2.8.2.1.3 No Response/Unable to Answer 8 2.8.3 CAS Impact on Female Student Employment Opportunities 49 2.8.4 Challenges 0 2.8.4.1 Internships not useful 8 2.8.4.2 Experienced asked for job placement 3 2.8.4.3 Handled at Program Level (not at CAS level) 2 2.8.4.4 Low Student Focus on Internships/Job search 6 2.8.4.5 Degree scope 33 2.8.4.6 Lack of Explicit System for Internship/ Job Placement 31 2.8.4.7 Minimum CAS Effort/ lack of support 35 2.8.4.8 Others 17 2.8.4.9 Solutions/Suggestions 16 2.9 Gender, Disadvantaged 0 2.9.1 US Partner Impact on 4 2.9.2 Disadvantaged Groups Equity 2 2.9.2.1 Perceptions On 5 2.9.2.2 Policies and Actions 2 2.9.3 Gender Equity 0 2.9.3.1 Perceptions On 0 2.9.3.1.1 No Female in Decision Making role 33 2.9.3.1.2 Active women in decision-making role 23 2.9.3.1.3 CAS doesn't support women employment 8 2.9.3.1.4 CAS support for women employment 20 2.9.3.1.5 Lab comfortable for female students 6 2.9.3.1.6 Equal treatment 35 2.9.3.1.7 Lack of female students 4 2.9.3.1.8 Lack of female faculty/admin 23 2.9.3.1.9 Trend and behavior change in energy 19 2.9.3.1.10 Engineering is male dominated field 15 2.9.3.1.11 Availability of female staff 19 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 154 Code System # 2.9.3.1.12 Organizational Environment and role 23 2.9.3.1.13 Cultural constraints 7 2.9.3.2 Policies and Actions 0 2.9.3.2.1 Quota for women 12 2.9.3.2.2 Gender policies 25 2.9.3.2.3 Merit Based selection (Students) 18 2.9.3.2.4 Merit based selection (Faculty) 6 2.9.3.2.5 Outreach Program for women 11 2.10 CAS Component Integration 0 2.10.1 Perceptions on Overall Integration of CAS Components 43 2.10.2 Integration Challenges and Solutions 10 2.10.2.1 Integration Challenges not resolved 17 3. Sustainability 0 3.1 Sustainability Plan 8 3.2 University Measures to Ensure CAS Financial Sustainability 7 3.2.1 Expenditure Management 1 3.2.1.1 Alternate financial arrangements (industry, govt., HEC) 10 3.2.1.2 Self- sufficiency measures 3 3.2.1.3 Financing through endowment fund 4 3.2.1.4 Financing from host university 4 3.2.2 CAS to exist after project 1 3.2.3.1 Yes 29 3.2.3.2 No 0 3.2.3.3 No Response/Unable to Answer 3 3.2.3 Alternate Funding Sources 0 3.2.3.1 Planned 0 Industry 9 Endowment Fund 5 Donor Funding 12 HEC 21 Host University 14 Government 7 Research grants 8 Generating funds through CAS resources 19 Collaboration with US partners 11 Alumni 6 3.2.3.2 Secured 0 Donor Funding 2 Other 1 HEC 2 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 155 Code System # Endowment fund 3 Government 2 Research grants 2 Generating funds through CAS resources 1 Industry 3 3.2.4 Revenue Generating Activities 0 3.2.4.1 Planned 6 Trainings / degrees 15 Research Funding 10 Commercialization of labs 30 Government 16 Non-Government 4 Private 13 3.2.4.2 Existing 2 Endowment Fund 4 Government 9 Non-Government 2 Private 7 3.3 CAS Components Most Valued by Sector 3 3.3.1 Student/Faculty Exchanges 1 3.3.2 Applied/Policy Research 19 3.3.3 Labs 7 3.3.4 Curriculum/Teaching 6 3.3.5 Graduates 3 3.3.6 Gender Issues 4 3.3.7 Components Willing to Support 8 3.4 CAS Components Most Difficult to Sustain 0 3.4.0 Potential Solutions/How to be Addressed 25 3.4.1 Exchange program 32 3.4.2 Applied and joint research 7 3.4.3 Scholarship program 4 3.4.4 All 1 3.4.5 Faculty retention 4 3.4.6 Labs 4 3.4.7 Curriculum reform 2 3.4.8 Others 3 3.5 CAS Governance Structures Supporting Sustainability 0 3.5.1 Sustainability Plan Exists 3 3.5.2 Measures being taken to support sustainability 7 3.5.3 Sustainability plan shared with stakeholders 3 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 156 Code System # Stakeholders unaware of any sustainability plan 4 3.6 Stakeholders Engaged in Sustainability Planning Process 0 3.6.1 US Partner University 36 3.6.2 Host University 30 3.6.3 HEC 52 3.6.4 Industry Partners 31 3.6.5 Other Stakeholders 15 3.7 Linkage between Research, Development & Patent Procurement 1 3.7.1 Yes/ Actual Examples 10 3.7.2 No Patent/ Minimal Effort 8 3.7.3 Actions to Support Linkage 26 3.7.4 Challenges/ Suggestions 6 3.8 Sharing Knowledge & Learning 0 3.8.1 How Knowledge is Shared 11 3.8.2 Examples across CAS University Departments (Internal) 12 3.8.3 Examples across CAS Universities (Project) 13 3.8.4 Examples across Non-CAS Universities (External) 18 3.8.5 Examples across Sector Organizations 14 3.8.6 Challenges to Knowledge Sharing and Learning 20 3.8.7 Suggestions to Improve Knowledge Sharing 17 3.9 CAS University Marketing Capacity as Center of Excellence 104 3.10 Alumni outreach- Issues and Recommendations 8 3.11 Challenges to Sustainability 0 3.11.1 Funding Issues 0 3.11.1.1 Lack of funds or Lack alternate funding sources 30 3.11.1.2 Lack of funds for scholarship 16 3.11.1.3 Lack of funds for salaries 10 3.11.1.4 Discontinuity of exchange program 9 3.11.1.5 Lack of capacity or resources to maintain labs 4 3.11.1.6 Challenge to get Funding 3 3.11.2 CAS Level Issues 0 3.11.2.1 Lack of action plan for sustainability 7 3.11.2.2 Lack of coordination with industry, partners, or stakeholders 20 3.11.3 Host university level issues 0 3.11.3.1 Role of host university 10 3.11.3.2 Inactivity on part of host university 7 3.11.3.3 Lack of autonomy / others 1 3.11.4 Employment Related Issues 0 3.11.4.1 Brain drainage or shortage of qualified personnel 6 3.11.4.2 Employment issues 10 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 157 Code System # 3.11.4.3 Uncertain future or lack of clarity 9 3.11.4.4 Lack of team (faculty) buy-in 2 3.11.5 Others 5 4. Governance 0 4.1 Effectiveness of CAS Governance Structures 0 4.1.1Effective Structures 11 4.1.2 Ineffective Structures 6 4.1.3 Uneven/Uncertain Effectiveness 11 4.1.4 Room for Improvement 11 4.2 Factors affecting Implementation of Governance 0 4.2.1 Enabling Environment 15 4.2.2 Leadership/Vision 21 4.2.3 Establishment of Support/Managerial Systems 18 4.2.4 Autonomy of CAS 30 4.2.5 Change Management Issues 11 4.2.6 Communication Issues 5 4.2.7 Collaborating, Learning, Adapting (CLA) Approaches 22 4.2.8 Stakeholder (Systems) Management (Internal) 35 4.3 Alignment of CAS Governance with Existing Uni Governance 0 4.3.1 Efforts to align with University’s existing structures 0 4.3.1.1 Yes 25 4.3.1.2 No 1 4.3.1.3 No Response/Unable to Answer 6 4.3.2 Examples of Efforts being made to Align 7 4.3.3 Strong Alignment 13 4.3.4 Weak Alignment 3 4.3.5 Uncertain or Uneven Alignment 8 4.3.6 Absorption into Existing Uni Systems instead of Alignment 21 4.3.7 Issues Affecting Alignment 14 4.3.8 Examples of Alignment 16 4.3.9 Planned Alignment (from Sustainability Plan) 2 4.4 Challenges Related to Governance 0 4.4.1 Weak or Ineffective Oversight Mechanisms 14 4.4.2 Difficult Enabling Environment 7 4.4.3 USAID Issues/Requirements Affecting Governance 8 4.4.4 CAS Leadership Issues 11 4.4.5 Weak CAS Managerial Capacity 20 4.4.6 Not Adapting Implementation Approach to Needs of Center 7 4.4.7 Weak Strategic Focus in Center 6 4.4.8 Weak Links to Sector 7 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 158 Code System # 4.4.9 Role of US Partner University in Governance 7 4.4.10 Systems Integration Challenges (CAS vs. Uni Systems) 16 4.4.11 Dominance of Existing Host University System/Policies 34 4.5 Women/Disadvantaged Groups in Decision Making Roles 10 4.5.1 Yes 10 4.5.1.1 Examples 12 4.5.2 No 37 4.5.2.1 Reasons why 19 5. Quotable 6 6. Case studies 1 7. Inductive code suggestions 11 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 159 ANNEX 9: LIST OF SECONDARY DOCUMENTS • Activity monitoring, evaluation, and learning (MEL) plans • Annual monitoring and evaluation (M&E) plans • Partners’ quarterly progress reports • Annual work plans • Sustainability plans • Performance indicator reference sheets (PIRS) • Partnership memoranda of understanding (MOUs) with sector organizations U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 160 ANNEX 10: DATA COLLECTION INSTRUMENTS Individual Interview Consent Form Introduction Thank you very much for meeting us today. Our names are: We represent the USAID/ Pakistan Centers for Advanced Study (CAS) midterm evaluation team. This independent midterm evaluation is being undertaken at the request of USAID by Management Systems International (MSI), an international consulting firm. USAID/Pakistan, in coordination with the Pakistan Higher Education Commission (HEC), is sponsoring three US￾Pakistan Centers for Advanced Studies (CAS) in energy (2 centers) and water (1 center) that are supported by two American universities. In conjunction with their U.S. partners, the centers are developing programs that produce graduates who are prepared to meet the needs of industry, government, and their communities through cutting edge knowledge and technologies. Project activities to support these objectives include: exchange programs, joint research initiatives focused on policy and/or applied research for sector organizations, building and equipping laboratories, developing/reforming curricula, promoting good university governance, and offering scholarships and support for graduate and post-graduate training programs. Key crosscutting themes of the project include gender equity and access for disadvantaged groups. To ensure that the project is achieving its objectives, USAID has requested that MSI conduct a performance evaluation of the CAS project. The midterm evaluation will assess the performance of each of the three CAS centers. Results will help improve the implementation over the remaining life of the overall project. The evaluation will assess the effectiveness of university partnerships; overall project management; and sustainability of results, including the role of governance structures. Confidentiality This interview is voluntary so you can choose not to participate if you prefer. With your permission, we would like to record this discussion so that we accurately capture your feedback and do not miss any important points. Please be assured that your responses will be kept confidential. In case we use quotations from this interview in our baseline report, you will not be identified by name, or official title, but in general terms as a student, faculty member, public/private enterprise representative, or higher education leader. Do you agree to let us interview you? CHECK Yes- O No – O Do you agree to let us to record our conversation to support our notes? CHECK Yes- O No – O Institution Date Sector Time Department Interviewer: Gender M/F Note Taker: Role Interview Code: Initials of Interviewer to Confirm Informed Consent has been Received/Declined: U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 161 Group Interview Consent Form Introduction Thank you very much for meeting us today. Our names are: We represent the USAID/ Pakistan Centers for Advanced Study (CAS) midterm evaluation team. This independent midterm evaluation is being undertaken at the request of USAID by Management Systems International (MSI), an international consulting firm. USAID/Pakistan, in coordination with the Pakistan Higher Education Commission (HEC), is sponsoring three US￾Pakistan Centers for Advanced Studies (CAS) in energy (2 centers) and water (1 center) that are supported by two American universities. In conjunction with their U.S. partners, the centers are developing programs that produce graduates who are prepared to meet the needs of industry, government, and their communities through cutting edge knowledge and technologies. Project activities to support these objectives include: exchange programs, joint research initiatives focused on policy and/or applied research for sector organizations, building and equipping laboratories, developing/reforming curricula, promoting good university governance, and offering scholarships and support for graduate and post-graduate training programs. Key crosscutting themes of the project include gender equity and access for disadvantaged groups. To ensure that the project is achieving its objectives, USAID has requested that MSI conduct a performance evaluation of the CAS project. The midterm evaluation will assess the performance of each of the three CAS centers. Results will help improve the implementation over the remaining life of the overall project. The evaluation will assess the effectiveness of university partnerships; overall project management; and sustainability of results, including the role of governance structures. Confidentiality This interview is voluntary so you can choose not to participate if you prefer. With your permission, we would like to record this discussion so that we accurately capture your feedback and do not miss any important points. Please be assured that your responses will be kept confidential. In case we use quotations from this interview in our baseline report, you will not be identified by name, or official title, but in general terms as a student, faculty member, public/private enterprise representative, or higher education leader. Do you agree to let us interview you? CHECK Yes- O No – O Do you agree to let us to record our conversation to support our notes? CHECK Yes- O No – O Respondent Code Degree Gender (M/F) R1 R2 R3 R4 Institution: Date: Sector: Time: Department: Interviewer: Note Taker: Group Interview Code: Initials of Interviewer to Confirm Informed Consent has been Received/Declined: U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 162 Group Interview Instrument for CAS Students QUESTIONS PROMPTS US Pakistani University Partnership 1. How has the CAS partnership benefitted your university? • What benefits, if any, do you anticipate that the partnership will provide in the future? • Can you give examples? Components - Exchange programs 2. If you have participated in an exchange program through CAS, how has it affected your skills? Prompt based upon respondent role • Applied Research • Problem Solving • Soft Skills (e.g., team work, communication, etc.) 3. What challenges, if any, did you/exchange scholars experience while participating in the exchange program? • How has CAS and/or the US Partner university addressed these challenges? 4. How effective would you say the selection process for the exchange programs is? • What criticisms, if any, have students or faculty members raised about the selection process? • How has CAS addressed these criticisms? 5. What measures have been taken to ensure gender equity and disadvantaged group inclusion in the exchange program? • Female Inclusion • Special Needs • Economically disadvantaged 6. This question is for those who did not participate Why did you not participate in the exchange program? Also probe cultural/ personal/ family issues that have emerged through students participating in exchanges. 7. What changes have you observed in (other) faculty and/or students who have participated in exchange programs? Prompt based upon different stakeholders • Applied Research skills • Problem Solving • Soft Skills • Lab Utilization Skills Components - Applied research 8. How is CAS applied research relevant to the needs of sector organizations (e.g. energy/water)? (Ask for specific examples) 9. How are CAS students involved in applied research focused on sector needs? 10. How has the U.S. partner university helped enhance your applied research capacity? (Ask for specific examples) 11. What challenges, if any, have you faced in conducting applied research? What solutions have been implemented to overcome these challenges? Components - Policy research 12. How has CAS research contributed to public policy making in the sector? What specific examples can you give of how CAS research has influenced policy? 13. What challenges have there been in developing CAS research that contributes to public policy making? U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 163 QUESTIONS PROMPTS Components - Lab utilization 14. How has use of lab facilities contributed to improvement in the following: • Skills needed by graduates to perform better at workplace • Practical teaching skills of faculty • Applied/policy research needs of sector organizations 15. How well trained are you to use lab facilities? How helpful are lab staff in equipment use? 16. What challenges, if any, have you faced in accessing and utilizing the lab facilities? How has CAS addressed any of these challenges? 17. How comfortable are women and special-needs students in using lab facilities? Are there female lab staff? Components - Curricula Reform and New Program Development 18. How effective are CAS curricula in preparing students to meet the ongoing needs of employers in sector? How relevant are the CAS Curricula to the skills in new employees sought by sector organizations? 19. What is your perspective on the teaching methodologies adopted by the faculty in delivering the revised curricula? Components - Graduate Training and Employment/Internships 20. How does CAS support students in securing: • Internships • Employment 21. What challenges, if any, have students faced in finding post graduate employment? How has CAS addressed these challenges in placing the students? 22. How has CAS enhanced employment opportunities for women, particularly in less traditional career paths? What role, if any, can sector organizations play in improving opportunities for females in this sector, particularly in roles traditionally filled by men? Components - Integration 23. How well do you think the different activities of CAS fit together to create value for sector organizations? e.g. curricula, labs, exchange, research opportunities, training, partnerships, etc. Governance 24. How has this CAS ensured the participation of women/disadvantaged groups in decision making roles? U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 164 Group Interview Instrument for CAS Alumni QUESTIONS PROMPTS US Pakistani University Partnership 1. How has the CAS partnership benefitted your university? • What benefits, if any, do you anticipate that the partnership will provide in the future? • Can you give examples? Components - Exchange programs 2. If you participated in an exchange program through CAS, how, if at all, did that affect your skills? (Prompt based upon respondent role) • Applied Research • Teaching (for faculty) • Problem Solving (students) • Are there any other areas? 3. What challenges, if any, did you/exchange scholars experience while participating in the exchange program? How, if at all, did CAS addressed these challenges? 4. How effective would you say the selection process for the exchange programs was? • What criticisms, if any, did students or faculty members raise about the selection process? • How, if at all, did CAS address these criticisms? 5. What measures were taken to ensure gender equity and disadvantaged group inclusion in the exchange program? • Female Inclusion • Special Needs • Economically disadvantaged 6. This question is for those who did not participate Why did you not participate in the exchange program? Also probe cultural/personal/family issues that have emerged through students participating in exchanges. 7. What changes did you observe in (other) faculty and/or students who participated in exchange programs? (Prompt based upon different stakeholders) • Applied Research skills (faculty and students) • Teaching (faculty) • Problem Solving (faculty and students) • Soft Skills (students) Components - Applied research 8. How, if at all, did the U.S. partner university help enhance your applied research capacity? (Ask for specific examples) 9. How, if at all, did your university engage sector partners in the applied research process? 10. What challenges did you face in conducting applied research? 11. What solutions, if any, were implemented to overcome those challenges? Components - Policy research 12. How, if at all, has CAS research contributed to public policy making in the sector? • What specific examples can you give of how CAS research has influenced policy? 13. What challenges have there been in developing CAS research that contributes to the public policy making? U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 165 QUESTIONS PROMPTS Components - Lab utilization 14. How, if at all, did use of lab facilities contribute to improving in the following: • Applied policy research needs of sector organizations • Skills needed by graduates to perform better at workplace • Practical teaching skills of faculty 15. How well trained are you to use lab facilities? • How helpful are lab staff in equipment use? 16. What challenges, if any, did you face in accessing and utilizing the lab facilities? • How did CAS address any of these challenges? 17. How comfortable are women and special-needs students in using lab facilities? • Are there female lab staff? Components - Curricula Reform and New Program Development 18. How effective was CAS curriculum in preparing you to meet current job needs? • How relevant was the CAS Curriculum to the skills sought by sector organizations in new employees? 19. How effectively has CAS involved sector organizations and/or U.S. university partners in revising and developing the curricula? 20. What is your perspective on the teaching methodologies adopted by the faculty in delivering the revised curricula? Components - Graduate Training and Employment/Internships 21. How does CAS support students in securing: • Internships • Employment 22. What challenges, if any, did you have in your efforts to find employment after graduation? • How, if at all, did CAS address those challenges? 23. How if at all, do you feel that CAS has changed students’ perceptions on career opportunities available to them? 24. How has CAS enhanced employment opportunities for women, particularly in less traditional career paths? Components - Integration 25. How well do you think the different activities of CAS fit together to create value for sector organizations? e.g. curricula, labs, exchange, research opportunities, training, partnerships, etc. Governance 26. How has this CAS ensured the participation of women/disadvantaged groups in decision making roles? U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 166 Interview Instrument for CAS Faculty QUESTIONS PROMPTS US Pakistani University Partnership 1. How has the CAS partnership with the American university benefited your university? • What benefits do you anticipate that the partnership will provide in the future? • Can you give examples? 2. What challenges, if any, has the CAS partnership encountered? How have challenges been addressed? Components - Exchange programs 3. If you have participated in an exchange program through CAS, how has it affected your skills? (Prompt based upon respondent role) • Applied Research • Teaching (for faculty) • Are there any other areas? 4. What challenges, if any, did you/exchange scholars experience while participating in the exchange program? How have challenges been addressed? 5. How effective would you say the selection process for the exchange programs is? • What criticisms, if any, have students or faculty members raised about the selection process? • How, if at all, has CAS addressed these criticisms? 6. What measures have been taken to ensure gender equity and disadvantaged group inclusion in the exchange program? • Female Inclusion • Special Needs • Economically disadvantaged 7. For those who have not participated Why did you not participate in the exchange program? Also probe cultural/ personal/ family issues that have emerged through students participating in exchanges. 8. What changes have you observed in (other) faculty and/or students who have participated in exchange programs? (Prompt based upon different stakeholders) • Applied Research skills (faculty and students) • Teaching (faculty) • Problem Solving (faculty and students) • Soft Skills (students) Components - Joint research 9. How has CAS affected the way your university engages international universities in joint research projects? 10. What challenges has your university faced in conducting joint research? What solutions have been implemented to overcome these challenges? 11. What is HEC doing to help facilitate more joint research projects? Components - Applied research 12. How relevant is CAS applied research to the needs of sector organizations (e.g. energy/water)? (Ask for specific examples) 13. How does your university engage sector partners in the applied research process? 14. How has the U.S. partner university helped enhance the applied research capacity of your university? (Ask for specific examples) 15. What challenges, if any, has your university faced in conducting applied research? What solutions have been implemented to overcome these challenges? U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 167 QUESTIONS PROMPTS Components - Policy research 16. How, if at all, has CAS research contributed to public policy making in the sector? • What specific examples can you give of how CAS research has influenced policy? 17. What challenges have there been in developing CAS research that contributes to public policy making? • How have challenges been addressed? Components - Public-private partnerships 18. What efforts has CAS made to establish formal partnerships with sector organizations: • Private/Commercial sector organizations • Not for profit/NGO • Startup organizations • Governmental organizations 19. What factors have contributed to successful partnerships through CAS? • What factors have inhibited the success of partnership development through CAS? Components - Lab utilization 20. How are CAS lab facilities being utilized? • Are they being used as effectively as they can? 21. What challenges have you faced in accessing and utilizing the lab facilities? • Are students able to make good use of labs? • How has CAS addressed any of these challenges? 22. How comfortable are women or special-needs students in using labs? • Are there female lab staff? Components - Curricula Reform and New Program Development 23. How effective are CAS curricula in preparing students to meet the ongoing needs of employers in the sector? 24. How has CAS involved sector organizations and/or U.S. university partners in revising and developing the curricula? • How has input from these groups been incorporated into curricula? 25. What training, if any, has faculty received to prepare them to teach the new curriculum? • 26. What challenges, if any, have you encountered in efforts to develop and implement CAS curricula (including faculty capacity to deliver)? • How have these challenges been addressed? Components - Graduate Training and Employment/Internships 27. How does CAS support students in securing: • Internships • Employment 28. What feedback have you received from sector organizations on skills of students coming from CAS? 29. What challenges, if any, have students reported in their efforts to find internships or post-graduation employment? • How has CAS addressed these challenges? U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 168 QUESTIONS PROMPTS 30. How has CAS enhanced employment opportunities for women, particularly in less traditional career paths? • What role, if any, can sector organizations play in improving opportunities for females in this sector, particularly in roles traditionally filled by men? Components – Integration 31. How well do you think the different activities of CAS fit together to create value for sector organizations? Refer respondent to the list of project objectives and components e.g. curricula, labs, exchange, research opportunities, training, partnerships, etc. 32. How, if at all, has CAS addressed any challenges that have emerged in integrating these activities? Sustainability 33. What measures has the university taken to ensure financial sustainability of CAS once donor funding ends? Probe alternate funding sources like: • University funds • Government • Private sector • Donors • Alumni support revenue generating activities are being identified and pursued? 34. Which components of CAS are the most valued to sector organization, either directly or indirectly? ● Exchange programs ● Lab utilization ● Curricula reform ● High quality teaching and research ● Gender equity 35. What particular components of CAS do you think will be the most challenging to sustain? Refer respondent to the list of project objectives and components How, if at all, is the university working to address any of these challenges? 36. How is CAS managing governance structures to ensure sustainability of outcomes once donor funding ends? How, if at all, has CAS engaged relevant stakeholders in its sustainability planning process? • U.S. Partner University? • Host University? • HEC? • Industry Partners? • Other stakeholders? 37. What actions has CAS taken to strengthen the linkages between research, development and patent registration? 38. How are knowledge and learning from CAS shared among relevant project stakeholders? • Across departments within each CAS university • Across all CAS universities • Across non-CAS universities U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 169 QUESTIONS PROMPTS 39. How would you improve the sharing of knowledge and learning? 40. How effectively is CAS contributing the university’s capacity to market itself as a centre of academic excellence in its respective sector? Governance 41. How effective are CAS governance structures in achieving overall project objectives? Policies, procedures, oversight committees 42. In your experience, what are the factors affecting implementation of governance structures (policies, procedures and oversight committees) in CAS? 43. What measures have been taken to overcome any challenges you have encountered? 44. How aligned are these structures with established university structures? 45. How has this CAS ensured the participation of women/disadvantaged groups in decision making roles? U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 170 Interview Instrument for CAS Administrator/USAID QUESTIONS PROMPTS U.S. Pakistani University Partnership 1. How has the CAS partnership with the American university benefited your university? • What benefits do you anticipate that the partnership will provide in the future? • Can you give examples? 2. What challenges has the CAS partnership encountered? How have challenges been addressed? Components – Exchange programs 3. Is exchange program valuable in the overall professional development of the awardees? [CIRCLE ONE NUMBER] If yes, could you please describe how? YES NO N/A 1 2 99 4. What suggestions do you have for improving any aspect of the exchange program (from selection through implementation and follow up)? 5. What measures have been taken to ensure gender equity and disadvantaged group inclusion in the exchange program? Components - Combined close ended section for Joint & Applied Research and PPPs/GDAs 6. Is the research conducted at this CAS in making significant contributions towards solving sector problems? [CIRCLE ONE NUMBER] If yes, could you please describe how? If not, why not? YES NO N/A 1 2 99 Components - Joint research 7. How has CAS affected the way your university engages international universities in joint research projects? 8. What challenges has your university faced in conducting joint research? What solutions have been implemented to overcome these challenges? Components - Applied research 9. How has the U.S. partner university helped enhance the applied research capacity of your university? (Ask for specific examples) 10. How does your university engage sector partners in the applied research process? 11. What challenges has your university faced in conducting applied research? What solutions have been implemented to overcome these challenges? U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 171 QUESTIONS PROMPTS Components - Policy Research 12. Does CAS research make significant contributions toward public policy making? [CIRCLE ONE NUMBER] If yes, how has CAS research contributed to public policy making in the sector? YES NO N/A 1 2 99 • What specific examples can you give of how CAS research has influenced policy? 13. How has CAS research contributed to public policy making in the sector? • What specific examples can you give of how CAS research has influenced policy? 14. What challenges have there been in developing CAS research that contributes to public policy making? How have challenges been addressed? Components - Public-private partnerships 15. What efforts has CAS made to establish formal partnerships with sector organizations: • Private/Commercial sector organizations • Not for profit/NGO • Startup organizations • Governmental organizations 16. What factors have contributed to successful partnerships through CAS? • What factors have inhibited the success of partnership development through CAS? Components - Lab utilization 17. How are lab facilities being utilized? Are lab facilities being used as effectively as they can? 18. What challenges have key stakeholders faced in accessing and utilizing the lab facilities? Stakeholders: o Faculty o Students o Partner Organizations • How has CAS addressed any of these challenges? 19. How comfortable are women or special-needs students in using labs? • Are there female lab staff? Components - Curricula Reform and New Program Development 20. How effective are CAS curricula in preparing students to meet the ongoing needs of employers in sector? 21. How has CAS involved sector organizations and/or U.S. university partners in revising and developing the curricula? • How, specifically, has input from these groups been incorporated into curricula? 22. What training, if any, has faculty received to prepare them to teach the new curriculum? U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 172 QUESTIONS PROMPTS 23. What challenges have you encountered in efforts to develop and implement CAS curricula (including faculty capacity to deliver)? • How have these challenges been addressed? Components - Graduate Training and Employment/Internships 24. Are employers satisfied with the quality of recruited CAS graduates? [CIRCLE ONE NUMBER] Could you please share some examples? YES NO N/A 1 2 99 25. How does CAS support students in securing: • Internships • Employment 26. What feedback have you received from sector organizations on skills of students coming from CAS? 27. What challenges have students reported in their efforts to find internships or post-graduation employment? • How has CAS addressed these challenges? 28. What examples can you provide that illustrate how CAS has had an impact on the career success of its graduates? 29. How has CAS enhanced employment opportunities for women, particularly in less traditional career paths? • What role, if any, can sector organizations play in improving opportunities for females in this sector, particularly in roles traditionally filled by men? Components - Integration 30. How well do you think the different activities of CAS fit together to create value for sector organizations? Refer respondent to the list of project objectives and components e.g. curricula, labs, exchange, research opportunities, training, partnerships, etc. 31. How has CAS addressed any challenges that have emerged in integrating these activities? Sustainability 32. Will CAS continue to exist after the current donor support ends? [CIRCLE ONE NUMBER] YES NO N/A 1 2 99 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 173 QUESTIONS PROMPTS 33. What measures has the university taken to ensure financial sustainability of CAS once donor funding ends? Probe alternate funding sources like: • University funds • Government • Private sector • Donors • Alumni support Revenue generating activities are being identified and pursued? 34. Which components of CAS are the most valued to sector organization, either directly or indirectly? ● Exchange programs ● Lab utilization ● Curricula reform ● High quality teaching and research ● Gender equity 35. What particular components of CAS do you think will be most challenging to sustain? Refer respondent to the list of project objectives and components How is the university working to address any of these challenges? 36. How is CAS managing governance structures to ensure sustainability of outcomes once donor funding ends? How has CAS engaged relevant stakeholders in its sustainability planning process? • Host University? • HEC? • Industry Partners? • Other stakeholders? 37. What actions has CAS taken to strengthen the linkages between research, development and patent registration? 38. How are knowledge and learning from CAS shared among relevant project stakeholders? • Across departments within each CAS university • Across all CAS universities • Across non-CAS universities What can HEC do to ensure that relevant knowledge and learning from CAS universities is shared across Non-CAS universities in Pakistan? 39. How would you improve the sharing of knowledge and learning? 40. How effectively is CAS contributing the university’s capacity to market itself as a centre of academic excellence in its respective sector? Governance 41. How effective are CAS governance structures in achieving overall project objectives? Governance Structures: Policies, procedures, oversight committees U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 174 QUESTIONS PROMPTS 42. What are the factors affecting implementation of governance structures (policies, procedures and oversight committees) in CAS? 43. Is Pakistani host university making efforts to align CAS Governance structures with university's existing governance structures? [CIRCLE ONE NUMBER] YES NO N/A 1 2 99 44. How aligned are these structures with established university structures? Examples of Structures: • Operational and academic autonomy, including autonomous policy development? • Employment and management of academic staff, including salary and incentives? • Revenue/fund generation? • Collaboration with industry/experts? 45. How has this CAS ensured the participation of women/disadvantaged groups in decision making roles? U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 175 Interview Instrument for CAS Lab Personnel QUESTIONS PROMPTS Components - Lab utilization 1. Have CAS labs contributed to the improvement of research quality? [CIRCLE ONE NUMBER] If yes, how? If no, why not? Could you please share some examples? YES NO N/A 1 2 99 2. How are CAS lab facilities being used? • Are they being used as effectively as they can? • Are students able to make good use of lab? • How frequently are lab facilities being used by: o Students o Faculty o Sector Organizations o Non-CAS universities 3. What challenges have students and faculty faced in accessing and utilizing the lab facilities? • How has CAS addressed any of these challenges? 4. How comfortable are women or special-needs students in using labs? • Are there female staff? U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 176 Interview Instrument for CAS University Leadership QUESTIONS PROMPTS U.S. Pakistani University Partnership 1. Are you satisfied with your university’s CAS partnership with its U.S. CAS partner university? [CIRCLE ONE NUMBER] If yes, how has the CAS partnership benefited your university? If not, why not? YES NO N/A 1 2 99 • What benefits, if any, do you anticipate that the partnership will provide in the future? • Can you give examples? 2. What challenges, if any, has the CAS partnership created? • How has university addressed any challenges? Components - Exchange programs 3. Are exchange programs valuable in the overall professional development of the awardees? [CIRCLE ONE NUMBER] If yes, how, if not, why not? YES NO N/A 1 2 99 4. What changes have you observed in faculty and/or students who have participated in exchange programs? (Prompt based upon different stakeholders) • Applied Research skills (faculty and students) • Teaching (faculty) • Problem Solving (faculty and students) • Soft Skills (students) Components - Applied Research 5. Is the research conducted at this CAS in making significant contributions towards solving sector problems? [CIRCLE ONE NUMBER] If yes, any examples of such contributions? YES NO N/A 1 2 99 6. Does CAS have an effective process in place to conduct applied research that meets needs of the sector? [CIRCLE ONE NUMBER] If yes, what processes are in place to ensure CAS applied research addresses the needs of sector organizations (e.g. energy/water)? If no, why not? YES NO N/A 1 2 99 (Ask for specific examples) 7. How has the U.S. partner university helped enhance the applied research capacity of your university? (Ask for specific examples) Components - Policy research 8. How has CAS research contributed to public policy making in the sector? • What specific examples can you give of how CAS research has influenced policy? • What challenges have there been in developing CAS research that contributes to the public policy making? U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 177 QUESTIONS PROMPTS Components - Joint research with international research institutions 9. How has CAS affected the way your university engages international universities in joint research projects? 10. What challenges has your university faced in conducting joint research? • What solutions have been implemented to overcome these challenges? 11. What is HEC doing to help facilitate more joint research projects? Components - Public-private partnerships 12. What efforts has CAS made to establish formal partnerships with sector organizations: • Private/Commercial sector organizations • Not for profit/NGO • Startup organizations • Governmental organizations 13. What factors, if any, have contributed to successful partnerships through CAS? • What factors have inhibited the success of partnership development through CAS? Components - Lab utilization 14. Have CAS labs contributed to the improvement of research quality? [CIRCLE ONE NUMBER] If yes, how? If not, why not? YES NO N/A 1 2 99 15. How are CAS lab facilities being utilized? • Are they being used as effectively as they can? 16. What challenges have you faced in accessing and utilizing the lab facilities? • Are students able to make good use of labs? • How has CAS addressed any of these challenges? 17. How comfortable are women or special-needs students in using labs? • Are there female lab staff? Components - Curricula Reform and New Program Development 18. Are employers satisfied with the quality of CAS students’ training (with respect to their preparedness for work)? [CIRCLE ONE NUMBER] If yes, give some examples? YES NO N/A 1 2 99 19. How effective are CAS curricula in preparing students to meet the ongoing needs of employers in the sector? 20. How has CAS involved sector organizations and/or U.S. university partners in revising and developing the curricula? • How has input from these groups been incorporated into curricula? 21. What challenges, if any, have you encountered in efforts to develop and implement CAS curricula (including faculty capacity to deliver)? • How have these challenges been addressed? U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 178 QUESTIONS PROMPTS Components - Graduate Training and Employment/Internships 22. What systems exist in CAS to support students in securing employment and internships? What examples can you provide that illustrate how CAS has had an impact on the career success of its graduates? 23. What feedback have you received from sector organizations on skills of students coming from CAS? 24. What challenges, if any, have students reported in their efforts to find internships or post-graduation employment? • How has CAS addressed these challenges? 25. How has CAS enhanced employment opportunities for women, particularly in less traditional career paths? • What role, if any, can sector organizations play in improving opportunities for females in this sector, particularly in roles traditionally filled by men? Components - Integration 26. How well do you think the different activities of CAS fit together to create value for sector organizations? Refer respondent to the list of project objectives and components e.g. curricula, labs, exchange, research opportunities, training, partnerships, etc. 27. How has CAS addressed any challenges that have emerged in integrating these activities? Sustainability 28. Will CAS continue to exist after the current donor support ends? [CIRCLE ONE NUMBER] YES NO N/A 1 2 99 29. How is CAS managing governance structures to ensure sustainability of outcomes once donor funding ends? How has CAS engaged relevant stakeholders in its sustainability planning process? • U.S. Partner University? • Host University? • HEC? • Industry Partners? • Other stakeholders? 30. What measures, if any, has the university taken to ensure financial sustainability of CAS once donor funding ends? What alternate funding sources like: • University funds, • government, • private sector, • donors, • alumni support, revenue generating activities are being identified and pursued? 31. What particular components of CAS do you think will be the most challenging to sustain? Refer respondent to the list of project objectives and components How is the university working to address any of these challenges? U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 179 QUESTIONS PROMPTS 32. What actions, if any, has CAS taken to strengthen the linkages between research, development and patent registration? 33. How effectively is CAS contributing the university’s capacity to market itself as a centre of academic excellence in its respective sector? 34. What can HEC do to ensure that relevant knowledge and learning from CAS universities is shared across Non-CAS universities in Pakistan? Governance 35. Is the Pakistani host university making efforts to align CAS governance structures with university's existing governance structures? [CIRCLE ONE NUMBER] If yes, how would you assess the university’s efforts to align CAS governance structures with the university’s existing structures with regard to the following areas? YES NO N/A 1 2 99 • Operational and academic autonomy, including autonomous policy development? • Employment and management of academic staff, including salary and incentives? • Revenue/fund generation? • Collaboration with industry/experts? 36. To what extent is the University willing to take over financial responsibility for CAS and its components once donor funding ends? • What mechanisms is the university putting in place to facilitate the transfer of financial responsibility of CAS and its components? 37. What components of CAS is the university most likely to retain? • What components of CAS, if any, is the university likely to discontinue or change? Why? 38. How has the university ensured the participation of women and disadvantaged groups in CAS Governance structures? 39. What efforts has the university made to ensure the participation of women and disadvantaged groups in CAS decision making roles? U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 180 Interview Instrument for U.S. Partner University QUESTIONS PROMPTS US Pakistani University Partnership 1. Are you satisfied with your university’s partnership with the CAS partner university? [CIRCLE ONE NUMBER] If yes, how? If not, why? YES NO N/A 1 2 99 2. How has the CAS partnership benefited your university? 3. How has the CAS partnership benefited the Pakistani Host university? 4. What challenges has the CAS partnership encountered? • How has your university addressed any current or anticipated challenges? Components - Exchange programs 5. Does the Pakistani CAS university adequately prepare exchange scholars before the start of the exchange program? [CIRCLE ONE NUMBER] YES NO N/A 1 2 99 6. Does the U.S. Host university support exchange scholars in adjusting to the new environment during the exchange program? [CIRCLE ONE NUMBER] YES NO N/A 1 2 99 7. What challenges, if any, did exchange scholars experience while participating in the exchange program? • What measures were taken to help overcome these challenges? o By U.S. CAS Partner o By Pakistan CAS Partner 8. What changes have you observed in faculty and/or students who have participated in exchange programs? • Applied Research skills (faculty and students) • Teaching (faculty) • Problem Solving (faculty and students) • Soft Skills (students) 9. What suggestions do you have for improving any aspect of the exchange program (from selection through implementation and follow up)? • Have you shared any feedback or suggestions, if there were any, on the selection process? • Have your suggestions been considered while improving the processes? 10. What measures have been taken to ensure gender equity and disadvantaged groups inclusion in the exchange program? • Female Inclusion • Special Needs • Economically disadvantaged Components - Applied research 11. How is CAS applied research relevant to the needs of sector organizations (e.g. energy/water)? • Pakistani sector organizations • International sector organizations 12. How has the U.S. partner university helped enhance the applied research capacity of Pakistani CAS partner university? (Ask for specific examples) 13. How has the U.S. CAS partner engaged sector partners in the applied research process? U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 181 QUESTIONS PROMPTS 14. What challenges has your university faced in supporting applied research in the Pakistani CAS partner university? • What solutions have been implemented to overcome these challenges? Components – Joint research 15. How has the U.S. partner university engaged with its Pakistani CAS partner in joint research projects? 16. What challenges your university faced in conducting joint research with it’s Pakistani partner? 17. What corrective measures were taken to overcome these challenges? Components - Public-private partnerships 18. What efforts has the Pakistani CAS partner made to establish partnerships in its sector of focus? • Private/Commercial sector organizations • Not for profit/NGO • Startup organizations • Governmental organizations 19. What factors, if any, have contributed to successful partnerships through CAS? • What factors have inhibited the success of partnership development through CAS? Components - Lab utilization 20. How are CAS lab facilities being utilized? • Are they being used as effectively as they can? 21. What challenges have you faced in accessing and utilizing the lab facilities? • Are students able to make good use of labs? • How has CAS addressed any of these challenges? Components - Curricula Reform and New Program Development 22. Is your University satisfied with the quality of CAS students’ training (with respect to their preparedness for work)? [CIRCLE ONE NUMBER] If yes, how? If no, why not? YES NO N/A 1 2 99 23. How effective are CAS curricula in preparing students to meet the ongoing needs of employers in the sector? 24. How has CAS involved the U.S. university partner in revising and developing the curricula? • How have inputs of U.S. university partner been incorporated into curricula? 25. How has CAS involved the U.S. university partner in developing faculty teaching capacity for the revised curriculum? 26. What training, if any, has faculty received to prepare them to teach the new curriculum? 27. How has CAS involved sector organizations/clients in revising and developing the curricula? • How have inputs from these various groups been incorporated into curricula? 28. What challenges, if any, have you encountered in efforts to help develop and implement CAS curricula? • How have these challenges been addressed? Components - Graduate Training and Employment/Internships 29. How does CAS support students in securing: • Internships • Employment U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 182 QUESTIONS PROMPTS 30. What challenges, if any, have students reported in their efforts to find internships or post￾graduation employment? • How has CAS addressed these challenges? 31. How has CAS affected students perceptions on career opportunities available to them? 32. How has CAS enhanced employment opportunities for women, particularly in less traditional career paths? Components - Integration 33. How well do you think the different activities of CAS fit together to create value for sector organizations? Refer respondent to the list of project objectives and components e.g. curricula, labs, exchange, research opportunities, training, partnerships, etc. 34. How has CAS addressed any challenges that have emerged in integrating these activities? Sustainability 35. Will CAS continue to exist after the current donor support ends? [CIRCLE ONE NUMBER] YES NO N/A 1 2 99 36. How is CAS managing governance structures to ensure sustainability of outcomes once donor funding ends? How has CAS engaged relevant stakeholders in its sustainability planning process? • Host University? • HEC? • Industry Partners? • Other stakeholders? 37. What measures, if any, has the Pakistani Host university taken to ensure financial sustainability of CAS once donor funding ends? 38. What specific efforts is your university is taking to ensure sustainability of CAS outcomes once funding ends? 39. What particular components of CAS do you think will be the most challenging to sustain? Refer respondent to the list of project objectives and components How is the university working to address any of these challenges? 40. Are there any specific components that your university is likely to support once donor funding ends? Technical or financial support for • Support in curricular reform • Supporting high quality teaching/research through endowed professorships • Scholarships for exchange programs • Funding for lab resources • Promoting gender equity 41. What actions, if any, has CAS taken to strengthen the linkages between research, development and patent registration? How has U.S. CAS partner supported this process? U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 183 QUESTIONS PROMPTS 42. How are knowledge and learning from CAS shared among relevant project stakeholders? • Across departments within each CAS university • Across all Pakistani CAS universities • Across different Pakistani non-CAS universities • Across international universities 43. How would you improve the sharing of knowledge and learning? 44. How effectively are efforts of CAS partners contributing the Pakistani CAS partner university’s capacity to market itself as a centre of academic excellence in its respective sector? Governance 45. How effectively do you feel the CAS governance structures (policies, procedures and oversight committees) implemented by Pakistani Host university are achieving overall project objectives? • In your experience, what are the factors affecting implementation of governance structures (policies, procedures and oversight committees) in CAS? • What measures have been taken to overcome any challenges you have encountered? 46. How has this CAS ensured the participation of women/disadvantaged groups in decision making roles? 47. Is the Pakistani host university making efforts to align CAS governance structures with university's existing governance structures? [CIRCLE ONE NUMBER] YES NO N/A 1 2 99 48. How would you assess the Pakistan host university’s efforts to align CAS governance structures with its existing structures? • Operational and academic autonomy, including autonomous policy development? • Employment and management of academic staff, including salary and incentives? • Revenue/fund generation? • Collaboration with industry/experts? 49. What challenges, if any, do you anticipate in aligning CAS governance structures to its Pakistani host university governance structures to sustain project results as CAS transitions from donor funding? U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 184 Interview Instrument for Higher Education Commission QUESTIONS PROMPTS U.S. Pakistani University Partnership 1. How have the CAS partnerships between American and Pakistani universities under CAS benefited Pakistani universities? 2. What challenges, if any, have the CAS partnerships encountered? How, if any, have these challenges been addressed? Components - Exchange programs 3. Are exchange programs valuable in the overall professional development of the awardees? [CIRCLE ONE NUMBER] If yes, how? If not, why not? YES NO N/A 1 2 99 Components - Joint research 4. How has CAS affected the way universities engage international universities in joint research projects? Which examples, if any, demonstrate this best? 5. What efforts have CAS universities made to engage other Pakistani Universities/research institutions in joint research projects? (Ask for specific examples). 6. What is HEC doing to help facilitate more joint research projects? Components - Policy research 7. Has research from CAS universities made significant contributions toward public policy development? [CIRCLE ONE NUMBER] If yes, how has CAS research contributed to public policy making in the sector? If not, why not? YES NO N/A 1 2 99 What specific examples can you give of how CAS research has influenced policy? 8. What challenges have there been in influencing policy development through CAS research? What is HEC doing to address these challenges? Components – Applied research 9. What is the HEC definition of Applied Research? Ask • Where it is published • How we can get copy of definition 10. How effective have CAS universities been in conducting applied research in this way? Why, Why not? 11. How does the applied research conducted by CAS meet the needs of sector organizations? • What, if anything, can CAS/HEC do to strengthen these efforts? U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 185 QUESTIONS PROMPTS Components - Public-private partnerships 12. What efforts have CAS universities made to establish partnerships with sector organizations: • Private/Commercial sector organizations • Not for profit/NGO • Startup organizations • Governmental organizations • What role can HEC play to enhance the development of partnerships with sector organizations? 13. What factors, if any, have contributed to successful partnerships through CAS? • What factors, if any, have inhibited the success of partnership development through CAS? Components - Curricula Reform and New Program Development 14. How effective have CAS curricula been preparing students to meet the ongoing needs of employers in sector? 15. What has HEC learned through CAS efforts to develop curricula that are aligned to the needs of sector organizations? 16. What challenges, if any, are you aware of regarding initiating new degree programs? 17. What has HEC learned from the way CAS has developed new programs? Components – Integration 18. How well do you think the different activities of CAS fit together to create value for: • Universities • Sector Organizations Refer respondent to the list of project objectives and components e.g. curricula, labs, exchange, research opportunities, training, partnerships, etc. 19. How, if at all, has CAS addressed any challenges that have emerged in integrating these activities? • What role has HEC played in addressing these challenges? Sustainability 20. How likely is CAS to continue to exist at the universities after the current USAID support ends? • Which components of CAS are most likely to be sustained once donor support ends? • Why are the others not likely to be sustained? 21. What specific efforts is HEC taking to ensure sustainability of CAS outcomes once funding ends? • Are there any specific components that HEC is likely to support financially once donor funding ends? 22. What actions, if any, have CAS universities taken to strengthen the linkages between research, development and patent registration? • How has HEC supported this process? U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 186 QUESTIONS PROMPTS 23. How effectively is learning and knowledge generated from the project be shared across universities (especially across Non-CAS universities)? What can HEC do to ensure that relevant knowledge and learning from CAS universities is shared across Non-CAS universities in Pakistan? 24. How effectively is CAS contributing the universities’ capacities to market themselves as Centres of Academic Excellence in their respective sectors? Governance 25. How effective are CAS governance structures in achieving overall project objectives? 26. What factors determine successful implementation of CAS governance structures (policies, procedures and oversight committees) within existing universities structures? 27. What measures has HEC taken to help overcome any challenges universities have encountered? 28. How would you assess the universities’ efforts to align CAS governance structures with existing structures? • Operational and academic autonomy, including autonomous policy development? • Employment and management of academic staff, including salaries and incentives? • Revenue/funds generation? • Collaboration with industry/experts? U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 187 Interview Instrument for Sector Organization QUESTIONS PROMPTS Components - Applied research 1. Is your organization satisfied with the quality of applied research from CAS? [CIRCLE ONE NUMBER] If yes, how? Please give some examples. If no, why not? YES NO N/A 1 2 99 2. Does CAS have an effective process in place to conduct applied research that meets the needs of your sector? [CIRCLE ONE NUMBER] If yes, how? Please give some examples. If no, why not? YES NO N/A 1 2 99 3. How, if at all, is CAS applied research relevant to the needs of sector organizations (e.g. energy/water)? 4. What challenges, if any, has industry faced while collaborating with CAS around applied research? Components - Policy research 5. How, if at all, has CAS research contributed to public policy making in your sector? ● What specific examples can you give of how CAS research has influenced policy? 6. What challenges have there been in developing CAS research that contributes to public policy making? Components - Public-private partnerships 7. Based upon your understanding, what efforts has CAS made to establish partnerships in your sector of focus? 8. What value, if any, does your organization see in partnering with CAS? 9. What factors, if any, have contributed to successful private partnerships through CAS? ● What factors have inhibited the success of public-private partnership development through CAS? Components - Lab utilization 10. How accessible, if at all, are CAS lab facilities and services in helping your organization address its research and development needs? 11. How effective are lab facilities and services in helping graduates develop the skills needed to perform better at workplace? U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 188 QUESTIONS PROMPTS Components - Curricula reform and development 12. Have you seen the CAS curricula? If yes, how relevant are CAS Curricula your organization seek in new employees? ● What systems are in place to ensure the ongoing relevance of the curricula to evolving sector needs (including international standards)? 13. How, if at all, has CAS involved your organization in the development or revision of curricula? ● What specific suggestions has your organization made regarding curricula? ● How, if at all, have these suggestions been addressed by CAS? Components - Graduate Training and Employment/Internships 14. Is your organization satisfied with the quality of recruited CAS graduates? [CIRCLE ONE NUMBER] YES NO N/A 1 2 99 15. What examples can you provide to illustrate the quality of the CAS graduates? ● How if at all, do you feel that CAS has changed students’ perspectives on career options available to them? 16. How has CAS enhanced employment opportunities for women, particularly in less traditional career paths? ● What role, if any, can sector organizations play in improving opportunities for females in this sector, particularly in roles traditionally filled by men? Sustainability 17. Which components of CAS are the most valued to your organization, either directly or indirectly? ● Exchange programs ● Lab utilization ● Curricula reform ● High quality teaching and research ● Gender equity 18. What components of CAS would your organization be willing to support in order to sustain the benefits to the sector? ● Scholarships for exchange programs ● Funding for lab resources ● Support in curricular reform ● Supporting high quality teaching/research through endowed professorships ● Opportunities for internships to enhance graduate training ● Promoting gender equity 19. How, if at all, will learning and knowledge generated from the project be shared among relevant project stakeholders? 20. How, if at all, would your organization improve the sharing of knowledge and learning? Governance 21. How, in your knowledge, have universities ensured the participation of women and disadvantaged groups in CAS Governance structures? U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 189 QUESTIONS PROMPTS 22. Has your organization been given the opportunity to participate in any CAS governance structures (such as oversight committees)? ● How have you participated? ● How has your feedback been used? 23. Do you feel that CAS governance structures sufficiently incorporate sector organizations’ views related to critical programmatic areas? U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 190 Survey Questionnaire for Faculty Serial number (FOR OFFICE USE ONLY) Start Time: hh:mm (AM/PM): Interviewer ID: End Time: hh:mm (AM/PM): Interview Date: DD/MM/YY Introduction: USAID/Pakistan, in coordination with the Pakistan Higher Education Commission (HEC) is supporting three U.S.- Pakistan Centers for Advanced Studies (CAS) in energy (2 centers) and water (1 center) that are supported by two American universities. In conjunction with their U.S. partners, the centers are developing programs that produce graduates who are prepared to meet the needs of industry, government, and their communities through cutting edge knowledge and technologies. Project activities to support these objectives include: exchange programs and joint and applied research with U.S. partner universities, building and equipping laboratories, developing/reforming curricula, improving university governance, and scholarships to support graduate and post￾graduate training programs. USAID has requested that Management Systems International (MSI) conduct an evaluation of the CAS project. Results will help improve implementation during the remaining project duration. The evaluation will assess the effectiveness of university partnerships; overall project management; and sustainability of the results, including the role of governance structures. As part of this evaluation to gauge the effectiveness of the CAS project, we are conducting a survey of CAS faculty. Since you are a CAS faculty member, we would appreciate your participation in this survey. This survey will take approximately 30-40 minutes. We will treat everything you say here confidentially. We will not use your names in reports or attribute anything you say to you. Module A: Background A1. Respondent name Refused to answer 99 A2. Gender Male 0 Female 1 A3. Which CAS are you associated with? NUST (Energy) 1 UET, Peshawar (Energy) 2 MUET, Jamshoro (Water) 3 A4. What is the highest degree you have received? M.S. 1 Ph.D. 2 A5. What is your designation Assistant Professor 1 Associate Professor 2 Professor 3 Other 4 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 191 A6. From where did you obtain your highest degree? North America or UK/Ireland 1 Continental Europe (excluding UK) 2 East Asia 3 New Zealand or Australia 4 Pakistan 5 Other 6 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 192 NOTE FOR TELEPHONIC INTERVIEWS: ➢ Enumerators should read every question along with responses aloud, unless instructed otherwise. Module B1: Partnership with US Universities and Exchange program Q1. Did you apply for a CAS exchange program? (CIRCLE ONE NUMBER) Yes No 1 0 GO TO Q11 Q2. Is the selection process for the CAS exchange program transparent? (CIRCLE ONE NUMBER) Yes No Don’t know Refused to Answer 1 0 98 99 Q3. Were you selected for a CAS exchange program? (CIRCLE ONE NUMBER) Yes No 1 0 GO TO Q11 Q4. Did you participate in the exchange program? (CIRCLE ONE NUMBER) Yes No 1 0 GO TO Q11 To what extent do you agree or disagree with the following statements? (CIRCLE ONE NUMBER) Question # Statement Strongly Agree Agree Disagree Strongly Disagree Q5. Your Pakistani CAS university adequately prepared you before the start of the exchange program. 4 3 2 1 Q6. The American host university adequately supported you in adjusting to the new environment during the exchange program. 4 3 2 1 Q7. Did the exchange program improve your applied research skills? (CIRCLE ONE NUMBER) (Applied research is research that meets the explicit needs of sector organizations) Yes No 1 0 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 193 Q8. Did the exchange program improve your applied research output? (CIRCLE ONE NUMBER) Yes No 1 0 Q9. Did the exchange program improve your teaching skills? (CIRCLE ONE NUMBER) Yes No 1 0 Q10. Did the exchange program improve your proposal writing skills? (CIRCLE ONE NUMBER) Yes No 1 0 Q11. To what extent are you satisfied/dissatisfied overall with your university’s partnership with its US CAS partner? (CIRCLE ONE NUMBER) Extremely Satisfied Satisfied Dissatisfied Extremely Dissatisfied 4 3 2 1 Module B2: Joint and Applied Research and Public-Private Partnerships Q12. Have you engaged in applied research since you have joined CAS? (CIRCLE ONE NUMBER) Yes No 1 0 GO TO Q. 14 To what extent do you agree or disagree with the following statements? (CIRCLE ONE NUMBER) Question # Statement Strongly Agree Agree Disagree Strongly Disagree Q13. The applied research you are conducting is relevant to the needs of sector organizations. 4 3 2 1 Q14. CAS applied research topics are aligned with the explicit needs of the organizations in our sector of focus (i.e., energy/water). 4 3 2 1 Q15. Research led by women scholars is accepted by organizations in the sector. 4 3 2 1 Q16. Sector organizations prefer males as compared to females in lead research roles. 4 3 2 1 Q17. At this CAS, women are involved in conducting research in your in the sector of focus (i.e., energy/water). 4 3 2 1 Q18. At this CAS, women are provided opportunities to take a lead role in conducting research in the sector of focus (i.e., energy/ water). 4 3 2 1 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 194 NOTE: CIRCLE ONE NUMBER Question # To what extent are you satisfied/dissatisfied with: Extremely Satisfied Satisfied Dissatisfied Extremely Dissatisfied Q19. The participation of sector organizations in conducting applied research? 4 3 2 1 Q20. The process used to select the applied research areas/topics. 4 3 2 1 Q21. The financial support you received from CAS to conduct applied research? 4 3 2 1 Q22. The technical support you received from the US university to conduct applied research? Technical support refers to such areas as research design, analytical tools, publishing support, guidance on use of lab/lab equipment, etc. 4 3 2 1 Q23. The level of autonomy you have to spend research money (grants) you receive. 4 3 2 1 Module B3: Lab Utilization at Pakistani CAS University Q24. Are CAS-supported laboratories at your university operational? (CIRCLE ONE NUMBER) Yes No 1 0 GO TO MODULE Q. B4 Q25. To what extent are the laboratories at your CAS accessible? (CIRCLE ONE NUMBER) Accessible means that laboratories are available to you when you need to use them. Easily Accessible Accessible Not easily accessible Not accessible at all 4 3 2 1 Q26. How frequently do you use CAS laboratories? (CIRCLE ONE NUMBER) Very Frequently Frequently Rarely Never 4 3 2 1 Q27. Do you use CAS laboratories for teaching? (CIRCLE ONE NUMBER) Yes No 1 0 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 195 Q28. Do you use CAS laboratories for research? (CIRCLE ONE NUMBER) Yes No 1 0 Q29. Are CAS laboratories sufficiently equipped to meet your teaching needs? (CIRCLE ONE NUMBER) Yes No 1 0 Q30. To what extent have CAS laboratories improved the quality of your applied research? (CIRCLE ONE NUMBER) Significantly Improved Somewhat Improved Minimally Improved No Improvement 4 3 2 1 Q31. To what extent are you satisfied with the adequacy of lab equipment for conducting applied research? (CIRCLE ONE NUMBER) Extremely Satisfied Satisfied Dissatisfied Extremely Dissatisfied 4 3 2 1 Module B4: Curricula Reform Q32. You have access to up-to-date resources/materials (academic journals, textbooks, other books, etc.)? (CIRCLE ONE NUMBER) Yes No 1 0 Q33. CAS has a mechanism in place to reform curricula periodically? (CIRCLE ONE NUMBER) Yes No 1 0 Q34. CAS curricula are aligned with the needs of sector organizations? (CIRCLE ONE NUMBER) Yes No 1 0 To what extent do you agree or disagree with the following statements? (CIRCLE ONE NUMBER) Question # Statement Strongly Agree Agree Disagree Strongly Disagree Q35. The CAS curricula adequately balance theory and practical application. 4 3 2 1 Q36. Faculty teaching methodologies adequately balance theory and practical application. 4 3 2 1 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 196 Module B5: Governance To what extent do you agree or disagree with the following statements? (CIRCLE ONE NUMBER) Question # Statement Strongly Agree Agree Disagree Strongly Disagree Q37. In CAS governance committees, all members have a voice in decision making. (Examples of governance committees include curriculum review committees etc.) 4 3 2 1 Q38. The CAS governance structures (policies, procedures, oversight committees) are open to suggestions for improvement. 4 3 2 1 Q39. The CAS governance structures (policies, procedures, oversight committees) are flexible in incorporating suggestions/feedback. 4 3 2 1 Q40. Mechanisms for addressing grievances (e.g., of faculty, students etc.) are in place. 4 3 2 1 Q41. CAS rules and regulations for curricula reform are clear/easily understandable. (CIRCLE ONE NUMBER) Yes No 1 0 Q42. CAS rules and regulations for conducting Applied Research are clear /easily understandable. (CIRCLE ONE NUMBER) Yes No 1 0 Module B6: Graduate and Postgraduate Training Programs and Employment NOTE: CIRCLE ONE NUMBER Question # Express your level of satisfaction/dissatisfaction with the following aspects of CAS Training and Degree programs: Extremely Satisfied Satisfied Dissatisfied Extremely Dissatisfied Q43. Preparation of students to pursue further studies. 4 3 2 1 Q44. Job placement efforts of the CAS. 4 3 2 1 Q45. Assistance students receive to conduct research. 4 3 2 1 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 197 Q46. To what extent are employers satisfied or dissatisfied with the quality of recruited CAS graduates? Extremely Satisfied Satisfied Dissatisfied Extremely Dissatisfied 4 3 2 1 To what extent do you agree or disagree with the following statements? (CIRCLE ONE NUMBER) Question # Statement Strongly Agree Agree Disagree Strongly Disagree Q47. The barriers to employment faced by women are GREATER than those faced by men in your sector (Energy/Water). 4 3 2 1 Module B7: Sustainability Q48. How likely is CAS to continue to exist at this university after the current donor support ends? (CIRCLE ONE NUMBER) Highly likely Somewhat likely Somewhat unlikely Highly unlikely 4 3 2 1 Q49. How likely is CAS to generate alternate financial resources after the current donor support ends? (CIRCLE ONE NUMBER) Highly likely Somewhat likely Somewhat unlikely Highly unlikely 4 3 2 1 Q50. How likely are partnerships for applied research with industry to continue after the current donor support ends? (CIRCLE ONE NUMBER) Highly likely Somewhat likely Somewhat unlikely Highly unlikely 4 3 2 1 Q51. How likely are joint research projects with other universities to continue after the current donor support ends? (CIRCLE ONE NUMBER) Highly likely Somewhat likely Somewhat unlikely Highly unlikely 4 3 2 1 Q52. In the future, how likely are you to continue applying the research skills learnt as part of your training? (CIRCLE ONE NUMBER) Highly likely Somewhat likely Somewhat unlikely Highly unlikely 4 3 2 1 Q53. How likely are you to continue working with CAS after the donor support ends? (CIRCLE ONE NUMBER) Highly likely Somewhat likely Somewhat unlikely Highly unlikely 4 3 2 1 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 198 Module C: Information / Consent Form C1. Can you share your mobile number? (CIRCLE ONE NUMBER) Yes GO TO C1A No GO TO C2 Do not Have Mobile GO TO C2 (DO NOT READ) Don’t Know GO TO C2 Refused to Answer GO TO C2 1 2 3 -88 -77 C1A. Record mobile number Number1: Number 2: C2. Can you share your landline number? (CIRCLE ONE NUMBER) Yes GO TO C2A No GO TO C3 Do not Have Landline GO TO C3 (DO NOT READ) Don’t Know GO TO C3 Refused to Answer GO TO C3 1 2 3 -88 -77 C2A. Record landline number City/Area Code Phone number City/Area Code Phone number Note: if response is “yes” in either C1 or C2, then ask C3. C3. Can we contact you via mobile or landline in the future if we need additional information? (CIRCLE ALL THAT APPLY) Cell Landline No 1 2 3 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 199 Survey Questionnaire for Graduates/Alumni Serial number (FOR OFFICE USE ONLY) Start Time: hh:mm (AM/PM): Interviewer ID: End Time: hh:mm (AM/PM): Interview Date: DD/MM/YY Introduction: USAID/Pakistan, in coordination with the Pakistan Higher Education Commission (HEC) is supporting three US-Pakistan Centers for Advanced Studies (CAS) in energy (2 centers) and water (1 center) that are supported by two American universities. In conjunction with their U.S. partners, the centers are developing programs that produce graduates who are prepared to meet the needs of industry, government, and their communities through cutting edge knowledge and technologies. Project activities to support these objectives include: exchange programs and joint and applied research with U.S. partner universities, building and equipping laboratories, developing/reforming curricula, improving university governance, and scholarships to support graduate and post￾graduate training programs. USAID has requested that Management Systems International (MSI) conduct an evaluation of the CAS project. Results will help improve implementation for the remaining project duration. The evaluation will assess the effectiveness of university partnerships; overall project management; and sustainability of the results, including the role of governance structures. As part of this evaluation to gauage the effectiveness of the CAS project, we are conducting a survey of CAS graduates. Since you are a CAS graduate, we would appreciate your participation in this survey. This survey will take approximately 30-40 minutes. We will treat everything you say here confidentially. We will not use your names in reports or attribute anything you say to you. Module A: Background A Respondent name Refused to answer 99 A2. Gender Male 0 Female 1 A3. Which year were you born? 19___ Refused to answer 99 A4. Which is your domicile province/territory? Punjab 1 Sindh 2 Khyberpakhyunkhwa 3 Balochistan 4 Gilgit-Baltistan/Northern areas 5 FATA 6 Azad Jammu-Kashmir 7 Capital Territory/ICT 8 A5. Which CAS did you graduate from? NUST (Energy) 1 UET, Peshawar (Energy) 2 MUET, Jamshoro (Water) 3 A6. Which CAS degree program were you in? Masters 1 Ph.D. 2 A7. Which year did you graduate? 20___ 1. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 200 NOTE FOR TELEPHONIC INTERVIEWS: ➢ Enumerators should read every question along with responses aloud, unless instructed otherwise. Do NOT read ‘Don’t Know’, ‘Refused to Answer’ or ‘NA’ options. Module B1: Partnership with US Universities and Exchange program Q1. Did you apply for a CAS exchange program? (CIRCLE ONE NUMBER) Yes No 1 0 GO TO Q11 Q2. Is the selection process for the CAS exchange program transparent? (CIRCLE ONE NUMBER) Yes No Don’t Know Refused to Answer 1 0 98 99 Q3. Were you selected for a CAS exchange program? (CIRCLE ONE NUMBER) Yes No 1 0 GO TO Q11 Q4. Did you participate in the exchange program? (CIRCLE ONE NUMBER) Yes No 1 0 GO TO Q11 To what extent do you agree or disagree with the following statements? (CIRCLE ONE NUMBER) Question # Statement Strongly Agree Agree Disagree Strongly Disagree Q5. Your Pakistani CAS university adequately prepared you before the start of the exchange program. 4 3 2 1 Q6. The American host university adequately supported you in adjusting to the new environment during the exchange program. 4 3 2 1 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 201 Q7. Did the exchange program improve your applied research skills? (CIRCLE ONE NUMBER) (Applied research is research that meet the explicit needs of sector organizations) Yes No 1 0 Q8. Did the exchange program improve your applied research output? (CIRCLE ONE NUMBER) Yes No 1 0 Q9. Did the exchange program improve your teaching skills? (CIRCLE ONE NUMBER) Yes No Not Applicable 1 0 98 Q10. Did the exchange program improve your proposal writing skills? (CIRCLE ONE NUMBER) Yes No 1 0 Q11. To what extent are you satisfied/dissatisfied overall with your university’s partnership with its US CAS partner? (CIRCLE ONE NUMBER) Extremely Satisfied Satisfied Dissatisfied Extremely Dissatisfied 4 3 2 1 Module B2: Joint and Applied Research and Public-Private Partnerships Q12. Were you engaged in applied research while at CAS? (CIRCLE ONE NUMBER) Yes No 1 0 GO TO Q. 14 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 202 To what extent do you agree or disagree with the following statements? (CIRCLE ONE NUMBER) Question # Statement Strongly Agree Agree Disagree Strongly Disagree Don’t Know/NA Q13. The applied research you are conducting is relevant to needs of sector organizations. 4 3 2 1 98 Q14. There is a process to align CAS applied research topics to the explicit needs of organizations in our sector of focus (i.e., energy/water). 4 3 2 1 98 Q15. Research led by women scholars is accepted by organizations in the sector. 4 3 2 1 98 Q16. Sector organizations prefer males as compared to females in lead research roles. 4 3 2 1 98 Q17. At your CAS, women are involved in conducting research in your in the sector of focus (i.e., energy/water). 4 3 2 1 98 Q18. At your CAS, women are provided opportunities to take a lead role in conducting research in the sector of focus (i.e., energy/ water). 4 3 2 1 98 Q19. While at CAS, I had the freedom to choose/identify the topic(s) I wanted to do research on. 4 3 2 1 98 Q20. Faculty members adequately advised me on my research so that it is of high quality. 4 3 2 1 98 NOTE: CIRCLE ONE NUMBER Question # To what extent are you satisfied/dissatisfied with: Extremely Satisfied Satisfied Dissatisfied Extremely Dissatisfied Don’t Know Q21. The financial support you received from CAS to conduct applied research? 4 3 2 1 98 Q22. The support you received from the partner US university to conduct applied research? The support could be in areas such as research design, analytical tools, publishing support, guidance on use of lab/lab equipment, etc. 4 3 2 1 98 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 203 Module B3: Lab Utilization at Pakistani CAS University Q23. During your studies at CAS, were CAS-supported laboratories at your university operational? (CIRCLE ONE NUMBER) Yes No 1 0 GO TO MODULE Q. B4 Q24. To what extent were the laboratories at your CAS accessible? (CIRCLE ONE NUMBER) Accessible means that laboratories were available to you when you need to use them. Easily Accessible Accessible Not easily accessible Not accessible at all 4 3 2 1 Q25. How frequently did you use laboratories? (CIRCLE ONE NUMBER) Very Frequently Frequently Rarely Never 4 3 2 1 Q26. To what extent did CAS laboratories improve the quality of your applied research. (CIRCLE ONE NUMBER) Significantly Improved Somewhat Improved Minimally Improved No Improvement 4 3 2 1 Q27. To what extent were you satisfied with the adequacy of lab equipment for conducting applied research? (CIRCLE ONE NUMBER) Extremely Satisfied Satisfied Dissatisfied Extremely Dissatisfied 4 3 2 1 Module B4: Curricula Reform Q28. You had access to up-to-date resources/materials (academic journals, textbooks, other books, etc.)? (CIRCLE ONE NUMBER) Yes No 1 0 Q29. CAS had a mechanism in place to reform curricula periodically? (CIRCLE ONE NUMBER) Yes No Don’t Know 1 0 98 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 204 Q30. Curricula were aligned with the needs of sector organizations? (CIRCLE ONE NUMBER) Yes No Don’t Know 1 0 98 To what extent do you agree or disagree with the following statements? (CIRCLE ONE NUMBER) Question # Statement Strongly Agree Agree Disagree Strongly Disagree Q31. The CAS curricula adequately balanced theory and practical application. 4 3 2 1 Q32. Faculty teaching methodologies adequately balanced theory and practical application. 4 3 2 1 Module B5: Governance To what extent do you agree or disagree with the following statements? (CIRCLE ONE NUMBER) Questio n # Statement Strongly Agree Agree Disagree Strongly Disagree Don’t Know Q33. The CAS governance structures (policies, procedures, oversight committees) were open to suggestions for improvement. 4 3 2 1 98 Q34. The CAS governance structures (policies, procedures, oversight committees) were flexible in incorporating suggestions/feedback. 4 3 2 1 98 Q35. Mechanisms for addressing grievances (e.g., of faculty, students etc.) were in place. 4 3 2 1 98 Q36. CAS rules and regulations for Curricula reform were clear/easily understandable? (CIRCLE ONE NUMBER) Yes No Don’t Know 1 0 98 Q37. CAS rules and regulations for conducting Applied Research were clear /easily understandable? (CIRCLE ONE NUMBER) Yes No Don’t Know 1 0 98 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 205 Module B6: Graduate and Postgraduate Training Programs and Employment NOTE: CIRCLE ONE NUMBER Question # Express your level of satisfaction/dissatisfaction with the following aspects of CAS Training and Degree programs: Extremely Satisfied Satisfied Dissatisfied Extremely Dissatisfied Q38. Quality of teaching. 4 3 2 1 Q39. Your preparation to pursue further studies. 4 3 2 1 Q40. Job placement efforts of the CAS. 4 3 2 1 Q41. Assistance students received to conduct research. 4 3 2 1 Q42. Are you currently employed? (CIRCLE ONE NUMBER Yes No 1 0 GO TO Q48 Q43. Is this work relevant to what you studied in your CAS degree program? (CIRCLE ONE NUMBER) [Alumni] Yes No 1 0 Q44. What is the nature of your job? (CIRCLE ONE NUMBER) Public sector 1 SKIP Q47 Private Sector 2 SKIP Q47 Self-employed 3 GO TO Q47 Q45. To what extent are employers satisfied or dissatisfied with the quality of recruited CAS graduates? (CIRCLE ONE NUMBER) Extremely Satisfied Satisfied Dissatisfied Extremely Dissatisfied 4 3 2 1 Q46. How long did it take you to get a job after graduation? (CIRCLE ONE NUMBER) Less than 3 months 1 Between 3 & 6 months 2 Between 6 & 12 months 3 More than 12 months 4 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 206 Q47. Is this a start-up venture? (CIRCLE ONE NUMBER) Yes No 1 0 To what extent do you agree or disagree with the following statements? (CIRCLE ONE NUMBER) Question # Statement Strongly Agree Agree Disagree Strongly Disagree Q48. I make my career decisions independently. 4 3 2 1 Q49. The barriers to employment faced by women are GREATER than those faced by men in your sector (Energy/Water). 4 3 2 1 Module B7: Sustainability Q50. In the future, how likely are you to continue applying the research skills learnt as part of your training at CAS? (CIRCLE ONE NUMBER) Highly likely Somewhat likely Somewhat unlikely Highly unlikely 4 3 2 1 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 207 Module C: Information / Consent Form C1. Can you share your mobile number? (CIRCLE ONE NUMBER) Yes GO TO C1A No GO TO C2 Do not Have Mobile GO TO C2 (DO NOT READ) Don’t Know GO TO C2 Refused to Answer GO TO C2 1 2 3 -88 -77 C1A. Record mobile number Number1: Number 2: C2. Can you share your landline number? (CIRCLE ONE NUMBER) Yes GO TO C2A No GO TO C3 Do not Have Landline GO TO C3 (DO NOT READ) Don’t Know GO TO C3 Refused to Answer GO TO C3 1 2 3 -88 -77 C2A. Record landline number City/Area Code Phone number City/Area Code Phone number Note: if response is “yes” in either C1 or C2, then ask C3. C3. Can we contact you via mobile or landline in the future if we need additional information? (CIRCLE ALL THAT APPLY) Cell Landline No 1 2 3 U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 208 Survey Questionnaire for CAS Current Students Serial number (FOR OFFICE USE ONLY) Start Time: hh:mm (AM/PM): Interviewer ID: End Time: hh:mm (AM/PM): Interview Date: DD/MM/YY Introduction: USAID/Pakistan, in coordination with the Pakistan Higher Education Commission (HEC) is supporting three US-Pakistan Centers for Advanced Studies (CAS) in energy (2 centers) and water (1 center) that are supported by two American universities. In conjunction with their U.S. partners, the centers are developing programs that produce graduates who are prepared to meet the needs of industry, government, and their communities through cutting edge knowledge and technologies. Project activities to support these objectives include: exchange programs and joint and applied research with U.S. partner universities, building and equipping laboratories, developing/reforming curricula, improving university governance, and scholarships to support graduate and post￾graduate training programs. USAID has requested that Management Systems International (MSI) conduct an evaluation of the CAS project. Results will help improve implementation during the remaining project duration. The evaluation will assess the effectiveness of university partnerships; overall project management; and sustainability of the results, including the role of governance structures. As part of this evaluation to gauge the effectiveness of the CAS project, we are conducting a survey of current CAS program students, and CAS graduates. Since you are a current student, we would appreciate your participation in this survey. This survey will take approximately 30-40 minutes. We will treat everything you say here confidentially. We will not use your names in reports or attribute anything you say to you. Module A: Background Respondent name Refused to answer 99 Gender Male 0 Female 1 Which year were you born? 19___ Refused to answer 99 Which is your domicile province/territory? Punjab 1 Sindh 2 Khyberpakhtunkhwa 3 Balochistan 4 Gilgit-Baltistan/Northern areas 5 FATA 6 Azad Jammu-Kashmir 7 Capital Territory/ICT 8 Which CAS are you associated with? NUST (Energy) 1 UET, Peshawar (Energy) 2 MUET, Jamshoro (Water) 3 Which CAS degree program are you in? Masters 1 Ph.D. 2 A A2. 1. A3. A4. A5. A6. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 209 NOTE FOR TELEPHONIC INTERVIEWS: • Enumerators should read every question along with responses aloud, unless instructed otherwise. Do NOT read ‘Don’t Know’, ‘Refused to Answer’ or ‘NA’ options. Module B1: Partnership with US Universities and Exchange program Did you apply for a CAS exchange program? (CIRCLE ONE NUMBER) Yes No 1 0 GO TO Q11 Is the selection process for the CAS exchange program transparent? (CIRCLE ONE NUMBER) Yes No Don’t Know Refused to Answer 1 0 98 99 Were you selected for a CAS exchange program? (CIRCLE ONE NUMBER) Yes No 1 0 GO TO Q11 Did you participate in the exchange program? (CIRCLE ONE NUMBER) Yes No 1 0 GO TO Q11 To what extent do you agree or disagree with the following statements? (CIRCLE ONE NUMBER) Question # Statement Strongly Agree Agree Disagree Strongly Disagree Your Pakistani CAS university adequately prepared you before the start of the exchange program. 4 3 2 1 The American host university adequately supported you in adjusting to the new environment during the exchange program. 4 3 2 1 Q1. Q2. Q3. Q4. Q5. Q6. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 210 Did the exchange program improve your applied research skills? (CIRCLE ONE NUMBER) (Applied research is research that meets the explicit needs of sector organizations) Yes No 1 0 Did the exchange program improve your applied research output? (CIRCLE ONE NUMBER) (Applied research is research that meets the explicit needs of sector organizations) Yes No 1 0 Did the exchange program improve your teaching skills? (CIRCLE ONE NUMBER) Yes No Not Applicable 1 0 98 Did the exchange program improve your proposal writing skills? (CIRCLE ONE NUMBER) Yes No 1 0 To what extent are you satisfied/dissatisfied overall with your university’s partnership with its US CAS partner? (CIRCLE ONE NUMBER) Extremely Satisfied Satisfied Dissatisfied Extremely Dissatisfied 4 3 2 1 Module B2: Joint and Applied Research and Public-Private Partnerships Have you engaged in applied research since you joined CAS? (CIRCLE ONE NUMBER) Yes No 1 0 GO TO Q. 14 Q7. Q8. Q9. Q11. Q12. Q10. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 211 To what extent do you agree or disagree with the following statements? (CIRCLE ONE NUMBER) Question # Statement Strongly Agree Agree Disagree Strongly Disagree Don’t Know/NA The applied research you are conducting is relevant to needs of sector organizations. 4 3 2 1 98 There is a process to align CAS applied research topics to the explicit needs of organizations in our sector of focus (i.e., energy/water). 4 3 2 1 98 Research led by women scholars is accepted by organizations in the sector. 4 3 2 1 98 Sector organizations prefer males over females in lead research roles. 4 3 2 1 98 At this CAS, women are involved in conducting research in your in the sector of focus (i.e., energy/water). 4 3 2 1 98 At this CAS, women have opportunities to take a lead role in conducting research in the sector of focus (i.e., energy/ water). 4 3 2 1 98 NOTE: CIRCLE ONE NUMBER Questio n # To what extent are you satisfied/dissatisfied with: Extremely Satisfied Satisfied Dissatisfied Extremely Dissatisfied Don’t Know/NA The participation of sector organizations in conducting applied research? 4 3 2 1 98 The process used to select the applied research areas/topics. 4 3 2 1 98 The financial support you received from CAS to conduct applied research? 4 3 2 1 98 The support you received from the partner US university to conduct applied research? The support could be in areas such as research design, analytical tools, publishing support, guidance on use of lab/lab equipment, etc. 4 3 2 1 98 The level of autonomy you have to spend research money (grants) you receive. 4 3 2 1 98 Q13. Q14. Q15. Q16. Q17. Q18. Q19. Q21. Q22. Q23. Q20. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 212 To what extent do you agree or disagree with the following statements? (CIRCLE ONE NUMBER) Question # Statement Strongly Agree Agree Disagree Strongly Disagree I have the freedom to choose/identify the topic(s) I want to do research on. 4 3 2 1 Faculty members adequately advise me on my research so that it is of high quality. 4 3 2 1 Module B3: Lab Utilization at Pakistani CAS University Are CAS-supported laboratories at your university operational? (CIRCLE ONE NUMBER) Yes No 1 0 GO TO MODULE Q. B4 To what extent are the laboratories at your CAS university accessible in Pakistan? (CIRCLE ONE NUMBER) Accessible means that laboratories are available to you when you need to use them. Easily Accessible Accessible Not easily accessible Not accessible at all 4 3 2 1 How frequently do you use CAS laboratories? (CIRCLE ONE NUMBER) Very Frequently Frequently Rarely Never 4 3 2 1 To what extent have CAS laboratories improved the quality of your applied research? (CIRCLE ONE NUMBER) Significantly Improved Somewhat Improved Minimally Improved No Improvement 4 3 2 1 To what extent are you satisfied with the adequacy of lab equipment for conducting applied research? (CIRCLE ONE NUMBER) Extremely Satisfied Satisfied Dissatisfied Extremely Dissatisfied 4 3 2 1 Q24. Q25. Q26. Q27. Q28. Q29. Q30. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 213 Module B4: Curricula Reform You have access to up-to-date resources/materials (academic journals, textbooks, other books, etc.)? (CIRCLE ONE NUMBER) Yes No 1 0 CAS has a mechanism in place to reform curricula periodically? (CIRCLE ONE NUMBER) Yes No Don’t Know 1 0 98 CAS curricula are aligned with the needs of sector organizations? (CIRCLE ONE NUMBER) Yes No Don’t Know 1 0 98 To what extent do you agree or disagree with the following statements? (CIRCLE ONE NUMBER) Question # Statement Strongly Agree Agree Disagree Strongly Disagree The CAS curricula adequately balance theory and practical application. 4 3 2 1 Faculty teaching methodologies adequately balance theory and practical application. 4 3 2 1 Module B5: Governance To what extent do you agree or disagree with the following statements? (CIRCLE ONE NUMBER) Questio n # Statement Strongly Agree Agree Disagree Strongly Disagree Don’t Know The CAS governance structures (policies, procedures, oversight committees) are open to suggestions for improvement. 4 3 2 1 98 The CAS governance structures (policies, procedures, oversight committees) are flexible in incorporating suggestions/feedback. 4 3 2 1 98 Mechanisms for addressing grievances (e.g., of faculty, students etc.) are in place. 4 3 2 1 98 Q31. Q32. Q33. Q34. Q35. Q36. Q37. Q38. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 214 CAS rules and regulations for Curricula reform are clear/easily understandable. (CIRCLE ONE NUMBER) Yes No Don’t Know 1 0 98 CAS rules and regulations for conducting Applied Research are clear /easily understandable. (CIRCLE ONE NUMBER) Yes No Don’t Know 1 0 98 Module B6: Graduate and Postgraduate Training Programs and Employment NOTE: CIRCLE ONE NUMBER Question # Express your level of satisfaction/dissatisfaction with the following aspects of CAS Training and Degree programs: Extremely Satisfied Satisfied Dissatisfied Extremely Dissatisfied Quality of teaching. 4 3 2 1 Your preparation to pursue further studies. 4 3 2 1 Job placement efforts of the CAS. 4 3 2 1 Assistance students receive to conduct research. 4 3 2 1 What are your employment plans upon graduating from your CAS program? To obtain a job in the government sector 1 To obtain a job in the private sector 2 To pursue self-employment 3 To pursue an academic career (teaching and research) 4 To pursue further study instead of employment 5 I do not intend to pursue a career 6 I do not plan for a particular sector 7 Other (Please specify) 8 Did you do an internship? (CIRCLE ONE NUMBER) Yes No 1 0 GO TO Q50 Where did you do the internship (WRITE THE NAME OF ORGANIZATION) Refused to answer 99 Q39. Q41. Q42. Q43. Q44. Q45. Q46. Q47. Q40. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 215 To what extent are you satisfied or dissatisfied with the internship experience? Extremely Satisfied Satisfied Dissatisfied Extremely Dissatisfied 4 3 2 1 To what extent was your employer satisfied or dissatisfied with the quality of your skills? Extremely Satisfied Satisfied Dissatisfied Extremely Dissatisfied 4 3 2 1 To what extent do you agree or disagree with the following statements? (CIRCLE ONE NUMBER) Question # Statement Strongly Agree Agree Disagree Strongly Disagree I make my career decisions independently. 4 3 2 1 The barriers to employment faced by women are GREATER than those faced by men in your sector (Energy/Water). 4 3 2 1 Module B7: Sustainability How likely is CAS to continue to exist at this university after the current donor support ends? (CIRCLE ONE NUMBER) Highly likely Somewhat likely Somewhat unlikely Highly unlikely 4 3 2 1 How likely are partnerships for applied research with industry to continue after the current donor support ends? (CIRCLE ONE NUMBER) Highly likely Somewhat likely Somewhat unlikely Highly unlikely 4 3 2 1 How likely are joint research projects with other universities to continue after the current donor support ends? (CIRCLE ONE NUMBER) Highly likely Somewhat likely Somewhat unlikely Highly unlikely 4 3 2 1 In the future, how likely are you to continue applying the research skills learnt as part of your training? (CIRCLE ONE NUMBER) Highly likely Somewhat likely Somewhat unlikely Highly unlikely 4 3 2 1 Q48. Q49. Q51. Q52. Q53. Q54. Q55. Q50. U.S.-PAKISTAN CENTERS FOR ADVANCED STUDIES – MIDTERM EVALUATION 216 Module C: Information / Consent Form C1. Can you share your mobile number? (CIRCLE ONE NUMBER) Yes GO TO C1A No GO TO C2 Do not Have Mobile GO TO C2 (DO NOT READ) Don’t Know GO TO C2 Refused to Answer GO TO C2 1 2 3 -88 -77 C1A. Record mobile number Number1: Number 2: C2. Can you share your landline number? (CIRCLE ONE NUMBER) Yes GO TO C2A No GO TO C3 Do not Have Landline GO TO C3 (DO NOT READ) Don’t Know GO TO C3 Refused to Answer GO TO C3 1 2 3 -88 -77 C2A. Record landline number City/Area Code Phone number City/Area Code Phone number Note: if response is “yes” in either C1 or C2, then ask C3. C3. Can we contact you via mobile or landline in the future if we need additional information? (CIRCLE ALL THAT APPLY) Cell Landline No 1 2 3 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523