May 2014 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Management Systems International under the Monitoring and Evaluation Program by Douglas Krieger, Ghazanfar Hoti, Sarah Tirmazi, Ahmad Jameel, and Sara Azmat Zaidi. Fulbright Student Program in Pakistan Midterm Performance Evaluation Report ACKNOWLEDGEMENTS The Monitoring and Evaluation Program (MEP) would like to thank Sarah Tirmazi (Evaluation Team Leader), Ahmad Jameel (Education Specialist and Evaluation Team Member), Ghazanfar Ali Khan Hoti (MEP Senior Evaluation Specialist and Team Member), and Sara Azmat Zaidi (MEP Evaluation Specialist and Evaluation Team Member) for undertaking desk reviews and field work for the evaluation and completing the evaluation report. Jennifer Mandel (Senior Evaluation Advisor, MEP) guided MEP staff and the evaluation team in the data collection design and analysis of qualitative data and provided training for conducting focus group discussions. Tariq Husain (Evaluation Director, MEP) provided detailed comments during the instrument design and report writing process to improve the clarity and presentation of the report. Bilal Hassan Khan (MEP Director Survey) guided the evaluation team data collection design and analysis of quantitative data, and provided training to enumerators for quantitative data collection. Muhammad Danish (Survey Analyst) assisted the evaluation team in analysis of quantitative data. Ghazanfar Hoti also managed the evaluation, supported by Sara Zaidi as co-manager during the data collection process and prior to the arrival of the evaluation team. Douglas Krieger (Management Systems International (MSI) Technical Director for MEP) provided detailed comments in the final steps of report writing to improve the evaluation logic, clarity, and the presentation of the report. MEP would also like to thank Voice Tel Tech and its staff for conducting online and telephonic survey for this evaluation. The evaluation team would particularly like to thank the United States Education Foundation in Pakistan for its support in conducting this evaluation. FULBRIGHT STUDENT PROGRAM EVALUATION IN PAKISTAN MIDTERM PERFORMANCE EVALUATION REPORT May 2014 Contracted under No. GS-23F-8012H and Order No. AID-391-M-11-00001 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i CONTENTS Project Summary .................................................................................................................................................................................... iv Executive Summary................................................................................................................................................................................. 1 Program Background .............................................................................................................................................................................. 7 Evaluation Purpose and Evaluation Questions ...............................................................................................................................10 Evaluation Methods and Limitations .................................................................................................................................................13 Findings and Conclusions ....................................................................................................................................................................17 Recommendations ................................................................................................................................................................................44 Annexes ...................................................................................................................................................................................................45 Annex 1: Evaluation Statement of Work ...................................................................................................................................45 Appendix 1: Initial Getting to Answers Table .....................................................................................................................62 Appendix 2: Criteria to Ensure the Quality of the Evaluation Report (USAID Evaluation Policy) .......................64 Annex 2: References .......................................................................................................................................................................65 Annex 3: Fulbright Alumni Survey Questionnaire ...................................................................................................................66 Annex 4: Post-stratification and Racking of Survey Data ......................................................................................................76 Annex 5: Instrument for Focus Group Discussions with Alumni........................................................................................78 Annex 6: Instrument for Personal Interviews with Supervisors ..........................................................................................81 Annex 7: Instrument for Personal Interviews with HEC Officials.......................................................................................84 Annex 8: Return on Investment Calculation .............................................................................................................................86 Annex 9: Disclosure of Any Conflicts of Interest ...................................................................................................................99 Annex 10: MEP Evaluation Team Bios ..................................................................................................................................... 101 Tables and Figures Table 1: Fulbright Scholarship Awards, 2005-2013 ........................................................................................................................ 9 Table 2: Focus Group Discussion Participants ..............................................................................................................................14 Table 3: Employers Interviewed and Sector of Employment .....................................................................................................15 Table 4: Duration of Stay in Pakistan ...............................................................................................................................................20 Table 5: Before and After Comparison of Monthly Salaries for Fulbright Alumni (in PKR) ..............................................30 Table 6: Average Annual Costs by Year and Degree (Nominal U.S. Dollar) ........................................................................36 Table 7: Average Annual Pre- and Post-Degree Earnings by Sex and Degree ......................................................................37 Table 8: Fulbright Program Benefits and Costs (U.S. Dollar) ....................................................................................................38 Table 9: Fulbright Scholarship Awards, 2005-2011 ......................................................................................................................50 Table 10: Tentative Evaluation Schedule .........................................................................................................................................60 Table 11: Level of Effort of Evaluation Team Members ..............................................................................................................61 Table 12: Difference Between Population and Sample Estimates Before and After Raking ..............................................77 Table 13: Average Annual Costs by Year and Degree (Nominal U.S. Dollar) ......................................................................88 Table 14: Total Enrollment by Year, Sex, and Degree Type .....................................................................................................88 Table 15: Pre- and Post-Degree Average Annual Earnings by Sex and Degree Type .........................................................89 ii Table 16: Estimated Effects of Experience on Annual Earnings................................................................................................. 89 Table 17: Estimates of Unemployment as a Function of Age ....................................................................................................91 Table 18: Historic U.S. Dollar/PKR Exchange Rates ....................................................................................................................92 Table 19: Number of Fulbright Alumni by Year, Sex, and Degree........................................................................................... 93 Table 20: Average Annual Earnings of Employed Alumni (PKR) ..............................................................................................94 Table 21: Benefits of a Fulbright Degree by Year, Sex, and Degree Type (Nominal U.S. Dollar) ...................................95 Table 22: Annual Program Costs by Year, Sex, and Degree Type (Nominal U.S. Dollar) ................................................96 Table 23: Present Values of Benefits by Sex, Degree Type, and Discount Rate (2012 U.S. Dollar) ...............................96 Table 24: Present Values of Costs by Sex, Degree Type, and Discount Rate (2012 U.S. Dollar) ...................................97 Table 25: ROI by Discount Rate, Sex, and Degree Type ...........................................................................................................97 Figure 1: Distribution of Scholarship Recipients by District ........................................................................................................ v Figure 2: Compliance with Residency Requirement ....................................................................................................................18 Figure 3: Reasons for Leaving Pakistan ............................................................................................................................................19 Figure 4: Employment Status by Degree and Sex .........................................................................................................................21 Figure 5: Fields of Study and Employment for Alumni ................................................................................................................23 Figure 6: Reasons for Working in a Field not Related to Academic Training .......................................................................24 Figure 7: Perception of the American People and the U.S. ........................................................................................................27 Figure 8: Effect of Fulbright Experience on Alumni Perceptions ..............................................................................................28 Figure 9: Average Salary Increases for Fulbright Alumni (Percent) ..........................................................................................30 Figure 10: Alumni Promotions After Fulbright (Percent) ...........................................................................................................31 Figure 11: Changes in Job Position After Fulbright ......................................................................................................................32 Figure 12: Change in Job Position .....................................................................................................................................................33 Figure 13: Change in Publication Behavior .....................................................................................................................................34 Figure 14: Total Number of Publications by Type ........................................................................................................................34 Figure 15: Ongoing Personal and Professional Linkages (Percent)...........................................................................................39 Figure 16: Nature of Ongoing Linkages and Collaboration .......................................................................................................40 Figure 17: Factors Affecting Partnerships with U.S. Entities ......................................................................................................41 Figure 18: Growth in Earnings as a Function of Sex and Education .........................................................................................90 Figure 19: Unemployment as a Function of Age (Post Graduate and PhD Holders) ..........................................................92 Figure 20: ROI by Discount Rate, Sex, and Degree Type ..........................................................................................................98 iii ACRONYMS DO Development Objective FATA Federally Administered Tribal Areas FCR Findings, Conclusions, and Recommendations FGD Focus Group Discussion GOP Government of Pakistan GRE Graduate Record Exam HEC Higher Education Commission ICT Islamabad Capital Territory IR Intermediate Result KP Khyber Pakhtunkhwa MEP Monitoring and Evaluation Program MRR USDOS Mission Resource Request MOU Memorandum of Understanding MSI Management Systems International MTDF Medium Term Development Framework PKR Pakistani Rupee PSLM Pakistan Social and Living Standards Measurement ROI Return on Investment SOW Statement of Work TPM Team Planning Meeting VTT Voice Tel Tech USAID United States Agency for International Development USDOS United States Department of State USEFP United States Educational Foundation in Pakistan iv PROJECT SUMMARY USAID Objectives Addressed DO 4: Improved Access to High Quality Education IR 4.1: Improved Educational Opportunities IR 4.1.1: Improved Educational Facilities IR 4.1.2: Increased Access to Scholarships Cross-cutting Objective 3: Improved Public Perception of the U.S. U.S. Department of State Mission Resource Request Objective No. 7: Public Diplomacy and Strategic Communications Performance Indicator No. 2: Increased number of Pakistani understand that the United States and Pakistan share similar broad values and objectives and can work together to assist each other’s goals and objectives in the region Implementing Partner United States Education Foundation in Pakistan Cooperative Agreement No. 391- G-00-02-00001-00 Project Dates First Grant: April 2005 Second Grant: May 2009 Third Grant: April 2010 Project Budget First Grant: US$15 million Second Grant: US$19.5 million Third Grant: US$19.5 million Fulbright Scholarship Support Program Higher Education Commission-USAID (HEC contribution) US$42 million over four years Project Location Fulbright Scholars selected nationwide in Pakistan for graduate studies (master’s and PhD) in the U.S. v FIGURE 1: DISTRIBUTION OF SCHOLARSHIP RECIPIENTS BY DISTRICT Figure 1 illustrates the geographic distribution of Fulbright scholarship recipients in terms of their home districts, as opposed to the districts where they now reside. 1 EXECUTIVE SUMMARY PROJECT BACKGROUND The Fulbright Student Program in Pakistan (henceforth called the Program) awards merit-based scholarships for both master’s and doctoral level study in the U.S. to early and mid-career professionals with high academic achievement and potential for leadership. The Program is intended to support awardees’ academic development, create mutual understanding between the people of Pakistan and the U.S., and facilitate linkages between American and Pakistani academic institutions and scholars.1 The Program in Pakistan is also intended to support the U.S. Government’s public policy and diplomacy strategy, as well as meet the priority needs of the Government of Pakistan (GOP) in higher education. Between 2005 and 2011, the United States Education Foundation in Pakistan (USEFP) awarded Fulbright scholarships to 910 individuals. By the end of 2011, 702 of these students had completed their degrees and returned to Pakistan, or had committed to a specific return date. Of the returned alumni, 606 earned master’s degrees and 96 earned PhDs; 418 were male and 284 female. PURPOSE AND EVALUATION QUESTIONS While the United States Agency for International Development (USAID) expects to continue funding Fulbright scholarships, it has substantial latitude to adjust program parameters (e.g., level of funding, priority fields of study, types, and distribution of degrees supported). The evaluation will assess the extent to which the program is meeting its objectives and provide recommendations aimed at improving performance relative to USAID and U.S. Government objectives. For ease of presentation, the seven evaluation questions, which are summarized below, are not reproduced here. EVALUATION DESIGN, METHODS, AND LIMITATIONS The evaluation relied on primary and secondary qualitative and quantitative data from a variety of sources. A comprehensive survey of 616 Fulbright alumni provided the primary quantitative data. Focus group discussions (FGDs) with alumni and personal interviews with employers and former and current senior Higher Education Commission (HEC) officials provided additional data and also supplied the qualitative data to complement and triangulate survey results. The primary limitation of the data is the potential for self-selection bias in survey, FGD, and personal interview results. Respondents to the alumni survey may have underrepresented alumni who had not satisfied the residency requirement, those who were not satisfied with the program, those who were less successful in their careers, or those who were less articulate. Similarly, employers interviewed may have represented alumni who were more comfortable having the evaluation team interview their employers. The survey achieved a response rate of 55 percent, with the online portion generating just over half of the responses (53 percent). While the response rate was relatively high for a survey of this type, especially for the online portion, it was low enough to raise concerns about the overall precision of results. Response rates in this range also increase the likelihood of significant differences between respondents and non-respondents due to non-response bias. 1 USAID, January 31, 2013, Fulbright Student Program Evaluation Statement of Work 2 SUMMARY OF FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS Question 1: To what extent do Fulbright alumni return to Pakistan and, for those who do return, how long do they stay in Pakistan? Findings  All 337 respondents to the survey of Fulbright alumni reported returning to Pakistan after completing their degree.  Of 337 survey respondents who provided data, 86 percent (83 percent of men and 87 percent of women) had remained in Pakistan since obtaining their degree. Among the 14 percent of respondents who reported leaving Pakistan after completing their degree, 11 percent eventually returned. Only 3 percent (3 percent of men and 4 percent of women) were currently working outside of Pakistan.  Sixty-one percent of respondents reported that they planned to remain in Pakistan.  Economic conditions were the single most important reason for planning to leave Pakistan.  Only 7 percent (10 percent of men and 3 percent of women) of survey respondents reported they had not completed the mandatory residency period in Pakistan after completing their degree, instead leaving Pakistan early to pursue economic opportunities. Conclusions Most recipients of Fulbright-sponsored degrees between 2005 and 2012 returned to and remained full-time in Pakistan after earning their degrees. Sixty-one percent reported no intention to leave Pakistan in the future. Pakistan’s current economic conditions, however, threaten to reduce retention of Fulbright graduates in Pakistan. More than one-third of survey respondents did not intend to remain in Pakistan and over 60 percent of these cited economic conditions as a primary reason. The 14 percent of graduates between 2005 and 2011 who had left Pakistan stayed in Pakistan for just over two years on average before leaving. Question 2: To what extent are Fulbright alumni productively employed in fields related to their academic preparation? Findings  Most (93 percent male and 81 percent female) respondents to the alumni survey reported being employed.  A large majority (92 percent male and 88 percent female) of employed alumni worked in a field related to their academic preparation.  Those not working in their chosen fields most often cited difficulty finding a job in the field as the reason.  With the exception of some specialized skills, most alumni participants in the FGDs reported that the education they received with the Fulbright scholarship was relevant to their work. Conclusions An overwhelming majority of Fulbright alumni are productively employed and working in fields related to their academic preparation. Women were more likely to be unemployed than men, although the reasons are unclear. It appears that a lack of opportunities in some fields (e.g., math and sciences, English, social sciences, economics, business) are largely responsible for the failure of some alumni to find jobs related to their fields of study. 3 Question 3: To what extent do Fulbright alumni contribute to achieving the goals and objectives of the U.S. Government’s development strategy in Pakistan as reflected in USAID’s relevant Development Objectives (DOs) and Intermediate Results (IRs)? Findings  HEC identified increasing the number of internationally trained PhD faculty at Pakistani universities as a priority.  Most families in Pakistan cannot afford the cost of a graduate degree at a U.S. university. The USEFP estimated the annual cost of a graduate degree in the U.S. at US$25,000 while average per capita income in Pakistan was US$1,372 in 2011-2012.  Between 2005 and 2011, the USAID-funded Fulbright program provided 26 percent of all scholarships for graduate study at foreign universities.  A majority (83 percent) of respondents to the alumni survey reported they would not have been able to obtain the Fulbright-funded degree without the scholarship.  Ninety-four percent of survey respondents had a favorable perception of the U.S. and 97 percent a favorable perception of the American people.  Thirty-eight percent of respondents reported a more favorable perception of U.S. foreign policy since obtaining their degree. Conclusions The Fulbright Program contributes to the goals and objectives of the U.S. Government’s development strategy in Pakistan. These goals and objectives are: development objective “improved access to high quality education” and intermediate results “improved educational opportunities; increased access to scholarships; and improved public perception of the U.S.” Most scholarship recipients could not have afforded a U.S. education otherwise. The Fulbright Program accounted for a substantial share of the foreign scholarships available to Pakistani students for graduate education. Most Fulbright alumni reported more favorable views of educational and U.S. policy and cultural characteristics, and were much more likely to have favorable views of the American people and the U.S. as a country than the Pakistan population as a whole. Question 4: To what extent do the alumni of the Fulbright Program actualize their potential for, and assume positions of leadership in, academia, government, industry, and business in Pakistan? Findings  Respondents to the alumni survey reported average increases in earnings (pre-degree compared to post￾degree) of 208 percent for male and 201 percent for female participants2 . FGD results corroborated the survey findings.  Ninety-two percent of the 274 employed Fulbright alumni who provided data on their field of employment reported working in a field related to their academic preparation.  More than 50 percent of alumni reported being promoted in their jobs after completing their Fulbright￾funded degree, and all of those who had been working for at least six years had been promoted at least once. 2 The comparison was made only for alumni who were employed both before and after the award of the scholarship. 4  A majority (83 percent among entry level and 68 percent among middle level) of alumni reported holding a higher position at the time of the survey than prior to receiving the scholarship.  The number of Fulbright alumni who had published peer-reviewed papers almost doubled from 15 percent pre-degree to 28 percent post-degree.  Nine out of 10 employers interviewed reported that Fulbright alumni showed more leadership potential than employees without a Fulbright-funded degree. Conclusions Fulbright alumni are assuming positions of responsibility and leadership within academia, government, industry and business in Pakistan. Most have substantially higher earnings than prior to earning their Fulbright-funded degrees. Over half have received at least one promotion since graduating with their Fulbright-funded degree. A majority work in mid-level positions and are in a higher position now than before. The percentage of alumni who have published almost doubled (from 15 percent to 28 percent) after completing their degrees. The quality of these academic publications has increased, with a greater number of alumni publications appearing in international and academic journals than in national journals. Qualitative information from personal interviews with employers and HEC officials suggests that Fulbright alumni are better prepared than their counterparts without a U.S. degree. Question 5: What is the Return on Investment (ROI) of the Fulbright Program? Findings The Fulbright-Hays Act, the Fulbright Program, and the USAID Results Framework suggest a variety of objectives, which include increased mutual understanding, educational and cultural advancement, improved access to higher education and scholarships, and improved public perception of the U.S. Training Pakistanis at U.S. universities may benefit the individual student, Pakistan, and the U.S. by increasing alumni incomes, contributing to development, increasing intellectual capacity in the public and private sectors (including strengthening Pakistani universities to the extent that alumni pursue academic careers within Pakistan), or contributing to cultural and academic exchange. This incomplete list of potential benefits emphasizes the difficulties inherent in identifying the full range of benefits of a Fulbright-sponsored degree. Furthermore, few of the potential benefits listed above are easily monetized. Even though the Fulbright program serves a variety of purposes above the level of the individual beneficiary, only private benefits, except for some of the sub-points mentioned above, are included in this calculation. The program's benefits actually accrue to the entire society or perhaps both societies, the U.S. and Pakistan. The ROI calculation, however, considers only the private benefits to alumni associated with higher incomes and only the cost to USEFP of supporting a student through a degree program, because these were the only data available. The findings are:  At a 12 percent discount rate, which is the rate the GOP uses for development projects as well as the standard rate used by multilateral and bilateral development agencies, the return on investment of a Fulbright-sponsored degree is 36 percent for the entire program.  The ROI for a master’s degree is higher than for a PhD, and higher for male than for female master’s candidates.  The ROI for a PhD becomes negative at discount rates greater than 10 percent (for men) and 11 percent (for women).This occurs because greater discount rates reduce the present value of benefits, which will accrue in the future, relative to costs, which occurred in the past The negative ROI means that the financial costs of the degree exceed the financial benefits at the specified discount rates. The result may be different if the ROI included the full range of costs and benefits. Conclusions Even considering a very restricted set of quantifiable benefits (i.e., increases in lifetime earnings attributable to a Fulbright-sponsored degree) and costs (i.e., the costs to USEFP to support students in their degree programs), the program generated a return of 36 percent over its costs. Consistent with findings elsewhere regarding the value of 5 graduate education, returns to a PhD are lower than returns to a master’s program. 3 The lower return to a PhD is a result of the longer time, and thus higher cost, required to obtain the degree. For instance, in the 2011-12 academic year, a two-year master’s degree costs US$117,310 while a four-year PhD costs US$186,144 or 59 percent more. Furthermore, the difference between pre-and post-degree earnings, i.e., the benefit of the degree, was lower for male PhDs than for male master’s degree holders. While female PhDs enjoyed a larger “benefit” than female master’s degree holders, it was not high enough to offset the higher cost relative to a master’s degree. Question 6: To what extent do Fulbright alumni contribute to achieving the goals and objectives of U.S. Government’s public diplomacy strategy in Pakistan, as reflected in the United States Department of State (USDOS’s) Mission Resource Request (MRR), in Pakistan? Findings  All 337 respondents to the alumni survey reported maintaining some linkages with U.S. individuals or institutions: 82 percent with former classmates, 62 percent with their universities, 60 percent with U.S. faculty members, and 16 percent with non-academic institutions.  Fulbright alumni also reported forming lasting relationships outside the academic environment. Several FGD participants fondly remembered host families and personal relationships and others recounted that American friends have visited Pakistan to attend weddings of their former classmates.  Sixty-four percent reported social interaction and a substantial number reported maintaining professional networks (43 percent) and collaborating on research (24 percent).  Alumni and HEC officials interviewed in FGDs agreed that travel, visa, and movement issues between Pakistan and the U.S. impede academic collaboration. Conclusions Fulbright alumni are contributing to U.S. diplomacy goals and objectives by increasing the professional and cultural exchange between Pakistani and American institutions and individuals. All surveyed alumni reported maintaining ties in the U.S. that strengthen professional collaboration and appreciation of American cultural values. Alumni have brought back new ideas, a new appreciation of American academic practices, and a better understanding of shared cultural values. In spite of these positive results, however, inadequate research facilities in Pakistan, limited funding for research, and travel restrictions for Pakistanis serve to restrict the Program’s potential contribution to professional, academic, and cultural exchange. Question 7: How relevant is the current Fulbright Program (e.g., mix of master’s and PhDs: U.S., regional, in-country study, and fields of study) to the priority needs of HEC? Findings  Increasing the number of Pakistani university faculty members trained in top U.S. graduate programs is a key HEC priority.  HEC also prioritizes degrees in science and technology, humanities, and social sciences. 3 Alan Stark, Economic Studies and Policy Analysis Division, Department of Finance, Canada, “Which Fields Pay, Which Fields Don’t? An Examination of the Returns to University Education in Canada by Detailed Field of Study,” February 2007. http://www.oecd.org/social/labour/37578152.pdf 6  All Fulbright-funded PhDs fell within these priority fields: 37 percent in science and technology and 63 percent in the social sciences and humanities.  HEC officials interviewed stated that U.S. universities produced the highest quality teachers and faculty in Pakistan’s universities. Conclusions The Fulbright Program is well-coordinated to support HEC’s strategy for higher education in Pakistan. Fulbright alumni increase the pool of available PhDs to fulfill the substantial anticipated demand of Pakistani universities in key fields of study. The fact that many alumni understand the importance of teaching to Pakistan’s economic and cultural development enhances the relevance of the Program. The fields of study and mix of degrees funded under the Program are generally consistent with the identified priorities of HEC. Furthermore, many stakeholders recognize the many facets of value attached to a degree from a U.S. university. Recommendations The evaluation concluded that the Fulbright Program has been very effective in supporting USAID and U.S. Government objectives. It has increased access to high quality education, improved access to scholarships, contributed to cultural understanding and academic collaboration, supported HEC objectives for improving the quality of faculty in Pakistani universities, and been responsive to changing HEC priorities in terms of supported fields of study. Fulbright alumni also have very favorable perceptions of the U.S. and the American people. The evaluation team could not identify opportunities for improving performance in areas addressed by the evaluation. 7 PROGRAM BACKGROUND THE PROBLEM STATEMENT One of the key public policy and diplomacy objectives of the Unites States Government has been to “increase understanding for American values, policies, and initiatives to create a receptive international environment”.4 One of the major sub-objectives is to “promote international educational exchanges and professional exchanges”.5 The Fulbright Scholarship Program compliments these public policy and diplomacy objectives. The Program has been functioning since 1951 in Pakistan through the auspices of USEFP, an organization which manages and executes several U.S. scholarship programs.6 HEC, which is a semi-autonomous organization of the GOP, has the mandate to increase access to higher education nationwide. The Medium Term Development Framework (MTDF) articulates the higher education priorities of the GOP. The 2005 MTDF identified the key problems as the less than ideal quality of faculty and the fact that many PhD faculty members were due to retire in the near future.7 The report recognized that it was important that university faculty should at least have a PhD.8 The latest MTDF (2011-2015) substantiated HEC’s requirement for well-qualified PhD faculty by stating that: “Faculty development was identified as the core aim in MTDF HE-I (2005-10), since it was recognized that it would not be possible to enhance access and improve the quality of higher education and research without the availability of sufficient, highly qualified faculty members.”9 In order to meet the objectives of both the U.S. Government and the GOP, USEFP has been administering the Fulbright Student Program in Pakistan since 2005.10 Funding for the Program is based on agreements between USAID and USDOS (f),11 as well as an agreement between USEFP and HEC.12 4 U.S. Department of State and U.S. Agency for International Development, Strategic Plan 2004-2009, p. 30. The more recent Strategic Plan for 2007-2012 does not mention the Fulbright Program specifically, but “Strategic Goal 6: Promoting International Understanding” highlights the need for positive public perceptions and the role of education in achieving this internationally (p. 34) 5 Ibid., p. 32 6 The initial agreement regarding educational exchange programs between the Government of the United States and the Government of Pakistan was signed in 1950 and later revised and signed in 1972. 7 Higher Education Commission, Medium Term Development Framework 2005-2010, p. 23 8 Ibid., p. 20 9 Higher Education Commission, Medium Term Development Framework 2011-2015, p. 27 10 The Fulbright Student Program is funded by USAID and started in 2005. USEFP has been administering other scholarship programs since 1951. 11 Memorandum of Agreement between the United States Agency for International Development (USAID) and the United States Department of State (USDOS) to Transfer Economic Support Funds, April 2005; Memorandum of Agreement between USAID and USDOS, Bureau of Educational and Cultural Affairs to Transfer Economic Support Funds, May 2009; and Memorandum of Agreement between USAID and USDOS to Transfer Economic Support Funds, April 2010 12 Agreement between USEFP and Higher Education Commission of Pakistan on the Fulbright/HEC/USAID Scholarship Program for Pakistani PhD Students in the United States, December 2005 8 THE DESIGN OF THE PROGRAM The Fulbright Student Program in Pakistan awards merit-based scholarships for both master’s and doctoral-level study in the U.S. to early and mid-career professionals with potential for leadership and high academic achievement. The Program is intended to support awardees’ academic development and create mutual understanding between the people of Pakistan and the U.S. It is expected to facilitate linkages between American and Pakistani academic institutions and to promote universities’ access to global education networks and job markets. The Program develops institutional capacity in Pakistan’s universities by developing the expertise of potential faculty members in various disciplines including finance, public administration, education, health, agriculture, environmental management, public policy, and media and communications. The Statement of Work (SOW) for the evaluation (Annex 1) provides further details. A 2008 evaluation of USAID/Pakistan’s higher education portfolio13 concluded that the Pakistan Fulbright Program is making a major contribution to HEC’s Medium Term Development Framework (MTDF I: 2005-2010),14 in terms of developing faculty, improving the quality of higher education, enhancing research capacity, forming linkages between Pakistani and U.S. universities, and promoting mutual understanding. PROGRAM IMPLEMENTATION Target Areas and Study Disciplines The Program accepts applications from qualified men and women throughout Pakistan, and awards scholarships on a merit-based process which includes acceptable Graduate Record Exam (GRE) scores and an interview. Fulbright interview panels include two Pakistanis who are Fulbright alumni volunteers, and two Americans, often U.S. Embassy professional staff volunteers. Table 1presents a summary of Fulbright awards by field between 2005 and 2013. Since 2005, the program has awarded 1,261 scholarships to Pakistani students in 20 disciplines. USEFP records at the time of evaluation indicate that 702 grantees have graduated and returned to Pakistan, or committed to specific return dates. Most of the remainder has not yet completed their studies. 13 Academy for Educational Development, 2008, Evaluation of USAID Higher Education Portfolio http://pdf.usaid.gov/pdf_docs/PDACM417.pdf 14 Higher Education Commission (HEC), (no date), Medium Term Development Framework I: 2005-10, http://www.hec.gov.pk/InsideHEC/AboutHEC/Documents/413_HEC_med_dev.pdf 9 TABLE 1: FULBRIGHT SCHOLARSHIP AWARDS, 2005-2013 Discipline Year Total by Gender Overall Total Percentage of Total 2005 2006 2007 2008 2009 2010 2011 2012 2013 M F M F M F M F M F M F M F M F M F M F Agriculture 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 2 2 2 4 0.3% Business 3 5 4 1 11 5 4 5 7 5 5 4 8 0 8 3 12 6 62 34 96 7.6% Computer Science 10 5 12 4 15 10 4 3 5 5 10 0 1 3 7 5 5 2 69 37 106 8.4% Economics 4 2 11 3 9 10 4 4 5 2 8 5 5 5 9 3 5 6 60 40 100 7.9% Education 3 2 0 1 3 3 2 2 2 0 2 4 3 7 2 3 0 4 17 26 43 3.4% Energy 0 0 0 0 0 0 0 0 0 0 0 1 7 0 0 2 4 0 11 3 14 1.1% Engineering 13 0 12 3 28 4 24 2 16 5 15 3 14 4 29 6 36 7 187 34 221 17.5% Environment 2 1 2 1 0 0 0 0 0 2 1 0 2 2 2 3 2 1 11 10 21 1.7% Finance 1 1 5 2 8 1 3 2 8 2 0 2 2 1 2 3 1 0 30 14 44 3.5% Fine Arts 1 2 1 3 4 7 3 6 7 10 2 5 3 3 2 9 5 4 28 49 77 6.1% Health 5 3 2 4 4 4 4 11 6 4 3 5 6 1 5 2 2 0 37 34 71 5.6% Humanities 0 0 0 1 1 1 0 0 0 0 2 1 0 0 0 2 0 2 3 7 10 0.8% Journalism 2 0 1 2 7 3 5 1 2 4 3 10 1 5 2 4 2 4 25 33 58 4.6% Law 5 1 0 0 0 0 3 1 2 0 4 1 4 2 2 2 2 3 22 10 32 2.5% Literature 4 3 1 2 1 1 2 5 3 7 2 2 0 2 0 2 2 5 15 29 44 3.5% Natural Science 3 1 2 2 5 1 7 1 8 4 4 2 2 1 2 3 4 4 37 19 56 4.4% Public Policy 12 2 8 5 10 5 13 6 9 1 10 2 8 3 10 11 14 17 94 52 146 11.6% Social Science 3 2 2 4 5 8 7 7 5 11 5 5 4 5 5 13 10 11 46 66 112 8.9% Social Work 0 0 0 0 0 1 0 0 0 0 0 0 0 2 0 2 0 0 0 5 5 0.4% Water Resources 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 1 0.1% Total 71 30 63 38 111 64 85 56 85 62 76 52 71 46 87 78 108 78 757 504 1,261 100% 10 EVALUATION PURPOSE AND EVALUATION QUESTIONS EVALUATION PURPOSE The Fulbright Student Program has been awarding scholarships to Pakistani students since 1951. While USAID expects to continue the program, it has substantial latitude to adjust program parameters (e.g., level of funding, priority fields of study, types and distribution of degrees supported). The purpose of this performance evaluation is to help USAID:  Assess the performance of the Fulbright Program;  Determine how well the program has supported HEC’s overall strategy and priorities; and  Tailor future programming to meet U.S. Government development and public diplomacy objectives. The evaluation will develop recommendations aimed at:  Improving the performance of the Fulbright Program;  Maximizing the degree to which it contributes to achieving U.S. Government strategic objectives; and  Helping USAID enhance the impact of its higher education interventions by prioritizing and balancing the portfolio of activities. EVALUATION QUESTIONS To address these objectives, the evaluation answers seven questions that MEP developed in collaboration with USAID. The questions are designed to provide a view of the broad effects of the Fulbright Program. The basic questions and associated explanations provided the information necessary for the evaluation team to develop the specific questions that guided data collection and analysis. The italics in the explanations indicate the key variables used as indicators to answer the questions. 1. To what extent do Fulbright alumni return to Pakistan and, for those who do return, how long do they stay in Pakistan? Explanation: Fulbright alumni are expected to return to Pakistan and use their advanced education and knowledge of American society to make long-term contributions to development in Pakistan. However, some scholarship recipients may not complete their degree or may not remain in Pakistan long enough to have a significant impact on development. This question will assess the extent to which Fulbright alumni return to Pakistan and their persistence rates.15 Because the evaluation will collect data only from relatively recent alumni, it will only be able to document the extent to which alumni remain in Pakistan for the first few years after graduation. USAID 15 Persistence rates refer to the length of time that Fulbright alumni stay in Pakistan after completing their degrees. 11 recognizes that it may be difficult to obtain data from alumni and that the evaluation team may have to rely on data provided by USEFP and HEC. 2. To what extent are Fulbright alumni productively employed in fields related to their academic preparation? Explanation: For various reasons, alumni may be unemployed, under-employed, or work in fields unrelated to their degree. The data collected in response to this question will document employment rates and the correspondence between fields of study and employment. The answer to this question will help USAID and USDOS ensure that the program selects students who are committed to working in fields aligned with their academic preparation. Indicators of productive and relevant employment include:  Employment status (i.e. whether alumni are employed);  Sector of employment and degree discipline (i.e. whether the sector of the graduates’ employment is related to the discipline in which they obtained their degree); and  To the extent possible, results disaggregated by degree type, discipline, sex, and other available and relevant demographic variables. 3. To what extent do Fulbright alumni contribute to achieving the goals and objectives of the U.S. government’s development strategy in Pakistan as reflected in USAID’s relevant DOs and IRs? Explanation: The U.S. Government is providing assistance to the people of Pakistan to achieve specific strategic objectives as articulated in USAID’s Results Framework (e.g. improved access to high quality education, increased access to scholarships, and improved public perception of the U.S.). This question will determine the extent to which the program contributes to these objectives. 4. To what extent do the alumni of the Fulbright Program actualize their potential for and assume positions of leadership in academia, government, industry, and business in Pakistan? Explanation: The evaluation will assess the extent to which Fulbright alumni assume positions of leadership by examining increases in salaries, promotions, publications, and job position. Data from a survey of alumni and interviews with employers and related stakeholders such as government and university officials provide the data to address this question. 5. What is the Return on Investment (ROI) of the Fulbright Program? Explanation: Calculating the ROI of the Fulbright Program will be extremely challenging. The “investment” per student is simply USAID’s fully-loaded costs, including overhead, of obtaining the degree. However, USAID recognizes the difficulty in reliably and credibly monetizing the returns to higher education programs. The calculated ROI will therefore necessarily understate the return on the investment. 6. To what extent do Fulbright alumni contribute to achieving the goals and objectives of the U.S. Government’s public diplomacy strategy in Pakistan, as reflected in the USDOS’s Mission Resource Request (MRR), in Pakistan? Explanation: The U.S. Government provides assistance to the people of Pakistan to achieve specific public diplomacy objectives, articulated in the USDOS’s MRR. These include:  “Objective No. 7: Public Diplomacy and Strategic Communications” and 12  “Performance Indicator No. 2: Increased number of Pakistanis understand that the United States and Pakistan share similar broad values and objectives and can work together to assist each other’s goals and objectives in the region.” Given the nature of the Fulbright Program, it is reasonable to expect that students, faculty members, and universities in both countries will form connections which will, in some cases, evolve into formal and sustainable partnerships. Keeping in view USDOS’s broad public diplomacy objective, the evaluation will answer this question by assessing the extent to which Fulbright Program alumni have formed and maintained relationships with their respective institutions and have fostered links between Pakistani and U.S. schools. These links include developing social ties, mutual understanding, and collaboration in academia and research between U.S. and Pakistani students and scholars. These relationships serve as a proxy for Performance Indicator No. 2 above. This question will help determine if there are “spin-off” benefits from the Fulbright Program that yield enduring U.S.-Pakistani partnerships. Evidence of sustainable partnerships includes formal university partnership agreements, joint research projects, follow-on faculty or student exchanges, and other indicators of ongoing collaboration. 7. How relevant is the current Fulbright Program (e.g., mix of master’s and PhDs: U.S., regional, in-country study, and fields of study) to the priority needs of HEC? Explanation: In the higher education sector, U.S. Government strategy aims to address Pakistan’s development needs by working with and through HEC to achieve mutually agreed upon development and diplomacy objectives. The Fulbright Program awards about 40 PhD scholarships to Pakistani students every year, thus potentially contributing to the number of Pakistani university faculty with doctorates. This question will assess the level of HEC’s commitment to the Fulbright Program, and will help determine the most desirable balance in scholarship types (i.e. U.S., regional, or in-country) to be supported by the U.S. Government. It will also provide information to determine the ideal balance of Fulbright awards among PhD and master’s scholarships. 13 EVALUATION METHODS AND LIMITATIONS Prior to assembling as a team, evaluation team members reviewed documents and data relevant to the evaluation. MEP then facilitated a three-day Team Planning Meeting (TPM) with the evaluation team to discuss the framework and components of the evaluation. During the TPM, the evaluation team developed a detailed plan for collecting, analyzing, and reporting data, and a work plan. A senior MEP staff member presented a workshop on conducting effective FGDs and analyzing FGD data. Following the data collection, MEP staff guided a findings, conclusions, and recommendations (FCR) workshop with the evaluation team and MEP staff. MEP then informally presented the evaluation’s findings, conclusions, and recommendations to officials of USDOS, USAID, USEFP, and HEC. The evaluation employed the following data collection methods: Secondary Data The evaluation utilized some secondary data collected by USEFP and HEC. The USEFP provided information related to scholarships awarded, contact information for alumni, and average annual costs for master’s and doctoral scholars. Relevant reports include HEC’s Medium Term Development Framework II: 2011-2015,16 the 2008 Academy for Educational Development Evaluation of USAID’s Higher Education Portfolio,17 USEFP annual program reports, program agreement documents, financial audit reports, program management manuals, Performance Management Plans, and evaluation and assessment reports. Annex 2 contains a complete list of documents used in the evaluation. Primary Data The evaluation relied largely on primary quantitative and qualitative data. A survey of alumni provided most of the quantitative data. The team collected qualitative data through FGDs with Fulbright alumni and personal interviews with employers and HEC officials. The use of a variety of techniques allowed the evaluators to triangulate results across multiple sources and methods. Alumni Survey USEFP provided a list of all 616 scholarship recipients who graduated between 2007 and 2011.18 The team used this list, linked to additional information on individual alumni including their email addresses, phone numbers, fields of study, home address, current address, and gender, as the sampling frame for the survey. The evaluation team designed a survey instrument (Annex 3) to collect data relevant to the data collection and analysis plan. A local survey firm and MEP partner, Voice Tel Tech (VTT), conducted the survey. USEFP first notified all 616 alumni by email of the upcoming evaluation and introduced VTT as the survey firm. Following the introductory email, VTT sent each alumnus an invitation to complete the survey online within seven days. At the end of that period, VTT telephoned all eligible respondents who had not responded to the survey and asked them to either complete the interview on the phone or to complete the online survey at their earliest convenience. 16 Higher Education Commission, (no date), Medium Term Development Framework II: 2010-15, http://hec.gov.pk/InsideHEC/Documents/MTDF%202011-15%20FINAL.pdf 17 Academy for Educational Development, 2008, Evaluation of USAID Higher Education Portfolio http://pdf.usaid.gov/pdf_docs/PDACM417.pdf 18 Because the shortest program requires two years, graduates in 2007 correspond to scholarship recipients from 2005. 14 A total of 337 alumni completed the survey: 304 earned a master’s degree, 21 earned a doctoral degree and the remaining 12 respondents had completed both degrees. The survey achieved a response rate of 55 percent, with the online portion generating just over half of the responses (53 percent). While the response rate was relatively high for a survey of this type, especially for the online portion, it was low enough to raise concerns about the overall precision of results. Response rates in this range also increase the likelihood of significant differences between respondents and non-respondents due to non-response bias. Potential sources of bias include the fact that respondents were those who voluntarily chose to participate in the survey. This self-selection may introduce bias since respondents may have felt more positive about their Fulbright experiences and achievements and wished to share that for evaluation purposes. Alumni who were unemployed, under-employed or otherwise not using their education productively may also have been less inclined to complete the survey. Survey respondents may also have been more likely than non-respondents to have complied with the Fulbright and HEC residency requirements. Non-respondents may also have been more likely than respondents to have already left the country. To address the problem of bias, MEP analysts developed weights to compensate for over-coverage or under￾coverage of some of the important proportions present in the population. The variables showing some imbalance between the population proportions and final sample proportions were gender, province, and city of origin. All analysis of the survey data presents weighted results. Annex 4 presents the details of the technical aspects of post￾stratification and weighting. Focus Group Discussions To add depth to the analysis, the evaluation team conducted six FGDs with Fulbright alumni, two each in Islamabad, Karachi, and Lahore. Annex 5 contains a copy of the FGD instrument. The team selected FGD participants from among respondents to the survey. The survey asked whether respondents would be interested in participating in a FGD and the evaluation team invited FGD participants from among those who gave a positive response. The evaluation team scheduled FGDs on specific dates and then contacted the alumni to see if they were available on those days. The final FGD participants were thus those who (1) responded to the survey, (2) were willing to participate in the FGDs, and (3) were available on the specified dates. Furthermore, the FGDs were held in the more “accessible” cities. FGD participants were not, therefore, a representative sample of the alumni population. These factors potentially biased the sample invited for the FGDs towards the more articulate alumni who may have felt more comfortable expressing their opinions in a group setting, those more eager to participate and share their experiences, those with more flexible schedules, and those based in or near the three cities in which the team conducted the FGDs. The evaluation presents FGD results as consensus opinion of the group rather than individual responses. The moderator encouraged participants to discuss the questions among the group and ideally come to a consensus. Since groups did not always express a consensus, however, the moderator had to use his or her judgment to note if there were major conflicts in the views expressed. Table 2 presents information on the numbers of FGD participants in each city. TABLE 2: FOCUS GROUP DISCUSSION PARTICIPANTS Date City Total number of Participants Male Female March 12, 2013 Islamabad 1 8 7 1 March 13, 2013 Islamabad 2 6 2 4 March 14, 2013 Lahore 1 8 5 3 March 15, 2013 Lahore 2 11 5 6 March 14, 2015 Karachi 1 9 3 6 March 15, 2013 Karachi 2 6 5 1 15 Personal Interviews with Employers The team conducted structured interviews with 10 employers of Fulbright alumni in Islamabad, Karachi, and Lahore, of which nine were direct supervisors. Annex 6 contains the interview guide. These interviews provided evidence of employers’ perceptions of the effect of the Fulbright Program on leadership and other qualities of the alumni. The alumni survey asked respondents for permission to contact their employers and the sample of employers for interviews was selected from among the positive responses. Hence, employers represented alumni (1) who took the survey, (2) who were actually employed, (3) who were willing to allow the evaluation team to interview their employers, and (4) whose employers were available to participate in the interviews on the specified dates. The employers interviewed, and the alumni they represent, are thus not representative samples. Furthermore, the interviews were held in the more “accessible” cities (the same cities where the FGDs were conducted). All these factors probably substantially biased the employer interviews towards those more likely to have a favorable view of their employee’s performance and attitude. Table 3 summarizes characteristics of the employers the team interviewed. TABLE 3: EMPLOYERS INTERVIEWED AND SECTOR OF EMPLOYMENT Name of Employer City Sector of Employment Iqra University Islamabad Higher education Assessment and Strengthening Program Islamabad Development Global Technologies Islamabad Media and communications State Bank of Pakistan (Public sector) Karachi Central bank Aahung (Non-governmental Organization) Karachi Women’s reproductive health Bank AlFalah Karachi Private bank Beacon House School System Lahore Basic and tertiary education Forman Christian College (Public sector) Lahore Higher education Conrad Labs Lahore Information technology Institute of Career and Professional Development Lahore Human resource development Personal Interviews with HEC Officials The team conducted personal interviews with four HEC officials in Islamabad: the former chairman, the former executive director, the current executive director, and the current member for operations and planning. The former officials oversaw the startup of the HEC component of the Fulbright Student Program in 2005, and they provided valuable historical perspective. The current officials provided insights into policy changes and current perspectives on the Program. Annex 7 contains the HEC personal interview instrument and Annex 8 documents the individuals the team interviewed. DATA ANALYSIS METHODS With the exception of the ROI calculation, quantitative data analysis consisted largely of summary frequency-tables and cross-tabulations. The qualitative information complemented the quantitative findings by triangulating results and adding depth. METHODOLOGICAL STRENGTHS AND LIMITATIONS One strength of the methodological approach was its reliance on multiple data sources and collection methods to triangulate and add depth to evaluation findings. Another strength was the two-pronged approach to the survey (i.e., online and telephone) which produced a relatively high response rate for this type of survey. However, the survey and interview data almost certainly suffer from some level of selection bias. Selection bias occurs when the 16 subjects of surveys or interviews are not representative of the population of interest. The process through which alumni decided to participate in the survey and FGDs and the limited geographic locations for the FGDs probably favored respondents and participants with more positive views of the Fulbright Program, more in compliance with Program requirements, more likely to be successful in their careers, and those working closer to Islamabad, Karachi, or Lahore. Similar sources of bias affect the interviews with employers. Another source of bias is that mobile alumni are harder to reach, and thus are likely underrepresented in the data relative to more stationary individuals. Respondents living in areas with less accessible internet may also be underrepresented. 17 FINDINGS AND CONCLUSIONS QUESTION 1 To what extent do Fulbright alumni return to Pakistan and, for those who do return, how long do they stay in Pakistan? The agreement between scholarship recipients and USEFP requires that “Pakistanis selected for Fulbright awards are obliged to return to Pakistan within 30 days of completing their programs and to serve a “residency requirement” of at least two years or equal to the length of their scholarship program.19 For master’s candidates, this agreement is a contract with USEFP, and for PhD candidates, a bond with HEC.20 All 337 respondents to the alumni survey reported returning to Pakistan after completing their programs. Participants in three of the six FGDs stated that the residency requirement was a major determinant of their decision to return to Pakistan. However, FGD participants also gave other reasons for returning, including:  To serve in Pakistan;  More opportunities and challenges in Pakistan;  Commitment to the spirit of Fulbright, the prestige associated with being a Fulbright scholar, and not wanting to spoil that impression;  Family reasons;  To gain some experience in Pakistan before returning to the U.S.; and  No jobs available in the U.S. Of the 337 respondents (187 men and 150 women) who provided data, 66 percent (68 percent of men and 64 percent of women) reported completing the USEFP residency requirement, 27 percent (22 percent of men and 33 percent of women) were still within the residency period and 7 percent (10 percent of men and 3 percent of women) had left the country before completing the residency requirement (Figure 2). Extrapolating results for those who are past their residency requirement to the 27 percent who are still in the requirement period suggests that about 91 percent of graduates will ultimately complete their residency requirement and 9 percent will not.21 19 USEFP website: http://www.usefpakistan.org/ 20 Ibid. 21 Some of those who are still within their residency requirement may leave before satisfying the requirement. One estimate of the percentage who will not satisfy the residency requirement is the percentage of other students who did not satisfy the requirement (i.e., 223/246=91 percent). Applying this to the 91 students still within their residency requirement period suggests that 9 additional students will fail to meet their residency requirement. This figure probably overestimates the actual number of students since it is not conditional on the number of years they have already remained in Pakistan. 18 FIGURE 2: COMPLIANCE WITH RESIDENCY REQUIREMENT Source: Alumni Survey, MSI A majority (61 percent) of survey respondents (N=207) reported that they intended to remain in Pakistan after completing their residency requirement. Family, serving the country, and job satisfaction dominated the reasons for staying. The reasons they gave included:  Family reasons (77 percent);22  Serving the country (68 percent);  Job satisfaction (44 percent);  Income (15 percent); and  Health condition (2 percent). Eight of the 10 supervisors the team interviewed believed that Fulbright graduates were likely to stay in Pakistan because job opportunities in the country were expanding, especially in teaching. Moreover, seven of 10 supervisors said they expected to retain the employees by offering them a challenging and positive work environment. However, 39 percent of survey respondents (71 male and 64 female) reported that they did not intend to remain in Pakistan. Economic conditions (i.e., poor economic situation in Pakistan, lower than expected earning potential, few jobs in field of study, have not found a job in Pakistan) dominated the reasons for their decision (Figure 3).23 Sixty-one percent cited at least one economic reason. Six (4 percent) reported that they did not intend to complete the residency requirement before going abroad. 22 Percentages do not sum to 100 because the question allowed multiple responses. 23 Percentages do not sum to 100 because the question allowed multiple responses. 19 FIGURE 3: REASONS FOR LEAVING PAKISTAN 49% 45% 41% 34% 25% 18% 7% 7% 42% 44% 61% 34% 23% 11% 6% 5% 0% 10% 20% 30% 40% 50% 60% 70% Poor economic situation in Pakistan Lower than expected earning potential Personal reasons Other Few jobs in field of study Career change Skills not relevant to Pakistan Have not found a job in Pakistan Percentage of Respondents Male (71) Female (64) Source: Alumni Survey, MSI Of the 337 (187 men and 150 women) respondents who provided data, 85 percent (83 percent of men and 87 percent of women) had been in Pakistan full-time since graduating, 3 percent (3 percent of men and 4 percent of women) were currently working overseas, and 12 percent (14 percent of men and 9 percent of women)) had left Pakistan for some period of time since graduation, but currently resided in Pakistan. On average, men and women currently working overseas had remained in Pakistan for 2.2 years following graduation, and those who had left, but returned had spent 2.6 years (men) and 2.3 years (women) in Pakistan since graduating. The duration of stay for those who have not resided full-time in Pakistan since graduating depends to some extent on the year they graduated (Table 4). It is not possible with the available data to reliably project how many Fulbright students who graduated between 2007 and 2012 will ultimately remain in Pakistan. 20 TABLE 4: DURATION OF STAY IN PAKISTAN Year of Graduation Left but Returned Currently Overseas Men Women Men Women N Mean (Years) N Mean (Years) N Mean (Years) N Mean (Years) 2006 1 1.0 0 -- 0 -- 0 -- 2007 8 3.9 1 4.0 3 2.8 2 2.5 2008 4 2.1 1 2.0 1 2.5 2 2.2 2009 0 -- 0 -- 1 2.5 2 2.8 2010 4 1.1 1 1.0 3 2.0 1 1.0 2011 0 -- 0 -- 0 -- 1 2.0 2012 0 -- 0 -- 1 0.2 0 -- All years 17 2.6 3 2.3 9 2.2 8 2.2 Source: Alumni Survey, MSI Other relevant findings from the survey, FGDs, and personal interviews include:  Only 5 percent of respondents to the alumni survey were working overseas at the time of the survey. However, this figure may understate the true percentage if those who left the country were less likely to respond to the survey.  Participants in all the FGDs said they appreciated the pre-departure orientation and facilitation provided by USEFP.  Participants in one of the FGDs urged the USEFP and alumni associations to play a more active role in supporting their job hunting efforts. USEFP officials reported that they held a seminar in the U.S. for scholars who have almost completed their studies, which included job search and networking skills. Conclusions Most recipients (86 percent) of Fulbright-sponsored degrees between 2005 and 2011 returned to and remained full-time in Pakistan after earning their degrees. A majority (61 percent) reported no intention to leave Pakistan in the future. Family, commitment to Pakistan, and good job prospects in Pakistan contributed strongly to the decision to stay. Pakistan’s current economic conditions, however, threaten to reduce retention of Fulbright graduates in Pakistan. More than one-third of survey respondents did not intend to remain in Pakistan and over 60 percent cited economic conditions as a primary reason. The 14 percent of graduates between 2005 and 2011 who had left Pakistan stayed in Pakistan for just over two years on average before leaving. QUESTION 2 To what extent are Fulbright alumni productively employed in fields related to their academic preparation? Of the 337 survey respondents who reported employment status, 298 (88 percent) were employed, 24 (7 percent) were unemployed, and 15 (5 percent) were still studying in Pakistan. Figure 4 summarizes employment status by degree and sex. A large majority of Fulbright alumni (93 percent of men and 81 percent of women) were employed. Women, however, were more likely than men to be unemployed with either degree. 21 FIGURE 4: EMPLOYMENT STATUS BY DEGREE AND SEX Source: Alumni Survey, MSI Ninety-two percent of males and 88 percent of females among the 274 employed Fulbright alumni who provided data on their field of employment reported working in a field related to their academic preparation. 22 Figure 5 illustrates the correspondence between field of study and field of employment for 14 fields. Alumni in journalism, art and architecture, education, law, and computer sciences were most likely to be working in fields related to their field of study while alumni in economics and development, English, social sciences, and math and sciences were least likely to be working in a field related to their field of study.24 24 The abbreviated fields of study in Figure 5 represent a much broader range of fields. The table below describes the range of fields included in each abbreviate field of study title. Field Description Journalism Journalism/Mass communication/Communication/Information sciences Art and architecture Art/Architecture/Media/Art history/Films studies/Theatre/Creative writing Education Education/Special education Law Law Computer sciences Computer Sciences/Information Technology Public policy Public Policy/Public administration Engineering Engineering/Civil engineering/Electrical engineering/Software engineering/Computer engineering/Chemical engineering/Industrial engineering/Telecommunication Health and medicine Public/Global Health/Epidemiology/Biological sciences/Entomology/Medical sciences/Micro Biology/Environmental studies/Physical Therapy Business Finance/Business administration/Marketing/Management/Human Resource Economics and development Economics/Development economics/Development studies Social sciences Political science/Sociology/Psychology/International relations/Social work/Archaeology/Anthropology/History (Non-U.S.) Other Other (Agriculture/Occupational Therapy/Religious studies/Statistics/Technology Management/Transportation Planning/Urban Planning/ etc. English English Language/English literature/Literature (Non-U.S.)/Language (Non￾U.S.)/Linguistics/American Literature/TEFL/Applied Linguistics Math and sciences Mathematics/Physics/Chemistry 23 FIGURE 5: FIELDS OF STUDY AND EMPLOYMENT FOR ALUMNI 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Social sciences (M=7 , F=13) Public Policy (M=29 , F=12) Other (M=7 , F=12) Math and sciences (M=2 , F=2) Law (M=8 , F=3) Journalism (M=7 , F=11) Health and medicine (M=10 , F=14) English Language (M=3 , F=2) Engineering (M=23 , F=5) Education (M=5, F=6) Economics/Development (M=15, F=7) Computer Sciences (M=22 , F=8) Business (M=28 , F=11) Art/Architecture (M=7 , F=16) Percentage of respondents with degree in field Female (n=122) Male (n=173) Source: MSI Survey Twenty-seven survey respondents (7 percent male and11 percent female) indicated that they are not employed in a field related to their degree. Figure 6 summarizes the reasons they gave for the mismatch between the fields of study and employment. Difficulty in finding a job related to their field of study was the single most important reason. 24 FIGURE 6: REASONS FOR WORKING IN A FIELD NOT RELATED TO ACADEMIC TRAINING Source: Alumni Survey, MSI A majority of participants in the six FGDs reported that the education they received with the Fulbright scholarship was relevant to their current work, and that their U.S. education equipped them with the knowledge and skills for their jobs. Of the 48 FGD participants, nine said that while the generic skills they learned such as work planning, analytical tools, and research techniques had been very helpful, some of the specialized technical skills had been less so. These individuals believed that the working environments and professional requirements in Pakistan did not demand these advanced skills. For example, an operations research graduate reported that most of the theory he studied had no practical application in Pakistan. Nine of the 10 employers (supervisors) the team interviewed also mentioned that their alumni employees were working in jobs very relevant to the degree they received under the Fulbright Program. They concurred that the 10 alumni they represented were productively employed in their current positions and delivering to their potential. All the employers rated the output and quality of work of Fulbright alumni much higher than that of their locally educated counterparts, and attributed the performance and effectiveness of these employees mainly to the education and living experience in the U.S. Conclusions An overwhelming majority of Fulbright alumni are productively employed and working in fields related to their academic preparation. Women were more likely to be unemployed than men, although the reasons are unclear. It appears that a lack of opportunities in some fields (e.g., math and sciences, English, social sciences, economics, business) are largely responsible for the failure of some alumni to find jobs related to the fields in which they studied. 25 QUESTION 3 To what extent do Fulbright alumni contribute to achieving the goals and objectives of the U.S. Government’s development strategy in Pakistan as reflected in USAID’s relevant DOs and IRs? USAID’s strategic plan for Pakistan uses the following DOs and IRs to articulate the goals and objectives of the U.S. Government’s development strategy in Pakistan:25  DO 4: Improved access to high quality education;  IR 1: Improved educational opportunities;  IR 1.3: Increased access to scholarships; and  IR 3: Improved public perception of the U.S. The evaluation used the following indicators to measure achievement of these goals and objectives:  For improved access to high quality education: the affordability of higher education in the U.S. for average Pakistanis  For increased access to scholarships: the numbers of scholarships offered under Fulbright as a percentage of other scholarships offered by HEC  For improved public perception of the U.S.: the impressions of Fulbright alumni about the U.S. and the American people and the changes in perception of alumni on some selected aspects of American society after studies in the U.S. funded by Fulbright program According to the HEC’s first Medium Term Development Framework (2005-10), “The present quality of higher education in Pakistan is very low. Not a single university of Pakistan is ranked among the top 500 of the world”. 26 No Pakistani university is yet among the top 500 of the world. However, in 2012, six Pakistani universities appeared on the QS (Quacquarelli Symonds) World University Rankings of 300 Asian universities.27 The USEFP estimated average expenses for one year of graduate level study in the U.S. at US$25,000.28 This level of expenditure places a higher education in the U.S. out of the reach for average families in Pakistan, where average per capita income was US$1,372 in 2011-2012. 29 Between 2005 and 2011, the Fulbright Program awarded scholarships to 919 Pakistanis. Eighty-three percent (281) of respondents to the alumni survey said that they would not have been able to afford graduate studies in the U.S. if they had not received Fulbright scholarships. Only one participant in the six FGDs stated that his family had the resources to pay for education in the U.S. Three alumni had other scholarship offers. All other FGD participants would not have had the opportunity to receive a high quality education from a U.S. university without Fulbright funding. Extrapolated from the survey respondents to all Fulbright students, this implies that an estimated 763 students would not have been able to obtain a graduate degree from a U.S. university without the Fulbright scholarship. The scholarship certainly increased access to a high quality U.S. education for these students. 25 USAID results framework, 2012. 26 HEC, MTDF, 2005-10, p iii 27 http://www.topuniversities.com/search?key=pakistan&type=All&subject=&location=&=Search 28 This is the amount mentioned on the USEFP website http://www.usefpakistan.org/gfaq.html. It is the average expense for any graduate level study in the U.S. and does not represent the average expense that USEFP has incurred. 29 Government of Pakistan, Ministry of Finance, Pakistan Economic Survey, Executive Summary, page ii http://www.finance.gov.pk/survey/chapter_12/ExecutiveSummary.pdf 26 Increased Access to Scholarships Apart from the various scholarships managed HEC, USEFP, and the U.S. Embassy, scholarships for graduate study in the U.S. are very scarce. HEC gives scholarships for graduate studies in the U.S. (and other countries) with a priority on faculty development for public sector universities.30 During 2005-12, HEC awarded 298 master’s and 2,679 PhD scholarships in addition to Fulbright. During the same period, the Fulbright Program awarded 792 master’s and 284 PhD scholarships, 26 percent of the HEC scholarships awarded for foreign study. The HEC’s foreign scholarship programs, other than Fulbright, have been in jeopardy in recent years due to funding constraints faced by HEC.31 Improved Public Perception of the U.S. A large majority of survey respondents had a favorable opinion of the American people and of the U.S. (Figure 7).32 Very few had no opinion. These findings stand in stark contrast to the results of the annual public opinion poll conducted by the PEW Research Center which reported in 2012 that 80 percent of Pakistanis had an unfavorable view of the U.S.33 30 http://www.hec.gov.pk/insidehec/divisions/hrd/scholarships/foreignscholarships/Pages/ForeignScholarship.aspx 31 Figures on the number of scholarships awarded by other agencies such as AusAid, DFID and individual universities are not available. 32 The figure classifies the responses of “very favorable” and “somewhat favorable” as “favorable” and “somewhat unfavorable” and “very unfavorable” as “unfavorable”. 33 http://www.pewglobal.org/2012/06/27/pakistani-public-opinion-ever-more-critical-of-u-s/ 27 FIGURE 7: PERCEPTION OF THE AMERICAN PEOPLE AND THE U.S. Source: Alumni Survey, MSI A majority of alumni survey respondents reported more favorable perceptions of six key academic and U.S. characteristics as a result of the Fulbright experience (Figure 8). However, only 39 percent male and 37 percent female respondents had a more favorable perception of U.S. foreign policy after their Fulbright experience while 42 percent male and 35 percent female respondents reported no change in perception. Many FGD participants reported being involved in part-time teaching at private and public universities and were very keen to share and demonstrate American academic values such as the freedom to ask questions. One alumnus had foregone invitations to teach at prestigious universities such as the Institute of Business Administration in Karachi, and had instead opted to teach children in less privileged circumstances. He reported having a “ball of a time” doing so, and was using material from his finance courses from the U.S. 95% 91% 98% 96% 3% 4% 2% 2% 5% 0% 3% 1% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% M (187) F (150) M (187) F (150) U.S. as a Country (N=337) American people (N=337) Percentage of Respondents Favorable Unfavorable No opinion 28 FIGURE 8: EFFECT OF FULBRIGHT EXPERIENCE ON ALUMNI PERCEPTIONS Source: Alumni Survey, MSI The FGDs confirmed that Fulbright experience had contributed to a better understanding of the American people and the U.S. among alumni. Participants expressed a belief that Americans and Pakistanis share similar family values and hospitality. All of them recounted experiencing a positive aspect of U.S. society. Some expressions used by the alumni to describe their experiences include:  I expected negativity, but found none  They have values like we have – family values  My social perceptions of Americans improved  Myths about the U.S. were broken  Both sides have stereotypes of each other  People in Pakistan know more about U.S. foreign policy than about the people living in America - these are two different things  I found the Americans very helpful  Americans have the same passion for food and more so for free food like us 90% 91% 87% 83% 84% 87% 72% 76% 65% 53% 66% 45% 42% 35% 2% 1% 4% 1% 4% 2% 7% 5% 13% 15% 16% 20% 21% 27% 8% 8% 10% 15% 11% 11% 20% 19% 22% 32% 18% 35% 37% 39% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% M (187) F (150) M (187) F (150) M (187) F (150) M (187) F (150) M (187) F (150) M (187) F (150) M (187) F (150) Research methods American culture Teaching methods U.S. philanthropy U.S. political system U.S. economic system U.S. foreign policy Percentage of Respondents More favorable Less favorable No Change 29 Conclusions The Fulbright Program contributes to achieving the goals and objectives of the U.S. Government’s development strategy in Pakistan. An estimated 763 of the 919 individuals who received scholarships between 2005 and 2011 could not have afforded a U.S. education otherwise. The Fulbright Program accounted for a substantial share of the foreign scholarships available to Pakistani students for graduate education. Most Fulbright alumni reported more favorable views of educational and U.S. policy and culture characteristics, and were much more likely to have favorable views of the American people and the U.S. as a country than the Pakistan population as a whole. The favorable view of the U.S. and the American people cannot be directly attributed to the Fulbright Program, since extrapolation from the very favorable views of Pakistani university students interviewed for the Merit and Needs￾Based Scholarship Program and HEC cash transfer evaluations suggests that scholarship recipients probably had largely favorable views prior to receiving the scholarship. QUESTION 4 To what extent do the alumni of the Fulbright Program actualize their potential for and assume positions of leadership in academia, government, industry, and business in Pakistan? Leadership is a difficult construct to measure. The evaluation assesses the leadership roles and potential of Fulbright alumni by examining a number of key indicators of professional recognition or leadership. These include salary, promotions, job position, publications, and honorary leadership positions. Salary The alumni survey asked about pre- and post-Fulbright earnings. Figure 9 illustrates average percentage increases in reported earnings by degree earned. Moving from a master’s to a PhD yielded the smallest percentage increase, with a master’s degree alone producing the next highest increase. Not surprisingly, earning both degrees (i.e., moving from a bachelor’s to a PhD) produced the largest increase in earnings. On average, Fulbright alumni reported earnings increases of 208 percent for male and 201 for female respondents over their pre-Fulbright earnings.34 34 The comparison was made only for alumni who were employed both before and after the award of the scholarship. 30 FIGURE 9: AVERAGE SALARY INCREASES FOR FULBRIGHT ALUMNI (PERCENT) 204% 237% 92% 148% 478% 275% 208% 201% 0% 50% 100% 150% 200% 250% 300% 350% 400% 450% 500% M (162) F (142) M (14) F (7) M (11) F (1) M (187) F (150) MA (304) Ph.D. (21) Both degrees (12) All Degrees (337) Average salary increase percentage Degree type Source: Alumni Survey, MSI TABLE 5: BEFORE AND AFTER COMPARISON OF MONTHLY SALARIES FOR FULBRIGHT ALUMNI (IN PKR) Degree Before (PKR) After (PKR) Percentage change Male Female Male Female Male Female MA 52,420 40,592 159,160 136,992 204 237 PhD 56,547 103,769 108,372 256,897 92 148 MA and PhD 28,122 12,000 162,454 45,000 478 275 Average across all three categories 50,214 43,600 154,670 131,381 208 201 The FGDs corroborated the survey findings. In three of the six FGDs, participants said their earnings had increased, with some reporting post-Fulbright earnings several multiples larger than pre-Fulbright earnings. None of the participants indicated that their earnings had not increased since obtaining their degree. Promotions Figure 10 presents survey results on the number of currently employed alumni who had been promoted after completing their degree. Forty-three percent of employed alumni had not been promoted, and most of the remainder had been promoted only once. Not unexpectedly, however, the likelihood of promotion increases with 31 the number of years since graduation. All of those who graduated in 2006 had been promoted, and half had been promoted twice. FIGURE 10: ALUMNI PROMOTIONS AFTER FULBRIGHT (PERCENT) 11% 50% 39% 16% 42% 42% 32% 48% 20% 34% 41% 25% 65% 30% 4% 73% 24% 2% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Never Once Twice or more Never Once Twice or more Never Once Twice or more Never Once Twice or more Never Once Twice or more Never Once Twice or more Male (38) Female (19) Male (56) Female (32) Male (69) Female (45) 2006 - 2008 2009 - 2010 2011 - 2012 Percentage of respondents by year Year of graduation Source: Alumni Survey, MSI In five of the six FGDs, participants said that a promotion is not dependent on a degree, especially in the public sector, and other factors such as tenure are responsible for promotions. The lack of comparable data on promotions for non-Fulbright employees with comparable levels of education makes it difficult to determine whether Fulbright graduates are promoted more quickly (i.e., are assuming leadership positions), more rapidly, or in greater numbers than other employees. Position The alumni survey asked respondents for their job positions35 (i.e., top, mid, or entry-level) before and after receiving the Fulbright-sponsored degree. It also asked them to compare their current position with their pre￾Fulbright position (i.e., lower, about the same, higher, much higher). At an aggregate (i.e., not individual) level, a much greater percentage of Fulbright alumni held top and mid-level positions at the time of the survey than before receiving their degrees (Figure 11). Prior to their Fulbright study, 58 percent male and 69 percent of female alumni reported that they held entry-level positions. Post-Fulbright, 67 percent male and 61 percent of female alumni reported holding mid-level positions. Since the questionnaire asked for current positions (i.e., at the time of the 35 These categories were defined in the survey questionnaire as: 1)Top or senior management, government servant (Grades 20 and above), university vice chancellor or full professor; 2) Mid-level professional, experienced specialist , government servant (Grades 18-19), associate professor; and 3) Entry-level professional, specialist , government servant (Grades 17 or less), assistant professor, lecturer. 32 survey), the responses reflect both changes in job position directly after receiving the degree and subsequent promotions. FIGURE 11: CHANGES IN JOB POSITION AFTER FULBRIGHT 58% 69% 24% 30% 42% 31% 67% 61% 1% 0% 9% 9% % 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Male (171) Female (127) Male (154) Female (112) Pre-Fulbright (298) Present (266) Percentage of Respondents Entry level Middle level Top level Source: Alumni Survey, MSI At the individual level, a majority (83 percent among entry level and 68 percent among middle level) of alumni reported holding a higher position at the time of the survey than prior to receiving the scholarship (Figure 12). Not surprisingly, those who held an entry-level position prior to receiving their degree were somewhat more likely than those who held a mid-level position to hold a higher position at the time of the survey. In the personal interviews, nine in 10 employers reported that their employees with Fulbright-funded degrees showed more leadership potential than those without Fulbright-funded degrees.36 36 One in 10 employers knew that that employee was a Fulbright scholar, and nine of 10 were direct supervisors. 33 FIGURE 12: CHANGE IN JOB POSITION Source: Alumni Survey, MSI Peer-Reviewed Publications Peer-reviewed publications are another proxy for leadership, especially in academia. Prior to receiving their Fulbright-funded degree, only 15 percent of the 334 respondents who provided data reported publishing a peer￾reviewed article. The figure almost doubled to 28 percent post-Fulbright. At an individual level, 21percent male and 19 percent female alumni published for the first time since receiving their Fulbright-funded degree.37 37 A change in publishing behavior may indicate a change in employer or job position rather than a change in leadership potential or responsibility. For example, a respondent who studied for a master’s degree in Pakistan before pursuing a Fulbright PhD may have published papers from his or her thesis. If that individual took a non-academic job after the Fulbright degree, publication opportunities may have been limited. 3% 14% 83% 0% 17% 83% 2% 15% 83% 5% 25% 71% 11% 29% 61% 6% 26% 68% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Lower position About the same position Higher position Lower position About the same position Higher position Lower position About the same position Higher position Male (n=160) Female (n=86) Total (n=246) Percentage of Respondents Entry level (153) Middle level (93) 34 FIGURE 13: CHANGE IN PUBLICATION BEHAVIOR 62% 21% 10% 7% 70% 19% 7% 4% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Never published Published for first time Continued to publish Stopped publishing Percentage of Respondents Male (187) Female (150) Source: Alumni Survey, MSI As a group, those who published, either before or after their Fulbright-funded degree, published a greater number of articles after receiving their Fulbright-funded degrees than before (Figure 13). The increase in publications in international and technological/academic outlets is particularly large relative to publications in national journals. FIGURE 14: TOTAL NUMBER OF PUBLICATIONS BY TYPE 28 165 23 30 31 115 8 46 14 26 15 71 0 20 40 60 80 100 120 140 160 180 Pre-Fulbright Post Fulbright Pre-Fulbright Post Fulbright Pre-Fulbright Post Fulbright International National Technological/academic Total number of published articles Type of publication Male (187) Female (150) Source: Alumni Survey, MSI 35 Honorary Leadership Positions Honorary leadership positions are another indicator of assuming leadership responsibility. Prior to receiving their Fulbright-funded degrees, 13 percent of alumni held honorary leadership positions. The percentage increased slightly to 16 percent post-Fulbright. All four of the HEC officials the team interviewed believed it was too early to talk about leadership, especially for fresh Ph.Ds. However, three of the four clearly stated that a U.S. education was the best for producing faculty who could teach students properly as well as be a role model for nationally-trained faculty. Conclusions Fulbright alumni are assuming positions of responsibility and leadership within academia, government, industry and business in Pakistan. Most have substantially higher earnings than prior to their Fulbright-funded degrees, and higher earnings usually correlate with greater responsibility and leadership. Over half have received at least one promotion and are advancing in their careers, i.e., gaining further promotions with increased experience. A majority now work in mid-level positions, while most were in entry-level jobs prior to earning their degrees and are in a higher position now than before. While just over a quarter of alumni have published, the indicator is relevant only for the subset of alumni who are in academia. Furthermore, the percentage of alumni who have published almost doubled (from 15 percent to 28 percent) after completing their degrees, and the quality of publications has increased, i.e., a greater number of publications in international and academic journals than national journals post-Fulbright. These results are not surprising since these leadership indicators probably correlate with any type of advanced degree. However, the qualitative information from personal interviews with employers and HEC officials suggests that Fulbright alumni are more prepared than their counterparts without a U.S. degree. QUESTION 5 What is the Return on Investment (ROI) of the Fulbright Program? The ROI is a common measure of the financial efficiency of an investment. It is the percentage return on an investment, and is defined algebraically as the net value of an investment divided by the investment cost.38 Calculating the ROI requires defining the components of costs and benefits to include in the calculation and then quantifying the cost and benefit components in monetary terms. Both of these steps are potentially challenging for development projects. The Fulbright-Hays Act, the Fulbright Program, and the USAID Results Framework suggest a variety of objectives, none of which are financial. These include increased mutual understanding, educational and cultural advancement, improved access to higher education and scholarships, and improved public perception of the U.S. Training Pakistanis in U.S. universities may benefit the individual student, Pakistan, and the U.S. by increasing alumni incomes, contributing to development, increasing intellectual capacity in the public and private sectors (including strengthening Pakistani universities to the extent that alumni pursue academic careers within Pakistan), or contributing to cultural and academic exchange. This incomplete list of potential benefits emphasizes the difficulties inherent in identifying the full range of benefits of a Fulbright-sponsored degree. Furthermore, few of the potential benefits listed above are easily monetized. Even though the Fulbright program serves a variety of purposes above the level of the individual beneficiary, only private benefits, except for some of the sub-points mentioned above, are included in this calculation. The program's benefits actually accrue to the entire society or perhaps both societies, the U.S. and Pakistan. 38 ROI=((Value of investment – Investment cost)/Investment cost)x100 36 Costs of the Fulbright Program may include the costs of tuition, fees, supplies, travel, and living expenses required to pursue the degree as well as the opportunity costs associated with investing time and resources in obtaining a degree. The financial costs incurred by USEFP to support a student through a degree program are relatively easy to measure. It is relatively less easy to measure or monetize many of the other costs. The ROI calculation in this evaluation defines benefits only in terms of the estimated increase in lifetime earnings of Fulbright alumni. On the cost side it considers only the costs incurred by USEFP to support students in their degree programs. The resulting ROI thus understates both benefits and costs and is not a particularly relevant measure of the efficiency of a program that has no stated financial objectives. It should be further noted that the ROI estimate represents a lower bound on the net benefits of the program, assuming that the average returned Fulbright scholar has a positive net externality effect on Pakistani society. Annex 9 describes in detail how the evaluation team calculated the ROI for the Fulbright Program. The broad strokes of the analysis are as follows. 1. To calculate program costs in each year between 2005 and 2011, the analysis multiplied the number of students enrolled in each year in master’s and doctoral programs by the average annual costs to USEFP of maintaining a student in either degree program (Table 6) USEFP was not able to provide specific cost information for each student or university but, instead, provided the average annual cost for each degree program (i.e., master’s and doctoral) in each year. TABLE 6: AVERAGE ANNUAL COSTS BY YEAR AND DEGREE (NOMINAL U.S. DOLLAR) Academic Year Master’s Doctoral 2005/06 $56,280 $46,464 2006/07 $55,992 $49,066 2007/08 $53,769 $47,127 2008/09 $54,693 $45,747 2009/10 $55,786 $45,358 2010/11 $58,442 $45,757 2011/12 $58,655 $46,536 Source: USEFP Note: The academic year is from September to August. 2. To calculate benefits (i.e., increases in estimated earnings over a 25-year career) the analysis: a. Determined, for each year between 2005 and 2011 and from the same pool of Fulbright students used to calculate program costs, the number who had completed their degrees, returned to Pakistan, and found employment. USEFP provided data on the number of students who had completed their degrees and returned to Pakistan. The MEP survey of Fulbright alumni provided estimates of baseline (i.e., immediately post-graduation) employment rates among Fulbright alumni. Since salaries and employment rates varied by sex and type of degree, the analysis disaggregated benefit data by sex and degree type. b. Projected employment rates (by sex and degree type) among Fulbright alumni over a 25-year career. The ROI analyst estimated the relationship between employment rates and job experience from 2010-11 Pakistan Labor Force Survey data39 and used the estimates to project 39 http://www.pbs.gov.pk/sites/default/files/Labour%20Force/publications/lfs2010_11/t34.pdf 37 employment rates for each alumnus cohort (defined by year of graduation, sex, and degree) over a 25-year career. Based on estimates from the Pakistan Labor Force Survey that showed employment rates increase with experience. c. Estimated aggregate (over all employed Fulbright alumni) post-degree earnings by multiplying the number of employed alumni in each year (by sex and degree type), by average post-degree earnings estimated from the alumni survey (Table 7). The analysis projected earnings over a 25- year career for each alumnus cohort by applying earnings growth rates for holders of graduate degrees obtained from analysis of 2010-11Pakistan Social and Living Standards Measurement (PSLM) survey data. 40 TABLE 7: AVERAGE ANNUAL PRE- AND POST-DEGREE EARNINGS BY SEX AND DEGREE Degree Male Female N (total) N (with earnings) Average Annual Earnings (Rs.) N (total) N (with earnings) Average Annual Earnings (Rs.) Master’s Pre-degree 162 148 614,926 142 123 499,122 Post-degree 144 1,877,246 109 1,530,848 Difference 1,262,320 1,031,726 PhD Pre-degree 25 21 515,429 8 7 946,286 Post-degree 17 1,676,471 7 2,288,572 Difference 1,161,042 1,342,286 Source: MEP Survey d. Estimated an aggregate pre-degree earnings counterfactual in an identical fashion. The counterfactual used pre-degree earnings reported by respondents to the MEP survey of Fulbright alumni as a baseline and applied earnings growth rates for college graduates to project earnings over a 25-year career.41 e. Calculated the aggregate (over all alumni) “benefits” of a Fulbright degree in each year, and for each sex and degree combination, by subtracting projected aggregate pre-degree earnings from projected aggregate post-degree earnings in each year over a 25-year career. f. Converted “benefits” from Pakistani Rupees (PKR) reported by survey respondents to U.S. Dollars based on historic exchange rates reported by the State Bank of Pakistan. To convert future earnings (i.e., post-2012), the analysis used the 2012 exchange rate. g. Discounted the streams of costs and benefits to 2012 values at a discount rate of 12 percent (the rate the Pakistan Planning Commission uses for social programs)42 to account for differences in the value of money over time. This calculation produced the present (i.e., 2012) value of program 40 http://www.pbs.gov.pk/content/pakistan-social-and-living-standards-measurement-survey-pslm-2010-11-provincial-district 41 All Fulbright students held bachelor’s degrees before starting their Fulbright programs. 42 The Planning Commission uses a 12 percent discount rate for social sector projects. http://www.pc.gov.pk/CH-5.htm 38 costs and benefits in each year by sex and degree type. The total (present value) of costs and benefits is then the sum of the present value of costs and benefits over the 25-year career of alumni. 3. The ROI of the Fulbright Program, or for any subset of alumni, is then: where the present value of benefits and costs can relate to the entire program or any subset of alumni as appropriate. Table 8 summarizes program benefits, costs, and ROI by sex and degree type and for the program as a whole for a discount rate of 12 percent. Annex 9 describes the ROI calculation in greater detail. TABLE 8: FULBRIGHT PROGRAM BENEFITS AND COSTS (U.S. DOLLAR) Master’s Doctoral All (N=632) Male (N=312) Female (N=228) Male (N=67) Female (N=25) Nominal benefits and costs (U.S. Dollars) Benefits $246,835,926 $163,913,937 $48,273,409 $21,167,939 $480,191,211 Costs $34,867,816 $25,531,803 $12,422,584 $4,636,572 $72,822,203 Present value of benefits and costs (2012 U.S. Dollars) – 12% discount rate Benefits $84,509,662 $53,117,343 $14,109,468 $6,116,930 $157,853,403 Costs $56,331,168 $39,687,448 $19,825,541 $7,400,661 $115,844,157 ROI 50% 34% -29% -17% 36% The ROI for the program as a whole is 36 percent. Conclusions Even considering a very restricted set of quantifiable benefits (i.e., increases in lifetime earnings attributable to a Fulbright-sponsored degree) and costs (i.e., the costs to USEFP to support students in their degree programs), the program generated a return of 36 percent over its costs. Consistent with findings elsewhere regarding the value of graduate education, returns to a PhD are lower than returns to a master’s program. 43 The lower return to a PhD is a result of the longer time, and thus higher cost, required to obtain the degree. For instance, in the 2011-12 academic year, a two-year master’s degree cost US$117,310 while a four-year PhD cost US$186,144 or 59 percent more. Furthermore, the difference between pre-and post-degree earnings, i.e., the benefit of the degree, was lower for male PhDs than for male master’s degree holders. While female PhDs enjoyed a larger “benefit” than female master’s degree holders, it was not high enough to offset the higher cost relative to a master’s degree. 43 Alan Stark, Economic Studies and Policy Analysis Division, Department of Finance, Canada, “Which Fields Pay, Which Fields Don’t? An Examination of the Returns to University Education in Canada by Detailed Field of Study,” February 2007. http://www.oecd.org/social/labour/37578152.pdf 39 Although the number of female PhD alumni is small, they reported substantially higher pre- and post-degree incomes than male PhD alumni and the largest “benefit” of a degree of any sex/degree combination (Table 7). QUESTION 6 To what extent do Fulbright alumni contribute to achieving the goals and objectives of the U.S. Government’s public diplomacy strategy in Pakistan, as reflected in the U.S. Department of State’s Mission Resource Request (USDOS’s MRR), in Pakistan? The answer to this question examines the extent to which the Fulbright Program has fostered enduring linkages, formal academic collaboration, and cultural exchange between Pakistani and American institutions and individuals. All 337 respondents to the alumni survey reported maintaining some linkages with U.S. individuals or institutions. A large majority (77 percent male and 84 percent female) reported maintaining ongoing personal linkages with former classmates in the U.S.; 62 percent with the university in which they earned their degree, 60 percent with U.S. faculty members, 16 percent with non-academic institutions, and 8 percent with a U.S. university in which they did not earn their degree (Figure 15). FIGURE 15: ONGOING PERSONAL AND PROFESSIONAL LINKAGES (PERCENT) 9% 17% 22% 58% 67% 77% 8% 15% 16% 61% 58% 84% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Other U.S. university Non-academic institution Any linkage or collaboration Faculty members Degree awarding university Former classmates Percentage of responses Famle (150) Male (187) Source: Alumni Survey, MSI Fulbright alumni also reported forming lasting relationships outside the academic environment. Several FGD participants fondly remembered host families and personal relationships and others recounted that American friends have visited Pakistan to attend weddings of their former classmates. Figure 16 documents the nature of ongoing engagements and collaboration reported by survey respondents. A small majority (55 percent) of alumni reported maintaining some sort of professional engagement, i.e., professional networking (43 percent male and 43 percent female), research (27 percent male and 19 percent female), or curriculum development (11 percent male and 7 percent female). Social interaction, reported by 57 percent male 72 percent of female respondents, represented the most common form of ongoing engagement. 40 FIGURE 16: NATURE OF ONGOING LINKAGES AND COLLABORATION 11% 11% 27% 43% 57% 5% 7% 19% 43% 72% 0% 20% 40% 60% 80% 100% Civic causes Curriculum development Research Professional networking Social interaction Percentage of responses Female (136) Male (201) Source: Alumni Survey, MSI Participants in three of the six FGDs with alumni corroborated the importance of research collaboration, and participants in three of the six FGDs also mentioned getting advice from former professors. Seven of the 10 supervisors interviewed said they were not aware of any formal or informal linkages between their employee and former classmates or professors. However, one of the 10 supervisors said his employee maintained an informal linkage with a former classmate regarding technical issues and one said his employee had informal links with a former professor asking for advice regarding his/her teaching course. The alumni survey also asked participants for their opinions about the factors that could promote partnerships or linkages between Pakistanis and Americans. Having interested partners in the U.S. emerged as the most important factor (68 percent male and 62 percent female respondents) with a majority also mentioning funding (60 percent male and 57 percent female), and improved facilities in Pakistan (48 percent male and 61 percent female). Only 41 percent male and 35 percent female mentioned time off (Figure 17). 41 FIGURE 17: FACTORS AFFECTING PARTNERSHIPS WITH U.S. ENTITIES 41% 48% 60% 68% 35% 61% 57% 62% 0% 20% 40% 60% 80% 100% Time/leave from employer Inadequate research/teaching facilities Funding Interested U.S. partners Percentage of responses Female (150) Male (186) Source: Alumni Survey, MSI In the FGDs and other interviews, alumni and HEC officials generally agreed that travel, visa, and movement issues between Pakistan and the U.S. impede academic collaboration. The alumni further noted that the inability to work in the U.S. for a specified time after the scholarship constrained the linkages (and learning specialization) they were able to form. Regarding the inadequate research facilities reported by alumni, HEC officials stated that there is online access to the top 25,000 international journals. Moreover, there is “Open Access Instrumentation”, which means that a researcher can send an experiment to any specially equipped laboratory in the country to get an analysis done. HEC pays for the analyses for public sector universities and the private sector pays for the cost of the work. According to USEFP officials, they try to disperse Fulbright scholars throughout the U.S. to broaden the cultural exchange aspects of the scholarship program. Conclusions In terms of engagement and cooperation with the U.S., Pakistani Fulbright alumni have very positively moved in a direction that meets U.S. diplomacy goals and objectives by increasing the professional and cultural exchange between Pakistani and American institutions and individuals. All surveyed alumni reported maintaining ties in the U.S. that strengthen professional collaboration and appreciation of American cultural values. Almost two-thirds reported maintaining social ties and just over half maintained active professional collaborations with former classmates, universities, and faculty. Alumni have brought back new ideas and a new appreciation of American academic practices and a better understanding of shared cultural values. Although not entirely attributable to the Fulbright Program, alumni hold a much more favorable view of the U.S. and American people than other Pakistanis. In spite of these positive results, however, insufficient research facilities in Pakistan, limited funding for research, and travel restrictions for Pakistanis serve to restrict the Program’s potential contribution to professional, academic, and cultural exchange. 42 QUESTION 7 How relevant is the current Fulbright Program (e.g., mix of master’s and PhDs: U.S., regional, in￾country study, and fields of study) to the priority needs of the HEC? The FGDs asked alumni participants how they believed they were contributing to Pakistan’s development needs. A majority believed they were contributing through their work, and participants in all the FGDs agreed that teaching (and many Fulbright alumni are teaching full- or part-time in public and private colleges and universities) makes a major contribution. One participant stated, “Putting an 18-year-old in a better direction and changing mindsets is a major contribution.” Participants expressed their love of teaching and their desire to pass on what they’ve learned in the U.S. in terms of content, the way people are allowed to speak and question freely, the work ethic, and integrity. One participant spoke very positively about the ethical standards of the U.S. media and teaching these standards to the media in Pakistan, in spite of facing dangerous situations because of his work. Priority Needs The MTDF and personal interviews with HEC officials identified training top quality PhDs to serve as faculty in Pakistan’s universities as a top priority. In terms of sectors, the MTDF (2005-2010) identified science and technology as the highest priorities. It also identified agriculture as a priority need, but one that was already being addressed, according to a former HEC executive director. All HEC officials interviewed agreed that these represented HEC priority needs. Around 2007-08, the priority needs changed to the social sciences and humanities, as there were significant shortages in these sectors. All Fulbright-funded PhDs fell within these priority fields: 37 percent in science and technology and 63 percent in the social sciences and humanities. For the sciences, HEC prefers U.S. PhDs, but will accept training in other countries, e.g., Hong Kong, China and Germany. For the social sciences and humanities, HEC has a stronger preference for U.S.-trained PhDs. HEC officials interviewed expressed the belief that PhDs from the top U.S. universities contribute best to the quality of teachers and faculty in Pakistan’s universities, and one stated that the “key to quality is faculty”. Provision of Well-Qualified Faculty Pakistan needs about 15,000 Master’s in Sciences and about 15,000 more PhDs in the sciences alone.44 However, HEC officials reported that Pakistan is only producing about 800-900 PhDs annually in national universities. Eventually, about 0.5 million PhDs are required to accommodate plans to build a knowledge-based economy.45 The Fulbright Program is “very helpful” in meeting some of this demand for PhDs according to some of the HEC officials. However, the officials stated that they need some mechanism to address equity and regional disparity issues as well, per the objectives of the MTDF 2011-15. For instance, students from poor families find it difficult to pay for the GRE (a Fulbright application requirement). Consequently, fewer students apply for Fulbright scholarships than for local scholarships. According to USEFP officials, the organization has come up with an “integrated approach to inform, motivate and assist students of public sector universities/colleges in preparing and taking the admission tests”, including the GRE. HEC officials also felt that Fulbright is not well-advertised in Pakistan’s universities. According to USEFP, the organization advertises opportunities through “national and regional daily newspapers, covering all the geographical locations in Pakistan.” Alumni are also asked to share information about the program with their friends, colleagues, and community, and USEFP sends the announcements and application forms to all the Vice Chancellors/Rectors/Presidents of all the public and private universities in Pakistan, as well as to the relevant ministries of the GOP. Finally, USEFP staff performs outreach activities by visiting universities and through media appearances. 44 Pakistan Institute of Development Economics, Higher Education Commission and COMSATS University, 2005, Science and Technology Based Industrial Vision of Pakistan’s Economy and Prospects of Growth, p.21 45 Ibid., p.31 43 Mix of Master’s and PhDs USEFP determines the mix of master’s and PhDs. In the revised PC-1 for the Fulbright Program, HEC envisioned 265 PhDs and 550 master’s degree holders. As of the date of the evaluation, 231 PhDs and 563 master’s students had gone to the U.S. USEFP officials stated that the PhD programs were more expensive than anticipated. Therefore, funding constraints, as well as the availability of qualified candidates, led to the lower-than-planned number of PhDs. All HEC officials agreed that, if funding is available, HEC would like to continue partnering with USEFP, and they were pleased with the program overall. Conclusions The Fulbright Program is well-coordinated to support HEC’s strategy for higher education in Pakistan. Fulbright alumni increase the pool of available PhDs to fulfill the substantial anticipated demand of Pakistani universities in key fields of study. The fact that many alumni understand the importance of teaching to Pakistan’s economic and cultural development enhances the relevance of the Program. The fields of study and mix of degrees funded under the Program are generally consistent with the identified priorities of HEC. Furthermore, many stakeholders recognize the many facets of value attached to a degree from a U.S. university. 44 RECOMMENDATIONS The evaluation SOW states that the evaluation will provide recommendations for:  Improving the performance of the Fulbright Program (in terms of USAID’s Development Objectives and Intermediate Results). These are:  Improved access to high quality education;  Improved educational opportunities;  Increased access to scholarships; and  Improved public perception of the U.S.  Maximizing the degree to which it contributes to achieving the U.S. Government’s strategic objectives; and  Helping USAID enhance the impact of its higher education interventions by prioritizing and balancing its portfolio of activities. The evaluation concluded that the Fulbright Program has been very effective in supporting USAID and U.S. Government objectives. It has increased access to high quality education, improved educational opportunities, improved access to scholarships, contributed to cultural understanding and academic collaboration, supported HEC objectives for improving the quality of faculty in Pakistani universities, and been responsive to changing HEC priorities in terms of supported fields of study. Fulbright alumni also have very favorable perceptions of the U.S. and the American people. The evaluation team could not identify opportunities for improving performance in these areas. Assessing the program management was beyond the scope of this evaluation and therefore this evaluation cannot present any recommendations on that. However, a number of issues emerged in the process of doing the evaluation that suggest that a management evaluation might be worthwhile. 45 ANNEXES ANNEX 1: EVALUATION STATEMENT OF WORK 46 FULBRIGHT STUDENT PROGRAM EVALUATION STATEMENT OF WORK JANUARY 31, 2013 This publication was produced for review by the United States Agency for International Development. It was prepared by the Management Systems International (MSI) under the Monitoring and Evaluation Program. 47 FULBRIGHT STUDENT PROGRAM EVALUATION STATEMENT OF WORK Contracted Under No. GS-23F-8012H and Order No. AID-391-M-11-00001 Monitoring and Evaluation Program (MEP) DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 48 ACRONYMS FCR Findings, Conclusions and Recommendations HEC Higher Education Commission IMEC Independent Monitoring and Evaluation Contract IUCN International Union for Conservation of Nature GOP Government of Pakistan MNBSP Merit and Needs-Based Scholarship Program MOU Memorandum of Understanding MSI Management Systems International MTDF Medium-Term Development Framework ROI Return on Investment SOW Statement of Work UK United Kingdom UNDP United Nations Development Program USDOS United States Department of State USEFP United States Educational Foundation in Pakistan USAID United States Agency for International Development 49 I. BACKGROUND INFORMATION A. Identifying Information about the Project The Fulbright Student Program has offered scholarships in Pakistan since 1951, albeit with periods during which there were no U.S.-sponsored Fulbright activities, and provides Pakistani students an opportunity for personal development and international experience. In 2005, USAID began funding the Fulbright Program through a $19.5 million per year Memorandum of Understanding (MOU) with the U.S. Department of State (USDOS). Since 2005, the United States Government has invested $163.5 million in the Pakistan Fulbright Student Program, making it the largest Fulbright Student Program in the world. USAID/Pakistan implements most of its higher education activities through the Higher Education Commission (HEC) – the government agency responsible for tertiary education in Pakistan. The HEC is equivalent to what most countries refer to as the Ministry of Higher Education. The Government of Pakistan (GOP) established the HEC in 2002 with a mandate to improve and promote higher education, research and development in Pakistan. The HEC manages public policy and funding for Pakistan's universities. In particular, the HEC is the agency of government accountable for the higher education development budget, including funding for scholarships. The HEC has the statutory authority for public higher education in Pakistan and is therefore the principal agency with which USAID works to implement and manage education activities. In cases where USAID provides assistance to individual public universities it is always within the context of, and under a MOU with, the HEC. In addition to the Fulbright Student Program, USAID provides support to the HEC to implement other programs. These include budget support to the HEC, support for Merit and Needs-Based Scholarships (MNBS), and a Financial Aid Development activity (HEC-FAD). USAID, working through the HEC, also plans to implement a program to develop three university Centers of Excellence in Agriculture, Energy and Water Resources. The Fulbright Program fits synergistically within this portfolio and must be considered within the full context of USAID’s higher education program. This detailed Statement of Work (SOW) describes a performance evaluation of the Pakistan Fulbright Student Program. The Fulbright Student Program provides merit-based scholarships to Pakistani nationals to obtain advanced degrees from U.S. universities. The program aims to “promote international cooperation for educational and cultural advancement” and also to contribute to USAID’s strategic objectives in Pakistan and the priority objectives of Pakistan’s Higher Education Commission (HEC). USAID funds the Pakistan Fulbright Program through the Department of State Bureau of Educational and Cultural Affairs. The program is administered by the U.S. Educational Foundation in Pakistan (USEFP). USEFP is a bi-national commission. Half of its eight-member board is appointed by the U.S. Ambassador. The Public Affairs Officer (PAO) and Cultural Affairs Officer (CAO) are among these appointees, with the CAO normally serving as the Treasurer. The other four members are appointed by the Government of Pakistan, normally drawn from a list of prominent Fulbright and Humphrey alumni. (Two alternates – one from each country – are also appointed.) The program also operates in coordination with the HEC. B. Development Context 1. Problem or Opportunity Addressed The Fulbright Student Program in Pakistan awards merit-based scholarships for both master’s and doctoral level study in the U.S. to early and mid-career professionals with potential for leadership and high academic achievement. The Fulbright Program is intended to support awardees’ academic development and create mutual understanding between the people of Pakistan and the U.S. The Fulbright Program facilitates linkages between American and Pakistani academic institutions and promotes universities’ access to global education networks and job markets. For Pakistan’s universities, the program develops institutional capacity by 50 developing the expertise of potential faculty members in disciplines such as public administration, education, health, agriculture, environmental management, public policy and media and communications. A 2008 evaluation of USAID/Pakistan’s higher education portfolio46 concluded that the Pakistan Fulbright Program is making a major contribution to the HEC’s Medium Term Development Framework (MTDF), 2011- 2015, in terms of faculty development, quality improvement, research capacity, linkages with U.S. universities and mutual understanding. 2. Target Areas and Study Disciplines The Fulbright program selects graduate students (male and female) among applicants from all over the country. They are selected based in merit-cum-interview process. Fulbright interview panels are composed of two Pakistanis, who are Fulbright alumni volunteers, and two Americans, most often U.S. Embassy professional staff volunteers. Table 9 presents a summary of Fulbright awards by field between 2005 and 2011. Since 2005, the program has awarded 919 scholarships to Pakistani students in 19 disciplines. Approximately 374 grantees have graduated and returned to Pakistan, representing a return rate by year of 96-100 percent for those students who have completed degrees. TABLE 9: FULBRIGHT SCHOLARSHIP AWARDS, 2005-2011 Discipline 2005 2006 2007 2008 2009 2010 2011 Total % of total Business 8 5 16 9 12 9 8 67 7% Computer science 15 16 25 7 10 10 4 87 9% Economics 6 15 18 8 7 13 11 78 8% Education 5 1 6 4 2 6 10 34 4% Energy 1 7 8 1% Engineering 13 15 32 27 21 18 20 146 16% Environment 3 3 2 1 4 13 1% Finance 2 7 9 5 10 2 3 38 4% Fine arts 3 4 11 9 17 7 6 57 6% Health 8 6 8 15 10 7 7 61 7% Humanities 1 2 3 6 1% Journalism 2 3 10 6 6 13 7 47 5% Law 6 4 2 5 6 23 3% Literature 7 3 3 7 10 4 2 36 4% Natural science 4 4 6 8 12 7 3 44 5% Public policy 14 13 15 19 10 12 14 97 11% Social work 1 14 2 17 2% Social science 5 6 13 16 10 9 59 6% Water resources 1 1 0% Totals 101 102 175 142 147 128 124 919 100% Percent of total 11% 11% 19% 15% 16% 14% 14% 100% 46 Academy for Educational Development. 2008. Evaluation of USAID Higher Education Portfolio. [http://pdf.usaid.gov/pdf_docs/PDACM417.pdf] 51 C. Intended Results The Fulbright Student Program was established in 1946 and is governed by the legislative provisions of the Mutual Educational and Cultural Exchange Act of 1961, as amended, commonly known as the Fulbright-Hays Act. The Fulbright-Hays Act seeks to “increase mutual understanding between the people of the United States and the people of other countries” by “promoting international cooperation for educational and cultural advancement”. The program provides scholarships to “nationals of foreign countries in American schools and institutions of learning located in or outside the United States”. D. Approach and Implementation In Pakistan, USAID/Pakistan funds the Pakistan Fulbright Program through the Department of State Bureau of Educational and Cultural Affairs. The program is administered by the U.S. Educational Foundation in Pakistan (USEFP). USEFP is a bi-national commission. Half of its eight-member board is appointed by the U.S. Ambassador. The PAO and CAO are among these appointees, with the CAO normally serving as the Treasurer. The other four members are appointed by the Government of Pakistan, normally drawn from a list of prominent Fulbright and Humphrey alumni. (Two alternates – one from each country – are also appointed.) The program also operates in coordination with the HEC. E. Existing Data  Academy for Educational Development, Evaluation of USAID Higher Education Portfolio, 2008, Islamabad  United States Educational Foundation in Pakistan, Annual Program Report, FY 2011  Agreement Between United States Educational Foundation in Pakistan and Higher Education Commission of Pakistan on the Fulbright/HEC/USAID Scholarship Program for Pakistani PhD Students in the United States, 2005  Audit of Financial Statements for the year ended September 30, 2010  Auditors Report to the Board of Directors, the United States Educational Foundation in Pakistan (USEFP)  Various spreadsheets of information about scholarship awards  United States Educational Foundation in Pakistan, Management Manual  Fulbright Degree and Humphrey Programs Selection, 2010, September 23, 2011 (presentation)  Memorandum of Understanding Between the United States Agency for International Development and the United States Department of State to Transfer Economic Support Funds, 2010  Memorandum of Understanding Between the United States Agency for International Development and the United States Department of State, Bureau of Educational and Cultural Affairs to Transfer Economic Support Funds, 2009  Investment of Surplus Funds  Program Implementation Letter No. 11: Financial Assistance for the USAID-Pakistan Fulbright Program between the United States Agency for International Development (USAID) and the Higher Education Commission of Pakistan (HEC) of the Government of Pakistan (GOP), 2009  USAID/Pakistan Fact Sheet, Fulbright Scholarship Program  Program Report FY 2010, United States Educational Foundation in Pakistan  Medium-Term Development Framework II 2011-15, Higher Education Commission of Pakistan 52 II. RATIONALE FOR EVALUATION A. Purpose of Evaluation The Fulbright Student Program has been awarding scholarships to Pakistani students since 1951. While USAID expects to continue the program, it has substantial latitude to adjust program parameters (e.g., level of funding, priority fields of study, types and distribution of degrees supported). The purpose of this performance evaluation is to help USAID:  effectively tailor future programming to meet the U.S. Government’s development and public diplomacy objectives;  determine how well the program has been supportive of HEC’s overall strategy and priorities;  assess the past performance of the program and provide recommendations for improving future performance. The evaluation will thus make recommendations aimed at:  improving the performance of the Fulbright Program;  maximizing the degree to which it contributes to achieving the U.S. government’s strategic objectives; and  helping USAID enhance the impact of its higher education interventions by prioritizing and balancing portfolio of activities. B. Audience and Intended Use The results of this evaluation will be shared with USAID, USDOS, USEFP and other implementing partners. The evaluation report will be utilized to tailor future programming to most effectively meet the U.S. Government’s development and public diplomacy objectives. It will also be used to determine how well aligned the Fulbright program is to the needs of the Centers of Excellence. C. Evaluation Questions This section presents the core evaluation questions based on the discussion of the previous section. The questions are designed to address discrete aspects of the program to provide a view of the broad effects of the Fulbright Program. The basic questions and the associated explanations will provide the information necessary for the evaluation team to develop the specific questions that will guide data collection efforts. 1. To what extent do Fulbright alumni return to Pakistan and, for those who do return, how long do they stay in Pakistan? Explanation: Graduates of Fulbright Program are expected to return to Pakistan and make long term contributions. Given their advanced education and knowledge of American society, the scholars are expected to contribute to development of Pakistan. However, a small number may not complete their degree or not remain in Pakistan long enough to have a significant impact on development. The answer to this question will yield information about Fulbright alumni return to Pakistan and their persistence rates. The evaluation will only be able to document the extent to which alumni remain in Pakistan for the first few years after graduation. USAID recognizes that it may be difficult to attain data from alumni and that the evaluation team may have to rely on the data available from the USEFP and the HEC. 53 2. To what extent are Fulbright alumni productively employed in fields related to their academic preparation? Explanation: For various reasons, alumni may be unemployed, under-employed or work in fields unrelated to their degree. The data collected in response to this question will document employment rates and will characterize the nature of the employment. The answer to this question will help USAID and USDOS ensure that the program selects students who are committed to working in fields aligned with their academic preparation. Subject to feasible methods of collection, data to answer this question may include:  employment status (i.e. whether the scholar upon graduation is employed?);  sector of employment along with degree discipline (i.e. whether the sector of the graduates’ employment is related to the discipline in which the degree was attained?); and,  to the extent possible results disaggregated by degree type, discipline, sex and other available and relevant demographic variables. 3. To what extent do Fulbright alumni contribute to achieving the goals and objectives of the U.S. government’s development strategy in Pakistan as reflected in USAID’s relevant DO’s and IRs? Explanation: The U.S. Government is providing assistance to the people of Pakistan in order to achieve specific strategic objectives as articulated in USAID’s Results Framework (e.g. improved access to high quality education, increased access to scholarships and improved public perception of the US). This question will determine the extent to which the programs contribute to these objectives. 4. To what extent do the alumni of the Fulbright Program actualize their potential for and assume positions of leadership in academia, government, industry and business in Pakistan? Explanation: The evaluation will assess demonstration of leadership by Fulbright alumni through increases in salaries, promotions, publications and job position in employed fields. Interviews with employers and related stakeholders such as government and university officials could provide the necessary qualitative data (e.g., current management position) to address this question. 5. What is the Return on Investment (ROI) of the Fulbright Program? Explanation: Calculating the ROI of the Fulbright Program will be extremely challenging. The “investment” per student is simply USAID’s fully-loaded costs, including overhead, of attaining the degree. USAID however recognizes the difficulty in reliably and credibly monetizing the ROI of education programs. 6. To what extent do Fulbright alumni contribute to achieving the goals and objectives of U.S. government’s public diplomacy strategy in Pakistan, as reflected in USDOS’s Mission Resource Request (MRR), in Pakistan? Explanation: The US. Government is providing assistance to the people of Pakistan in order to achieve specific public diplomacy objectives, articulated in the USDOS’s MRR as follows:  “Objective No.7: Public Diplomacy and Strategic Communications”. “Performance Indicator No.2: Increased number of Pakistani understand that the United States and Pakistan share similar broad values and objectives and can work together to assist each other’s goals and objectives in the region.” A major trend in global higher education is a systematic pattern of trans-national collaboration. Based on the available research and the nature of the Fulbright Program, it is reasonable to expect that the 54 connections made between students, faculty members, and universities in both countries will be maintained and, in some cases, evolve into formal partnerships that can be sustained. Keeping in view USDOS’s broad public diplomacy objective, the evaluation will answer this question to the extent Fulbright Program alumni have maintained relationships with their respective institutions and have fostered links between Pakistani and U.S. schools. These links include developing social ties and mutual understanding, and collaboration in academia and research among U.S. and Pakistani students and scholars. It is expected that these relationships developed and links made directly contribute to Performance Indicator No. 2 mentioned above. This question will help determine if there are “spin-off” benefits from the Fulbright that yield enduring U.S.-Pakistani partnerships. The question may also explore potential mechanisms to foster and sustain partnerships. Evidence of sustainable partnerships may include formal university partnership agreements, joint research projects, follow-on faculty or student exchanges, and other indicators of ongoing collaboration. 7. How relevant is the current Fulbright Program (e.g., mix of master’s and PhDs; U.S, regional, in-country study and fields of study) to the priority needs of the HEC? Explanation: In the higher education sector, the U.S. government strategy aims to address the country’s development needs by working with and through the HEC to achieve mutually agreed upon development and diplomacy objectives. The Fulbright Program awards about 40 PhD scholarships to Pakistani students every year thus potentially contributing to the number of Pakistani university faculty with doctorates. This question will assess the level of the HEC’s commitment to the Fulbright Program and will help determine the most desirable balance in scholarship types (i.e. U.S., regional or in-country) to be supported by the U.S. Government. It will also provide information to determine the ideal balance of Fulbright awards among PhD and master’s scholarships. III. EVALUATION DESIGN AND METHODOLOGY A. Evaluation Design The evaluation will use a mixed-methods approach to systematically investigate and document the outcomes of the Pakistan Fulbright Program for the period 2004 to 2010. The evidence-based methodology will establish clear and defensible findings, conclusions, and recommendations and comply to the greatest possible extent with the USAID evaluation policy for performance evaluations. The evaluation team will probe the outcomes and results as deeply as feasible within time and resource constraints. To facilitate analysis, the team will collect and report data in a way that enables disaggregation across multiple dimensions including, but not limited to, sex, academic discipline, degree type, institution and sector of employment. Since the outcomes of a Fulbright award may not be evident for several years after a student graduates and enters the workforce, the evaluation will, to the extent that it is cost-effective to do so, examine historical data prior to 2004.47 The evaluation team will likely survey all scholarship recipients for which the USEFP can provide contact information and select from among the same pool for interviews and Focus Group Discussions (FGD). However, response and participation rates will likely decline with the length of time since an individual obtained a degree – because they may be more difficult to locate and their connection to the program will be less immediate. As the evaluation progresses, the evaluators will weigh the value of obtaining historical information against the difficulty and cost of doing so. The evaluation will rely on both quantitative and qualitative evidence to answer the evaluation questions and draw these data from both primary and secondary sources. Potential sources of secondary data include USEFP 47 USAID believes that the USEFP has reliable data back to 1991. 55 reports and databases, program assessments, and other research or data on education and employment in Pakistan. Primary sources include interviews with USEFP, HEC, USAID, USDOS, and employers of Fulbright alumni and surveys of and interviews with Fulbright alumni. The evaluation team will develop appropriate survey and other protocols to ensure the quality of primary data. B. Data Collection Methods The evaluation expects to employ the following data collection methods. Secondary Data The evaluation will rely heavily on secondary data collected by program management – USEFP and HEC. The USEFP will provide data on scholarships awarded, contact information for alumni, program expenditures and data collected from alumni. The evaluators will attempt to collect participation data for as far back as possible to understand trends in participation. The evaluation team will also identify and validate other sources of secondary data such as labor market surveys that may contribute to the evaluation. The evaluation team will also collect and review all relevant project documents from USAID, USEFP, HEC, USDOS and other sources identified during the evaluation. Relevant reports include the HEC’s Medium Term Development Framework (2011-2015), the 2008 Evaluation of USAID’s Higher Education Portfolio, USEFP annual program reports, program agreement documents, financial audit reports, program management manuals, Performance Management Plans, and evaluation and assessment reports. Primary Data The primary data will include both qualitative and quantitative data. Quantitative data will come from telephonic and mail surveys. Qualitative data will come from FGD and semi-structured interviews. Following is a brief explanation of these sources of data: Online/Telephonic – The quantitative portion of the evaluation will rely on a large-scale online/telephonic/survey of the universe of Fulbright Student Program alumni. The survey will depend on the extent to which USEFP maintains contact information for all Fulbright Student Program alumni and that it is available to the evaluation team. MEP evaluation team will design the survey questionnaire which Voice Tel Tech (VTT) will implement. VTT will pretest the questionnaire for both online and telephonic surveys. After pretesting VTT will send emails to all Fulbright alumni for whom email contacts are available, with a URL for the questionnaire. VTT will simultaneously implement telephonic survey. Focus group discussions – The majority of Fulbright awardees come from Pakistan’s major cities and it is likely that after returning to Pakistan they will be working in them as well. The evaluation team will therefore conduct FGDs with samples of Fulbright alumni in Islamabad/Rawalpindi, Lahore, and Karachi, and if possible in Faisalabad, Peshawar and Quetta. Based on the alumni records provided to the evaluation team, a random sample of 20 alumni will be selected from each city and will be invited to a FGD (with an expectation that 8-12 will participate). Semi-structured interviews – The evaluation team will conduct semi-structured interviews with USEFP, HEC, USAID, USDOS, and employers of Fulbright scholarship alumni48 . 48 Answer to the evaluation questions involving employers will come from interviews with senior staff members of those organizations who have worked closely with them. These interviews will depend on two things: whether contact information is available; and second, permission from the employee (who in this case would be Fulbright alumni) may be required. If this information is not readily available with USEFP, it implies that it can possibly come from survey data. In the latter case, field work will have to wait until such information is available to the team. 56 C. Data Analysis Methods Data will be disaggregated, to extent possible, by discipline, sex and other economic (e.g., employment status/salaries) and non-economic indicators. Data analysis will include a combination of frequency-tables and cross-tabulation on the following aspects of the program:  number of Fulbright applicants;  number of Fulbright awards;  number of degrees attained;  percentage of Fulbright graduates who return to Pakistan;  number of years that Fulbright alumni have remained in Pakistan;  for alumni who are no longer in Pakistan, country of destination; and  sector of employment, cross-tabbed with degree discipline (i.e. is the sector of employment related to the discipline in which the degree was attained) and leadership (as defined in evaluation question 4). The quantitative information from FGDs with alumni and semi-structured interviews with employers will be used to construct frequency and crosstab tables. The qualitative information will be subject to content analysis for providing answers to the above evaluation questions and triangulating the information from the surveys. The evaluation will also include a return on investment analysis, to the extent possible as explained in evaluation question 5. The Getting to Answers table in Error! Reference source not found.1 summarizes how the evaluation team is expected to address the evaluation questions. The table represents a first draft of the approach that the evaluation team will refine during the initial team planning meeting. D. Methodological Strength and Limitations The evaluation methodology relies on triangulation of sources and methods to ensure validity and reliability of the results. The scope of this evaluation is limited to answering the evaluation question and to the extent that they can be operationally defined and data are available. The evaluation will use a two pronged approach to increase the response rate of survey by using both online and telephonic survey. The methods proposed for collecting and analyzing evaluation information are potentially subject to selection bias. Selection bias occurs when the subjects of surveys or interviews are not representative of the population of interest. In this case, selection bias is most likely to result from online survey respondents self-selecting49 themselves. Fulbright alumni who are not complying with their agreements (e.g., have not returned to Pakistan, are not working) will be less likely to respond to the survey. Similarly selection bias may also occur in selection of Fulbright alumni for FGDs, where participation of alumni will depend on their availability50. Selection of alumni for FGDs may also be limited to the cities visited by the team for fieldwork: the selected sample may not be representative of the population. The evaluation relies heavily on the timely response of Fulbright alumni and employers to our survey. Low response rates for these two groups due to their professional and personal time commitment will be outside M&E project’s control. This limitation can affect the representativeness of the sample and reliability of results. To counter this limitation the evaluation will use multiple data collection methods including online, phone and mail survey as explained in the data collection section. 49 Bias caused by self-selection is beyond the control of the evaluation team. However, to illustrate the representative of the sample data, evaluation will compare the characteristics of sample survey with population characteristics for available indicators. 50 It has been the experience of evaluation team in evaluation of MNBSP that alumni turnout for GDs was low despite several invitations from universities. 57 IV. PRODUCTS A. Deliverables The deliverables for this evaluation are 1) final SOW, including final evaluation questions, clear methodology and approach for each component of the evaluation linked with the evaluation questions, and sampling methods and confidence levels; approved by USAID following the TPM 2) survey instruments and interview guides approved by USAID/PMU, 3) a debriefing presentation to USAID, 4) a draft evaluation report, and 5) a final evaluation report. Note that field work will not commence until deliverables 1 through 2 are completed. B. Reporting Guidelines The final report will be delivered by the team leader to USAID in printed and electronic forms along with the annexes mentioned in this SOW. The report will follow standard guidelines as laid out in Appendix 1 of USAID’s Evaluation Policy (attached as Appendix 2 here) and ADS.51 The evaluation report will follow the structure given below:  Title page;  Table of Contents;  Tables of tables and figures;  Acknowledgements or preface (optional);  Executive summary: the executive summary will be 3-5 pages in length that summarizes key points (project purpose and background, key evaluation questions, methods, findings etc.)  Introductory chapter;  The Development Problem and USAID’s Response: this section will describe the development problem USAID wanted to address. This will include USAID’s response to the problem, the development hypothesis and project implementation;  Purpose of the Evaluation: this section will include the purpose of evaluation and state all evaluation questions;  Research Design and Evaluation Methodology: a written design which includes key questions, methods, main features of data collection instruments, and data analysis plan;  Findings and Conclusions: this section will include findings and conclusions for each evaluation question;  Recommendations;  References; and  Annexes  The Evaluation Scope of Work;  Any “statements of differences” regarding significant unresolved difference of opinion by funders, implementers, and/or members of the evaluation team;  All tools such as questionnaires, checklists, survey instruments, discussion guides; and 51 www.usaid.gov/policy/ads/200/220mab.pdf. 58  All sources of information properly identified and listed. V. TEAM COMPOSITION The evaluation will require the following areas of expertise: 1. A team leader (short-term consultant) with experience in program evaluation, student support programs, tertiary education, and development needs in Pakistan. He/ She will guide all tasks listed under section VI of SOW and will be responsible for guiding the evaluation team members listed below. The team leader will be responsible for all deliverables and most importantly, will author and present the draft and final reports; 2. A full time evaluation manager with experience in program evaluation. The evaluation manager will be a full-time staff of M&E Project; 3. A full-time co-manager with experience in program evaluation will assist team leader/manager in all the functions mentioned above. He/ She will be a part of team for field visits and with the consent of the team leader contribute to writing parts of the report; 4. A full-time MEP director of survey unit with experience in sampling and launching surveys, conducting FGDs for coordinating the online/telephonic. 5. An education expert (short-term consultant) who will provide advice and assistance in developing data collection instruments, interpreting results, and writing specific sections of the evaluation report. The consultant will also analyze the quantitative and qualitative data from the survey of alumni, employers, GDs and semi-structured interviews. In addition to the above team members, MSI will contract out a firm to conduct online/telephonic survey. All evaluation team members will be required to provide a written disclosure of conflict of interest. VI. EVALUATION MANAGEMENT A. Logistics The evaluation will proceed in four main phases – planning, online/telephonic/mail survey, field work and reporting. This section describes the general tasks in each of these phases. 1. Evaluation planning – During the planning phase, MEP will develop a detailed SOW based on a draft SOW provided by USAID. The detailed SOW will serve as the work plan for the evaluation. Once USAID approves the detailed SOW, MEP will request background documents and data from USAID and the implementing partners. For this evaluation, it will be critical that MEP receive contact information for Fulbright alumni from USEFP as soon as possible. 2. Online/telephonic survey – As soon as USAID approves the detailed SOW, MEP will begin to develop and deploy the online/telephonic of Fulbright alumni. 3. Field work – The evaluation field work will begin after the online/telephonic survey is launched. Field work consists of the document review, site visits, semi-structured interviews, FGDs and secondary data collection described in the methodology section of this SOW. At the beginning of the field work the evaluation team will conduct a team planning meeting to plan the evaluation field work, refine field data collection methods and develop interview guides and other field protocols. Before starting data collection, the evaluation team will meet with USAID to clarify evaluation objectives, review evaluation questions, discuss details of the field work, and obtain approval for the field work plan. The evaluation team will also meet with the implementing partners (e.g., USDOS, HEC and USEFP) early in the field work phase to gain a deeper understanding of the program prior to beginning data collection. Field work will require approximately four weeks. 59 4. Reporting – At the conclusion of the field work, the evaluation team will prepare and deliver a debriefing presentation on FCR to USAID and to implementing partners or other parties with USAID approval. The evaluation team will incorporate comments from the presentation(s) into a draft report. After a thorough technical review, MEP will deliver the draft report to USAID – and to implementing partners if appropriate - for review and comment. Once MEP receives comments on the draft report, it will ask the evaluation team to incorporate comments, send the report to the MEP home office for a final technical review, editing and branding and then deliver the final report to USAID. 60 B. Scheduling The tentative evaluation schedule is shown in the form of a Gantt chart below. MEP will start the evaluation process with a document review on December 31, 2012. MEP will submit a draft report to USAID on April 1, 2013 and the final report on April 22, 2013. TABLE 10: TENTATIVE EVALUATION SCHEDULE Activity Dec. January February March April May w4 w 1 w 2 w 3 w 4 w 1 w 2 w 3 w 4 w 1 w 2 w 3 w 4 w 1 w 2 w 3 w 4 w 1 w 2 w 3 Document Review (Dec 31–Jan 8) TPM (Jan 9–1852) Finalizing evaluation SOW and vendor SOW for data collection (Jan 23–Feb 1) Field Survey (Feb 4–Mar 4) Team Field Visit (Mar 4–Mar 15) Analysis/FCR Workshop (Mar 18–Mar 25) Debriefing Presentation/Report Writing (Mar 26–Apr 5) Internal Review and Revision (Apr 6–16) Branding and Editing (Apr 17–24) Draft Report submission to USAID (Apr 26) Comments from USAID (May 6) Final Report submission to USAID (May 18) 52 This includes three days of field holidays due to security situation in Islamabad. 61 C. Budgeting The following table highlights Level of Effort (LOE) of each evaluation team member. TABLE 11: LEVEL OF EFFORT OF EVALUATION TEAM MEMBERS Tasks Level of Effort (days) Team Members (CCNs) Team Leader (STTA) Evaluation Manager (LTTA) Co-Manager LTTA Education Specialist (STTA) Director Survey (LTTA) Stage I  Developing Draft SOW, Preliminary Meetings and Hiring 7 Stage II:  Document Review 5 - - 5 - Stage III: (Islamabad)  Fulbright Evaluation Orientation, information sharing and Team Plan Meeting (TPM) 4 4 2 4 2 Stage IV:  Development of questionnaires, pre-test, and online survey of host universities and telephonic/mail survey of alumni 3 2 2 3 10 Stage V:  Field Work 12 12 12 12 - Stage VI:  Analysis, FCR workshop, Debriefing and Draft Evaluation Report 16 10 10 14 4 Stage VII:  Final Products 5 5 2 - Total LOE Days 45 40 26 40 16 62 APPENDIX 1: INITIAL GETTING TO ANSWERS TABLE Data Collection Evaluation Question Type of Answer/Evidence Methods Source Sampling/ Selection Data Analysis Methods 1. To what extent do Fulbright alumni return to Pakistan and, for those who do return, how long do they stay in Pakistan? Quantitative/statistical – number or percentage of Fulbright alumni who return to Pakistan by length of time they stay and disciplines in which they are employed. Survey; document review; FGDs Online/telephonic/mail survey of Fulbright awardees; FGDs with Fulbright alumni; project records if available Online/telephonic/ mail survey of universe of Fulbright alumni % of students who return by year and discipline, % of students staying in Pakistan by length of stay and discipline. Account for those who do not complete their degrees or who return to Pakistan for only a short period. 2. To what extent are Fulbright alumni productively employed in fields related to their academic preparation? Quantitative/statistical – number or percentage of Fulbright alumni currently employed by position, field, and preparation (e.g., degree and field). Survey; document review; FGDs; interviews with employers Online/telephonic/mail survey of Fulbright awardees; project records if available; FGDs with Fulbright alumni; employers Online/telephonic/ mail survey of universe of Fulbright alumni; purposive sampling Document employment status, field of work, and position, reasons for not working in relevant field 3. To what extent do the alumni of the Fulbright contribute to achieving the goals and objectives the U.S. government’s development strategy in Pakistan as reflected in USAID’s relevant DOs and IRs? Descriptive Semi-structured interviews; survey; FGDs Interviews with USAID personnel, USEFP and HEC officials, and Fulbright alumni; online survey of Fulbright alumni; FGDs with Fulbright alumni Purposive sample of USAID and HEC personnel; online survey of universe of Fulbright alumni Use content analysis of interviews to determine how Fulbright contributes to goals and objectives. Support with quantitative data from survey where applicable. 63 Data Collection Evaluation Question Type of Answer/Evidence Methods Source Sampling/ Selection Data Analysis Methods 4. To what extent do the alumni of the Fulbright Program actualize their potential for and assume positions of leadership in academia, government, industry and business in Pakistan? Statistical Semi-structured interviews; surveys; FGDs Interviews with employers of Fulbright alumni; online/telephonic/mail survey of Fulbright alumni; FGDs with Fulbright alumni Random sample of employers (sample size?); online/telephonic/mai l survey of universe of Fulbright alumni Develop indicators of leadership and/or performance. % of alumni by index value and discipline/sector 5. What is the ROI of the Fulbright Program? Financial analysis – Return on Investment, Cost Benefit/Effectiveness Analysis Document review; literature review; survey Cost data from program records; framework for return to education from literature; estimates of monetary benefits from survey of alumni Online/telephonic/ma il survey of universe of Fulbright alumni If it is possible to determine the monetary return to investments in education, then calculate ROI. Otherwise, calculate cost per output (e.g., graduate, employed graduate, etc.) 6. To what extent do Fulbright alumni contribute to achieving the goals and objectives of U.S. government’s public diplomacy strategy in Pakistan, as reflected in USDOS’s MRR, in Pakistan? Statistical and descriptive Surveys; semi￾structured interviews; FGDs Surveys and interviews with participating Pakistani institutions and Fulbright alumni; FGDs with Fulbright alumni Random sample of Pakistani institutions; online/telephonic/mai l survey of universe of Fulbright alumni Describe (quantitatively and qualitatively) the number and nature of interactions between Fulbright alumni, the institutions in which they work, and U.S. institutions and individuals 7. How relevant is the current Fulbright Program (e.g., mix of master’s and PhDs; U.S, regional, in-country study and fields of study) to the priority needs of the HEC? Descriptive/narrative Semi-structured Interviews Interviews with USEFP, HEC, and public university officials Sample of convenience depending on travel requirements and other movements of the evaluation team. Content analysis of interview findings 64 APPENDIX 2: CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT (USAID EVALUATION POLICY)  The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why.  Evaluation report shall address all evaluation questions included in the statement of work.  The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer.  Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an annex in the final report.  Evaluation findings will assess outcomes and impact on males and females.  Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.).  Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence.  Sources of information need to be properly identified and listed in an annex.  Recommendations need to be supported by a specific set of findings.  Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. 65 ANNEX 2: REFERENCES Academy for Educational Development, 2008, Evaluation of USAID Higher Education Portfolio Agreement between USEFP and Higher Education Commission of Pakistan on the Fulbright/HEC/USAID Scholarship Program for Pakistani PhD Students in the United States, December 2005 Bashir, Malik Fahim, et al, 2011, “Estimating Private Rate of return on Higher Education in Pakistan,” Review of Economic and Business Studies, Volume 4, Issue 1, pp. 179-197 Colclough, Christopher, et al, April 2009, “The Pattern of Returns to Education and its Implications”, RECOUP Policy Brief Number 4 Hayward, Fred M., [not dated] “Higher Education Transformation in Pakistan: Political and Economic Instability” Higher Education Commission, Pakistan Institute of Development Economics and COMSTECH, 2005, Science and Technology Based Industrial Vision of Pakistan’s Economy and Prospects of Growth Higher Education Commission, Medium Term Development Framework 2005-2010 Higher Education Commission, Medium Term Development Framework 2011-2015 Memorandum of Agreement between USAID and USDOS to Transfer Economic Support Funds, April 2005 Memorandum of Agreement between USAID and USDOS to Transfer Economic Support Funds, April 2010 Memorandum of Agreement between USAID and USDOS, Bureau of Educational and Cultural Affairs to Transfer Economic Support Funds, May 2009 Rahman, Atta-ur, 2013, “Silent Revolution in Higher Education” USDOS and USAID, Strategic Plan 2004-2009 USDOS and USAID, Strategic Plan 2007-2012 USAID, January 31, 2013, Fulbright Student Program Evaluation Statement of Work USEFP, Annual Program Report FY 2011 USEFP, Program Report FY 2010 Websites USDOS http://www.state.gov/ USAID/Pakistan http://transition.usaid.gov/pk/ USEFP http://www.usefpakistan.org/ HEC http://www.hec.gov.pk/ 66 ANNEX 3: FULBRIGHT ALUMNI SURVEY QUESTIONNAIRE Evaluation and Survey Unit Evaluation of Fulbright Student Program (Questionnaire for Alumni) Version 25-2-2013 (English) February 2013 An evaluation of the USEFP Fulbright Program is being conducted. You have been chosen to participate in this survey because you received a Fulbright scholarship. Your responses to this survey will help to improve the Program. Your response is very important. We have sent the questionnaire to Fulbright recipients and it is important that we hear from everyone who received the questionnaire in order for our results to be meaningful. Answering the questions will take about 20 minutes. Please be assured that your answers will be kept strictly confidential. We will not share the survey data with anyone and will not identify individual responses in reports. Thank you in advance for your assistance with this important survey. Note:  Circle / check / click one option only unless multiple answers are allowed by the question 67 SECTION A: BACKGROUND INFORMATION Q1. What is your gender? [Kindly observe and note; don’t ask] 1. Male 2. Female Q2. Which is your domicile province/ Territory? 1. Punjab 2. Sindh 3. KP 4. Balochistan 5. Gilgit-Baltistan/Northern areas 6. FATA 7. Azad Jammu-Kashmir 8. Capital Territory/ICT Q3. What is the type of city you currently reside in? 1. Major Metropolitan; (Provincial headquarter / capital city Islamabad) 2. Large City / Divisional headquarter (but not Provincial headquarter / capital city Islamabad) 3. Small town / cities (District headquarter and Tehsil headquarter / small town) 4. Rural/ remote areas/ far flung areas (as classified by 1998 census as rural areas) Q4. What is the type of city of your origin? 1. Major Metropolitan; (Provincial headquarter / capital city Islamabad) 2. Large Cities / Divisional headquarter (but not Provincial headquarter / capital city Islamabad) 3. Small town / cities (District headquarter and Tehsil headquarter / small town) Q5. What is your age? [in years]_________________________ Q6. Which degree have you received with financial assistance from the Fulbright Program? (Check all that apply) 1. Master’s program Year of Degree Completion:______________ 2. PhD program Year of Degree Completion: ______________ Q7. What was the field of study of your Fulbright funded degree? [Do not read each choice, check the relevant option] 1. Computer Sciences/Information Technology 2. Engineering/Civil engineering/Electrical engineering/Software engineering/Computer engineering/Chemical engineering/Industrial engineering/Telecommunication 3. Economics/Development economics/Development studies 4. Political science/Sociology/Psychology/International relations/Social work/Archaeology/Anthropology/History (Non-U.S.) 5. Public Policy/Public administration 6. Finance/Business administration/Marketing/Management/Human Resource Management 7. Public/Global Health/Epidemiology/Biological sciences/Entomology/Medical sciences/Micro Biology/Environmental studies/Physical Therapy 8. Law 68 9. Education/ Special education 10. Art/Architecture/Media/Art history/Films studies/Theatre/Creative writing 11. Journalism/Mass communication/Communication/Information sciences 12. Mathematics/Physics/Chemistry 13. English Language/English literature/Literature (Non-U.S)/Language (Non-U.S)/Linguistics/American Literature/TEFL/Applied Linguistics 14. Other (Agriculture/Occupational Therapy/Religious studies/ Statistics/Technology Management/Transportation Planning/Urban Planning/ etc. SECTION B: EMPLOYMENT / SELF-EMPLOYMENT BEFORE FULBRIGHT SCHOLARSHIP Q8. What was your primary employment / academic status in the three months before you were accepted into the Fulbright degree program? [Check only one which you consider as primary status and then follow instructions to the next question] 1. Employed having a fixed monthly salary [go to Q.9 ] 2. Self-employed with variable monthly income [go to Q.10] 3. Unemployed or looking for employment [go to Q.12] 4. Student at Bachelor, Master or PhD level [go to Q.12] Q9. Which position did you hold in your organization when you left your job to pursue the Fulbright funded degree? 1) Top or senior management, government servant (Grades 20 and Above), university vice chancellor or full professor 2) Mid-level professional, experienced specialist , government servant (Grades 18-19), associate professor 3) Entry level professional, specialist , government servant (Grades 17 or less), assistant professor, lecturer Q10. What was your approximate monthly salary or income from your last job / self-employment before you went on the Fulbright scholarship (monthly salary/income in rupees from the job or self-employed professional income only)? Rs. __________________ Q11. How long did you work for this organization /remained self-employed before you went on Fulbright program? 1. Less than one year 2. More than 1 year but less than 2 years 3. More than 2 years but less than 3 years 4. More than 3 years but less than 4 years 5. More than 4 years but less than 5 years 6. More than 5 years but less than 6 years 7. More than 6 years but less than 7 years 8. More than 7 years Q12. What was your total work experience in years with respect to your job/ self-employment before you left to pursue Fulbright funded degree? 1. Less than one year 69 2. More than 1 year but less than 2 years 3. More than 2 years but less than 3 years 4. More than 3 years but less than 4 years 5. More than 4 years but less than 5 years 6. More than 5 years but less than 6 years 7. More than 6 years but less than 7 years 8. More than 7 years. Q13. How many papers did you publish in the form of the following, before receiving the Fulbright scholarship [write 0 if no publication for each of the category]? 1) International journals with peer review (No.)_____ 2) National journals with peer review (No.) _____ 3) Technical/scholarly publication with peer review (No.) ____ Q14. Were you a contributor of newspaper/magazine articles before receiving Fulbright scholarship? 1) No 2) Occasional (less than once a month) 3) Frequent (at least once a month) Q15. Did you hold any leadership position in the following types of organizations in the last five years before going for Fulbright scholarship? [Check all that apply] 1. Civic Organizations 2. Trade Associations 3. Technical Societies 4. Other organization/association/society SECTION C: EMPLOYMENT / SELF-EMPLOYMENT AFTER FULBRIGHT SCHOLARSHIP (CURRENT) Q16. What is your current primary employment or academic status? 1) Employed having a fixed monthly salary on a contract of at least one year [go to: Q17] 2) Self-employed with variable monthly income[go to: Q22] 3) Unemployed or looking for employment [go to: Q28] 4) Student at Master and PhD level [go to: Q28] 5) Post-doc [go to: Q28] Q17. [If employed having a fixed monthly salary in Q16], what is the type of organization you are currently working for? 1. Government including public sector research and educational institutions 2. Private sector (for profit) including private sector research and educational institutions 3. Non-profit 4. United Nations or Multilateral Q18. [If employed having a fixed monthly salary in Q16], to which of the following fields is your employment most closely related? 1) Computer Sciences/Information Technology 2) Engineering/Civil engineering/Electrical engineering/Software engineering/Computer engineering/Chemical engineering/Industrial engineering/Telecommunication 3) Economics/Development economics/Development studies 70 4) Political science/Sociology/Psychology/International relations/Social work/Archaeology/Anthropology/History (Non-U.S.) 5) Public Policy/Public administration 6) Finance/Business administration/Marketing/Management/Human Resource Management 7) Public/Global Health/Epidemiology/Biological sciences/Entomology/Medical sciences/Micro Biology/Environmental studies/Physical Therapy 8) Law 9) Education/ Special education 10) Art/Architecture/Media/Art history/Film studies/Theatre/Creative writing 11) Journalism/Mass communication/Communication/Information sciences 12) Mathematics/Physics/Chemistry 13) English Language/English literature/Literature (Non-U.S)/Language (Non-U.S)/Linguistics/American Literature/TEFL/Applied Linguistics 14) Other (Agriculture/Occupational Therapy/Religious studies /Statistics/Technology Management/Transportation Planning/Urban Planning/ etc. Q19. [If employed having a fixed monthly salary in Q16], what position do you currently hold in your organization? 1) Top or senior management, government servant (Grades 20 and Above), university vice chancellor or full professor 2) Mid-level professional, experienced specialist , government servant (Grades 18-19), associate professor 3) Entry-level professional, specialist , government servant (Grades 17 or less), assistant professor, lecturer Q20. [If employed having a fixed monthly salary in Q16], how would you compare your current position with the position you held before receiving the Fulbright scholarship? 1) Lower 2) About the same 3) Higher 4) Much higher Q21. [If employed having a fixed monthly salary in Q16], how many times have you been promoted in your job since obtaining Fulbright scholarship? 1) Never 2) Once 3) Twice 4) More than twice Please skip Q.22, if employed having a fixed monthly salary in Q16 Q22. [If self-employed in Q 16], what type of organization or business is your main source of income (Only for currently self-employed) ? 1) Government including public sector research and educational institutions 2) Private sector (for profit) including private sector research and educational institutions 3) Non-profit 4) United Nations or Multilateral Q23. Where do you currently work? 1) Punjab 71 2) Sindh 3) KP 4) Balochistan 5) Gilgit-Baltistan/Northern areas 6) FATA 7) Capital territory/ ICT 8) Azad Jammu Kashmir 9) Overseas Q24. Are you employed / self-employed in a field that is related to your degree under Fulbright program? 1) No [go to Q.25] 2) Yes [go to Q.26] Q25. What are the reasons for not working in a field related to your academic training? (Check all that apply) 1) Difficult to find a job in the field 2) Did not learn relevant skills 3) Employers did not view the degree as high quality 4) Changed careers 5) Earning potential in field was too low 6) Other Q26. What is your approximate monthly salary or income from your current job / self-employment (Monthly salary/income in rupees from the job or self-employed professional income only)? Rs. ________________________________________________ Q27. How long have you been employed with your current organization /remained self-employed after you completed your Fulbright funded degree? 1. Less than one year 2. More than 1 year but less than 2 years 3. More than 2 years but less than 3 years 4. More than 3 years but less than 4 years 5. More than 4 years but less than 5 years 6. More than 5 years but less than 6 years 7. More than 6 years but less than 7 years 8. More than 7 years. Q28. What is your total work experience in years with respect to your job/ self-employment after you completed your Fulbright funded degree? 1. Less than one year 2. More than 1 year but less than 2 years 3. More than 2 years but less than 3 years 4. More than 3 years but less than 4 years 5. More than 4 years but less than 5 years 6. More than 5 years but less than 6 years 7. More than 6 years but less than 7 years 8. More than 7 years. 72 Q29. How many papers have you published in the form of the following after receiving Fulbright scholarship [write 0 if no publication for each of the category]? 1) International journals with peer review (No.) _____ 2) National journals with peer review (No.) _____ 3) Technical/scholarly publication with peer review (No.) ____ Q30. Have you been a contributor of newspaper/magazine articles since obtaining Fulbright funded degree? 1) No 2) Occasional (less than once a month) 3) Frequent (at least once a month) Q31. Have you held any leadership position in the following types of organizations since obtaining Fulbright funded degree? [Check all that apply] 1. Civic Organizations 2. Trade Associations 3. Technical Societies 4. Other organization/association/society SECTION D: FULBRIGHT LINKAGES Q32. Do you have any formal/informal professional linkages in the U.S with the following: [Check all that apply] 1. University from where you obtained Fulbright funded degree 2. Another university in US 3. Non-academic institution including business/industry in U.S. 4. Former fellow students 5. Faculty members 6. Other Q33. Are you currently engaged in the following activities in cooperation with an individual or institution in the U.S? [Check all that apply] 1. Research 2. Developing Curriculum or teaching aides 3. Professional networking 4. Pursuit of civic causes 5. Social interaction 6. Other Q34. Which of the following do you think are needed to establish partnership with a U.S individual or institution? [Check all that apply] 1. Funding 2. Interested Partners in U.S 3. Time/ leave from current employer 4. Improved research/ teaching facilities in Pakistan 5. Other Q35. In what way has your Fulbright experience altered your understanding of the following? No Change Perception is now less favorable Perception is now more favorable 73 Research methods 1 2 3 Teaching methods 1 2 3 U.S. economic system 1 2 3 U.S. political system 1 2 3 U.S. foreign policy 1 2 3 U.S. philanthropy 1 2 3 American culture 1 2 3 SECTION E: USAID’S DEVELOPMENT OBJECTIVES Q36. Had you not received the Fulbright scholarship, would you have still been able to enroll in this degree program in the US? 1. Yes, I would have still enrolled in the degree without Fulbright scholarship [go to:Q.37] 2. No, I would not have been able to enroll in the degree without Fulbright scholarship [go to:Q.38] Q37. How would you have financed your education if you had not received Fulbright scholarship? [Check all that apply] 1. Other Scholarship 2. Personal savings 3. Students loans 4. Personal Loans 5. Family Funding 6. Part-time Job 7. Other Sources Q38. What is your perception of the United States as a country? 1. Very favorable 2. Somewhat favorable 3. Somewhat unfavorable 4. Very unfavorable 5. Don’t know/Unwilling to answer Q39. What is your perception of the American people? 1. Very favorable 2. Somewhat favorable 3. Somewhat unfavorable 4. Very unfavorable 5. Don’t know/Unwilling to answer SECTION F: POST GRADUATION RESIDENCE STATUS Q40. Have you completed your residency requirement under the Fulbright Program? 1) Yes 2) No Q41. After completing your Fulbright degree, how many years have you stayed in Pakistan? [Write “0” if the alumni has not returned ] (Number of years) _________ Q42. Do you intend to remain in Pakistan? 74 1) Yes [go to Q.45] 2) No [go to Q.43] Q43. [If ‘No’ in Question 42], do you intend to complete your residency requirement in Pakistan before going abroad? 1) Yes 2) No Q44. [If ‘No’ in Question 42], what are your reasons for leaving Pakistan?(Check all that apply) 1) Difficult to find a job related to the field 2) Did not learn skills (during Fulbright program) relevant to the local job market 3) Career change 4) Earning potential less than expectation 5) Have not found any job in Pakistan 6) Poor economic situation in Pakistan 7) Personal reasons 8) Other Q45. [If ‘Yes’, in Q42], what are your reasons for continuing to remain in Pakistan after completion of your residency requirement? (Check all that apply; only for those who said yes in Q 42) 1) Job Satisfaction 2) Income 3) Family reasons 4) Serving the country 5) Health condition 6) Other SECTION G: PERMISSION TO MEET WITH YOUR EMPLOYER/ CONSENT TO PARTICIPATE IN FGD We would like to have a short meeting with your supervisor regarding how your performance and contribution at work was enhanced by your Fulbright experience. It is very important that we do so to assist the Fulbright program in better meeting the needs of Fulbright scholars in the future. Q46. Do you agree to MSI contacting your employer/supervisor? 1. Yes 2. No In addition, we would also like to conduct a Focus Group Discussion (FGD) with the Fulbright alumni in the first and second week of March 2013 to gather qualitative information regarding the Fulbright Program. Q47. Would you like to participate in the Focus Group Discussion (FGD)? 1. Yes [go to Q.48] 2. No -> [End Survey] Q48. [If ‘Yes’ in Question 47], Kindly suggest your time of convenience? 1) 9 am-12 pm 2) 12 pm-3 pm 3) 3 pm-6 pm 4) 6 pm-9 pm 75 Q49. [If ‘Yes’ in Question 47], Do you have any preference for day of the week? If yes, which day? ----------------------- Q50. [If ‘Yes’ in Question 47], Kindly provide your preferred contact Number or an email address? Contact No. /Email Address: ____________________ 76 ANNEX 4: POST-STRATIFICATION AND RACKING53 OF SURVEY DATA Table 12 below presents characteristics of the alumni population and sample before and after raking or weighting. These reveal slight differences in proportions of men and women Males, as compared to females, were slightly underrepresented (-4.41 percentage points) in the sample relative to the population. Thus, females were overrepresented in the final sample (+4.41). Similarly, with respect to province of origin, Capital Territory was underrepresented in the sample while Sindh was overrepresented in the final sample. Furthermore, Fulbright alumni belonging to Islamabad and the four provincial headquarters were highly underrepresented (-18.40) in the final sample while alumni from smaller regions were overrepresented (+12.12) in the sample. To correct potential any biases in the survey estimates arising from such unit non-response, the survey data was racked using “calibrate” in STATA to create weights that were used in the analysis. Gender and province of origin were almost completely balanced after raking, while differences between population and sample estimates with respect to city of origin decreased considerably. The weights originally obtained increased the frequencies, which could have underestimated the margins of error. To solve this problem, the weights were trimmed down by dividing the weights by the mean of weight variable. All of the survey analysis presents weighted analysis using post￾stratification weights. 53 This procedure makes a sample representative and corrects for non-response errors by weighting the survey data so that responses are adjusted to the known population trends. For example, if the population is 50 percent male and 50 percent female, but the survey contains many more male respondents compared to female respondents, raking adjusts through weighting the low response of females. 77 TABLE 12: DIFFERENCE BETWEEN POPULATION AND SAMPLE ESTIMATES BEFORE AND AFTER RAKING Population Data Survey Data Difference Before Raking Survey Estimates After Raking Difference After Frequency Raking Valid Percent Frequency Valid Percent Frequency Valid Percent Gender Male 369 59.9 187 55.5 -4.413 203 59.9 0.00 Female 247 40.1 150 44.5 +4.413 136 40.1 0.00 Total 616 100.0 337 100.0 338 100.0 Province of Origin Balochistan 26 4.2 11 3.3 -0.964 14 4.2 0.01 Islamabad Capital Territory 76 12.4 22 6.5 -5.830 42 12.3 0.02 FATA 2 .3 1 .3 -0.028 1 .3 0.00 Khyber Pakhtunkhwa 37 6.0 27 8.0 +1.996 20 6.0 0.01 Gilgit-Baltistan and Azad Jammu and Kashmir 7 1.1 6 1.8 +0.642 4 1.1 0.00 Punjab 310 50.4 169 50.1 -0.258 170 50.3 0.08 Sindh 157 25.5 101 30.0 +4.442 87 25.6 +0.12 Total 615 100.0 337 100.0 338 100.0 City of Origin Major metropolitan/ provincial headquarters and capital cities 475 79.8 207 61.4 -18.408 261 77.1 -2.72 Large city/ divisional headquarters 65 10.9 58 17.2 +6.286 36 10.6 0.37 Small towns/cities 55 9.2 72 21.4 +12.121 42 12.3 +3.09 Total 595 100.0 337 100.0 338 100.0 78 ANNEX 5: INSTRUMENT FOR FOCUS GROUP DISCUSSIONS WITH ALUMNI MSI-MEP Fulbright Student Program Evaluation Focus Group Discussion Guide - March 15 version Date: Time: City where FGD held: No. of FGD Participants: Men: Women: Total: Moderator: Note taker:  Thank you filling the survey forms and participating in the FGD  Confidentiality and anonymity  Permission for recording the FGD NOTETAKER – PLS NOTE DOWN GOOD QUOTES, EXPLANATIONS AND EXAMPLES Questions Prompts INTRODUCTION – ICE BREAKER  Introduce yourself and the U.S. education program you attended under Fulbright  Two people can talk and then introduce each other to the group 1. How was your experience in the U.S.?  What did you like the most about the program?  What did you like least about the program?  What do you miss about the U.S.?  What do you not miss about the U.S.?  How has Pakistan changed while you were in the U.S.? 2. After completing your U.S. degree, what has encouraged your continued stay in the Pakistan?  What are your long term plans for living and working in Pakistan?  Why? 79 Questions Prompts 3. How did your U.S. degree prepare you for the work you are currently doing?  How has your U.S. degree helped you in your current work?  How is your job aligned with the degree you pursued in the U.S.?  How satisfied are you with your current employment? 4. How has U.S. education helped your leadership potential?  How have your U.S. studies prepared you for leadership positions in the current field/career?  What are the indications that you are on track to attain a leadership position in your field (promotions, salary raise, increased responsibilities etc.)? 5. In what ways did your study and stay in the U.S. contribute to your perception of the U.S. and the Americans?  What were your perceptions of the U.S. and Americans before leaving for the U.S.?  How did your perceptions change as a result of your studies in the US?  How has your experience of living in the U.S. influenced your understanding of American people and society?  Public diplomacy objective: “Increased number of Pakistani understand that the United States and Pakistan share similar broad values and objectives and can work together to assist each other’s goals and objectives in the region” [Can read out to the participants]  In your view what values and objectives do Americans and Pakistanis share?  How have you been able to contribute to a better understanding of these values and objectives in the U.S.? In Pakistan?  Please give examples of how the common understanding of values and objectives has been achieved. 6. What kind of linkages were you able to develop with fellow students, faculty, researchers and the American people more generally during your stay in the U.S.?  Discuss your success in maintaining links with U.S. based institutions and individuals since your return to Pakistan?  With whom and for what purpose have you maintained links with U.S. based institutions and individuals since your return to Pakistan?  Formal institutional links?  Social ties and networks?  How have these linkages helped you professionally?  How can these linkages be promoted and made more useful or productive?  Are you involved in Alumni Association’s? How effective are they? How can they be made more effective? 80 Questions Prompts 7. How would you have financed your degree without a Fulbright Scholarship?  What other scholarships did you consider before going abroad on the Fulbright scholarship?  How helpful did you find your family and friends in helping you study abroad?  What other sources of funding did you consider?  Why did you ultimately choose Fulbright? 8. How can you contribute to Pakistan’s development needs?  What are the needs as related to your degree?  How are you and can you contribute? Total time – 2 hours Approx. 81 ANNEX 6: INSTRUMENT FOR PERSONAL INTERVIEWS WITH SUPERVISORS MSI-MEP Fulbright Student Program Evaluation Personal Interview Questionnaire for SUPERVISOR Interviewer Note taker Gender of employee: Gender of employer: Degree Level: Field: (to be completed by interviewer beforehand) Employee Sponsored by HEC: Yes No (to be completed by interviewer beforehand) Date Time: City: Venue NOTETAKER – PLEASE NOTE DOWN GOOD QUOTES, EXAMPLES AND EXPLANATIONS Subject/Topic Questions 1. INTRODUCTION  Introduction of the team and purpose of the evaluation  This is an evaluation of the Fulbright program. Fulbright sends master’s and PhD students to the U.S.  How long have you known this employee and do you know that the employee has a U.S. education and is a Fulbright scholar? 2. Organization and employees  Please tell us a little bit about your organization and your employees.  Number of employees: M= F=  U.S. educated or not?  Type of work the organization conducts 3. Productive employment of Fulbright alumni Current job  What is the employee’s current job?  What are the employee’s duties and responsibilities? Education  Does the employee’s job require an advanced education (MA and/or PhD)? Match of degree and job  The employee’s U.S. degree is in __________________________. 82 Subject/Topic Questions  How is the employee’s current job relevant to her/his U.S. degree?  How are you utilizing the employee’s advanced skills and learning?  What does the employee bring to the organization that is striking to you? 4. Leadership potential of Fulbright employee General  Do you have other employees who have studied overseas?  What do you think are the differences in a U.S. education versus the education of other countries?  How does a U.S. education make this employee different from other employees?  Does a U.S. education help encourage or enhance leadership potential? The scholar  Has U.S. education helped the employee in demonstrating leadership? If so, how (promotion, salary, responsibility and research)?  Does the employee use his/her U.S. education to coach or mentor other employees? How? 5. Objectives of Fulbright Program: improve public perception about U.S.  Has the employee talked about their experience in the U.S. with you and/or other employees?  If yes, what did you think about her/his experience?  To your knowledge, what do other employees think about the U.S. based on the (Fulbright) employee’s experience?  Do you think perceptions about the U.S., within the organization, have changed due to the (Fulbright) employee? Social perceptions? Political perceptions? Perceptions about shared values? Positively or not in each instance? 6. Linkages with U.S.  Are you aware if the employee is connected to friends, peers and faculty from the U.S.?  Do you think these linkages would be helpful (or are helpful) for your organization? How?  Is the employee using U.S. linkages (professional or social) to help this organization, and in what way? 83 Subject/Topic Questions 7. Likelihood of staying and working in Pakistan  Do you think employees with U.S. degrees are more likely get employment abroad?  If yes, then why invest in hiring this and other U.S. educated employees?  Do U.S. educated employees need additional motivation to continue to work in your organization and in Pakistan?  If yes, do you provide additional motivation and how?  Do you find exceptional or ordinary commitment in U.S. educated employees to work for Pakistan and its development?  What about in this particular employee?  If yes, what do you think is the extra commitment? Do you think it comes from the employee’s upbringing and background? Do you think U.S. education has contributed to this extra commitment? 84 ANNEX 7: INSTRUMENT FOR PERSONAL INTERVIEWS WITH HEC OFFICIALS MSI-MEP Fulbright Student Program Evaluation Personal Interview Questionnaire for HEC OFFICIALS (Current and former) Interviewer: Note taker: Date: Time: City: Venue: Subject/Topic Questions  Introduction and Background  Introduction of the team and purpose of the evaluation  Tell us about the professional relationship between HEC and USEFP regarding the Fulbright Program.  HEC Priority Needs  How have the priority needs of HEC changed since 2005, and why, and what are the current priorities?  How did/does HEC ensure that its priorities are adequately reflected in the selection of Fulbright scholars?  Did any of the scholars identified by HEC or USEFP meet HEC’s priorities? How?  How could the process of selection of candidates have been different to better meet HEC’s priority needs?  Achievement of Objectives  In what ways has the Fulbright program enabled HEC to achieve its objective of improved access to high quality education? Examples.  Relevance of current Fulbright Program (e.g., mix of Master’s and Ph.Ds. and fields  To what extent has HEC been able to achieve a desired mix of master’s and Ph.Ds.’ that is relevant to its priority needs? 85 Subject/Topic Questions of study to priority needs of HEC  What were your discussions with USEFP regarding the desired mix and how did HEC assess the scholars with HEC/USAID joint funding?  Based on your experience, is U.S. education more relevant to HEC priority needs than education in other countries?  How would the HEC meet its needs of high quality education in the absence of a Fulbright Program?  Given that funding was available, what would be HEC’s interest in reviving its involvement in the Fulbright program?  In your assessment, what motivates the Fulbright Alumni to continue working in Pakistan? What is the role of HEC in motivating them to stay and work in Pakistan?  Leadership potential  To what extent you think has the U.S. education prepared the Fulbright Alumni for assuming leadership positions in academia?  Linkages  In what manner have the Fulbright Alumni capitalized on the linkages with U.S. institutions/individuals (if any) that support them in their professional pursuits?  In what ways the HEC foster and sustain partnerships with U.S. institutions, especially through Fulbright alumni?  How can these and any other linkages between Pakistan and U.S. scholars be further encouraged and supported? Total time: 75 minutes approx. 86 ANNEX 8: RETURN ON INVESTMENT CALCULATION The return on investment (ROI) is a common measure of the financial efficiency of an investment. It is the percentage return on an investment, and is defined algebraically as the net value of an investment divided by the investment cost.54 Calculating the ROI requires defining the components of costs and benefits to include in the calculation and then quantifying the cost and benefit components in monetary terms. Both of these steps are potentially challenging for development projects. The Fulbright-Hays Act, the Fulbright Program, and the USAID Results Framework suggest a variety of objectives, none of which are financial. These include increased mutual understanding, educational and cultural advancement, improved access to higher education and scholarships, and improved public perception of the U.S. Training Pakistanis in U.S. universities may benefit the individual student, Pakistan, and the U.S. by increasing alumni incomes, contributing to development, increasing intellectual capacity in the public and private sectors (including strengthening Pakistani universities to the extent that alumni pursue academic careers within Pakistan), or contributing to cultural and academic exchange. A report by the College Board in the U.S., identifies a much wider range of possible benefits of higher education including: 1) college graduates are more likely to have employer-provided health insurance and pension benefits; 2) college graduates enjoy better health and provide their children with more opportunities; 3) higher levels of education are associated with lower unemployment and poverty rates; 4) college graduates in the workforce increase overall wages; 5) college graduates are less likely to smoke and have healthier lifestyles overall; and 6) college graduates are more likely to be civically active, volunteer, donate blood, and vote. While many of the “benefits” may be correlated, rather than caused by, college education, they nevertheless demonstrate a wider range of potential benefits of higher education.55 This incomplete list of potential benefits emphasizes the difficulties inherent in identifying the full range of benefits of a Fulbright-sponsored degree. Even though the Fulbright program serves a variety of purposes above the level of the individual beneficiary, only private benefits, except for some of the sub-points above, are included in this calculation. The program's benefits actually accrue to the entire society or perhaps both societies, the U.S. and Pakistan. Furthermore, few of the potential benefits listed above are easily monetized. The ROI calculation in this evaluation defines benefits only in terms of the estimated increase in lifetime earnings of Fulbright alumni. The resulting ROI thus substantially understates benefits and is not a particularly relevant measure of the efficiency of a program that has no stated financial objectives. Costs of the Fulbright Program may include the costs of tuition, fees, supplies, travel, and living expenses required to pursue the degree as well as the opportunity costs associated with investing time and resources in obtaining a degree. The financial costs incurred by USEFP to support a student through a degree program are relatively easy to measure. It is not so easy to measure or monetize many of the other costs. It should further be noted that the ROI estimate represents a lower bound on the net benefits of the program, assuming that the average returned Fulbright scholar has a positive net externality effect on Pakistani society. The remainder of this annex describes how the evaluation team calculated the ROI for the Fulbright Program. The Fulbright Program incurs costs and generates benefits in overlapping but different time periods. For the cohort included in this evaluation, USEFP incurred the costs between 2004 and 2012. Alumni started earning income 54 ROI=((Value of investment – Investment cost)/Investment cost)x100 55 Baum, Sandy and Jennifer Ma. 2007. Education Pays: The Benefits of Higher Education for Individuals and Society. The College Board. Retrieved from: http://trends.collegeboard.org/education-pays 87 (benefits) in 2006, the year the first recipient graduated, and 2037, 25 years after the date the last of the cohort graduated.56 Given the difficulties noted above, the analysis considers only the costs incurred by USEFP (including overhead and administrative costs) to fund the scholarships. On the benefit side, it includes only estimates of increased earnings projected over a 25-year career from the date a scholarship recipient enters the workforce. To calculate the ROI, the analysis had to make the following assumptions and judgments about the parameters of the calculation:  Growth of earnings over time – The analysis defines benefits as increased lifetime earnings attributable to a Fulbright-sponsored degree. Since both with- and without-degree earnings will change over the course of a career, the analysis had to apply a growth rate to each income stream. The analyst estimated growth rates in earnings for alumni with bachelor’s, master’s, and PhD degrees from 2010-11 Pakistan Social and Living Standards Measurement (PSLM) data. No data were available to estimate differential growth rates for employees with degrees from foreign universities – estimates that would have been more relevant to Fulbright alumni.  Changes in unemployment rates over time – At any point in time, some employees will be unemployed and thus not earning income (benefits). Furthermore, unemployment rates may change with experience or job tenure. The analysis incorporates changes in unemployment rates over time based on estimates from 2010-11 Pakistan Labor Force Survey data.  Exchange rates – Exchange rates between the Pakistani Rupee (PKR) and the U.S. Dollar have changed substantially over the 2004-2012 time period. The analysis used historic exchange rates to convert earnings, reported in PKR, to U.S. Dollars.  The discount rate – Costs incurred and benefits received at different times must be discounted to a common present value. The ROI is very sensitive to the choice of a discount rate. The analysis calculates the ROI for a range of discount rates to explore the sensitivity of results to the choice of a discount rate. This annex describes the rationale for the judgments and assumptions used in the analysis. Program Costs USEFP provided the team with cost estimates for each year covered by the evaluation. The cost estimates provided by USEFP represent average annual costs per student by degree type, i.e., master’s and PhD, and program year. Changes in the cost of education and the mix of universities cause costs to vary slightly from year to year. The costs represent the full costs USEFP incurred to provide the scholarship including administrative and overhead costs. Table 13 summarizes the cost data provided by USEFP. 56 For the sake of projecting benefits, the analysis assumes a 25-year career. 88 TABLE 13: AVERAGE ANNUAL COSTS BY YEAR AND DEGREE (NOMINAL U.S. DOLLAR) Academic Year Master’s Doctoral 2005/06 $56,280 $46,464 2006/07 $55,992 $49,066 2007/08 $53,769 $47,127 2008/09 $54,693 $45,747 2009/10 $55,786 $45,358 2010/11 $58,442 $45,757 2011/12 $58,655 $46,536 Source: USEFP Note: The academic year is from September to August. To estimate total program costs for each year covered by the evaluation, the team multiplied the average per student cost, by degree, by the number of Fulbright recipients enrolled in each year. The data provided by USEFP did not indicate precise start and end dates for each student. Therefore, the evaluation team could not determine exactly how many years each student had been enrolled. However, USEFP told the team that most master’s programs lasted two years and most PhD programs lasted four years. The evaluation team used these numbers, along with data on the academic year in which each student received his or her scholarship, to estimate enrollment in each year by sex and program. Table 14 summarizes enrollment estimates used in the ROI analysis. TABLE 14: TOTAL ENROLLMENT BY YEAR, SEX, AND DEGREE TYPE Academic Year Master’s Doctoral Male Female Male Female 2005/06 56 24 10 5 2006/07 97 54 26 10 2007/08 108 80 64 22 2008/09 123 89 67 25 2009/10 107 88 59 21 2010/11 92 85 43 17 2011/12 0 36 5 5 Program Benefits Because monetizing the ROI analysis defines benefits only in terms of the estimated increase in earnings of Fulbright alumni over a 25-year career commencing in the year they obtain their degree. The survey of 337 Fulbright alumni provided estimates of pre- and post-degree earnings as well as rates of unemployment. Table 15 summarizes these data. Earnings estimates reflect data from employed respondents only. Respondents who were not employed were either unemployed and looking for work or they were in school. 89 TABLE 15: PRE- AND POST-DEGREE AVERAGE ANNUAL EARNINGS BY SEX AND DEGREE TYPE Degree Male Female N (total) N (with earnings) Average Annual Earnings (Rs.) N (total) N (with earnings) Average Annual Earnings (Rs.) Master’s Pre-degree 162 148 614,926 142 123 499,122 Post-degree 144 1,877,246 109 1,530,848 PhD Pre-degree 25 21 515,429 8 7 946,286 Post-degree 17 1,676,471 7 2,288,572 Source: MEP Survey Growth in Earnings over Time The ROI analysis projects earnings over a 25-year career starting in the year a student graduates. Earnings are expected to increase over time as employees gain experience and seniority. Furthermore, it is possible that an employee’s sex and level of education may affect the rate at which earnings increase. The analysis used data from the 2010-11 Pakistan Social and Living Standards Measurement (PSLM) survey to estimate increases in annual earnings as a function of length of employment, sex, and level of education. Table 16 reports results of a regression of the logarithm of income on a set of explanatory variables. TABLE 16: ESTIMATED EFFECTS OF EXPERIENCE ON ANNUAL EARNINGS Variable Coefficient p-value Confidence Interval Constant 11.0337 (0.0161) 0.0000 11.0021 11.0653 Years of employment 0.0678 (0.0009) 0.0000 0.0660 0.0696 Years of employment squared -0.0011 (0.0000) 0.0000 -0.0011 -0.0010 Undergraduate degree (dummy) 0.9870 (0.0217) 0.0000 0.9444 1.0296 Graduate degree (dummy) 1.3343 (0.1583) 0.0000 1.0240 1.6446 Years of employment x undergraduate degree 0.00323 (0.0011) 0.0030 0.0011 0.0053 Years of employment x graduate degree 0.0092 (0.0070) 0.1900 -0.0045 0.0229 Male (dummy) -0.2794 (0.0124) 0.0000 -0.3037 -0.2551 Dependent variable: Log of annual income Source: Analysis of 36,930 observations from the 2010-11 PSLM data Note: Numbers in parentheses are standard errors Note: The p-value represents the probability of obtaining the observed result by chance. A small p-value thus implies that the estimated coefficient is significantly different from zero (i.e., no effect). 90 The results reported in Table 16 represent data from 36,930 observations taken from the 2010-11 PSLM for which respondents were formally (i.e., received a regular salary) and fully (i.e., employed for the entire 12 months prior to the survey) employed. This subset of respondents corresponds most closely to the likely employment situation of Fulbright alumni. Also, the PSLM did not include data on length of employment. The analysis thus used the number of years elapsed since completing school as a proxy for work experience. It assumed that respondents started work at the age of 15 years, if they did not pursue an education past grade 10, and started work immediately after earning their degrees if they did continue with their education. The results suggest that experience has a large and significant effect on earnings, but the effect tapers off with years of experience. Those with undergraduate degrees start their careers with higher earnings than those without a college degree, and their earnings increase more rapidly with experience. Similarly, those with a graduate degree start their careers with higher earnings than those with no college or with undergraduate degrees, and their earnings increase slightly faster than either of the other two groups. Men start with slightly lower earnings than women at all levels of education.57 The data differentiate only between no college degree, undergraduate, and graduate degrees. The analysis, therefore, applied the graduate degree growth rate to both master’s and PhD degrees. FIGURE 18: GROWTH IN EARNINGS AS A FUNCTION OF SEX AND EDUCATION 0 100,000 200,000 300,000 400,000 500,000 600,000 700,000 800,000 900,000 0 2 4 6 8 10 12 14 16 18 20 22 24 Annual income (2012 Rs.) Years of experience Male - No college degree Male - Undergraduate Male - Graduate degree Female - No college degree Female - Undergraduate Female - Graduate degree 57 Although not reported here, additional analysis found no significant difference between men and women in rates of growth in earnings. 91 The ROI analysis used the estimated growth rates for earnings to project pre- and post-degree earnings over a 25- year career. It applied the “undergraduate” growth rates to pre-degree earnings to estimate the counterfactual earnings trajectory, i.e., how earnings would have changed without the Fulbright-sponsored degree, and used the “graduate degree” growth rates to project post-degree earnings. The team was not able to find secondary data to estimate growth rates in earnings for those holding an advanced degree from a foreign university. Unemployment Some portion of Fulbright alumni will be unemployed at any time and thus not earning. The analysis adjusts benefit estimates to account for unemployment. Only five of the 316 respondents (1.6 percent) who were not students reported being unemployed prior to pursuing their Fulbright-sponsored degrees. After receiving their degrees, 25 of the 320 Fulbright alumni (7.8 percent) who were not still in school reported being unemployed. The survey covered Fulbright alumni who graduated between 2006 and 2012. The earliest graduates thus had only six years of experience in the job market. Data from the 2010-11 Labor Force Survey58 suggest that unemployment rates are much higher in the first 10 years of a career than later on. To estimate unemployment rates as a function of job tenure over a 25-year career, the analysis estimated unemployment rates by age (a proxy for experience) from summary data from the Labor Force Survey. TABLE 17: ESTIMATES OF UNEMPLOYMENT AS A FUNCTION OF AGE Variable Coefficient p-value Confidence Interval Constant 0.311087 (0.026794) 0.000000 0.245523 0.376650 Age (years) -0.018549 (0.002021) 0.000000 -0.023494 -0.013604 Age squared 0.000362 (0.000048) 0.000000 0.000245 0.000478 Age cubed -0.000002 (0.000000) 0.001000 -0.000003 -0.000001 Dependent variable: Unemployment rate Source: Analysis of summary data from the 2010-11 Labor Force Survey Note: Numbers in parentheses are standard errors Note: The p-value represents the probability of obtaining the observed result by chance. A small p-value thus implies that the estimated coefficient is significantly different from zero (i.e., no effect). The ROI analysis used this function to estimate unemployment rates for Fulbright alumni during a 25-year career. The analysis shifted the function upwards (by adding 0.024 to the constant term) to adjust the initial unemployment rate from the 5.4 percent rate reported in the Labor Force Survey to the 7.8 percent initial unemployment rate reported by Fulbright alumni. Figure 19 illustrates both the estimated unemployment rate function (based on the Labor Force Survey) and the function used to estimate unemployment rates for the ROI analysis. 58 http://www.pbs.gov.pk/sites/default/files/Labour%20Force/publications/lfs2010_11/t34.pdf 92 FIGURE 19: UNEMPLOYMENT AS A FUNCTION OF AGE (POST GRADUATE AND PHD HOLDERS) -2.00% 0.00% 2.00% 4.00% 6.00% 8.00% 10.00% 20-24 25-29 30-34 35-39 40-44 45-49 50-54 55-59 60-64 65 and older Unemployment rate (%) Age in years Labor Force Survey ROI analysis Exchange Rates The analysis used historic exchange rate data from the State Bank of Pakistan59 to convert earnings denominated in PKR to U.S. Dollars for comparison with costs which were incurred in U.S. Dollars. TABLE 18: HISTORIC U.S. DOLLAR/PKR EXCHANGE RATES Academic Year Average Exchange Rate (PKR/U.S. Dollar) 2005/06 59.59 2006/07 60.27 2007/08 60.73 2008/09 70.62 2009/10 81.58 2010/11 85.12 2011/12 86.28 Source: State Bank of Pakistan Note: Exchange rates are the average of monthly exchange rates for the academic year. 59 http://www.sbp.org.pk/ecodata/HER-USDollar.xls 93 Discount Rate A discount rate is necessary to adjust costs incurred and benefits received at different times to a common time period. The adjustment is necessary because, for a number of reasons, a rupee next year is not worth as much as a rupee now. Reasons for this discrepancy may include inflation which reduces the value of money, opportunity cost (i.e., what a person could do with the money in the intervening year), or personal needs or preferences. Because the Fulbright program generates most of its benefits well after it incurs costs; the ROI is particularly sensitive to the choice of a discount rate. Ideally, the discount rate should reflect at least the expected rate of inflation over the next 25 years. But it may also incorporate returns from alternative investments. Because the choice of a discount rate is somewhat subjective, the analysis explored the sensitivity of the ROI to the discount rate by calculating ROI for a range of discount rates. ROI Based on the information provided previously in this annex, the analysis calculated the ROI as follows: Program Benefits 1. It determined the number of scholarship recipients and graduates by year, sex, and degree type from data provided by USEFP. 2. Table 19 summarizes data on the number of graduates by year, sex, and degree. All alumni are assumed to have continued working for a 25-year career, adjusted for unemployment as described in the next step. TABLE 19: NUMBER OF FULBRIGHT ALUMNI BY YEAR, SEX, AND DEGREE Male – Master’s Female – Master’s Male – PhD Female – PhD Total 2005 0 0 0 0 0 2006 56 24 0 0 80 2007 41 30 0 0 71 2008 67 50 10 5 132 2009 56 39 16 5 116 2010 51 49 38 12 150 2011 41 36 3 3 83 Total 312 228 67 25 632 Source: USEFP. Note: The adjustment factor is the percentage of scholarship recipients that said they could not have obtained their U.S. degrees without the scholarship. 3. It then estimated the number of beneficiaries with earnings each year by multiplying the number of beneficiaries by one minus the unemployment rate. The unemployment rate applicable to each year after graduation was calculated from the equation of Table 17 and adjusted as described in the narrative accompanying the table. 4. Average earnings in each year after graduation were calculated from the MEP survey data adjusted for experience using the estimates of Table 15. 5. Table 20 summarizes estimates of baseline pre- and post-degree earnings by sex and degree type. 94 TABLE 20: AVERAGE ANNUAL EARNINGS OF EMPLOYED ALUMNI (PKR) Pre-degree Post-degree Male Female Either Sex Male Female Either Sex Master’s 614,926 499,122 562,365 1,877,246 1,530,848 1,728,008 PhD 515,429 946,286 623,143 1,676,471 2,288,572 1,855,000 Either degree 602,562 523,200 568,057 1,856,047 1,576,573 1,739,011 Source: MEP Survey of Fulbright Alumni 6. Total post-degree earnings in each year after graduation, by sex and degree, was then the number of graduates in the work force in that year multiplied by the relevant post-degree earnings adjusted for experience. 7. The analysis then estimated total pre-degree earnings by year, sex, and degree type in an identical manner to that used for post-degree earnings. It used a constant 1.48 percent unemployment rate, the unemployment rate reported by Fulbright alumni before they started their programs. 8. The “benefit” of the Fulbright program in each year is total post-degree earnings minus total pre-degree earnings with the totals taken over all alumni in the work force that year. 9. Benefits in each year were then converted to U.S. Dollars from PKR using historic exchange rates obtained from the State Bank of Pakistan (Table 18). The 2012 exchange rate was applied to all years after 2012. Table 21 summarizes benefits by year, sex, and degree type in nominal (i.e., undiscounted) U.S. Dollars. 95 TABLE 21: BENEFITS OF A FULBRIGHT DEGREE BY YEAR, SEX, AND DEGREE TYPE (NOMINAL U.S. DOLLAR) Benefits (Nominal U.S. Dollar) Year Male - Master’s Female - Master’s Male - PhD Female - PhD Program 2005 0 0 0 0 0 2006 0 0 0 0 0 2007 1,023,661 0 358,675 0 1,382,336 2008 1,855,981 0 834,639 0 2,690,620 2009 2,788,882 144,025 1,422,122 81,379 4,436,408 2010 3,391,006 335,880 1,901,252 147,827 5,775,966 2011 4,201,030 805,821 2,538,439 317,402 7,862,692 2012 5,030,929 903,383 3,284,710 383,055 9,602,077 2013 5,050,248 909,686 3,295,143 387,127 9,642,204 2014 5,465,689 987,555 3,564,173 421,692 10,439,109 2015 5,898,411 1,068,986 3,844,478 457,916 11,269,791 2016 6,347,484 1,153,834 4,135,479 495,738 12,132,535 2017 6,811,807 1,241,918 4,436,484 535,081 13,025,291 2018 7,290,102 1,333,025 4,746,690 575,853 13,945,670 2019 7,780,920 1,426,906 5,065,180 617,945 14,890,950 2020 8,282,645 1,523,276 5,390,928 661,233 15,858,082 2021 8,793,502 1,621,821 5,722,805 705,578 16,843,706 2022 9,311,566 1,722,193 6,059,580 750,826 17,844,165 2023 9,834,772 1,824,013 6,399,933 796,808 18,855,527 2024 10,360,929 1,926,876 6,742,458 843,345 19,873,609 2025 10,887,738 2,030,352 7,085,675 890,243 20,894,007 2026 11,412,805 2,133,988 7,428,039 937,298 21,912,131 2027 11,933,666 2,237,313 7,767,957 984,300 22,923,236 2028 12,447,799 2,339,839 8,103,795 1,031,028 23,922,461 2029 12,952,654 2,441,071 8,433,894 1,077,258 24,904,877 2030 13,445,670 2,540,504 8,756,585 1,122,762 25,865,522 2031 13,924,300 2,637,632 9,070,205 1,167,311 26,799,448 2032 14,386,035 2,731,951 9,373,107 1,210,675 27,701,768 2033 11,985,771 2,822,965 8,668,379 1,252,628 24,729,744 2034 10,260,260 2,910,189 7,678,208 1,292,949 22,141,606 2035 7,148,636 2,528,838 5,816,651 1,054,031 16,548,156 2036 4,500,926 1,853,669 4,010,319 806,459 11,171,373 2037 2,030,100 135,899 1,977,952 162,192 4,306,144 Total 246,835,926 48,273,409 163,913,937 21,167,939 480,191,211 Note: Years run to 2037 to accommodate a 25-year earnings projection for alumni who graduated in 2012 and started their careers in 2013. 96 Program Costs The analysis then calculated annual program costs by multiplying the number of students enrolled in each year in each degree program by the average cost per student for that relevant degree program and year (Table 13). Table 22 summarizes annual program costs in nominal U.S. Dollars. TABLE 22: ANNUAL PROGRAM COSTS BY YEAR, SEX, AND DEGREE TYPE (NOMINAL U.S. DOLLAR) Year Male - Master’s Female - Master’s Male - PhD Female - PhD Program 2005 $3,151,680 $1,350,720 $464,640 $232,320 $4,967,040 2006 $5,431,224 $3,023,568 $1,275,716 $490,660 $9,730,508 2007 $5,807,052 $4,301,520 $3,016,128 $1,036,794 $13,124,700 2008 $6,727,239 $4,867,677 $3,065,049 $1,143,675 $14,659,965 2009 $5,969,102 $4,909,168 $2,585,406 $907,160 $13,463,676 2010 $5,376,664 $4,967,570 $1,876,037 $686,355 $12,220,271 2011 $2,404,855 $2,111,580 $139,608 $139,608 $4,656,043 Totals $34,867,816 $25,531,803 $12,422,584 $4,636,572 $72,822,203 The seven-year (2005-2011) stream of costs and 25-year stream of benefits were then converted to present values, using discount rates ranging from 1 percent to 12 percent, to facilitate analysis of the sensitivity of results to the choice of a discount rate. The present value of the cost and benefit streams is the sum of present values overall years. Tables 23 and 24 summarize present values of benefits and costs, respectively. TABLE 23: PRESENT VALUES OF BENEFITS BY SEX, DEGREE TYPE, AND DISCOUNT RATE (2012 U.S. DOLLAR) Discount Rate Present Value of Benefits by Discount Rate, Sex, and Degree Type (U.S. Dollar) Male - Master’s Female - Master’s Male - PhD Female - PhD Program 1% $218,213,866 $144,261,470 $42,162,847 $18,469,773 $423,107,956 2% $194,174,551 $127,794,574 $37,059,346 $16,217,603 $375,246,074 3% $173,906,729 $113,940,820 $32,778,499 $14,329,596 $334,955,644 4% $156,754,812 $102,239,057 $29,172,281 $12,740,092 $300,906,242 5% $142,187,126 $92,316,440 $26,121,452 $11,396,221 $272,021,238 6% $129,770,976 $83,870,449 $23,529,611 $10,255,255 $247,426,291 7% $119,153,000 $76,654,753 $21,318,551 $9,282,544 $226,408,847 8% $110,043,627 $70,468,076 $19,424,595 $8,449,889 $208,386,187 9% $102,204,757 $65,145,383 $17,795,721 $7,734,261 $192,880,122 10% $95,439,951 $60,550,889 $16,389,279 $7,116,786 $179,496,905 11% $89,586,602 $56,572,494 $15,170,182 $6,581,944 $167,911,223 12% $84,509,662 $53,117,343 $14,109,468 $6,116,930 $157,853,403 97 TABLE 24: PRESENT VALUES OF COSTS BY SEX, DEGREE TYPE, AND DISCOUNT RATE (2012 U.S. DOLLAR) Discount Rate Present Value of Costs by Discount Rate, Sex, and Degree Type (U.S. Dollar) Male - Master’s Female - Master’s Male - PhD Female - PhD Program 1% $36,311,653 $26,502,033 $12,930,334 $4,825,520 $75,744,020 2% $37,811,257 $27,506,457 $13,456,028 $5,021,250 $78,773,741 3% $39,368,568 $28,546,170 $14,000,206 $5,223,971 $81,914,944 4% $40,985,589 $29,622,297 $14,563,421 $5,433,901 $85,171,307 5% $42,664,376 $30,735,992 $15,146,241 $5,651,260 $88,546,609 6% $44,407,048 $31,888,441 $15,749,247 $5,876,277 $92,044,736 7% $46,215,785 $33,080,861 $16,373,034 $6,109,184 $95,669,680 8% $48,092,828 $34,314,502 $17,018,212 $6,350,221 $99,425,542 9% $50,040,484 $35,590,647 $17,685,405 $6,599,632 $103,316,536 10% $52,061,122 $36,910,613 $18,375,252 $6,857,670 $107,346,988 11% $54,157,181 $38,275,752 $19,088,408 $7,124,592 $111,521,340 12% $56,331,168 $39,687,448 $19,825,541 $7,400,661 $115,844,157 ROI The ROI associated with each discount rate, sex, and degree type was then calculated using the formula: Table 25Table 25 summarizes ROI by discount rate, sex, and degree type for discount rates ranging from 1 to 12 percent. Figure 20 illustrates the data in Table 22. TABLE 25: ROI BY DISCOUNT RATE, SEX, AND DEGREE TYPE Discount Rate Male - Master’s Female - Master’s Male - PhD Female - PhD Program 1% 501% 444% 226% 283% 459% 2% 414% 365% 175% 223% 376% 3% 342% 299% 134% 174% 309% 4% 282% 245% 100% 134% 253% 5% 233% 200% 72% 102% 207% 6% 192% 163% 49% 75% 169% 7% 158% 132% 30% 52% 137% 8% 129% 105% 14% 33% 110% 9% 104% 83% 1% 17% 87% 10% 83% 64% -11% 4% 67% 11% 65% 48% -21% -8% 51% 12% 50% 34% -29% -17% 36% 98 FIGURE 20: ROI BY DISCOUNT RATE, SEX, AND DEGREE TYPE -100% 0% 100% 200% 300% 400% 500% 600% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% Return on Investment Discount rate Male - Masters Female - Masters Male - Ph.D. Female - Ph.D. Program 99 ANNEX 9: DISCLOSURE OF ANY CONFLICTS OF INTEREST USAID Evaluation Policy requires that evaluation reports include a signed statement by each evaluation team member regarding any conflicts of interest. Name Sarah Tirmazi Title Team Leader Organization Management Systems International Evaluation Position? Team Leader Team Member Evaluation Award Number (contract or other instrument) Order No. AID-391-M-11-00001 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Fulbright Program I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date June 12, 2013 100 101 ANNEX 10: MEP EVALUATION TEAM BIOS Ms. Sarah Tirmazi is a senior consultant with considerable experience in education sector in Pakistan. She has previously had long term assignments with ActionAid, USAID/Pakistan and the World Bank. She holds a PhD in Economics from the American University, USA. Mr. Ahmad Jameel has considerable experience in the issues of tertiary education in Pakistan and worked as a consultant in the evaluation of MNBSP. He has also been associated with the training and management of university staff under USAID programs. Mr. Ghazanfar Ali Khan Hoti is a full-time Senior Evaluation Specialist at MEP. He has expertise in bank examining and project evaluations. At MEP, he has managed and participated in several evaluations of USAID/Pakistan projects. Previously he has worked as a consultant with the Independent Evaluation Group of the World Bank in Washington DC. He holds Master of Public Administration (Economic Policy Management) and Master of Science (Operations Research) degrees from Columbia University, USA. Ms. Sara Azmat Zaidi is a full-time Evaluation Specialist at MEP. She has four years of experience in education policy, trainings, and media. She holds a Master’s in Education Policy, Planning and Administration (International Educational Development) from Boston University, USA. 102 U.S. Agency for International Development/Pakistan American Embassy RAMNA 5, Diplomatic Enclave Islamabad