Pakistan-United States Science and Technology Cooperation (S&T) Program: Mid-term Performance Evaluation Report September 30, 2014 This publication was produced for review by the United States Agency for International Development by Tariq Husain, Zameer Haider, Aftab Ismail Khan, and Babar Mufti. It was prepared by Management Systems International (MSI) under the Monitoring and Evaluation Program (MEP). ACKNOWLEDGMENTS The authors would like to express their thanks to all those who facilitated the work of the team and enabled it to complete this evaluation. These include, but are not limited to, the following:  Lisa Slifer-Mbacke, MSI’s Education Practice Area Leader and Technical Director, reviewed earlier drafts of the report and led data collection and analysis in the U.S. with the help of Aleksandra Markovich, Julian Glucroft, and Gwynne Zodrow, all of whom deserve special thanks for conducting and analyzing interviews with U.S. grant recipients and program managers;  Officials of Pakistan’s Higher Education Commission and the Ministry of Science and Technology, as well as the United States Department of State, USAID, and the National Academy of Sciences, who facilitated the team’s work and provided a wealth of information through interviews, discussions, and relevant documents; and,  The Pakistani and American grantees of the S&T Program and the Heads of Department of some of the Pakistani institutions, for sharing their knowledge and understanding of the program with the evaluation team. PAKISTAN-UNITED STATES SCIENCE AND TECHNOLOGY COOPERATION (S&T) PROGRAM MID-TERM PERFORMANCE EVALUATION REPORT September 30, 2014 Contracted under Order No. AID-391-C-13-00005 Monitoring and Evaluation Program DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i CONTENTS Executive Summary......................................................................................................................................................................... 1 Project Summary ............................................................................................................................................................................. 4 Program Background ...................................................................................................................................................................... 6 Evaluation Purpose and Evaluation Questions ......................................................................................................................... 9 Evaluation Methods and Limitations .........................................................................................................................................10 Findings and Conclusions ............................................................................................................................................................13 Recommendations ........................................................................................................................................................................26 Bibliography ....................................................................................................................................................................................27 Annexes ...........................................................................................................................................................................................28 Annex 1: Overview of Phases 1-5 of the S&T Program ............................................................................................28 Annex II: S&T Evaluation Statement of Work .............................................................................................................30 Annex III: Sampling Frame for Interviews with Researchers and Heads of Departments of Pakistani Institutions ............................................................................................................................................................................60 Annex IV: Sampling Frame for Interviews with Researchers in the U.S. .............................................................. 63 Annex V: Data Collection Instruments .........................................................................................................................66 Annex VI: Disclosure of Conflict of Interest ...............................................................................................................70 Tables and Figures Table 1: Project Summary ............................................................................................................................................................. 4 Table 2: Number of S&T Grants by Phase and Focus Area ................................................................................................. 6 Table 3: S&T Results Framework ............................................................................................................................................... 8 Table 4: Number of Researchers Interviewed in Pakistan by Phase and Focus Area .................................................10 Table 5: Number of Researchers Interviewed in The U.S. by Phase and Focus Area ................................................11 Table 6: Number of Heads of Department Interviewed in Pakistani Institutions by Phases and Focus Areas ....11 Figure 1: Map of S&T Program Area .......................................................................................................................................... 5 Figure 2: Improvements in Pakistani Researchers' Skills .....................................................................................................14 Figure 3: Factors Contributing to Improved Pakistani Researchers' Skills .....................................................................14 Figure 4: Indicators of Capacity Development at Pakistani Institutions ..........................................................................16 Figure 5: Factors Contributing to Capacity Development at Pakistani Institutions .....................................................16 Figure 6: Results of Steps Taken to Sustain Improved Capacity at Pakistani Institutions ..........................................17 Figure 7: Dissemination of Research Findings........................................................................................................................20 Figure 8: Challenges/Barriers to Increasing the Benefits of Research .............................................................................21 Figure 9: Factors Contributing to Enhanced Pakistan-U.S. Understanding and Goodwill ..........................................23 Figure 10: Factors Hindering Timely Project Performance ................................................................................................25 ii ACRONYMS BEP Bio-Engagement Program CSP Chemical Security Program DNA Deoxyribonucleic Acid DoS Department of State HEC Higher Education Commission HOD Head of Department M&E Monitoring and Evaluation MEP Monitoring and Evaluation Program MoST Ministry of Science and Technology MSF Mission Strategic Framework MSI Management Systems International MTDF Medium-Term Development Framework NAS National Academy of Sciences NIBGE National Institute of Biotechnology and Genetic Engineering PCR Polymerase Chain Reaction PI Principal Investigator RFP Request for Proposals S&T Pakistan-United States Science and Technology Program SOW Statement of Work U.S. United States USAID United States Agency for International Development USDA United States Department of Agriculture 1 EXECUTIVE SUMMARY Background Started in 2005, the United States Agency for International Development’s (USAID’s) Pakistan-United States Science and Technology Cooperation (S&T) Program has been funded by USAID, the Department of State (DoS), and the Government of Pakistan to provide research grants to Pakistani and American researchers to carry out joint research projects. It is implemented in the United States (U.S.) by the U.S. National Academy of Sciences (NAS) and in Pakistan by the Higher Education Commission (HEC) and the Ministry of Science and Technology (MoST). The program entered its sixth phase of funding in Fall 2014, but activities have been completed (and included in the evaluation) only for the 73 grants awarded in phases 1-4. The objective of supporting research partnerships is to build capacity at the institutional level in Pakistan and strengthen U.S.-Pakistan cooperative relationships in one or more of the following ways: (1) enhance the ability of Pakistan's science and technology community to spur human and economic development; (2) improve the quality, relevance, or capacity of education and research at Pakistani institutions of higher education in the field of science and technology; or (3) increase the capacity of Pakistani research institutions to support industry competitiveness.1 The S&T Program has not been evaluated independently since 2008. As the program approaches its tenth year, USAID and DoS are interested in determining whether the purpose and approach of the S&T Program model remain relevant, effective, and vital as contributors to improved U.S.-Pakistan relations and enhanced research capacity in Pakistan. The evaluation focuses on four main questions that address: (1) sustainable strengthening of Pakistan’s scientific research capacity; (2) the benefits of research for industry, government, or individual quality of life in Pakistan; (3) mutual understanding and goodwill between institutions and individuals in the U.S. and Pakistan as a result of academic collaboration; and, (4) the timely achievement of project targets. The evaluation addressed each of these and a large number of sub-questions in detail, using a qualitative methodology based on multiple sources of information for triangulation. The methodology included a review of documents for 27 grants from phases 1-4, and 50 individual interviews (29 in Pakistan and 21 in the U.S.) with grant recipients, associated heads of department in Pakistan, and U.S. and Pakistani program managers from USAID, DoS, NAS, HEC, and MoST. Conclusions The S&T Program has achieved much of what it was designed and resourced to accomplish. The challenges that remain have their root causes both in the research environment of Pakistan, which is beyond the control of the program, and the absence of specific measures aimed at meeting certain expectations. The following conclusions summarize program achievements and challenges: 1. As judged by the program participants interviewed for the evaluation, the S&T Program has enhanced the skills of Pakistani researchers, including soft skills (such as critical thinking, creative problem solving, innovation, and analytical reasoning) and the ability to use new laboratory equipment and research methods, in carrying out applied research. The exchange visit component, in particular, has contributed to improved skills, and so has working with new equipment. 2. Pakistani program participants believe that the program has also strengthened the capacity of the institutions where Pakistani researchers are based by investing in researchers’ skills, financing new 1 Pakistan-United States Science and Technology Cooperation Program, Program Solicitation 2009 (Proposal Deadline: October 31, 2009). 2 laboratory equipment, and helping improve students’ research opportunities and the teaching imparted to them.2 3. The S&T Program has a strong thrust toward academic publications and conference presentations. Almost all of the grants have resulted in academic publications, most of the research work has been presented at international conferences, and the average output per grant is impressive: 5.2 publications and 7.1 conference presentations per project for the 73 phase 1-4 projects. Achievements such as these are associated with enhanced international recognition for Pakistani researchers and their partners; they are also achievements in Pakistan-U.S. collaboration. 4. Improved skills among researchers and students represent human capital that will yield a flow of benefits in the future. Thus, these skills, supported by laboratory equipment that is maintained, can contribute to international standards research in the years to come. Sustainability of results will be enhanced wherever researchers receive follow-up grants from S&T and other sources (as many have among the researchers interviewed) and establish effective linkages with government and business entities (which a few of the interviewed researchers have done). 5. The S&T Program is recognized by program participants as an avenue for mutual understanding and goodwill between researchers from Pakistan and the U.S. This recognition, combined with the involvement of researchers’ students in the program, the addition of laboratory equipment at Pakistani institutions, and continuing efforts by some Pakistani researchers to work with U.S. institutions, represents progress in collaboration between the Pakistani and U.S. research establishments. 6. The benefits of S&T-funded research for business, government, and the general public have been limited and the program confronts challenges in generating such benefits. Researchers identified elements of the macro environment that limit impact beyond the research setting, but these elements are beyond the control of S&T. The evaluation also found that research projects are not designed (and not required to be designed) in partnership with the ultimate owners, sellers, and users of research products, an omission that can be rectified by the program. 7. The program has also been unable to overcome some of the challenges to sustaining its results: there is little evidence that Pakistani researchers’ institutions have allocated resources for maintaining equipment, or introduced policies for sustaining research that harnesses improved researcher skills and institutional capacity. The program has not developed instruments for ensuring the sustained operation and maintenance of equipment. 8. The S&T Program is avowedly merit-based and Principal Investigators (PIs) are expected to include women in their projects. In practice, women are included in S&T research to a limited extent, and this could be a reflection of male dominance in scientific research. The program has not introduced some of the means that are available for establishing more equal gender participation (outlined in the evaluation recommendations below). 9. Funding, procurement, visa, and other problems have continued to affect researchers involved in the program for several years, and many of these problems lie beyond the control of the program. Significant problems in program management have evidently not been addressed in a systematic manner so far, although many researchers expressed the need for resolving them and the program managers in Pakistan felt that solutions are possible. Recommendations The evaluation offers four recommendations. Three of these can be implemented by modifying the grant award process to enable the S&T Program to generate greater practical benefits, better sustain results, and include more women in program activities. The fourth recommendation concerns timely project performance. It is recommended that the grant award process be modified in three ways to reflect the following suggestions: 2 U.S. researchers deal with individual Pakistani researchers and it was hard for them to observe and assess increased institutional capacity. 3 1. The S&T Program objectives are aimed at both basic and applied research, the latter including research that spurs economic development and supports industry competitiveness. The program should add three methods for encouraging applied research to the grant announcement, proposal, and selection processes. These recommended measures are: (a) clarify in the request for proposals how the program intends to promote applied research; (b) require researchers interested in applied research to demonstrate partnerships at the proposal/application stage with specific business and government entities for which they intend to produce certain products or solutions; and provide written confirmation of common objectives and contribution from such entities. In other words, proposals should describe not only partnerships between U.S. and Pakistani researchers, but also between researchers and the end-users of the proposed research. Research intended to benefit communities (for example, farmers, home builders, patients, and water users) should also be proposed as partnerships between researchers and relevant service providers in the public or private sector (from example, government line departments and regulatory agencies or private manufacturers and organizations that represent community groups); and (c) give bonus points during the grant selection process to proposals/applications that demonstrate promising partnerships with or for the end users. 2. The S&T Program should require Pakistani researchers who include equipment in their proposals to submit plans for the sustainable operation and maintenance of the equipment, including their institutions’ written commitment to allocate the space, staff, and finances required for this purpose. In addition, USAID should request the Higher Education Commission of Pakistan to ensure that institutions that will own the equipment allocate the required staff and finances for its operation and maintenance before they purchase the equipment. 3. The S&T Program should consider various options, in a stand-alone or mutually reinforcing mode, which could lead to greater inclusion of women in the program. One option is to specify a progressively increasing quota for female PIs in terms of the proportion of funding or number of grants. The quota could be for the program as a whole or for specified focus areas, including those (for example, agriculture) in which there are few women at present. Another option is to give weight or bonus points in the grant selection criteria to the number of women, including PIs, staff, and students, who will participate in the proposed research. A third option is to establish a support group consisting of program alumni that would seek out and mentor female grant applicants from Pakistan and the U.S. throughout the application and research process. In this scenario, the S&T Program should provide operational funds to support group members for their time, travel, communication, photocopying and similar costs during the application process. It could also arrange training in proposal development and grant management. The fourth recommendation is aimed at improving the timeliness of project performance: 4. In consultation with program sponsors and implementing partners from Pakistan and the U.S., USAID should commission a management review focusing on known problem areas, including those highlighted in the evaluation. It was beyond the scope of the evaluation to determine the causes of delay in releasing funds to Pakistani PIs, the link between reporting and release of funds, or the causes of delays in hiring and procurement. An earlier (2008) evaluation was “not able to find a defensible reason for the delays [in payment]” but recommended that S&T should “streamline the administration of this program and ensure that payments are made in a timely fashion.” Streamlining has evidently not taken place, and a management review would be useful in clarifying ways and means for solving problems. 4 PROJECT SUMMARY The S&T Program provides research grants to Pakistani and American universities and research institutions to carry out joint research projects. The program has been funded by USAID, DoS and the HEC and the MoST. It is implemented in the United States by the NAS and in Pakistan by the HEC and MoST. The objective of these research partnerships is to build capacity in the sciences and technology at the institutional level in Pakistan and to strengthen U.S.-Pakistan cooperative relationships. The program strives to meet this objective in one or more of the following ways: 1. Enhance the ability of Pakistan's science and technology community to spur human and economic development; 2. Improve the quality, relevance, or capacity of education and research at Pakistani institutions of higher education in the field of science and technology; or 3. Increase the capacity of Pakistani research institutions to support industry competitiveness. TABLE 1: PROJECT SUMMARY Title/Field Program Information Agreement Number 391-ESP-A-00-05-00001 Agreement Officer’s Representative Jerry O’Brien, Office of Afghanistan and Pakistan Affairs Start Date June 1, 2005 Completion Date June 29, 2018 Location Nationwide Name of Implementing Partner Implementing Partner in the United States:  National Academy of Sciences, Washington Implementing Partners in Pakistan:  Higher Education Commission  Ministry of Science and Technology USAID/Pakistan Mission Strategic Framework Linkages Development Objective 4: Improved Opportunities of Learning and Work Budget United States Government:  USAID: $10.00 million  Department of State: $9.77 Government of Pakistan:  Higher Education Commission: $12.40 million  Ministry of Science and Technology: $0.26 million Total: $32.43 million 5 FIGURE 1: MAP OF S&T PROGRAM AREA 6 PROGRAM BACKGROUND USAID’s Pakistan-U.S. S&T Program provides research grants to Pakistani and American researchers to carry out joint research projects. The objective of these research partnerships is to build capacity at the institutional level in Pakistan and strengthen U.S.-Pakistan cooperative relationships in one or more of the following ways: (1) Enhance the ability of Pakistan's science and technology community to spur human and economic development; (2) Improve the quality, relevance, or capacity of education and research at Pakistani institutions of higher education in the field of science and technology; or (3) Increase the capacity of Pakistani research institutions to support industry competitiveness.3 The program is implemented in the United States by the NAS and in Pakistan by the HEC and MoST. Funding for U.S. researchers has been provided by USAID and DoS, and for Pakistani researchers by the Government of Pakistan’s HEC and MoST, with the exception of the eight Phase 4 projects on biological sciences supported on both sides with DoS funds. Five phases of funding were initiated between 2005 and May 2014, 4 and the call for proposals under the sixth phase, which will continue through 2018, will be issued in October 2014. USAID has funded all phases, with the exception of Phase 4, for which DoS provided NAS with all funds for U.S. institution grants, with HEC providing the majority of funds on the Pakistani side. An overview of Phases 1-5 is given in Annex I of the report. The U.S.-side grant budgets have ranged from $40,000 for a one-year project to $350,000 for three-year projects. On the Pakistani side, budgets have ranged from $30,000 to $500,000. HEC awards grants to Pakistani researchers and covers all in-Pakistan logistical expenses such as travel, meeting participation, and communication. MoST has not provided funds for any of their grants since early 2008, but has remained active in providing other support for the project such as participation in planning, reviews, and peer reviews of proposals. The list and value of grants is included in Appendix 4 of the evaluation Statement of Work (SOW) (Annex II of the report). The number of S&T grants by phase and focus area is reported in the table below. TABLE 2: NUMBER OF S&T GRANTS BY PHASE AND FOCUS AREA Focus Area Phase 1 Phase 2 Phase 3 Phase 4 Phase 5 Total Agriculture and Food Science 1 2 1 6 3 13 Earthquake and Disaster Management - 3 1 1 1 6 Engineering and Information Technology 3 1 5 3 1 13 Environment and Energy 2 1 1 3 1 8 Health 2 5 5 6 3 21 Physics/Chemistry/New Materials - - - 1 - 1 Plants 2 2 4 5 - 13 Water 1 2 2 2 1 8 Grand Total 11 16 19 27 10 83 While not a stated goal of the S&T Program, like the Fulbright Program, the DoS sees this program as promoting international cooperation for educational and cultural advancement. It also contributes to USAID’s strategic objectives in Pakistan and the priority objectives of Pakistan’s HEC. 3 Pakistan-United States Science and Technology Cooperation Program, Program Solicitation 2009 (Proposal Deadline: October 31, 2009). 4 Phase 5 started in mid-2013 and is not included in the evaluation. 7 PROBLEM OR OPPORTUNITY ADDRESSED In Pakistan, about 3 percent of 17-23 year-olds participate at the higher education level, while the participation rate in higher education in developed countries is 40 percent.5 Higher education in Pakistan suffers from problems such as the quality of faculty, students, libraries, and laboratories. Pakistan’s higher education facilities do not meet international standards.6 In 2001, Pakistan was ranked among the lowest in the world in higher education enrollment rates. Other developing countries in Asia, such as India and Korea, stood at 10 percent and 68 percent, respectively. According to a report by the Steering Committee for Higher Education, in 2001, only 2.6 percent of the students between ages 17-23 were enrolled in universities. The total increased to 2.9 percent in 2005,7 and by 2008, stood at 3.9 percent. The target for the Government of Pakistan was to almost double the percentage of college-age students by 2010 to 5 percent of the 17-23 age group.8 Under these circumstances, the S&T Program has been a timely intervention that helped the massive effort on the part of the Pakistan government to improve higher education in the country since 2003, when DoS and MoST signed a comprehensive Science and Technology Cooperation Agreement establishing a framework to increase cooperation in science, technology, engineering, and education for mutual benefit and peaceful purposes between the science and education communities in both countries. For actual implementation, the USAID Mission in Pakistan in 2005 agreed to assume funding and management responsibilities for the activity stemming from this agreement, termed the Pakistan-U.S. Science and Technology Cooperation Program.9 TARGET AREAS AND GROUPS The S&T Program is designed to strengthen the research capacity of public and private universities and research institutions across Pakistan. Research projects are not restricted to any particular geographic area of the country. As shown in the map in Figure 1, S&T has funded research at institutions located in all four provinces of Pakistan, the state of Azad Jammu and Kashmir, and all three federally-administered areas (Islamabad Capital Territory, the Federally Administered Tribal Areas, and the Gilgit-Baltistan region). Seventy￾five of the 83 grants have been awarded to researchers located in five main cities (Karachi, Faisalabad, Lahore, Peshawar, and the twin cities of Rawalpindi-Islamabad). The selection of researchers is based on the determined utility and relevance of the research proposal, and is not limited by any demographic characteristic. INTENDED RESULTS AND THE THEORY OF INTERVENTION The S&T Program contributes to the USAID/Pakistan Mission Strategic Framework (MSF) Development Objective 4 – Improved Opportunities of Learning and Work. By providing research grants to eligible candidates and establishing sustainable partnerships between U.S. and host country higher education and research institutions, the project will improve the quality of applied research produced by local Pakistani researchers in key sectors. The project contributes to USAID’s development objective for education in Pakistan by providing support to strengthen the capacity of university research laboratories and independent research organizations to produce high quality and relevant research. At universities, these investments also enable institutions to deliver a higher quality educational experience and develop research skills in their students through experiential, hands-on learning in the laboratory. These education quality improvements at the higher education level are complemented by U.S. Government investments in education quality and access at the primary level and strong policy engagement with the Government of Pakistan to ensure education provides each new generation of Pakistani children and youth with the foundational and specialized skills 5 This section is taken from the evaluation SOW provided to MEP by USAID/Pakistan. 6 “Problems and Prospects of Higher Education in Pakistan”, Higher Education Commission Pakistan, Source: http://eprints.hec.gov.pk/440/1/243.html 7 “Government of Pakistan, Report of the Steering Committee for Higher Education,” 2001, Government of Pakistan, Economic Adviser’s Wing, Finance Division. Pakistan Economic Survey 2005-06. Islamabad: 2006, p. 168. 8 “Ministry of Education Policy 1998-2010,” . 9 “Pakistan-United States Science and Technology Cooperation Program,” Program Solicitation 2009 (Proposal Deadline: October 31, 2009). 8 required to thrive, contribute, and compete in the local and global economy. In this way, the S&T Program contributes to the USAID/Pakistan mission to improve the opportunities of Pakistanis to access work-relevant learning. A summary of the S&T results framework is presented in Table 3. TABLE 3: S&T RESULTS FRAMEWORK Development Objective 4 Improved Opportunities for Learning and Work Intermediate Result (IR) 4.2 Improved Quality of Education Sub IR 4.2.3 Improved Teaching and Research at Higher Education Institutions in Key Sectors Indicators  Number of partnerships between U.S. and host country higher education institutions that address development needs  Number of host-country individuals who completed U.S. Government-funded short-term training or exchange programs involving higher education institutions APPROACH AND IMPLEMENTATION NAS works with the Pakistani-side counterparts (HEC and MoST) and the U.S. co-sponsors (USAID and DoS) to prepare and issue a joint call for proposals. Proposals must meet program criteria (relevance to development, modest overhead costs, true collaboration for research and training) and are jointly prepared and submitted by U.S. and Pakistani partner institutions. Research may be proposed in areas including but not limited to education, health, nutrition, water/sanitation, agriculture, democracy and governance, environment, energy (especially renewable forms), social sciences, economic development, technology transfer, entrepreneurship, innovation, and commercialization. NAS and HEC/MoST recruit technical experts to conduct peer reviews of the proposals. The proposals rated most highly by the reviewers on each side are jointly assessed by the U.S. and Pakistani co-sponsors, and a subset is selected by consensus for funding. The duration of most projects is three years. Criteria for awarding research grants are included in Appendix 5 of the evaluation SOW. NAS is the implementing agent for the S&T Program in the United States. In Pakistan, the main implementing partner is HEC, which actively works with NAS in determining the areas of research, the modalities, and the selection of the grantees. NAS disburses United States Government funds to U.S. researchers and HEC disburses Government of Pakistan funds to Pakistani researchers. 9 EVALUATION PURPOSE AND EVALUATION QUESTIONS The S&T Program has not been evaluated independently since 2008. As the program approaches its tenth year, USAID and DoS are interested in determining whether the purpose and approach of the S&T Program model remain relevant, effective, and vital as contributors to improved U.S.-Pakistan relations and enhanced research capacity in Pakistan. The evaluation is expected to assess if U.S. Government investment in S&T research partnerships has: 1. Sustainably strengthened scientific research capacity in Pakistan; 2. Developed applied research products that have yielded benefits to industry, government, or individual quality of life; 3. Fostered mutual understanding and goodwill between institutions and individuals in the United States and Pakistan as a result of academic collaboration; and, 4. Yielded the achievement of project targets against the project’s MSF indicators10 in a timely manner. The evaluation SOW (Annex II of the report) includes four main questions corresponding to the above￾mentioned expectations. It also provides explanations for each of these questions, which are summarized below and in the report. 1) How well have USG investments in the S&T project resulted in sustainably strengthened scientific research capacity in Pakistan? 2) To what extent have the research projects funded through the S&T program yielded benefits to industry, government or quality of life for the Pakistani people? 3) How successfully have research collaborations through the S&T project fostered mutual understanding and goodwill between institutions and individuals in the United States and Pakistan? 4) What factors affected timely project performance against its MSF indicators? 10 Refer to Table 3 for these indicators. 10 EVALUATION METHODS AND LIMITATIONS EVALUATION METHODOLOGY The methodology for this evaluation was developed through a process of consultation with USAID/Pakistan during and after the approval of the evaluation SOW.11 The methodology was qualitative and included review of relevant documents and primary data from individual interviews with U.S. and Pakistani S&T program managers, PIs of S&T grants, and Heads of Department (HODs) of Pakistani institutions that received S&T grants. MEP carried out a document review and analysis during and after the SOW preparation. This included review of project reports pertaining to 27 of the 73 S&T grants given during phases 1-4 of the program, which are included in the evaluation, and other documents (listed in the bibliography). MEP selected interview participants purposively according to a sampling plan agreed upon with USAID/Pakistan that included grants from all four phases, all eight focus areas, five major locations, and six grants for which women are the PIs. MoST and HEC facilitated access to PIs and HODs in Pakistan, and NAS facilitated access to PIs in the U.S. In total, the MEP evaluation team conducted 50 individual interviews, 29 in Pakistan and 21 in the U.S. The Pakistani side of the evaluation team conducted individual (personal) interviews in Faisalabad, the twin cities of Rawalpindi-Islamabad, Karachi, Lahore, and Peshawar12 with the Pakistani PIs for 16 of the 27 grants, nine associated HODs, and one representative each from USAID/Pakistan, DoS, HEC, and MoST. The U.S. team conducted individual (telephone) interviews with the PIs of 18 of the 27 grants as well as in￾person interviews with one representative each of USAID/Washington, DoS, and NAS. Eleven of the interviews in both Pakistan and the U.S. were paired PIs who were joint recipients on the same grants. The number of researchers interviewed in Pakistan and the U.S. is reported in Tables 4 and 5, and the number of HODs in Table 6. Annexes III and IV of the report provide additional information on the grants and locations of the PIs and HODs interviewed. TABLE 4: NUMBER OF RESEARCHERS INTERVIEWED IN PAKISTAN BY PHASE AND FOCUS AREA Focus Area Phase 1 Phase 2 Phase 3 Phase 4 Total 1. Agriculture and Food Science 2 1 3 2. Engineering and Information Technology 1 1 2 3. Environment and Energy 1 1 2 4. Health 1 1 1 3 5. Physics/Chemistry/New Materials 1 1 6. Plants 2 1 3 7. Water 1 1 2 Grand Total 2 3 5 6 16 11 By agreement with USAID/Pakistan, the methodology used in the evaluation differs from that proposed in the evaluation SOW. 12 The other locations include towns and cities (such as Gilgit, Hyderabad, Jamshoro, Quetta, Sargodha, and Topi) where very few researchers were based, and there are travel limitations due to remoteness or security concerns. 11 TABLE 5: NUMBER OF RESEARCHERS INTERVIEWED IN THE U.S. BY PHASE AND FOCUS AREA Focus Area Phase 1 Phase 2 Phase 3 Phase 4 Total 1. Agriculture and Food Science 1 1 2 2. Earthquake and Disaster Management 1 1 3. Engineering and Information Technology 1 1 4. Environment and Energy 1 2 3 5. Health 1 1 2 6. Physics/Chemistry/New Materials 1 1 7. Plants 1 3 1 5 8. Water 1 1 1 3 Grand Total 3 4 4 7 18 TABLE 6: NUMBER OF HEADS OF DEPARTMENT INTERVIEWED IN PAKISTANI INSTITUTIONS BY PHASES AND FOCUS AREAS Focus Area Phase 1 Phase 2 Phase 3 Phase 4 Total 1. Agriculture and Food Science 1 1 2. Earthquake and Disaster Management 1 1 3. Health 1 1 4. Physics/Chemistry/New Materials 1 1 5. Plants 2 1 3 6. Water 1 1 2 Grand Total 1 4 4 9 The evaluation team used a single instrument (agreed upon with USAID/Pakistan) consisting of 21 questions for all interviews, which is reproduced in Annex V; in seven instances, however, a question in the instrument was phrased in two different ways to address different groups of respondents. The instrument questions revolved around the four main evaluation questions and several sub-questions (including some that were implicit and not stated explicitly in the SOW). Based on knowledge of the project and the evaluation SOW, the team listed prompts for some of the instrument questions to ensure that interviewers would keep relevant issues in mind during the interviews. For all interviews, the team prepared summary interview notes consisting of bullet points that captured the responses. For data analysis, the evaluation team prepared a tally sheet (spreadsheet) consisting of themes that corresponded to evaluation sub-questions and sub-themes based on the prompts and additional issues brought up by the respondents. Responses encountered during the interviews and recorded in the summary notes were marked for counting purposes into the tally sheet, separately for U.S. and Pakistani respondents, under relevant sub-themes. To minimize intra-evaluator bias, the Pakistani members of the evaluation team cross￾checked the tally sheet prepared by the U.S. team members, and the two Pakistani teams (of two members each) cross-checked each other’s tally sheets. The basic approach to triangulation is to report the tally of Pakistani PIs’ responses for the sub-themes and then report the extent to which these responses were 12 supported by Pakistani HODs and U.S. PIs.13 Program managers’ responses have been reported mainly for questions 3 and 4. Documentary sources were also used for triangulation wherever they provided additional information on specific sub-themes. METHODOLOGICAL STRENGTHS AND LIMITATIONS As indicated above, the respondents interviewed for this evaluation represented 27 grants awarded in all the four phases included in the evaluation, all eight focus areas in which grants have been awarded, all five major locations where Pakistani grant recipients are based, and all six instances (among the 27 grants) in which the Pakistani PI was a woman. Four stakeholder groups – Pakistani and U.S. PIs, Pakistani HODs, and program managers from both countries – added further diversity to the sample. The sample allowed systematic triangulation across diverse groups of stakeholders, which was enhanced with the help of available documents to ensure the validity and reliability of findings. The evaluation team also employed cross-checks to minimize intra-evaluator bias. A high degree of dependence on grant receiving individuals and closely-linked heads of department is perhaps the main limitation of the methodology. To mitigate the possible effects of bias resulting from this, the evaluation team cross-checked interviewee responses against available documentation wherever possible (for example, to verify responses about academic publications and research products that have yielded benefits to industry, government, or ordinary people). It was not possible, however, to employ such cross-checks for some of the themes and sub-themes, such as gauging attitudes and mutual goodwill among Pakistani and U.S. program participants (question 3), and the extent and causes of payment and procurement problems (that emerged in response to question 4). 13 Responses that did not directly address evaluation questions and sub-questions were not included in the report. Sub-themes that elicited only one or two responses were also excluded. 13 Pakistani Researchers’ Examples of Skill Development  “The project contributed towards enhancing our soft skills and skills in proposal writing, creating a positive work environment, decision-making, and sharing ideas.”  “The project participants learned and applied advanced statistical analysis and computer modeling for the very first time in Pakistan.”  “The program encourages researchers to solve problems. It is not simply academic but hits the real issues through applied research.”  “Learned to write more effectively in scientific language and conduct experiments in sophisticated laboratories.” FINDINGS AND CONCLUSIONS FINDINGS AND CONCLUSIONS FOR QUESTION 1 Question 1: How well have U.S. Government investments in the S&T Program resulted in sustainably strengthened scientific research capacity in Pakistan? The answer to this question will assess improvements in the skills of Pakistani researchers (including research procedures/processes, use and maintenance of specialized equipment, and soft skills like critical thinking, creative problem solving, innovation, and analytical reasoning), S&T elements and experiences contributing to these improvements, and steps taken by the researchers and their institutions to sustain improvements in research capacity.14 The evaluation was also expected to identify adjustments that could be made to the program to achieve greater gender equity in academic research. The findings presented below focus first on researchers, then on their institutions, then on sustainability, and finally, on gender issues. FINDINGS Almost all Pakistani PIs (15 out of the 16 interviewed) reported that their skills had improved as a result of the S&T research grant (see box for examples). Almost all of the U.S. PIs interviewed (16 out of 18) and the HODs of Pakistani institutions (8 out of 9 interviewed) that received S&T grants supported this assessment. Most of the Pakistani PIs (11 out of 16) and HODs (6 out of 9) acknowledged improvements in their soft skills (such as critical thinking, creative problem solving, innovation, and analytical reasoning); some (5 out of 18) U.S. PIs agreed with this assessment. Most (12 out of 16) of the Pakistani PIs reported improvements in their ability to use and maintain laboratory equipment, and many (7 out of 18) U.S. PIs shared this perception. Most (11 out of 18) of the U.S. PIs reported that Pakistani researchers had adopted improved research procedures; some (4 out of 16) of the Pakistanis PIs, but none of the HODs, concurred with this view.15 Examples of improved research procedures cited by Pakistani respondents include a protocol for Polymerase Chain Reaction screening, Deoxyribonucleic Acid (DNA) screening, extraction and identification, and data recording. Some of the U.S. PIs observed that their Pakistani counterparts had adopted more rigorous scientific methodologies and planning. Examples given by the U.S. PIs include enhanced understanding of advanced scientific concepts, methodologies, and procedures (e.g. genetic variability assessments, DNA fingerprinting, and the use of controls), improved field research practices, better documentation of the work, and better use of advanced technologies and equipment. Most (10 out of 16) of the Pakistani PIs, almost all the HODs (8 out of 9), and most (11 out of 18) of the U.S. PIs felt that exchange visits, networking, and collaboration with U.S. researchers and institutions were beneficial in improving the skills of Pakistani researchers. Pakistani respondents felt that this aspect of the 14 The sense of question 1, as discussed in a meeting between USAID/Pakistan on June 11, 2014, is that “research capacity” includes researchers’ skills as well as the capacity of their institutions. 15 There are indications in the next paragraph and the box that Pakistani respondents reported some aspects of skill development as soft skills that U.S. respondents considered to be research procedures and processes. 14 program provided opportunities to interact with researchers, share ideas, and learn new techniques to undertake international standard research. Most of the Pakistani (10 out of 16) and U.S. (12 out of 18) PIs also appreciated the value of working with U.S. researchers for skill improvement. Most (9 out of 16) of the Pakistani PIs identified new laboratory equipment (for example, equipment for Polymerase Chain Reaction screening and DNA screening, extraction, and identification) as a particularly useful contributor to skill improvement.16 Some Pakistani (6 out of 16) and U.S. (5 out of 18) researchers felt that laboratory work in the U.S. had also contributed. Figures 2 and 3 present the interview findings reported above. FIGURE 2: IMPROVEMENTS IN PAKISTANI RESEARCHERS' SKILLS FIGURE 3: FACTORS CONTRIBUTING TO IMPROVED PAKISTANI RESEARCHERS' SKILLS 16 The “Evaluation of USAID Higher Education Portfolio, Volume 1, Assessment Report” (p. 51) by the Academy for Educational Development (2008) also mentioned Pakistani researchers’ appreciation of these elements of the S&T program: “All researchers were highly complimentary of the international exchanges and networking process, expertise and cooperation of the U.S. resource persons, facilities visited, laboratory techniques observed, computer simulation models, and workshop training received while in the U.S.” 15 11 12 4 16 5 7 11 0 2 4 6 8 10 12 14 16 18 Skills Improved Soft Skills Improved Skills to Use and Maintain Lab Equipment Improved Ability to Use Improved Research Procedures Improved Number of Responses Pakistan PIs (n=16) U.S. PIs (n=18) 10 10 9 6 5 11 12 1 5 0 0 2 4 6 8 10 12 14 Exchange Visits, Networking, Collaboration with U.S. Working with U.S. Researchers New Laboratory Equipment Laboratory Work in the U.S. Field Work in Pakistan Number of Responses Pakistan PIs (n=16) U.S. PIs (n=18) 15 Some (5 out of 16) Pakistani researchers with field-based research projects also appreciated the benefits of field work in Pakistan, a perspective that was completely missing among the U.S. PIs.17 Examples of field work include the installation of tube wells powered by solar panels in the Federally Administered Tribal Areas, identification of families in which genetic blindness occurs, and identification of buffalos with high milk production. Only two respondents from Pakistan said that the exchange visits were relatively less beneficial than other elements in building researchers’ capacity, while one from the U.S. said that the provision of laboratory equipment was relatively less beneficial. The evaluation encountered several ways in which respondents indicated institutional capacity development. Most (10 out of 16) of the Pakistani PIs associated it with improved laboratories, a view shared by some (5 out of 18) of the U.S. PIs. Most of the Pakistani (11 out of 16) and U.S. (10 out of 18) PIs also felt that improved research procedures indicated institutional capacity development. Most (9 out of 16 PIs) of the Pakistani PIs and some (5 out of 18) from the U.S. associated the improved soft skills of Pakistani researchers’ students with enhanced institutional capacity. Some (7 out of 16) of the Pakistani PIs also associated institutional capacity development with the improved teaching they claimed had resulted from the S&T grants, the improved quality and higher number of student dissertations (6 out of 16 PIs), new courses (5 out of 16 PIs), the researchers’ improved ability to secure grants (6 out of 16 PIs), and the enhanced status of departments and centers that received S&T grants (5 out of 16 PIs). For example, the National Institute of Biotechnology and Genetic Engineering (NIBGE) in Faisalabad started teaching a new course on genomics and its laboratory was strengthened with state-of-the-art equipment for undertaking research in cotton genomics. A reference laboratory was established in the institute through another S&T project for performing tests for finger prints and DNA. The relevant HOD reported that the project had helped multi-disciplinary institutional build-up and that two laboratories in the institution had been strengthened. A PI from the Center of Excellence in Molecular Biology in Lahore mentioned that at the start of the S&T grant, the institute could only perform rudimentary work in the field of forensic DNA, but after completion of the grant, the institute was capable of performing 100 percent of the work within the institution. He further added that 1,000 cases of forensic DNA analysis, including some high profile cases, had been solved by the institute. It was more difficult for U.S. PIs to assess whether the capacity of Pakistani research institutions to carry out research meeting international standards had increased.18 Nevertheless, through direct observation and continued contact with their Pakistani counterparts, some of the U.S. PIs identified areas in which Pakistani institutional capacity to conduct high quality scientific research had been strengthened. Specific examples pertained to the higher level of sophistication of scientific projects, expansion of subjects being researched, elevation of research standards, and availability of more advanced equipment and laboratories. Most of the Pakistani PIs and HODs (12 out of 16 and 5 out of 8, respectively) felt that working with U.S. researchers, exchange visits, and networking with U.S. researchers and institutions, were beneficial in developing institutional capacity; nine out of 18 of the U.S. researchers agreed with this view. Eight of the 16 Pakistani researchers also attributed institutional capacity development to new laboratory equipment, and five out of 16 of them to their students’ laboratory and research work in the U.S. For example, a mineral processing laboratory was established in the Department of Geology at the University of Peshawar, with S&T support. Another laboratory was established with S&T support in the Department of Chemistry at the Quaid-e-Azam University in Islamabad and is being used to conduct pioneering work in this field. 17 This could be due to U.S. PIs facing limitations on travel to and within Pakistan. 18 This is primarily because U.S. PIs deal with individual Pakistani researchers and it is hard for them to observe and assess increased institutional capacity. 16 Figures 4 and 5 illustrate the above-mentioned findings on institutional capacity development. FIGURE 4: INDICATORS OF CAPACITY DEVELOPMENT AT PAKISTANI INSTITUTIONS FIGURE 5: FACTORS CONTRIBUTING TO CAPACITY DEVELOPMENT AT PAKISTANI INSTITUTIONS 10 11 9 7 6 6 6 5 5 5 10 5 0 0 0 0 5 0 0 2 4 6 8 10 12 Improved Laboratories Improved Research Procedures Improved Soft Skills of Students Improved Teaching More and Better Student Dissertations New Courses Enhanced Researcher Ability to Secure Grants Enhanced Status of Department/Center Publications in High Impact Factor Journals Number of Responses U.S. PIs (n=18) Pakistan PIs (n=16) 12 10 8 5 4 3 3 3 9 9 4 6 0 4 0 0 0 2 4 6 8 10 12 14 Working with U.S. Researchers Exchange Visits, Networking, Collaboration with U.S. New Laboratory Equipment Students' Training and Laboratory Work in the U.S. Publications Researchers' Laboratory Work in the U.S. Students' Training and Laboratory Work in Pakistan New Curriculum Number of Responses Pakistan PIs (n=16) U.S. PIs (n=18) 17 One-half or more (8-10 out of 16) of the Pakistani researchers said they had built networks, approached industry for linkages, designed new courses, and trained students in research methods as steps toward the sustainability of improved research capacity. Some (6 out of 18) of the U.S. researchers were also aware of network-building. Some (6 out of 16) of the Pakistani PIs felt that teachers trained in improved research methods and staff trained in managing laboratory equipment were also relevant steps. A few (3 out of 16) of the Pakistani PIs had also applied for follow-up grants from S&T and other sources. Some (5 out of 16) of the Pakistani researchers reported that new curricula had been approved as a result of the steps they had taken19 and that they had received follow-up grants from S&T and other sources. A few (3 out of 16) of them reported successes in establishing linkages with government institutions and receiving funds for equipment maintenance. Eight of the 16 Pakistani PIs claimed greater use of their institution’s services (for example, higher numbers of graduate students) as a result of the S&T grant and the steps taken for sustaining the resulting improvements. For example, the Center of Excellence in Marine Biology at the University of Karachi started offering a new course in aquaculture based on knowledge gained from the S&T grant. The Department of Geology at the University of Peshawar built networks for collaboration with the Director General of Mines and Minerals at the Geological Survey of Pakistan and the COMSATS Institute of Information Technology to continue working on the project in the future. The Abdul Qadir Khan Institute of Bio-Technology and Genetics at the University of Karachi has made an arrangement with the Missouri Botanical Gardens and Tropicos (a Missouri-based database on plants) to ensure the sustainability of the project. Figure 6 is a visual summary of the reported results of the steps taken to sustain improved institutional development. FIGURE 6: RESULTS OF STEPS TAKEN TO SUSTAIN IMPROVED CAPACITY AT PAKISTANI INSTITUTIONS None of the respondents said that their institutions made any changes in the existing policies and procedures to sustain improvements in research capacity. Some (5 out of 16) Pakistani and half (9 out of 18) U.S. researchers emphasized the need to continue programs such as S&T. A review of the S&T requests for proposals suggests that researchers are not required to provide adequate assurances of institutional support 19 Changes in curriculum require a lengthy process culminating in HEC approval. 5 5 4 3 8 0 1 3 0 0 0 1 2 3 4 5 6 7 8 9 New Curriculum Introduced Follow-up Grant Received Funds Allocated for Equipment Maintenance Linkages Established with Government Greater Use of Institution's Services Number of Responses Pakistan PIs (n=16) U.S. PIs (n=18) 18 for the operation and maintenance of equipment proposed to be purchased with S&T funds and located in their institutions.20 All four Pakistani program managers and many of the PIs and HODs (6 out of 16 PIs and 4 out of 8 HODs) noted that the program is merit-based and does not discriminate on the basis of sex. Six of the 27 grants included in the field work for this evaluation are led by Pakistani women. The 2014 NAS Fact Sheet21 reports that “women represented 30 percent of the total exchange visitors.” It was beyond the scope of the evaluation to compare the proportion of women grant recipients and exchange visitors with the proportion of researchers in the relevant discipline who are women, which could provide a better basis for commenting on gender equity than the information reported above. A review of the S&T requests for proposals reveals that the PIs are expected to include women in their projects but there is no statement to the effect that female researchers are encouraged to apply for grants. A few of the Pakistani and U.S. researchers and HODs offered options for affirmative action such as increasing the number of grants for female researchers, targeting women researchers in some way, and setting a quota for women researchers. CONCLUSIONS The S&T Program strengthened scientific research capacity in Pakistan in two ways. First, judging by the views expressed by Pakistani and U.S. researchers and Pakistani HODs, S&T grants enhanced the skills of Pakistani researchers in carrying out applied research. There is evidence, in particular, of improvements in the soft skills of Pakistani researchers and their ability to use new laboratory equipment and research methods. Second, research funded through S&T helped develop the capacity of the institutions where the research was carried out. Capacity improved in terms of laboratories, research procedures, and the education imparted to students. Exchange visits, networking, and collaboration with U.S. researchers and institutions were particularly beneficial in improving the skills of Pakistani researchers and developing institutional capacity; working with new laboratory equipment also contributed to skill enhancement and institutional development. Improved skills among researchers and students represent human capital that will yield a flow of benefits in the future. Thus, these skills, supported by laboratory equipment that is maintained, can be expected to continue generating improved research in the years to come. Steps taken by Pakistani researchers for sustaining improved research capacity – obtaining follow-up grants from S&T and other sources (as many have among the researchers interviewed) and establishing effective linkages with government and business entities (which a few of the interviewed researchers have done) – suggest that many, if not most of them, will be able to continue in their endeavors. At the institutional level, however, there has been no change in policies to encourage researchers to continue with international standard research, and there is little evidence that researchers’ institutions have allocated resources for maintaining S&T-funded laboratory equipment made available by the institutions where it is located. The S&T grant award process does not address the need for sustainable operation and maintenance of laboratory equipment. Women are involved in S&T research to a limited extent, and this could be a reflection of male dominance in scientific research. The program has not introduced some of the means available for establishing more equal gender participation (outlined in the evaluation recommendations). 20 This is in contrast to standard government practice in Pakistan, whereby every project proposal must describe satisfactory operation and maintenance arrangements before a new project can be approved. 21 NAS, Pakistan-U.S. Science and Technology Cooperation Program Fact Sheet, January 2014. 19 FINDINGS AND CONCLUSIONS FOR QUESTION 2 Question 2: To what extent have the research projects funded through the S&T Program yielded benefits to industry, government, or quality of life for the Pakistani people? The evaluation identified products that have been developed, patented, and/or commercialized or otherwise used outside the laboratory through S&T research grants; the benefits of these products to businesses, government, or the public; academic publications and/or conference presentations on findings resulting from the grants; other benefits; efforts to generate or expand the impact of S&T-funded research; and the principal barriers or challenges to developing the capacity of Pakistani researchers to produce research that yields benefits to industry, government, or quality of life for the Pakistani people. FINDINGS A review of the project summaries prepared by NAS shows that some researchers provided training to government functionaries, disseminated information to them, and engaged them in discussion during the grant period, but none of the grants were initiated as partnerships in which researchers and the intended ultimate sellers, users, or owners22 of research products agreed at the outset to pursue a common objective and forge a partnership for this purpose. Project summaries sometimes speak of the expectation that a certain government department, private entity, or group of people may benefit from the researchers’ efforts, but there is no evident plan to this effect. While improving the well-being of ordinary Pakistanis is a general S&T objective, there is no evidence of specific, operational criteria that were used in the grant selection process to focus research on identifiable industry, government agencies, or segments of the population and promote appropriate partnerships for these purposes. Interviews with Pakistani and U.S. researchers suggest that they had developed or discovered 12 products in addition to research papers and presentations, which are: new method of plant disease control; new crop variety; identified anti-cancer plants; healthy seeds and feed; new method of medical treatment; wastewater treatment system; new chemical product (organic/inorganic); new approach for safer buildings; upgraded materials; sources/deposits of minerals; alternative source of energy; and DNA identification and extraction. Accounting for overlapping statements among respondents, Pakistani and U.S. researchers reported that 10 projects were benefitting businesses, government organizations, or segments of the public. In response to another question, however, only three Pakistani researchers and one from the U.S. reported that their products were in use by a community, business, or government entity. The 2014 NAS Fact Sheet23 notes that seven projects “reported developing prototypes, products, or submitted patents (one project developed 2 prototypes).” According to the Pakistani PIs of projects that have generated benefits outside the research setting:24  The project “Integrated Genetic/Physical Mapping of Gossypium” at the National Institute of Biotechnology and Genetic Engineering in Faisalabad helped a PI in the final stages of developing seven new cotton varieties. The PI estimated that 30 percent of the area planted to cotton in Pakistan (almost all of it in Punjab and Sindh Provinces) was under these varieties. He attributed the development of an eighth variety entirely to his S&T grant.  The project “Development of a Materials Connection Center” at the University of Peshawar reported that the project helped establish a laboratory that is being used by a government department in the Khyber Pakhtunkhwa Province to augment the mineral materials it uses in its studies. 22 This refers, for example, to businesses that sell products, consumers (such as farmers) that buy them, and owners such as government departments that adopt and enforce certain standards and tests that emerge from research. 23 NAS, Pakistan-U.S. Science and Technology Cooperation Program Fact Sheet, January 2014. The seven projects mentioned here are among the 73 undertaken during phases 1-4. 24 Independent verification of these reports was beyond the scope of the evaluation. 20  The project “Intensification of Forensic Services and Research at the Center for Applied Molecular Biology” in Lahore demonstrated techniques and processes that were adopted by government law enforcement agencies in the Punjab Province.  The project “Small-Scale Sewage Treatment and Wastewater Reuse System for Pakistan” at the Quaid-e-Azam University in Islamabad established a wastewater treatment plant that is being used to treat wastewater from the university’s residential colony. Two other projects have also generated benefits for Pakistani people but they are initiatives for capacity building and service delivery rather than research projects. One of them is the project “Enhancing the Minimally Invasive Surgery Skills of General Surgeons and Allied Surgical Specialists at the National Level,” which is located at the Holy Family Hospital in Rawalpindi. The second one is “Improving the Lifestyles of Villagers in Remote Areas of the Federally Administered Tribal Areas of Pakistan Using Renewable Energy,” a project of the National University of Science and Technology” in Islamabad, which provided water pumps powered by solar energy to six villages. In addition, there is one research project that is reported to have generated benefits for industry and farmers in the U.S., but not in Pakistan. This is the project “Development, Optimization and Application of a High-Performing Engineered Fertilizer” 25 of the National Institute of Food and Agriculture in Peshawar, whose Pakistani PI has moved to the U.S. Fifteen of the 16 Pakistani PIs and 14 out of 18 U.S. PIs, representing 21 of the 27 grants included in the evaluation, reported publishing their research work in journal articles. Eleven grants produced presentations for conference proceedings, 10 of them (excluding double counting across grants) in an international forum. Interview responses illustrating the range of dissemination products are summarized in Figure 7. The 2014 NAS Fact Sheet26 reports that the S&T Program as a whole had produced 377 publications (e.g., research papers and book chapters) and 519 conference presentations – an average of 5.2 publications and 7.1 conference presentations per project for the 73 phase 1-4 projects. FIGURE 7: DISSEMINATION OF RESEARCH FINDINGS A few (3-4 out of 16) of the Pakistani researchers felt that the S&T-assisted research promoted a culture of research in Pakistani institutions and improved Pakistan’s image in the U.S.; four out of 18 U.S. researchers agreed with the second part of this statement. A few (3 out of 18) U.S. researchers also mentioned that the S&T program enabled Pakistani researchers to gain international recognition and compete internationally. 25 This engineered fertilizer is reportedly patented in the U.S. and a U.S. firm is commercializing it. A South African firm has also contracted the PI to establish an industrial unit to produce fertilizer through this technique in their country. Pakistani researchers expect 1-4 more products to be patented or commercialized in due course (the projection varies across U.S. and Pakistani researchers). 26 Ibid. 15 2 7 4 1 14 0 5 0 2 0 2 4 6 8 10 12 14 16 Published in Journal Article Published in Book Presented in International Forum Presented in National Conference Presented in University Seminar Number of Responses Pakistan PIs (n=16) U.S. PIs (n=18) 21 During the grant period, a few of the Pakistani PIs initiated discussion with government (3 PIs) and business entities (2 PIs) and applied for follow-up grants (3 PIs). For example, NIBGE, Faisalabad, established linkages with waste management companies to sell technology for the production of biogas from solid waste, and the Quaid-e-Azam University, Islamabad, is collaborating with the Capital Development Authority for replicating their water treatment plant in the Islamabad Capital Territory. A few of the researchers held discussions with government and business entities and some received follow-up grants from S&T and other sources after the grant period. One researcher applied for a patent after the project ended. Many (9 out of 25) of the Pakistani PIs and HODs believed that the enabling environment and lack of funding for research and development are barriers to developing the capacity of Pakistani researchers; many of the U.S. PIs supported these views. Three of the 16 Pakistani PIs cited lack of cooperation and trust between researchers and business, and four mentioned the lack of support from industry, as factors that limit the benefits of research. In this vein, some (5 out of 16) Pakistani researchers also identified the lack of infrastructure (including electricity) and trained people who could maintain sophisticated new laboratory equipment. These findings are reflected graphically in Figure 8, which presents the combined responses of PIs and HODs. FIGURE 8: CHALLENGES/BARRIERS TO INCREASING THE BENEFITS OF RESEARCH CONCLUSIONS The S&T Program has a strong thrust toward academic publications and conference presentations: almost all of the grants have resulted in academic publications, most of the research work has been presented at international conferences, and the reported average output per grant is impressive. In line with some of its objectives, however, the program also expects research to spur economic development and support industry competitiveness. In practice, the benefits of research for business, government, and the general public have been limited. As S&T research projects are not designed (and not required to be designed) in partnership with clearly identified ultimate owners, sellers, and users of research products, it is understandable that products that can be used outside the research setting have emerged infrequently and more by chance or individual intent than program design. The results of follow-up research and ongoing discussion between some of the researchers and businesses are uncertain. Moreover, it is questionable whether a program such as S&T can 9 9 7 5 5 5 7 9 0 5 0 0 0 1 2 3 4 5 6 7 8 9 10 Lack of Enabling Environment Lack of Funding for Research and Development Lack of Support from Industry Lack of Infrastructure Lack of Cooperation between Researchers and Business Lack of Trust between Business and Research Number of Responses Pakistan PIs and HODs (n=25) U.S. PIs (n=18) 22 promote significant change to overcome macro-level barriers (identified by Pakistani researchers)27 to producing impact outside the research and academic settings. FINDINGS AND CONCLUSIONS FOR QUESTION 3 Question 3: How successfully have research collaborations through the S&T Program fostered mutual understanding and goodwill between institutions and individuals in the United States and Pakistan? The evaluation assessed the extent to which Pakistani project participants expressed positive views of the United States, and improved attitudes toward the United States, specific experiences that contributed toward changes in attitudes, and the extent to which U.S. and Pakistani institutions established partnerships and/or other forms of formal and informal cooperation as a result of the S&T project. FINDINGS Evidence suggests that the program fostered mutual understanding and collaboration between U.S. and Pakistani researchers and their departments. Twenty-seven out of the 29 Pakistani respondents and 15 of the 20 U.S. respondents felt that the S&T Program had improved mutual understanding and goodwill between Pakistan and the U.S. Fourteen of the 16 Pakistani PIs and all nine HODs identified exchange visits as a factor that led to positive views of and improved attitudes toward the U.S. Twelve of the 18 U.S. PIs expressed the same opinion. Most of the examples they gave pertained to project activities that took place in the U.S. Some (6 out of 16) of the Pakistani and most (10 out of 18) of the U.S. PIs specifically mentioned the contribution of working with U.S. researchers to improved attitudes. Some (5 out of 16) of the Pakistani PIs also appreciated the value of their laboratory work in the U.S. in building goodwill. Many (7 out of 18) of the U.S. PIs and a few (4) of the Pakistanis mentioned students’ laboratory work in the U.S. as a contributor. Figure 9 illustrates the kinds of responses mentioned here. 27 These include the enabling environment, lack of funding for research and development, and lack of cooperation between researchers and businesses. 23 FIGURE 9: FACTORS CONTRIBUTING TO ENHANCED PAKISTAN-U.S. UNDERSTANDING AND GOODWILL According to U.S. researchers, regular Skype calls, similar academic interests, collaborative laboratory work, social events that were not part of the project-funded activities, and general coordination of activities together were also important factors. These experiences helped build relationships and foster mutual respect, as well as instill a deeper appreciation among U.S. researchers of the scientific operating environment in Pakistan. Eight of the 16 Pakistani researchers reported that they have maintained post-project informal collaboration with U.S. researchers as well as formal institutional collaboration; they have also promoted formal or informal collaboration between Pakistani students and U.S. researchers (primarily for review of dissertations). Many (7 out of 18) of the U.S. researchers confirmed post-project informal collaboration with Pakistani counterparts. Examples of formal collaboration include: the University of Veterinary and Animal Sciences in Lahore established partnerships with Cornell University, the United States Department of Agriculture (USDA), and the University of Wisconsin that have continued even after the completion of the S&T grant. The Nuclear Institute for Food and Agriculture in Peshawar established linkages with USDA. The Center of Excellence in Marine Biology at the University of Karachi signed a Memorandum of Understanding with the U.S. National Institute of Oceanography for collaboration on further research. The University of Karachi signed a Memorandum of Understanding with a U.S. organization dealing in medicinal and herbal plants with the support of Missouri Botanical Gardens. CONCLUSIONS Feedback from Pakistani and U.S. academics and program managers is consistent with widespread mutual understanding and goodwill among individuals associated with S&T, with researchers in both countries highlighting the role of exchange visits in this outcome. Some Pakistani researchers continued to collaborate with U.S. researchers after completing their S&T research. The involvement of students in the program, the teaching and research from which they reportedly benefited, and the acquisition of laboratory equipment can be said to be ways in which the program would have appeared attractive to Pakistani institutions, and not only individual researchers. 14 6 5 4 3 3 0 12 10 0 7 4 0 3 0 2 4 6 8 10 12 14 16 Exchange Visits Working with U.S. Researchers Researchers' Laboratory Work in the U.S. Students' Training and Laboratory Work in the U.S. Students' Training and Laboratory Work in Pakistan Field Work in Pakistan Working Together on Publications / Dissertation Review Number of Responses U.S. PIs (n=18) Pakistan PIs (n=16) 24 FINDINGS AND CONCLUSIONS FOR QUESTION 4 Question 4: What factors affected timely project performance against its USAID/ Pakistan Mission Strategic Framework indicators? The evaluation was expected to identify factors that supported or hindered timely project performance and suggest how hindrances could be avoided and their effects mitigated. FINDINGS Some (4 out of 16 in each instance) of the Pakistani PIs reported that availability of funds, timely provision of funds, and support from partner researchers and research institutions supported timely performance. Some of the U.S. PIs mentioned timely provision of funds (4 out of 18 PIs) and support from partner researchers (5 out of 18 PIs) in this connection. They noted enabling factors such as the timely disbursement of funds, support of NAS, NAS reminders about reporting deadlines, and the assistance of the U.S. embassy staff (especially regarding processing and issuing visas). All four Pakistani program managers (including U.S. and Pakistan representatives) and most of the Pakistani (11 out of 16) and U.S. (12 out of 18) PIs identified delays in the release of funds to Pakistani researchers as a factor that hindered timely project performance. This is evidently an old and continuing problem, also highlighted in an earlier assessment.28 Some (4 out of 16) of the Pakistani PIs mentioned the lack of funds, referring to the reduction of scope and budget they faced at the application stage, and many (7 out of 16) highlighted delays caused by their institution’s procurement process. Some researchers (4 from Pakistan and 5 from the U.S.) mentioned visa problems,29 and five in total from Pakistan and the U.S. mentioned the law and order situation in Pakistan. According to the U.S. PIs, factors that hindered timely project performance were unreliable payment from HEC and MoST, language issues (e.g. understanding each other’s accents and difficulties writing in English, which is required for peer-reviewed scientific journals), infrastructure issues in Pakistan (unreliable access to electricity), procurement issues (e.g. import restrictions, unavailability of goods, and the underestimation of the cost of equipment), security and travel issues, and a management system in Pakistan which prevented access to newly-acquired equipment. Responses regarding factors hindering timely project performance are illustrated in Figure 10. 28 “Evaluation of USAID Higher Education Portfolio, Volume 1, Assessment Report” (p. 56) by the Academy for Educational Development (2008). 29 The “Evaluation of USAID Higher Education Portfolio, Volume 1, Assessment Report” (pp. 3-4) by the Academy for Educational Development (2008) also highlighted visa problems: “[E]xchange visits continue to be hampered by U.S. delays in issuance of visas and security issues for U.S. researchers.” 25 FIGURE 10: FACTORS HINDERING TIMELY PROJECT PERFORMANCE Many (10 out of 28) of the Pakistani and U.S. researchers wanted the timely release of funds to be ensured. Four of the 16 Pakistani and six of the 18 U.S. PIs felt that procurement rules and procedures were beyond their control. All four Pakistani program managers felt that the program could help with the timely release of funds, and three of them felt that procurement rules and procedures could be made less intensive. CONCLUSIONS Few researchers attributed timely project performance to the timely availability of funds, and a few appreciated the support provided by partner researchers and institutions in overcoming delays in funding and travel. There has been continuing concern among researchers about delays in the release of funds to Pakistani researchers that have hindered timely project performance. Other management issues that affected timeliness include delays in procurement. Important problems in program management have evidently not been addressed in a systematic manner so far, although many researchers expressed the need to resolve them and the program managers in Pakistan felt that solutions were possible. 11 4 7 4 1 12 6 0 5 4 0 2 4 6 8 10 12 14 Delay in Release of Funds Lack of Funds Procurement Issues and Delays Delays in Getting Visas Law and Order Situation Number of Responses Pakistan PIs (n=16) U.S. PIs (n=18) 26 RECOMMENDATIONS The evaluation offers four recommendations. Three of these can be implemented by modifying the grant award process to enable the S&T Program to generate greater practical benefits, better sustain results, and include more women in program activities. The fourth recommendation concerns timely project performance. It is recommended that the grant award process be modified in three ways to reflect the following suggestions: 1. The S&T Program objectives are aimed at both basic and applied research, the latter including research that spurs economic development and supports industry competitiveness. The program should add three measures for encouraging applied research to the grant announcement, proposal and selection processes. These measures are to: (a) clarify in the request for proposals/application how the program intends to promote applied research; (b) require researchers interested in applied research to demonstrate partnerships at the proposal/application stage with specific business and government entities for which they intend to produce certain products or solutions; and provide written confirmation of common objectives and contribution from such entities. In other words, proposals should describe not only partnerships between U.S. and Pakistani researchers but also between researchers and the end users of the proposed research. Research intended to benefit specific groups of people (for example, farmers, home builders, patients, and water users) should also be proposed as partnerships between researchers and relevant service providers in the public or private sector (for example, government line departments and regulatory agencies, or private manufacturers and organizations that represent community groups); and (c) give bonus points during the grant selection process to proposals/applications that demonstrate promising partnerships with or for the end users. 2. The S&T Program should require researchers who include equipment in their proposals to submit plans for the sustainable operation and maintenance of the equipment, including their institutions’ written commitment to allocate the space, staff, and finances required for this purpose. In addition, USAID should request the Higher Education Commission of Pakistan to ensure that institutions that will own the equipment allocate the required staff and finances for its operation and maintenance before they purchase the equipment. 3. The S&T Program should consider various options, in a stand-alone or mutually reinforcing mode, that could lead to greater inclusion of women in the program. One option is to specify a progressively increasing quota for female PIs in terms of the proportion of funding or number of grants. The quota could be for the program as a whole or for specified focus areas, including those (for example, agriculture) in which there are few women at present. Another option is to give weight or bonus points in the grant selection criteria to the number of women, including PIs, staff, and students, who will participate in the proposed research. A third option is to establish a support group consisting of program alumni that would seek out and mentor female grant applicants from Pakistan and the U.S. throughout the application and research process. In this scenario, the S&T Program should provide operational funds to support group members for their time, travel, communication, photocopying, and similar costs during the application process. It could also arrange training in proposal development and grant management. The fourth recommendation is aimed at improving the timeliness of project performance: 4. In consultation with program sponsors and implementing partners from Pakistan and the U.S., USAID should commission a management review focusing on known problem areas, including those highlighted in the evaluation. It was beyond the scope of the evaluation to determine the causes of delay in releasing funds to Pakistani PIs, the link between reporting and release of funds, or the causes of delays in hiring and procurement. An earlier (2008) evaluation was “not able to find a defensible reason for the delays [in payment]” but recommended that S&T should “streamline the administration of this program and ensure that payments are made in a timely fashion.” Streamlining has evidently not taken place and a management review would be useful in clarifying ways and means of solving problems. 27 BIBLIOGRAPHY  Agreement of Science and Technology Cooperation between the Government of the United States of America and the Government of the Islamic Republic of Pakistan, 2003.  Cooperative agreement of Pakistan-United States Science and Technology Cooperation Program, 2005.  Proposals, annual reports, and final technical reports of the 27 projects included in the evaluation.  Evaluation of USAID Higher Education Portfolio, Volume 1, Assessment Report, prepared by Academy for Educational Development.  Pakistan-United States Science and Technology Cooperative Program, Program Solicitation, 2005, 2006, 2009, and 2012.  Minutes of joint review and selection meetings for Phase I, II, III, and IV.  Project Summaries with U.S. Administrative Evaluation Comments (Phases 1, 2, and 3).  Pakistan-U.S. Science and Technology Cooperation Program, Phase Four, Project Summaries, August, 2012.  Annual and quarterly reports of the S&T Program. 28 ANNEXES ANNEX 1: OVERVIEW OF PHASES 1-5 OF THE S&T PROGRAM Between 2005 and May 2014, five rounds of funding were completed, and the call for proposals under the upcoming sixth phase, which will continue through 2018, is anticipated for October 2014. USAID has funded all phases, with the exception of Phase 4. In Phase 4, DoS provided NAS with all funds for U.S. institution grants, with HEC providing the majority of funds on the Pakistani side (with the exception of the eight projects on biological sciences supported on both sides with DoS funds). Phase 1 (Application deadline June 15, 2005): No special focus areas were included in the Request for Proposals (RFP). All 11 projects selected in this phase were funded by USAID on the U.S. side and HEC or MoST on the Pakistani side. General topical guidelines were established for this phase and used for all succeeding phases, as follows: “Projects funded under this solicitation should contribute to building capacity in Pakistan, and U.S.-Pakistan cooperative relationships, with one or more of the following goals: 1. Improve the quality, relevance, or capacity of education and research at Pakistani institutions of higher education in science and technical fields. 2. Improve the capacity of Pakistani public and private science institutions to support industry competitiveness. 3. Increase the capacity of science and technology to improve the well-being of ordinary Pakistani people. Topics include, but are not necessarily limited to: basic education, basic health, nutrition, water/sanitation, environment, and economic development.” Phase 2 (Application deadline July 31, 2006): No special focus areas were included in the main RFP, but a special supplementary RFP was issued with a pre-proposal application deadline of June 30, 2006, for earthquake-related research. “Projects funded under this solicitation should contribute to building capacity in Pakistan and strengthening Pakistan-U.S. cooperative relationships in earthquake-related research and development. Potential fields include, but are not limited to engineering; construction management; geology; seismology; risk assessment; building code design and compliance; urban planning; and sociological, psychological, and health impacts. Pre-proposals must explain and demonstrate relevance to earthquake preparedness, damage mitigation and/or remediation, and earthquake recovery and reconstruction.” All 16 projects were funded by USAID on the U.S. side and HEC or MoST on the Pakistani side. Phase 3 (Application deadline June 30, 2007): No special focus areas were included in the RFP. However, at the time projects were selected for funding, the Bio-Engagement Program (BEP) of DoS provided funds to support the U.S. sides of seven projects in their areas of interest (biological and biomedical research, particularly on infectious diseases affecting humans or animals). The remaining 11 projects were funded by USAID on the U.S. side and by HEC and MoST on the Pakistani side. Phase 4 (Application deadline November 16, 2009): In addition to the regular broad areas of eligibility, the RFP also included the following special focus areas, which were supported with funds from BEP and its related Chemical Security Program (CSP): “As part of the current call for proposals, we are pleased to offer a special opportunity for proposals in the biological sciences, with emphasis on human and animal infectious disease, as well as best practices in laboratory management. Another special opportunity exists for proposals in the chemical sciences that focus on best practices in chemical management, pesticide management, and chemical safety education.” BEP and CSP funded a total of eight projects (BEP funds covered both the U.S. and Pakistani sides of the projects, while CSP funds covered only the U.S. side). The rest of the projects supported in this phase (18) were funded on the U.S. side by public diplomacy funds from DoS. No USAID funds were used for this phase. 29 Phase 5 (Application deadline November 30, 2012): The following language explains the special focus areas for this phase: “The program welcomes proposals across a wide range of topics, including, but not limited to, education, health, nutrition, water/sanitation, agriculture, democracy and governance, environment, energy (especially renewable forms), social sciences, and economic development. As part of the current call for proposals, we are pleased to offer a special opportunity for proposals focused on technology transfer, entrepreneurship, innovation, and commercialization. Proposals submitted in response to this special focus area should be designed to (1) build the capacity of university and public sector researchers and technology entrepreneurs to connect with the private sector to support technology commercialization; (2) identify new technologies with potential market value; (3) scale up a prototype or process for full-scale production; or (4) provide training and mentorship on technology transfer, administration, and other relevant functions related to technology development and commercialization. Private companies may not apply or receive funding, but may participate as partners on projects.” Following the completion of the parallel peer review process on the Pakistani and U.S. sides and the joint review and selection meeting involving the co-sponsors on both sides, 10 research proposals were awarded under Phase 5 grants in August 2013. USAID funds were used to support the U.S. side of the selected projects, and HEC funded the Pakistani side of the projects (MoST did not have funds available). 30 ANNEX II: S&T EVALUATION STATEMENT OF WORK PAKISTAN-UNITED STATES SCIENCE AND TECHNOLOGY COOPERATION (S&T) PROGRAM MIDTERM PERFOMANCE EVALUATION EVALUATION STATEMENT OF WORK MAY 2014 31 PAKISTAN-UNITED STATES SCIENCE AND TECHNOLOGY COOPERATION (S&T) PROGRAM MIDTERM PERFORMANCE EVALUATION EVALUATION STATEMENT OF WORK 32 ACRONYMS BEP Bio-Engagement Program CAS Centers for Advanced Studies DoS Department of State FCR Findings, Conclusions and Recommendations HEC Higher Education Commission IMEC Independent Monitoring and Evaluation Contract IP Implementing Partner GoP Government of Pakistan MNBSP Merit and Needs-Based Scholarship Program MOU Memorandum of Understanding MoST Ministry of Science and Technology MSI Management Systems International NAS National Academy of Science RFA/P Request for Applications/Proposals ROI Return on Investment S&T Science and Technology Program SOW Statement of Work USAID United States Agency for International Development USG United States Government 33 I. BACKGROUND INFORMATION A. Identifying Information about the Program This Statement of Work (SOW) outlines the requirements for a consulting firm (the consultant) to conduct a midterm performance evaluation of the United States Agency for International Development (USAID)’s Pakistan￾United States Science and Technology Cooperation (S&T) Program. The S&T Program provides research grants to Pakistani and American universities and research institutions to carry out joint research projects. The objective of these research partnerships is to build capacity at the institutional level in Pakistan and strengthen U.S.-Pakistan cooperative relationships in one or more of the following ways: (1) Enhance the ability of Pakistan's science and technology community to spur human and economic development; (2) Improve the quality, relevance, or capacity of education and research at Pakistani institutions of higher education in the field of science and technology; or (3) Increase the capacity of Pakistani research institutions to support industry competitiveness. The program is implemented in the United States by the American National Academy of Sciences (NAS) and in Pakistan by the Higher Education Commission (HEC) and the Ministry of Science and Technology (MoST). NAS works with the Pakistani side counterparts (HEC and MoST) and U.S. co-sponsors (USAID and Department of State) to prepare and issue a joint call for proposals. Proposals must meet program criteria (relevance to development, modest overhead costs, true collaboration for research and training) and are jointly prepared and submitted by U.S. and Pakistani partner institutions. Research may be proposed in areas including but not limited to education, health, nutrition, water/sanitation, agriculture, democracy and governance, environment, energy (especially renewable forms), social sciences, economic development, technology transfer, entrepreneurship, innovation, and commercialization. NAS and HEC/MoST both recruit technical experts to conduct peer reviews of the proposals. The proposals rated most highly by the reviewers on each side are jointly assessed by the U.S. and Pakistani co-sponsors, and a subset is selected by consensus for funding. The duration of most projects is three years. Criteria for the awarding of research grant proposals are provided in Attachment 3. Additional information may be accessed at the NAS S&T website: http://sites.nationalacademies.org/PGA/dsc/pakistan/index.htm Between 2005 and May 2014 five rounds of funding were completed, and the call for proposals under the upcoming sixth phase, which will continue through 2018, is anticipated for October 2014. USAID has funded all phases (through the State Department), with the exception of Phase 4. In Phase Four, the State Department provided NAS with all funds for US institution grants, with HEC providing the majority of funds on the Pakistani side (with the exception of the eight projects on biological sciences supported on both sides with State funds). The U.S.-side budgets of the grants have ranged from $40,000 for a one-year project to $350,000 for three-year projects. On the Pakistani side, budgets have ranged from $30,000 to $500,000. HEC covers all in-Pakistan logistical expenses such as travel, meeting participation, and communication. Pakistan’s Ministry of Science and Technology is a stakeholder to the program and initially committed a total of $1.5 million for Phases 1-3 but has only contributed $260,000. MoST has not provided funds for any of their grants since early 2008. Due to their failure to honor previous grant commitments, no grants requiring MoST co-funding were selected in Phases 4 or 5. However, MoST has remained active in providing other support for the project such as participation on planning, reviews, and peer reviews of proposals. While not a stated goal of the program, like the Fulbright Program, it is seen by the Department of State (DoS) as promoting international cooperation for educational and cultural advancement. It also is seen to contribute to USAID’s strategic objectives in Pakistan and the priority objectives of Pakistan’s HEC. 34 TABLE 1: PROJECT SUMMARY B. Development Context: 1. Problem or Opportunity Addressed Pakistan ranks among South Asia’s lowest performing countries on many education indicators and will not achieve the Millennium Development Goal (MDG) of universal primary education by 2015. Pakistan’s poor performance in education persists despite a vast body of research that demonstrates a clear link between education and poverty alleviation, attainment of long-term development goals, and sustainable economic growth. In other words, the relative lack of educational attainment in Pakistan does more than just minimize job opportunities; it is also a key contributor to the nation’s ongoing political and social instability. In the mid-2000s, Pakistan’s adult literacy rate was 54 percent compared to 53 percent in Bangladesh, 57 percent in Nepal, 66 percent in India and 92 percent in Malaysia. Although Pakistan’s Tertiary Gross Enrollment Ratio (GER) more than doubled during the 2005-2010 period, it still sits at only 5 percent, compared to Bangladesh at 9 percent, India at 15 percent, and Malaysia at 37 percent. Pakistan’s expenditure on higher education has been declining in recent years, from 2.6 percent of GDP in 2009 to 2.1 percent of GDP in 2012. According to the HEC of Pakistan, Pakistani universities award about 700 Ph.D. degrees per year and lack qualified research faculty. Only 20 percent of the faculty in Pakistan’s higher education institutions have Ph.D. degrees. Universities are poorly governed, and the management structures and practices are ineffective. For example, until recently there has been an ineffective monitoring of staff credentials, such as previous degrees and research. A lack of sufficient facilities in remote/rural areas prevents most rural students from pursuing higher education. Girls have been particularly disadvantaged, as evidenced by Pakistan’s 2008-09 Gender Parity Index of 0.37 for degree (college & university) enrollment. The difference between male and female enrollment is even more acute in rural areas, which have a Gender Parity Index for degree enrollment of 0.24. Universities often serve as incubators for new ideas and innovative solutions through institutional relations (i.e. partnerships, networks, resource sharing, problem-solving assistance). However, for the most part, higher education in Pakistan is characterized by poorly trained researchers who are focused more on theoretical rather than applied research that often does not meet international standards. Poorly trained faculty produce poorly trained students and thus university faculties generally are not well qualified to impart the technical knowledge and research skills to graduates. This knowledge, and particularly the skills developed in a rigorous applied research program – critical thinking, innovation, creative problem solving and more – form the core competencies of a 21st century workforce capable of responding to a rapidly changing global marketplace, successfully acting on Title / Field Program Information Agreement Number 391-ESP-A-00-05-00001 Agreement Officer’s Representative (AOR) Jerry O’Brien, Office of Afghanistan and Pakistan Affairs Start Date June 1, 2005 Completion Date June 29, 2018 Location Nationwide Name of Implementing Partner (IP) National Academy of Sciences, Washington USAID/Pakistan Mission Strategic Framework Linkages Development Objective 4: Improved Opportunities of Learning and Work. Budget US $12.4 million 35 opportunities for new business creation, introducing innovation into stagnant industries, and solving Pakistan’s most intransigent development challenges. A 2011 USAID/Pakistan assessment of the leading Pakistani universities in agriculture, water, and energy found a significant lack of applied research undertaken to support small farmers, urban dwellers, or the private commercial and industrial sectors, despite businesses’ demand for applied research and entrepreneurs’ expressed desire to collaborate with these universities. In addition, Pakistan universities are typically constrained by limited funding, which prevents modernization of outdated laboratory equipment and restricts scholarship opportunities for talented but disadvantaged students. The study found that even Pakistan’s highest ranking universities lack the capacity to conduct in-depth research to identify applied science and technology solutions to pressing development challenges. In the view of the problems and opportunities outlined above, in 2003, the U.S. Department of State (DoS) and Government of Pakistan (GoP) Ministry of Science and Technology (MoST) signed a comprehensive Science and Technology Cooperation Agreement that established a framework to increase cooperation in science, technology, engineering, and education for mutual benefit and peaceful purposes between the science and education communities in both countries. In January 2005, the USAID Mission in Pakistan agreed to assume funding and management responsibilities for the activity stemming from this agreement, termed the “Pakistan-U.S. Science and Technology Cooperation Program”. This activity is intended to increase the strength and breadth of cooperation and linkages between Pakistan’s scientists and institutions with counterparts in the U.S. Each country has contributed funds to support projects that would enhance the ability of the science and technology community to positively contribute to human and economic development in Pakistan 2. Target Areas and Groups The S&T Program is designed to strengthen research capacity at public and private universities and research institutions across Pakistan. Research projects are not restricted to any particular geographic area of the country. The selection of Pakistani researcher beneficiaries is dependent on the determined utility and relevance of the research proposal, and is not limited by any demographic characteristic. Research topics are limited to applied science and technology fields and topics that stand to contribute significantly to Pakistan’s national development. C. Development Hypothesis The S&T Program contributes to the USAID/Pakistan Mission Strategic Framework Development Objective 4 – Improved Opportunities for Learning and Work. By providing research grants to eligible candidates and establishing sustainable partnerships between US and host country higher education and research institutions, the project will improve the quality of applied research produced by local Pakistani researchers in key sectors. The project contributes to the Agency’s development objective for education in Pakistan by providing support to strengthen the capacity of university research labs and independent research organizations to produce high quality and relevant research. At universities, these investments also enable institutions to deliver a higher quality educational experience and develop research skills in their students through experiential, hands on learning in the laboratory. These education quality improvements at the higher education level are complemented by USG investments in education quality and access at the primary level and a strong policy engagement with the government of Pakistan to ensure education provides each new generation of Pakistani children and youth with foundational and specialized skills required to thrive, contribute and compete in the local and global economy. In this way, the S&T Program contributes to the USAID/Pakistan Mission to improve the opportunities of Pakistanis to access work-relevant learning. D. Intended Results The S&T Program contributes to USAID/Pakistan’s Mission Strategic Framework (MSF) Objective 4: “Improved Opportunities of Learning and Work”. A summary of the S&T results framework is presented in Table 2. 36 TABLE 2: S&T RESULTS FRAMEWORK Development Objective 4 Improved Opportunities for Learning and Work Intermediate Result (IR) 4.2 Improved Quality of Education Sub IR 4.2.3 Improved Teaching and Research at Higher Education Institutions in Key Sectors Indicators  Number of partnerships between U.S. and host country higher education institutions that address development needs  Number of host-country individuals who completed U.S. Government-funded short-term training or exchange programs involving higher education institutions E. Approach and Implementation NAS is the implementing agent for the S&T program. In Pakistan, its main partner is the HEC which actively works with NAS in determining the areas of research, the modalities, and the selection of the grantees. All funds for the program go to NAS, which provides the grant funds to the American institutional counterpart of each grant, which – in turn – disburses the funds to the Pakistani partner. HEC does not receive any support for its participation in the program and covers its related staff and other costs. F. Current Status of Activities - Pakistan-U.S. Science and Technology Cooperation Program, Phases 1-5 Phase 1 (Application deadline June 15, 2005): No special focus areas were included in the RFP. All 11 projects selected in this phase were funded by USAID on the U.S. side and HEC or MoST on the Pakistani side. General topical guidelines were established for this phase and used for all succeeding phases, as follows: “Projects funded under this solicitation should contribute to building capacity in Pakistan, and U.S.-Pakistan cooperative relationships, with one or more of the following goals: 1. Improve the quality, relevance, or capacity of education and research at Pakistani Institutions of higher education in science and technical fields. 2. Improve the capacity of Pakistani public and private science institutions to support industry competitiveness. 3. Increase the capacity of science and technology to improve the well-being of ordinary Pakistani people. Topics include, but are not necessarily limited to: basic education, basic health, nutrition, water/sanitation, environment, and economic development.” Phase 2 (Application deadline July 31, 2006): No special focus areas were included in main RFP, but a special supplementary RFP was issued with a pre-proposal application deadline of June 30, 2006, for earthquake-related research. “Projects funded under this solicitation should contribute to building capacity in Pakistan and strengthening Pakistan-U.S. cooperative relationships in earthquake-related research and development. Potential fields include, but are not limited to engineering; construction management; geology; seismology; risk assessment; building code design and compliance; urban planning; and sociological, psychological, and health impacts. Pre￾proposals must explain and demonstrate relevance to earthquake preparedness, damage mitigation and/or remediation, and earthquake recovery and reconstruction.” All 16 projects were funded by USAID on the U.S. side and HEC or MoST on the Pakistani side. 37 Phase 3 (Application deadline June 30, 2007): No special focus areas were included in RFP. However, at the time projects were selected for funding, the Bio-Engagement Program (BEP) of the Department of State provided funds to support the U.S. sides of seven projects in their areas of interest (biological and biomedical research, particularly on infectious diseases affecting human or animals). The remaining eleven projects were funded by USAID on the U.S. side and by HEC and MoST on the Pakistani side. Phase 4 (Application deadline November 16, 2009): In addition to the regular broad areas of eligibility, the RFP also included the following special focus areas, which were supported with funds from BEP and its related Chemical Security Program: “As part of the current call for proposals, we are pleased to offer a special opportunity for proposals in the biological sciences, with emphasis on human and animal infectious disease, as well as best practices in laboratory management. Another special opportunity exists for proposals in the chemical sciences that focus on best practices in chemical management, pesticide management, and chemical safety education.” BEP and CSP funded a total of eight projects (BEP funds covered both the U.S. and Pakistani sides of the projects, while CSP funds covered only the U.S. side). The rest of the projects supported in this phase (18) were funded on the U.S. side by public diplomacy funds from the Department of State. No USAID funds were used for this phase. Phase 5 (Application deadline November 30, 2012): Following is the language explaining the special focus areas for this phase: “The program welcomes proposals across a wide range of topics, including but not limited to education, health, nutrition, water/sanitation, agriculture, democracy and governance, environment, energy (especially renewable forms), social sciences, and economic development. As part of the current call for proposals, we are pleased to offer a special opportunity for proposals focused on technology transfer, entrepreneurship, innovation, and commercialization. Proposals submitted in response to this special focus area should be designed to (1) build the capacity of university and public sector researchers and technology entrepreneurs to connect with the private sector to support technology commercialization; (2) identify new technologies with potential market value; (3) scale up a prototype or process for full-scale production; or (4) provide training and mentorship on technology transfer, administration, and other relevant functions related to technology development and commercialization. Private companies may not apply or receive funding, but may participate as partners on projects.” Following the completion of the parallel peer review process on the Pakistani and U.S. sides and the joint review and selection meeting involving the co-sponsors on both sides, 10 research proposals were awarded under Phase 5 grants in August 2013. USAID funds were used to support the U.S. sides of the selected projects, and HEC funded the Pakistani sides of the projects (MoST does not have funds available). Phase 6: This phase is expected to launch in summer 2014. An application deadline and selection criteria have not yet been announced. This evaluation will focus exclusively on phases one through five. II. RATIONALE FOR EVALUATION A. Purpose of the Evaluation The S&T program was envisioned as a means of furthering Pakistan-U.S. scientific cooperation as well as strengthening Pakistan’s research capacity, with the overarching purpose of enhancing the ability of the science and technology community to positively contribute to human and economic development in Pakistan. A 2008 evaluation of USAID/Pakistan’s higher education portfolio concluded the following regarding the S&T Program: The Pakistan-U.S. Science and Technology Cooperation Program has provided significant capacity building for staff at universities and institutes in applied research, assisted with critical technology transfer, and holds the promise of important contributions to national development in critical areas such as agriculture, health, and water resources. It has contributed to HEC goals of faculty development, quality improvement in science and technology, and expansion of the research sector. Links with U.S. universities and research institutions were very effective, although exchange visits continue to be hampered by U.S. delays in issuance of visas and security issues for U.S. researchers. 38 The S&T Program has awarded 83 research grants to US and Pakistani researchers since 2005 and has not been evaluated independently since 2008. As the program approaches its tenth year, USAID and State are interested in determining whether the purpose and approach of the S&T Program model remain relevant, effective, and vital as contributors to improved US-Pakistan relations and enhanced research capacity in Pakistan. The primary purpose of this evaluation is to independently evaluate whether the project has performed as anticipated against its indicators, and whether the project’s approach has delivered the expected higher level outcomes inherent in the development hypothesis that informed project design. In short, this evaluation will determine if USG investment in S&T research partnerships has: 1) Sustainably strengthened scientific research capacity in Pakistan; 2) Developed applied research products that have yielded benefits to industry, government, or individual quality of life; 3) Fostered mutual understanding and goodwill between institutions and individuals in the United States and Pakistan as a result of academic collaboration; and 4) Yielded the achievement of project targets against the project’s MSF indicators30 in a timely manner. This evaluation will assess the past performance of the S&T program, propose opportunities to improve future performance, and identify promising areas for future investments to build scientific research capacity in Pakistan while strengthening positive relations between Pakistan and the United States. The findings of this evaluation will inform both the design of future science and technology partnerships and the implementation of USAID/Pakistan’s Centers for Advanced Studies project, which will seek to develop comparable capacity at four selected Pakistani universities beginning in June 2014. The analysis within this evaluation should therefore identify lessons learned and recommend promising opportunities for future programming to further advance points 2-4 above. The evaluation will help USAID and DoS tailor future programming to most effectively meet the USG development and public diplomacy objectives. B. Audience and Intended Use The results of this evaluation will be shared with all stakeholders and the general public. USAID, NAS, MoST, and HEC will review the final drafts for comments. The evaluation team will provide a debriefing to USAID and the American Embassy in Islamabad at the time the draft is completed, followed by a separate teleconferenced debrief with NAS and HEC. The evaluation report will be utilized to tailor future programming to most effectively meet USAID’s objectives for higher education in Pakistan, including the CAS Project. C. Evaluation Questions This section presents the core evaluation questions based on the previous section. The questions are designed to address discrete aspects of the program to provide a view of the broad effects of the Science and Technology Program. The basic questions and the associated explanations will provide the information necessary for the evaluation team to develop the specific questions that will guide data collection efforts. The evaluation team should respond to the following key questions for completed studies. 1) How well have USG investments in the S&T project resulted in sustainably strengthened scientific research capacity in Pakistan? a. To what extent do project participants believe Pakistani researchers have improved skills in developing and carrying out applied research to international standards, including research procedures/processes, use and maintenance of specialized equipment, and soft skills like critical thinking, creative problem solving, innovation, and analytical reasoning? 30 Refer to the indicators list [in section III.D]. 39 b. What specific elements or experiences within the S&T project were most beneficial in developing the above skills, and which elements were less beneficial? c. What adjustments to the program (if any) can be made to achieve greater gender equity in academic research? d. What steps have project participants and their institutions taken that have best supported the sustainability of improvements in research capacity (e.g. policy work, network building, champions, fundraising, equipment maintenance investments etc.) and what USG support may be required to support further adoption of these good practices? What are the results of these efforts? 2) To what extent have the research projects funded through the S&T program yielded benefits to industry, government or quality of life for the Pakistani people? a. What specific benefits have each of the products developed through S&T research grants yielded to businesses, government, or the public? b. What proportion of applied research products have been developed, patented and/or commercialized or otherwise used outside the lab, and what are the characteristics of the beneficiaries of these applied research products (types of beneficiaries [businesses, government, or the public], regional distribution of beneficiaries, if applicable)? c. What proportion of the research grants has led to academic publication and/or conference presentations of the findings? d. In what ways have the S&T research projects yielded benefits beyond those identified above? e. What efforts are underway to generate or expand the impact of this research, and by what processes do these efforts occur after the end of the research grant period? f. What are the principal barriers or challenges to developing the capacity of Pakistani researchers to produce research that yields benefits to industry, government, or quality of life for the Pakistani people? 3) How successfully have research collaborations through the S&T project fostered mutual understanding and goodwill between institutions and individuals in the United States and Pakistan? a. What proportion of Pakistani project participants express positive views of the United States, and improved attitudes toward the United States? b. What specific experiences contributed toward improvements in attitudes? c. To what extent have U.S. and Pakistani institutions established partnerships and/or other forms of formal and informal cooperation as a result of the S&T project? 4) What factors affected timely project performance against its MSF indicators? a. What factors supported timely project performance? b. What factors hindered timely project performance, and how can these be mitigated? III. EVALUATION DESIGN AND METHODOLOGY A. Data Collection Methods The evaluation will use a snapshot mixed methods design to analyze and document the outcomes of the Pakistan S&T Program for Phases I through IV. This design recognizes that research grants made in Phases V and VI are ongoing, requiring that the evaluation questions above remain focused on the research projects undertaken in 40 Phases I-IV. The evidence-based methodology should establish clear and defensible findings, conclusions, and recommendations and comply to the greatest possible extent with the USAID evaluation policy for performance evaluations. The evaluation team will probe the outcomes and results as deeply as feasible within time and resource constraints. To facilitate analysis, the team will collect and report data in a way that enables disaggregation across multiple dimensions including, but not limited to, sex, academic discipline, and institution. The evaluation will rely on both quantitative and qualitative evidence to answer the evaluation questions and draw these data from both primary and secondary sources. Potential sources of secondary data include NAS, USAID, HEC, MOST, DoS, and relevant documents coming from the research itself and/or host institutions. The evaluation expects to employ the following data collection methods. Document review - The evaluation team will collect and review all relevant project documents from USAID, NAS, HEC, MOST, DoS, the project records of the AOR for the S&T activity, and other sources identified during the evaluation. Relevant reports include the HEC’s Medium Term Development Framework (2011-2015), the 2008 Evaluation of USAID’s Higher Education Portfolio, NAS reports, program agreement documents, previous evaluations, and program management manuals. Survey - A questionnaire is suggested as the instrument for collecting both quantitative and qualitative information about each of the research questions above. This survey will be administered to identified participants in each of the research grants issued in Phases I-IV; participants in Phase V and VI will be omitted from data collection. In designing this survey instrument, certain questions will lend themselves to a Likert-style question with responses provided in the form of descriptive statistics, while others will require an analysis of open-ended question responses. The proposed division between these two forms of survey question is as follows:  Likert scale question - answer presented as descriptive statistics: 1a, 1c, 1d, 2b, 2c, 3a, 3b  Open-ended question - answer presented as analysis of qualitative data: 1b, 1e, 1f, 2a, 2d, 2e, 2f, 3c, 4a, 4b This survey may be administered through a combination of media, including online, in written format (email) or over the telephone. HEC maintains contact information for all S&T participants. NAS also has contact information, although it is unclear how current it is. If the evaluation team discovers certain projects have become inactive or are incomplete, this should be documented and shared in the evaluation. Group Interviews - USAID anticipates the evaluation team will organize one or more group interviews with Pakistani researchers to discuss the four overarching evaluation questions in greater depth to add richness to the evaluation findings. Three such discussions, including one in Islamabad and two in the provinces, are recommended; participants should be selected from large and small institutions, and institutions located in urban and rural settings should be represented. Although group interviews with U.S. researchers may not be feasible, U.S. researchers may be contacted by internet, telephone, or post as required. The evaluation team should propose the parameters and guidelines for the selection of participants approach. Recommended evaluation participants include: i) Researchers at Pakistani universities and research and development (R&D) institutions (group interviews and surveys), ii) Researchers at participating U.S. universities and R&D institutions (group interviews and surveys), Key Informant Interviews - Time and budget allowing, the evaluation team may also wish to meet with Pakistan officials at HEC, MoST and participating universities and R&D institutions, and officials at NAS, USAID/Pakistan and the State Department to further enrich their understanding of the project’s impact. B. Data Analysis Methods The evaluation team will use the results of the performance data analysis, surveys, interviews, and document reviews to provide the analysis in response to the objectives of this evaluation and the questions raised above. Each of the sub-questions will be explored in detail and the findings synthesized into the response to the key question they feed into. Graphs and other visuals will be used to illustrate the descriptive statistics provided. Person level data will be disaggregated by gender, and by other variables as appropriate and informative. 41 C. Methodological Strength and Limitations The methods proposed for collecting and analyzing evaluation information are potentially subject to selection bias. Selection bias occurs when the subjects of surveys or interviews are not representative of the population of interest. In this case, selection bias is most likely to result from online survey respondents self-selecting themselves. S&T researchers who were not successful for various reasons will be less likely to respond to the survey. The evaluation relies heavily on the timely response of S&T researchers and officials to the survey and every effort should be made to ensure these responses are received. The evaluation team should follow up closely with non-responders to encourage participation in the evaluation. D. Existing Data Detailed data regarding the grant funded research under each of the first five phases of the S&T project are available through three sources: 1) A table listing all funded grants by category, amount and other information for phases 1 – V is attached in Appendix1; 2) the 2012 NAS report on the USAID funded projects for phases 1 - IV which will be forwarded to MSI; and 3) the independent evaluation of the S&T Program conducted by the Academy for Education Development in 2008, referenced previously, which will be provided to the evaluation team. Other data and documents are available and will be provided by USAID/Pakistan, NAS, and HEC as required. These include Agreements between the USG and GoP, and between USAID and NAS as well as more detailed information and data on the program. Several websites containing information about the S&T program are also available. These include the following the NAS website and the HEC websites: http://sites.nationalacademies.org/PGA/dsc/pakistan/index.htm; http://www.hec.gov.pk/Pages/HECMain.aspx According to the USAID/Pakistan Performance Indicator Reference Sheet (PIRS), the Washington-based AOR for the S&T program should have data (ideally housed on the PakInfo data management system) on the following indicators, which are part of the Mission Strategic Framework (MSF):  Number of U.S.-host country joint development research projects  Number of new policies to which USG-supported organizations have contributed  Number of partnerships between U.S. and host country higher education institutions that address development needs  Number of host-country individuals who completed USG-funded short-term training or exchange programs involving higher education institutions  Number of US-supported tertiary education programs that included experiential and/or applied learning opportunities HEC officials will be available to meet with the evaluation team. HEC may also be helpful in providing contact information on the Pakistani universities, research institutes and individuals involved in the grants program. OAPA and the E3 bureau will assist the evaluation team with contacting US partners. E. Evaluation Process The evaluation process consists of the following main stages: Stage 1: Initial planning and preparation by the evaluation team (two weeks). The consultant and evaluation team members will finalize the SOW in collaboration with USAID during a team planning meeting. Following USAID approval of the SOW, the evaluation team will identify and obtain key documentation and conduct a thorough desk review. The team will organize key informant interviews with USAID, NAS, HEC, and other S&T stakeholders to inform the development of data collection instruments. The team will then gather, verify, and update if needed all contact information for survey and interview participants, schedule individual and group interviews as required, and organize travel arrangements as needed. The written surveys and interview guides will be developed during 42 this stage and shared with USAID for review and approval. Prior to beginning survey and interview data collection, the evaluation team will provide an in-brief to USAID and other key stakeholders detailing the evaluation approach and timeline at the end of week two. Stage 2: Interview and survey data collection (four weeks). Following USAID approval of the survey instrument, the evaluation team will begin to collect survey data over a four week period, with outreach to non-respondents beginning in the third week of data collection. Key informant and group interviews will be conducted simultaneously by two or more field teams (two to three researchers per team) as needed over the first three weeks of this period. Stage 3: Data analysis and reporting (10 weeks, excluding Eid holiday week). Data analysis will begin immediately following data collection and carried out simultaneously for all data types; two weeks will be required for quantitative survey data analysis, while qualitative data analysis will require three weeks. Following completion of fieldwork and analysis, the evaluation team will prepare and deliver a debriefing presentation to USAID and other key S&T Project stakeholders with USAID approval. The evaluation team will incorporate comments from the presentation(s) into a draft report. After a thorough technical review, the evaluation team will deliver the draft report to USAID – and to implementing partners if appropriate - for review and comment. Once the evaluation team receives comments on the draft report, it will incorporate the comments, send the report to the evaluation team home office for a final technical review, editing, and branding, and then deliver the final report to USAID at the end of week sixteen. IV. TEAM COMPOSITION A. Evaluation Team Positions and Skills The contractor should propose a mixed team of local and consultant staff, which may include the following: - A Team Leader/Evaluator of the evaluation team will require expertise in the areas of applied research and university-market linkages. This expertise, in addition to a strong background in evaluation and research methods, will enable the evaluation team leader to fully comprehend the nuances of the questions and responses to be explored through this study. S/he also must have good oral and written communication skills. USAID considers this to be a key personnel position and must approve her/his selection. - A Quantitative Research Specialist with experience in sampling and launching surveys for designing, coordinating, and administering the online/telephonic/mail surveys and group interviews. - A Qualitative Research Specialist will analyze the quantitative and qualitative data from the questionnaires, group interviews, and semi-structured interviews. - A short-term Education Specialist - An independent National Expert (short-term) in education to provide advice and assist in developing data collection instruments, interpreting results, and writing specific sections of the evaluation report. - Two or more local short-term Qualitative Research Assistants will be recruited to assist in leading key informant and group interviews, including facilitation, data collection, data coding/cleaning, and data analysis. In addition to this evaluation team, the consultant will work closely with all the potential stakeholders, including USAID, implementing partners, beneficiaries (grantees), host government (HEC a/o MoST), and others. Disclosure of conflict of interest: All evaluation team members will provide a signed statement attesting to a lack of conflict of interest, or describing an existing conflict of interest relative to the project being evaluated. 43 V. EVALUATION MANAGEMENT A. Logistics In terms of logistics, this assignment requires inputs and actions from the evaluation partners as summarized below: USAID/Pakistan  The Program Office’s Performance Management Unit (PMU) along with Education team will facilitate the preparation of the evaluation SOW in accordance with USAID standards and good practices, review the instruments and the draft report, and provide technical inputs on the contractual matters.  The education team will provide relevant information and, as required, facilitate on-the-ground meetings and interviews set up by NAS and the evaluation team with beneficiary organizations and other stakeholders. National Academies of Science  NAS will provide contact lists, documents, and other data as required to the evaluation team, and will dedicate adequate time to ensure the evaluation team has the information required to complete its work. Beneficiary researchers and institutions  The staff from selected beneficiary organizations is expected to cooperate with the evaluation team by making time to share relevant documents and provide data as required through surveys, meetings and/or interviews. The Consulting Firm (MSI)  The consulting firm will provide support for travel, lodging, and other arrangements related to the evaluation team’s work and will take the lead in scheduling meetings with participants to collect data. 44 B. Scheduling The complete process, including finalization of the SOW, review of documents, developing survey and interview tools, data collection and analysis, and report writing and finalization will require approximately 16 weeks. Activity W1 W2 W3 W4 W5 W6 W7 W8 (Eid) W9 W10 W11 W12 W13 W14 W15 W16 Planning, document review Scheduling, instrument development, in-brief, individual interviews (USAID), SOW finalization Data collection - survey Field work - interviews Data analysis - quantitative Data analysis - qualitative Initial findings debriefing Report writing Submission of draft report USAID review and comments Revisions, home office review Final report submission to USAID 45 C. Budgeting Level of Effort (days - the consultant may propose additional staff) Tasks Team Leader Qualitative Research Specialist Quantitative Research Specialist Education Specialist (STTA) Qualitative Research Assistant 1 (STTA) Qualitative Research Assistant II (STTA) Planning, Document Review 5 5 5 5 Scheduling, instrument development, in-brief, individual interviews (USAID), SOW finalization 5 5 5 5 Data collection - survey 10 20 Field work – interviews 10 15 15 15 Data analysis - quantitative 5 10 2 Data analysis - qualitative 5 15 3 8 8 Initial findings debriefing 1 1 1 1 Report writing 13 9 9 5 Submission of draft report 1 USAID review and comments 1 Revisions, home office review 13 2 2 2 Final report submission to USAID 1 Total 70 52 52 23 23 23 46 VI. EVALUATION DELIVERABLES A. Deliverables 1. Briefing(s) to USAID, State Department, NAS, HEC and other stakeholders regarding preliminary findings, conclusions and recommendations after conclusion of fieldwork (week of August 4, 2014); 2. Draft report to USAID (August 29, 2014); 3. Final report to USAID (September 26, 2014). B. Report Content The evaluation report will follow standard guidelines as laid out in Appendix 1 of USAID’S Evaluation Policy and operationalized in ADS 203.3.1.8 (Documenting Evaluations), reproduced in Annex 2. The evaluation report will follow the structure given below (the section titles and order are illustrative):  Title page  Table of Contents;  Table of tables and figures;  List of acronyms  Acknowledgements or preface (optional);  Program summary  Map showing the location of program activities  Executive summary which will be 3-5 pages in length that summarizes key points (project purpose and background, key evaluation questions, methods, findings, etc.)  Introductory chapter;  The Development Problem and USAID’s Response (1-3 pages): This section will describe the development problem USAID wants to address. This will include USAID’s response to the problem, the development hypothesis and theory of change, results framework, and project implementation (including the current status of the project or activity);  Purpose of the mid-term evaluation and evaluation questions (1-2 pages): This section will include the purpose of the Study and state all questions;  Evaluation Design, Methodology and Limitations (1-3 pages): A written design which includes key questions, methods, main features of data collection instruments; an explanation of why these methods were chosen, with additional information in the annex as necessary; limitations of the methodology and how these have been accounted for; and data analysis plan;  Findings and Conclusions: If there are a large number of findings, there will be a synthesis or summary of findings for each question that establishes the connection with the conclusions that follow.  Recommendations  References; and  Annex  Evaluation Statement of Work  Evaluation Methods and Limitations  Data Collection Instruments  Bibliography of Documents Reviewed  List of individuals and agencies contacted and places visited  Meeting notes of all key meetings with stakeholders.  Disclosure of Any Conflicts of Interest  Statement of Differences (only if applicable)  Evaluation Team Bios 47 SOW APPENDICES APPENDIX 1: TABLE OF EVALUATION QUESTIONS BY DATA SOURCES, COLLECTION AND ANALYSIS METHODOLOGIES Data Collection Evaluation Question Type of Answer/ Evidence Method Sources Sampling/Selection Data Analysis Methods Question No. 1: How well have U.S. Government investments in the S&T project resulted in sustainably strengthened scientific research capacity in Pakistan? Descriptive  Description of overall trends, with illustrative examples  Descriptive statistics  Quantitative analyses  Objective assessment with respect to evaluation question, will illustrative examples  Survey  Group interview  Document review  Key informant interviews Survey:  Online/telephone/mail survey of grantee beneficiaries Group interview  Relevant beneficiaries (researchers, private sector etc.) Document review:  Project data records  Assessment reports Key informant interview:  Relevant USAID, NAS and university officials  Purposive/ Convenience sampling for selection of survey, key informant, and group interview participants  Frequency tables and cross-tabs of survey data, with disaggregation  Identification of trends and themes across data sources  Linking qualitative data to quantitative survey analyses to explain quantitative findings Question No. 2: To what extent have the research projects funded through the S&T program yielded benefits to industry, government, or quality of life for the Pakistani people? Descriptive  Description of overall trends, with illustrative examples  Descriptive statistics  Quantitative analyses  Objective assessment with respect to evaluation  Survey  Group interview  Document review  Key informant interviews Survey:  Online/telephone/mail survey of grantee beneficiaries Group interview  Relevant beneficiaries (researchers, private sector etc.) Document review:  Project data records  Assessment reports  Purposive/ Convenience sampling for selection of survey, key informant and group interview participants  Frequency tables and cross-tabs of survey data, with disaggregation  Identification of trends and themes across data sources  Linking qualitative data to quantitative survey analyses to explain quantitative findings 48 Data Collection Evaluation Question Type of Answer/ Evidence Method Sources Sampling/Selection Data Analysis Methods question, will illustrative examples Key informant interview:  Relevant USAID, NAS, and university officials Question No. 3: How successfully have research collaborations through the S&T project fostered mutual understanding and goodwill between institutions and individuals in the United States and Pakistan? Descriptive  Description of overall trends, with illustrative examples  Objective assessment with respect to evaluation question, will illustrative examples  Descriptive statistics  Survey  Group interview Survey:  Online/telephone/mail survey of grantee beneficiaries Group interview  Relevant beneficiaries (researchers, private sector etc.)  Purposive/ Convenience sampling for selection of survey and group interview participants  Frequency tables and cross-tabs of survey data, with disaggregation  Identification of trends and themes  Linking qualitative data to quantitative survey analyses to explain quantitative findings Question No. 4: What factors affected timely project performance against its MSF indicators? Descriptive  Description of overall trends, with illustrative examples  Objective assessment with respect to evaluation question, will illustrative examples  Group interview  Key informant interviews  Document review Group interview  Relevant beneficiaries (researchers, private sector etc.) Key informant interview:  Relevant USAID, NAS and university officials Document review  Project records  Purposive/ Convenience sampling for selection of survey, key informant and group interview participants  Frequency tables of grant completion time from project record data  Identification of trends and themes across data sources 49 APPENDIX 2: REPORTING GUIDELINES According to ADS 203.3.1.8 (Documenting Evaluations), evaluation reports must meet the following criteria: 1. Evaluation reports must represent a thoughtful, well-researched, and well-organized effort to objectively evaluate what worked in the project, what did not work, and why. 2. Evaluation reports must address all evaluation questions included in the scope of work. The evaluation report should include the evaluation statement of work as an annex. The technical officer (who is the COR when the evaluation is conducted by a contractor) must agree upon, in writing, all modifications to the statement of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline. 3. Evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists, and discussion guides will be included in an annex in the final report. 4. When evaluation findings address outcomes and impact, they must be assessed on males and females. 5. Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). 6. Evaluation findings must be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. 7. Sources of information must be properly identified and listed in an annex. 8. Recommendations must be supported by a specific set of findings and should be action-oriented, practical and specific, with defined responsibility for the action 50 APPENDIX 3: LIST OF AVAILABLE DOCUMENTS - Evaluation of S&T Program, 2008 - S&T Project Summaries for Phases 1, 2, 3, 4, and 5 - Descriptions of all active and concluded S&T projects since 2008 51 APPENDIX 4: CURRENT STATUS OF ACTIVITIES USAID-funded projects in Pakistan-U.S. Science and Technology Cooperation Program (Phases 1, 2, 3, 4, and 5). Project Title Start Date End Date USAID Funds PAK Funds (not USAID) Pakistanis Trained Agriculture and Food Science Establishment of Extrusion Center of Institute of Food Science and Technology 1/1/2006 12/31/2007 $320,000 $389,000 21 Development of Biosecure, Sustainable, and Cost-Effective Culture Technologies for Edible Shrimp (Fenneropenaeus merguiensis) in Pakistan 1/1/2007 12/31/2011 $116,003 $271,677 6 Development, Optimization, and Application of a High-Performing Engineered Fertilizer 5/1/2007 4/30/2011 $116,250 $100,000 117 Discovery of Genetic Variation that Enhances Improvement of Dairy Production and Health in Cattle and Buffaloes 4/1/2008 3/31/2011 $183,700 $351,000 117 **New Approaches of Estrus Synchronization to Improve Reproductive Performance in Dairy Animals 11/15/2010 11/14/2012 $88,780 $98,093 395 **Establishment of Functional and Nutraceutical Food Research Section at the National Institute of Food Science and Technology, U. of Agriculture, Faisalabad 11/15/2010 11/30/2013 $127,409 $123,086 109 **Establishing a Biostatistical Consulting Center 11/15/2010 9/30/2014 $338,332 $76,113 (DoS funds passed through HEC) 224 **Arthropod Functional Genomics Initiative: Building Community Resources for Animal Health 11/15/2010 9/30/2014 $351,671 $206,146 (DoS funds passed through HEC) 15 **Characterization of Mycolplasma gallisepticum Isolates from Pakistan and their Use in Production of Diagnostic Antigen and Vaccine 11/15/2010 9/30/2014 $63,450 $225,440 (DoS funds passed through HEC) 100 52 **Capacity Building, epidemiology, and Risk Assessment of Endemic and Emerging Tick-Borne Disease in KPK and FATA, Pakistan 11/15/2010 9/30/2014 $172,186 $140,303 (DoS funds passed through MoST) 250 Targeted Engineering of Brassica Juncea Seed Biochemistry to Produce Reduced-Viscosity Plant Oils for Direct Use as Biofuel 10/1/2013 9/30/2016 $ 114,488 $ 78,315? 0 The Establishment of Fruits and Vegetables Processing Center in The Institute of Food Science and Nutrition, University of Sargodha 10/1/2013 9/30/2015 $81,987 $159,625? 0 Collaborative Research for Genetic Conservation and Improvement of Pakistani Goats 10/1/2013 9/30/2016 $295,688 $531,433? 0 Subtotal $2,369,944 $2,750,231 1,354 Health Intensification of Forensic Services and Research at Center for Applied Molecular Biology 1/1/2006 12/31/2007 $160,000 $118,650 88 Antimicrobial Resistance in Pakistan 1/1/2006 3/31/2010 $75,000 $500,000 1,282 Multiplex Immunoassays for the Detection of Tuberculosis 2/1/2007 1/31/2009 $189,500 $173,945 14 Capacity Building in Research Ethics and for Research on Ethics 2/1/2007 11/30/2009 $150,000 $232,732 160 Nanomedicine for Cancer Research 2/1/2007 10/31/2011 $250,000 $137,219 317 Establishment of Virtual Trainer Lab for Improving Minimally Invasive Surgery Skills 3/1/2007 3/31/2011 $213,000 $270,000 146 Building Molecular Biology Capacity for Preventing Tick-Transmitted Diseases in Pakistan 3/1/2007 4/30/2012 $290,000 $225,451 227 Development of DNA Database for Convicted Offenders in Pakistan (project terminated, funds returned) 5/1/2008 4/30/2010 $0 $0 0 **Hepatitis C Virus Management in Pakistan 7/1/2008 6/30/2011 $250,000 $300,000 2 **Establishment of an Integrated Technology, Pathology Imaging, and Immunohistochemistry Facility in Pakistan 7/1/2008 6/30/2010 $164,500 $143,000 5 **Capacity Building of Lady Health Workers in Rural Mardan, KP, Through the Use of ICT-Based 7/1/2008 1/30/2012 $38,325 $111,000 30 53 Technologies **Novel Triple Acting Chimeric Antimicrobials for Eradication of Multi-Drug Resistant Not available Not available (U.S. project was not managed by NAS but directly by State ISN/CTR $80,300 0 **A Molecular Approach to Prevent Hereditary Blindness in Pakistan 11/15/2010 11/14/2014 $270,326 $236,801 12 ** Water, Sanitation, Health and Hygiene Interventions in a Northern Pakistani Village 11/15/2010 8/31/2014 $146,372 $138,020 91 **Prevalence, Species Distribution, and Trends in Resistance of Fungi Responsible for Invasive Mycoses in Pakistan 11/15/2010 9/30/2014 (DoS funds sent directly to U.S. agency) $256,088 (DoS funds passed through HEC) 25 **Rapid Detection of Infection and Drug Resistance in Tuberculosis Patients by Multiplex Analysis 11/15/2010 9/30/2014 $267,360 $262,342 (DoS funds passed through HEC) 7 **Hepatitis B Virus-Associated Hepatocellular Carcinoma in Pakistan 11/15/2010 9/30/2013 $311,990 $274,984 (DoS funds passed through HEC) 0 **Enhancing the Minimally Invasive Surgery Skills of General Surgeons and Allied Surgical Specialists at the National Level 2/1/2011 1/31/2013 $93,150 $204,000 (DoS funds passed through MoST) 309 Saving Maternal and Infant Lives with Affordable Technology 10/1/2013 9/30/2015 $154,080 $148,779? 0 Development and Commercialization of a Blood-based Tuberculosis Diagnostic Test 10/1/2013 9/30/2016 $255,748 $298,500? 0 Study of Magnetic Materials for Hyperthermia Treatment of Cancer 10/1/2013 9/30/2016 $166,060 $402,085? 0 Subtotal $3,538,561 $4,413,246 2,715 Earthquake and Disaster Management Education and Learning after the Pakistani Earthquake: Can the Children Recover? 2/1/2007 1/31/2011 $83,700 $240,000 162 Development of a Framework for Probabilistic Seismic Hazard Maps for Pakistan 2/1/2007 10/31/2011 $174,705 $130,000 69 Building Pakistan's Capacity for Instruction, Research, and Practice in Earthquake Engineering and 2/1/2007 10/31/2011 $241,595 $220,000 388 54 Retrofit Response Modification Factors of Typical Pakistani Reinforced Concrete and Masonry Buildings for Pakistani Seismic Code Development 4/1/2008 10/31/2011 $100,750 $66,000 9 **Capacity Building in Disaster Risk Assessment and Management through Training and Research in Geoinformatics and Hydrometerological Hazard Risk Reduction Strategies 11/15/2010 6/30/2014 $230,000 $147,513 8 Neotectonic and Earthquake-Hazard Studies of the Chaman Fault, Western Pakistan 10/1/2013 9/30/2016 $249,819 $210,719? 3 Subtotal $1,080,569 $1,014,232 639 Engineering and Information Technology Development of a Strategic Model for Improvement of Construction Project Management Education, Research, and Practice in Pakistan 1/1/2006 12/15/2009 $130,000 $275,000 780 Development of Guidelines for Asphalt Pavement Recycling in Pakistan 1/1/2006 12/31/2009 $350,000 $500,000 376 Development of Computational Mechanics Infrastructure and Human Resources for Advancing Engineering Design Practices in Pakistani Industry 1/1/2006 1/31/2010 $220,000 $302,000 153 Development of an ITS-Based Traffic Management Model for Metropolitan Areas of Pakistan with Karachi as a Pilot Study 2/1/2007 9/30/2010 $94,000 $195,988 1,369 Synthesis and Characterization of Smart Polymer Microgels for Biomedical Applications 4/1/2008 7/31/2011 $178,645 $100,000 12 Development of a Materials Connection Center 4/1/2008 9/30/2011 $254,000 $188,000 88 Implementation of SuperPave Binder and Asphalt Mix Specification to Improve Pavement Performance in Pakistan 4/1/2008 11/30/2011 $189,000 $411,000 100 55 Telephone-based Speech Interfaces for Access to Information by Non￾Literate Users 6/1/2008 1/31/2011 $125,000 $60,000 43 **Technology for the Poor: Low￾Cost Information and Computing Technology 7/1/2008 6/30/2010 $220,000 $96,000 6 **Establishment of a Center of Excellence to Conduct and Promote Construction Safety Research, Education, and Training in Pakistan 11/15/2010 8/31/2014 $208,035 $107,661 205 **Integration of Geological, Geochemical, and Remote Sensing Data for Finding Source Rocks for Gold in the Northern Areas of Pakistan 11/15/2010 5/31/2014 $200,000 $159,914 63 **Ambient Air Quality Monitoring Using Integrated Secure Wireless Sensor and Vehicular Networks 11/15/2010 4/30/2014 $145,608 $35,997 57 Capacity Building for Pakistan in Fire Risk Management 10/1/2013 9/30/2016 $172,184 $215,771? 0 Subtotal $2,486,472 $2,647,331 3,252 Plants Understanding and Control of Plant Viral Disease Complexes in Pakistan 1/1/2006 12/15/2009 $175,000 $142,000 18 Gene Pyramiding through Genetic Engineering for Increased Salt Tolerance in Wheat 1/1/2006 6/30/2010 $350,000 $47,880 24 Secure Pakistan Wheat Production through Controlling Rusts 2/1/2007 1/31/2011 $100,001 $241,884 7 Identification and Cloning of Drought-Related Genes in Wheat (T. aestivum) 2/1/2007 4/30/2012 $109,963 $100,000 9 Biotechnological Approaches to the Control of Ascochyta Blight of Chickpea 3/1/2008 9/30/2011 $127,219 $67,000 4 **Integrated Genetic/Physical Mapping of Gossypium 7/1/2008 6/30/2012 $160,000 $140,000 5 **Enhance Sugarcane Production in Pakistan by Modern Breeding Technology 7/1/2008 6/30/2013 $85,000 $71,000 6 **Management of Greening by Producing Healthy Plants, Monitoring Vectors, and Identification of Tolerance 7/1/2008 6/30/2011 $159,500 $245,000 5 56 **Characterization and Utilization of Gossypium arboreum as a Source of Resistance Against Begomoviruses that are a Threat to the Cotton Crop in Pakistan and the United States 7/18/2011 7/17/2014 $115,522 $106,000 6 **Bioactive Genes and Peptides/Proteins from Medicinal Plants 11/15/2010 11/14/2013 $317,000 $117,565 13 **Inventory of the Plants of Pakistan 11/15/2010 3/31/2014 $46,021 $45,306 10 **Building Capabilities for the Molecular and Biochemical Characterization of Photosynthesis and Oxidative Stress Gene Expression in Halophytes with Potential Use As Nonconventional Crop 11/15/2010 2/28/2015 $180,000 $165,283 29 **Standardization and Quality Assurance of Medicinal Plants 11/15/2010 8/31/2014 $281,000 $104,502 18 Subtotal $2,206,226 $1,593,420 154 Environment and Energy Improving the Lifestyle of Villagers in Remote Areas of Federally Administered Tribal Areas of Pakistan Using Renewable Energy 1/1/2006 12/31/2006 $40,000 $100,000 14 Determination of Heavy Metals and Polycyclic Aromatic Hydrocarbons in Airborne Particulates in Lahore, Pakistan, and Madison, Wisconsin, USA 1/1/2006 12/31/2008 $80,000 $107,625 12 Association of Particulate Matter with Daily Morbidity in Urban Population 2/1/2007 10/31/2011 $126,295 $148,739 71 Capacity Building and Collaborative Research for Assessing Impact of Climate Change on Glaciers of the Karakoram Himalaya 4/1/2008 10/31/2011 $230,000 $90,000 11 **A Hybrid Solar Water Heating System using CO2 As Working Fluid 11/15/2010 12/31/2013 $164,115 $76,566 9 **Solid Waste Management for Bioenergy Production 10/1/2011 2/28/2015 $200,000 $257,000 3 57 **Total Solution Based Organic￾Inorganic Solar Cells for Enhanced Efficiency and Stability 11/15/2010 10/31/2014 $215,790 $239,714 118 New Approaches for Lower Cost, Longer Ability, and Higher Efficiency of Dye-Sensitized Solar Cells 10/1/2013 9/30/2016 $236,410 $393,559? 0 Subtotal $1,292,610 $1,413,203 238 Water Capacity Building for Research, Education, and Training in Water Resources Management in Pakistan 1/1/2006 12/31/2009 $100,000 $500,000 125 Capacity Building and Quality Assurance/Quality Control Procedures for the Network of Water Quality Laboratories in Pakistan 2/1/2007 6/30/2008 $45,000 $30,000 29 Assessment and Development of Renewable Ground Water Resources in the Quetta Valley, Pakistan 2/1/2007 1/31/2011 $199,986 $254,590 27 Upper Indus River - Flow Reconstruction Using Tree Rings: Implications for Agriculture and Hydroelectricity 4/1/2008 3/31/2011 $100,000 $140,000 77 Bioremediation of Chromium and Arsenic from Industrial Wastewater 4/1/2008 9/30/2011 $237,460 $268,000 149 **Removal of Arsenic from Drinking Water Using Iron Ores as Low￾Cost Reactive Absorbent Media 11/15/2010 3/31/2014 $130,916 $267,152 178 **Small Scale Sewage Treatment and Wastewater Reuse System for Pakistan 11/15/2010 3/30/2014 $122,175 $202,986 24 Development of Ultrasensitive, Robust, and Affordable Nanoparticle-based Strips for Detecting Bacteria 10/1/2013 9/30/2016 $271,930 $362,748? 0 Subtotal $1,207,467 $2,025,476 609 Physics/Chemistry/New Materials **Carbazole and Fluorene-Based Polymeric and Molecular Materials for Optoelectronic Applications: Synthesis, Spectroscopic Studies, Device Fabrication, and Characterization 11/15/2010 11/14/2014 $123,030 $100,343 6 58 Subtotal $123,030 $100,343 6 Total $14,304,879 $15,957,482 8,967 Note: $1,645,416 of State/BEP funds were passed through to either HEC or MoST and included in the Pakistan side of funds. 59 APPENDIX 5: TYPICAL RESEARCH GRANT SELECTION CRITERIA Detailed information regarding selection criteria, instructions for the applicants, for the first four phases of the research grants and other information can be found on the S&T website: http://sites.nationalacademies.org/PGA/dsc/pakistan/index.htm#eligibility - Types of activities supported. Applicants are instructed to request support for collaborative research grants, which may be focused on any topic listed in the solicitations that are published with each phase of solicitation, with especially welcomes proposals corresponding to the special interests identified in each phase. To date there have been five phases since 2005 with the latest coming out in 2011. Some of the main focus areas have been agriculture and food science, health, disaster management, engineering, plants, water, environment and energy. Each set of partners may receive no more than one grant in a given phase of the program. - Eligibility. Proposals are to be developed by partnerships that should include at least one Pakistani partner and one U.S. partner. Pakistani partners must be based at a Pakistani university or research institute officially recognized by HEC. In developing proposals, proposing partners are urged to take advantage of the widest range of Pakistani talent capable of contributing to the achievement of objectives. In particular, partners should explain in their proposals how they will encourage the participation of women and ensure that they are provided equal opportunities for involvement in the proposed projects, whether as co-investigators, post￾doctoral researchers, students, trainees, or workshop participants. - Topical eligibility. Due to U.S. government regulations, the program cannot fund certain research activities. Specifically, “testing or breeding, feasibility studies, variety improvement or introduction, consultancy, publication, or training in connection with the growth or production [of a crop] in a foreign country for export if such export would compete in world markets with a similar commodity grown or produced in the United States." Before preparing their proposals, applicants intending to submit proposals involving research on citrus crops, sugar cane, or cotton are advised to e-mail a brief explanation of their proposed projects to pkcontract@nas.edu for an advance determination of eligibility by program staff in consultation with USAID and DOS. - Anticipated type of award. Requests for funding can vary in amount and length but should not exceed three years. Project funding for collaborative research grants is generally expected to be in the range of $100,000 to $300,000 per year (combined total for the Pakistani and U.S. partner institutions). Capital costs for the construction of new buildings are not covered. Salary costs for senior individual participants are expected to be covered from other sources, but salary support for junior team members and technical support staff may be included as appropriate. An annual progress report is required. Continuing-year funding of multi-year projects will be contingent upon a merit-based annual evaluation of progress. Proposals should include all activities and costs necessary to achieve the results stated in the application. Leveraging of resources is encouraged; therefore, the entire cost of the project should be included. Applicants are required to disclose any other funding received or applied for from other sources that would support the proposed project. - Selection criteria. Proposals are evaluated in separate peer review processes organized by the Pakistani and U.S. sponsors according to 1) relevance to the goals enumerated above under “Program Objectives,” 2) the scientific and technical merit of the proposal, 3) the cost-effectiveness of the project, 4) the capabilities of the participating institutions and individuals to successfully complete the project, and 5) the nature and quality of the collaboration, including the proposing partners’ commitment to ensuring the participation of women in the project. 60 ANNEX III: SAMPLING FRAME FOR INTERVIEWS WITH RESEARCHERS AND HEADS OF DEPARTMENTS OF PAKISTANI INSTITUTIONS Title of the Grant/Project Sector Phase Name of the Pakistani Institution Location No. of Interviews31 Gene Pyramiding through Genetic Engineering for Increased Salt Tolerance in Wheat Plants Phase 1 National Institute for Biotechnology and Genetic Engineering, Faisalabad Faisalabad 1 **Integrated Genetic/Physical Mapping of Gossypium Plants Phase 3 National Institute for Biotechnology and Genetic Engineering, Faisalabad Faisalabad 1 **Management of Greening by Producing Healthy Plants, Monitoring Vectors, and Identification of Tolerance Plants Phase 3 University of Agriculture, Faisalabad Faisalabad 1 **Solid Waste Management for Bioenergy Production Environment and Energy Phase 4 National Institute for Biotechnology and Genetic Engineering, Faisalabad Faisalabad 1 Improving the Lifestyle of Villagers in Remote Areas of Federally Administered Tribal Areas of Pakistan Using Renewable Energy Environment and Energy Phase 1 National University of Sciences and Technology, Islamabad Islamabad 1 Capacity Building and Quality Assurance/Quality Control Procedures for the Network of Water Quality Laboratories in Pakistan Water Phase 2 Pakistan Council of Research in Water Resources, Islamabad Islamabad 1 **Small Scale Sewage Treatment and Wastewater Reuse System for Pakistan Water Phase 4 Quaid-i-Azam University, Islamabad Islamabad 1 **Carbazole and Fluorene-Based Polymeric and Molecular Materials for Optoelectronic Applications: Synthesis, Spectroscopic Studies, Device Fabrication, and Characterization Physics/Chemistry/ New Materials Phase 4 Quaid-i-Azam University, Islamabad Islamabad 1 **A Hybrid Solar Water Heating System using CO2 as Working Fluid Environment and Energy Phase 4 COMSATS Institute of Information Technology, Islamabad 1 31 The evaluation team conducted interviews with 16 Pakistani researchers and nine HODs of Pakistani institutions from this oversampled list of 27 projects. 61 Title of the Grant/Project Sector Phase Name of the Pakistani Institution Location No. of Interviews31 Islamabad Nanomedicine for Cancer Research Health Phase 2 University of Karachi, Karachi Karachi 1 Building Pakistan's Capacity for Instruction, Research, and Practice in Earthquake Engineering and Retrofit Earthquake and Disaster Management Phase 2 NED University of Engineering and Technology, Karachi Karachi 1 Development of Biosecure, Sustainable, and Cost-Effective Culture Technologies for Edible Shrimp (Fenneropenaeus merguiensis) in Pakistan Agriculture and Food Science Phase 2 University of Karachi, Karachi Karachi 1 Upper Indus River - Flow Reconstruction Using Tree Rings: Implications for Agriculture and Hydroelectricity Water Phase 3 Federal Urdu University of Arts, Sciences, and Technology, Karachi Karachi 1 **Inventory of the Plants of Pakistan Plants Phase 4 University of Karachi, Karachi Karachi 1 Capacity Building for Research, Education, and Training in Water Resources Management in Pakistan Water Phase 1 University of Engineering and Technology, UET, Lahore Lahore 1 Intensification of Forensic Services and Research at Center for Applied Molecular Biology Health Phase 1 Centre for Applied Molecular Biology, Lahore Lahore 1 Determination of Heavy Metals and Polycyclic Aromatic Hydrocarbons in Airborne Particulates in Lahore, Pakistan, and Madison, Wisconsin, USA Environment and Energy Phase 1 University of Engineering and Technology, UET, Lahore Lahore 1 Discovery of Genetic Variation that Enhances Improvement of Dairy Production and Health in Cattle and Buffaloes Agriculture and Food Science Phase 3 University of Veterinary and Animal Sciences, Lahore Lahore 1 **Technology for the Poor: Low￾Cost Information and Computing Technology Engineering and Information Technology Phase 3 Lahore University of Management Sciences, Lahore Lahore 1 **Enhance Sugarcane Production in Pakistan by Modern Breeding Technology Plants Phase 3 University of the Punjab, Lahore Lahore 1 **Total Solution Based Organic￾Inorganic Solar Cells for Enhanced Efficiency and Stability Environment and Energy Phase 4 GC University, Lahore Lahore 1 **A Molecular Approach to Prevent Hereditary Blindness in Pakistan Health Phase 4 1) Centre of Excellence in Molecular Biology 2) Allama Iqbal Medical College, Lahore Lahore 1 62 Title of the Grant/Project Sector Phase Name of the Pakistani Institution Location No. of Interviews31 3) Layton Rehmatullah Benevolent Trust Hospital, Lahore Development, Optimization, and Application of a High-Performing Engineered Fertilizer Agriculture and Food Science Phase 2 Nuclear Institute of Food and Agriculture, Peshawar Peshawar 1 Development of a Framework for Probabilistic Seismic Hazard Maps for Pakistan Earthquake and Disaster Management Phase 2 NWFP University of Engineering and Technology, Peshawar Peshawar 1 Development of a Materials Connection Center Engineering and Information Technology Phase 3 University of Peshawar, Peshawar Peshawar 1 Response Modification Factors of Typical Pakistani Reinforced Concrete and Masonry Buildings for Pakistani Seismic Code Development Earthquake and Disaster Management Phase 3 NWFP University of Engineering and Technology, Peshawar Peshawar 1 **Integration of Geological, Geochemical, and Remote Sensing Data for Finding Source Rocks for Gold in the Northern Areas of Pakistan Engineering and Information Technology Phase 4 University of Peshawar, Peshawar Peshawar 1 Total 27 63 ANNEX IV: SAMPLING FRAME FOR INTERVIEWS WITH RESEARCHERS IN THE U.S. Title of the Grant/Project Sector Phase Name of the U.S. Institution Location No. of Interviews32 Gene Pyramiding through Genetic Engineering for Increased Salt Tolerance in Wheat Plants Phase 1 University of California, Davis Davis, CA 95616 1 **Integrated Genetic/Physical Mapping of Gossypium Plants Phase 3 University of Georgia, Athens Athens, GA 30602 1 **Management of Greening by Producing Healthy Plants, Monitoring Vectors, and Identification of Tolerance Plants Phase 3 University of California, Riverside Riverside, CA 92521 1 **Solid Waste Management for Bioenergy Production Environment and Energy Phase 4 U.S. Department of Agriculture, Agricultural Research Service Washington DC, 20250 1 Improving the Lifestyle of Villagers in Remote Areas of Federally Administered Tribal Areas of Pakistan Using Renewable Energy Environment and Energy Phase 1 Solar Energy International Carbondale, CO 81623 1 Capacity Building and Quality Assurance/Quality Control Procedures for the Network of Water Quality Laboratories in Pakistan Water Phase 2 U.S. Geological Survey Reston, VA 20192 1 **Small Scale Sewage Treatment and Wastewater Reuse System for Pakistan Water Phase 4 George Washington University NW, Washington, DC 20052 1 **Carbazole and Fluorine-Based Polymeric and Molecular Materials for Optoelectronic Applications: Synthesis, Spectroscopic Studies, Device Fabrication, and Characterization Physics/Chemistry/ New Materials Phase 4 University of Central Florida Orlando, FL 32816 1 **A Hybrid Solar Water Heating System using CO2 As Working Fluid Environment and Energy Phase 4 North Dakota State University Fargo, ND 58102 1 Nanomedicine for Cancer Research Health Phase 2 University of Illinois at Urbana￾Champaign Champaign, IL 1 32 The evaluation team conducted interviews with 18 U.S. researchers from this oversampled list of 27 projects. 64 Title of the Grant/Project Sector Phase Name of the U.S. Institution Location No. of Interviews32 Building Pakistan's Capacity for Instruction, Research, and Practice in Earthquake Engineering and Retrofit Earthquake and Disaster Management Phase 2 GeoHazards International, Palo Alto Menlo Park, California 94025 1 Upper Indus River - Flow Reconstruction Using Tree Rings: Implications for Agriculture and Hydroelectricity Water Phase 3 Columbia University, New York New York, NY 10027 1 **Inventory of the Plants of Pakistan Plants Phase 4 Missouri Botanical Garden St Louis, MO 63110 1 Capacity Building for Research, Education, and Training in Water Resources Management in Pakistan Water Phase 1 University of South Carolina Columbia, SC 29208 1 Intensification of Forensic Services and Research at Center for Applied Molecular Biology Health Phase 1 Strand Analytical Laboratories, LLC Indianapolis, IN 46241 1 Determination of Heavy Metals and Polycyclic Aromatic Hydrocarbons in Airborne Particulates in Lahore, Pakistan, and Madison, Wisconsin, USA Environment and Energy Phase 1 University of Wisconsin-Madison Madison, WI 53706 1 Discovery of Genetic Variation that Enhances Improvement of Dairy Production and Health in Cattle and Buffaloes Agriculture and Food Science Phase 3 USDA Agricultural Research Service, Beltsville, MD Beltsville MD 20705 1 **Technology for the Poor: Low-Cost Information and Computing Technology Engineering and Information Technology Phase 3 University of California, Berkeley Berkeley, CA 1 **Enhance Sugarcane Production in Pakistan by Modern Breeding Technology Plants Phase 3 American Sugar Cane League, Thibodaux, LA and USDA Agricultural Research Service, Houma, LA Thibodaux, LA 70301 1 **Total Solution Based Organic￾Inorganic Solar Cells for Enhanced Efficiency and Stability Environment and Energy Phase 4 University of Delaware Newark, DE 19716 1 **A Molecular Approach to Prevent Hereditary Blindness in Pakistan Health Phase 4 The Johns Hopkins University and National Eye Institute MD, 21218 Bethesda, MD 20892- 2510 1 Development, Optimization, and Application of a High-Preforming Engineered Fertilizer Agriculture and Food Science Phase 2 USDA Agricultural Research Service, Western Regional Research Center Albany, CA 94710 1 Development of Biosecure, Sustainable, and Cost-Effective Agriculture and Food Science Phase 2 Texas A&M University Texas 1 65 Title of the Grant/Project Sector Phase Name of the U.S. Institution Location No. of Interviews32 Culture Technologies for Edible Shrimp (Fenneropenaeus merguiensis) in Pakistan Development of a Framework for Probabilistic Seismic Hazard Maps for Pakistan Earthquake and Disaster Management Phase 2 University of Illinois at Urbana￾Champaign Champaign, IL 1 Development of a Materials Connection Center Engineering and Information Technology Phase 3 Boise State University Boise, ID 83725 1 Response Modification Factors of Typical Pakistani Reinforced Concrete and Masonry Buildings for Pakistani Seismic Code Development Earthquake and Disaster Management Phase 3 University of Illinois at Urbana￾Champaign Champaign, IL 1 **Integration of Geological, Geochemical, and Remote Sensing Data for Finding Source Rocks for Gold in the Northern Areas of Pakistan Engineering and Information Technology Phase 4 University of Houston Houston, TX 77004 1 Total 27 66 ANNEX V: DATA COLLECTION INSTRUMENTS Pakistan-United States Science and Technology Cooperation Program (S&T) Mid-term Performance Evaluation Instrument for Individual Interviews Introduction Thank you very much for meeting us today. My name is __________________ and I represent the USAID/ Pakistan Monitoring and Evaluation Program (MEP). This program is being implemented by Management Systems International, an international consulting firm. The program helps the USAID/ Pakistan Mission monitor and evaluate a range of its programs, including those in the areas of science, innovation and higher learning. USAID/ Pakistan has contracted MSI to conduct a mid-term evaluation of the Pakistan United States Science and Technology Cooperation Program (S&T) and identify possible improvements for the remainder of the project. We would like to ask you a few questions about the program and Mr./Ms. _____________________ will take notes. We will need to cover several aspects of the S&T program with your help. Confidentiality With your permission, we would like to record this discussion so that we accurately capture your feedback and do not miss any important points. Please be assured that your responses will be kept confidential. In case we use quotations from this interview in our evaluation report, you will not be identified by name, or official title, but in general terms as a researcher, official, or manager. Individual Interview Code: Name of Organization33 Position of Interviewee With S&T Since: Month: Year: Date of Interview Venue City Name and Position of Interviewer Name and Position of Note-taker Evaluation question 1: How well have USG investments in the S&T project resulted in sustainably strengthened scientific research capacity in Pakistan? Question 1 (Relates to evaluation question 1.a) For Pakistani PIs: Looking back from when you received your S&T grant to the time you completed the research, what changes did you observe in your skills and ability to meet international standards for conducting research? For USAID, NAS, HEC, MoST, US PIs and Pakistani HODs: Keeping in view your experience with the program, please tell us what changes you have observed as a result of the S&T program in the skills of Pakistani researchers in relation to international standards for research. [Prompts: research procedures/processes, use and maintenance of specialized equipment, and soft skills like critical thinking, creative problem solving, innovation and analytical reasoning] Answer Question 2 For Pakistani PIs: Which elements or experiences in the program were most 33 The highlighted information has been kept separate from the interview notes (in a document called “List of Sources”) to maintain confidentiality. 67 (Relates to evaluation question 1.b) beneficial in developing your skills, and which ones were relatively less beneficial? For USAID, NAS, HEC, MoST, US PIs and Pakistani HODs: Which elements or experiences in the program were most beneficial in developing researchers’ skills, and which ones were relatively less beneficial? [Prompts: partnering with US researchers, laboratory work in Pakistan and/or the US, field work, seminars, the publication process, exchange visits, provision of laboratory equipment] Answer Question 3 (Relates to evaluation question 1.a) For Pakistani PIs and HODs: What kind of changes, if any, have you noticed in research capacity at your institution? For USAID, NAS, HEC, MoST, and US PIs: What kind of changes, if any, have you noticed in research capacity at Pakistani institutions that have received S&T grants? [Prompts: changes in teaching, curricula, student dissertations and research procedures/processes, use and maintenance of specialized equipment, critical thinking, creative problem solving, innovation and analytical reasoning] Answer Question 4 (Relates to evaluation question 1.b) For Pakistani PIs and HODs: Which elements or experiences in the program were most beneficial in developing capacity in your institution, and which ones were relatively less beneficial? For USAID, NAS, HEC, MoST, and US PIs: Which elements or experiences in the program were most beneficial in developing capacity in Pakistani institutions, and which ones were relatively less beneficial? [Prompts: partnering with US researchers, curriculum development, dissertation supervision by US and/or Pakistani researchers, laboratory work in Pakistan and/or the US, field work, seminars, the publication process, exchange visits] Answer Question 5 (Relates to evaluation question 1.d) For Pakistani PIs and HODs: What kinds of measures, if any, have you and your institution taken to sustain improvements in the research capacity of researchers and their institutions? For USAID, NAS, HEC, MoST, and US PIs: What kinds of measures, if any, have Pakistani researchers and their institution taken to sustain improvements in the research capacity of researchers and their institutions? [Prompts: changes in policy, curriculum development, network building, new courses, revised curriculum, trained teachers, champions, fundraising, equipment maintenance investments] Answer Question 6 (Relates to evaluation question 1.d) What are the results of these measures in terms of sustaining researcher and institutional capacity for international research standards? Answer Question 7 (Relates to evaluation question 1.d) What kind of support from the United States Government may be required to support these measures? Answer 68 Question 8 (Relates to evaluation question 1.c) What measures have been taken to promote gender equity in the program? Answer Question 9 (Relates to evaluation question 1.c) What can the S&T program do in the future to better promote gender equity in the science and technology fields? Answer Evaluation question 2: To what extent have the research projects funded through the S&T program yielded benefits to industry, government or quality of life for the Pakistani people? Question 10 (Relates to evaluation question 2.a) For Pakistani PIs: Please describe the kinds of research products that you have developed with the support of the S&T grant. For USAID, NAS, HEC, MoST, HODs and US PIs: Please describe the kinds of research products that Pakistani researchers have developed with the support of S&T grants. Answer Question 11 (Relates to evaluation question 2.b) What benefits have the products developed through S&T research grants yielded to Pakistani businesses, government or the public? Please also identify the beneficiaries. [Prompts: products patented or put to commercial use or use outside the lab, research leading to new technologies or practices adopted by specific population groups (community, city and country level)] Answer Question 12 (Relates to evaluation question 2.c) For Pakistani and US PIs: Please tell us about the academic publications and conference presentation resulting from your research grant. For USAID, NAS, HEC, MoST, and HODs: Please share any information you have on academic publications and conference presentations resulting from the research grants. Answer Question 13 (Relates to evaluation question 2.d) Please tell us about any additional benefits resulting from the research grants that you may not have mentioned in response to the earlier questions. [Prompt; introducing community based solutions through existing technologies/products with the help of S&T grant(s)]. Answer Question 14 (Relates to evaluation question 2.e) Please tell us about any steps that have been taken to generate or expand the impact of S&T-funded research. Answer Question 15 (Relates to evaluation question 2.f) What are the principal barriers or challenges to developing the capacity of Pakistani researchers to produce research that yields benefits to industry, government or quality of life for the Pakistani people? Answer 69 Evaluation question 3: How successfully have research collaborations through the S&T project fostered mutual understanding and goodwill between institutions and individuals in the United States and Pakistan? Question 16 (Relates to evaluation question 3.a) How and to what extent did research collaboration through the S&T project foster mutual understanding and goodwill between institutions and individuals in the United States and Pakistan? Answer Question 17 (Relates to evaluation question 3.b) What kind of experiences have contributed toward changes in attitudes? [Prompts: exchange visits, training, working with US/Pakistani researchers/institutions] Answer Question 18 (Relates to evaluation question 3.c) To what extent have US and Pakistani individuals and institutions established partnerships and other forms of formal and informal cooperation as a result of the S&T project? Answer Evaluation question 4: What factors affected timely project performance against its MSF indicators? Question 1934 (Relates to evaluation question 4.a) What factors supported timely project performance? Answer Question 20 (Relates to evaluation question 4.b) What factors hindered timely project performance? Answer Question 21 (Relates to evaluation question 4.b) How can project performance be improved in terms of timeliness? Answer 34 Questions 19-21 will not be addressed to the Heads of Department. 70 ANNEX VI: DISCLOSURE OF CONFLICT OF INTEREST Name Tariq Husain Title Senior Evaluation Consultant Organization Management Systems International (MSI) Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Pakistan-United States Science and Technology (S&T) Cooperation Program National Academy of Sciences (NAS), Washington I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date July 14, 2014   71 Name Zameer Haider Title Evaluation Specialist Organization Management Systems International (MSI) Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Pakistan-United States Science and Technology (S&T) Cooperation Program National Academy of Sciences (NAS), Washington I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 7. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 8. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 9. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 10. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 11. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 12. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date July 14, 2014   72 Name Babar Mufti Title Monitoring Coordinator Organization Management Systems International (MSI) Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Pakistan-United States Science and Technology (S&T) Cooperation Program National Academy of Sciences (NAS), Washington I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 13. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 14. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 15. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 16. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 17. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 18. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date July 14, 2014 X X   73 Name Mohammed Aftab Ismail Khan Title Evaluation Specialist - STTA Organization Management Systems International (MSI) Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Pakistan-United States Science and Technology (S&T) Cooperation Program National Academy of Sciences (NAS), Washington I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 19. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 20. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 21. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 22. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 23. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 24. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date July 14, 2014   74 Name Nikhat Ambreen Title Researcher - STTA Organization Management Systems International (MSI) Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Pakistan-United States Science and Technology (S&T) Cooperation Program National Academy of Sciences (NAS), Washington I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 25. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 26. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 27. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 28. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 29. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 30. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date July 14, 2014   75 Name Lisa Slifer-Mbacke Title Technical Director Organization Management Systems International (MSI) Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Pakistan-United States Science and Technology (S&T) Cooperation Program National Academy of Sciences (NAS), Washington I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 31. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 32. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 33. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 34. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 35. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 36. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date July 14, 2014   76 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523