1 MID-TERM EVALUATION OF THE PROGRAM TO EXTEND SCHOLARSHIPS AND TRAINING TO ACHIEVE SUSTAINABLE IMPACTS – PHASE II (PRESTASI-II) May 2015 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Colin Davies, Dwiagus Stepantoro, Sherry L. Mueller, Ph.D., Erlinda Panisales, and Dian Rachmawati of International Business & Technical Consultants, Inc (IBTCI). 2 MID-TERM EVALUATION OF THE PROGRAM TO EXTEND SCHOLARSHIPS AND TRAINING TO ACHIEVE SUSTAINABLE IMPACTS PHASE II (PRESTASI-II) May 18, 2015 Contract No. GS-10F-0309P Order No. AID-497-M-15-00003 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 3 ACKNOWLEDGEMENTS The International Business and Technical Consultants, Inc. (IBTCI) team conducting the Mid-Term Evaluation of the PRESTASI-II Program thanks the following groups of respondents and officials for being so generous with their time, expertise, reactions, and suggestions: USAID Indonesia program and technical staff, particularly in the Education Office, who were always ready to respond to the team’s many questions and requests for documents. Current PRESTASI scholars and alumni including representatives of ALPHA-I. Staff of the implementing partners – the Indonesia International Education Foundation and the Institute of International Education. Officials of the Government of Indonesia including: Coordinating Ministry of Human Development and Culture Lembaga Pangelola Dana Pendidican/Institute of Management Education Fund (LPDP) Ministry of Health Ministry of Marine Affairs and Fisheries Ministry of National Development Planning/National Development Planning Agency (BAPPENAS) Indonesian educational institutions that graciously received the visiting interview teams, including: Airlangga University, Surabaya Ar-Raniry Islamic State University, Banda Aceh Bandung Institute of Technology, Bandung Gadjah Mada University, Yogyakarta Sanata Dharma University, Yogyakarta Sunan Kalijaga Islamic State University, Yogyakarta University of Indonesia, Jakarta University of Syiah Kuala, Banda Aceh US educational institutions that hosted PRESTASI participants and reflected on their experiences to the evaluation team, including: California State University-Los Angeles Eastern Mennonite University Michigan State University Ohio State University Oregon State University SUNY Buffalo University of Georgia University of South Florida Western Michigan University Representatives of other agencies sponsoring scholarship programs. 4 TABLE OF CONTENTS ACKNOWLEDGEMENTS TABLE OF CONTENTS ACRONYMS EXECUTIVE SUMMARY 1. EVALUATION PURPOSE & EVALUATION QUESTIONS 1.1. EVALUATION PURPOSE 1.2. EVALUATION QUESTIONS 1.3. PROJECT BACKGROUND 2. EVALUATION METHODS & LIMITATIONS 2.1. EVALUATION METHODS 2.2. EVALUATION LIMITATIONS 3. FINDINGS, CONCLUSIONS & RECOMMENDATIONS 3.1. GENERAL RECOMMENDATIONS 3.2. COMPONENT-SPECIFIC RECOMMENDATIONS 3.2.1. OUTREACH AND RECRUITMENT 3.2.2. SELECTION PROCESS 3.2.3. SCHOLAR PREPARATION 3.2.4. SCHOLARS EXPERIENCE IN THE UNITED STATES 3.2.5. INDIVIDUAL AND INSTITUTIONAL BENEFITS 3.2.6 RE-ENTRY AND ALUMNI ACTIVITIES 3.3. PROGRAM MANAGEMENT 3.3.1. IIEF’S MANAGEMENT OF THE PROGRAM 3.3.2. PARTICIPANT INTERACTIONS WITH IIEF 3.3.3. IIE's MANAGEMENT OF THE PROJECT 3.3.4. PARTICIPANT INTERACTION WITH IIE 3.3.5. COST-SAVINGS 3.4. EXTERNAL FACTORS ANNEXES ANNEX 1: ADDITIONAL DETAILS ON EVALUATION METHODS ANNEX II: DATA COLLECTION INSTRUMENTS ANNEX III: SOURCES OF INFORMATION ANNEX IV: STATISTICS – INDONESIAN STUDENTS STUDYING IN THE UNITED STATES ANNEX V: STAFFING 5 ACRONYMS ADS The Automated Directives System, USAID regulations and procedures AED Academy for Educational Development AETR Academic and Enrollment Term Reports ALPHA-I Asosiasi Alumni PRESTASI HICD Amerika Indonesia (PRESTASI Alumni Association) AMCHAM American Chamber of Commerce AMINEF American Indonesian Exchange Foundation BAPPENAS Badan Perencanaan Pembangunan Nasional (Agency/Ministry of National Development Planning) COR Contracting Officers Representative CSO Civil Society Organization DIKTI Direktorat Jendral Pendidikan Tinggi (Directorate General of Higher Education) DOT Development Objective Teams EAP English for Academic Purposes ELT English Language Training GBV Gender Based Violence GOI Government of Indonesia GPA Grade Point Average GRE Graduate Record Examination HAC Health and Accident Coverage Insurance HDI Human Development Index HEI Higher Education Institution HICD Human and Institutional Capacity Development IALF Indonesia Australia Language Foundation IBTCI International Business and Technical Consultants, Inc. ICT In-country Training IIE Institute of International Education IIEF Indonesian International Education Foundation KII Key Informant Interview KSA Knowledge, Skills and Attitudes LPDP Lembaga Pangelola Dana Pendidican (Institute of Management Education Fund) LTT Long-Term Training 6 MMA Monthly Maintenance Allowance M&E Monitoring and Evaluation NESO Netherlands Education Support Office OECD Organization for Economic Cooperation and Development PAT Pre-Academic Training PDO Pre-Departure Orientation PLC Professional Learning Communities PNS Pegawai Negeri Sipil (Permanent Civil Servant) PPR Periodic Progress Report PRESTASI-II Program to Extend Scholarship and Training to Achieve Sustainable Impact Phase II PT Participant Training RFP Request for Proposals SOP Standard Operating Procedure STT Short-Term Training TOEFL Test of English as a Foreign Language USAID United States Agency for International Development USG United States Government UST United States Training 7 EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS The purpose of this activity is to conduct a mid-term evaluation of the United States Agency for International Development (USAID) Program to Extend Scholarships and Training to Achieve Sustainable Impacts Phase II (PRESTASI-II), assessing its performance from August 2012 to date, identifying options for improvement, and providing recommendations for adjustment. This mid-term evaluation reviewed the effectiveness of – and the scholar and alumni satisfaction with - all aspects of the program, including recruitment and selection (in particular, outreach to underserved populations), pre￾academic training, pre-departure orientation, US-based orientation, in country/US monitoring, the overall educational experience and the acquisition of academic knowledge and leadership skills, re-entry and alumni support, and the overall management of the program. PROJECT BACKGROUND PRESTASI is USAID/Indonesia’s flagship participant training program. The goal of PRESTASI Phase II is to improve the performance and leadership skills of Indonesian professionals, which, in turn, will help to promote development in Indonesia and the achievement of the Mission’s Development Objectives under the 2009-2014 Mission Strategic Plan. PRESTASI Phase II provides training and technical services required to strengthen and expand the base of skilled, high-performing professionals and institutions in Indonesia’s public and private sectors. Currently there are 59 participants in training in the United States and Indonesia. Participants in the United States are studying throughout the country, from California to New York. The states of Michigan and New York have the highest number of participants with eleven and seven scholars, respectively. Ten participants are studying in Indonesia, at the University of Indonesia, Gadjah Mada University, University of Airlangga and Brawijaya University. Sixty-eight PRESTASI alumni are residing throughout Indonesia, with the majority on the island of Java. EVALUATION QUESTIONS, DESIGN, METHODS AND LIMITATIONS The IBTCI evaluation team conducted a thorough analysis of all available project documents and reports, followed by a detailed review of project-relevant statistics, articles, and external reports that provided contextual understanding. This was followed by key informant interviews (KII) and group discussions with relevant stakeholders: USAID officials, implementing partners, Asosiasi Alumni PRESTASI HICD Amerika Indonesia (PRESTASI Alumni Association or, ALPHA-I), scholars and alumni, Government of Indonesia (GOI) officials (Coordinating Ministry, Badan Perencanaan Pembangunan Nasional (Agency/Ministry of National Development Planning – BAPPENAS), Lembaga Pengelola Dana Pendidikan (Institute of Management Education Fund, or LPDP), employers, and other providers of scholarship programs (Indonesian, Australian, American and other sponsors). The team conducted a total of 120 interviews (61 female and 59 male). Concurrently, the team distributed an online questionnaire using the SurveyMonkey online platform, sending email invitations to 127 current scholars and alumni. This online questionnaire, translated into Bahasa Indonesia, queried respondents about their PRESTASI experiences, degree of satisfaction, knowledge and skills gained, and suggestions for program improvement. In addition, alumni were asked about their application of knowledge and skills, job performance, and career benefits. 8 The survey response rate was 64% (81 out of 127). The team entered the survey responses into an Excel spreadsheet for further data processing and analysis including, but not limited to, tabulation and cross-tabulation. In the analysis, data that was gathered were triangulated across different sources and methods. The primary limitation was the fact that, given time and financial constraints, the evaluation team could only cover the scholars/alumni from the western part of Indonesia (Aceh, Yogyakarta, Surabaya, and Jakarta with its surrounding cities). This limits the generalizations that can be made about the findings, as alumni and employers from the eastern part of Indonesia might have had different perceptions that would have been instructive to add to the mix. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS The evaluation team reached and/or developed a total of 46 findings, 42 conclusions, and 29 recommendations. The key findings, conclusions and recommendations are included below. Key Finding: A large majority (92%) of the current and alumni survey respondents rated their overall experience as a PRESTASI-II scholar as “Excellent” (33%) or “Very Good” (59%). Of these, 100% said they would fully recommend this program to their colleagues. All survey respondents reported that their educational experience had either a Very Positive (50%) or Positive (50%) impact on their careers. Conclusion: PRESTASI-II is a well-designed and remarkably well-integrated scholarship program that is achieving its primary objectives. Recommendation: Increase the size of the PRESTASI-II program. PRESTASI-II fills an important gap in the scholarship availability to students who represent underserved geographic areas and populations in Indonesia. Key Finding: A frequent question from scholars during interviews was whether there were any scholarship opportunities to complete a Ph.D. According to LPDP, the current ratio of Ph.D.s to the general population in Indonesia is 143 Ph.D.s per one million Indonesians. In the United States, this ratio is 10,000 Ph.D.s per one million Americans. Conclusion: There is a demand for educational opportunities beyond the Master’s degree level and a need for more scholars. Recommendation: For similar future training programs, it is recommended that a limited number of Ph.D. scholars in disciplines key to Indonesia’s growth be funded, and the possibility of transferring some scholars from one contract to the next be investigated. For the current PRESTASI program, a one-week course, such as “An Introduction to Studying for a Ph.D.”, could be offered. Key Finding: The program has conducted successful outreach toward women, but has had less success in its outreach within underserved areas and in reaching people with disabilities. Conclusion: USAID’s implementer, the Institute of International Education, or IIEF, can expand its outreach and take greater advantage of existing networks of appropriate non-governmental organizations in each region to reach underserved areas and the disabled. Recommendation: Continue to increase efforts to reach people from underserved areas and people with disabilities. The possibility of conducting a joint outreach program should be discussed with the relevant Fulbright association (American Indonesian Exchange Foundation or AMINEF) and with LPDP, both of which expressed interest in this type of collaboration. Key Finding: Sixty-five percent of scholars said that their limited English language skills had been a barrier to getting the most out of their program. PRESTASI scholars who receive a score of 450 on the Test of English as a Foreign Language (TOEFL) currently study for six months. By contrast, scholars with the same score who are selected for other scholarships -- by Australia Awards, Direktorat Jendral 9 Pendidikan Tinggi (the Directorate General of Higher Education – referenced later as DIKTI), the Badan Perencanaan Pembangunan Nasional (Agency/Ministry of National Development Planning, or BAPPENAS) and LPDP -- study for at least nine months, which is often followed by further study on arrival at their host campus. Conclusion: Some scholars are constantly playing catch-up with their course-work because they are spending their credit hours taking courses to improve their English language and study skills during their first and second semesters of graduate work. Recommendation: The length of in-country English language study for those scholars who score between 450 and 500 on the TOEFL should be increased, followed by a period of study on their US campus. Key Finding: The team received positive feedback from the participants who had attended a short￾term training (STT) program. Several employers requested that their staff be sent to STTs in the US to focus their study on topics such as strategic management. The Institute of International Education (IIE) has experience in managing STTs. Conclusion: There is a demand for focused STTs. Recommendation: PRESTASI-II should consider increasing the number of STTs, building on IIE’s expertise in this area. Key Finding: ALPHA-I is a relatively new organization with great potential to service PRESTASI alumni and the broader community. Alumni are seeking ways to share their newly acquired expertise. Conclusion: Given the nature of the community building activities already hosted by ALPHA-I, working with Prestasi Junior – part of the global Junior Achievement network – would provide alumni relevant volunteer opportunities to share their areas of expertise. Recommendation: Encourage ALPHA-I to develop a partnership with Prestasi Junior. In addition, an ALPHA-I mentoring program should be implemented that pairs an alumnus with a newly selected scholar. Key Finding: Data gathered through meetings at GOI ministries indicate that there has been little communication between PRESTASI and the Coordinating Ministry for Human Development and Culture or with BAPPENAS. A representative from the Ministry for Human Development and Culture stated that the most recent communication from PRESTASI was two years ago. Conclusion: There is a lack of communication, consultation and coordination between GOI ministries and the PRESTASI program. Recommendation: IIEF and a USAID representative should pay a courtesy visit to BAPPENAS, LPDP, and particularly to the Coordinating Ministry at least once a year. ADDITIONAL RECOMMENDATIONS ● IIEF should develop a standard template for reporting. A consultant, perhaps an IIE information technology specialist, should be brought in to set up a database. Current staff should be trained in data input and generating reports. Until the database is established and tested, the requirement to submit Semi-Annual Reports should be removed. ● There is a perceived need for an informal periodic meeting among all those organizations managing scholarship programs for an exchange of best practices. The PRESTASI program should lead this effort and coordinate the first meeting. 10 ● The program should consider developing a pilot “Meet America” program to take groups of scholars, including several American students, during a Holiday break to have an enrichment experience similar to the Mid-Winter Seminars of the 1980s and 1990s. Participants would be placed in homes with US families, and take short workshops that would provide them with management training supplemental to their academic programs. ● More time should be given to explaining the procedure for medical examinations and how to access health insurance coverage once the scholars arrive in the United States. ● In addition to the Pre-Departure Orientation (PDO) hosted by the Indonesian International Education Foundation (IIEF), USAID should give serious consideration to resuming the practice of having IIE host a 3-day arrival orientation in the United States. If this is not possible given budget constraints, two IIE Program Officers, with experience at a wide range of US universities and/or colleges should be invited to take part in the PDO in Indonesia. ● IIEF should develop a more structured leadership program with several components linked to the re-entry activities for when the scholars return to Indonesia. ● A series of webinars (with paper copies for those without reliable internet access) for scholars should be developed and offered during the months prior to their departure from Indonesia to the United States. This would help to sustain the scholars’ post-selection enthusiasm. ● The PRESTASI program should encourage more ownership by the scholar’s employer. The role and responsibilities of the employer should be more clearly defined in the Stakeholder Compact. Additionally, each scholar should be required to prepare a preliminary action plan in consultation with his/her supervisor outlining how both parties anticipate the scholar will use the knowledge and skills gained after completing their graduate work. ● All alumni should automatically be enrolled in ALPHA-I. The program should develop a more structured re-entry program that occurs three months after scholars return and incorporate the recommendation above regarding the leadership program development. 11 1. EVALUATION PURPOSE & EVALUATION QUESTIONS 1.1 EVALUATION PURPOSE The purpose of this activity is a mid-term evaluation of the Program to Extend Scholarships and Training to Achieve Sustainable Impacts Phase II (PRESTASI-II) program, assessing the performance of the project from August 2012 to date, identifying areas for improvement, and providing recommendations for adjustment. This mid-term evaluation assesses the PRESTASI-II program in general as well as in the following specific areas: outreach, selection and placement; monitoring of scholars; program administration; and post-training support and follow-up; inclusive of gender across these areas. 1.2 EVALUATION QUESTIONS This mid-term evaluation focused on answering the six evaluation questions posed in the amended Request for Proposals (RFP), in addition to a set of four evaluation questions posed by IBTCI. These questions are: 1. What aspects of PRESTASI are proving to be most and least effective in the implementation of PRESTASI relating to recruitment, selection, management, and support of students? (RFP Q.2). 2. To what extent has the project been able to reach underserved populations? What can be done to increase this effort? (RFP Q.6) 3. To what extent are participants and alumni satisfied with project service delivery, including recruitment and selection, pre-academic instruction, pre-departure orientation, US-based orientation, in country/US monitoring, educational experience, re-entry and alumni support. (IBTCI Q.1) 4. To what extent are the participants and alumni satisfied with the long-term and short-term instruction received? (IBTCI Q.2) 5. To what extent has US and Indonesian long-term and short-term training directly contributed to the knowledge and skills of Indonesian professionals? (IBTCI Q.3) 6. To what extent has US and Indonesian long-term and short-term training directly contributed to the performance and leadership skills of Indonesian professionals. (RFP Q.1) 7. To what extent have PRESTASI stakeholders benefited from the project’s activities and what specific value has been added? (RFP Q.3) 8. To what extent can improvements be made that will facilitate the attainment of the planned results? (RFP Q.5) 9. To what extent are the PRESTASI project’s resources being implemented and managed efficiently and cost-effectively? (RFP Q.4) 10. What external factors have affected project implementation and results and how did they affect them? (IBTCI Q.4) 1.3 PROJECT BACKGROUND Despite rapid social and economic progress, Indonesia still faces a number of development problems. Among others, these include: a high maternal mortality rate, a weak education system, high unemployment and environmental management challenges. For example, the United Nations Development Program (UNDP) 2010 Human Development Index report ranked Indonesia as number 12 111 in the world and categorized it among the 10 lowest in East Asia and the Pacific. The report measures life expectancy, literacy, education, and standard of living. Based on this report, many sectors need improvement in order to not only move Indonesia to a higher HDI rating, but also to foster Indonesian development. In response to these challenges, USAID/Indonesia, through the PRESTASI program, provides opportunities for Indonesians to study for an advanced degree in the United States or Indonesia and to attend short-term technical training programs. PRESTASI is designed to address the needs of Indonesians who meet the technical qualifications for higher level training and have demonstrated the proven leadership capacity to apply the knowledge, skills and experience gained toward Indonesia’s development. PRESTASI is USAID/Indonesia’s flagship participant training (PT) program. PRESTASI-II has built upon earlier PT and PRESTASI Phase I programs. The goal of PRESTASI-II is to improve the performance and leadership skills of Indonesian professionals, which, in turn, will help to promote development in Indonesia and the achievement of the Mission’s Development Objectives under USAID’s 2009-2014 Mission Strategic Plan. Hence, PRESTASI-II provides training and technical services required to strengthen and expand the base of skilled, high-performing professionals and institutions in Indonesia’s public and private sectors. Training includes long-term education (degree and non-degree) and short￾term opportunities (conferences, visits, seminars, and programs tailored to the needs of particular groups) primarily in the United States. Selection for participation in Indonesia and other countries this is determined on a case-by-case basis. The overarching goal of the PRESTASI-II program is to develop individuals and entities that are better equipped to provide leadership in the public and private sectors. The objectives of the program are: 1. Implement and support policies important to Indonesia’s development; 2. Exercise equity, accountability and transparency in managing public and private sector resources; 3. Provide better delivery of public services; 4. Participate more effectively in and contribute more broadly to the country’s economic and social development; and 5. Support the achievement of key objectives in several of USAID’s priority sectors. The following major outcomes are expected from the implementation of PRESTASI-II: 1. Creation of a cadre of skilled leaders and managers who can share new knowledge and skills within priority sectors and therefore advance key development objectives; 2. Development of new advanced management skills, technical capacity, and knowledge within targeted public/private institutions and Government of Indonesia (GOI) ministries to enable more efficient, transparent, and accountable practices across sectors, and greater support for economic growth; 3. Creation of a group of returned participants committed to: (1) applying newly acquired skills at their places of employment; and (2) sharing new skills, knowledge and best practices with colleagues, thereby maximizing the impact of training and creating a climate of growth, wider access to information and an on-going learning environment in their institutions, as well as the broader community; 4. Increased coordination of efforts among GOI, local institutions, United States Government (USG) agencies and other donor organizations through the establishment of participant networking teams to mutually enhance capabilities, participate in community service activities, 13 reinforce leadership potential and promote positive trends throughout the country; 5. More efficient, transparent and accountable practices in managing resources across sectors leading to better service delivery; 6. Increased goodwill and cultural understanding engendered through academic study in the United States; 7. Increased number of Indonesian future leaders holding advanced degrees (Master’s) from US and Indonesian higher education institutions (HEI); and 8. Increased number of women and individuals from disadvantaged and/or under represented geographic areas participating in the program. PRESTASI-II participants are Indonesian professionals recruited from all the major regions in Indonesia. The program puts special emphasis on reaching both women and promising Indonesians from undeserved and disadvantaged areas with limited resources. Training subject areas include: Basic and Higher Education, HIV/AIDS, Infectious Diseases, Maternal and Child Health, Sustainable Management of Forests and Marine Ecosystems, Clean Energy, Climate Change Adaptation, Science and Technology, Economics, Entrepreneurship, and Democracy and Governance. English language training with a writing component is offered in order to provide the proper foundation skills for participants planning to study in the United States. The distribution of training activities is based on the technical priorities of the USAID Mission in addition to available funding streams (e.g. Basic and Higher Education, HIV/AIDS, Infectious Disease, Forestry Management, Rule of Law, Economics, etc.). Currently $9,676,929 has been obligated to the contract. The source of funding, by sector, is as follows: Democracy and Governance ($1,716,344) supports 28 participants for Master’s and short-term training programs; Education ($2,293,298; basic education $1,510,164; and higher education $783,134) supports 34 participants for Master’s degrees; Economic Growth ($701,042) supports nine Master’s degree participants and short-term training; Environment ($2,033,245) supports 23 participants for Master’s degree and short-term training program; and Health ($2,933,000) supports 56 participants. The higher number of health participants in relation to funds expended reflects the fact that many of the health participants study in Indonesia. To date, PRESTASI-II has successfully managed scholarship services for 150 scholars studying in the United States and Indonesia for Master’s and Doctoral degrees. In addition to long-term degree scholarships, PRESTASI-II manages short-term training and supports technical offices throughout the Mission. Four short-term training programs have been implemented: one in Democracy and Governance for 12 participants, one in Economic Growth for two participants and two short-term training (STT) programs in Environment for two participants. PRESTASI has also successfully increased female participation by encouraging women to apply, via the use of various outreach materials including targeted site presentations. It has been instrumental in the development of the alumni association (Asosiasi Alumni PRESTASI HICD Amerika Indonesia (PRESTASI Alumni Association – ALPHA-I). Currently, there are 59 participants in training in the United States and Indonesia. Participants in the United States are studying throughout the country, from California to New York. Michigan and New York have the highest number with 11 and seven scholars respectively. Ten participants are studying in Indonesia at University of Indonesia, Gadjah Mada University, University of Airlangga and Brawijaya University. Sixty-eight PRESTASI alumni are residing throughout Indonesia with the majority on the island of Java. 14 2. EVALUATION METHODS & LIMITATIONS 2.1 EVALUATION METHODS Based on the Statement of Work, IBTCI developed a work plan for the mid-term evaluation. The work plan was submitted to USAID for approval and approved on February 19, 2015. The evaluation activities were conducted from mid-January through April 2015, starting with consultations with USAID, document review, interviews with scholars currently studying in United States followed by interviews with alumni, implementing partners, and other stakeholders in Indonesia as outlined below. A detailed description of evaluation activities appears below. Table 1: Evaluation Timeline Dates Activity Personnel Objective/ Notes Relevant Evaluation Questions Phase One Information Gathering and Planning 1/15/15 – 2/18/15 Prepare, Submit, Revise, and Re-submit Work Plan C. Davies with review by T. (Tyers) Dixon 1/15/15 – 1/30/15 Compile test questionnaires and interview/focus group questions for: current students, alumni, international student offices, faculty advisors, USAID officials, Indonesian university staff, Indonesian employers, GOI Ministry officials C. Davies with review by T. (Tyers) Dixon To prepare for surveys, interviews and focus groups. 1, 2, 3, 4, 5, 6, 7, 8, 9,10 2/2/15 – 2/9/15 Review reports and other documents C. Davies To gather data for evaluation 1, 2, 5, 6, 7, 9, 10 Phase Two Data Collection and Field Visits 2/18/15 – 2/20/15 Visit IIE/Washington and IIE/New York C. Davies, S. Mueller To review procedures and practices 1, 2, 8, 9, 10 2/23/15 – 2/27/15 Visit Lansing, MI and Kalamazoo, MI. Meet with Michigan State University and Western University students. Meet with international student advisors and appropriate faculty. Conduct interviews with students and faculty on 6 other US campuses C. Davies, S. Mueller Conduct interviews and focus groups. Gather data. 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 15 3/1/15 Fly to Jakarta C. Davies and S. Mueller 3/6/15 – 3/11/15 Meet with USAID and IIEF C. Davies, S. Mueller, D. Stepantoro, E. Panisales Conduct focus groups and interviews. Gather data 1, 2, 3, 4, 5, 6, 7, 8, 9,10 3/6/15 - 3/30/15 Meet with Jakarta-based alumni, in￾country scholars, scholars’ supervisors, GOI Ministry officials, training providers, representatives of other international scholarship programs, ALPHA-I Evaluation Team Conduct focus groups and interviews. Gather data 1, 2, 3, 4, 5, 6, 7, 8, 9,10 3/6/15- 3/21/15 Finalize, test, and translate Survey Monkey. Distribute Survey to current scholars in Indonesia and the United States, and to alumni Evaluation Team To gather Survey quantitative and qualitative data 1, 2, 3, 4, 5, 6, 7, 8, 9,10 Phase Three Data Tabulation and Analysis 3/23/15 – 3/30/15 In Jakarta: data review Evaluation Team Review and analyze data from the survey and from the interviews and focus groups. Conduct data runs, analyses of variance tests as appropriate 1, 2, 3, 4, 5, 6, 7, 8, 9,10 3/23/15 – 4/4/15 Prepare draft report and MS PowerPoint “No Surprises” presentation. Translate report Evaluation Team, in coordination with T. (Tyers) Dixon 1, 2, 3, 4, 5, 6, 7, 8, 9,10 Phase Four Presentation of Findings and Recommendations 4/8/15 Deliver Evaluation Debriefing “No Surprises” oral presentation C. Davies, S. Mueller, D. Stepantoro, E. Panisales, Receive comments for inclusion in Draft Report and Lessons Learned Report 1, 2, 3, 4, 5, 6, 7, 8, 9,10 4/6/15 – 4/10/15 Prepare Draft Main Report and Lessons Learned Report The Team with approval by T. (Tyers) Dixon 1, 2, 3, 4, 5, 6, 7, 8, 9,10 Phase Five Completion of Final Report and Delivery 4/17/15 USAID Provides IBTCI Comments about the Draft Main Evaluation Report and Lessons Learned Report USAID 4/17/15 – 4/24/15 IBTCI Team Prepares Final Report and Lessons Learned T. (Tyers) Dixon with input from the All comments received from COR and other 1, 2, 3, 4, 5, 6, 7, 8, 9,10 16 Team stakeholders will be incorporated into the Final Report 4/24/15 Submit Final Evaluation Report and Lessons Learned Report T. (Tyers) Dixon 1, 2, 3, 4, 5, 6, 7, 8, 9,10 The evaluation employed a mixed-methods approach consisting of a combination of quantitative and qualitative data collection methods. Taken together, these methods provide an accurate understanding of project performance and enabled the team to generate reliable and valid answers for each of the evaluation questions. Document Review During the document review period, the evaluation team read and analyzed reports, such as quarterly and semi-annual reports, quarterly performance monitoring reports, the performance management and evaluation plan, annual work plans, any external assessment reports, participant academic and enrollment team reports (AETRs), procurement plan, comprehensive alumni plan, training reports, pipeline analysis reports, quarterly financial reports and status reports, and stakeholder agreements. The document review process supplied to the team key quantitative data and qualitative data as well as contextual information. The team also reviewed related statistics, articles, and external reports that related to the project and contextual understanding. Annex III contains a list of documents and publications reviewed as part of the team’s research and due diligence efforts. Semi-structured Questionnaire The evaluation team distributed an online semi-structured questionnaire, using the SurveyMonkey web￾based survey software, to 127 respondents including 59 current scholars and 68 PRESTASI-II alumni. The 68 alumni were divided into groups A, B, and C. Group A consisted of 27 alumni recruited and placed by the Academy for Educational Development (AED) in Fall 2011. This group was monitored by the Institute of International Education (IIE). Most students graduated in 2013. Group B, a group of 33, was recruited by AED, placed, and monitored by the Institute of International Education (IIE) and b the Indonesian International Education Foundation (IIEF) in the Fall 2012. Most Group B students graduated in 2014. Group C was comprised of eight students recruited by IIEF, and then placed and monitored by IIE and IIEF. Most are scheduled to graduate during the summer of 2015. There are eight students who have graduated and returned. This online survey provided both qualitative and quantitative data. The questionnaire can be found in Annex II. It was translated into Bahasa Indonesia and it queried respondents about their experience, degree of satisfaction, knowledge and skills gained, suggestions for improvement, and it asked alumni about application of knowledge and skills, job performance, and career benefits. All survey responses were entered into an Excel spreadsheet for further data processing and analysis (tabulation, cross￾tabulation). Data gathered were triangulated across different sources and methods. (Triangulating involves comparing and cross checking the results of data collected from different sources.) The online survey received a 64% (81 out of 127) response rate. Among those who responded, 47 were female and 34 male. See Figure 1 for a breakdown of respondents. 17 Figure 1: Number of Respondents to the Online Survey by Group and Gender Source: PRESTASI-II Online Survey Key Informant Interviews (KIIs) and Group Discussions The team conducted interviews and small group discussions with stakeholders, including USAID officials (Contracting Officer Representative (COR) and technical officers), implementing partners, IIE, IIEF, current and alumni participants, international student advisors, faculty/deans/department heads, GOI officials at the Ministry of Education and Culture, the Indonesian scholarship program, the Coordinating Ministry for Human Development and Culture, and Indonesia’s National Development Planning Agency – (or BAPPENAS). In addition, follow-up interviews and group discussions were held with employers who had knowledge of alumni job performance, Indonesian providers of short-term training, representatives of the Alumni Association ALPHA-I, Fulbright alumni, and the American Indonesian Exchange Foundation (AMINEF) staff. For comparative purposes, the team also interviewed representatives from the Netherlands, Australia, and the United Kingdom who offer similar scholarships. Annex IIl contains detailed information about the persons interviewed in each category. Through key informant interviews and group discussions, the evaluation team collected qualitative data to help answer the evaluation questions, deepen the understanding of the quantitative data and bring it to life with illustrative anecdotes and quotations. The two teams of interviewers initially conducted interviews together in Washington, D.C., New York, East Lansing, Bandung, Yogyakarta, and Surabaya, and were in constant communication when the teams operated separately, to ensure consistency of questioning and response coding. In total, the team interviewed 120 key informants; 59 male and 61 female respondents. 18 Following the collection of all the quantitative surveys and analysis of the qualitative data the team conducted a consolidated data analysis. In a series of team discussions, the team reviewed the findings and analysis of the qualitative data to inform and explain the findings from the quantitative data and vice versa. The evaluation team presented preliminary reports to key audiences including USAID on April 8, 2015; IIEF on April 9, 2015; and relevant GOI agencies on April 10, 2015. The team factored in the feedback received during each presentation into the preparation of the final report. The IBTCI evaluation team was composed of six members led by Team Leader, Mr. Colin Davies. Annex V includes a description of staffing. 2.2 EVALUATION LIMITATIONS LIMITATION 1: For practical reasons, such as distance and time availability, telephone interviews were occasionally used as an alternative to face-to-face interviews. Interviews via telephone preclude observation of the expression of emotion and feelings through body language and limit the interviewer’s capacity to sense the genuineness of the response. Skype was used whenever possible and mitigated this limitation to some degree. LIMITATION 2: There was a relatively small number of employers (six) in the group of respondents interviewed. This resulted in limited perspectives from employers that could be used to compare alumni perceptions as revealed in interviews and the online survey. However, information gathered for this report will be complemented by Tracer Study findings. Along with other topics, the Tracer team is studying employer perceptions of USAID-funded trainees between 1995 and 2012. LIMITATION 3: Given time and financial constraints, the evaluation team could only cover the scholars/alumni located in the western part of Indonesia (Aceh, Yogyakarta, Surabaya and Jakarta with its surrounding cities). This limits the generalizations that can be made from the findings, as people from the eastern part of Indonesia may have different perceptions of PRESTASI-II’s achievements. LIMITATION 4: Despite the evaluation team’s efforts to schedule a meeting with DIKTI, the key person at DIKTI was not reachable for an interview. USAID staff advised the evaluation team that the DIKTI official is unlikely to be knowledgeable about PRESTASI. In addition, the function of managing and monitoring scholarship programs has been transferred to the Indonesia Endowment Fund for Education or LPDP. The team had a particularly informative meeting with LPDP but would have preferred to have been able to include the perspective of DIKTI officials as well. LIMITATION 5: The team recognizes that the individuals who choose to complete survey questionnaires or consent to be interviewed were self-selected. This limitation is endemic to all evaluation research and limits the representativeness of the data. Nonetheless the team is confident that the large number of respondents has provided valid information. 19 3. FINDINGS, CONCLUSIONS & RECOMMENDATIONS 3.1 GENERAL RECOMMENDATIONS Size of Program “I had the opportunity to go to the United Kingdom but decided to go to the US and get a new perspective. PRESTASI is a great program -- one of the most scholar-friendly programs available.” Alumnus, University of California, San Diego One of the current PRESTASI-II scholars interviewed said in a follow up email said, “It was a great time to share with both of you the experience I have here funded by the USAID-PRESTASI Scholarship. I hope it will be continued for the coming years and more Indonesian people will get this great opportunity to embrace graduate student life in the United States. Although it’s tough and challenging, I am so blessed for having this experience in my life journey.” Current scholar, Michigan State University: “It’s a win-win for everybody. We hope these programs keep going” said a Faculty advisor, Oregon State University. With these comments, the scholars captured the overall reaction of both current scholars and alumni of the PRESTASI-II Program. Current scholars and alumni appreciated the extraordinary opportunity they had been afforded and hoped it would be extended to many other Indonesians. Everyone interviewed said they would encourage a colleague to apply. Findings A large majority (92%) of the survey respondents rated their overall experience as a PRESTASI-II scholar as “Excellent” (33%) or “Very Good” (59%). Fully 100% said that they would recommend this program to their colleagues. All survey respondents said that their educational experience has had a Very Positive (50%) or Positive (50%) impact on their careers. Figure 2: Overall experience as PRESTASI-II Scholars Source: PRESTASI-II Online Survey, n=27 20 Conclusions As subsequent findings and recommendations in this report demonstrate, PRESTASI-II is a well-designed and remarkably well-implemented scholarship program that is achieving its primary objectives including gender balance. As two faculty advisors put it, “Thank you to USAID for the opportunity to host a PRESTASI student. I think this program is great.” And “It’s a win-win for everybody. We hope these programs keep going.” Three current scholars put it thus: “A friend told me about her Fulbright. This is better” and, “They recruited people who really want to be change agents.” “It is like a miracle for me to get this scholarship. I will be able to apply this new knowledge at home.” Well-selected scholars are honing their leadership abilities and gaining knowledge and skills that will enrich their institutions and contribute to the development of sectors of activity in Indonesia that reflect USAID priorities. However, the program is not taking full advantage of the available economies of scale. Much current effort is expended to attract a large number of applications for only relatively few scholarships. There are thus many disappointed, qualified applicants who are not involved. Recommendation 1 Increase the size of the program. PRESTASI is filling an important gap in scholarship availability to students representing underserved geographic areas and populations at a time when the potential for educational programs to contribute to achieving USAID and Indonesian mutual development goals and to strengthening the US-Indonesian bilateral relationship is at an all-time high. Increasing its modest size would also reduce cost per participant. The team recommends that USAID should be the party responsible for implementing this recommendation. Ph.D. Awards Findings Almost every scholar interviewed asked whether there were any further opportunities for studying for a Ph.D. Here are some comments from past and current scholars; “If you want to be somebody in the Ministry of Agriculture, you have to continue and get your Ph.D.” and “I am in close contact with my boss who actually visited me on campus. He is urging me to go onto a PhD.” A faculty advisor said, “I really wish I could help them continue to a PhD.” This is a highly prized credential in a country with few alumni of doctoral programs. Those university deans and vice-deans interviewed also described their respective institution’s need for more overseas-trained Ph.D. faculty scholars. The pressing need for better-trained university faculty, with an understanding of how to produce valid and reliable research, and the ability to guide research was often mentioned. A third of the scholars studying under BAPPENAS are pursuing a Ph.D. as are 25% of the LPDP awardees, and15% of the Australian Awards recipients. An extended stay in the United States would enable scholars to absorb and internalize many of the strengths associated with US academic life, such as a rigorous intellectual enquiry, academic freedom, capacity to articulate and adopt solutions, ability to work in teams, and the development of critical thinking. The study and research involved in a Ph.D. can increase the scholars’ access to data sources and their exposure to rigorous research methodology, and allow them to collaborate with peers pursuing similar research topics, thereby building lasting professional networks. It is also recognized that it is often hard to recruit women for Ph.D. programs because of the extended absences from their families. One often cited concern about Ph.D. study for international scholars is the issue of their non￾return to their home countries. However, all evidence collected in support of this mid-term evaluation 21 demonstrates that this is not an issue with Indonesian scholars. Since the 1980s, only one USAID-funded student or scholar has failed to return. A 2015 report by the Organization for Economic Cooperation and Development (OECD) on Education in Indonesia states: ”No Indonesian university is highly placed among the various ranking of world universities….Academic teaching staff are under qualified by international standards….Only a very small proportion of the Indonesian workforce, including in higher education and research, are qualified at the Ph.D. level. The shortage is acute outside Java.” A representative of LPDP told the evaluation team, "Our universities need more Ph.Ds." He pointed out that the ratio of Ph.D. recipients to the general population is: Indonesia – 143 per one million population, Malaysia – 509 per one million, United States - 10,000 per one million. Conclusions To support Indonesian development in sectors of primary interest to USAID, more scholars should be afforded the opportunity to earn Ph.D. degrees in the United States. Recommendation 2 Expand the PRESTASI program to include opportunities for promising Indonesian scholars to pursue their Ph.D. degrees. The team recognizes the contractual difficulties involved in funding Ph.Ds. since their period of study will in most cases extend beyond the length of a contract/task order. However, it is recommended that the possibility of transferring scholars from one contract to another (a common practice under many USAID academic programs) be reviewed. It is also recommended that a test case be carried out involving an Indonesian scholar currently pursuing Ph.D. studies at an Indonesian university. The team recommends that s/he be sent to a US university for a 6-month period to conduct field research and learn more about research methodology. In addition, it is recommended that the possibility of offering a one-week course, "An Introduction to Studying for a Ph.D." be held in Indonesia for scholars who have successfully completed a Master‘s program. The course would include topics such as "Are you Ph.D. material?", "The theoretical basis for your discipline," "How to apply for a Ph.D. program," "Research methodology," " Using primary and secondary sources," "Institutional Review Board ethics and practices regarding research on human subjects" and "Confirming Data". 3.2 COMPONENT-SPECIFIC RECOMMENDATIONS 3.2.1 Outreach and Recruitment Findings The IIEF engages in outreach by utilizing various methods, which include: talk shows, presentations to government and academic institutions in different cities in Indonesia (three cities in the western part of Indonesia and three cities in the eastern part of Indonesia), the project website, use of social media (Facebook and Twitter), print media (local and national newspapers, brochures, etc.), radio and television to promote the project. IIEF also disseminates information and encourages applications from women, candidates from underserved areas, and people with disabilities. To ensure that more women would apply and take advantage of the program, IIEF visited women’s organizations and the GOI Ministry of Women’s Empowerment. When possible female alumna were involved in outreach activities and served as examples to motivate and encourage women to apply. More women than men applied for PRESTASI-II scholarships. The percentage of women applicants from Java is much higher than outside Java, if application data are quantified proportionately according to provinces. 22 Interviews through information and communication technologies (ICT) and alumni confirmed that scholars learned about the program through the Internet, advertisements, friends, and in the case of some, national and local government officials and university staff, or through their employers. In some cases, their employers even suggested they apply. Recent recruitment efforts include staff on-site presentations in various regions around the country. Interviews in Aceh and East Java suggested that potential scholars in the non-governmental (NGO) sector and the private sector were not being approached. Based on online survey results, the majority of the 81 respondents learned about PRESTASI from articles/news on web site (38%), advertisements (21%) and from friends/word of mouth (17%). Figure 3: Means/Media used to learn about PRESTASI-II Source: PRESTASI-II Online Survey, n=64 A further breakdown of data by region shows that people from Java learned about PRESTASI-II mainly from the Internet and from recommendations from their supervisors, while people from outside Java (Eastern and Western part of Indonesia) learned about it mostly from the Internet and advertisements. Figure 4: Means/Media used to learn about PRESTASI-II by geographic location of Scholars 23 Source: Online Survey, n=64 More applicants from government institutions learned about PRESTASI from the Internet and recommendations from their supervisors than applicants from NGOs/CSOs. Figure 5: Means/Media used to learn about PRESTASI-II broken down by type of institution Source: Online Survey, n=64 Conclusions This project is both gender and geographically inclusive. Efforts have been made by exercising affirmative action to ensure more women and those in geographically remote and underserved areas can participate and take advantage of opportunities provided by the program. Regional considerations were 12% 9% 6% 0% 20% 7% 6% 4% 10% 4% 6% 6% 5% 2% 1% 1% 0% 10% 20% 30% 40% Article on internet Advertisement Friend/Acquintance (word of mouth) Program Presentation/Visit Recommendation from supervisor Flyer/circulation letter in workplace Western-Indonesia Java Eastern Indonesia 12% 9% 5% 5% 4% 1% 17% 9% 10% 5% 6% 4% 9% 5% 2% 1% 1% 0% 10% 20% 30% 40% Article on Internet Advertisement Friend/Acquintance (word of mouth) Program Presentation/Visit Recommendation from Supervisor Flyer/Circulation Letter in workplace NGO/CSO Government Institution 24 taken into account by involving regional representatives as members of the selection panels. The project was successful in reaching out to women but could valuably direct additional effort to encourage applicants from outside of Java, especially those from the Eastern part of Indonesia, and from people with disabilities. IIEF can expand its outreach and take greater advantage of networks of appropriate non-governmental organizations in each region. It should be noted that outreach has expanded significantly as the program has matured as illustrated by the fact that both PRESTASI and ALPHA I participated in a recent Am Cham event. Recommendation 3 Continue to increase efforts to reach people from underserved areas and people with disabilities. More outreach should target regional associations of NGOs and business organizations. The possibility of conducting a joint outreach program should be discussed with the Fulbright implementing agency, American Indonesian Exchange Foundation (or AMINEF), and with LPDP. The directors of both organizations expressed interest in conducting joint recruitment activities. The party responsible for implementing this recommendation is IIEF. Finding Permanent civil servants (Pegawai Negeri Sipil or “PNS”) continue to receive their basic salary during their period of study, and are assured of their employment status though not a specific position when they return. Those civil servants not in this category (Non-PNS) are offered no such security. Employees of NGOs and businesses do not receive a salary during their period of study, and are less likely to be guaranteed employment upon return. Conclusion Non-PNS government staff members are less likely to apply for the program. 3.2.2 Selection Process Findings The regional panels recommend applicants for Final Selection by USAID. These panels are convened in cities where the most applicants reside. During the first round of recruitment the regional panel consisted of two panelists from the region and one from USAID. In the second round panel make-up was changed to two people from USAID (one from a Development Objective Team heavily involved in the region and one permanent representative to attend all panels), and one local person with relevant field expertise. (The Fulbright program uses a four-person panel, one from the State Department, one from the Fulbright office, and two local among Fulbright alumni.) There is a list of standardized questions in use that several stakeholders who were interviewed believed does not allow for a sufficiently free-ranging discussion or probing of the candidates to insure their suitability and adaptability. The scholars had some interesting and positive comments to make about the Selection Process: “I thought it was an excellent process. The application form was about the right length and the criteria were reasonable.” “They should look beyond the numbers [such as scores from the Graduate Record Examination (GRE) and Test of English as a Foreign Language (TOEFL)], and rather pay attention to whether the candidate has carefully researched option and try to honor preferences.” “The process was very fair. They interviewed me for more than one hour.” An IIEF official said, “The interview process is 25 very dynamic. It should not be constrained by standardized questions. Panelists must be able to probe and let discussion evolve organically.” Comments from faculty advisors and international student advisors were equally positive: “The selection process is good. The students are dedicated and work hard.” “Selection is excellent. The scholar is very engaged and diligent.” “The selection process is excellent. I am extremely impressed with both scholars. They are both obviously leaders; both are committed to the idea of culture-sharing and are actively involved on campus.” Conclusions Local factors – social, political, economic – may not be taken sufficiently into account during the Selection Process. The standardization of questions tends to preclude more in-depth probing of the interviewee’s personality, adaptability, and knowledge of the proposed field of study. Recommendation 4 Introduce a four-person selection panel, two from USAID, and two with local expertise (preferably one with experience working on a USAID project). Continue to use the standardized questions as a guideline but allow the panel to probe and conduct an in depth discussion with each candidate. Involve alumni whenever possible. USAID and IIEF are the recommended responsible parties for implementing this recommendation. 3.2.3 Scholar Preparation The Placement Process Finding Despite the fact that 93% of survey respondents said they would have preferred more involvement in the selection of educational institutions, 90% said that their wishes, suggestions, and requests were listened to. Conclusion The current placement process is working well and should be maintained. Pre-Academic Training and Orientation Finding The response to the question, “in general, how well prepared were you for your educational experience,” was: 50% considered themselves Well-Prepared and 49% Prepared. Conclusion Even allowing for some possible courtesy bias, this high rate of preparedness is a strong endorsement of the Pre-Academic Training component of the program. English Language Training Finding More than two-thirds (68%) of survey respondents took a course in English at Lembaga Bahasa International-University of Indonesia (Question 19). Of these, 63% found it Very Useful and 35% Useful. 26 In particular, half of them judge that Academic Writing is the most useful skill they gained from the training. Figure 6: The Usefulness of English Language Training Source: PRESTASI-II Online Survey, n=52 More than half (56%) found the length of the course “About Right” and 32% found the course “Too Short.” In the comments section respondents requested “more focus on IBT TOEFL content with reasonable timing/length for improving the TOEFL score”, while some suggested greater focus on academic writing and presentation/public speaking skills, with more native speakers providing the training. (See figure 6) Figure 7: Suggestions for Improving English Language Training Source: PRESTASI-II Online Survey, n=45 Despite the finding that 98% of the respondents found the classes Useful or Very Useful, 65% of the scholars said that their limited English language proficiency had been a barrier to getting the most out of their program while on campus. This finding was confirmed by almost all of scholars and alumni interviewed. Many said they would have benefitted significantly from studying English in the United States the summer before starting their course of study. They also believed that this would have 27 The ‘Debriefing Notes from USAID/Indonesia’ March 5-9, 2007 ofthe Training Future Leaders report, describe the GPT-ll Project: “Candidates were placed in three tracks according to their scores on the TOEFL….The entire program lasted 12-14 months, and some participants attended an additional 2-3 months of English “topping off” in the US… The model was found to provide an adequate pool of English speaking participants prepared for US study.” accelerated their cultural adjustment process and understanding of classroom culture in the United States. From the survey and from interviews with current scholars, alumni, and faculty advisors, it is clear that the lack of sufficient English language and academic study skills is putting many students at a disadvantage when they start their graduate study. The online survey found that 51% respondents experienced difficulties with English language as a barrier, particularly due to limited conversation and reading skills. Other difficulties included home-sickness (missing home), becoming accustomed to cold winter weather, and becoming comfortable participating actively in class discussions. There were many comments from current and past scholars and faculty advisors that demonstrated this finding. From students: “I had to take two English classes my first semester, and one my second. This meant that I fell behind the other students who took three subject classes right from the start.” “The English course taught us how to pass the test. We did very little academic English.” “There were some U.S. guest lecturers, but we didn’t really acquire any study skills.” “By the time I’ve thought in my head what I want to say, someone else has already said it.” From faculty advisors: “All my Indonesian students definitely struggle at first because of their limited English.” “The students know the science content but they need to learn to write better. They would definitely benefit from taking English upon arrival.” “I know the Indonesian students have a lot to say but they lack confidence in their ability to say it.” “It is better to plug their language gaps earlier in the summer before the program starts.” “We love these students and would love to have more. They are wonderful students – but their English lets them down.” Figure 8: Academic Challenges Source: PRESTASI-II Online Survey, n=54 Students who achieve a score of 550 or more on the TOEFL exam currently take an English language program for two months or 30 hours a week/240 hours. Students who achieve a 500+ TOEFL score study for four months or 480 hours. 50% 21% 8% 8% 6% 2% 6% 0% 10% 20% 30% 40% 50% 60% English language (reading/speaking/conversation) Study workload/time management Active participation in class Academic writing & presentation skill Homesick/Family issue Culture, food and weather Others 28 Students who achieve a score of 450-500 study for six months or 720 hours. In contrast, students studying at the IALF (Indonesia Australia Language Foundation) study for significantly longer. (Note: The Team is not proposing a particular vendor for the provision of EFL services, but rather comparing the lengthy amount of time normally dedicated to English language preparation for Master’s students by various scholarship-funding organizations.) Those with a score of 500 study for at least six months and those with a score of 450 study for at least nine months. This is followed up on arrival in Australia with an additional one-month course in Introductory Academic Preparation. The current provider of language instruction, the University of Indonesia, recommended the same length of programs to DIKTI. Their recommendation was accepted, and the research showed an increased rate of success. Students who are studying at BAPPENAS also study English for similarly longer periods. Those who score a 550 are entitled to study for three months, those who score a 500 for six months, and those with a 450 for nine months. For certain scholars LPDP offers up to 12 months of English language training. In many cases, PRESTASI students who arrive at their universities not ready to take a full course load because they lack the necessary English language skills. In some cases, they also lack the necessary academic skills include, but are not limited to: note-taking, writing reports, citing and referencing, structuring an assignment, developing an argument, writing critical reviews. Several of the students interviewed had only taken two classes in English, and only one in their subject of study in their first semester at a US university, which meant they would have to play “catch up” in the subsequent semesters. Conclusion Despite the fact that scholars are enthusiastic about their English classes, in many cases they and their faculty advisors judged that they still needed more English language training before starting their substantive course work. The current two-month program of English study is sufficient for those students who receive at least a 550 TOEFL score. For those who receive less than 550, more English language preparation is required before they embark on a full course of academic study. Some students are playing catch-up on their course work because they are still spending credit hours taking courses to improve their English language and study skills during their first and second semesters of graduate work. Recommendation 5 Increase significantly the length of time that scholars study English language before beginning their other academic work. Students who score between 450 and 500 on the TOEFL should study English nine months. In addition, the evaluation team recommends that students would benefit from arriving on their US campus earlier (i.e. in June) and taking a two-month summer course in English language and in academic skills. These students should be given the opportunity to observe classes during the summer semester in order to gain a better understanding of U.S. classroom practices and behavior. The team understands the time limitations placed by the current contract date and therefore recommends that for this contract the period of in-country English language training be increased to eight months. The possibility of longer nine-month training should be considered for future programs. In addition, the team recommends a period of English language and pre-academic study in the U.S. for three months whenever possible. USAID and IIEF are the recommended responsible parties for implementing this recommendation. Finding U.S. universities require that complete applications (including a TOEFL score) be received by December. Currently, many students are not taking the TOEFL test until January, which means that their applications cannot be considered until after that date. If applications are not received until February, 29 university placements in the student’s chosen field may not available. In addition, other applicants may have already claimed tuition reductions and financial aid for international scholars. In one interview, a current student complained, “I did not get into my first choice of school [Michigan State University] because I was told my application was too late.” Conclusion The timeline of the process should be shifted by one or more months in order to ensure the earlier submission of complete applications. Recommendation 6 Students should start their English language classes earlier and take the TOEFL test in December at the latest. The Team recognizes the restraints of the current contract, but whenever possible, and for future contracts, recommends a rearrangement of the recruitment, selection and pre-academic training timetable to allow complete applications to be submitted earlier in the academic year. This was a request made by both IIE and the US campus advisors, since it would increase the scholars' possibilities for being accepted and for receiving tuition reductions. IIEF and its vendor are the recommended responsible parties for this recommendation Statistics Training Finding The survey showed that 49% of the respondents took a course about data analysis and statistics at Atmajaya University. Of these, 11% found it “Very Useful” and 68% “Useful”. Most (87%) said that the course was too short. Some respondents recommended a longer course and more practical exercises. See figure 8. Figure 9: Recommendations for Statistics Training Source: PRESTASI-II Online Survey, n=34 Almost all scholars and alumni interviewed agreed that the training was too rushed to be of real use. Some scholars noted that they had no use for statistics. Others already had taken a course. One of the respondents said: “It was a very short period for learning many statistical topics from morning until afternoon. Many scholar friends of mine fell sick at that time because of that. It is better to make the training period a bit longer, so scholars will not have to be forced, especially to memorize 30 all the statistical formula in a very short time.” Other scholars said, “The second day, half the students didn’t show up.” “The statistics course was simply not helpful. The presenter tried to cover too much in the time available.” “It was far too basic for me, and too advanced for some of the other students. We were all at different levels.” “It had limited utility. I did not have the background to learn much. Too many topics were covered in too short a time.” “The course was not so useful. I had studied statistics already.” Conclusion There is not enough of a common denominator of need to justify using the time of potential scholars for this activity. Even the vendor currently delivering the workshop said that the “different level of capacity of the students” made it a challenge to design the workshop. There is little support for the course as it is currently presented among scholars or alumni interviewed. All U.S. universities offer courses in statistics. Recommendation 7 The statistics training should be discontinued. USAID and IIEF are the recommended parties responsible for implementing this recommendation. Leadership Workshop Finding Three-quarters of the scholars completing the survey took a course in Leadership Training at the Daya Insani/Universitas participant training. Almost a third (29%) found it Very Useful and 65% Useful. Also, most (78%) found it to be “About Right in its Length”. Interviews suggest that the Leadership Workshop was particularly helpful in increasing self-confidence, building relationships among scholars, and honing their ability to work in teams. Facilitators did a good job of building self-awareness. While appreciating these aspects of the training, respondents suggested that the leadership dimension of the training should be enhanced. Not enough attention was paid to identifying the characteristics of good leaders in various cultural contexts and developing leadership skills. Comments from participants included, “It was a great way to bond with other students.” “The Leadership Workshop was very useful. It was more like character building. There were many stimulations. Some were designed to help us handle stress.” “The Workshop helps to build relationships with peers. It builds confidence to speak up. It pushed me to get out of my comfort zone.” Figure 10: Perceived benefits from Leadership Workshop 31 Source: PRESTASI-II Online Survey, n=48 Conclusion The course in its current format is popular among the scholars, particularly because it builds rapport among a specific cohort or “batch”. However, it could be even more effective were it to be linked to an ongoing study of leadership and also related to the re-entry process. Recommendation 8 Develop a more structured leadership program with several components linked to the re-entry program. While the Leadership Workshop component of the Pre-Academic Training was universally appreciated, linking it to re-entry issues and assuring consistency over time would make it more valuable and easier to administer. Identifying a vendor and working with that organization during a three year period to polish and perfect the program would be even more useful. Consideration should be given to involving employers as speakers in the training, well as developing a “Leadership Lessons Learned” diary that each scholar would use as a workbook during the leadership training and then as a place to record observations made in the United States to be shared during re￾entry activities. Topics may include the qualities of good leadership, leaders they admire, and the extent to which there are universal principles of leadership and the extent to which leadership principles are culturally specific. Pre-departure Information and Orientation Finding The online survey queried scholars about the information received prior to traveling to the U.S and the adequacy of their various logistical arrangements, including medical examinations, visa applications, USAID regulations, travel and about the Pre-Departure Orientation. Almost all were satisfied. Almost one-third (32%) of respondents were “Extremely Satisfied”, 40% “Very Satisfied” and 25% “Satisfied.” In the comments section, “More Information about the Health Examination” was the most frequently cited request. Some mentioned the difficulty faced by those coming from remote areas outside of Java to take the health examination in Jakarta. 32 Thirty-nine percent of respondents reported they were “Very Satisfied” and 55% were “Satisfied” with their orientation. Only four respondents (6%) expressed their dissatisfaction, which tended to be due to specific occurrences such as a delay in visa-processing that led to missing pre-departure orientation and little involvement in making decisions regarding placement. Figure 11: Satisfaction of Respondents with Pre-Departure Orientation Source: PRESTASI-II Online Survey, n=74 Respondents were asked which pre-departure topics should be addressed or explained in greater detail. The most frequent responses were: Health Insurance (50%), the Stakeholders Compact (31%) and Learning Methodologies (30%). 33 Figure 12: Topics requiring more detailed explanation during Pre-Departure Orientation in Indonesia Source: PRESTASI-II Online Survey, n=70 Interviews confirmed these findings. One alumnus interviewed experienced a serious health crisis during a winter holiday break and reported that he had to spend three hours on various buses to reach a health care provider that would accept his insurance. Conclusion The responses about logistics and preparedness were generally, but not overwhelmingly, positive. Scholars require more explanation about medical examinations, accessing health insurance providers, and more information about studying in the United States. One student commented: “I found it very hard to understand the health system in the U.S. It is completely different from ours.” Recommendation 9 More time should be given to explaining the procedure for medical examinations and accessing health insurance coverage when in the United States. A review should be made to ensure the network of providers is accessible by public transportation. More information should be included about the classroom culture and interaction with faculty at a US university. Alumni should be involved in the orientation for them to share their experiences around those issues of concern. The recommended responsible parties for implementing this recommendation are USAID and IIEF. Finding The survey shows that many students thought that several topics had not been covered in sufficient depth in the Indonesia PDO (Pre-Departure Orientation). In addition, it has been observed in other exchange programs that students do not really focus on the information when it is presented to them in the abstract. One student noted: “We needed to have more information about the U.S. There isn’t much public transport and you have to take taxis.” It is recognized that information needs to be presented several times, in different contexts, in order for it to be reinforced and adequately absorbed. Many students said in interviews that they found their US orientation from IIE to be extremely useful in forming the bond between the students and the IIE program staff they will depend on for the next two years. One student said, “Orientation on the US 34 would be ideal. Scholars will pay more attention if information is conveyed in the place it will be used.” Another said, “The D.C. Orientation gave us the opportunity to bond with our program officer.” Conclusion A strengthening of the PDO is required, and should include greater involvement of the IIE staff, who possess extensive knowledge of the U.S. educational system, experience on particular campuses (through their regular program of campus visits), and who will assume the role of mentor (and friend) to the students for the next two years. Recommendation 10 USAID should give serious consideration to resuming the practice of having IIE host a three-day arrival orientation in the United States (in addition to the PDO hosted by IIEF). If this is not possible given budget constraints, two IIE Program Officers, with experience of a wide range of US campuses, should be invited to take part in the PDO in Indonesia. Orientation should include a briefing on USAID, its mission, history, and current objectives in Indonesia. It is understood that a budget analysis needs to be made before making a decision regarding the location of the orientation program. The recommended responsible parties for implementing this recommendation are USAID and IIE. Finding Some students reported a feeling of unfulfilled expectations during the period between their acceptance in February/March and their arrival on campus in August. Other stakeholders also mentioned the challenge of building a relationship with their new students during this critical period. Conclusion Activities should be designed and offered to maintain students’ enthusiasm and to provide the necessary pre-departure information in easier to digest chunks and better prepare them for studying in the United States. The Evaluation Team recognizes the usefulness of Massive Open On-line Courses (MOOCs), but appreciates the wide variations in the type, quality, and relevance of such programs and their requirement for fast and reliable internet connections which are often unavailable in Indonesia. The Team therefore recommends that materials be designed that are specific to the pre-departure needs of PRESTASI scholars, and that these materials be developed in both webinar and hard copy format (for those scholars for whom internet access is a problem. Recommendation 11 An earlier recommendation (#6) suggested that students should go to their host university in the U.S. for a summer program earlier than is current practice so that they can focus on English language training and related academic skills. A series of webinars for scholars should also be developed and offered during the months prior to their departure. This would better enable them to sustain the high level of enthusiasm experienced immediately following selection. Topics might include American classroom culture, adaptation skills, cross-cultural communication, an assignment to read a selection from the novel The Killer Angels and discuss the nature of national identity, etc. Even though much of the material could be covered in a later orientation session, these webinars may provide opportunities to practice English and begin to grapple with topics relevant to making the most of their US experiences. Postal mailings could be substituted for the webinars if internet access is an issue in a particular geographic region. IIEF is the recommended responsible party for implementing this recommendation. Finding 35 The responsiveness of the scholar’s employer to new ideas and the employer’s willingness to effectively utilize the new knowledge and skills of returned PRESTASI scholars are not major considerations in the application process. The signing of the Stakeholder Compact for U.S. Training can be a pro forma procedure. In several cases scholars complained about returning to an institution that was not particularly interested in giving them the increased responsibility or the opportunity that they sought in order to incorporate new ideas or practices. In contrast, the BAPPENAS program has a particularly impressive process in offering this. The human resources department has the responsibility of assigning each student a project upon return that is relevant to the student’s topic of study and also to the needs of the organization. For one year the returned scholar works on this task under the direct supervision of their superior. While this is not a practical model in all cases, it does recognize that greater employer buy-in would be advantageous. Conclusion The PRESTASI program should seek more ways to encourage expanded ownership by each scholar’s employer. Ideally the program would become a true Human and Institutional Capacity Development (HICD) program with a greater focus on institutional strengthening. There should be the concomitant recognition that training may not necessarily be the solution. A truism in the corporate training profession is that more than 80 percent of training programs are trying to solve problems for which the solution is not in fact a lack of skills but rather a broader problem in corporate organizational structure, mission, incentives, relationships, or policy. Even if the focus of the PRESTASI program is to remain on the scholar, rather than on the institution, the involvement of the institution should be increased, to ensure that the maximum development benefits are derived from each scholar’s course of study. Recommendation 12 The role and responsibilities of the employer should be more clearly defined in the Stakeholder Compact. Each student should be required to draw up a preliminary Action Plan in consultation with her supervisor which demonstrates how they plan to use the scholar’s newly acquired skills and knowledge to the maximum benefit of the employing organization upon their return. The Compact should also state that each scholar is required to send the AETR to the employer each semester, and to maintain a dialogue with the employer, including sending papers and articles of interest to their colleagues. It is additionally recommended that the IIE Newsletter be sent to employers each month. The recommended responsible parties for implementing this recommendation are USAID and IIEF. Recommendation 13 The program should identify ways to involve employers in Pre-Academic Training leadership training, orientation, and ALPHA I activities by inviting them to serve as speakers or interviewing them and featuring their profiles on the PRESTASI and ALPHA I web sites. The recommended responsible parties for implementing this recommendation are IIEF and ALPHA-I. The intent of this recommendation is to give employers more ownership of the program. In addition to inviting employers to speak at leadership workshops, either as panelists or as plenary speakers, it is recommended that employer interviews be included in the PRESTASI and the ALPHA-I websites. Interview topics could include a discussion of leadership, examples of leaders they admire, traits those leaders embody, the qualities of leadership they look for in alumni, and how scholars might acquire those qualities during their period of study. 36 Orientation in the United States Finding Most survey respondents (88%) attended an orientation in the United States (these U.S. orientations have now been discontinued). Of these, all said they were useful: 67% considered it Very Useful and 33% just Useful. The topic most frequently cited as needing further explanation was Health Insurance. Figure 13: Topics of Orientation in the United States requiring more detailed explanation Source: PRESTASI-II Online Survey Conclusion There is a high level of satisfaction with the IIE orientation program. More information needs to be included in the orientation about how to access health insurance in the U.S. 3.2.4 Scholars’ Experience in the United States Campus Experience “We have in the foreign student community in this country something that could be a terrible time bomb or a tremendous source of international understanding— both in what they come to know about us and in what American students learn from them. We are training a generation not only of foreign leaders but of American leaders, and it is terribly important, therefore, that our foreign students not be isolated, that they mix, from a part of the community of the universities where they are studying, for their sakes, but even more for our own.” Moorhead Kennedy, The Ayatollah in the Cathedral: Reflections of a Hostage, 1986, page 172. Despite the passing of three decades, Moorhead Kennedy’s observation is still relevant. It is important to find out whether international scholars in the United States consider themselves an integral part of their communities and campus life. Finding In survey responses, 26% of scholars described their interactions with their fellow students as “Excellent,” 36% as “Very Good,” and 42% as “Good.” In addition, 75% reportedly participate in cultural and recreational activities or clubs at their universities, and 70% are involved in community or 37 volunteer activities. International student advisors interviewed repeatedly stressed the importance of encouraging PRESTASI scholars to take advantage of opportunities for “culture sharing.” Several current scholars interviewed reported that they made presentations about Indonesia at local schools. Others participated in various events and field trips. Despite these good reports, interviews revealed that some students had difficulty making friends with American students. Many suffered from homesickness particularly during the first semester. Being lonely and missing their families was a common theme during interviews. Two current students said, “It is not easy to make friends with domestic students. Most spend their weekends in a bar. You do not realize that you don’t need to drink on a ‘pub crawl’. It is easier to get to know other international students.” “American students are open to friendship. You just have to start it.” And from an alumna, “To be honest, it was very difficult to get American friends. That changed when they learned that I like to cook. The provided the ingredients and I taught them how to cook Indonesian dishes.” Conclusion A majority of PRESTASI scholars are well integrated into US campus life. Some play leadership roles in campus organizations. For example, one current scholar interviewed at Michigan State University was serving as president of the Indonesia Student Association. Another used cooking Indonesian dishes as her vehicle for making friends with US students. However, some interviewees admitted that they had a difficult time making American friends. As one current scholar phrased it: “It is difficult to make friends with domestic students.” Reasons proffered include language barriers (“Sometimes I am not understood by Americans”), how busy everyone is, and how many social activities on US campuses revolve around drinking. Some found it hard to identify easily accessible opportunities to interact with Americans and to make American friends. Recommendation 14 Given the need to build stronger relationships with U.S. students, develop a pilot “Meet America” program to take three groups of 10-12 scholars, including several American students, during a holiday break, to have an enrichment experience similar to the Mid-Winter Seminars of the 1980s and 1990s. Participants would be placed in homes with US families. It is understood that meeting and engaging with American families and the community is an important goal of the PRESTASI program. This proposed activity - as well as providing the scholars with valuable skills - would be a further way of achieving this goal. Many USAID participants in the 1980s and 1990s enjoyed workshops or seminars, lasting between one or two weeks, as part of their US program to provide them with management training supplemental to their academic programs. One Evaluator, Andrew Gilboy, found that “participants repeatedly and forcefully stated that work attitudes, critical thinking, and other ‘non-technical’ tools were the major benefits of their training….” The relevant evaluation report by Gilboy et al. recommended that “every future long-term participant….returns with a toolkit of non-technical, managerial, and attitudinal solutions to the myriad challenges to be faced at the workplace at home” (Gilboy, 2004. P.52). Workshops might feature sessions about general management skill areas, for example, leadership, decision-making, management communications, conflict resolution, and team building. The workshops would be designed to help participants apply their learning to their own institutions and situations. They would also include a re-entry component with case studies, group discussions, and some individual planning for participants on how to utilize their newly acquired knowledge and skills when they return home. The parties responsible for implementing this recommendation are USAID and IIE. Finding 38 Separation from their families was the difficulty mentioned most often by scholars in their interviews and in the open-ended questions in the online survey. Many scholars, both men and women, spoke plaintively of their unhappiness at being separated from their spouses and especially children. They spoke of homesickness and even of overwhelming feelings of guilt. This topic was not included in the questionnaire, but was one that emerged during the interviews. It was mentioned frequently by both the married students (who spoke of their anxieties and distress at being separate from their families) and by the single students who noted that their married colleagues frequently spoke of this issue. One female scholar who left a small son at home in Indonesia said she “felt like an irresponsible mother.” Interviews with married women alumni who appeared to be economically advantaged said that when they were planning to go abroad to study, it was discussed within the family. They identified problems that might arise and discussed how to solve them. It was found that women from well-to-do families were financially able to hire domestic workers and child-care providers to provide support in their absence. This is not possible for women from less financially advantaged families, where the mother traditionally take care of the children. One alumna spoke of a woman she knew whose husband did not allow her to go abroad to study. Regarding the emotional implications of the scholars, following are some of the comments expressed by officials, scholars, and alumni:  IIE Official: “There is a need to rethink the ‘no family’ policy. Indonesians are a very family oriented society.”  Current scholar: “I miss my wife in Indonesia. She has trouble adjusting. If we have family with us, we feel stronger.”  Faculty advisor: “It is hard for older professional students with young children to be separated from their families. USAID could be more family friendly. The student I supervise is going home for spring break, after many interventions. His wife and child who is sick are not coping well.”  Alumna: “The hardest part was being away from my husband and son. I lost 5 kilos. My family and son came the second year. I learned a lot about the U.S. education system because of my son’s experience in school. It was another window into American life.”  Alumna: “It seems to me that USAID does everything it can to discourage us from bringing our families.”  Alumna: “I cried when someone said I wasn’t a good mother.”  Alumnus: “I feel very sorry for my colleagues who are married with children. I know how sad they feel.”  Alumna: “It was very hard to leave my son. This time will not come back. I felt that I was not a responsible mom.”  Alumna: “I was very homesick for my family and after three months wanted to come home. A very good American friend persuaded me to stay on.”  Alumna: Females who leave their children have a hard time. Having their children with them would make them more involved in the community.”  Alumna: “It is important to have family accompany the student. We need to think about maintaining the balance of soul, body, and mind.” Although there is minimal statistically-based evidence that this separation adversely affected the scholars’ study, many scholars mentioned in interviews that they had colleagues who had not applied for PRESTASI-II because they did not want to be separated from their families. 39 In contrast to USAID’s approach, the Australian Awards program allows scholars to bring their spouses after a semester, and, under Australian law, spouses (and the scholars) are able to take paid employment. LPDP pays 25% of the living allowance for each dependent, to a maximum of two dependents (but does not pay for dependents’ health insurance). The BAPPENAS program also allows scholars to bring their spouses after the first semester if they have good academic records, as does the Fulbright Program. Neither of these two programs pays any dependent allowances. The evaluation team appreciated one 1992 report by Gillies, John, et al, “Training for Development: A Review of Experience” that explores the issue of “Spouse Training in Indonesia.” The Spouse Training Program was introduced in the second phase of the Western Universities Agricultural Education Project to increase the pool of qualified candidates for the regular training program, especially women, who might otherwise not be able to participate. It was seen as a way to maximize project resources by providing appropriate training for more people. The primary participants, at least half of whom were supposed to be women, had to achieve a grade point average (GPA) of 3.0 in the first semester in order for the spouse plan to be approved. The project financed 50 percent of in-country ELT, 50 percent of the maintenance allowance, 100 percent of tuition and other training costs, 100 percent of return airfare if the training objectives were met, and 100 percent of health and accident insurance during the training period. The report found that “Experience with other participants indicates that those at the Ph.D. level complete their programs more quickly when accompanied by their families. There is no indication that academic performance is affected as measured by the GPA.” The report’s authors concluded, “A spouse training program has the potential for reducing program time (and costs), increasing the number of women participants, and increasing the number of trained persons.” Conclusion It is understood that the USAID/PRESTASI policy towards spouses follows the USAID regulation, and although it does not specifically forbid them from accompanying the scholar, the prevailing perception is that approval is never given. Recommendation 15 The current practice should be reviewed. Scholars should, on a case-by-case basis, be shown more lenience in response to their requests to bring their spouses at their own expense after the successful completion of the first semester. Families should also be given more support in their efforts to visit the United States for their spouse’s graduation or during a holiday break (though this will not, of course, ease the homesickness during academic year). USAID is the recommended parties responsible for implementing this recommendation. Academic Experience Finding Ninety-four percent of survey respondents said their university placement matched their academic needs and expectations “A Great Deal” (56%) or “Some” (37%). A large majority, 88%, rated the support, which they received from their faculty as “Excellent” (33%) or “Very Good” (55%). In addition, 89% rated the quality of education they received as “Excellent” (29%) or “Very Good” (60%). During interviews with current scholars and alumni, faculty advisors received high marks for their involvement, encouragement, and hands-on academic training. One scholar enthusiastically stated: “My faculty advisor is really awesome.” Most had weekly interactions with their advisees and were 40 thoroughly acquainted with the current status of the work of the scholars under their direction. Often they identified additional university funding to help supplement the funds provided in the grant. For example, one MSU faculty advisor acquired expensive tropical tree samples for a PRESTASI scholar studying forestry so that relevant research could be conducted. One of the respondents offered this typical response: “I had a quality academic experience, especially learning about critical thinking, expressing opinions based on evidence from research, and developing my writing and publication skills.” Another respondent observed: “It is very special, not only to learn about the study that I am interested in but also about finding myself.” One respondent interviewed articulately described this multilayered learning: “I not only learned about my field of Marine Ecology, but I also learned how to communicate scientific findings to non-scientists.” Impressed by the creativity of his professors, this alumnus described his professors as “really great – one day I want to be a lecturer, too.” Conclusion The respondents are very satisfied with the academic training they receive, and with the support from their faculty members. Finding During the interviews, the team received very positive feedback from the trainees who had attended a STT program. The interviewees said that the study was directly relevant to their work, and they had received good support from IIEF. Several employers also requested that their staff be sent on STTs to the United States to focus on topics such as strategic management. One university dean said, “We need our employees to take short, job-related focused courses.” Another employer said, “I would like a short course in strategic management for my staff.” The representative from Australia Awards said they managed several STTs (averaging between two to six weeks) for government employees and were planning to extend these programs to representatives from NGOs and the private sector. The BAPPENAS representative also expressed great interest in STTs in the United States on topics such as energy, tourism, food security, maritime issues, and the environment. IIE has experience in managing STTs on all these subjects, both ‘off-the-shelf’ training programs and “custom tailored’ programs, and also has experience in recruiting and hiring interpreters. Conclusion There is a clear demand for focused STTs, both off-the-shelf and those designed with a specific group in mind. Recommendation 16 USAID is encouraged to consider increasing the number of STTs by tapping IIE’s expertise in this area. Additional courses could be either for groups from the same organization and in the same office, or for employees in similar positions but in different geographic regions. Both these approaches can form a critical component of institutional strengthening. Where applicable, off-the-shelf courses may be used, but there may be occasions when it is necessary for U.S. training providers to design specific training programs, for which a limited competition would be managed by IIE. U.S.-based interpreters, with familiarity with the topic, would be hired as needed. Finding Interviews showed that there is interest in ‘joint Master’s’ programs, whereby scholars take part of their course at an Indonesian university and part in a US university. Research has shown that there are already a number of university linkage programs in place, which could be used to facilitate joint programs. These programs include the USAID-funded Higher Education for Development, the 41 University Partnership Program, National Science Foundation Program, and the Higher Education and Leadership Program managed by Chemonics. There are various models for such ‘joint’ and ‘split’ Master’s programs evidenced by the following. Under the BAPPENAS-Australian Awards program, students study for the first year at an Indonesian university (funded by BAPPENAS) and the second year in Australia (funded by Australia Awards) earning two degrees. Other programs have the reverse approach; scholars take their first year in the United States and return to an Indonesian university to conduct their research. The Team also noted the successful joint program conducted by Sanata Dharma University and Loyola University, Chicago. Similar models exist for Ph.D. programs: there are ‘sandwich’ programs, wherein scholars are enrolled in a Ph.D. at a university in their home country, conduct research and coursework at a US institution, and return home to complete their studies and dissertation presentation. An LPDP representative said, “We fund all types of degrees – joint, split, sandwich.” Conclusion Joint degree programs (particularly the Master’s, with only one year in the United States) would put less of a strain on families. Programs such as these would give a strong professional grounding in the home country, enrichment through immersion in a foreign setting, and reduce international costs. The Team does understand, however, the limitations and constraints involved in joint programming. Recognizing that a previous program was not a success, the Team recommends that this Recommendation be placed 'on the back burner’ for possible future review. Recommendation 17 Serious consideration should be given to joint/split/sandwich programs at the Master’s and Ph.D. levels. USAID is the recommended responsible party for implementing this recommendation. 3.2.5 Individual and Institutional Benefits Finding In response to the question regarding which new knowledge and/or skills they had acquired during their period of study, the primary skills cited were: “Critical thinking” (88% of respondents); English language (85%); “Research Skills” (85%); “Becoming self-reliant” (85%); Time management (85%); and Technical Knowledge in their field (77%). All respondents asserted they had been able to apply their new knowledge and skills in their jobs. 42 Figure 14: New knowledge and/or skills that scholars acquired during study period Source: PRESTASI-II Online Survey, n=26 Conclusion The respondents placed a high value on those acquired skills that they were able to apply in their work settings. Findings A full 100% of respondents said their leadership skills had improved as a result of their graduate study experience, and 100% also said they had applied newly acquired skills in their jobs. Among those respondents who have returned home, 88% said that their degrees strengthened their job performance. A full 100% said their employers were “Very Supportive” (50%) or “Supportive” (50%); while almost two-thirds (62%) were promoted, on average between three and 12 months after return. Ninety-five percent of respondents said they applied new methodologies in carrying out their work. More than three-quarters (76%) said they improved existing training programs in their place of work, and 71% said they improved management systems. The same percentage said they contributed to their organization’s expansion and that they mentored colleagues (Question 100). One alumna said, “My colleagues love it. They say they are proud of me and they listen to my ideas.” An alumnus: “My University just opened a new center for training elementary school teachers, and so I was able to apply exactly what I had learned in the U.S.” Other positive comments include: “I learned about hard work, time management, and I became more disciplined and able to think critically. I also learned how to teach math to blind students. I want to develop this in Indonesia and share the methods I learned throughout the country.” “The experience already changed my personality a lot. It built my self-confidence and motivation and developed my cross-cultural communication skills. It also developed my capacity for analysis, taught me how to make partnerships, and to compare educational systems in a rigorous way.” “I have more self￾assurance. I am not afraid to try new things.” “I will go back and share this new knowledge with my students. My teaching has improved as well. I will make my classes more interactive.” “I learned time 43 The ‘Debriefing Notes for USAID/Indonesia’ March 5-9, 2007 of the Training Future Leaders report reached the following conclusion: “While it is undeniable that the role of one individual can be crucial to change, it is more likely that having a critical mass within an institution will produce the greatest impact. Having two or more US- trained colleagues at an institution gives them a base of support, the ability to pool ideas and mutual reinforcement for introducing and implementing changes. In deeply entrenched bureaucracies and systems that are resistant to change, a team of like-minded professionals can be powerful advocates for change and can serve as role models and mentors for younger staff joining the system.” management, self-management. I am more patient now. And organizational skills.” A vice dean at one university said simply of one of his faculty members, “He returned from the U.S. more caring.” Those respondents who have had difficulties in applying their skills attributed their difficulties to several factors, the leading factor being “poor management of human resources” within the organizations employing them (73%), and a lack of financial resources (55%). Conclusion PRESTASI-II has been successful in its goal of building the leadership and job skills of the scholars, thus fulfilling the objective of the program “to develop a better cadre of skilled leaders and managers.” Figure 15: Application and Contribution in the Workplace after Study Source: PRESTASI-II Online Survey, n=21 Conclusion PRESTASI scholars have received recognition from their employers, and have been very successful in applying their skills in their workplace. Finding Almost all (96%) of survey respondents have been able to share their knowledge and skills with others. Examples of such sharing beyond the workplace include being active in professional organizations and working pro bono for a community-based organization. Conclusion Based on the Survey and the interviews, there 5% 62% 71% 71% 71% 76% 95% 0% 50% 100% 150% Others Taking new leadership role in professional organisation Improving the management system and procedure Institutional development/enhancement Becoming supervisor/mentor to colleagues Improving the exising training program Applying new methodology for conducting job/task 44 are impressive success stories and many examples of the application of newly acquired skills in the workplace. However, some alumni interviewed reported no promotions, changing organizational structures and personnel, and the inability to move to a permanent position. Recommendation 18 Please see Recommendation 12. Finding There was a lot of discussion during the interviews with scholars and faculty, and with the US Implementing Agency about the issue of clustering. In other words: Should groups of scholars from the same institution be sent in the same cohort? Should groups of PRESTASI scholars be placed on the same campus? Many scholars were of the opinion that a group of scholars returning to the same organization would have a greater chance of affecting real change within that organization. Many of the students currently studying thought that having more PRESTASI students on their campus increased the awareness among the faculty of particular Indonesian development concerns. One current scholar said, “It makes the faculty more aware of Indonesian students and they give us more attention. It is also helpful for us to share our ideas with other Indonesians.” On the other hand, some respondents who were the lone PRESTASI scholar on a campus said he had been forced to become more self-sufficient, and to integrate more with other international students and American students. As one scholar phrased it: “I get out of my cocoon and experience new things.” Another said, “Being on my own with no other Indonesians on campus forced me to get out of my comfort zone.” The IIE Placement Office policy is to place the scholar in the program that best matches their academic aspirations, regardless of how many other Indonesians are being placed. A key IIE staffer avowed: “The best fit is our goal.” Another said, “There is not deliberate attempt to have more than one participant from each institution. It is better not to cluster.” Conclusion The issue of clustering scholars on a campus was thoroughly discussed with various stakeholders. Selecting clusters of scholars from one particular institution (or from regional branches of the same institution) can in some instances be a powerful engine for change. Recommendation 19 PRESTASI-II should explore recruiting groups of scholars from the same institution, to maximize the possibility of the scholars having an impact within that institution. The parties responsible for instituting this change are recommended as USAID and IIEF. 3.2.6. Re-entry and Alumni Activities General Finding The survey responses to the question “How do/did you expect your colleagues and supervisor to react when you return with a new degree and new skills were positive?” found that most respondents expect to be welcomed enthusiastically by their colleagues and supervisors especially if they bring new ideas for improving the work of the organization. One respondent said: “I wish that my working colleagues can be supportive and accepting new approaches that I bring in to facilitate/promote more improvements within our organization.” 45 Similarly, 98.5% answered that ‘this is the best program for my needs/my organization’s needs. Almost all (99%) of the respondents thought that their graduate study would be “Very Useful” or “Useful” for their future careers. Conclusion The respondents were almost unanimously optimistic about their return home to their work. ALPHA-I Activities Finding From the online survey, only 10 responded to the questions related to ALPHA-I. This represents less than 15% of the alumni who responded to the Survey. Mostly, the alumni involvement is limited to attending the launching and general assembly meeting of ALPHA-I, and only three of the respondents were involved in the development of the ALPHA-I proposed activities. Periodic progress reports on the ALPHA-I activities and results of interviews with its members suggest ALPHA-I is doing significant work on gender issues that encompasses activities addressing Gender-based Violence, increasing access of the poor and women to government sponsored health insurance, and women’s reproductive health and rights. Interviews suggest that respondents appreciated any re-entry activities that were available to them. Those that consisted of more than a casual conversation with USAID colleagues were most valued. One alumna interviewed phrased it well, stating: “Stress the moral obligation to give back – to share what we have learned.” Indeed, one alumnus said, “It is time for us to pay back. ALPHA-I is a place where all alumni can share ideas and contribute to the development of Indonesia.” It is instructive to hear what an Australian Awards Official had to say on the subject of alumni activities: “We engage in year-round branding and relationship building through email blasts and events throughout the country. Professional development activities are especially popular. We also host an annual dinner that we view as one of our key public diplomacy activities. We are building people-to-people relationships and professional linkages.” Conclusion The ALPHA-I has sponsored a range of worthwhile activities, but only a small proportion of PRESTASI alumni are currently involved. Recommendation 20 All alumni should automatically be enrolled in ALPHA-I and an alumni directory should be posted on the ALPHA-I website including contact information, fields of expertise, and areas of potential mentoring. The program should develop a more structured re-entry program that occurs three months after scholars return to Indonesia. The scholars should be asked to prepare a 30-minute presentation based on their primary research followed by questions and answers. These could be posted on the USAID and ALPHA-I web sites. An additional component of the program should be a follow-on discussion to a similar session at the Pre-Academic Training (PAT) Leadership Workshop that guides the new alumni to examine how their ideas about leadership changed, what they learned, and how they are trying to apply that learning on the job. The workshop will be followed by webinars and mailings. IIEF and ALPHA-I are the recommended parties responsible for implementing this recommendation. Finding 46 ALPHA-I is a relatively new organization with great potential to serve PRESTASI alumni and the broader community. It has already sponsored valuable activities that give alumni volunteer opportunities that develop their network of like-minded colleagues. Alumni are seeking ways to share their newly acquired expertise. Interviews suggest that they are eager to “give back.” Conclusion Given the nature of the community building activities already hosted by ALPHA I, working with Prestasi Junior would provide relevant volunteer opportunities and reinforce the “giving back” ethic that is so essential to development. Junior Achievement is an NGO founded in 1919 in the United States, and has grown into a global network. Its Mission is to inspire and prepare young people to succeed in a global economy. One of the current scholars interviewed at Michigan State University was, when interviewed, considering an internship with Junior Achievement in East Lansing during the coming summer. Recommendation 21 Encourage ALPHA I to develop a partnership with Prestasi Junior. This Indonesian NGO is part of the worldwide network of Junior Achievement organizations that started in the United States to train young people in financial literacy and work readiness. The Team considered the possibility of working with other country alumni but decided that the cost in effort in trying to set up such a program outweighed the potential results. The Team believes it is more important to develop ties with other students and scholars who have studied in the US. And so, the possibility of such joint programming with Fulbright alumni is recommended. Furthermore a mentoring program should be implemented that pairs an alumnus with a newly selected scholar. IIEF and ALPHA-I are the recommended parties responsible for implementing this recommendation. 3.3. PROGRAM MANAGEMENT 3.3.1. IIEF’S Management of the Program Finding A thorough document review revealed a lack of consistency in the PPRs (Periodic Progress Reports submitted quarterly) and in the Semi-Annual Reports submitted by the prime contractor. Some, but not all, include information about site visits and a paragraph on each student’s progress. Information about workshops is often confusing: sometimes the same workshop is referred to using different names (perhaps as a result of the information being translated into English by different people). There is an inconsistency in the inclusion of lists of students and their status. Not all reports include the monthly student Newsletters produced by IIE. Not all information included in the PPRs is included in the Semi￾Annuals. The purpose of the Semi-Annual Reports is unclear. Conclusion Some useful information is not being reported. Some of the significant achievements and accomplishments of the program are not being captured and disseminated. Recommendation 22 There should be a standard template for reporting. Each report should, at minimum, include: a list of current scholars (both in the United States and in Indonesia), with subject and name of host institution, and report on site visits and a paragraph on each student’s progress. Workshops should be named consistently, with date and place. The monthly newsletters should be included with each report. The 47 requirement to submit Semi-Annual Reports should be removed, to be replaced by an Annual Report. USAID and IIEF are the recommended responsible parties for implementing this recommendation. Finding The evaluation team spent a great deal of time trying to acquire information about alumni, their period of study, and their accurate contact information. The way in which data are currently stored makes it difficult to conduct searches efficiently and to quickly sort the information into fields such as city of residence and type of employment. Data appear to be stored in various Excel spreadsheets each with various categories of information. Much of the contact information – email addresses, telephone numbers - is out of date. Conclusion There is a serious deficiency in the gathering, entry, storage, and updating of scholar data and a resultant limited capacity to retrieve information quickly. There is no integrated database management system that produces consistent and reliable results about the program. IIEF staff has assured the Evaluation Team that they are making good progress resolving these issues. Recommendation 23 A consultant (perhaps an IIE information technology specialist) should be engaged in to set up a database, and current staff should be trained on data input and generating reports. The fields should mirror those in the IIE database and should include: Name, Contact Information, Employer, Type of Organization, Region, Host Institution, Subject of Study, and Period of Study. A mechanism/standard operating procedures should be developed and implemented (with reasonable time/space allocated) for regular data quality assurance process (data checking and updating). IIEF and IIE are the recommended responsible parties for implementing this recommendation. Finding The previous financial reporting system used by IIEF, whereby expenditures are reported by contract line item numbers, was cumbersome and made it difficult to extract clear, simple, and usable information from the financial reports. It is understood that a new financial system for management and reporting has now been installed. Conclusion The previous system of reporting by CLINs (in addition to reporting by DOTs) added a layer of complexity to the process. This is but one factor restricting the ability to extract simple information from the reported expenditures. Recommendation 24 The Team recommends that the newly installed system of reporting be reviewed regularly by USAID and IIEF to ensure that a simpler system, capable of providing financial data upon request, is operating effectively. Finding IIEF currently bids out to contractors the implementation of the PAT training activities on an annual basis. This is not required by USAID regulations. Since ALPHA-I is funded through a three-year mechanism, a three-year subcontract – with a built-in cancellation clause along with the flexibility to renegotiate the precise Scope of Work each year – could reduce the costs associated with an annual solicitation while still ensuring the program offered is responsive to the responsible party’s concerns. 48 “We want them to think of us as their family.” IIE program officer “I feel like they (IIE) are my family here.” Current scholar “You welcomed a stranger and sent home a friend.” Scholar from Yogyakarta Conclusion The current process of bidding out vendor sub-contracts annually may result in increased expenditures of time and of money for both IIEF and the vendors. Recommendation 25 The procurement of the sub-contracted PAT activities (English language instruction and leadership training) should be conducted every three years, with contracts including cancellation/modification clauses awarded for this period. Finding Meetings the team held with GOI Ministries indicated that there has been little communication between PRESTASI and either the Coordinating Ministry for Human Development and Culture – previously the Coordinating Ministry for People’s Welfare (the PRESTASI-II Implementing Ministry) - or BAPPENAS. One of the representatives from the Coordinating Ministry stated that the most recent communication that they had with PRESTASI was at least two years ago. He also said that his Ministry “had been promised 20 PRESTASI scholarships.” (There had plainly been miscommunication or misunderstanding). He said that the Ministry had not received reports, either written or in person, from PRESTASI-II (and had not been consulted about PRESTASI-III) and requested better communication in the future. Although there is no formal relationship between PRESTASI-II and BAPPENAS, it is worth noting that the BAPPENAS’ head of planning, education, and training center also described a similar lack of communication, lack of knowledge about PRESTASI-II, and lack of collaboration (in contrast with the very close collaboration with Australian Awards, whose representatives “visit us regularly”). On the same note, the director of LPDP said he was not very familiar with the PRESTASI-II program. Conclusion There is a lack of communication, consultation, and coordination between GOI ministries and the PRESTASI program. Recommendation 26 IIEF and a USAID representative should pay a courtesy visit to each of these Ministries at least once a year. In addition, IIEF should send the Ministries the PPRs and copies of IIE’s monthly PRESTASI student newsletter, which provides informative snapshots of the program and its successes. USAID and IIEF are the recommended responsible parties for implementing this recommendation. Finding Discussions with Indonesian ministries (Coordinating Ministry, BAPPENAS) and with other scholarship￾funding programs (Australia Awards, British Council) found that there is no formal mechanism for all scholarship donors to meet, to compare outreach and selection policies, and to share lessons learned. The Australian scholarship team previously hosted a valuable annual Roundtable, but no longer does so. A representative from LPDP described these roundtables: “We are partners. We complement each other. We should sit together." Conclusion There is a perceived need for an informal periodic meeting among all those organizations managing scholarship programs for an exchange of information and best practices. 49 “IIE staff are very supportive and attentive. They regularly communicate, conduct campus visits, and even a visit to my apartment. They always respond my email, answering my questions and attending to my requests very well. When I was sick before returning home, they are also helpful in arranging my flight schedule that needed to be changed. IIE also often communicate with the International Affairs officer in my university. Scholar from Bandung We want them to think of us as their family.” - IIE Program Officer Recommendation 27 The PRESTASI program takes the initiative in this effort and coordinates the first and subsequent meetings. Conversely, the meeting could be held on a rotating basis, with each scholarship sponsor hosting and chairing in turn. USAID, IIE and IIEF are the recommended responsible parties for implementing this recommendation. 3.3.2 Participant Interactions with IIEF Finding Two-thirds (66%) of survey respondents were “Very Satisfied” with the level of support from IIEF, and 32% were “Satisfied.” Almost all (92%) respondents said they were given the cell phone numbers of key staff. Most (96%) said IIEF was Responsive to their concerns, and only 11% reported that there were occasions when they did not receive the support they were expecting. In the case where they perceived that did not receive support, it was more related to consultation about university placement and the transparency of the placement process. One current student said, “IIEF did not give me clear advice regarding how to select the best school for me.” Another said, “I wanted to know about the US community college system, but IIEF did know anything about that.” Another: “IIEF doesn’t know about studying in the US. They need to understand the students’ psychology and the stress involved in studying away from home.” One complained that IIEF staff “are not in touch with campus advisors, I don’t think any of them had actually studied in the US.” Of the scholars who were studying in-country, 86% said they received their allowances in a timely manner (Questions 43 and 44). Alumni and ICTs interviewed stated that IIEF is proactive in asking what scholars need. Conclusion Overall scholars were satisfied with the support they received from IIEF. The support could be improved in the area of consultation during university placement. Finding From discussions with IIEF, the Team found that no one in the organization working on PRESTASI-II has first-hand experience studying in the United States. Scholars and alumni reported in interviews that IIEF lacked knowledge regarding certain aspects of the U.S. educational system. Conclusion The ability of IIEF to provide educational guidance during the whole outreach, application, selection, and PAT activities should be strengthened. Recommendation 26 Given the pivotal role that IIEF plays in program administration and the fact that none of the current staff has studied in the United States, the program should, in collaboration with the Cultural Affairs Officer, develop a three week “orientation to higher education in the United States” single country project. Selected IIEF staff members would have a valuable professional development opportunity under the auspices of the U.S. Department of State International Visitor Leadership Program. It would equip them to play their respective roles more effectively. A staff exchange program, through which programming staff from IIE and IIEF exchanged jobs for a short (perhaps two weeks) period should also be considered. It is understood that the Chief 50 of Party and the international training specialist will be attending the 2015 NAFSA annual conference. The Team also believes that deeper exposure to U.S. campuses will give IIE and IIEF staff an enhanced understanding of the range, complexities, and qualities of the U.S. system of higher education. IIEF, IIE and USAID are the recommended responsible parties for implementing this recommendation. 3.3.3. IIE's Management of the Project Finding Every request for information – programmatic or financial – that was made by the Evaluation Team received an immediate response. This seemed illustrative of the extremely efficient and professional manner in which the program is managed by IIE. Their database is current and well-maintained and relevant data can be immediately accessed. Clear procedures are in place, and all staff members interviewed had a very clear idea of their role. Student files were reviewed and found to be in order, and all other documentation was clearly organized. Financial data was readily available. Conclusion IIE, in both its Washington, D.C. and New York office are managing the program extremely efficiently. 3.3.4. Participant Interaction with IIE Findings All of the survey respondents reported were “Very Satisfied” (71%) or Satisfied (29%) with the level of support from IIE. All but one reported that they received the staff cell numbers. One hundred percent said that IIE had been responsive to their requests. One hundred percent had received their allowances in a timely manner. Only four percent of respondents said there were occasions when they had not received the support they were expecting. Interviews confirmed these findings and interviewers were impressed with the uniform high level of satisfaction with IIE. Both survey responses and interviews with current scholars and alumni confirmed that scholars have had relatively little interaction with the international student offices on campus. Eighty-nine percent said they interact with that office only monthly. Following are a few more very positive comments about IIE from current and past students: “I really appreciated the IIE campus visit. The IIE person met with me and my faculty advisor and we talked about everything.” “All my questions to IIE were answered. I asked my colleagues and they all said the same thing.” “IIE helps us step by step. Every month they check in with me.” I feel I am supported by IIE. It is good to know program administrators personally. They are very encouraging.” “IIE is very responsive. When I requested a tutor, they responded immediately.” Faculty advisors were equally positive: “I wish all program officers paid that much attention to their charges. When can I get my next student from IIE?” “I have more interaction with IIE than with any other program agency.” An international student advisor echoed these sentiments: “IIE is great about keeping me in the loop as well as building relationships with academic departments.” Conclusion IIE is the agent to whom scholars turn with their problems and concerns. The scholars are extremely satisfied with the support they are receiving. Illustrative quotations follow:  “I feel I was very supported by IIE. It is good to know the program administrators personally.”  “IIE is like our family here.” 51  “I had a problem with registration. Lindsay told me what to do. She solved the problem on the same day.” 3.3.5. Cost Savings Findings Both IIE and IIEF have in place, cost-saving and cost-sharing mechanisms – particularly regarding U.S. university tuition and fees (other scholar costs – MMA, computer and book allowances, return baggage allowance etc. – are all set by U.S. ADS253 regulations). IIE, with its strong connections and relationships with U.S. universities, which have been built up over more than seven decades, is able to garner significant cost-sharing in the form of tuition waivers, tuition reduction, and in-state tuition. Of the four scholars currently enrolled at MSU, two were receiving a cost waiver for the full tuition. Of the seven attending Western Michigan University, all were paying lower in-state tuition rates (both of these discounts meet or exceed the U.S. Automated Directives System (ADS) regulations’ recommended target of 25% cost sharing). The staffing size in both the IIEF and the IIE offices is reasonable, based on the size of the program and standard practices in placement organizations. There were no obvious signs of extravagance. Based upon a review of the project budget and the expenditures to date, the current percentage of administrative support costs vs. participant costs is between 36% (actual to date) and 44% (budgeted). This ratio is within the standard range for a project of this complexity. Conclusion Based upon observation of the offices and a review of the financial reports provided by IIEF, appropriate cost-saving and cost-sharing mechanisms are utilized. 3.4 EXTERNAL FACTORS Finding In some cases the opportunity for the returned scholars to implement change back in their organizations and try out new practices is limited by their inability to assume roles with greater responsibility. This is particularly marked in the case of employees in the private sector (non-PNS). During interviews several alumni of the program expressed their frustration at the fact that they were still hourly-paid employees with little chance to have an influence on policy and practices within their organization Conclusion Without discriminating against non-PNS employees, PRESTASI should consider whether their efforts would be better directed towards employees in the NGO and private sector who – like the non-PNS employees – will not receive their basic salary, but who will more often have the opportunity to have an impact within their organizations. Recommendation 29 Broaden the outreach of the program to attract more applicants from NGOs and private businesses. IIEF is the recommended responsible party for implementing this recommendation. Finding The current Indonesian government is putting a great emphasis on improving education and adjusting its national budget to reflect that fact. This is a hospitable context in which to manage an educational exchange program. However, the United States is no longer the only popular destination for Indonesian 52 students and scholars seeking higher education. In the mid-1980s, there were 20,000 USAID participants studying in the United States annually. From 1997-98, there were more than13,000 Indonesians studying at U.S. institutions of higher education. In subsequent years, these levels have declined significantly, although there have been some recent gains. For exact figures please see Annex IV. BAPPENAS statistics show that the United States is now the fourth destination of choice (after the United Kingdom, Australia, and The Netherlands) for scholars in the World Bank-funded Scholarships Program for Strengthening the Reforming Institutions program – despite the fact that studying in Australia and Europe is more expensive than in the United States. Conclusion With an increasing number of higher education scholarships now available – through LPDP, BAPPENAS, DIKTI, various ministries, the Australians, the British, the Dutch and others – talented Indonesian students are now able to engage actively in “scholarship shopping.” Despite its decades-long history of providing high quality programs in higher education, the United States – for various reasons (the distance to be traveled, perceptions following 9/11) – continues to face challenges when it comes to attracting high quality applicants and potential future leaders. 53 ANNEXES ANNEX I: ADDITIONAL DETAILS ON EVALUATION METHODS KEY EVALUATION CONSIDERATIONS The IBTCI Evaluation Team conducted its research aware of these key considerations: 1. The need for clear communication between our team, IIE/IIEF and the training participants, to avoid any confusion or misunderstandings. 2. The importance of engendering an understanding by the scholars of the role of evaluators in relation to IIE and IIEF, the USG, and the students’ host institutions. 3. The need for discretion among our team members when discussing PRESTASI-II with all parties – all key informants and particularly the GOI and USAID staff. 4. In all interviews and discussions with respondents, the team clarified that this evaluation relates solely to PRESTASI-II, (and not to PRESTASI-III or to earlier scholarship programs). 5. It is recognized that although access to the Internet in Indonesia is increasing exponentially, only about 74.6 million people (29% of the population) currently have access. Online access varies across the country, with the greatest access in the western parts of Indonesia and with more limited access in the eastern provinces such as Papua and West Papua. In order to mitigate this issue of access the team used in person and telephone interviews to gather data as appropriate. 6. There is always the danger of “courtesy bias” -- culturally weighed responses from respondents, who do not wish to appear ungrateful, or to give offense. By stressing the anonymity of this survey, by gaining the trust of the respondents, by using highly experienced interviewers, and by posing questions in a variety of ways, the team believes that valid responses have been elicited. 7. In an evaluation of this type there is always an issue of translation distortion, arising from differences in the meanings of words, from differences in syntactical contexts, and from differences in the cultural perceptions of the respondents. In order to mitigate these issues related to translation, the team included two experienced bilingual interviewers who will be continually conferring to ensure consistency in terminology and annotation in the interviews, and in the analyses and interpretations of the data. 54 ANNEX II: DATA COLLECTION INSTRUMENTS SURVEY-MONKEY QUESTIONNAIRE Questionnaire for PRESTASI-II Scholars: “Dear PRESTASI-II Scholar, USAID has hired IBTCI, a private firm specializing in evaluation, to conduct a mid-term evaluation of the PRESTASI-II Program. As a current or past scholar your input will be invaluable. Please note we will be seeking to identify trends and patterns. The responses will be aggregated. No response or comment will be attributed to a specific individual so your answers are confidential. We need your help. By completing this questionnaire and sharing your reactions to the program and describing its impact, you will not only greatly benefit future scholars participating in PRESTASI-II but also those participating in any USAID scholarship program. We also believe that you will benefit as well from reflecting on your experience and distilling lessons you learned from it. We thank you in advance for your candid and thoughtful responses. Thank you for your cooperation. Colin Davies, Team Leader The Evaluation Team” 55 Demographic Questions 1. Name: ___________ 2. Sex: o Female o Male 3. Age: o 16-24 o 25-34 o 35-44 o 45-54 o 55+ 4. Degree Program: __________ 5. Area of Study/Major: ___________ 6. At which university did you study? ______________ 7. On what date did you begin your degree program? ____________ 8. On what date did you complete your degree program? ______________ Background 9. Where did you go to elementary school? ______________ 10. Where did you go to high school? ______________ 11. Where did you go to university? ______________ 12. What motivated you to apply for the PRESTASI-II program?__________________ 13. How did you learn about this program? o I saw an advertisement o I attended a presentation o I read about it in a newspaper/on-line o My supervisor recommended me o A workplace committee selected me o Others. Please describe ______________ 14. In general, how prepared were you for your educational experience? o Well-prepared o Prepared o Not prepared Overall Rating 15. In general, please rate your overall experience as a PRESTASI-II Scholar. o Excellent o Very good o Satisfactory o Fair o Poor Questions on the Pre-Master’s Program Experience 16. Pre-departure information and logistical arrangements (including medical examinations, visa application, USAID regulations, travel). Please describe your level of satisfaction. o Extremely satisfied o Very satisfied o Satisfied 56 o Somewhat satisfied o Not satisfied o What recommendations do you have to improve the pre-departure information and logistical arrangements?____________ 17. Pre-departure orientation. Please describe your level of satisfaction. o Very satisfied o Satisfied o Not satisfied o What recommendations do you have to improve the pre-departure orientation?____________ 18. Which topics would you like to have explained in greater detail? o Health insurance o Allowances o Housing o Learning methodologies o The participant-stakeholder compact o US culture and customs o US employment regulations o US visa regulations o Culture and adaptation strategies 19. Did you take a course in English at Lembaga Bahasa International-Universitas Indonesia? o Yes o No o If yes, length of course: ______________ 20. If yes, how useful was it? o Very useful o Useful o Not useful 21. Please provide a specific example of a skill(s) you acquired on the course that you have used in your program of study?________________ 22. Did you find the course too long, about right or too short? o Too long o Too short o About right 23. How could the course be improved? ______________ 24. Did you take a course in data analysis and statistics at Almajaya University? o Yes o No 25. If yes, how useful was it? o Very useful o Useful o Not useful 26. Please provide a specific example of a skill(s) you acquired on the course that you have used in your program of study?________________ 27. Did you find the course too long, about right or too short? o Too long o Too short o About right 57 28. How could the course be improved? ______________ 29. Did you take a course in leadership training at PT Daya Insani/Universitas? o Yes o No 30. If yes, how useful was it? o Very useful o Useful o Not useful 31. Please provide a specific example of a skill(s) you acquired on the course that you have used in your program of study?________________ 32. Did you find the course too long, about right or too short? o Too long o Too short o About right 33. How could the course be improved? ______________ Questions on the US Educational Experience 34. What is your level of satisfaction with your educational experience? o Very satisfied o Satisfied o Not satisfied 35. What are your recommendations for improvement?____________ 36. Thinking about your course and institution selection, how involved were you in the process? ______________ 37. Would you prefer to have been more involved or less involved? o More o Less 38. Were your wishes, suggestions, and requests listened to? o Yes o No o Please explain____________________ Questions on Participant Interactions with IIEF: 39. Are/were you satisfied with the level of support from IIEF? o Very satisfied o Satisfied o Not satisfied 40. Were you given the staff cell phone numbers? o Yes o No 41. Are/were they responsive to your requests? o Yes o No o Please provide an example____________ 42. Are/were there any occasions when you did not receive the support you were expecting? o Yes o No o If yes, please describe this occasion(s)___________ 58 43. Are/were you studying in Indonesia? o Yes o No 44. If yes, did you receive your allowances in a timely manner? o Yes o No 45. (Linking question for question logic) Are/were you studying in the US? o Yes o No Questions for Participants Studying in the United States: 46. Did you attend a US arrival orientation? o Yes o No 47. If yes, how useful was it? o Very useful o Useful o Not useful 48. What topics would have liked explained in greater detail?____________ 49. Are/were you satisfied with the level of support from IIE? o Very satisfied o Satisfied o Not satisfied 50. Were you given the staff cell phone numbers? o Yes o No 51. Are/were IIE responsive to your requests? o Yes o No o Please provide an example(s)____________ 52. Do/did you receive your allowances in a timely manner? o Yes o No 53. Are/were there any occasions when you did not receive the support you were expecting? o Yes o No o Please describe this occasion(s)________ 54. Do/did you think you were sufficiently prepared for US teaching and studying methods? o Yes o No 55. Please describe any challenges you encountered adjusting to the US educational system: _____________ 56. To what extent has/was your level of English language proficiency has been a barrier to your getting the most out of your program? o A great deal o Sometimes o Very little 57. Describe the level of support the international students’ office provides/provided? 59 o A lot o Some o Not much 58. How often would you say you interact(ed) with them? o Almost every day o Once a week o Once a month 59. Have there been or were there occasions when the international student office failed to help you? o Yes o No o If yes, please describe the occasion(s)_____________ 60. Please describe the quality of your university housing? o Excellent o Very good o Satisfactory o Fair o Poor 61. Is/was it clear to you which issues are handled by IIE and which by the international students office? o Yes o No 62. Are/were you sufficiently prepared for your return home? o Yes o No Questions on the Campus Experience: 63. How well do/did your university placement match your academic needs and expectations? o A great deal o Some o Not a lot o Please comment______________ 64. Please describe the quality of your academic coursework: ______________ 65. Please talk with us about the quality of the overall academic experience: ______________ 66. How would you rate the support you receive from university faculty? o Excellent o Very good o Satisfactory o Fair o Poor 67. Please describe your accommodations. o Dormitory o Apartment on campus o Apartment off campus o Other______________ 68. Did you have (a) roommate(s)? o Yes o No 69. If yes, what is their nationality? o Indonesian 60 o Other International Student o US Student 61 How do/did you rate the meal plan? o Excellent o Very good o Satisfactory o Fair o Poor 70. How would you describe your interactions with your fellow students? o Excellent o Very good o Satisfactory o Fair o Poor 71. How would you rate the quality of the education you are receiving/received? o Excellent o Very good o Satisfactory o Fair o Poor 72. Did you participate in cultural and recreational activities or clubs at your university? o Yes o No o If yes, please provide some examples of these activities________________ 73. Were you involved in any community or volunteer activities? o Yes o No o If yes, please provide some examples of these activities________________ Overall: 74. How do/did you anticipate using your new skills and knowledge upon your return?_______________ 75. If you are/were returning to work at the same organization, what changes do/did you want to introduce as a result of this learning experience?______________ 76. How do/did you expect your colleagues to react when you return with a new degree and new skills?______________ 77. How do/did you expect your supervisor to react when you return with a new degree and new skills?______________ 78. Do/did you think this is the best program for your needs/your organization’s needs? o Yes o No o Please explain______________ 79. How useful do/did you think the study will be for your future career? o Very useful o Useful o Not useful 80. Have/did your career plans changed as a result of this educational experience? ____________ 81. What are your recommendations regarding ways to increase the impact of this educational experience on your work and career?______________ 62 82. Please list three events or aspects of your educational experience that are/were the most enjoyable, memorable, or useful________________ 83. What advice would you give to future students on this program?____________ 84. (Linking question for question logic)Please choose one of the options below for further questions o I’m a female participant o I’m a male participant Questions for FEMALE PRESTASI-II Students: 85. What issues arose during the program and during your study that affected you as a woman that might not have risen for male participants?______________ 86. How has being a female affected your position within your organization?____________ 87. How has your participation in the program affected how people, including your employer, view you?______________ 88. (Linking question for question logic)Have you finished your study or are you still studying? o I have finished my study o I’m still studying 89. What was your job before your study? Title______________ Organization______________ 90. Did you return to that job? o Yes o No Overall Rating 91. In general, please rate your overall experience as a PRESTASI-II Scholar. o Excellent o Very good o Satisfactory o Fair o Poor Questions on the Program of Study and Its Impact: 92. What new knowledge and/or skills did you acquire during your program of study? You may choose more than one. o Technical knowledge in your field o Organization and management o Research skills and techniques o Becoming self-reliant o Computer skills o Teamwork o Negotiating with colleagues o Strategic planning o Critical thinking o Time management o Public speaking o English language skills o Other (describe) ______________ 63 93. Have you been able to apply your new knowledge and skills in your job? o Yes o No o If yes, which skills are the most valuable?______________ 94. Have your work responsibilities changed since you received your degree? o Yes o No 95. Did you receive any promotions or special assignments? o Yes o No o If yes, after how long did you get promotion upon return to your work?______________ 96. Do you think your program of study has affected your performance at work? o Yes o No o If yes, why? If no, why not? Please provide an example______________ 97. Do you think your leadership skills have improved as a result of your program of study? o Yes o No 98. Have you been able to apply these new leadership skills on the job? o Yes o No o If yes, why? If no, why not? Please provide an example______________ 99. What specific contributions have you made to your workplace as a result of your study? You may choose more than one response. Of your responses, please rank them numerically by importance, with #1 being the most important contribution. o I improved existing training programs o I improved management systems and/or procedures o I applied new methodologies in carrying out my work o I contributed to my organization’s expansion o I mentored my colleagues o I assumed a leadership role in a professional organization o Other. Please describe____________ 100. Within what period of time were you able to apply your new skills and knowledge after your return? 7 o Immediately o Within a month o Within 6 months o Within a year o Longer 101. Did you experience any difficulties or constraints in applying your knowledge and/or skills in your workplace? o Yes o No If yes, what difficulties did you face? You may select more than one response. 10 o My study was not applicable to my work____ 64 o I faced indifference by my supervisors____ o My co-workers were not interested in accepting new methods____ o Poor management of human resources____ o Weak communication____ o External interference____ o Lack of financial resources____ o Lack of incentives____ o Other ____ 102. Has your degree strengthened your job performance? o Yes o No o If yes, why? If no, why not? Please explain______________ 103. Have you been able to share your new skills and knowledge with others? o Yes o No 104. If yes, did you share your knowledge and/or skills with individuals: o within your organization o with those outside your organization o both o Please explain_____________ 105. In what ways have you contributed to your organization since you completed your degree and to what extent do your attribute these contributions to your program of study?______________ 106. What recognition have you received since you completed your degree and what recognition would you have liked to receive? ______________ 107. How interested and supportive has your employer been since you completed your degree? o Very supportive o Generally supportive o Not supportive 108. Are there other USAID scholars at your workplace? o Yes o No 109. If yes, was it helpful to have colleagues who had studied under a USAID program? o Very helpful o Generally helpful o Not helpful 110. Has your educational experience in the United States and Indonesia affected your professional relationships in your workplace? o Yes o No o If yes, why? If no, why not? Please explain____________ 111. Has your educational experience in the United States and Indonesia affected your professional relationships outside your workplace? o Yes o No o If yes, why? If no, why not? Please explain____________ 112. What is your ideal job in 5 years?____________ o To what extent are you presently on your chosen/preferred career path?______________ 65 o Where do you see yourself in 5 years?______________ 113. What impact has your PRESTASI-II educational experience had on your career? 6 o Very positive o Positive o No real impact o Negative o Please explain 114. What recommendations do you have for ways to improve the impact of your educational experience on your work and career?____________ 115. Do you still communicate with people you met on your program? o Yes o No 116. If yes, how often are you in contact with them? o Several times a week o Once a month o Every few months 117. If yes, has this been beneficial for you and your career? o Yes o No o If yes, why? If no, why not? Please explain ____________ 118. What ALPHA-I activities (workshops, meetings) have you attended? o Activity One: ______________ o Activity Two: ______________ o Activity Three: ______________ 119. For activity one, please give rate: o Extremely Useful o Very useful o Useful o Somewhat useful o Not Useful o Please explain:______________ 120. For activity two, please give rate: o Extremely Useful o Very useful o Useful o Somewhat useful o Not Useful o Please explain:______________ 121. Activity Three___________ o Extremely Useful o Very useful o Useful o Somewhat useful o Not Useful o Please explain:______________ 122. Are there any other activities for alumni you would like to attend?_________________ 123. (Linking question for question logic)Are/were you going to study in the United States? 66 o Yes o No 67 Questions for participants going to US 124. Some people who have studied in the United States return home with views that are different from those of their friends and colleagues. Do you think your views have changed as a result of your US experience? o Yes o No o If yes, please describe: ______________ 125. Do you talk to others about the United States and express your views? o Yes o No 126. What did you learn about the United States that you had not known before you studied there? ____________ 127. Did you have a mentor to help you through the re-entry process? o Yes o No o If not, do you think a mentor could have been helpful to you? ● Yes ● No Questions about the PRESTASI-II Program: 128. Overall, what parts of the PRESTASI-II program are most beneficial? ________________ 129. Which parts are the least useful? ______________ 130. Would you recommend this program to your colleagues? o Yes o No 131. Do you have any recommendations for improving the program?___________________ 68 QUESTIONNAIRES FOR DIFFERENT STAKEHOLDERS Key Informant Interview Questions for Faculty and International Student Advisors 1. Describe the preparation of the PRESTASI-II scholars to study at your university: a. Language capability? b. Academic background? c. Adaptability to the US system? 2. Are the PRESTASI-II scholars relating well to administrative staff, faculty, and fellow students? a. How are they adapting to US society and a US campus? Please provide an example(s). 3. Describe the content and quality of your relationship with IIE and IIEF. a. As far as you are aware, has all necessary support (including payment of stipend and allowances) been provided to students without delay? Please provide an example(s), if applicable. 4. What type of support do the scholars need? a. Do you believe this support is being provided by IIE? By IIEF? And, by your office? b. It is clear to the scholars to whom they should turn for support in cases of need? c. Are there any areas of scholar concern that you think are not being adequately addressed? 5. Has there been any aspect of the PRESTASI-II scholar’s program that has been particularly successful, or, alternatively, caused you to worry? 6. How are the scholars performing in absolute terms, and relative to other students? 7. To what extent do you think this program of study has contributed to the knowledge and skills of these scholars? 8. What external factors have affected these scholars’ academic progress? 9. Have you encouraged the scholars to take part in supplementary programs, such as leadership training, gender awareness programs, community engagement programs, and re-entry programs? Have the students shown interest in these activities? 10. Have you encouraged the scholars to take part in volunteer and community activities? Have they shown interest? 11. Have any of the scholars expressed concern about their return home, and how they will re-adjust to their workplace? How have you responded to these concerns? 12. Please state how satisfied you are – and what improvements could be made – in the following areas: a. Selection and quality of students? b. Course selection? c. Student preparedness? d. Support for students on an on-going and an emergency basis? e. Monitoring of students? f. Preparation for re-entry into Indonesian society? 13. Are there any particular gender issues that you have noticed? Please describe these in more detail. 14. What is your overall impression of the management of the PRESTASI-II scholarship program? 69 Key Informant Interview and Focus Group Discussion Questions for USAID Officials 1. Please describe your role at USAID. 2. Please describe your specific involvement in the PRESTASI-II program. 3. Describe the objectives of the PRESTASI-II program. 4. Do you consider PRESTASI-II to be an organizational strengthening program, or a ‘best and brightest’ program? a. If a ‘best and brightest’ program, what do you believe can/will bring about greater organizational transformation? 5. How does the PRESTASI-II program fit into the overall Mission Plan? CDCS? 6. In what ways is this program complementary to or duplicative of Fulbright and/or other State Department programs? 7. How much coordination is there with other scholarship programs – e.g. the Australian scholarship programs? 8. How is the program publicized? 9. What type of outreach is there? a. To women and people with disabilities? b. Do you have any suggestions for outreach and/or program improvement? 10. What do you consider the relative merits of targeting the PRESTASI-II program to either recent graduates or to mid-career professionals? Let’s take each one at a time. 11. Are you satisfied that the PRESTASI-II program is reaching underserved populations? a. How do you define ‘underserved populations’ in the context of PRESTASI-II? b. What more could be done to attract a wide range of participants? 12. What changes – if any – would you make to the selection process? 13. Is the selection process weighted more towards technical expertise or leadership potential? Or, other factors? 14. What type of needs assessment is carried out at the scholar’s place of work? 15. For each program, has a training need been identified at the sectoral, institutional, or individual level? Please describe this training needs assessment process. 16. Are clearly defined objectives written for each scholar’s program of study? 17. How involved are you in the selection of host universities? a. Do you prefer students to be sent in clusters or to be spread throughout the country? 18. Are you satisfied with the pre-departure program in-country? a. To what extent are you involved? 19. Have you received any feedback regarding the US orientation program? 20. Overall, do you think the PRESTASI-II scholars are prepared for their educational experience? 21. How satisfied are you with the placements that are made in Indonesia and in the United States? a. How much placement subject-matter expertise do the programming agencies possess? 22. Do you think that cost-containment measures are rigorously applied? 23. From the financial reports, and your observations, are resources being effectively managed? 24. Please describe the re-entry debriefing of returned participants? a. Do you believe this to be a sufficient process 25. What type(s) of recognition are PRESTASI-scholars given by USAID and by their employers? 26. To what extent do you try to maintain contact with returned participants and their employers? 27. What sort of attitudinal and job performance changes do you hope to see in returned participants? 28. Please share with us specific examples of changes have you seen. 29. Please describe the extent to which you have you noticed differences in the skills and attitudes of those who studied in the United States and those who studied in Indonesia under this program? 70 30. What factors – internal and external – have prevented these scholars from having more impact? 31. How active is the alumni association? What more could be done? 32. How involved are returned scholars in outreach, selection, pre-departure orientations, and mentoring of new PRESTASI-II scholars? Questions about Relations with IIEF: 33. Are you generally satisfied with IIEF’s management of the program? 34. What is your approach to managing/overseeing IIEF? 35. Are you satisfied with the reports you receive from IIEF, both narrative and financial? To what extent would you say you are satisfied? Please provide us an example(s). 36. How responsive do you find IIEF to your ad hoc questions and requests? 37. What suggestions do you have for improvement? Questions about Relations with IIE: 38. Are you generally satisfied with IIE’s management of the program? 39. What is your approach to managing/overseeing IIE? 40. Are you satisfied with the reports you receive from IIE, both narrative and financial? Please provide us an example(s). 41. How responsive do you find IIE to your ad hoc questions and requests? 42. What suggestions do you have for improvement? 71 Employer Key Informant Interview Questions Questions for Employers whose PRESTASI-II Alumni Worked at the organization Prior to the PRESTASI-II Scholarship Award: 1. Please describe your role in this organization. 2. Please describe your specific involvement in the PRESTASI-II program. 3. Please describe your specific involvement/relationship to your PRESTASI-II scholar(s)? 4. Did you play any role in the selection of scholars? Please describe this role. 5. Were you involved in the scholar’s choice of degree program? Please describe your involvement. 6. To what extent was the selection of scholar and degree program based upon a needs assessment within your organization? 7. What did you expect from the scholars following their degree completion? 8. How has the PRESTASI-II scholar’s performance changed since his/her return to work? a. Describe the extent to which you believe the knowledge and skills gained during their degree program has affected his/her job performance. b. What about their work unit and the organization as a whole? 9. Describe the extent to which you believe that PRESTASI-II involvement has helped or hindered the alum’s career advancement? Questions for all Employers (Relating to specific Alums): 7 What knowledge, skills and/or value does he/she now bring to his/her job? a. In addition to academic and technical knowledge, have you noticed whether the alum acquired any new skills or ways of working during his/her period of study? b. Prompts for Question 7a include: Discuss with us the PRESTASI-II alum’s organization and management; Research skills and techniques; Becoming self-reliant; Computer skills; Teamwork; Negotiating with colleagues; Strategic planning; Critical thinking; Time management; Teaching; Public speaking; English language skills; Others? 8 What knowledge, skills and/or value does he/she now bring to the work unit that you can link to their involvement in the PRESTASI-II program? 9 Describe the alum’s job performance in terms of performance, output, productivity, collegiality, leadership, ambition. 10 Have you noticed a specific change in terms of the scholar’s leadership skills? Please provide us an example(s). 11 How has the alum’s education affected his/her leadership or management opportunities within the organization? Please provide an example(s). 12 To what extent has he/she received promotion or been given new responsibilities? 13 How has he/she contributed to change in the overall organizational performance and culture? Please provide us a specific example(s) of this contribution. 14 In what ways has he/she shared the newly acquired skills and knowledge with colleagues? 15 To what extent do you attribute these contributions to your organization to the PRESTASI-II program? Questions for all Employers about the Program in General: 16 To what extent do you think that a Master’s degree scholarship program is the best and most relevant type of program for your employees? a. How might these programs be more relevant to your organization’s goals? 17 To what extent do you believe an organizational assessment should be conducted prior to aligning a PRESTASI-II scholarship degree program with your organizational needs? 72 18 How should returning participants’ skills be used? 19 What recommendations do you have regarding ways to improve the impact of the educational experience on the alumni’s work and career? a. On his/her work unit? b. On your organization? 20 What acknowledgement do you give returning participants? 21 Do you notice a difference between those who studied in the United States and in Indonesia? Please provide an example(s) of this difference(s). 22 Would you recommend the PRESTASI-II program to other employers? 23 What recommendations would you make to ensure that this scholarship program makes the maximum contribution to the employing organization? Key informant Interview Questions for Government of Indonesia Officials 1. How involved are you with the PRESTASI-II program? 2. What role do you play in: a. Outreach? b. Selection? c. Choice of educational institution? 3. How does the PRESTASI-II program fit into an overall government strategy of leadership training and institutional strengthening? 4. Is this program complementary or does it duplicate a. Other Indonesian government programs? b. US State Department programs? c. Australian scholarship programs? d. What about other scholarship programs? 5. Do you think the PRESTASI-II program has been successful in reaching underserved populations? a. Do you, in fact, think that this should be an objective of the program? b. What do you believe to be the priority ‘underserved populations’? 6. Which populations would you like to see more included? 7. How do you think the PRESTASI-II program has contributed to the performance and leadership skills of Indonesian professionals? 8. Please share some examples of contributions made by professionals trained under this program. 9. Have you noticed differences in the skills and attitudes of those who studied in the United States and those who studied in Indonesia and in other countries under this and other scholarship programs? Please describe these differences. 10. What factors – internal and external - limit the success of such a program? 11. How would you rate the overall benefit of this program to your country? 12. What recommendations do you have for improving the program? Key Informant Interview Questions for PRESTASI-II Implementers 1. Outreach (for IIEF). Outline your outreach policy. How are you taking advantage of different forms of social media? How successful do you think you have been in reaching underserved populations? To what do you attribute the decline in applications in the latest round? To what extent do you coordinate with other sponsored scholarship programs? What challenges have you experienced, and how have you overcome them? Some changes were made in the Application Form (fields of study changed; information about spouse/dependents/parents was 73 removed; physical impairment was added’ type or employer organization was added): why, and how have these changes affected the numbers and profile of applicants? 2. Selection and review (for IIEF). Outline your selection, review, and interview process, and the modifications you have made to it during the course of this program. How does this vary by geographic region? Also, please talk a little about the students’ choices of degree subjects, and any variance by region or gender. 3. Preparation for departure (for IIEF). Describe how and why you chose your current subcontractors for Pre-Academic Training? How satisfied are you (and the students) with these courses? What modifications have you proposed? Despite PAT, which types of students have been the hardest to get placed? Talk about the students’ experiences with the GRE. 4. Other pre-departure activities (for IIEF): visas, security clearances, medical examinations, flights. What changes have you made to your procedures during your management of PRESTASI I and II? What further modifications would you recommend? 5. Pre-departure, arrival, and reentry programming (for IIEF and IIE-DC): explain why the decision was made to change the location. What evaluations have been conducted of these programs? How satisfied are you with them, and what changes would you recommend? 6. US Placement (for IIE – New York). Describe any challenges you may have faced with placing Indonesian students? Do you take into account gender when making placements? What other factors do you consider? Does the student’s geographic location play a part in choice o institution? What would be your ideal number of PRESTASI-II students at any one institution? What type of cost-sharing/cost-containment have you been able to secure? 7. In-country Placement (for IIEF). What factors go into your choice of venue? 8. Student monitoring. For IIEF and IIE-DC: how often, and on what basis, do you conduct site visits? What are the most common issues regarding accommodation and meals? What are the most common academic problems faced by these students? How have you dealt with them? Describe the type and extent of interaction you have with faculty and students advisors? Talk a little about the newsletter, and also about efforts and successes in developing a networking ‘group identity’ among these students. For IIE-DC: what problems of cultural adaptation have you found? What strategies have you developed for dealing with them? How much do you encourage these students to become involved in social and volunteer/community activities? How interested are they in general? Have you noticed any particular differences in the students’ US education experience based on gender or geographic origin? 9. Alumni activities. For IIEF: Describe your relationship with ALPHA-1. How successful do you think they have been? Describe their management. What recommendations would you make? Which do you think have been their most successful activities? Are you satisfied with the amount of deliverables (reports, seminars, training modules, discussion groups, training or trainers, evaluations) they have produced? What future activities would you like to see? For IIE￾DC: to what extent are you involved in alumni activity planning? 10. Recommendations. For IIEF and IIE-DC: a Reentry Meeting (PPR#9) produced three recommendations: A larger conference allowance; greater use of professional memberships and international journals; US pre-study English language training. What is your response to these recommendations? 11. USAID compliance. For IIEF and IIE-DC and IIE-NY: How compliant do you consider your organization with regards to quarterly and semi-annual reporting; financial reporting; student and alumni tracking and reporting. 12. ADS 253 compliance: For IIEF and IIE-DC and IIE-NY: Are you familiar with the required file documentation and record keeping? How compliant do you consider your organization? 74 ANNEX III: SOURCES OF INFORMATION LIST OF DOCUMENTS REVIEWED PRESTASI Specific Materials ALPHA-I Brochure ALPHA-I Gender Based Violence Module (Modul untuk Petugas Kesehatan. Percakapan Pemberian Bantuan Untuk Penanganan Kekerasan Terhadap Perempuan. Kerjasama antara Yayasan Pulih dan ALPHA I) ALPHA-I Newsletter dated January 2015 ALPHA-I Presentation on Financial Literacy for Youth IIE The PRESTASI Newsletter #1,2,3,4,5,6,7,8 IIE PRESTASI Scholarship Participant Files IIEF Annual Work Plan Years 1,2,3 IIEF Outreach Plan IIEF Performance Monitoring Plan IIEF Pre-Departure Plan IIEF PRESTASI Scholarship Participant Files IIEF PRESTASI Scholarship Participant Files PRESTASI Alumni List PRESTASI-II Detailed Fiscal Report - Major Subcategories 08 August 2012 - 28 February 2015 PRESTASI-II Detailed Fiscal Report - Funding Earmark 08 August 2012 - 07 August 2017 PRESTASI-II Fiscal Report by CLIN 08 August 2012 - 28 February 2015 PRESTASI-II List of Participants in Training PRESTASI-II Periodic Progress Reports: ● PPR #1 (July - September 2012) ● PPR #2 (October - December 2012) ● PPR #3 (January - March 2013) ● PPR #4 (April - June 2013) ● PPR #5 (July - September 2013) ● PPR #6 (October - December 2013) ● PPR #7 (January - March 2013) ● PPR #8 (April - June 2014) ● PPR #9 (July - September 2014) ● PPR #10 (October – December 2014) ● SAPR #1 - #2 (August 2012 - June 2013) ● SAPR #3 (July - December 2013) ● SAPR #4 (January - June 2014 PRESTASI-II Pipeline Analysis 08 August 2012 - 28 February 2015 PRESTASI-II Scope of Work and Amendments USAID Indonesia Scholarship PRESTASI Program Application Form, versions 1 & 2 Joint Outreach Materials Cycle 1 ● Brochure USAID - English.100913.2 ● Brochure USAID - Indonesia.110913 75 ● Flyer 140213 ● Flyer FAQ Indo ● Panduan Pendaftaran / Registration Guidelines ● PRESTASI Application form 2013 Cycle 2 ● Flyer 14.8x21cm - Rev 20131210 (1)a ● Revised Panduan Pendaftaran/ Registration Guidelines ● PRESTASI Application Form ● USAID - Poster 29.7x42cm - Rev 20131217 Websites http://www.iie.org/en/Who-We-Are/News-and-Events/Press-Center/Press-Releases/2013/2013-01-07- USAID-Renews-Prestasi-IIEF-Contract; visited 26 February 2015 at 16.25 pm USAID (2014) Program to Extend Scholarships and Training to Achieve Sustainable Impacts Web 20 Feb 2015 www.prestasi-iief.org Website of ALPHA-I, “Asosiasi Alumni Program Beasiswa Amerika Serikat-Indonesia”, http://www.alpha￾i.or.id/; visited on 19 March 2015, 13.00 pm Other Documents Reviewed Books and Articles AMEX International Inc. (1995) The Development Impact of Participant Training. Washington, DC: AMEX Board for International Food and Agricultural Development (BIFAD) (2003) Renewing USAID Investment In Global Long-Term Training and Capacity Building In Agriculture and Rural Development. Washington, DC: BIFAD Buchori, Mochtar, Moser, Tom, and Sanders-Smith, Melanie (1994) Final Evaluation of the General Participant Training Project II 497-032.Washington, DC: Creative Associates International, Inc. Creative Associates International, Inc. (1993) An Integrated Methodological Framework for Enhancing and Evaluating the Development Impact of Training. Washington, DC: Creative Associates International, Inc. Doyle, Aloyae (1975) Project Appraisal Report: FY 69-Mid-FY 75 Indonesia General Participant Training (summary). Washington, DC: USAID. Ervin, Susan and Bower, Robert (1952) “Translation Problems in International Surveys” Public Opinion Quarterly, Volume 16, Number 6 Fischer Karin “Many Foreign Students Are Friendless in the US, Study Finds,” The Chronicle of Higher Education, June 14, 2012. Available online at http://chronicle.com/article/Many-Foreign-Students￾Find/132275/?cid=pm&utm_source=pm&utm_medium=en Gareis, Elisabeth, (2012) Intercultural Friendship: Effects of Home and Host Region, Journal of International and Intercultural Communication 76 Gilboy, Andrew C. (1998) Quick Reference to Results-Oriented Training: Best Practices Abridged for USAID Strategic Objective Team: Rosslyn, VA. Aguirre International Gilboy, Andrew C. (2002) Why Training Has Not Produced the Desired Results – and What to Do about It. Washington, DC: Associates for Global Change. Gilboy, Andrew et al. (2004) Generations of Quiet Progress, an Impact Assessment of ATLAS/AFGRAD, Volumes I, II and III. Washington, DC: Aguirre International Gilboy, Andrew et al. (2010) Agriculture Long-Term Training Assessment and Design Recommendation. Bethesda, MD: Aguirre Division of JBS International Gilboy, Andrew C., et al. (2001) Evaluation and Options for Development Training 2: Lessons Learned from an Ambitious Training Management Experiment and Options for Promoting Global Performance Improvement. Rosslyn, VA: Aguirre International Gillies, John et al. (1992) Training for Development: A Review of Experience. Washington, DC: Juarez and Associates, Inc., Management Systems International, Inc., Research Triangle Institute, Academy for Educational Development Hodson, Jeremy, et al. (1989) An Evaluation of USAID/Senegal Training Projects and Proposal for Five Year Training Program. Waltham, MA: Education Development Center, Inc. InterMedia (2009), Evaluation of the Youth Exchange and Study Program. Washington, DC: US Department of State International Cooperation Administration & Interdepartmental Coordinating Committee of the Indonesian Government (1959) An Evaluation of the Participant Training Program in Indonesia. Washington, DC: USAID. International Institute of Education (IIE) (2014) Open Doors, Report on International Educational Exchange. Washington, DC: International Institute of Education Keilson, Jerrold (2001) “Building Human Capacity through Training” SIT Occasional Papers Series, Issue no.2, Spring, pp.15-29. Brattleboro, VT: World Learning Kluger, Rona (2006) “Increasing Women’s Participation in International Scholarship Programs” IIE Research Report Number 27 Washington, DC: Institute of International Education: Washington, DC Mashburn, Robert, et al. (1990) Evaluation of General Participant Training. Washington, DC: TvT Associates McCloud, Pamela et al. (2007), Training Future Leaders: Assessment for a Development Leadership Initiative. Asia and the Near East Region. Arlington, VA: Development Associates Merrill and Burrola (2015) “Indonesia’s Mental Revolution” The Indonesian Journal of Leadership, Policy and World Affairs, Volume 5, Number 1, January – March, pp. 20-26 77 Moorhead Kennedy (1986) The Ayatollah in the Cathedral: Reflections of a Hostage p. 172. New York: Hill & Wang Pub. Morris, Robert and Susanne Morris (1992) “Making Their Way: Some Critical Variables Affecting Professional Reintegration of International Students into Their Countries and Organizations” NAFSA Working Paper #27, Arlington, VA: Management Training and Development Institution. OECD Economic Survey of Indonesia 2015 OECD/Asian Development Bank (2015), Education in Indonesia, Rising to the Challenge. OECD Publishing, Paris. Skelton, Ann et al. (2009) Malawi Participant Training Program Outcomes Study. Bethesda, MD: Aguirre Division of JBS International US Higher Education Leaders Mission to Indonesia (2009) Report and Recommendations. Washington, DC: US-Indonesia Society Wein, Gerald and Auten, Sarah (2008) Evaluation of the Clinton and Master’s Degree and the Presidential Scholarship Programs. Bethesda, MD: Aguirre Division of JBS International USAID Documents USAID (2010), Performance and Monitoring & Evaluation TIPS USAID (2011), USAID Evaluation Policy USAID (2012), Alumni Association: A Guide for USAID Missions USAID ADS Chapter 203 Assessing and Learning, USAID ADS Chapter 252 Visa Compliance for Exchange, USAID ADS Chapter 253 Participant Training and Exchanges for Capacity Development USAID Bureau for Economic Growth, Education, and Environment, Conditions of Scholarship for US￾Based Activities USAID Evaluation Report Template USAID Indonesia Mission Order for Participant Training USAID Indonesia Stakeholder Compact for US Training Journals and Newsletters AMINEF – Fulbright Brochure 2015 – 2016 Study Year Australian Aid (03/2014) Newsletter. Building Brighter Futures Blog INDONESIA MENGGLOBAL British Council Flyer Websites A Glimpse of LPDP Scholarship, www.beasiswalpdp.org, visited on 20 February 2015 at 3.24 pm Website of Australian Awards Indonesia, http://www.australiaawardsindonesia.org/, visited on 25 March 2015 at 09:00 am Website of LPDP (Lembaga Pengelola Dana Pendidikan), http://www.lpdp.kemenkeu.go.id/, visited on 3 April 2015 at 9.00 am 78 Website of NESO (The Netherlands Education Support Office) – Indonesia, http://www.nesoindonesia.or.id/, visited on 30 March 2015 at 15.00 pm 93 List of SurveyMonkey Respondents Name Gender University Group 1. Bernie Endyarni Medise Female University of Nebraska Medical Center Group A - 2011 placed. Graduated 2. Ella Masita Female Eastern Connecticut State University Group A - 2011 placed. Graduated 3. Dwi Elfrida Martina Simanungkalit Female University of Massachusetts Group A - 2011 placed. Graduated 4. Redhi Setiadi Male University of Delaware Group A - 2011 placed. Graduated 5. Yossa Nainggolan Male Oregon State University Group A - 2011 placed. Graduated 6. Evi Fatimah Female West Chester University of Pennsylvania Group A - 2011 placed. Graduated 7. Vitri Widyaningsih Female Michigan State University Group A - 2011 placed. Graduated 8. Muhammad Wahyudi Male Michigan State University Group A - 2011 placed. Graduated 9. Nurhidayat B. Female Universitas Indonesia Group A - 2011 placed. Graduated 10. Dwi Adi Maryandi Male Saint Louis University Group A - 2011 placed. Graduated 11. Damayanti Sari Rohmaningtyas Female Florida Atlantic University Group A - 2011 placed. Graduated 12. Sarman Oktovianus Gultom Male University of Minnesota Group A - 2011 placed. Graduated 13. Ashari Cahyo Edi Male University of Delaware Group A - 2011 placed. Graduated 14. Riana A. Arief Female Colorado State University Group B - 2012 placed. Graduated 15. Marlina Meilani Simbolon Female Universitas Indonesia Group B - 2012 placed. Graduated 16. Jusly Adrianus Lakapu Female Universitas Indonesia Group B - 2012 placed. Graduated 17. Isnain Evilina Dewi Female Murray State University Group B - 2012 placed. Graduated 18. Nila Kusumawati Elison Female Georgia State University Group B - 2012 placed. Graduated 19. Trisno Ikhwanudin Male University of Minnesota Twin Cities Group B - 2012 placed. Graduated 20. Rina Nelly Jowei Female Michigan State University Group B - 2012 placed. Graduated 21. Henny Merizawati Female Central Michigan University Group B - 2012 placed. Graduated 22. Safarudin Male Universitas Indonesia Group B - 2012 placed. Graduated 94 23. Kristian Adi Putra Male The University of Arizona Group B - 2012 placed. Graduated 24. Riana Murianty Female Saint Michael's College Group B - 2012 placed. Graduated 25. Dirna Mayasari Female University of California Los Angeles Group B - 2012 placed. Graduated 26. Dewi Kartikawati Ningsih Female University of Nebraska Medical Center Group B - 2012 placed. Graduated 27. Sophia Kurniasari Purba Female Fordham University Group C - 2013 placed. Graduated 28. Erlina Mariana Rosada Sari Siregar Female Western Michigan University Group C - 2013 placed. Graduated 29. Auliya Ridwan Male Florida State University Group C - 2013 placed. Graduated 30. Agustin Indracahyani Female Brandeis University Group C - 2013 placed. Graduated 31. Angga Rachmansah Male State University of New York, Syracuse Group C - 2013 placed. Graduated 32. Ahmad Hafizh Adyas Male Scripps Institution of Oceanography, UC San Diego Group C - 2013 placed. Graduated 33. Esty Haryani Female Western Michigan University Group C - 2013 placed. Currently in training 34. Fajar Rochadi Male University of Illinois at Chicago (UIC) Group C - 2013 placed. Currently in training 35. Putri Pandarangga Female The University of Georgia Group C - 2013 placed. Currently in training 36. Aisah Putri Budiatri Female State University of New York at Albany Group C - 2013 placed. Currently in training 37. Fransiska Renita Anon Basundari Female Michigan State University Group C - 2013 placed. Currently in training 38. Stephanie Perdana Ayu Lawalu Female Michigan State University Group C - 2013 placed. Currently in training 39. Erly Kueain Female New Mexico State University Group C - 2013 placed. Currently in training 40. Testriono Male Northern Illinois University Group C - 2013 placed. Currently in training 41. Kadek Ridoi Rahayu Female Montclair State University Group C - 2013 placed. Currently in training 95 42. Rio Jati Kusuma Male University of Nebraska Lincoln Group C - 2013 placed. Currently in training 43. Miranti Triana Zulkifli Female Yale University Group C - 2013 placed. Currently in training 44. Heri Hermawan Male Yale University Group C - 2013 placed. Currently in training 45. Irma Hidayana Female Montclair State University Group C - 2013 placed. Currently in training 46. David Kuntel Male Troy University, Troy, Alabama Group C - 2013 placed. Currently in training 47. Emy Mariana Female Saint Louis University Group C - 2013 placed. Currently in training 48. Muhtar Ahmad Male Indiana University Group C - 2013 placed. Currently in training 49. Nugrohojati Male Tulane University Group C - 2013 placed. Currently in training 50. Wiranta Yudha Ginting Male Brandeis University Group C - 2013 placed. Currently in training 51. Gusti Ayu Fransiska Sri Rahajeng Kusuma Dewi Female State University of New York College of Environmental Science And Forestry Group C - 2013 placed. Currently in training 52. Mohammad Meidiansyah Male Portland State University Group C - 2013 placed. Currently in training 53. Reinardus Liborius Male Michigan State University Group C - 2014 placed. Currently in training 54. Arifatul Khorida Female Universitas Gadjah Mada Group C - 2014 placed. Currently in training 55. Bakhtiyar Salam Male Western Michigan University Group C - 2014 placed. Currently in training 56. Nurwasya Endang Suhendar Male Oregon State University Group C - 2014 placed. Currently in training 57. Diani Indah Rachmitasari Female Western Michigan University Group C - 2014 placed. Currently in training 58. Silvia Anastasia Landa Female Utah State University Group C - 2014 placed. Currently in 96 training 59. Cyrillus Raja Bhoja Male Western Michigan University Group C - 2014 placed. Currently in training 60. Endyka Erye Frety Female Universitas Brawijaya Malang Group C - 2014 placed. Currently in training 61. Bertimuliadi Male Baruch College, City University of New York Group C - 2014 placed. Currently in training 62. Qurratul Aini Female State University of New York, Buffalo Group C - 2014 placed. Currently in training 63. Maretha Dellarosa Female Ohio State University Group C - 2014 placed. Currently in training 64. Rudi Hermawan Male Georgia State University Group C - 2014 placed. Currently in training 65. Paulus Paramma Male Eastern Mennonite University Group C - 2014 placed. Currently in training 66. Wisnu Trianggono Male The University of North Carolina at Chapel Hill Group C - 2014 placed. Currently in training 67. Fitri Lapau Female University of Illinois at Urbana-Champaign Group C - 2014 placed. Currently in training 68. Mareta Pratiwi Female State University of New York, Buffalo Group C - 2014 placed. Currently in training 69. Dina Rafidiyah Abdul Mannan Female Pennsylvania State University Group C - 2014 placed. Currently in training 70. Wiesye Violent Pelupessy Female State University of New York, Syracuse Group C - 2014 placed. Currently in training 71. Hasiholan Tiroi Simorangkir Male California State University Los Angeles Group C - 2014 placed. Currently in training 72. Wiwik Mulyani Female Florida State University Group C - 2014 placed. Currently in training 73. Merly Aclin Nuasizta Klaas Female University of Wisconsin Madison Group C - 2014 placed. Currently in training 74. Catherine Setiawan Female Cornell University Group C - 2014 placed. Currently in training 97 75. Frans Judea Samosir Male The University of Georgia Group C - 2014 placed. Currently in training 76. Indah Sukmawati Manti Putri Female Universitas Indonesia Group C - 2014 placed. Currently in training 77. Arie Arizandi Kurnianto Male Universitas Airlangga Group C - 2014 placed. Currently in training 78. Ira Ryski Wahyuni Female Institut Teknologi Bandung Group C - 2015 placed. Currently in training 79. Samsriyaningsih Handayani Female University of Illinois at Chicago Other - Ph.D. 80. Akim Dharmawan Male Saint Louis University Other - Ph.D. 81. Andre Abraham Male Universitas Indonesia Other - Ph.D. Persons interviewed disaggregated by groups/institutions and gender Groups/ Organizations Men Women Total Current Scholars in the US 4 9 13 Current Scholars In-country 1 5 6 Alumni 13 9 22 Academic Advisors 6 5 11 Employers/Supervisors 4 2 6 International Students Advisors - 5 5 Implementing Agencies 3 9 12 United States Government 7 5 12 Government of Indonesia 3 2 5 Service Providers 4 1 5 Other Scholarship Sponsors 7 4 11 Other Interviewees 5 3 8 57 59 116 98 Respondents to online survey disaggregated by group and gender Group of respondents Male Female Total Group A - 2011 placed. Graduated 6 7 13 Group B - 2012 placed. Graduated 3 10 13 Group C - 2013 placed. Graduated 3 3 6 Group C - 2013 placed. Currently in training 9 11 20 Group C - 2014 placed. Currently in training 11 14 25 Group C - 2015 placed. Currently in training (In-country) 0 1 1 Other - Ph.D.. Currently in training 2 1 3 Total 34 47 81 99 ANNEX IV: STATISTICS - INDONESIAN STUDENTS STUDYING IN THE UNITED STATES Historical trends: The number of Indonesian students studying in the United States grew steadily in the 1980’s and 1990’s, peaking at 13,282 students in 1997/98. It then declined through 2006/07 and then experienced some fluctuations until leveling out in 2010/11 at just under 7,000. There has been slow but steady growth over the last three years, with the 2013/14 level reaching the highest number of Indonesian students in ten years. Year # of Students From Indonesia % Change from Previous Year # of US Study Abroad Students Going to Indonesia 2013/14 7,920 3.3% n/a 2012/13 7,670 7.6% 493 (up 33.6%) 2011/12 7,131 2.7% 369 (up 65.5%) 2010/11 6,942 0% 223(up 0.9%) 2009/10 6,943 -7.5% 221 (up 25.6%) 2008/09 7,509 -2.4% 176 2007/08 7,692 4.8% 74 2006/07 7,338 -3.1% 132 2005/06 7,575 -2.4% 57 2004/05 7,760 -12.6% 28 2003/04 8,880 -14.9% 24 2002/03 10,432 -10.2% 26 2001/02 11,614 -0.1% 52 2000/01 11,625 2.9% 213 1999/00 11,300 -6.9% 189 1998/99 12,142 -8.6% 201 1997/98 13,282 6.6% 182 1996/97 12,461 -2.8% 209 1995/96 12,820 8.0% 170 Source: IIE Open Doors fact sheet on Indonesia 2014 100 ANNEX V: EVALUATION STAFFING Team Leader Colin Davies has more than 40 years of experience managing participant training, education, and institutional strengthening programs including 15 years of work experience in majority￾Muslim countries. He is responsible for managing this evaluation project under the supervision of the Project Director. This includes acting as the primary liaison with USAID/Jakarta, supervising the staff, scheduling travel, drafting the survey questions, overseeing all interviews, leading the final presentation and the writing of the draft evaluation report, the final evaluation report and lessons learned report. Evaluation Expert Dwiagus Stepantoro has more than 15 years of experience in monitoring and evaluation (M&E) planning and implementation. He is experienced in planning and designing M&E systems, data analysis and database management, and project management and facilitation. He is responsible for data analysis and data reporting on this project, and will also conduct interviews in Indonesia. Higher Education Expert Sherry Mueller served as Director for Professional Exchange Programs at the Institute of International Education (IIE) and also as President of the National Council for International Visitors (now Global Ties US). In both positions her responsibilities included the ongoing evaluation of international exchange programs. She is currently an adjunct professor at American University’s School of International Service. She conducted US campus visits and the Indonesian site visits, and – as a published author – played a major role in editing the draft evaluation report, final evaluation report and lessons learned report. Evaluation Specialist Erlinda Panisales brings more than 20 years of experience in monitoring and evaluation, capacity building and gender mainstreaming. She has expertise in designing data collection tools and methodologies, and is responsible for designing the Survey Monkey quantitative analysis tool. She conducted interviews and group discussions. Project Director Traci Tyers oversees the mid-term evaluation team’s efforts from the IBTCI Home Office in Vienna, Virginia, USA. She has more than 15 years of experience in qualitative and quantitative data collection, M&E and program management. She is primarily responsible for team management and quality assurance and quality control of all deliverables. The team also includes Ian Nugrahane, Logistics Specialist and Dian Rachmawati, Data Analyst. 101 US Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523