March 2015 This mid-term evaluation was produced at the request of the United States Agency for International Development. It was independently prepared by Dr. Valerie Haugen, Dr. Nancy Lloyd Pfahl, Dr. Dwatmadji, Ms. Rina Arlianti, and Ms. Erlyn Yuli Astuti of International Business and Technical Consultants, Inc. (IBTCI). Mid-Term Evaluation of the Indonesian Higher Education Leadership and Management (HELM) Project [Type text] 2 [Type text] Photo Credits: The cover photograph was taken at Politeknik Negeri Padang. The banner in the middle of the photograph is a Higher Education Leadership and Management project commitment banner that was signed by management of the polytechnic. Surrounding the banner are: the HELM Chief of Party (on the left), HELM Quality Assurance Specialist (on the right), and lecturers from Politeknik Negeri Padang. [Type text] 3 [Type text] MID-TERM EVALUATION OF THE HIGHER EDUCATION LEADERSHIP AND MANAGEMENT PROJECT March 3, 2015 FINAL EVALUATION REPORT Contract No. GS-10F-0309P (MOBIS Contract) Order No: AID-497-M-15-00001 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. [Type text] 4 [Type text] ACKNOWLEDGEMENTS “From islands of innovation to seas of change” (Wageningen UR) The Mid-Term Evaluation (MTE) Team would like to thank the personnel of the Government of Indonesia (GOI), in particular the many individuals in the Ministry of Education and Culture (MOEC) and the personnel of the 18 higher education institutions who gave so generously of their time, who provided the team with excellent and candid commentary, and who expressed a genuine interest in ensuring that the benefits being derived from HELM are expanded and sustained. Thanks are also extended to personnel from the donor and development partner community whose views helped to inform a more complete understanding of the context in which HELM is being implemented and the changes beginning to emerge because of HELM. The team would like to thank our USAID/Indonesia colleagues, Ms. Jona Lai (HELM MTE Contracting Officer’s Representative) and Mr. Remy Rohadian (HELM Contracting Officer’s Representative and Alternate Contracting Officer’s Representative for the HELM MTE), who provided ongoing advice and guidance. Last, but most certainly not least, the MTE team would like to thank the staff of Chemonics International for their dependable and accommodating assistance with administrative aspects and on-the-ground logistical assistance, especially Dr. Andrea Bosch (HELM Chief of Party) and Mr. Iskandar Nataamijaya (Deputy Chief of Party) for their excellent preparation work setting the parameters of the MTE just prior to the team’s mobilization and during the in-country work. Their professional and collegial assistance to the team was very much appreciated. 5 CONTENTS MID-TERM EVALUATION OF THE HIGHER EDUCATION LEADERSHIP AND MANAGEMENT PROJECT ............................................................................................... 3 ACKNOWLEDGEMENTS................................................................................................. 4 ACRONYMS........................................................................................................................ 6 GLOSSARY........................................................................................................................ 12 Supporting Concepts....................................................................................................................... 15 PROJECT LOCATION..................................................................................................... 17 FIGURE 1: MAP OF INDONESIA AND HELM HIGHER EDUCATION SITES...... 17 FIGURE 2: MAP OF HELM MID-TERM EVALUATION HEI SITES.......................... 17 PROJECT DATA TABLE................................................................................................. 18 EXECUTIVE SUMMARY................................................................................................. 21 1. INTRODUCTION......................................................................................................... 33 1.1 CONTEXT ............................................................................................................................ 33 1.2 PROJECT BACKGROUND................................................................................................. 35 1.3 EVALUATION PURPOSE AND QUESTIONS................................................................ 36 1.3.1 Evaluation Purpose .................................................................................................................... 36 1.3.2 Evaluation Questions................................................................................................................. 37 1.4 EVALUATION METHODOLOGY .................................................................................... 38 1.4.1 Type of Evaluation ..................................................................................................................... 38 1.4.2 Participants ................................................................................................................................ 38 1.4.3 Data ........................................................................................................................................... 39 1.5 EVALUATION LIMITATIONS .......................................................................................... 40 2. KEY FINDINGS AND CONCLUSIONS.................................................................... 42 2.1 HELM ACHIEVEMENTS..................................................................................................... 42 2.1.1 Achievements Against HELM Targets ....................................................................................... 42 2.1.2  Achievements  Against  HELM  Theory  of  Change........................................................................ 44 2.1.3  Achievements  in  Core  Management  Areas  and  Special  Initiatives ........................................... 52 2.1.4  Achievements  in  Gender  and  Marginalized  Groups.................................................................. 64 2.2.1 Workshops.................................................................................... Error!  Bookmark  not  defined. 2.2.2 Forum with Special Proceedings................................................................................................ 68 2.2.3 Technical Assistance.................................................................................................................. 69 2.3.4 Mentoring and “Advisor to Go”................................................................................................ 70 2.2.5 Blended Learning ....................................................................................................................... 70 2.2.6 Institutionalization and Sustainability ......................................................................................... 72 3. LESSONS LEARNED ................................................................................................... 77 4. RECOMMENDATIONS............................................................................................... 79 ANNEXES.......................................................................................................................... 83 6 ACRONYMS Acronym Description A2G Advisor on the Go ADS Automated Directives System AEC Asian Economic Community AIPT Accreditation of Higher Education Institution/ Akreditasi Institusi Perguruan Tinggi AK Kolaka Kolaka Community College/ Akademi Komunitas Kolaka APS Accreditation of Study Program/ Akreditasi Program Studi AR Action Research ARP Action Research Project ASEAN Association of South East Asian Nations AusAID Australian Agency for International Development AusDFAT Australian Department of Foreign Affairs and Trade BAN-PT National Accreditation Board for Higher Education/ Badan Akreditasi Nasional Pendidikan Tinggi BAPPENAS National Planning Board/ Badan Perencanaan Nasional BKMA Academic Quality Control Division/ Badan Kendali Mutu Akademik BLU Public Service Agency/ Badan Layanan Umum BPK Audit Board of the Republic of Indonesia/ Badan Pemeriksa Keuangan BPKP Financial Development and Supervisory Board/ Badan Pengawasan Keuangan dan Pembangunan CA Core Area CDCS Country Development Cooperation Strategy CEDAW Convention on the Elimination of Discrimination Against Women COP Chief of Party CVA Harvard Capacities and Vulnerabilities Framework DCOP Deputy Chief of Party DG Director General 7 DGHE/DIKTI Directorate General for Higher Education/ Direktorat Jenderal Pendidikan Tinggi ECPDM European Center for Development and Policy Management EGAT Economic Growth and Trade ESP Education Strategic Plan F/C/R Findings, Conclusions, Recommendations F2F Face-to-Face FBO Faith Based Organization FGD Focus Group Discussions FMIS Financial Management Information System FMS Financial Management System FY Fiscal Year GAL General Administration and Leadership GER Gross Enrollment Rate GIZ Gesellschaft fur Internationale Zusammenarbeit GOI Government of Indonesia GDP Gross Domestic Product HE Higher Education HEI Higher Education Institution HELM Higher Education Leadership and Management Project HESP Higher Education Strategic Plan HICD Human and Institutional Capacity Development IBTCI International Business and Technical Consultants Incorporated ICT Information and Communications Technology ID Institutional Development IDEAL Institute for Development Education and Learning IDF Institutional Development Framework I-MHERE Indonesia Managing Higher Education for Relevance and Efficiency IPB Bogor Agricultural University/ Institut Pertanian Bogor Irjen Inspector General/ Inspertorat Jenderal 8 ISO International Standards Organization ISSLA Indonesian Survey of Student Activities ITB Bandung Institute of Technology/ Institut Teknologi Bandung JBS JBS International – Aguirre Division KABBS Knowledge, Attitudes, Beliefs, Behaviors, Skills KJM Quality Assurance Office/ Kantor Jaminan Mutu KOPERTIS Coordinator of Private Higher Education Institutions/ Koordinasi Perguruan Tinggi Swasta LAKIP Government Accountability Performance Report/ Laporan Kinerja Instansi Pemerintah LOP Life of Project LPPM Research and Community Service Center/ Lembaga Penelitian dan Pengabdian pada Masyarakat M&E Monitoring and Evaluation MIS Management Information System MOEC/Kemdikbud Ministry of Education and Culture/ Kementerian Pendidikaan dan Kebudayaan MOF/Kemkeu Ministry of Finance/ Kementerian Keuangan MONE Ministry of National Education MORA/Kemenag Ministry of Religious Affairs/ Kementerian Agama MOSA Ministry of Social Affairs MOU Memorandum of Understanding MOWECP/KEMENPPPA Ministry of Women’s Empowerment and Child Protection/ Kementerian Pemberdayaan Perempuan dan Perlindungan Anak MTE Mid-Term Evaluation NUFFIC Netherlands Organization for International Cooperation in Higher Education OCAI Organizational Capacity Assessment Instrument OD Organizational Development OVC Opportunities for Vulnerable Children PAG Project Advisory Group PD Vice Dean/ Pembantu Dekan 9 PMP Program Monitoring Plan PNM State Polytechnic of Medana/ Politeknik Negeri Medana PNP State Polytechnic of Padang/ Politeknik Negeri Padang Poli Aceh Aceh Polytechnic/ Politeknik Aceh POLMED State Polytechnic of Medan/ Politeknik Negeri Medan POLNES State Polytechnic of Samarinda/ Politeknik Negeri Samarinda PR Vice Rector/ Pembantu Rektor PTN BH Higher Education Institution Legal Entity/ Perguruan Tinggi Negeri Badan Hukum PTN BLU Higher Education Institution Public Service Agency/ Perguruan Tinggi Negeri Badan Layanan Umum PTN Satker Higher Education Institution Work Unit/ Perguruan Tinggi Negeri Satuan Kerja PwD People with Disabilities QA Quality Assurance R&D Research and Development RENSTRA Strategic Planning/ Rencana Strategis SEAMEO South East Asian Ministers of Education Organization SED-TVET Sustainable Economic Development through Technical and Vocational Education and Training SOW Scope/ Statement of Work SPI Internal Audit Unit/ Satuan Pengawasan Internal STTA Short-Term Technical Assistance TA Technical Assistance TOR Terms of Reference UGM Gajah Mada University/ Universitas Gajah Mada UHO Halu Oleo University/ Universitas Halu Oleo UKSW Satya Wacana Christian University/ Universitas Kristen Satya Wacana UI University of Indonesia/ Universitas Indonesia UIN SUKA Sunan Kalijaga Islamic University/ Universitas Islam Negeri Sunan Kalijaga 10 UKy University of Kentucky UKT Single Tuition Fee/ Uang Kuliah Tunggal ULP Procurement Services Unit/ Unit Layanan Pengadaan UM State University of Malang/ Universitas Negeri Malang UMM Muhammadiyah University of Malang/ Universitas Muhammadiyah Malang UMS Surakarta Muhammadiyah University/ Universitas Muhammadiyah Surakarta UNAIR Universitas Airlangga UNAND Andalas University/ Universitas Andalas UNCEN Cenderawasih University/ Universitas Cenderawasih UNESA State University of Surabaya/ Universitas Negeri Surabaya UNHAS Hasanuddin University/ Universitas Hasanuddin UNICEF United Nations International Children’s Education Fund UNIDO United Nations International Development Organization UNIPA State University of Papua/ Universitas Negeri Papua UNLAM Lambung Mangkutat University/ Universitas Lambung Mangkurat UNM State University of Makassar/ Universitas Negeri Makassar UNMUL Mulawarman University/ Universitas Mulawarman UNNES State University of Semarang/ Universitas Negeri Semarang UNP State University of Padang/ Universitas Negeri Padang UNPATTI Pattimura University/ Universitas Pattimura UNS Eleven March University/ Universitas Sebelas Maret UNSYIAH Syiah Kuala University/ Universitas Syiah Kuala UNTAD Tadulako University/ Universitas Tadulako UNTAN Tanjungpura University/ Universitas Tanjungpura UNY Yogyakarta State University/ Universitas Negeri Yogyakarta UPI Indonesia University of Education/ Universitas Pendidikan Indonesia US United States of America USAID United States Agency for International Development 11 USG United States Government USU University of North Sumatera/ Universitas Sumatera Utara 12 GLOSSARY1 Term Definition Capability The collective ability of a group or system to do something either inside or outside the system.2 Capacity The overall ability of organizations or systems to add value to others.3 Capacity Enhancement A process that focuses on increasing the abilities of specific types of personnel within an organization.4 Collaborative Evaluation A collaborative evaluation “implies a varying level of involvement that considers the extent to which program staff and other stakeholders should be included as part of the evaluation team...is often empowering to participants...(and) enhances their understanding of evaluation so they gain new skills...promotes utilization of evaluation findings”5 Conflict Sensitivity Is “…the ability of an organization to develop and use the sum of its human and organizational capital to minimize negative and maximize positive impacts on the conflict dynamics of the environment(s) where it works. This means an awareness of the causes of historical, actual or potential conflict, and the likelihood of further conflict, and its likely severity; and the capacity to work with all parties to minimize the risk of further conflict.6” Counterpart Relationship “A counterpart is an individual or a collectivity (e.g., a group or even and organization) who contributes to a relationship designed to exchange knowledge and support as part of a deliberate effort to induce development results in a partner country.”7 Direct Beneficiary The people for who the project is being undertaken. 1 A number of these definitions, including those for Effectiveness, Efficiency, Impact, Lessons Learned and Sustainability, are taken from AusGuide. Canberra: Austrailian Agency for International Development (AusAID). 2 Baser, Heather and Morgan, Peter. (2008) Capacity, Change and Performance: Synthesis Report. Maastrich, Netherlands: European Center for Development Policy Management. 3 Ibid. 4 John M. Cohen (1993) “Building Sustainable Public Sector Managerial, Professional, and Technical Capacity: A Framework for Analysis and Intervention,” Harvard Institute for International Development, Harvard University. Development Discussion Papers, n. 473. October. 5 O’Sullivan, Rita M. (2008) Practicing Evaluation: A Collaborative Approach. SAGE Publications, Inc.: Thousand Oaks, CA. 6 Waqo, Halakhe and Onyango, Rachael. (2008). Conflict Sensitive Programming. Kenya Humanitarian Forum, United Nations, Office for the Coordination of Humanitarian Affairs, New York. 7 Morgan, Peter. (2008) Improving Counterpart Relationships in Papua New Guinea, A Study For the Governments of Papua New Guinea and Australia (Draft), July. 13 Effectiveness The extent to which the development intervention’s objectives were achieved, or expected to be achieved, taking into account their relative importance—progress in achieving objectives, standard of outputs, extent of benefit to the target population. Efficiency A measure of how economically resources/inputs (funds, expertise, time, etc.) are converted to results—timeliness and appropriateness of preparation and implementation processes, including appraisal and peer review; standard of the contract and activity implementation by the contractor; strength of partner government support and the value of dialogue in country; USAID management including risk management and use of external expertise; activity monitoring and communication. Endogenous Processes Capacity processes that appear to be essentially internally driven (by the organization), and not driven by the concerns of an external donor. Evaluation A social science activity directed at collecting, analyzing, interpreting and communicating information about the workings and effectiveness of social programs. Gender Equality Gender equality means that women and men enjoy the same status. Women and men have equal conditions for realizing their full human rights and potential to contribute to national, political, economic, social and cultural development, and to benefit from the results. Gender equality is the equal valuing by society of both the similarities and differences between women and men, and the varying roles that they play.8 Gender Equity Gender equity is the process of being fair to women and men. To ensure fairness, measures must often be available to compensate for historical and social disadvantages that prevent women and men from otherwise operating on a level playing field. Equity leads to equality.9 Impact Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended (inter alia, impacts may be economic, institutional, technological, environmental, socio-cultural, gender-related); measurement of extent of impacts (if possible, a cost-benefit analysis should be undertaken); assessment of effect on development policies. Indirect Beneficiary Indirect stakeholders or other parties that benefit from a project. Institution Institutions are humanly devised constraints that shape human interaction. They are the ‘rules of the game’ in a society, the rules that facilitate human interaction and societal life. They are the arrangements human have made for governing their lives. They may be formal arrangements, such as legal systems and property rights, or informal arrangements, like moral standards. In some cases, they take the form of implicit work views or mental maps (i.e., cognitive frameworks for looking at the 8 Status of Women-Canada. (1996) Gender-Based Analysis: A Guide for Policy-Making. Ottawa. Available at: http://www.pacificwater.org/userfiles/file/IWRM/Toolboxes/gender/gender_based_analysis.pdf. See also 2003 Update to this policy making guide available at: http://www.gnb.ca/0012/Womens-Issues/Genderanalysis2003.pdf 9 Ibid. 14 world around you). These arrangements or institutions operate at different levels, ranging from an international level (such as trade arrangements) to community and individual levels (e.g., the values that determine the way in which people interact with each other).10 Institutional Development Institutional development is the processes by which institutions evolve and perish, (i.e., ongoing endogenous and autonomous processes in society). 11 Lessons Learned Generalizations based on evaluation experiences with activities, programs or policies that draw from the specific circumstances and apply to broader situations. Frequently lessons learned highlight strengths or weaknesses in preparation, design and implementation that affect performance, outcome and impact. Levels of Analysis Approach The approach used within this evaluation. This approach focuses not only on the individual level of analysis, but also how that individual works within a larger organization and/or set of organizations. Narrative Processes A set of connected pieces of information or stories that illustrate or inform a process. Objective A concrete statement describing what the project is trying to achieve. The objective should be written at a low level, so that it can be evaluated at the conclusion of a project to see whether it was achieved or not. A well-worded objective will be Specific, Measurable, Attainable/Achievable, Realistic and Time Bound (SMART). http://www.tenstep.com/open/miscpages/94.3Glossary.htmlhttp://www.ten step.com/open/miscpages/94.3Glossary.html Organization(al) Development The practice of changing people and organizations for positive growth. This can take on many forms including, but not exclusively: team-building, organizational assessments, career development, training, e-learning, coaching, innovation, leadership development, talent management, and change management.12 10 European Center for Development Policy Management (ECDPM) (no date provided). Institutional Development: Learning by Doing and Sharing. Maastricht, Netherlands: Netherlands Ministry of Foreign Affairs, Poverty Policy and Institutional Development Division. Available at: http://portals.wi.wur.nl/files/docs/msp/Institutional_dev_tools_ECDPM.pdf 11 Ibid. Also, for an excellent historical perspective on Organizational Development and Institutional Development, See Van der Velden, Fons and Leenknegt, Anne-Marie. (2006) Facilitation of Organizational Change: Beyond Organizational and Institutional Development. Contextuals, No. 5: December. http://contextinternationalcooperation.files.wordpress.com/2007/12/contextuals-no-5.pdf 12 For a general overview of organizational development, visit the Organizational Development Portal. Available at: http://www.odportal.com/OD/whatisod.htm. 15 Regional Industry Clusters Regional industry clusters—geographic concentrations of interconnected firms and supporting organizations—represent a potent source of productivity at a moment of national vulnerability to global economic competition. An industry clusters program stimulates the collaborative interactions of firms and supporting organizations in regional economies to produce more commercial innovation and higher wage employment.13 Relevance The extent to which the objectives of a development intervention are consistent with the beneficiaries’ requirements, country needs, global priorities and partners’ and donors’ policies. This includes evaluation of the relevance of the activity’s objectives (i.e., were they clear, realistic and measurable?) and the adequacy of documented activity design to achieve the objectives. Stakeholder Specific people or groups who have a stake in the outcome of the project. Normally stakeholders are from within the company and include internal clients, management, employees, administrators, etc. A project may also have external stakeholders, including suppliers, investors, community groups and government organizations.14 Supportive Leadership Supportive Leadership is one of the leadership styles found in path-goal theory. A supportive leader attempts to reduce employee stress and frustration in the workplace. This method is effective when your work tasks are dangerous, tedious and stressful. It is not really effective if your work tasks are intrinsically motivating because you don't need to be motivated to do the work. Supporting Concepts: In order to fully understand supportive leadership, you need to place it within the larger context of path-goal theory developed in large part by Robert House. According to the theory, a manager establishes the goal for his employees and sets forth the path for achieving that goal. Tasks for a manager include the clarification of tasks, clarification of the employee's role and responsibilities, clarification of the criteria for success, provision of guidance and coaching, removal of obstacles that can prevent task completion and provision of psychological support and awards when appropriate. The theory proposes that you should use certain leadership styles in different situations. Leadership styles available to you include: directive, participative, achievement-orientated, and of course, supportive. Supportive leadership is one leadership approach that you use in any given situation depending on the nature of the task and the nature of the employees.15 Sustainability This is the continuation of benefits from a development intervention after major development assistance has been completed, including the sustainability of benefits (technological, social, environmental, gender); sustainability of institutional capacity; and maintenance of a future recurrent budget (financial sustainability). 13 Mills, Karen G., Reamer, Andrew, and Reynolds, Elisabeth B. (2008) Clusters and Competitiveness: A New Federal Role for Stimulating Regional Economies. Brookings Institution: Washington, DC. Available at: http://www.brookings.edu/research/reports/2008/04/competitiveness-mills. 14 Stakeholder Forum for a Sustainable Future. (No date) Frequently Asked Questions. Available at: http://www.stakeholderforum.org/sf/index.php/about-us/faqs 15 Education Portal. (No date) Supportive Leadership Style: Definition, Lesson and Quiz. http://education￾portal.com/academy/lesson/supportive-leadership-style-definition-lesson-quiz.html#lesson Accessed Jan 1, 2015 at 12:18am from Jakarta, Indonesia. 16 Workforce Development Activities that increase the competence of individual staff so that they can contribute effectively to public service throughout their whole working life and which simultaneously increase the capacity of public service agencies to adopt high￾performance work practices that support their staff to develop the full range of their potential skills and value. 17 PROJECT LOCATION Figure 1: Map of Indonesia and HELM Higher Education Institution (HEI) Sites Figure 2: Map of HELM Mid-Term Evaluation HEI Sites LEGEND C1|C2                        University – Public University – Private Polytechnic – Public Polytechnic – Private Community College – Public Community College – Private LEGEND C1|C2 University – Public University – Private Polytechnic – Public Polytechnic – Private Community College – Public Community College – Private 18 PROJECT DATA TABLE Program Name Higher Education Leadership and Management (HELM) Project Region/Country Asia/Indonesia (Authorized Geographic Area 937) Government of Indonesia Counterpart Ministry of Education and Culture, Directorate General for Higher Education (Direktorate Jenderal Pendidikan Tinggi) Funder United States Agency for International Development (USAID) Prime Managing Contractor Chemonics International Subcontractors Indiana University (in alliance with Ohio State University, University of Illinois–Urbana-Champaign) JBS International, Inc. Aguirre Division University of Kentucky Contract Number Contract No. AID-497-C-12-00001 Agreement Information Modification #07 (October 24, 2014) Modification #06 (May 9, 2014) Modification #05 (June 10, 2014) Modification #04 (March 18, 2013) Modification #03 (January 10, 2013) Modification #02 (June 21, 2012) Modification #01 (April 5, 2012) Key Program Dates Final Evaluation Mission TBD Mid-Term Evaluation Mission – Field activities December 1, 2014 to January 23, 2015 HELM Implementation Period November 28, 2011 to November 30, 2016 Total Contract Value (Nov 28, 2011-Nov 30, 2016): (As of Jan 1, 2015): (As of Nov 28, 2011): USD$19,678.939.00 USD$19,678.939.00 USD$19,678.939.00 Evaluation Provider International Business & Technical Consultants, Inc. (IBTCI) 19 MTE Team Members: Ms. Erlyn Astuti (Research Specialist) Ms. Rina Arlianti (Indonesia Higher Education Specialist-Technical and Vocational Education) Dr. Dwatmadji (Indonesia Higher Education Specialist) Dr. Valerie Haugen (Team Leader) Dr. Nancy Lloyd-Pfahl (Evaluation Specialist) MTE Mission Site Visits Aceh, Central Java, East Java, Maluku, North Sumatera, South Sulawesi, Papua, West Papua, West Kalimantan, and West Sumatera Provinces, Jakarta List of 17 Higher Education Institutions Visited by the MTE Team 16 Kolaka Community College/ Akademi Komunitas Kolaka (AK Kolaka) State Polytechnic of Padang/ Politeknik Negeri Padang (PNP) State Polytechnic of Samarinda/ Politeknik Negeri Samarinda (POLMED) Satya Wacana Christian University/ Universitas Kristen Satya Wacana (UKSW) Sunan Kalijaga Islamic University/ Universitas Islam Negeri Sunan Kalijaga (UIN SUKA) State University of Malang/ Universitas Negeri Malang (UM) Muhammadiyah University of Malang/ Universitas Muhammadiyah Surakarta (UMM) Andalas University/ Universitas Andalas (UNAND) Cenderawasih University/ Universitas Cenderawasih (UNCEN) Hasanuddin University/ Universitas Hasanuddin (UNHAS) State University of Papua/ Universitas Negeri Papua (UNIPA) State University of Makassar/ Universitas Negeri Makassar (UNM) State University of Semarang/ Universitas Negeri Semarang (UNNES) State University of Padang/ Universitas Negeri Padang (UNP) Pattimura University/ Universitas Pattimura (UNPATTI) Syiah Kuala University/ Universitas Syiah Kuala (UNSYIAH) 16 The MTE team physically visited 17 HEI. The MTE team cancelled confirmed meetings with Gajah Mada University/ Universitas Gajah Mada (UGM) and Aceh Polytechnic in HELM Cohort 1, and the State University of Medan/ Universitas Negeri Medan (UNIMED) in HELM Cohort 2. Interviews with UGM and Aceh Polytechnic were rescheduled and conducted remotely by telephone from Jakarta. There was a total of 19 HEIs included in the sample. 20 Tanjungpura University/ Universitas Tanjungpura (UNTAN) 21 EXECUTIVE SUMMARY BACKGROUND Indonesia’s higher education landscape is complex and varied. Currently, 30% of Indonesian youth aged 19-24 (or five million young people) are enrolled in one of the country’s estimated 3,700 higher education institutions (HEI). One out of every five studies economics, law or a social science discipline, with the majority of public HEI students enrolled in pre-service teacher education programs. There are too few graduates in the hard sciences required for private sector growth and too few graduates with the critical thinking skills and creativity to contribute to overall socio-economic development. Approximately 4% (or 92 public institutions) are under the management of Ministry of Education and Culture (MOEC) and a small array of faith-based institutions are under the management of the Ministry of Religious Affairs (MORA), but are obligated to adhere to MOEC policies and regulations. These institutions serve 38% of the student population. The bulk of the institutions (Core Area 3,600 or 96%) is private and serves about 68% of the student population. A small percentage of the HEIs - known as the “Big 10” - are able to consistently provide high quality education services. This small number is inadequate to keep pace with the demands of Indonesia’s middle-income status17 and the country’s economic and social development aspirations. Private institutions are largely unregulated and of questionable value and quality. The Government of Indonesia (GOI) spends relatively little on the tertiary education sub-sector. Of the 25% of Gross Domestic Product (GDP) allocated by law to the education sector, the tertiary education sub-sector captures very little of these funds. Of the 1.2% of GDP allocated to the sub-sector, 0.3% comes from public sources and 0.9% (largely tuition) comes from private sources. As a result, Indonesia is among those countries with the highest shares of private funding for tertiary education in the world. In an effort to improve its colleges and universities as student enrollment continues to increase, the GOI has embarked on an ambitious course. In 2012, the GOI passed Law No. 12 (the Higher Education Act) that sets the stage for ongoing reforms that address areas such as: academic quality assurance and relevance; university management and governance; university financial management; and efforts to provide greater opportunities for Indonesia’s young women and poorer students. Given the immensity of the higher education sub-sector and the high demand for tertiary education, the GOI is faced with a daunting task as it seeks to bring the sub-sector in line to meet the requirements of a middle-income country. PROJECT DESCRIPTION USAID/Indonesia’s 2014-2018 Country Development Cooperation Strategy (CDCS), “Investing in Indonesia: A Stronger Indonesia Advancing National and Global Development,” notes that “The higher education sector (which includes polytechnics, community colleges, and teacher training institutions as well as universities) has a critical role to play in both training those who manage essential services and educating future managers, technical specialists, and leaders.” The “ultimate goal of USAID education programs is to collaborate with the Government and people of Indonesia to improve the academic 17 The World Bank categorizes Indonesia, Malaysia, the Philippines and Thailand as middle-income countries based on a range of socio-economic development factors. 22 performance of their basic and higher education sectors, so that the education services will be at high quality, and more relevant to the economic and social growth of the country. This goal is articulated as the Assistance Objective statement of USAID Indonesia’s 2010-2014 strategic plan for education: to help ‘students [be] better prepared for success in learning and work.’”18 The HELM project is one of two USAID higher education projects initiated between 2010 and 2014. HELM is a five-year USD$19.678 million project under implementation from November, 2011 to November, 2016. The HELM contract was awarded to Chemonics International and its subcontractors, Indiana University in alliance with The Ohio State University and University of Illinois at Urbana￾Champaign (hereafter referred to as the Indiana Alliance), and JBS International – Aguirre Division (JBS) and the University of Kentucky (UKy) on November 28, 2011. HELM aims to build the capacity of higher education leaders and managers from 50 public and private HEIs, including universities, polytechnics and community colleges. These 50 institutions are located across Indonesia that are grouped in two cohorts, each with 25 HEIs respectively. Engagement with Cohort 1 institutions commenced in 2012 and Cohort 2 engagement commenced in mid-2014. HELM seeks to increase the capacity of these HEIs in 4 core management areas: 1. General Administration and Leadership 2. Financial Management 3. Quality Assurance; and 4. Collaboration with External Stakeholders. HELM uses a number of strategies to enhance capacity in these institutions. Four components comprise the delivery strategy: 1. Design technical assistance approaches to achieve effective implementation of key reforms across the higher education system, coordinating with Directorate General for Higher Education/ Direktorat Jenderal Pendidikan Tinggi (DIKTI) and maximizing opportunities to internalize best practice within the Higher Education (HE) system; 2. Provide technical assistance to increase management capacity and improve performance at partner HEIs and disseminate best practices; 3. Strengthen graduate-level programs in higher education leadership and management; and 4. Support Special Initiatives by providing assistance to advance reforms and innovation within management of HEIs. EVALUATION PURPOSE AND QUESTIONS HELM is at the end of the third year of its project life cycle. In 2014, USAID/Indonesia issued a request for proposals to carry out the mid-term evaluation (MTE) of the project. The contract was awarded to International Business & Technical Consultants, Inc. (IBTCI). IBTCI was tasked with evaluating HELM’s performance to date, making an assessment of the sustainability and ‘replicability’ of the project’s work, and providing recommendations to help guide USAID and HELM to improve performance where required for the remainder of the project. MTE findings and recommendations should also provide guidance to Ministry of Education and Culture (MOEC), the DIKTI, and other relevant agencies, 18 HELM Contract document. 2011. 23 including the donor community such as the Australian Department for Foreign Affairs (DFAT) and the leadership of other HEIs. IBTCI fielded a five-person team from December 1, 2014 to January 23, 2015 to evaluate HELM’s progress in: 1. Improving the knowledge and skills of HE staff in the four core areas 2. Supporting the partner HEIs in implementing their thematic action plans, and 3. Contributing to improved processes and systems at the institutional level. The team was asked to provide findings, conclusions and recommendations related to 12 evaluation questions that focused on HELM’s performance to date and looked forward over the next two years of HELM implementation and beyond. Question Set 1: Looking Back: Review of Achievement of Objectives/Outcomes at Mid-point 1. To what extent has HELM achieved its stated mid-point objectives and outcomes? 2. To what extent have HELM resources been managed and utilized efficiently and effectively across HEI subsets to accomplish the project’s mid-point objectives and outcomes? Why/Why not? 3. What specific value has been added through HELM activities for beneficiaries at the output level, the short-medium term outcome level and the longer-term impact level, and to what extent? 4. What aspects of HELM are proving most and least effective in building the capacity of individuals participating in HELM activities? Why/Why not? 5. What aspects of HELM are proving most and least effective in contributing to the organizational development of the HEIs participating in HELM activities? Why/Why not? 6. How effective has HELM been in improving women’s participation across components? Why/Why not? 7. Are the effects of HELM on 50 HEI contributing to improving the overall enabling environment for higher education? Question Set 2: Looking Ahead to Project Endpoint and Post-Project Impact (Sustainability and Replication) 1. Is HELM on track to achieve its overall end-of-project goals across all HEI subsets? Why/Why not? 2. What HELM activities and results are likely to be sustained and/or replicated after the project is completed? Why/Why not? 3. Is the HELM project design, structure and approach suitable for achieving the desired results? Why/Why not? 4. If HELM design, structure and approaches are not suitable, how should these be reorganized? 5. What, particularly, is needed to promote Phase 3 institutionalization? EVALUATION METHODOLOGY This was a quasi-collaborative evaluation. HELM personnel were invited to attend most interviews and were given the opportunity to ask questions. The evaluation team made an effort to ensure that personnel were able to provide their perspectives on the data derived during interviews and to enhance the evaluation team’s understanding of HELM implementation. The team used several analytical frameworks to provide a systematic way to examine HELM’s implementation and present data including: the levels of analysis, and Brinkerhoff’s enabling environment and training program quality frameworks. 24 The team used a mixed methods approach to collect and analyze primary and secondary source data. The methods consisted of: a literature review (including project documentation and literature derived from Internet searches), key informant interviews, group discussions, and an opinion survey. Data were triangulated and the team ensured that primary data took account of a set of variables (e.g., HEI type, the type of HELM activity – core area, special initiative, etc. – sex, and region). The team visited 17 HEIs located in the following provinces throughout the Indonesian archipelago: Aceh, Central Java, East Java, Maluku, Papua, South Sulawesi, West Papua and West Sumatera. These 17 HEIs are representative of the overall group of 50 HELM HEI partners. The site visits focused on Cohort 1 institutions since HELM has only recently begun to work with its 25 new Cohort 2 partners. The team interviewed 277 individuals (90 women/187 men). The majority of the MTE participants were HELM alumni working in the 17 HEIs as well as personnel from two additional HEIs that the team did not visit but interviewed by telephone or in Jakarta. 158 individuals (29 women and 129 men) are in management positions and 34 individuals (15 women and 19 men) are lecturers. HELM personnel and other stakeholders from the GOI and donor and development partners were also interviewed. LIMITATIONS Several limitations constrained the evaluation. The two most prominent were: (1) the MTE time frame that included a major holiday period (December 18, 2014 to January 5, 2015) during which opportunities to meet beneficiary HEIs and their faculty were highly constrained. This notwithstanding, the Team managed to visit 17 sites across the archipelago and interview 277 individuals. and (2) issues with the quality of project data quality and documentation that required time to rectify and that prevented a thorough early grasp of the project. DISCUSSION HELM Achievements According to HELM-reported achievements, the project is on target or has exceeded targets for the project’s17 performance indicators. This finding is consistent with the 2013 external audit of HELM. The evaluation team found solid evidence (beyond the data associated with the largely output-based indicators and the “outcome”-based custom indicators) that HELM is producing positive short-term outcomes. The MTE data show that there are positive results being derived from the various HELM activities, particularly with respect to the enhancement of individual knowledge and skills and to a certain extent, to the application of the new knowledge and skills. Certain HELM activities such as the Action Research Program (ARP) are building a cadre of individuals within each institution that understand the importance of technical knowledge and skills, soft skills such as those typically addressed in training programs (i.e., communication skills, time management, etc.), and those that are seldom incorporated with any intentionality (e.g., group processes that are essential for organizational transformation). Other HELM activities such as the workshops carried out under each of the 4 core areas have seen some widespread traction with respect to the: • Use of new instruments (such as the Triple Helix methodology19 and a checklist to support accreditation processes) 19 Etzkowitz, Henry. (2008). The Triple Helix: University-Industry-Government Innovation In Action. London: Routledge.; Etzkowitz. Henry. (2003). Innovation in Innovation: The Triple-Helix of University-Industry-Government Relations. Social Science Information, 42(3), 293-337.; Etzkowitz, Henry and Leydesdorff, Loet. (1995). The Triple Helix: 25 • Introduction of new processes or the improvement of existing processes (such as internal processes for preparing accreditation submissions and improving procurement processes) • Knowledge and skills transfer within units and, in some cases, to units in other HEIs. Core Area 4 – partnerships and collaboration – stands out with respect to contributing to obvious development results. Alumni were inspired by the Triple Helix model for developing partnerships between research units, business and industry, and government. Several alumni reported a change in their attitudes and skills toward how to set up business partnerships and student internships. Results are actually quantifiable, although HELM has not undertaken any effort to date to capture these important contributions. Notable examples include, but are not limited to: • Lecturers joined and worked with other university partners in West Sulawesi and in West Java to approach the Ministry of Industry to change its regulations for the cement industry. As a result of the changes in regulations in response to practices advocated by the university personnel, the cement corporation was able to save IDR12,000,000,000 (about USD$950,000). • Collaboration with a cacao plantation and the local government in West Sumatra is underway to develop new products for the beans since there are no chocolate factories in Indonesia. • Another partnership has been developed to produce and market tea from agar wood and sandalwood for health purposes. • A partnership between the pharmaceutical faculty and corporations producing traditional oils will apply new technologies to increase the efficiency of production processes without sacrificing quality. An improved profit margin is expected to result. • Another collaboration with the government focuses on expanding the production of a new tropical species of wheat seed that should increase revenue for the growers. • Seven staff members are collaborating with the local government and rice farmers on Montawai Island in a new partnership to analyze the soil, assist in choosing the correct varieties of rice to plant in 2015, and manage their rice fields to increase rice yields. • In another new collaboration, researchers and a palm oil corporation are experimenting with the production of composite wood from the coconut shells using local materials. The other three core areas all have success stories, but they do not lend themselves to showcasing or quantification of immediate development results. Please see section 2.1.3 in the report for a discussion of the efforts and results specific to the other core areas. In addition to the ARP, the project has introduced two other ambitious initiatives: the Post-Graduate Program that aims to develop study programs for master and doctoral degrees in higher education leadership and management, and the transformation of the four core areas’ workshop content into an executive/professional development program. It is expected that the executive development program will be integrated with the post-graduate degree program. Both initiatives utilize a blended learning delivery platform. These three initiatives are being carried out with the assistance of the HELM University-Industry-government Relations: A Laboratory for Knowledge Based Economic Development. EASST Review, 14(1), 11-19.; Leydesdorff, Loet. (2011). “Meaning” as a Sociological Concept: A Review of the Modeling, Mapping, and Simulation of the Communication of Knowledge and Meaning. Social Science Information, 50(3-4), 1- 23.; Leydesdorff, Loet and Etzkowitz, Henry. (1996). Emergence of a Triple Helix of University-Industry￾Government Relations. Science and Public Policy, 23(3), 279-286. Ranga, Marina. and Etzkowitz, Henry (2013). Triple Helix Systems: An Analytical Framework for Innovation Policy and Practice in the Knowledge Society, Industry and Higher Education, 27(4), 237-262. 26 subcontractors. The ARP is implemented across the 25 Cohort 1 institutions and the post-graduate degree program is being undertaken by four targeted HEIs. One of these institutions, the University of Gadjah Mada/ Universitas Gajah Mada (UGM), is also working on the executive/professional development program. Implementation of these initiatives is on track. The ARP has generated widespread excitement and commitment among the ARP alumni. The most important dimension of this program has been the opportunity to visit the Philippines and Thailand, which enabled alumni to be exposed to examples of progressive, successful institutions that operate in contexts similar to that of Indonesia. The post-graduate program aims to ultimately produce a common core curriculum for higher education leadership and management and the program will continue to be supported through a collaborative relationship with Indiana University even after HELM finishes. The executive development program has produced an impressive number of materials in a highly structured framework. Of these three initiatives, the one with the greatest potential to transform the organizational culture and practices of Indonesian higher education institutions is the ARP. This program necessitates crossing work siloes to work collaboratively and interdependently in groups and requires changes in mindsets and behaviors. Nearly all of the ARP projects are on track. This impressive progress has all been accomplished with very limited investment of time and money. The program could benefit from providing a framework that would enable identification of clusters of benefits through a meta￾analysis of the 25 HEIs’ actual and anticipated results. It may be possible to quantify these results as well. The post-graduate degree program is also an ambitious undertaking, not least because Indonesian HEIs do not have a tradition of collaboration and healthy competition. To date, HELM has undertaken a separate venture with each institution. The move toward a common core curriculum will require a clear understanding of what motivates an institution to collaborate with another and a convincing argument about the benefits of collaboration. If HELM is successful in establishing a good quality master’s and doctoral degree programs and if the project is successful in selling the degree programs to potential students, the project will have provided a great service to Indonesia. Getting to a common core and ensuring an adequate supply of qualified students and developing a brand for the study programs remain significant hurdles. The executive development program is, arguably, the most ambitious of the three initiatives and also the most challenging to bring to fruition. The program materials are based on the content developed and presented in the various HELM workshops that have been carried out over the past three years with Cohort 1 alumni. As is the case with the post-graduate degree program, HELM intends to deliver the program through various modalities including face-to-face (F2F) instruction and Internet-based instruction. A rapid quality assessment of the materials and the teaching methods reveals some significant quality issues. Given the challenges with establishing the blended learning platform as a legitimate and viable delivery platform, the quality issues of the materials and methods and the marketing required to ensure that professionals engage with the program, the likelihood of success of this initiative is questionable. Monitoring and Evaluation HELM has experienced significant flux in its monitoring and evaluation. The year one Performance Monitoring Plan included 17 indicators; in year two, seven of the 17 were dropped and 10 of the 17 were reported on; in year three, the 10 year two indicators were carried over and seven new indicators directly linked to measuring achievements in the four core areas and community colleges were added. The relevance and appropriateness of many of these indicators (including standard and custom indicators), particularly with respect to the extent they enable HELM to tell the real story of its achievements and to identify underperformance and inform strategic planning, is questionable. Additional 27 measurements and data are needed to support these indicators; these would enable HELM to tell its story more convincingly and plan more strategically. The data that underpin many of the 17 indicators also suffer from a lack of quality control. The challenges associated with the project’s monitoring and evaluation have an effect on the extent to which HELM can demonstrate short- and medium-term outcomes. At present, there is no structured way to capture and report on real impact. This is not to say that HELM has not contributed to short￾and medium-term outcomes or that HELM may be putting place elements that will show an impact further down the road. It is to say that at present there is no way to systematically capture and report on the short- and medium-term outcomes that the evaluation team’s data demonstrate are occurring. An exception might be the Action Research (AR) special initiative that has a clearly articulated agenda and well-defined expectations and systematic method of monitoring and evaluation at the institutional level. Yet even this well-run activity does not aggregate the real effects of the initiative. A meta-analysis of effects across the institutions would allow HELM to showcase outcome patterns and trends. MTE alumni consistently urged HELM to help set performance targets associated with the core area workshops. The evaluation team concurs with this recommendation from HELM stakeholders. Sustainability and Institutionalization HELM core area workshops, the diverse methods of post-workshop follow-up, the blended learning initiative, the ARP and the post-graduate strengthening program, among others, each contain some inherent assumptions about sustainability. The strength of these various assumptions needs to be tested. The lack of a coherent and cohesive sustainability strategy that is underpinned by a clear notion of institutionalization is a liability that could have a detrimental effect on the project’s achievements. There does not appear to be a fundamental understanding of the concept of institutionalization and all that it could entail, either within HELM or its HEI partners. There is a strong risk that the investment in HELM alumni who attended the four core area workshops will result in limited (if any real) institutional change, let alone transformation. The use of HELM technical personnel and Indonesian short-term consultants to help alumni make real progress in the four core areas is an inadequate response to the size of the need. Within the HELM results framework, improved capacity in the four core areas and the special initiatives results in increased management capacity in Indonesian HEIs that in turn results in improved education quality, which then results in students who are better prepared for learning and work.20 From an organizational development perspective, this results sequence jumps up the ladder of inference by omitting the need to integrate HELM components as a system within HELM and the HEIs. The extent to which institutionalization of increased management capacity will occur is dependent on the integration of the disconnected components into a system and on the extent of integration and cross fertilization both within and across implementing units of the HEIs during the next two years. Gender and Social Equity HELM has introduced the Women in Leadership initiative and has carried out two workshops for approximately 50 individuals who developed action plans. A book on female role models is also being produced. These activities are in the absence of a USAID/Indonesia contractual obligation. An external project review in 2013 recommended that HELM undertake a gender analysis. At the time of the MTE 20 Performance Management Report Version 4, p. 3. 28 one year later, this analysis had still not been carried out. Using the gender marker coding system, the HELM gender marker code would be1b, which is defined as “The project’s gender marker code21 would be Gender Code 1a: “Boiler-plate” treatment. The project documentation mentions gender, but there is not a purposeful and informed treatment of any or all of the following: 1) gender analysis in the needs assessment that leads to 2) gender responsive activities and 3) gender-related outcomes.” (See Annex X for the complete gender marker-coding framework.) Despite the very limited investment to date in analytical efforts or activities, implementation specific to gender has yielded some startling results. For example, in one HEI, prior to the HEI’s Women in Leadership seminar, six out of 40 members of the HEI’s senate were women. Following the seminar, a large group of women stepped forward and ran for and won senate seats. Today, women comprise 40% (just under half of the members) of the senate. In those HEIs where alumni were able to carry out their action plans, alumni recounted similar results. The fact that a minimal investment can generate change of this scale suggests that women in the higher education system are ready and able to assume leadership positions and that the system and broader enabling environment is now sufficiently welcoming for women to move upwards. While women in the higher education system have increased their visibility and possibilities, people with disabilities in the higher education system (either as staff or as students) remain invisible, so much so that they are not even factored into the discussion of equity. HELM has an opportunity to make a significant contribution in the disability integration, especially that some HEI ARPs are focused on diversity. Other marginalized groups and the specific and unique needs of particular regions of the country have not been factored into HELM implementation to any noticeable extent. HELM Design The evaluation team was asked to consider the appropriateness of the HELM design. HELM is built on the following theory of change (TOC): “HELM support to the Directorate of Higher Education will enable DIKTI to 1) promote the development of key reforms in four core management areas and 2) facilitate the implementation of those key reforms by disseminating strategies tried and tested in targeted HELM HEIs to other higher education institutions across Indonesia to enable systematic improvements in teaching, research, and service so that student achievement and employability are improved so that the students graduating from these institutions contribute to Indonesia’s economic development.”22 This theory of change is still appropriate and relevant to the current Indonesian context. Unfortunately, fundamental links in the chain that should be within HELM’s control have been largely omitted from HELM implementation, even though they were taken into account in the Chemonics proposal. The most glaring omission is the support to the Directorate of Higher Education that would enable DIKTI to manage, sustain and roll out the HELM activities more broadly across the higher education system. Except for predominantly consultative meetings, DIKTI (and National Accreditation Board for Higher Education/ Badan Akreditasi Nasional Pendidikan Tinggi (BAN PT)) personnel have not been engaged in 21 The gender marker codification presented in the IASC Humanitarian Assistance Handbook was adapted by V. Haugen, Y. Zaidi, A. Abdullah in 2012 for a USAID/Pakistan gender assessment and gender analysis. The adaptation was undertaken in order to provide greater obvious distinctions between the gender marker codes. 22 This is the first time, to the MTE team’s knowledge, that the logic chain underpinning the program has been presented as a single, coherent chain. It may be that some adjustment needs to be made by USAID, the GOI and the HELM team to ensure that the logic model presented in historical documents still holds. 29 a substantive way in HELM implementation. While two of the three major initiatives will be embedded several HEIs, DIKTI’s role in quality oversight and further strategic planning with respect to these initiatives is unclear. While the design and the Chemonics proposal are both of very good quality conceptually and practicality, the distortion of the HELM design and the Chemonics strategic implementation plan in the first year of the project has had detrimental effects. The distortion doubled the number of HELM partner institutions without a commensurate increase in the budget or staffing; condensed research activities that should have taken several months to accomplish adequately into very constrained timeframes (e.g., in one instance, the time frame went from seven months to three weeks); and pushed forward activities that had been spread over a sensible and realistic timeframe. In addition, there was significant churn in personnel, including the Chief of Party, Deputy Chief of Party, technical positions and the home office project director role. RECOMMENDATIONS Given the fact that HELM has roughly 18 months of solid implementation time remaining before the processes for the closing of the project begin, the MTE team has included only those recommendations that are the most vital to ensuring that the project leaves a solid legacy for the higher education sub￾sector. Each of the recommendations included needs to be acted on immediately and with urgency. RECOMMENDATION 1: Hold an intensive workshop to focus HELM implementation on consolidation of results from activities, sustainability and institutionalization for the final two years of the project. Without an all-out effort to regroup, develop and embed a strategic plan into HELM, the end results of the project and the overall development impact will suffer. The parameters of this workshop should include the following: Participants • All HELM technical personnel, including sub-contractor experts, for the full-time frame of the workshop. This strategy should be developed collaboratively with HELM’s high-level technical experts from JBS International (JBS), the University of Kentucky (UKy) and the Indiana Alliance to identify synergies and ways to consolidate returns on investment; • Key senior DIKTI stakeholders/counterparts who understand HELM in depth for as much of the workshop timeframe as they can attend; and • Key senior BAN-PT stakeholders and counterparts for selected sessions. Workshop Facilitation • Bring in a skilled and experienced individual capable of facilitating and moderating the process. This individual should also understand HELM and the Indonesian higher education landscape well. Suggested Timeframe • Five days minimum. Workshop Products 1. Action plan: The workshop outputs should include an action plan with a timeline that addresses the 2013 Deliverable 11 (External Review) recommendations. These recommendations were relevant one year ago and they are still relevant and appropriate. The MTE team has made some minor adjustments to ensure that the recommendations reflect the current situation. For ease 30 of reference, these recommendations are included in Annex XVI. 2. Prioritized and rationalized set of activities: Given the limited time remaining for HELM implementation, the significant activities already solidly underway, and the extent to which HELM technical personnel are already stretched, it is prudent to prioritize the project’s activities. Those initiatives that are currently in a formative state and that show little potential for generating sustainable results over the next 18 months should be dropped, even if a significant investment of resources has been directed toward these initiatives. Initiatives that are in this initial formative state and that have little time to come to fruition include the Women in Leadership initiative, the Google E-University and Institute for Development Education and Learning (IDEAL). The exception to this recommendation is the Women in Leadership initiative. Despite the fact that the investment of resources – human and financial – has been limited and the initiative has not been located within a strategic approach to gender integration, the efforts to date have demonstrated the potential to generate significant dividends with respect to gender and development. It is highly likely that the initiative will become the showcase for successful transformation and significant social dividends within and beyond the 50 HELM HEIs, if expert strategic and technical advice is sought and the initiative is handled correctly over the next 18 months. The approach to date has been low-cost at the HELM implementation level and low or no cost at the HEI level. Accordingly, this initiative has the potential to be sustained beyond HELM. In addition to the book under development, it is recommended that a sequel be produced that captures the results that should continue to emerge from the initiative. 3. Sustainability and Institutionalization Strategy: The strategy must: a. Define what these two concepts mean; b. Revisit the original HELM theory of change and make explicit in text and visual form any modifications to the original theory of change; c. Capture and systematize the disparate ways and means HELM is seeking to address sustainability and institutionalization; and d. Include a risk matrix and a set of assumptions that underpin the strategy. 4. Strategic Communications for Development Strategy and Work Plan. Of special importance is the need to take urgent action on a strategic communications platform immediately so that the extensive repository of information and knowledge that has accrued under HELM is disseminated effectively upon platform design and development to different audiences. This strategy should: a. Include a Gallery Walk conference for all 50 HEIs at the end of 2015 where best practices and deep institutional change is shared. This initial effort should be expanded to a national conference, managed by a professional conference organizing company, that showcases the best of HELM results, has pre-conference workshops where individuals can participate in condensed versions of HELM workshops or undertake part of the executive/professional development courseware, is a marketing platform for the executive development courseware and the postgraduate degree programs, and stimulates networking. This conference should be open to any interested HEI personnel and specifically target the participation of associations and their members. High-level consultation with the GOI could result in this conference becoming an annual event. It is strongly recommended that HELM contact STTA with significant experience in strategic communications for development results, including relevant skills in social 31 media and social network analysis; and b. Not include the development of any more video testimonials – unless they are used as a marketing tool for encouraging potential students to engage with the postgraduate program and/or the executive/professional development courses. 5. Scaled down set of indicators: Evaluate the 17 current indicators including some of the standard indicators. For the custom indicators, evaluate their merit against best principles in indicator design and eliminate those indicators that do not conform or restructure them so that they do conform, keeping in mind the challenges of collecting appropriate data. RECOMMENDATION 2: Act immediately to ensure the quality, relevance and appropriateness of the executive/professional development for higher education leadership and management materials. HELM’s assumption is that the executive/professional development courseware for higher education leadership and management that is delivered via a blended learning platform will stand the test of time and sustain HELM investments. This assumption needs to be tested. HELM should engage an internationally recognized expert in cross-cultural instruction design for distance and open/blended learning.23 This individual will: 1. Provide technical input to improve the quality of the instructional materials and teaching methods utilized on the University of Gajah Mada (UGM) web site. The expert should ensure that a recognized quality review instrument is used for the review of all existing materials. 2. Develop and oversee a rolling testing and revision plan for these materials that takes into account use of the materials in different regions of Indonesia, particularly locations where infrastructure concerns prohibit internet-based delivery platforms. 3. Design and conduct a training of trainers program for individuals from all 25 Cohort 1 HEIs so that they can become internal HEI and local technical experts on the HELM content for the core areas and special initiatives. As a possible part of this effort, HEI personnel could be grouped according to institution types/characteristics to bring a differentiated lens to the common core of the HELM materials. HELM could use this engagement with Cohort 1 alumni to introduce performance metrics for the various core areas to consolidate and deepen the benefits of the project. The STTA should work with HELM technical specialists and targeted individuals from with DIKTI and from DIKTI’s stable of external technical trainers. These individuals will be the master trainers of the HELM alumni trainers. RECOMMENDATION 3: Bring the HELM databases up to a generally accepted standard. To do so, contract STTA with bona fide credentials in database design that can apply accepted principles and norms for database design, including the development of unique identification numbers. HELM monitoring and evaluation personnel will then need to clean and re-interrogate the existing HELM data. RECOMMENDATION 4: Prioritize the approach to the 25 new Cohort 2 institutions in the following ways: 23 During the in-country fieldwork period, the MTE team did review the document produced by the HELM expert on blended learning. The document focused on the delivery platforms. This work needs to be framed in an in￾depth review of the quality of the materials and teaching methods. 32 1. Make the ARP initiative the cornerstone of HELM work with these institutions. This could be done by the international STTA working with the 25 Cohort 1 HEIs to institutionalize the ARP process. Establishing an action research unit within the Quality Assurance unit with the intention that the ARP process becomes an institutional vehicle for change and change management could likewise do this. This would build the capacity of the ARP participants to serve as master mentors within their institution to replicate the ARP process to others within their home institution and to colleagues within the 25 Cohort 2 HEIs; and 2. Roll out the executive/professional development courses in the 25 Cohort 2 HEIs using the blended learning platform. Involve the Cohort 1 alumni in the delivery of the courses at their home institutions and as mentors for Cohort 2 HEIs. (See Recommendation 2) These courses complement the action research process and provide more detail about specific aspects of the various core areas. RECOMMENDATION 5: Produce a gender and social equity analysis and gender and social equity integration strategy in collaboration with the 50 HEIs and DIKTI and BAN-PT. An immediate action that could be carried out is to ensure that the Women in Leadership action plans are initiated at all 50 HEIs. The strategy should recognize that HELM has only two years remaining. While the gender and social equity analysis may not make a significant contribution to HELM implementation at this point in time, the output of the exercise may help inform future USAID investments in higher education. Beyond the self￾efficacy indicator, ensure that the HEIs track and report on the number of women who move up the ladder as a result of HELM empowerment activities and change in men’s attitudes about and advocacy for female empowerment. 33 1. INTRODUCTION 1.1 CONTEXT24 Indonesia is the world’s most populous predominantly Muslim nation and is one of the world’s top 20 largest economies. The Indonesian archipelago consists of more than 18,000 islands according to a 2002 study by the Indonesian National Institute of Aeronautics and Space. The Government of Indonesia (GOI) has named 8,844 islands and 922 of those are permanently inhabited within Indonesia’s 34 provinces. Many of these are populated with a highly diverse ethnic population. Indonesia’s higher education landscape is also complex and varied. Currently, there are an estimated 3,700 higher education institutions (HEI) that consist of a mix of approximately 92 public institutions that predominantly fall under Ministry of Education and Culture (MOEC) management and a small array of faith-based institutions that fall under the management of the Ministry of Religious Affairs (MORA) but that are obligated to adhere to MOEC policies and regulations. Private HEIs constitute 96% of Indonesia’s institutions and thus dominate the tertiary education sub-sector. These HEIs are largely unregulated and may be for-profit or not-for-profit entities. The Government spends relatively little on the tertiary education sub-sector. Of the 20% of Gross Domestic Product (GDP) that is allocated by law to be directed to the education sector, the tertiary education sub-sector captures very little of these funds. Of the 1.2% of GDP allocated to the sub-sector, 0.3% comes from public sources and 0.9% comes from private sources (largely tuition) placing Indonesia in the rank of countries internationally that have the highest shares of private funding for tertiary education in the world. A small percentage of the more than 3,700 HEIs - known as the “Big 10” - are able to consistently provide modern education services. Unfortunately, this small number is inadequate to keep pace with the demands of Indonesia’s middle-income status25 and the country’s economic and social development aspirations. The vast majority of Indonesia’s HEIs face significant issues that have deleterious effects on the quality of the education provided and, therefore, ultimately on the quality of the graduates and those graduates’ capacity to contribute to Indonesia’s development. With about 3,000 institutions of higher education under its purview, the MOEC has embarked on an ambitious course to improve its colleges and universities as student enrollment continues to increase. Ongoing reforms are addressing areas such as academic quality assurance and relevance, university management and governance, and university 24 References used to inform the content in this section of the report include: (1) United States Agency for International Development. (2012) HELM: Deliverable 6 – Analysis of Approaches to Improved Quality and Relevance for Higher Education Institution Academic Programs: September 28; (2) World Bank. (2012) Putting Higher Education to Work: Skills and Research for Growth in East Asia. World Bank East Asia and Pacific Regional Report. Jakarta: World Bank; (3) Sakellariou, Chris. (2010) Labor Market Outcomes of Higher Education in East Asia. Department of Economics, Nanyang Technological University: Singapore; (4) Lopez-Claros, Augusto and Mayat, Yasmina N. (2009) The Innovation Capacity Index: Factors, Policies and Institutions Driving Country Innovation. In The Innovation for Development Report 2009-2010. Palgrave Macmillan: New York; (5) Asian Development Bank. (2011) Higher Education Across Asia: An Overview of Issues and Strategies. Manila: Asian Development Bank; (6) Asian Development Bank. (2008) Indonesia: Technological and Professional Skills Development Sector Project. Jakarta: Asian Development Bank; and (7) the website for the Ministry of Education and Culture for the Republic of Indonesia accessed through http://www.kemdiknas.go.id 25 The World Bank categorizes Indonesia, Malaysia, the Philippines and Thailand as middle-income countries based on a range of socio-economic development factors. World Bank Data available at: http://data.worldbank.org/country/indonesia 34 financial management, and include efforts to provide greater opportunities for Indonesia’s young women and its poorer students. USAID supports these areas of policy reform while also helping to strengthen individual institutions so they can excel within the changing environment. Many HEIs also lack the human resources necessary to ensure the relevance and viability of HEIs. Administrators are generally sourced from among existing HEI personnel who do not necessarily have the qualifications or background in administration and management in general. Given the lack of professional development programs and academic degree programs in the field of higher education administration and management in Indonesia, most HEI administrators do not have the knowledge, skills and behaviors associated with effective governance of an HEI. In addition, administrators also continue to carry a teaching load that can create challenges around financial and organizational management and quality assurance related to teaching and learning. More than half of HEI lecturers do not hold a post-graduate degree and receive limited, if any, professional development support to improve their teaching methods. Lecturers typically teach the way they were taught, using a lecture-based format that focuses on a curriculum that is theoretical in nature rather than applied and practical. Pay raises and advancement are based in part on the lecturer’s demonstration that he/she devotes time and effort to community service as well. Both of these aspects can affect a lecturer’s capacity to focus on providing high quality instruction. Research skills are lacking as is the research infrastructure (hard and soft) needed to support rigorous research. The government’s 4.6% expenditure on research and development (R&D) originating in universities is notably low; other Southeast Asian countries invest between 14-16% in R&D. This disparity is correlated to the very high percentage of R&D that is government-generated in Indonesia (81%, compared to 11-23% in other Southeast Asia countries) and to Indonesia’s relatively low rank in the Innovative Capacity Index, where it holds the penultimate place in Southeast Asia only ahead of Cambodia. 26 These issues compromise Indonesia’s ability to be competitive internationally in terms of knowledge creation and to contribute to the development of the country through applied research. HEI lecturers are not well paid and it is not uncommon for these personnel to work several jobs to make ends meet. About 30% of Indonesian youth aged 19-24 (or five million young people) are enrolled in an HEI. 62% of these five million students are enrolled in private HEIs and 38% are enrolled in Indonesia’s 92 public HEIs. One out of every five studies economics, law or a social science discipline, with the majority of public HEI students enrolled in pre-service teacher education programs. There are too few graduates in the hard sciences required for private sector growth and too few graduates with the critical thinking skills and creativity to contribute to overall socio-economic development.27 Employers report that graduates are not well prepared and lack the knowledge, skills and practices necessary to enable them to succeed. Graduates find that it can take up to two years to find a job and 31% of graduates are unemployed, compared to 3-12% of graduates in neighboring countries. 26 United States Agency for International Development. (2012) HELM: Deliverable 6 – Analysis of Approaches to Improved Quality and Relevance for Higher Education Institution Academic Programs: September 28. 27 World Bank. (2014) Indonesia’s Higher Education System: How Responsive is it to the Labor Market? World Bank Policy Brief 89222. Washington, DC: World Bank. Available at: http://www￾wds.worldbank.org/external/default/WDSContentServer/WDSP/IB/2014/07/15/000442464_20140715132755/Rend ered/PDF/892220BRI00P120abor0Market0May02014.pdf. See also Moeliodihardjo, Bagyo Y. (2010). Equity and Access in Higher Education: The Case of Indonesia. Available at: http://siteresources.worldbank.org/INTEASTASIAPACIFIC/Resources/Indonesia￾EquityandAccessinHigherEducation.pdf. 35 Despite previous and current government efforts to improve equitable access to higher education, significant disparities still exist. 51% of university students come from the top 2 income quintiles (Q4 and Q5) and only 14% come from the bottom two quintiles (Q1 and Q2). Despite the expectation of the Directorate General for Higher Education (DIKTI) that 20% of all students in public universities should be from the lower income quintiles, gross enrolment rates (GER) reflect the disparity in access: while the overall GER is 26.6%, the Q1 GER is 1.1%, Q2 is 2.72% and Q3 is 5.64%. Merit scholarships are available students from the bottom quintiles, but these scholarships can be difficult to secure. Poor families are often unwilling to cover the costs associated with applying to an HEI if there is no certainty that the applicant will receive a scholarship: without the scholarship, the applicant would not be able to afford to attend the HEI and the family would have sacrificed limited resources for no gain.28 Financial aid only covers 3% of the cost of tertiary education in Indonesia and student loans are limited. For private universities, per student spending sits at USD$1,200 and USD$2,200 for public universities. These costs put higher education well beyond the reach of many potential students. In addition, poor quality primary and secondary education means that a high proportion of HEI students, particularly those from poor and marginalized parts of the country and from poor families enter the HEI unprepared for the demands associated with higher education studies. Drop out rates can be high and internal efficiency related to on-time student graduation rates is often low in many institutions. Efforts on the part of the government to improve the quality, access and relevance of the higher education sub-sector are noteworthy. These efforts include the passing of in 2012 of Law No. 12 (Higher Education Act). 1.2 PROJECT BACKGROUND The USD$19.678 million Higher Education Leadership and Management project (HELM) is one of two USAID higher education projects initiated between 2010 and 2014. The HELM contract was awarded to Chemonics International and its subcontractors, the Indiana University in alliance with The Ohio State University and University of Illinois at Urbana-Champaign (hereafter referred to as the Indiana Alliance), and JBS International – Aguirre Division (JBS) and the University of Kentucky (UKy). The HELM contract was signed on November 28, 2011 and continues through November 30, 2016. According to the HELM contract, “[t]he ultimate goal of USAID education programs is to collaborate with the Government and people of Indonesia to improve the academic performance of their basic and higher education sectors, so that the education services will be at high quality, and more relevant to the economic and social growth of the country. This goal is articulated as the Assistance Objective statement of USAID Indonesia’s 2010-2014 strategic plan for education: to help ‘students [be] better prepared for success in learning and work.’” HELM sits squarely within USAID/Indonesia’s 2014-2018 Country Development Cooperation Strategy (CDCS), “Investing in Indonesia: A Stronger Indonesia Advancing National and Global Development,” that notes, inter alia, “The higher education sector (which includes polytechnics, community colleges, and teacher training institutions as well as universities) has a critical role to play in both training those who manage essential services and educating future managers, technical specialists, and leaders.” 28 Ibid. 36 HELM targets increased capacity in four core management areas: (1) General Administration and Leadership (2) Financial Management (3) Quality Assurance; and (4) Collaboration with External Stakeholders. Sitting across these four core management areas are four key components: A. Design technical assistance approaches to achieve effective implementation of key reforms across the higher education system, coordinating with DIKTI and maximizing opportunities to internalize best practice within the Higher Education (HE) system; B. Provide technical assistance to increase management capacity and improve performance at partner HEIs and disseminate best practices; C. Strengthen graduate-level programs in higher education leadership and management; and D. Support Special Initiatives by providing assistance to advance reforms and innovation within management of HEIs. HELM serves to build the capacity of higher education leaders and managers at different levels of the institutional hierarchies and representing 50 public and private HEI (i.e., universities, polytechnics and community colleges, a newly established modality for higher education in Indonesia). As illustrated in Figure I, these institutions are located throughout Indonesia and comprise two cohorts of 25 HEI each, one of which started when HELM began and the second of which started in 2014. 1.3 EVALUATION PURPOSE AND QUESTIONS 1.3.1 Evaluation Purpose HELM is at the end of year three of its project life cycle. Accordingly, in mid-2014, USAID/Indonesia issued a request for proposals to carry out the mid-term evaluation (MTE) of the project. The contract was awarded to International Business & Technical Consultants, Inc. (IBTCI), and a five-member MTE team carried out the evaluation between December 1, 2014 and January 23, 2015. Team members included two international consultants, Dr. Valerie Haugen (Team Leader) and Dr. Nancy Lloyd-Pfahl (Evaluation Expert), and three Indonesian consultants, Dr. Dwatmadji (Higher Education Specialist), Ms. Rina Arlianti (Higher Education Specialist-Technical and Vocational Education) and Ms. Erlyn Astuti (Research Specialist). The MTE team was tasked with evaluating HELM’s performance to date, making an assessment of the sustainability and ‘replicability’ of the project’s work, and making recommendations to help guide USAID and HELM to improve performance where required for the remainder of the project. It is anticipated that the MTE findings and recommendations will also provide guidance to the MOEC, DIKTI, and other relevant agencies, including the donor community such as the Australian Agency for International Development (AusAID), and the leadership of other HEIs. The team was requested specifically to evaluate HELM’s progress in: (1) Improving the knowledge and skills of HE staff in the four core areas (2) Supporting the partner HEIs in implementing their thematic action plans, and (3) Contributing to improved processes and systems at the institutional level. 37 1.3.2 Evaluation Questions The MTE questions are organized into two sets: questions that look retrospectively at HELM’s performance to date and questions that look forward over the next two years of HELM implementation. Question Set 1: Looking Back: Review of Achievement of Objectives/Outcomes at Mid-point 1. To what extent has HELM achieved its stated mid-point objectives and outcomes? 2. To what extent have HELM resources been managed and utilized efficiently and effectively across HEI subsets to accomplish the project’s mid-point objectives and outcomes? Why/Why not? 3. What specific value has been added through HELM activities for beneficiaries at the output level, the short-medium term outcome level and the longer-term impact level and to what extent? 4. What aspects of HELM are proving most and least effective in building the capacity of individuals participating in HELM activities? Why/Why not? 5. What aspects of HELM are proving most and least effective in contributing to the organizational development of the HEIs participating in HELM activities? Why/Why not? 6. How effective has HELM been in improving women’s participation across the components? Why/Why not? 7. Are the effects of HELM on 50 HEI contributing to improving the overall enabling environment for higher education? Question Set 2: Looking Ahead to Project Endpoint and Post-Project Impact (Sustainability and Replication) 1. Is HELM on track to achieve its overall end-of-project goals across all HEI subsets? Why/Why not? 2. What HELM activities and results are likely to be sustained and/or replicated after the project is completed? Why/Why not? 3. Is the HELM project design, structure and approach suitable for achieving the desired results? Why/Why not? 4. If the HELM design, structure and approaches are not suitable, how should these be reorganized? 5. What, particularly, is needed to promote Phase 3 institutionalization? To the extent possible, the MTE team also attempted to identify commonalities and distinctions between various types of HEIs with respect to the way in which HELM activities are undertaken and the changes in the respective HEI types. The team was also asked to identify commonalities and distinctions between HEIs (regardless of the type of HEI) in different regions of Indonesia and in (ultra) urban versus more removed locations. The evaluation was to be conducted in accordance with USAID’s policies and procedures for evaluations and take into consideration the principles underlying USAID’s Human and Institutional Capacity Development (HICD) policy. 38 1.4 EVALUATION METHODOLOGY 1.4.1 Type of Evaluation The MTE was a formative evaluation study that utilized a semi-collaborative approach.29 Although the MTE was not originally intended to be a collaborative evaluation, the MTE team made an effort to introduce this dimension to enhance Chemonics International’s understanding and ownership of the MTE findings and conclusions as well as engender a commitment to implement the proposed recommendations. The MTE utilized a combination of qualitative document review, key informant interviews, and semi￾structured surveys to collect the data used to inform the findings, conclusions and recommendations presented in this report. These methods were chosen to enhance the quality of the data collected because each method emphasizes a contextual approach. The MTE team emphasized qualitative data collection to ensure that differences across HEI types, core areas and gender were captured, and triangulated the data as primary and secondary data collection sources were analyzed and reported. Please refer to section 1.4.3 Data, below, and the Work Plan in Annex II for a detailed description of the methodology employed to execute this mid-term evaluation. 1.4.2 Participants The team visited a sample of 18 of the 50 participating HEI, purposively selected to represent all types of HEI and both cohorts. As illustrated by Figure 2, above, these HEI are located throughout Indonesia. The team interviewed 277 individuals during the course of the MTE, including 90 women and 187 men from Aceh, Central Java, East Java, Maluku, North Sumatera, Papua, South Sulawesi, West Sumatera, West Kalimantan, and West Papua provinces and Jakarta. The majority of the MTE participants were HELM alumni from 18 HEIs with 158 individuals (29 women and 129 men) in management positions and 34 individuals (15 women and 19 men) in lecturing positions. Figure 3 below provides a breakdown of the participant demographics. 29 A collaborative evaluation “implies a varying level of involvement that considers the extent to which program staff and other stakeholders should be included as part of the evaluation team...is often empowering to participants…[and] enhances their understanding of evaluation so they gain new skills...promotes utilization of evaluation findings.” See O’Sullivan, Rita G. (2004) Practicing Evaluation: A Collaborative Approach. SAGE Publications, Inc: Thousand Oaks, CA. 39 Figure 3: MTE Participants Disaggregated by Sex and Position Title 1.4.3 Data 1.4.3.1 Primary Source Data Collection and Analysis Qualitative and quantitative data were collected and analyzed. These data were gathered using a variety of research methods including group discussions using a semi-structured questionnaire and individual participant interviews using a semi-structured questionnaire. The MTE team adapted the recently distributed HELM online opinion survey and used the categories included in that survey to ensure that HELM participants’ perspectives on human and institutional capacity development dimensions considered important by HELM were obtained. A Findings Matrix was developed and populated based on these categories as well. This Findings Matrix formed the basis of an in-depth pattern analysis to identify the Key Findings included in this report. The MTE collected quantitative data through: (1) an opinion survey that was distributed to all MTE participants; and (2) a Gender and Social Equity questionnaire that was distributed to all 18 MTE HEIs in order to obtain a picture of the demographic profile of HEI personnel and students. Unfortunately, only seven HEIs returned the questionnaire. In addition, the MTE team cleaned and re-interrogated the raw data contained in the HELM Individual Trained Tracking Summary for Fiscal Years (FY) 2012-2014 database and the Summary of Participants by Activity and Core Program for FY 2012-2014 database. 1.4.3.2 Secondary Source Data The MTE team reviewed a wide range of secondary source materials including, but not limited to: those specific to the project (e.g., USAID request for proposals, Chemonics’ technical proposal, contract and amendments, project annual and quarterly reports, documents related to training and similar activities specific to each of the four core areas and special initiatives, special studies, etc.); studies and project documentation produced by other donors and development partners; documents produced by the Government of Indonesia (GOI); and international literature on human and institutional capacity Rector Director Vice Rector Vice Director Vice Rector Team Manager International Office - Chair International Office - QA Unit - Chair QA Unit - Member Institution Center - Chair Institution Center - Member Research and Service - Research and Service - Bureau - Chair Internal Supervisory Unit - Procurement Unit - Chair Procurement Unit - Dean Vice Dean Department - Chair Department - Member Study Program - Chair Lecturer Unit - Chair Unit - Member Total HELM Alumni Others HELM Staff 6   4   20   4   6   2   4   1   11   9   10   3   7   1   5   3   2   2   13   5   2   0   8   19   13   9   9   9   0   0   2   1   0   0   0   2   6   3   1   1   6   1   0   0   0   0   3   7   2   1   6   15   5   8   7   13   Male   Female   40 development including adult education, distance/open/blended learning, evaluation and higher education sub-sector and systems development. 1.4.3.3 Validity and Reliability The MTE team captured data in written and oral forms with strict attention paid to recording the informant’s speech exactly. Where the speech or the meaning was not clear, the team sought clarification with non-leading, non-evaluative follow-up questions. Data were triangulated and qualitative data were subjected to an intensive pattern analysis to identify key findings that could form the basis of the conclusions and recommendations presented in this report. The MTE team cleaned (to the extent possible) the quantitative data, then analyzed and represented the data in graphs, charts and tabular form. 1.5 EVALUATION LIMITATIONS Limitation One: The MTE timeframe included several weeks over the holiday period (December 18, 2014 to January 5, 2015). Internal USAID/Indonesia considerations precluded postponing the evaluation until after the holiday season. The MTE team managed to undertake technical preparation, communication with HELM HEI partners, logistical arrangements and the bulk of the site visits in a highly constrained timeframe. The constrained timeframe placed significant stress and demands on HELM, HEI and MTE personnel. Further, this resulted in a bifurcated rollout of the MTE overall and made the formulation of a coherent view of HELM implementation challenging. Limitation Two: Internal USAID/Indonesia constraints, including coordination with the Office of Acquisition and Assistance and the delayed release of the HELM MTE Scope of Work (SOW) to the HELM staff hampered full participation and understanding on the part of all parties in the early stages of the evaluation. The lack of early, informed involvement by all parties and full ownership in planning for the evaluation further complicated the issues raised in Limitation One. Limitation Three: The inherent nature of HELM with respect to its geographic scope, contextual differences that are dependent on geographical location, the range of type of institutions involved in the project and the limited number of certain types of institutions (such as three community colleges) made purposive sampling unrealistic and necessitated a plan that accommodated site visits to around 40% of HELM’s partner institutions. In ordinary circumstances, where there is greater homogeneity among entities, targeting 19 out of 50 institutions for site visits would be considered more than ample. The allocation of time for future monitoring and evaluation work should take these issues into account. Limitation Four: The MTE team had intended to visit two additional HEIs for a total of 19 site visits. However, due to scheduling constraints and conflicts, the site visits to the Aceh Polytechnic and the University of Gadjah Mada / Universitas Gadjah Mada (UGM) in Cohort 1, and the State University of Medan/ Universitas Negeri Medan (UNIMED) in Cohort 2 were cancelled. The team interviewed the Director of the Aceh Polytechnic in Jakarta over the Christmas break and had several telephone conversations with one of the key individuals at UGM. While these circumstances were not ideal, the team was able to collect in-depth, contextual information from these two institutions. No interviews with UNIMED personnel were attempted. Although the team extended apologies to all concerned, the situation was not ideal given the limitations cited above. Limitation Five: The 2 HELM databases were not easily searchable and required a thorough cleaning and re-analysis of the data by the MTE research specialist in order for the MTE team to utilize the data during the fieldwork and to provide USAID/Indonesia with reliable conclusions about HELM’s reported achievements. 41 Limitation Six: Despite significant, ongoing efforts to ensure that each of the HEIs visited by the MTE team returned the Gender and Social Equity Questionnaire, only 7 HEIs returned the form by the end of the in-country work. Consequently, these data are less comprehensive than expected and provide only a partial picture of certain gender and social equity considerations. Limitation Seven: The MTE team’s reliance on HELM personnel as interpreters placed an unnecessary and avoidable burden on HELM personnel and resulted in a less than ideal data collection process. 42 2. KEY FINDINGS AND CONCLUSIONS 2.1 HELM ACHIEVEMENTS 2.1.1 Achievements Against HELM Targets Key Findings: On the basis of HELM-reported achievements, the project is on target or has exceeded the targets for its current 17 indicators. The November 2013 external evaluation report also mentioned the same finding for indicators used in years one and two. In some cases, the achievements have been far in excess of Life of Project (LOP) targets. Indicator 3 – the proportion of women who report increased self-efficacy at the conclusion of United States Government (USG) supported training/programming – is a case in point. The LOP target was 17 women and by the end of year three, a cumulative figure of 277 women had reported increased self-efficacy. Annex IV includes a detailed summary of HELM indicators reported against between project years one and three. Indicators used by HELM to report progress have varied from year one to year three. In year one, HELM had 17 indicators. Ten of these indicators were carried over to year two and the other eight were dropped. These 10 year two indicators were then carried over into year three and another seven indicators were added for a total of 17 indicators. These 17 indicators are a mix of standard and custom output-based indicators.30 A number of issues with data quality and reporting arose over the course of the evaluation. The indicators used previously and currently by HELM are predominantly output indicators with several outcome indicators. Although several of these indicators are standard USAID-designated indicators, the construct of both the standard and the custom indicators does not conform to good practice in indicator development. For example, new year three indicators related to accreditation status are beyond HELM’s sphere of influence since accreditation processes may be interrupted due to lack of budget, lack of assessors, etc. There have been discrepancies in reporting of achievements. HELM personnel recognize the issue and HELM has worked to improve its data quality and reporting of data. In 2013, HELM notified USAID of problems associated with reporting data for Indicator 1 – the number of host-country individuals trained in Indonesia as a result of USG investments involving HEIs – and rectified the incorrect reporting of data. In 2014, HELM reanalyzed its databases in a further effort to enhance data quality and reporting against indicators. Despite the efforts to improve the quality of data entered and reported on, issues remain. There are a number of database entries that appear to be duplicates or triplicates of one data point. In preparing for institution visits, the MTE team assembled lists of HELM alumni. It became apparent that an alumna/alumnus was oftentimes listed twice, thrice or even four times on the alumni list because of 30 In recognition of the limitation of the majority of standard indicators to contribute to project strategy review and implementation decisions, USAID/Washington encourages projects to craft custom indicators. These custom indicators can be used to capture and report against outputs and outcomes of particular relevance to the respective project in the short-, medium- and long-term 43 variations in the recording of the individual’s name. The MTE team undertook a data cleaning exercise to produce a reliable list per institution. As such, the HELM database now has cleaned alumni data for 17 HEI. Data associated with workshops are of poor quality and difficult to interrogate both within the databases and from the quarterly reports. For example, there is variation in the dates reported for workshops. The evaluation team worked with HELM colleagues to produce a comprehensive table of workshop and forum activities. However, the table has not been fully and accurately populated due to a lack of data available to the evaluation team. The results and LOP targets are not consistently reported across the annual reports. The MTE team was unable to conduct an exhaustive assessment of the HELM databases. Best practices include, but are not limited to, advice below. Based of the team’s use of the HELM databases to extract key data, there are indications that the databases show do not conform to best practices regarding database development.: • Always have a primary and succinct key identified within the database that is unique to the record, mandatory for all entries and immutable to changed,– preferably not using a tax identification number;31 • Design and/or maintain the database using carefully followed normalization procedures;32 • Relational databases such as SQL or the ANSI core SQL are preferable to using other database formats because they are relational and stable platforms; • Use consistent column name suffixes and prefixes; • Avoid using one table to house all domains of information; and • Avoid using identity/grid columns as the only primary key.33 Conclusions: The MTE data suggest that HELM has indeed produced outputs and outcomes and it is very likely that HELM has achieved beyond the annual or even the LOP targets set for the various indicators. However, for a number of indicators, the evaluation team lacked reliable data to validate the HELM figures reported. While the latter is a matter for concern, a more thorough upgrading of the databases and a re-interrogation of data should rectify the specific issues raised above. Distinct from issues around data quality and reliability is the concern that current and previous HELM indicators, overall, do not enable HELM to demonstrate (or evaluators to validate) achievements that go beyond a numeric tally of outputs. The issue of reporting “numbers of” (i.e., the bulk of F-drive indicators are actually tallies of numbers) and the inadequacy of these numbers to enable a project to reveal its actual achievements are not unique to HELM. Since USAID/Washington requires projects to select a set of these standards indicators, a project is contractually obligated to report against such indicators. However, the introduction of custom indicators enables a project to utilize indicators that are more relevant to its strategic planning and implementation needs. There are, however, two issues with respect to outcome monitoring and reporting. First, the custom indicators introduced for HELM in 31 If the table constructed to hold the database has multiple rows that reference an entity, a unique identifier should be maintained for each entity to ensure the integrity of the database. 32 Normalization will define the domains so that they can easily be mapped to the objects in the database. Normalization is difficult but not impossible to achieve the Microsoft Excel spreadsheets that HELM uses to store its databases. As a general rule of thumb, if a table contains more than 20 fields it may be improperly normalized. 33 For more information on this topic, see Petrenko, Maksym; Rada, Amyris; Fitzdimons, Garrett; McCallig, Enda; and Zuzarte, Calisto. (2012) Best Practices: Physical Database Design for Data Warehouse Environments. IBMCorp, among other sources available online and through the federal government’s knowledge management initiatives. 44 Q2 of year two do not conform to best practices in indicator development. This issue means that the data associated with these indicators, thus, become problematic as well. Second, those data that are more pertinent to monitoring and measuring important outcomes and impact are not easily accessible in the HELM monitoring and evaluation system. This is not to say that such data are not collected. Rather, it is to say that these data are buried in the system, are not captured coherently or mined comprehensively and systematically against analytical frameworks, and then presented in reports. The matters above contribute to the inability of HELM to: (1) tell its story convincingly through its data; and (2) use these data to inform strategic planning and implementation considerations. There is a lack of coherence between the annual and LOP targets in a number of cases is concerning. This situation suggest that there are problems either with the indicator(s) as it is constructed and/or a lack of understanding of the features and characteristics of the respective context. These problems then result in setting an under-ambitious and implausible LOP target. This, in turn, leads to the opportunity for a project to claim significant success; however, the claim may not be credible for the reasons given above. For example, in a context where many personnel, particularly, lecturers, have limited opportunities to participate in professional development activities and where the number of personnel in a given organization is large, it is not particularly difficult to attract individuals to participate. While it may be necessary for HELM to include some standard indicators to enable USAID/Washington to aggregate data globally for reporting to the USG and a project has no room to modify these indicators, there is an opportunity to get at indicators that sit underneath the broad USAID/Washington standard indicators. For example, while HELM may report on Indicator I to serve USAID’s overall needs, it could set indicators that are fit for project purposes. Examples include data on the persistence rates of the participants and the positions held by participants – disaggregated by sex and disability status – to gain a clear understanding of who is participating in what professional development activities. Both of these dimensions are important for gauging project success and for strategic and tactical planning. Both are also easy to capture, provided the project has a viable method for doing so such as a unique identifier number that captures a range of data within the number itself. In addition, these dimensions would provide HELM with more substantive data on the effects of HELM on women, if they were tracked over time. Overall, HELM could tell its story more convincingly and to greater effect by: (1) utilizing custom indicators that sit beneath the standard indicators; (2) ensuring that indicators that are within the control of HELM to construct conform to best practices in indicator development; and (3) identifying more plausible annual and LOP targets. 2.1.2 Achievements Against the HELM Theory of Change In order to examine the extent to which HELM has addressed or could address the various links within links within the theory of change underpinning the project, it was necessary to actually construct the theory of change. The theory of changed is typically described in a number of paragraphs in various USAID and HELM project documents. When these paragraphs are condensed into a single logic chain, the result is presented below in Figure 4. The HELM theory of change is driven by the 4 core areas: general administration and leadership; financial management; quality assurance; and partnerships and external collaboration. Through each of these core areas, HELM targets specific, individuals, institutional units and HEIs through which to enact change. The HELM theory of change is further supported by continued collaboration between DIKTI, HELM, the HEIs, and the individuals that populate these organizations. Figure 4 below provides a visual illustration of a somewhat scaled down representation of the logic chain. 45 “HELM support to the Directorate of Higher Education will enable DIKTI to: (1) promote the development of key reforms in four core management areas; and (2) facilitate the implementation of those key reforms by disseminating strategies tried and tested in 50 targeted HELM HEIs to other higher education institutions across Indonesia to enable systematic improvements in teaching, research, service so that student achievement and employability are improved so that the students graduating from these institutions contribute to Indonesia’s economic development.” Since HELM’s design is formulated around the principles of human and institutional capacity development, the MTE team selected a “levels of analysis” framework that provides a viable construct for discussing change in individuals and change in institutions. The levels of analysis are captured in Figure 5 below and discussed in the section that follows. 34 2.1.2.1 Individual Level Achievements More than 1,200 HELM alumni have taken part in HELM capacity development activities. The MTE team examined changes in individuals’ knowledge, attitudes, beliefs, skills, behaviors and application of knowledge and skills. The majority of alumni participated in predominantly two-part F2F workshops targeted to addresses relevant issues among the four core areas of general administration and leadership, financial management, quality assurance, partnerships and external collaboration. In Part 1 of the workshop model, alumni were exposed to new content, methods and instruments and developed action plans to be implemented on their return to their respective institutions. Part 2 of the workshop, scheduled approximately three months after Part 1, provided alumni with the opportunity to share their progress against their action plans and to obtain advice and guidance from other HEI and HELM personnel. Key Findings:35 All but four respondents out of 249 HEI personnel interviewed reported that HELM alumni had improved their knowledge, kills, behaviors, and undergone some changes in attitudes and beliefs. Of the 194 individuals who submitted responses to the MTE opinion survey, a majority of individuals reported that their knowledge, attitudes, beliefs and behaviors improved as a result of participating in HELM workshops. Respondents reported that they felt the greatest gains were in their attitudes and beliefs about their work and the knowledge they had gained. 66% (128 people) of respondents reported that they were applying their new knowledge and skills regularly. No respondents reported that they were applying their skills and knowledge very little. Between 1 and 4% of respondents reported that there was very little change in their attitudes and beliefs. Data from the F2F 34 This is the first time, to the MTE team’s knowledge, that the logic chain underpinning the program has been presented as a single, coherent chain. It may be that some adjustment needs to be made by USAID, the GOI and the HELM team to ensure that the logic model presented in historical documents still holds. 35 As noted in the Evaluation Methodology, the MTE team used a levels of analysis approach to understand the effects of HELM and when considering the effects on individuals, the team considered changes in knowledge, attitudes/beliefs, behaviors and skills (KABBS) and how these changes were being utilized on the job. Figure 4: HELM Scaled Theory of Change 46 Figure 5: Levels of Analysis Approach interviews strongly support the results of the opinion survey. When asked whether their knowledge increased as a result of their HELM participation, 85% of respondents stated that their knowledge had increased. 69% (133 people) reported their knowledge increased quite a bit and another 16% (32 people) stated their knowledge increased a “huge amount.” Speaking to increased skills as a result of HELM participation, the results are similar. 66% (127 people) of respondents stated that their skills increased quite a bit and another 9% (18 people) reported their skills increased a “huge amount.” Often these skills were linked to the introduction by HELM of instruments and tools (e.g., the Triple Helix). Across respondents, it is interesting to note that regardless of how the respondent self￾reported their HELM skills and knowledge, 163 of 194 core area participant respondents to the MTE opinion survey identified themselves as learners. A lesser total number within the core area participants,(149 of 194 respondents) saw themselves as disseminators of information to other colleagues in their implementing unit. These data are captured in Table 1 below. The sphere of influence that individuals noted they could affect lessened the further away people moved from themselves and their implementing units to beyond the walls of their institution. A greater percentage of participants in the core area activities felt they had been able to disseminate knowledge and skills to colleagues in other HEIs than those participants in the special initiatives. Table 1: Participant Perspective on Their Role Because of HELM Participants Who Saw Themselves as: Core Area Participants Special Initiative Participants Learner 163 65 Disseminator to Implementing Unit Colleagues 149 52 Disseminator to Colleagues Outside Implementing Unit 131 35 Disseminator to Colleagues in Other HEIs 102 29 N=194; Source: MTE Opinion Survey The results of the opinion survey also show that of the 179 responses to the question about whether other non-HELM alumni have taken up new ways of working, 35% of the responses were, “quite a bit.” During F2F interviews, alumni consistently noted that they would be able to apply their skills and knowledge even more if HELM and their HEI management had set performance targets and had introduced dissemination strategies. There were some cases mentioned by alumni of individuals who had participated in core area activities being transferred to another area of the institution where they could not use what they had learned from HELM. There were numerous instances mentioned where substitutes were sent in place of the desired attendee. On return to their institution, these substitutes were often unable to apply what they learned for a variety of reasons including lack of authority, work in another part of the institution, lack of interest, etc. For example, one alumnus attended workshops in all 47 four of the core areas and often went as a substitute workshop participant for the rector of his institution. He noted that he will be retiring in a few months and he did not think that it had been a good investment having him attend the workshops. Three rectors and directors who had attended workshops noted that they found the workshops very useful and their staff noted that they saw improvements in the individuals’ leadership style and practices. All rectors and directors who were interviewed said that they have not had time to learn through the HELM approach. Aside from HELM’s interaction with a limited number of senior personnel, there is little evidence of HELM’s impact on individuals within DIKTI or BAN PT. This lack of intensive engagement with personnel from these two organizations was raised as a concern by the leadership of the two organizations. Conclusions: HELM has clearly had a positive effect on individuals who attended the HELM core area activities. In addition to the opinion survey results, there is strong anecdotal evidence of the positive effects of HELM on the ways individuals are carrying out their work responsibilities. There is also strong evidence that individual HELM alumni are disseminating what they have learned to other colleagues both within and without their respective institutions. The absence of a systematic approach by HELM to capture these positive changes at the individual level as learners and disseminators makes it impossible at this point in time to quantify results and benefits. A systematic approach would have enabled HELM to recognize the importance of engaging intensively with a range of DIKTI personnel to embed the range of HELM activities within the directorate’s strategic plan and to work toward a scale up and roll out of these activities more widely. A systematic approach would also have: (1) provided a structure for building interdependencies between the four core areas; (2) lessened or eliminated the ad hoc nature of workshop design and follow-up; and (3) set out an integrated training agenda that took account of sound training design, implementation and knowledge transfer principles and practices. While HELM ultimately has no control over the individuals who attend the activities, a needs assessment could have helped mitigate problems that emerged around the relevance of workshop content and substitutes attending activities instead of targeted individuals. Analyses of reliable data on participation would have enabled a better understanding of who was attending what and why. The situation is regrettable; these changes that HELM has helped to foster constitute a significant contribution to the development of the respective HEIs and the higher education sub-sector more broadly. More benefits could have emerged with more intentional and systematic planning. 2.1.2.2 Implementing Unit and Institution Level Achievements Key Findings: Results from the MTE opinion survey show that 59% of respondents felt that their work group improved “quite a bit” because of HELM and 35% felt their group improved “somewhat”. These results are consistent with the data from interviews. With respect to HELM’s contribution to broader institutional change, 54% of participants felt that their respective institution had improved “quite a bit” and 35% felt it had improved “somewhat”. Aside from some examples of widespread efforts to improve accreditation processes across all faculties and an effort to familiarize all deans to procurement processes and the ARP studies, it was difficult for participants to provide examples of activities that have led to widespread change at the institutional level. HELM documentation does not provide any systematic capture and reporting on institutional level change, so triangulation of the survey data was difficult. There are several examples of good practices with respect to enhancing the capacity of implementing units and effecting HEI transformative change. Alumni gave the following examples of initiatives that have made a real difference to their units and to the institution more broadly. 48 § State Polytechnic of Medan / Politeknik Negeri Medan (POLMED): Procurement personnel asked the Internal Audit Unit personnel to join in on the procurement process. By doing this, the procurement unit felt more secure in carrying out procurement processes and the negative opinions held by staff of each unit about the other were reduced. § Andalas University / Universitas Andalas (UNAND): Strong support from the Rector and coordination between the Vice Rector II (General Management and Finance Affairs), Vice Rector I (Academic Affairs), and Quality Assurance (QA) has resulted in the development of the first set of courses that apply student-centered, adult learning approaches. § State University of Semarang / Universitas Negeri Semarang (UNNES): Close collaboration between the Finance Management, QA and Information and Communications Technology (ICT) units has resulted in the development of an integrated Academic and Finance Management Information System. § Muhammadiyah University of Malang / Universitas Muhammadiyah Malang (UMM): The finance unit collaborated with the HEI management to implement and begin using a new accounting system that has eight budget line items, compared with old system which only had one. When the MTE team asked if these change initiatives were due to HELM, alumni said the initiatives had come about independently of HELM (usually before HELM’s input). Personnel at nearly all 18 of the HEIs noted that HELM activities built on prior and, in some cases, current investments in Human and Institutional Capacity Development (HICD) made by the GOI, the senior and middle level administrators of the respective HEI and other donors and development partners. For example, several of the HEIs located in Eastern Indonesia noted that they had benefited significantly from the Government of the Netherlands’ Netherlands Organization for International Cooperation in Higher Education (NUFFIC) project and personnel from several other HEIs mentioned the World Bank Indonesia Managing Higher Education for Relevance and Efficiency (I-MHERE) project. Many alumni noted that while HELM has not introduced any completely new ideas and practices, the inputs have been a catalyst and an enhancer – “HELM has helped us move things forward faster.” Alumni at most HEIs visited said they do not have intentional, structured follow-up programs in place to ensure the greatest benefits are derived at the implementing unit and HEI levels from HELM inputs. If follow-up does happen, it is usually because an individual alumnus/alumna has a personal commitment to sharing and improving work processes. In some HELM HEIs, groups within implementing units have formed in order to share and apply the knowledge and skills acquired from HELM workshops. However, alumni do not attribute these actions to HELM or to efforts of HEI leadership. At each of the HEIs visited, a majority of HELM alumni met with the MTE team. In all but three HEI cases, alumni were startled to realize that their institution had so many HELM alumni and alumni from across the four core areas had had no interaction with each other until the MTE group discussion. In a few cases, alumni from within the same core area were unaware that other core area colleagues were HELM alumni. One alumnus said, “I thought only the three of us attended a HELM workshop on QA. I did not know there were others who attended other HELM workshops.” This was a commonly expressed view. Most alumni wondered why HELM had not brought alumni within an institution together. All alumni participating in the MTE group discussions noted that they learned a lot from the MTE group discussion and developed new ideas about how they might collaborate with other alumni in their institution. Some HELM personnel noted that it was the responsibility of the HEI leadership to ensure that such collaboration took place. The Chemonics proposal includes a strategy for establishing change agent groups at each HEI. To date, this strategy has not been implemented. The MTE team was unable to get a clear understanding of why this was the case. 49 There were some notable exceptions in at least three of the 19 HEIs, including in one of the Cohort 2 institutions where HEI leadership had a clear vision of how to exploit the HELM investment in human capacity development for the overall benefit of the institution. One rector mentioned that, “After the alumni return from the workshops, they have to write a report and then disseminate what they have learned to other staff as part of their follow up plan.” Some HEIs have provided funding for HELM alumni to disseminate and embed their new knowledge, skills and ways of working within the respective implementing unit or even beyond. However, there are also several instances of funding having been earmarked but then shifted to another HELM activity instead. There are also numerous instances of funding being attached to an individual who is in a position of authority with some control over a budget; if that individual is no longer is in the position of authority, or if the section that is financing the HELM activity closes down, the funding for the respective activity is no longer available. For example, a Vice Rector who was participating in the ARP was funding the activity out of his unit’s budget. When his unit was shut down due to the need of the HEI to conform to a specific DIKTI policy, the ARP activity funding ceased. Since this situation occurred, the HEI has not been able to make any progress on its ARP plan. While this example is anecdotal and does not indicate that funding systematically "disappears," it suggests that to ensure implementing unit consistency in plan implementation, the budget must also align with such efforts Specific suggestions were made by HEI leadership based on their experience with other projects that they would like HELM to consider: a more rigorous eligibility and selection process for participants; assistance in developing performance targets linked to an institutionalization and sustainability plan; joint planning sessions between rectors/directors and HELM personnel early on in the budgeting cycle to ensure HELM activities are included in the HEI annual plan and budget; and a clear agenda of activities available well in advance of the implementation date. They also noted that the top leadership of the HEIs is extremely busy and other methods of engagement beyond long F2F workshops would be of benefit. Several leaders mentioned that they would welcome professional development information through social media such as What’s App, LinkedIn, Facebook, etc. A number of personnel also noted that they received and read interesting articles from USAID/Indonesia through email. Conclusions: There is solid anecdotal evidence that HELM inputs have had an effect on implementing units across the four core areas at all the HEIs visited. However, the lack of performance targets and a structured mechanism for monitoring and evaluating the results of the inputs and outputs is a liability that needs immediate attention. The suggestions from HELM alumni are sensible and based on prior experience with other donor initiatives. The MTE team felt that a project such as HELM that aims to enhance the functionality of higher education institutions that are notoriously insular and that work internally in siloes would strive to make greater efforts to ensure that the project itself does not mirror those siloes. The lack of interplay between and across the four core areas and with and between the special initiatives results in a situation where the sum of the parts is much less than the whole. Pigeon holing leadership in Core Area 1 rather than embedding it as a cross cutting programmatic theme along with group processes and gender and social equity diminishes HELM impact at both the implementing unit and the institutional level. The most effective institutional capacity development activity – the Action Research Program (ARP) – provides a model for working around siloes, utilizing group processes and focusing on tangible performance targets. The ARP is discussed at length in section 2.1.3 below. 2.1.2.3 Enabling Environment Level Achievements The MTE team focused on the enabling environment within which HELM must execute its contract given that: (a) the influence of actors and other factors within the enabling environment external to projects like HELM impact a project’s ability to generate the type of long-term benefits HELM seeks to create; and (b) the HELM contract, as discussed below, specifically references HELM’s relationship to actors within the enabling environment. 50 The enabling environment is “a set of interrelated conditions – such as legal, bureaucratic, fiscal, informational political, and cultural – that impact on the capacity of development actors to engage in development processes in a sustained and effective manner.”36 Annex XII contains a table of commonly agreed upon features of the enabling environment divided into five categories or factors: economic, political, administrative, socio-cultural and resources.37 These categories provide the framework within which the MTE team conducted its analysis. Derick Brinkerhoff38 utilizes this framework to understand the fit between donor projects and the environment within which the project is situated.39 Brinkerhoff (2007) examines the intersection of economic, administrative, political and socio-cultural aspects of the policy sphere in a country, as well as the available resources to better isolate and understand the barriers and/or boosters within the environment that might hinder and/or foster implementation. In utilizing this framework, the MTE team recognizes that HELM is neither fully responsible nor directed to influence the entirety of the enabling environment within Indonesia. This is an unreasonable expectation and one not supported by the HELM contract. Based on the HELM contract cited below, however, HELM plays a critical capacity building role not just within the HEI but also through its relationship with DIKTI. The HELM contract notes the following expectations: “As further specified in Section C.4.4, ‘Expected Results, Required Activities and Deliverables’, Activities 8-13, the project will need to address the above assumptions by strengthening the capabilities at DIKTI to ensure that: • effective approaches to leadership and management are developed and implemented at a select set of strategically identified higher education institutions; • that such approaches are disseminated to and incorporated by other higher education institutions; 36 Thindwa, Jeff. (2001) Enabling Environment for Civil Society in CDD Projects. Washington, DC: Word Bank, Social Development Family, CDD Learning Module, April 19, page 3. Available at: http://www.worldbank.org/participation/enablingenvironment/EnablingenvironmentCECDD.pdf. Within the literature there is a ubiquitous use of the term “enabling environment” as it is the topic of conferences, World Bank training programs, and USAID and other donor-supported technical assistance projects. A few examples include: (1) InterAction Symposium. (2002) Creating an Enabling Environment for Achievement of Millennium Development Goals, Washington, DC; (2) The Nonprofit Partnership Conference. (2003) An Enabling Environment: The Legal and Policy Framework Required for a Vibrant NPO Sector, Johannesburg, South Africa; (3) United States Agency for International Development/Nigeria. (2004) Request for Assistance No. 620-04-003, Enabling Environment; and (4) Brinkerhoff, Derick. (2007) Capacity Development and Fragile States. Maastrich, Netherlands; European Center for Development Policy Management. 37 For useful treatments that aggregate and synthesize research findings, see: World Bank. (1985) Sustainability of Projects: First Review of Experience. Report No. 5718. Washington, DC: Operations Evaluation Division; World Bank. (1997) The State in a Changing World. World Development Report 1997. New York: Oxford University Press; World Bank. (2002) Building Institutions for Markets. World Development Report 2002. New York: Oxford University Press; and United Nations Development Program. (2002) Deepening Democracy in a Fragmented World. Human Development Report 2002. New York: Oxford University Press. 38 Brinkerhoff, Derick. (2007) Capacity Development and Fragile States. Maastrich, Netherlands; European Center for Development Policy Management. 39 See also Eade, Deborah. and Williams, Suzanne. (1995) The Oxfam Handbook of Development and Relief. Oxford: Oxfam; and Morgan, Peter R; Land, Anthony; and Baser Heather. (2005) Study on Capacity, Change and Performance: Interim Report. Discussion Paper No. 59A. Maastricht, The Netherlands: European Centre for Development Policy Management. 51 • that the approaches are applied in a manner that improves teaching, research and service in one or more academic disciplines at each institution involved in the project; • that DIKTI develops strategies, plans and capacity supportive of the above and the broader higher education reform effort in Indonesia; • that DIKTI develops and implements policies that provide incentives for the above; • that DIKTI develops the capacity to monitor implementation of the above; and • that DIKTI develops the capacity to directly assist the dissemination and incorporation of the management and quality higher education approaches developed under this project. (Contract # AID-497-C-12-00001 Higher Education Leadership and Management – HELM - Project Page 32 of 103). The enabling environment in Indonesia is supported and fostered by the Higher Education Act40 and other regulations, MOEC, DIKTI, BAN-PT and HELM among others. These entities provide the participant HEIs the opportunity to interact with a diverse and geographically dispersed set of governing bodies and HEIs despite the HEI’s funding source. Speaking about HELM’s role, some MTE participants cited an expectation that HELM should play a bridging role between the HEIs and their experiences as policy implementers, and DIKTI and BAN-PT as policy developers. MTE participants also expected that HELM would work with and through already established networks such as the Rectors of the University Assembly of Indonesia/ Majelis Rectors to foster communication regarding HELM activities and the potential for regulations (e.g. to affect those activities). Drawn from the MTE team’s F2F interviews, examples of HELM’s effect on the enabling environment include increased application of the Triple Helix model, which resulted in: (1) a collaborative grant between UNIDO, the University of Idaho and the private sector to increase the number of programs that attract students and fulfill local employment needs; (2) the UMM development of a Triple Helix Unit directed to increase small scale activities involving lecturers and private industry; and (3) engagement with bee hunters and beekeepers, local government officials and HELM participant faculty. As a result of HELM’s interaction with participant faculty, the income of beekeepers has doubled from IRD 100,000 to over 200,000 monthly. Additionally, with HELM knowledge about research and collaboration, UNP's cement industry partnership joined its lecturers who worked with other university partners in West Sulawesi and in West Java to approach the Ministry of Industry to change its regulations for the cement industry. As a result of the changes made, the cement corporation was able to save IDR12,000,000,000 (about USD$950,000) by implementing the practices recommended by the university. Conclusions: While HELM actively coordinates and collaborates with the HEIs to manage the expectations of HEI leadership and HELM participants, HELM’s focus and efforts have been at the individual- and implementing-unit levels and it has not directly addressed the enabling environment level. Although HELM has not fully leveraged its unique positioning with respect to DIKTI, BAN-PT and the HEIs to execute the requirements of section C.4.4 of its contract, there is still an opportunity to play a bridging and capacity building role between DIKTI and BAN-PT, and the HEIs. HELM is also uniquely positioned to influence, albeit to a limited extent, the greater enabling environment by working collaboratively with DIKTI to further build DIKTI’s capacity to understand the needs of the HEIs and to support DIKTI’s provision of resources and/or infrastructure grants to meet the four core area needs of the HEIs. HELM could specifically increase its efforts to increase DIKTI’s capacity building capabilities in 40 The Indonesian name of this regulation is Undang-Undang Pendidikan Tinggi. 52 the area of financial management, for example, by championing MOF and DIKTI budget cycle coordination, post-HELM follow-on activities budget support, competitive research support, and national-level scholarship assistance. 2.1.3 Achievements in Core Management Areas and Special Initiatives While Section 2.1.2 above looks comprehensively at achievements that have accrued collectively at the individual level, the implementing unit level, the institution level and beyond, this section identifies key findings and conclusions specific to each distinct core area and special initiative. 2.1.3.1 Core Management Areas USAID/Indonesia is interested in understanding the extent to which the four core areas have been successful at bringing about change. Each core area is discussed in turn below. 2.1.3.1.1 Core Management Area 1 – General Administration and Leadership (GAL) According to HELM documentation, HELM provides “technical support and training on public outreach, general management and supportive leadership for selected higher education institutions across the country. The project invited 24 leaders from 12 universities in Indonesia to attend a focused workshop on leadership in higher education. Seven courses were provided to the participants, including: Effective Listening, Listening with Empathy, Supportive Leadership in Higher Education, Managing Change in Higher Education, Developing Information System for Decision Making, Effective Communication in Change Management, and Community and Public Outreach. Two resource persons from UGM, Prof. Dr. Sahid Susanto and Prof. Sudjarwadi, were invited to share their knowledge and experience with the participants in higher education leadership.”41 Key Findings: Drawing from the HELM workshops, MTE participants noted the following improvements to their HEI: • Rectors requiring that staff that attends HELM workshops must share their knowledge with their colleagues. After which, the HEI will put a plan in place to apply the knowledge and skills shared. • Regular monthly meetings conducted among staff in implementing units to increase collaboration and share updates regarding staff activities.42 • Improved administration processes within research and community development centers. • Increase in confidence of female HELM alumni and initiative to lead ad-hoc activities within the HEI. HELM activities are relevant to the HEIs’ knowledge and capacity building needs particularly where the HELM-selected staff’s role within key HEI units directly aligned with the workshop content/core area. While HELM programming provides knowledge improvement and capacity building, some MTE participants expressed concern that the HEIs are not ready with the budget and infrastructure necessary 41 HELM Success Story. (no date) Higher Education Leaders Improve their Communication Skills. 42 The data presented in this bullet point speaks to general improvements (i.e., the regularity of meetings and increased collaboration) in the HEI that can be tied to HELM. It does not speak to the awareness of fellow HELM alumni among their respective implementing unit colleagues. See section 2.1.2.2 of this report for more information on the latter data point. 53 for the implementation of follow-up activities. One MTE participant, however, mentioned that it would be feasible to integrate follow-up activities into the regular HEI programs for which there is already budget available. Small changes within the implementing unit can make the HEI’s work more efficient and effective. One MTE participant stated: “Initially I did not know how to evaluate RENSTRA (Rencana Strategis or Strategic Planning), but after the workshop I know – knowing that LAKIP (Laporan Kinerja Instansi Pemerintah or Government Accountable Performance Report) can be used – then know that the State Polytechnic of Padang / Politeknik Negeri Padang (PNP) mission has not been achieved, some has been achieved, and some has over the target. Some indicators is (sic) difficult to be measured. Now the PNP Strategic Plan / Rencana Strategis (RENSTRA) is expired 2014 – now [I] understand to set performance indicators…” Quoted on a HELM advertising sheet, Dr. Yuni Pantiwati, MM, Head of the Biology Study Program at UMM stated “What we have done in our study program should be published through different channels. We have the UMM website, Twitter, Facebook, and YouTube, but they are not utilized properly. After the workshop, I sent a message to my team, asking them to check the website, Twitter, Facebook, YouTube and update the communication channels by providing text, photo and video (sic).” Some MTE participants also noted that transparency within an HEI and among HEIs could be improved. MTE participants cited miscommunication and/or confusion regarding whether budget proposals were approved, which impacted planning and future implementation/follow-up based on HELM workshops. MTE participants also cited a one-size fits all approach taken both by the GOI and HELM in so much as there is not a clear acknowledgement of the differences between university and polytechnic curriculum and the corresponding needs in HELM workshops. Conclusions: Better planning is needed between HELM and HEI partners, especially with Satker (non￾autonomous) status, so that HEIs can plan and budget for follow-up activities in the fiscal year that follows their respective workshop(s) attendance(s). The MTE team recognizes that the GOI budget cycle and how non-autonomous HEIs budget for HELM attendance and plan for HELM follow-up activities is beyond HELM’s control. Increased coordination to assist the planning process, however, is needed to ensure the HEIs are able to participate in HELM follow-up activities and institutionalize HELM best practices. While specific HEIs are implementing best practices, the instances cited by our MTE participants are isolated and not systematic. 2.1.3.1.2 Core Area 2 – Financial Management HELM helps DIKTI to improve HEI financial management through: (1) Empowering the Procurement Service Unit (ULP) and Internal Audit Unit (SPI) through training, socialization of the role and function of the ULP and SPI to the Rector and his staff, in order to improve systems and procedures; (2) Supporting public HEIs with regular Satker status updates, which are controlled by DIKTI, to become Satker BLU (semi-autonomous so as to use and develop revenue); and (3) Piloting some HEIs for technical assistance in developing strategic plans, business unit plans and standard operation procedures. In support of Core Area 2, HELM has conducted: 8 workshops (Quality Improvement for Procurement of Goods and Services for Finance and Procurement Staff, Strategy Audit and Follow Up Auditor Findings – 2 times; Summative - Quality Improvement for Procurement of Goods and Services - 2times; Business Plan and Budgeting – 2 times; and Transparent and Accountable Financial Management), 3 Forum Discussions (Expanding Higher Education Access Through Financial Assistance: Scholarships and Student Loans, Student Loan Modeling in Indonesia, and Problem Solving in Financial Management), Provided mentoring to 11 HEIs (State University of Makassar/ Universitas Negeri Makassar (UNM), Halu Oleo University/ Universitas Halu Oleo (UHO), Tadulako University/ Universitas Tadulako (UNTAD), Surakarta 54 Muhammadiyah University/ Universitas Muhammadiyah Surakarta (UMS), Eleven March University/ Universitas Sebelas Maret (UNS), Satya Wacana Christian University/ Universitas Kristen Satya Wacana (UKSW), Sunan Kalijaga Islamic University (UIN SUKA), Pattimura University/ Universitas Pattimura (UNPATTI), State Polytechnic of Samarinda/ Politeknik Negeri Samarinda (POLNES), Mulawarman University/ Universitas Mulawarman (UNMUL), and Lambung Mangkutat University/ Universitas Lambung Mangkurat (UNLAM)), and provided advisory services to three HEIs (UNPATTI, UNLAM, and UINSUSKA). Key Findings: Most of the financial and procurement personnel acknowledged that they improved their knowledge and improved their self-confidence through attendance at and completion of the HELM workshops. Although financial personnel in most HEIs mentioned that the implementing unit and university level are slow to change, there are two HEIs that showed significant improvement in their financial management. Some of the changes in implementing units are illustrated by these examples from UNNES Semarang and UMM Malang. 1. UNNES has been developing their several Management Information Systems (MIS), including QA, finance, academic, assets, etc., although little to no integration of these disparate systems currently exists. HELM participants from the QA and Financial Management Units, mentioned that after joining the HELM workshop on finance they were triggered to integrate two MIS into one integrated MIS. By using this integrated MIS every lecturer in UNNES can monitor their real-time activities linked with their main salary and their additional income. By having this integrated MIS, there is paradigm shift for the lecturers, from “trying to avoid additional jobs” to “looking for additional jobs,” given the incentive of additional income. 2. When the UNNES QA unit wanted to implement a system that all lecturers should upload their teaching modules, the QA personnel first approached the rector to implement the system. The rector and vice rectors successfully uploaded their modules under QA unit personnel guidance. Once the top managers completed this task, the dean, heads of departments and study programs and all lecturers followed suit. 3. UNNES has effected change beyond its HEI through collaboration with HELM UIN SUSKA Riau to learn the MIS on academics. UNNES also collaborated with other HEIs including Yogyakarta State University / Universitas Negeri Yogyakarta (UNY) and UNESA Surabaya to study UNNES MIS in Finance and Academics. 4. A participant from the UMM financial unit has developed the university’s audit system from having just one single source of audit, called general budget, to become 8 classifications/groups: operational budget, student, research block grant, community services block grant, student entry income, activity, laboratory, and autonomy. By having these classifications, the finance system is better appreciated by and accessible to most units. HELM has improved the UNPATTI Public Service Agency/ Badan Layanan Umum (BLU) team to make better documentation and preparation for Higher Education Institution Public Service Agency/ Perguruan Tinggi Negeri Badan Layanan Umum (PTN BLU) status. Represented by 3 UNPATTI BLU members, the team mentioned that UNPATTI has improved its preparation to submit formal BLU documents to DIKTI and the Ministry of Finance (MOF). Because of HELM, UNPATTI also mentioned that they are prepared to initiate the next steps to become PTN BLU. UNPATTI management expects that MOF approval to be PTN BLU will be issued this year. Most HELM participants agreed that they gained knowledge and self-confidence in dealing with financial auditors after joining HELM workshop entitled Strategy Audit and Follow-Up Auditor Findings. Most participants said that there exists confusion for the financial/procurement personnel when the 3 GOI auditing agencies voice 3 different perspectives or sets of findings after an audit. This confusion has led to uncertainty among the financial/procurement personnel as to whether the HEI is in full compliance 55 with GOI financial regulations. HELM has helped most HEI participants to better understand how to deal with auditors and the process of an audit. Scholarships and student loans are intended to help students from poor backgrounds access higher education. In the Indonesian context, these support mechanisms are not well developed. The MTE team recognizes that scholarships and student loans are a relatively discreet aspect of the HELM intervention. To this end, HELM financed a special study to help inform decision-makers’ actions regarding scholarships and loans and shared these findings at a high-level forum. The research showed that families would be willing to cover application costs if they could be certain that the child would receive a scholarship. In an effort to help ensure that HEIs were increasing the number of students from the lowest 2 income quintiles and were effectively using scholarships and student loans to enable access, Core Area 2 invited HEI personnel to attend sessions on this topic. MTE participants noted that these sessions, while interesting, were not useful and that they had not been able to demonstrate any progress on these topics at their respective HEIs. The barriers noted by MTE participants include the enabling environment including restrictive government policies and regulations regarding the scholarships. Additionally, there are no student loan providers available. A few MTE participants noted that their HEI had already taken action by itself prior to HELM and would continue to do so. HEIs personnel struggle with late GOI budget disbursements. This situation leads to challenges for budget absorption, as the time needed for conducting proper procurement is significantly shorter. The normal financial year should run from January to mid-December in any given year. When the budget disbursement comes as late as September, the procurement timeframe is only 3 months. HELM alumni have raised this as an area of concern; although they recognize that it is not within HELM’s control, they have asked that HELM function as an intermediary with government stakeholders. Conclusions: The HELM workshops and activities related to Core Area 2 are having a direct and largely positive impact on HEI participants and their institutions, including increased collaboration across HEIs to foster problem solving and understanding of GOI audit processes and outcomes and MIS applications. The UNNES and UMM practices highlighted above have increased each respective institutions’ financial management capacity and could be used as a model for other HEIs that comparatively face similar financial management challenges or have similar financial management practices. Across the HEIs, emphasis on sound financial management, audit preparation, data storage and management are in need of improvement. In addition to the outcomes referenced above, the implementation of sound financial management practices would also support transparency, help to ensure compliance with government regulations, combat corruption, and support the determination as to whether an HEI is eligible to receive funding from outside donors and the private sector. The HELM workshops and activities related to Core Area 1 are having a direct and largely positive impact on HEI participants and their institutions; including increased collaboration across HEIs to foster problem solving and understanding of GOI audit processes and outcomes and MIS applications. The lack of scholarships and student loans made available through the GOI and/or higher education system due largely to the restrictive nature of the Indonesian regulatory environment presents a prime case for HELM to leverage its role as a bridge between the HEI and DIKTI, and to meet it contract obligations under Section C.4.4, however again, the MTE team recognizes that scholarships and student loans are a discreet aspect of the HELM intervention. HELM’s increased efforts in this area is but one way it would help to fulfill the contractual activity that requires HELM to increase DIKTI’s capacity building capabilities such “that DIKTI develops strategies, plans and capacity supportive of …the broader higher education reform effort in Indonesia.” 56 2.1.3.1.3 Core Management Area 3 – Quality Assurance The first workshop on Quality Assurance (QA) was conducted in 2012. This workshop addressed the change from voluntary to mandatory HEI accreditation as required by Law No. 12 (Higher Education Act of 2012). Besides the accreditation of study programs, there is also HEI or institutional-level accreditation. The law also stipulated that to improve the quality of the HEI, the HEIs must develop an internal quality assurance system. In the forum on Managing Higher Education Quality Assurance System to Reach Better Performance, participants noted that the HEIs need to implement data management policies and procedures to better manage data to support planning for and the accreditation process itself. HELM has provided technical assistance in the area of quality improvement, with particular attention to the new requirements and challenges of the internal and external quality assurance systems. Achievements thus far include 96% of the HEIs are accredited by BAN-PT, which exceeds the target of 90% in Indicator 13.1. Key Findings: MTE participants cited that the HELM QA workshops improved their understanding of what constitutes quality and quality improvement and the importance of data management. There is a strong commitment to conduct internal QA and data sharing in the HEIs. As a result of the HELM QA workshops, one director of academic affairs has integrated five quality assurance areas within the HEI. These include academics, student affairs, human resources, financial management and infrastructure development QA. One PNP staff person stated, “After we came back from two workshops, the director established a new team to support the accreditation team – if the accreditation team needs data then the other team will support the data. Now PNP prepares to seek high/er accreditation status from BAN-PT (Civil Engineering – A and Accountancy – high B).” MTE participants also cited that HELM QA unit staff mentors the personnel at the faculty and study program levels to prepare the accreditation forms and support simulations of the visitation of BAN-PT assessors. This has resulted in increased participation and cooperation among staff across units and levels aimed at improving the accreditation process. HELM alumni also noted that while there were many QA activities going on within their HEI that not all activities were recorded or reported making it difficult to draw upon data to complete the accreditation forms and process. The HEIs recognize the need for QA, citing the lack of integrated data management systems across HEI units that impact the HEIs ability to gather, process and report data in support of accreditation, for example. They also mentioned particular barriers including the instability of electricity, deficient skill sets among computer center staff, limited access to the Internet, limited server capacities and lack of servers dedicated to QA data storage Conclusions: To date, HELM QA workshops have largely focused on the accreditation processes and targets and efforts have been aimed at improving the competencies of the QA staff so that they can support their home institutions in this regard.. HELM workshops have clearly triggered the interest of HEI leadership to aim for higher accreditation levels and QA alumni play an important role in their HEI internal processes to gain BAN-PT accreditation. QA dimensions that go beyond strengthening processes to achieve a certain accreditation level are limited at this point in time. HELM is well positioned, given the positive regard in which HEI personnel hold the prior HELM QA initiatives, to branch out into other dimensions of QA. For example, assisting HEI personnel to engage with special initiatives like the Action Research Program and blended learning are an obvious next step. 57 2.1.3.1.4 Core Management Area 4 – Partnerships and External Collaboration Through Core Area 4, HELM is to assist HEIs to collaborate with external partners to build new partnerships that will contribute to achieving the tri dharma aspirations (i.e., teaching, research, and community service). The government is interested in identifying how the universities of Indonesia can contribute to supporting business and industry and with local and provincial governments by directing part of their research capacity to include more applied research that will contribute to meeting the socio-economic needs of the Indonesian people. HELM staff carried out seven events for 282 HELM alumni, and 94 additional attendees from other HEIs at the national seminar on research and community service, a program that reinforced the Triple Helix model and marketing. Findings: For its revised 2014 indicators HELM has defined two different forms of partnerships. The first is an altogether new collaboration between partner HEIs and external stakeholders that may come from the private sector, government, non-governmental organizations, other Indonesian universities, international universities, and research centers. A partnership being actively pursued by an HEI is one that can be tracked by documenting formal HEI meetings with potential stakeholder partners, proposal development, joint proposals, such as between research department and business, and other activities pursued jointly by the HEIs and their partners. During this MTE, polytechnics also indicated they are developing partnerships with professional associations related to technical areas of study. HELM's May 2014 questionnaire to set a baseline for the number of partnerships being pursued actively reported that at the end of 2013, 164 new partnerships were in progress, compared to 190 in 2014. There is an increase of 26 new partnerships in development, or 14% from 2013 to 2014. From field data it appears that HELM is on track to exceed the number of actively pursued partnerships as well as the number of HE partnerships between US and host country HEIs that address regional, national or local development needs. There is no disaggregation of the number of Indonesian-US HEI partnerships even though MTE participants at one institution referenced a partnership in forestry with the University of Idaho that meets the third year target and from IBTCI’s evaluation of 11 partnerships between US and 18 Indonesian HEIs, we know of several involving HELM HEI. HELM's fourth year target is two more partnerships with US HEIs. Data from the MTE team’s interviews substantiate the quality and breadth of partnership activity. Regional workshops on the Triple Helix model have generated motivation and enthusiasm for developing external partnerships at all (100%) of the HEI sites that the MTE team visited where HELM alumni are engaged actively in developing and expanding external partnerships. The Triple Helix model has demonstrated a doable way of collaborating in partnership with business and industry and government at all levels. It has provided not only knowledge, but also an understanding of the importance of such partnerships to productivity and effectiveness of impact. HELM has exceeded its targets for new partnerships that are in progress. Several alumni reported a change in their attitudes and skills toward how to set up business partnerships and student internships. HELM training complemented what was going on already at University of Padang / Universitas Negeri Padang (UNP). Reaching out to the community to identify opportunities for sharing faculty expertise, has increased the productivity of new and pre-existing partnerships that have expanded in a number of different areas, including the following: • Although UNP's partnership with the cement industry was established in 2007, with HELM knowledge about research and collaboration, lecturers joined and worked with other university partners in West Sulawesi and in West Java to approach the Ministry of Industry to change its regulations for the cement industry. As a result of the changes made, the cement corporation 58 was able to save IDR12,000,000,000 (about USD$950,000) by implementing the practices recommended by the university. • Collaboration with a cacao plantation in West Sumatra and with the local government is underway to develop new products for the beans since there are no chocolate factories in Indonesia. • Another partnership has developed tea from agar wood and sandalwood for health purposes. • A fourth partnership between the pharmaceutical faculty and corporations making traditional oils will apply new technologies to make the production of traditional oils more efficient and profitable while maintaining their high quality. • Another collaboration with the government focuses on expanding the production of a new tropical species of wheat seed that will increase revenue for the growers. • Seven staff members are collaborating with the local government and rice farmers on Montawai Island in a new partnership to analyze the soil, assist in choosing the correct varieties of rice to plant in 2015, and manage their rice fields to increase rice yields. • Another new collaboration with a palm oil corporation is to research how to make composite wood from the shells using local materials. As a result of HELM, the long-range plan of UNP is to engage every academic department at the university in producing productive partnerships such as these modeled after the pre-existing partnership with the cement industry. HELM training helped UNP to focus on capacity building and knowledge transfer through interacting with other universities, community businesses, and corporations to exchange experience and to ask how to integrate input from these organizations into UNIPA’s programs. These partnerships also have impacted the academic program that now includes entrepreneurship classes for students to set up business teams funded by the university. The university is collaborating with the Ministry of Agriculture and with the Department of Economic Affairs to expand cooperation. Pattimura University's/ Universitas Pattimura (UNPATTI) level of collaborative activity is as diverse as at UNIPA. As reported by one of the four faculty members interviewed, as a result of HELM training, they realize that "it is the lecturers' responsibility to share what they have learned through HELM with others." For example: 1. Lecturers engaged "external stakeholders and faculty to sit together to develop [a new] International Office." Another lecturer from the university reported that faculty and external stakeholders met to plan and to initiate the new International Office because they intend to engage international university partners. By conducting a labor market survey with the assistance of HELM, they then planned and submitted a grant application to the United Nations International Development Organization (UNIDO) in partnership with a US university and private sector partners to increase the number of students qualified to bridge the gap in local employment needs. Writing the proposal required a significant amount of internal collaboration across implementing units, as well as collaboration with external partners. 2. Lecturers in forestry who are concerned with environmental conservation advocated with the local government "to declare new regulations" limiting deforestation and encouraging the use of non-wood instead of natural forest products. As a result the local government has established new environmental regulations. 3. Another participant reported that as a result of HELM he has a different vision of how to improve the local economy by emphasizing community empowerment. The lecturer expanded his research to support increased honey production and community empowerment, sharing his findings and collaborating with bee hunters and beekeepers in the local area and on four small islands where communities are dependent upon sales of honey and in need of community empowerment training. He then collaborated with other professionals to improve branding, 59 marketing, and packaging their products and to improve traditional practices and produce higher quality honey for a larger market. As a result, the income of the beekeepers and bee hunters doubled from IDR100,000 to 200,000 (from about USD$7.90 to $15.80) per hive, and the annual income of the lecturer who initiated the project has increased by 15 times what he had earned before building partnerships and collaborating. He succeeded in changing the bee hunters' traditional practice from cutting down trees that housed beehives to smoking out the bees and to replanting the trees that already had been destroyed. HEIs have found that the skills of collaboration and partnership building can strengthen other new initiatives. For example, when Medan Polytechnic/ Polyteknik Medan (POLMED) established its Career Development Center, applying what they had learned in workshops enabled them to increase student internship opportunities from their initial partnership with the National Electricity Company to include a retail mini-market and Garuda Airlines by signing MOUs. As a result of HELM they also have initiated help from the Ministry of Labor and National Board for Employment of Workers MTE participants identified specific HELM assistance related to partnership building and collaboration that they believe could enhance the benefits derived from Core Area 4. Their suggestions included: 1. Support for civil engineering faculty and other faculty to facilitate industrial partnerships, including those overseas. 2. Assistance in planning and conducting discussions with the Ministry of Labor in the Division of Manpower to understand how to keep students trained in Indonesia to work in Indonesia rather than working elsewhere in southeast Asia, the US, or Europe. 3. Localized or even regional meetings of HEIs, local and provincial government officials, and business and industry representatives to discuss how to structure partnerships and work collaboratively. 4. Adjustment of the “one size does not fit all" model and introduce customized training to account for HEI differences relative to the Triple Helix model (e.g., medium-sized HEIs versus large HEIs); differences between the vision and mission of polytechnics, community colleges and universities and the special needs of Islamic universities that have limited intellectual property specific to industry and the hard sciences. 5. Provision of additional training for HEIs that need further assistance on how to initiate collaborative relationships with external partners. MTE participants also noted barriers that have interfered with partnership development including the following: 1. Sluggishness related to following complex regulations and long waiting times for multiple levels of external approval to move forward with HELM-related activities (i.e., "bureaucratic bottlenecks"). 2. Delays in setting up partnerships because of the slowness of HEI leaders in approving them. 3. Limited financial resources, lack of equipment, and lack of supportive policies to move ahead with partnership development. 4. A claim by one HEI of an 11-month delay in follow-up from HELM staff. Conclusions: The experiences of HEIs engaged in building productive partnerships and community collaboration substantiate the relevance and contribution of these efforts to continuous quality improvement. The Triple Helix model struck a chord in most HELM HEIs and motivated staff to become more innovative in thinking about the relationships they could establish and how they could leverage the partnerships to improve academic programs. However, there is a wide range of readiness among HELM HEIs to build external partnerships and collaborate, given that some institutions need assistance in the basics (for example, understanding how to initiate a conversation with potential partners that they have 60 identified and researched) while other HEIs already have multiple well-established and productive partnerships that they want to exploit even further. However, without a serious effort in the next 18 months to: (1) customize the Triple Helix model to respond to the differences and diverse needs of the various types of Indonesian HEIs whose missions and capacities are not homogeneous; (2) ensure that the HEIs are positioned to sustain their pre-HELM as well as their several hundred new external partnerships; and (3) establish a viable metrics to track and capture results from partnerships and disaggregate those benefits and results according to characteristics of various partnership varieties, HELM’s systematic and intentional contribution to this important dimension of growth in the higher education sub-sector and to Indonesia’s development will be minimal. In addition, there will be no systematic, structured process and no instruments that can be taken up by DIKTI and implemented more widely across the system both to introduce other HEIs to new approaches and for DIKTI to monitor and measure the maturity and results of these partnerships. 2.1.3.2 Special Initiatives43 2.1.3.2.1 Action Research Program (ARP) HELM subcontractor, UKy, is implementing this dimension of HELM using a recognized framework to structure and guide the action research work that takes place over a six-month period. Based on international experience, HELM expects that the implementation of an action research process will contribute to developing the capacity of higher education personnel and will result in innovations for quality improvement that are tied to an institution’s core mission. It is also expected that the action research process will introduce a replicable process for change management. The foundational principles of action research are broad-based collaboration, data collection and analysis to inform decisions, continuous assessment of progress and results, and leadership and change management skills that connect an initiative to the core mission of an HEI. All 25 Cohort 1 HEI are participating in the ARP. The cohort was divided into two groups with 10 HEIs in the first group and 15 in the second group.44 As of the date of the MTE, all Batch 1 HEIs have met their targets and completed their projects. Batch 2 HEIs have planned their projects and begun to carry out their projects. Assessment of progress and results by the STTA is continuous and ongoing from the start of the action research process to the end. See Annex XI for a summary of the respective action research projects. The core approach consists of five stages of focus that are intended to ultimately result in innovation and adoption of the innovation by the institution. The five stages are: knowledge, persuasion, decision, implementation, and continuation. Inquiry and discussion are components of the research. Participants must identify their action research topic, develop an action research plan and carry out the research needed. Core elements of the ARP approach are: topic identification and proposal development, a “Change Management for Emerging Leaders” workshop, and a study visit to the Philippines (Batch 1) and Thailand (Batch 2) – two countries display attributes similar to those of Indonesia, - and ongoing support and a sharing and dissemination workshop. The change management workshop focused on the theory 43 Given the formative state of the IDEAL and Google initiatives, the MTE team did not review these activities but focused instead on the more established HELM activities and what might be done with these activities over the next 18 months. 44 HELM refers to these two Cohort 1 groups as Cohort 1 and 2. This nomenclature is confusing since the 50 HELM HEIs are referred to as Cohort 1 and Cohort 2. It is suggested that HELM uses the term “batch” or “group” as in ARP Batch 1, Batch 2, Batch 3, etc., to avoid confusion. 61 and practices of change management through action research and the links between research, individual leadership growth, and institutional change management. It also provided the opportunity for each ARP research team to update their topic, develop their ARP proposal and revise ARP team membership. The international study program covered a wide range of topics relevant to the ARPs including: culture change; challenge; identifying champions; learning outcomes; customer focus; leadership commitment; and communication and a process approach to management, with input and output documentation to track change; systems approach to teaching minds, touching hearts, and transforming lives; data driven decisions; and collaboration and partnerships. Participants also engaged with higher education bodies including the Office of the Higher Education Commission of the Thai Ministry of Education, the South East Asian Ministers of Education Organization (SEAMEO) Regional Center for Education, Innovation, and Technology and the Philippines Accrediting Association. These meeting were intended to enable participants to establish a network of contacts that they could tap into in the future. The sharing and dissemination workshop enabled Batch 1 participants to present their projects and to share lessons learned with Batch 2. Findings: Interview data reveal patterns of action across the ARP institutions that participants believe have contributed to their success in bringing about change in their institutions in a deeper and more pervasive way than before HELM. The following examples illustrate the change. Alumni from six of the 11 ARP institutions interviewed (55% of institutions) report an institution-wide emphasis on quality that had framed and contributed to different student learning and research activities. In one case, for example, the new theme of nurturing a culture of quality and competitiveness first prompted research on what academic and soft skills students needed. This investigation led to sharing the findings with lecturers to change their perspectives on how soft skills and active, student-centered learning could contribute to improving the quality of their teaching and student learning. At another ARP institution, the rector reported that HELM’s quality assurance program had been a real boost to the institution’s quality assurance efforts. He allocated additional funds for faculty research and six areas were identified where the institution could address quality of teaching and learning by focusing on active, student￾centered learning, peer and student evaluations of lecturers, and mentoring for lecturers “who are low.” Alumni from another institution reported that companies were hiring more student than before because their graduates were higher quality. Quality audits have prompted three of the 11 institutions (27%) to consider how to reduce the very heavy faculty workloads in order to improve the quality of their classes. All of the ARP institutions are using data to inform decision-making. Another pattern of action across the ARP institutions is an increase in informal learning, as well as formal in-house learning opportunities. Five of the 11 ARP institutions reported various forms of informal learning. They include "sharing with other universities," “sharing that is motivational;" "informal meetings for alumni to share empowering efforts... and to address other issues" and "meetings to collaborate." Other HELM alumni have "held many activities and meetings with their co-workers to share what we have learned." Formal learning activities have included "conducting a focus group discussion with businessmen and government forestry staff," expanding a pre-existing departmental research incubator to become the Bureau of Innovation and Research to serve all departments as a place for both informal discussion and formal learning opportunities, and "an event on lessons learned from senior researchers for other faculty." Data from the opinion survey completed by interviewees substantiates these findings described. The largest group of learners (65%) and disseminators (65%) within their units are ARP alumni, and the ARP alumni also are the largest group of disseminators within their HEIs (63%) and outside their HEIs (62%). This level of activity stands out in a context where the data that include all alumni indicate that 51% report that HELM has had the biggest impact on individuals, and 47% report that HELM has had the least impact on the unit group (13%) and the HEI (34%). 62 Alumni and STTA alike consistently noted their enthusiasm for and excitement about the ARP. They also consistently mentioned that implementation of the action research process is time consuming for the alumni and for the STTA. Although HELM does not document the ongoing communication between the alumni from the 25 HEIs and the STTA, there is a constant stream of communication between the practitioners and the STTA, and STTA are performing formal and informal coaching. While alumni and STTA were asked about the interface of the ARP and other HELM activities, they noted that there is no interface of which they were aware. The program was introduced and then extended beyond the original conception. Some personnel noted that there are issues around contracting and working in siloes. Conclusions: HELM has succeeded in developing a small cadre within each of the 25 HEIs in an intentional and structured manner. The success of this process, documented in both the quantitative and qualitative data, can be traced to the systematic and intentional way that the action research process has been introduced and the respective action research projects are being carried out. The systematic planning and design of the action research projects not only has informed the activities, but also, and perhaps more importantly, given coherence to what is happening. There is a synergy generated by the shared understanding and vision that develops while planning the project. The qualitative evidence is clear that these individuals have an in-depth understanding of the application of an internationally credible and accepted process-driven approach to addressing and redressing institutional issues. An essential element for the success of the ARP has been the study visits to the Philippines and Thailand. The opportunity to witness high quality institutions in countries similar to Indonesia and understand firsthand the processes that underpin their excellence was transformative for the action research participants. While these individuals will continue on in the higher education system, the achievement of their action plan, while certainly a commendable accomplishment, should not be the raison d’etre of the ARP or the end point of HELM’s intervention. Given the quality of the ARP, the alignment of the approach with the best principles and practices of organizational change and change management, and the commitment to the process that the initiative has generated among ARP participants, it would be a sensible move on HELM’s part to make the ARP a centerpiece of its sustainability and institutionalization strategy. It would also be a sensible move to ensure that the 25 Cohort 2 HEIs are exposed to the action research process. HELM has an opportunity to use the cadre of Cohort 1 action research practitioners more strategically and intentionally over the next two years to ensure that a strong legacy for this important initiative is left behind. By further extending the capacity of this selected group of individuals to undertake one or several of the following, HELM will have made an important contribution to improving the quality of Indonesia’s higher education system. This contribution has a good chance of becoming institutionalized within the system more broadly, if HELM is forward thinking. HELM has not yet made an effort to aggregate the results emerging from the Cohort 1 projects demonstrate development results more substantively, Potential follow-on actions are described in the recommendations section of the report. 2.1.3.2.2 Postgraduate Strengthening Program in Higher Education Leadership and Management Key Findings: A key HELM initiative is the Postgraduate Strengthening Program in Higher Education Leadership and Management. In May 2013, the Indiana Alliance produced “Methodology for Strengthening of Graduate Programs in Higher Education and Leadership in Indonesia.” This publication was discussed with and agreed to by DIKTI leadership. This publication provides the blueprint for HELM 63 engagement with 4 Indonesian universities – UGM, the Bogor Institute of Technology/ Institut Pertanian Bogor (IPB), the Indonesia University of Education/ Universitas Pendidikan Indonesia (UPI), and the State University of Padang/ Universitas Negeri Padang (UNP) - to develop new study programs that fills the gap in expertise in higher education leadership and management. UGM initiated a master’s degree program in 2012. Twelve master’s degree candidates graduated from UGM at the beginning of 2015. IPB and UPI began their programs in 2013. A UPI administrator stated that the university has graduated master’s degree candidates in education administration with a concentration in higher education administration. An IPB administrator stated that the Institute has graduated master’s degree candidates in business administration with a concentration in higher education management. HELM has also invited and is working with UNP, UPI, UGM and IPB to establish doctoral programs as part of its sustainability strategy to increase the number of Indonesian higher education leaders with advanced degrees. Doctor of Philosophy or Ph.D. programs at UGM, IPB, UPI and UNP are in development, but have not yet begun accepting students. UGM, IPB and UPI will enroll their first students in 2015 and UNP is waiting for approval from DIKTI for its program. IPB will conduct an international symposium on higher education management and UPI plans to travel to other HEIs in other parts of West Java. In 2014, the Indiana Alliance signed memoranda of understanding (MOU) with IPB, UGM, UPI, and UNP for teaching, research, exchange of faculty and students, and staff development. At this point, each HEI has prepared three staff member to begin their doctoral studies in the United States in 2015. At present, there is no common core of curriculum across the four postgraduate programs. HELM plans to help the institutions to work collaboratively to develop this common core. HELM is also undertaking work with UGM to roll out the delivery of an executive/professional development curriculum in higher education leadership and management through a range of modalities. It is anticipated that there will be an articulation between these courses and the postgraduate degree programs. The blended learning initiative is described in greater detail in section 2.2.4 below. Conclusions: The Postgraduate Strengthening Program is on strong footing for further development. The methodology blueprint is well conceptualized and the inputs from Indiana University are resulting in solid progress and products. The ongoing collaboration with the Indiana Alliance past the HELM closing data is a significant benefit. Additionally, the long-time engagement of Alliance members in Indonesia bodes well for sustainability of HELM investments. Sustainability will also require close collaboration among HELM, DIKTI and the Indiana Alliance. Given that there is not a strong tradition in Indonesia for higher education institutions to work collaboratively and for the common good of all HEIs, it is not clear why HELM did not make an effort to embed collaboration as a key and critical aspect of program and HEI development from the start of the program. HELM worked at cross-purposes on collaboration ultimately affecting program efficiency and its ability to demonstrate results. The distinct, albeit related, work done with each institution to strengthen its agenda now has to be pulled and stitched together to develop the common core. There is no risk mitigation strategy apparent; this is a risk in and of itself. For example, during the next 2 years the Indiana Alliance and the four HEI partners will be challenged to address 2 barriers to program sustainability: (1) the sustained enrollment of an adequate number of students to enable programs to justify funding and continue to develop. This is especially true, if the lack of perceived status of the social sciences negatively effects the enrollment of graduate students in the higher education leadership and management masters and doctoral programs; and (2) the challenges associated with the various learning modalities, particularly modalities that involve contemporary information and communication technologies. 64 2.1.4 Achievements in Gender and Marginalized Groups Key Findings: There is no specified requirement in the Chemonics contract with USAID that a gender analysis or broader analysis of marginalized groups including People with Disabilities (PwD) be undertaken by HELM. USAID does, however, have a Disability Policy Paper and a policy, elaborated by ADS 205, addresses gender in USAID programming. The November 2013 internal evaluation of HELM45 recommended that HELM undertake a gender analysis immediately. To date, HELM has not conducted either a gender analysis or a disability analysis, however. BAN PT does not require higher education institutions to provide sex-disaggregated data or data on PwD (and other marginalized groups) as part of the accreditation applications (at the institution or study program levels) for personnel or students, which means that there is no readily available benchmark against which relative participation by HELM participants can be measured. The MTE team was unable to locate such data elsewhere and attempted to fill this gap by producing and distributing its own questionnaire. Seven HEIs out of 18 returned the questionnaire. (See Annex VIII Gender and Social Equity Data Analysis for additional information.) Moreover, as part of HELM’s implementation of USAID’s empowerment policy, HELM has introduced a special initiative entitled Women in Leadership and to date has conducted a two-part “Women in Leadership” workshop. The workshop content consisted of several women who were considered to be role models were invited to present their stories. The MTE team interviewed at least 1 and, in most cases, several individuals from each of the HEIs visited who had attended the workshop. Several patterns were consistent across the interviewees. In institutions where participants had been able to carry out their action plan, a number of results were observed: (1) women were inspired by the stories shared by role models, they felt more confident about their capacity to compete in the public arena, and they stepped forward and took risks in the public arena that previously they would not have considered pursuing; (2) women began to intentionally encourage other women to be confident and to take risks; (3) the number of women in leadership positions increased. For example, in one HEI, prior to the HEI’s Women in Leadership seminar, six out of 40 members of the HEI’s senate were women. Following the seminar, a large group of women stepped forward and ran for and won senate seats. Today, women comprise 40% (just under half of the members) of the senate. No consolidated data were available through the HELM database that quantify and document such results. HELM administered a self-efficacy questionnaire during Part B of the workshop. The results show positive gains for women and some positive gains for men. The questionnaire does not conform to good or best practice in survey design and this gap creates concerns about the validity and reliability of the data and the results. In HEIs where participants were not successful in carrying out their action plans, the reasons for the lack of success consistently mentioned include the following: the Women in Leadership initiative had no institutional “home.” This meant that no unit within the HEI was willing to incorporate the action plan into its budget; HELM alumnae went “door-knocking” at various units and were turned away and told to go to such and such a unit instead. This was even the case where the HEI had a women’s studies center, which HELM alumni believed would have been a logical institutional home for the initiative; HELM alumni 45 United States Agency for International Development. (2013) Higher Education Leadership and Management (HELM): Deliverable 11 – External Assessment of HELM Progress Toward Objectives and Targets: November 26. 65 submitted a proposal for action to the financial management unit, but no response was forthcoming; in a few cases, the proposal was accepted and a budget had been allocated only to be withdrawn and shifted to another HELM activity. To date, HELM has not undertaken either a gender or a disability analysis. There is no gender and social equity strategy available. Some HELM personnel noted that HELM is gender-sensitive because of the number of women in leadership positions in higher education is low and yet HELM has been successful in having a good representation of women at workshops. The 2013 external review of HELM also noted the absence of a gender analysis and strategy and recommended that this gap be addressed immediately. Conclusions: Although HELM has taken measures to promote the empowerment of women and there appears to be a relatively high ratio of female participation, the absence of a HELM gender integration strategy informed by substantive and thoughtful analyses represents a significant gap in HELM implementation and strategic planning. Implementing contractors should be well past the point where simply reporting data disaggregated by sex can be considered sufficient for demonstrating gender sensitivity and should be well aware of USAID/Washington policy regarding gender integration and the associated ADS requirements. With respect to PwD, HELM is in a position to help guide HELM HEI to take steps to address the needs and potential of this (and other) marginalized constituencies. Although PwD are largely invisible in Indonesian society in general and in the higher education system and the work force in particular, some provinces and districts are particularly proactive in promoting inclusion; and at least one of the HEIs visited by the team expressed a strong interest in looking into ways and means of promoting inclusive education. 2.2 HELM APPROACHES TO HUMAN AND INSTITUTIONAL CAPACITY DEVELOPMENT HELM has utilized a variety of approaches to Human and Institutional Capacity Development (HICD), primarily workshops and forums. From 2012 to 2014, HELM conducted 60 types of such gatherings (2012= 7; 2013= 30; and 2014=23), and involved 2,453 participants (1,766 men/72% and 687 women/28%). These data indicate that, on average, 41 participants attended each HELM gathering. When the data are disaggregated into each core management areas and the two dominant special initiatives, ARP and the Post-Graduate Strengthening Program, they show that Core Area 1 – General Administration and Leadership had 9 gatherings or 15% of the total. Core Area 2 – Financial Management had 12 or 20% of the 60 total gatherings. Core Area 3 – Quality Assurance had 8 gatherings or17% of the total. And, Core Area 4 – External Collaboration had 10 gatherings or 17% of the 60 total gatherings. The ARP had 11 gatherings or 18%, and the Post Graduate Strengthening Program had 10 gatherings or 17% of the 60 total gatherings. Human and Institutional Capacity Development (HICD): A series of structured and integrated processes designed to remove significant barriers to the achievement of an institution’s goals and objectives. HICD involves the systematic analysis of all the factors that affect performance, followed by specific interventions that address gaps between desired and actual institutional behaviors. HICD interventions include training to address skill and knowledge gaps, and to deal with other performance barriers such as dysfunctional organizational structure, unsupportive work atmosphere, or lack of necessary tools and incentives. Success of training and other capacity development interventions is measured by improvement in overall organizational performance and output, not the number of individuals trained. (USAID HICD Policy) 66 Figure 6: Training Success Factors Model 2.2.1 Workshops Traditional F2F workshops are the cornerstone of the HELM HICD approaches. These workshops have been delivered predominantly by Indonesians and somewhat by international short-term technical experts. A workshop typically consists of two parts. In Part 1 of the workshop model, participants are exposed to new content, methods and instruments, and develop action plans to be implemented on their return to their respective institutions. Part 2 of the workshop occurs approximately three months later and provides the opportunity for the alumni to share their progress against their action plans and to obtain advice and guidance from other HEI and HELM personnel. For example, during the summative meeting for the Women in Leadership workshop, some female rectors were invited to talk about their experiences as HEI leaders. Approximately 1,200 HELM alumni have taken part in HELM capacity development activities, predominantly the two-part F2F workshops that addressed topics that HELM technical specialists felt were of relevance to HEI personnel associated with the four core management areas (general administration and leadership, financial management, quality assurance, partnerships and external collaboration). Key Findings: Alumni mentioned that besides the formal schedule of the workshop, they valued the opportunity to discuss and share experiences with their colleagues from other HEIs. Furthermore, during the summative meetings, the presence of additional resource persons who had been invited to share their experiences was very much appreciated. For example, a number of alumni who participated in the QA workshops talked about the value in listening to and connecting with representatives from universities that had achieved the A level status for institutional accreditation. HELM alumni show an overwhelming preference for F2F workshops as a learning modality. MTE participants in leadership positions consistently urged that HELM provide sessions at their respective HEI. They felt that wide scale change would not happen with only 2-3 individuals attending a specialized event. They suggested that those who had attended the events could receive training to become in-house trainers. Given the importance of the workshop modality to individual capacity building under the project, the MTE team examined the quality of the workshops and asked MTE participants about the workshops using the analytical framework depicted Figure 6. For the most part, MTE participants were satisfied with the quality of the materials and the facilitators. Most participants had only been involved in lecture-based training delivered by government-contracted external presenters. Generally, the approaches used in the HELM workshops were seen as appropriate to the needs of adults who were professionals in various fields. One of the most frequently mentioned aspects of the workshops, regardless of the core area discussed, was the exposure to new instruments and tools that could be used repeatedly and also shared with colleagues. Examples of these instruments and tools include the Triple Helix model, accreditation forms, guidance on financial management, the case study approach, and the ARP approach. 67 The MTE team attempted to examine the curricula associated with each core area, but this proved challenging for several reasons. The approach to workshop course design is not standardized and does not meet internationally accepted standards for adult education curriculum development. 46 There is no overarching scope and sequence of topics, syllabi, and consistency in what are categorized as learning outcomes, for example. Workshop topics have apparently emerged organically over time. For example, there are workshops conducted based on topical needs linked to the four core areas that were identified during workshops. Data integration is an example of just such a topic. Terminology used is non-standard, as are planning documents such as a Terms of Reference and an agenda, both of which apparently constitute the respective course design. There is an extensive amount of content on the UGM website for professional development of HEI personnel in higher education leadership and management. This content is discussed in the Blended Learning section below. Because documentation of the course design/content for workshops and forums is not well organized or systematically presented, it is not clear how the professional development materials on the UGM website link to the workshops and forums. It is also not clear whether HELM worked with DIKTI personnel to adapt or upgrade existing course materials for topics that are already being addressed through DIKTI or other providers such as the World Bank-sponsored center on procurement practices at one of the Indonesia Managing Higher Education for Relevance and Efficiency (I-MHERE) universities. Conclusions: HELM has a limited number of technical and operational personnel who competently handled the huge demands associated with carrying out such a large number of individual capacity development activities. The staff is to be commended for accomplishing all that they have and for generating such enthusiasm for the project and what it offers. However, the workshops are a training program and, as such, should have been developed more systematically with greater clarity about the learning outcomes and expected results and documented and packaged professionally for broader distribution and use. Training programs have the potential to become an unending stream of attempts to fill ongoing gaps. To prevent this situation from arising, a structured needs assessment is typically undertaken. The assessment considers both individual and organizational capacity gaps. The results of the assessment are then used to inform the framework for what the training program can and will address. The absence of a highly qualified and skilled specialist in adult education and training and organizational change management who could have worked alongside the core area technical staff has likely contributed to the issues associated with overall course design, oversight of facilitators and documentation. Alumni could have benefited more from the forums and workshops is they had been clustered into small coherent teams that had consistent members, more along the lines of the ARP approach. Working in teams at the institutional level is the start to institutional change and also improves the capacity of individual team members. 46 Kolb, David A. (1984) Experiential Learning: Experience as the Source of Learning and Development. New Jersey: Prentice-Hall; and Kolb, Alyce Y. and Kolb, David A. (2005) Learning Styles and Learning Spaces: Enhancing Experiential Learning in Higher Education. Academy of Management Learning & Education, 4(2): 193-212. 68 2.2.2 Forum with Special Proceedings The contract requires implementation of “a series of collaborative forums in which DIKTI and BAN PT, other higher education personnel and representatives from the private higher education institutions can interact with each other and with representatives from relevant U.S. higher education institutions, accrediting bodies, and university advisory councils around substantive issues of relevance to the Indonesian higher education landscape. Topics are to include, at a minimum, quality assurance, external stakeholder collaboration, financial management, and university leadership and management.” The forum is typically a one off high-level event that is attended by the top management of an HEI (e.g., rectors/vice rectors and deans/vice deans, senior personnel from DIKTI and other government ministries, and senior personnel from other stakeholder organizations such as banks). There were no data available on overall numbers of forum participants. It is not clear how HELM helps rectors/directors utilize the benefits derived from the high-level forums with the individuals, predominantly lecturers, who attend the core area workshops.47 Key Findings: According to Deliverable 1148, “The [forum] topics are relevant to the 4 core areas and support DIKTI’s priorities. Attendance at the forums has been high and feedback is positive. The management of the forums is strong and the participants have appreciated the opportunity to network and learn from their colleagues and international experts through presentations and discussions.” The MTE team found no evidence that is contradictory to these findings. There appears to have been no documented follow-on to address the issues and opportunities raised in the gatherings and captured in the forum proceedings documents. This was pointed out in the November 2013 external review document as well and it was recommended that this situation be remedied. As was found by the external review team in 2013, the MTE team also found little evidence of a structured plan that identifies how forum proceedings will be disseminated to various additional audiences/users and how the opportunities and issues emerging from the various forums will be taken forward systematically. Individuals who were interviewed by the evaluation team either did not have much engagement with the forums – possibly since they were not at high enough levels in their institutions and so were not invited to attend – or those who had attended did not recall specific benefits or results coming out of the forums. There does appear to have been some effort to link the topical foci of some of the forums with the foci of some core area workshops. For example, the topic, Student Financial Assistance and Student Loans, was the subject of a forum in October 2012 and also the focus of a two-part workshop under Core Area 2. It is not clear whether the forum content/proceedings were shared with the Core Area 2 workshop participants. Conclusions: HELM has been successful in attracting a range of high-level stakeholders to the various forums and this is a notable achievement in itself, given the positions that these individuals hold. The 47 HELM established the forum series in year one and continues to be on schedule for forum delivery. Topics to date include: Roundtable Forum (July 2012); Supportive and Scientific Leadership to Improve HE Performance (September 2012); Managing Higher Education Quality Assurance Systems to Achieve Better Performance (September 2012); Student Financial Assistance and Student Loans (October 2012); Collaboration with External Stakeholders (November 2012); Autonomy and Governing Boards (March 2013); Student Loan Modeling Forum (April 2013) and the Action Research Forum (June 2013). 48 United States Agency for International Development. (2013) Higher Education Leadership and Management (HELM): Deliverable 11 – External Assessment of HELM Progress Toward Objectives and Targets: November 26. 69 various for a have been framed around the high quality research studies and reports – HELM deliverables – and these session have produced some excellent insights and suggestions for further action that are captured in write-ups of the forum proceedings. However, HELM is missing a clear opportunity to exploit the investment in these forums both in terms of communicating the results of the forums to the range of interested stakeholders, to take forward the suggestions for further actions to address issues and to ensure that the content of these forums are accessible to and understood and utilized by workshop participants. The lack of a clear strategy for ensuring that the content and results of these forums extend beyond a one-off information event is the primary problem. It is unfortunate that the very sensible recommendation from the 2013 external review has not been taken forward more intentionally and systematically. 2.2.3 Technical Assistance Key Findings: HELM technical assistance is delivered by a range of individuals including HELM technical specialists, international STTA, local STTA, and mobile advisors. In the past year, a mentoring program entitled “Advisors on the Go” (A2G) has been introduced. In this approach, an individual from an HEI puts in a request to HELM for one-on-one or one-on-small group mentoring that is provided by a HELM technical specialist or Indonesians who are external to the project, but who are known to have expertise in specific areas. The mentoring visits are conducted on-site and are to be related to the workshops and are intended to strengthen the implementation of the action plans and beyond. The advisor addresses other HEI needs that are approved by HELM. HELM has also introduced a blended learning platform for HEI personnel interested in professional development in higher education leadership and management. This platform is described in greater detail in section 2.2.4 below. Nearly all of the alumni interviewed expressed an interest in being mentored by a HELM or Indonesian expert. Mentoring, advising and coaching are very specific activities that have a body of literature around the theory and practice of these endeavors. The expressed demand for mentoring and advising far outstrips current and projected HELM capacity and resources. There does not appear to be a mentoring and A2G documented strategy that provides specific parameters for meeting the demand and the results to be achieved. According to HELM personnel and some HEI alumni participating in the MTE, there has been increased engagement by the HELM home office technical specialists with HEI clients. This development is seen to be beneficial. HELM has established two regional offices, one in Makassar and one in Medan to serve the HEIs in eastern and western Indonesia, and each is staffed with a regional coordinator and support personnel. It is expected that the regional coordinators will fill at least part of the gap in client engagement on technical matters. In addition, a mentoring facility has been introduced where HEI clients can request specific assistance for emerging issues through a number of A2G; these advisors can be HELM technical specialists or external local STTA. Results from the MTE opinion survey show that HEI clients are enthusiastic about receiving technical support from local Indonesian experts. Many HELM alumni participating in the MTE consistently mentioned that they would like individual mentoring by HELM technical specialists or local STTA. According to feedback from some HELM alumni and other individuals participating in the MTE, there are ongoing issues regarding acknowledgement of requests for technical support and the actual provision of that support. There were several instances cited where a request had been made nine, six or three months prior with no acknowledgement from HELM and no assistance. This problem was also mentioned in the 2013 external review report. 70 Some HEI clients have had (and continue to have) the opportunity to interact consistently with international STTA, particularly clients from the four institutions involved with the Indiana Alliance and those clients participating in the Cohort 1 ARP initiative. Cohort 2 ARP clients will also have the same opportunity. There is little evidence to suggest that personnel within DIKTI or BAN PT have benefited from the provision of international STTA. Individuals from DIKTI and BAN PT felt that such an opportunity would be beneficial. Conclusions: The demand for mentoring is high. This is a positive sign since it signals the interest of personnel in improving their own capacity and the capacity of their institution. However, the demand far exceeds the supply of mentoring opportunities and even meeting the requests already received in a timely manner is proving to be problematic. The lack of evidence regarding the effectiveness of the mentoring that has been provided is problematic as well – there is currently no vehicle for determining whether the approach is worth the investment. In addition, some of the requests for assistance may deal with issues or situation for which the HELM technical personnel and/or the Indonesian external STTA may not have the level of expertise needed to provide credible advice. Few individuals have the opportunity to make use of HELM’s international STTA. The lack of opportunity for DIKTI managers and lower tier personnel to learn and benefit from these STTA who bring a wealth of knowledge and insights about higher education system development is especially problematic. Without a well crafted and highly selective mentoring and advising strategy with clear goals and metrics, the mentoring approach will not be fit for purpose, may create problems for HELM, and frustration on the part of the stakeholders. 2.2.4 Blended Learning HELM has introduced the Blended Learning Initiative that combines traditional and contemporary delivery modes. According the a HELM advertising panel titled, USAID HELM – UGM Blended Learning, “Blended learning techniques include a combination of face-to-face training and technology-supported training and virtual interaction such as internet-based webinars and the delivery of videos and other instructional materials. This approach has been successful when accompanied by high quality instructional design. It is important to employ locally grown approaches and strategies in order to lay the groundwork for internationally proven methods.” The HELM advertising panel also notes that the website has had 300,000 hits since it sent live. Key Findings: Within Indonesia, there are several institutions that have developed a significant level of experience and expertise in the delivery of distance and open education, including for professionals who want to engage in further learning. Of note are the Open University, IPB and SEAMEO. UGM is a relative newcomer to blended learning platforms. Despite this, UGM has been collaborating with the World Bank to utilize blended learning for upgrading the capacity of medical school personnel.. On the international front infoDev, based in the World Bank/Washington, is spearheading an international effort to enhance the use of information and communication technologies in developing countries. HELM has introduced the Blended Learning Initiative in collaboration with UGM, building from the university’s recent experience. . According to the HELM advertising panel titled, USAID HELM – UGM Blended Learning, “Blended learning techniques include a combination of face-to-face training and technology-supported training and virtual interaction such as internet-based webinars and the delivery of videos and other instructional materials. This approach has been successful when accompanied by high quality instructional design. It is important to employ locally grown approaches and strategies in order to lay the groundwork for internationally proven methods.” The HELM advertising panel also notes that the website has had 300,000 hits since it went live. Viewers can access a range of materials and instructional media including 71 webinars and videos that are specific to content covered during workshops under the four core areas. For prospective learners in eastern Indonesia where Internet connectivity is an issue, HELM will provide compact disks loaded with the materials. According to HELM documentation and one HELM staff person, the blended learning initiative is one aspect of the HELM sustainability approach. Many alumni were interested in the blended learning delivery platform since they believed the content offered a way for them to improve their management competence. However, they expressed reservations about the practicality of the platform. The results of the MTE opinion survey of HELM alumni from the 18 HEI sites show that HEI clients do not feel that a blended learning platform that utilizes online learning will be effective. Qualitative data from interviews of these alumni show that alumni are concerned about the infrastructure necessary to access the platform, the timing of interactive webinars and the time blocks needed to devote to study, in particular. Individuals who have already participated in the delivery platform noted that they had difficulties with the platform, especially with respect to the webinars. Some of the problems mentioned were: electrical blackouts, unreliable internet access, and the timing of the webinars. For example, during the set time for the webinar, participants could not attend because of other activities or duties (e.g., giving lectures or attending meetings). HELM’s own publicity material acknowledges that the blended learning initiative has some significant challenges that must be overcome. Scans of the web content by the MTE team show that the teaching and learning methods do not conform to best practices for adult education in general or for blended learning in particular. Scans of the web content by the MTE team show that the teaching and learning methods do not conform to good or best practices for adult education, in general, and for blended learning, in particular. 49 Conclusions: HELM has invested a significant amount of work in its collaboration with the UGM to establish the platform for the delivery of professional development courses in higher education leadership and management to HEI personnel across Indonesia. It would have been prudent for HELM to explore the possibility of engaging other more experienced open and distance education providers in Indonesia to provide the delivery platform and to work with UGM to ensure the quality of the teaching and learning and the materials. While the collaboration with UGM is established, there may be things to learn from other organizations that have been dealing with the blended learning platform for many years. HELM faces an uphill challenge over the next 18 months if it is to: improve the quality of the teaching and learning approaches and the curriculum, including respective materials, present online; convince prospective learners of the viability of the platform; and address the infrastructure constraints. Without immediate attention, these issues 49 This finding is based on information drawn from the follow sources: Bersin, Josh. (2004) The Blended Learning Book: Best Practices, Proven Methodologies and Lessons Learned. San Francisco: Pfeiffer, a John Wiley & Sons, Inc. publisher; Garrison, Randy D. and Vaughn, Norman D. (2007) Blended Learning in Higher Education: Frameworks, Principles and Guidelines. San Francisco: Jossey-Bass, a John Wiley & Sons, Inc. publisher; Glazier, Francine S. and Rehm, James. (2011) Blended Learning: Across the Disciplines, Across the Academy (New Pedagogies and practices for Teaching in Higher Education). Sterling: Stylus; Picciano, Anthony G.; Dziuhan, Charles D.; and Graham, Charles R. (Eds.) (2013) Blended Learning: Research Perspectives, Volume 2. New York: Routledge; Stein, Jared and Graham, Charles R. (2013) Essentials for Blended Learning: A Standards-Based Guide. New York: Routledge; Vai, Marjorie and Sosulski, Kristen. (2011) Essentials of Online Course Design: A Standards-Based Guide. New York: Routledge. See also Kolb, David A. (1984) Experiential Learning: Experience as the Source of Learning and Development. New Jersey: Prentice-Hall; and Kolb, Alyce Y. and Kolb, David A. (2005) Learning Styles and Learning Spaces: Enhancing Experiential Learning in Higher Education. Academy of Management Learning & Education, 4(2): 193-212. 72 have the potential to negatively affect prospective learners’ interest in the platform and the content. The short timeframe remaining to address these issues raises concerns about uptake and sustainability of the initiative post-HELM. 2.2.5 Institutionalization and Sustainability Key Findings: According to the HELM Scope of Work, “It is essential that the Contractor seek to create and leave a legacy of Indonesian human and institutional capacity that can continue implementing and improving higher education reforms and quality in Indonesia (emphasis added). Achieving program results will hinge upon the ability of the Contractor to understand Indonesian higher education priorities, needs, capacities, practices and challenges. In addition, the effective development and implementation of models and approaches will depend on Indonesian participation and ownership of the endeavors. Therefore, it is critical that the Contractor establish and maintain close and collegial relations with the project’s Indonesian partners, stakeholders and experts (including the DIKTI and higher education institutions) and ensure processes that increase Indonesian ownership of the program.”50 The Chemonics proposal notes that the Chemonics consortium is to undertake “an ambitious set of interventions to increase management capacity within Indonesia’s higher education institutions and build the capacity of the Directorate General for Higher Education (DIKTI), part of the Ministry of National Education (MONE), to promote and monitor these interventions. … (and) offer(s) decades of experience supporting Indonesians to effect sustainable institutional change within universities and other public institutions. (The Chemonics) Indonesian- driven approach emphasizes sustainability, identifies incentives to trigger change, and produces replicable and scalable methodologies to improve university performance (emphases added). Chemonics’ strategy and approach to ensure HELM leaves a legacy of stronger management systems in Indonesian HEIs is to first ensure (that the consortium partners) support the leadership of Indonesians, including champions for reform within DIKTI, MONE (sic), the higher education (HE) community, and Indonesian society more broadly. To do so, in HELM’s first stage, (Chemonics) local and expatriate staff will help DIKTI inform themselves in detail about the state of Indonesia’s HE sector as well as the range of models and approaches that have been and are being used by HEIs to achieve their missions. Second, (Chemonics) will support DIKTI in identifying the policy mechanisms and incentives that can effect the changes needed to achieve their strategic goals for the sector. Doing so will involve identifying where (at DIKTI, at the HEI, etc.) decisions are made on key issues of university management and ensuring that those whose behavior DIKTI wishes to change have the authority and incentive to decide differently. Finally, (Chemonics) will support DIKTI in developing successful models and approaches to all 50 Section C.5.2.a. 73 aspects of university management and in generating the tools and resources that HEIs will need to replicate them effectively.”51 According to the HELM contract and subsequent project documentation, years four and five are the points in time when the project focuses on institutionalization. HELM Deliverable 652 notes that, “…while not all higher education institutions can immediately transform themselves into world-class universities, they can focus on becoming regional knowledge hubs.53 In this regard, HEIs across the region can profitably learn from the exemplar programs within their own and neighboring countries. We define “exemplar programs” as areas where the institution has chosen to invest faculty and administrative time, funding, and planning, in order to combine academics with regional responsiveness. Often these programs are interdisciplinary in nature, serving as relevant and innovative knowledge hubs that play a key role in addressing local, national, regional, and global needs. Such programs can only exist through the strategic integration of the 4 areas at the core of the HELM project (emphasis added): sound leadership and management, transparent financial responsibility at the service of academic priorities, attention to quality assurance and the use of data, and responsiveness to the needs of external stakeholders.” HELM documentation mentions a Program Coordinator who “…is responsible for managing the cross-cutting team, ensuring program coordination, and providing technical supervision. The program coordinator will support the writing of all reports and project documents.” This is a part￾time, resident hire position for a native English speaker based in Jakarta.”54 There was no evidence of this position having been filled. Within the first six months of HELM implementation, HELM international and local experts carried out institutional assessments in selected HEIs. The assessment period was shortened through a contract amendment (#02) from several months to three weeks and was also moved forward in the project work plan. These assessments were qualitative in nature and did not utilize a prescribed institutional audit or institutional development framework. Based on these specifications,55 DIKTI and HELM identified a group of HEIs that enabled the intent of the specifications listed above to be honored. This pre-selected group of HEIs was invited to submit proposals expressing interest in and commitment to participating in HELM DIKTI and HELM personnel reviewed the proposals and HELM began to work with 10 HEIs 51 Chemonics Higher Education Leadership and Management Project Technical Proposal. March 28, 2011. 52 United States Agency for International Development. (2012) HELM: Deliverable 6 – Analysis of Approaches to Improved Quality and Relevance for Higher Education Institution Academic Programs: September 28. 53 Sharma, Yojana. (2012) Universities Need to Serve Regional Economy, Society (July 17). AsiaEngageNetwork. 54 C.8.1.e HELM SOW and Contract Amendment #03 January 10, 2013 55 According to the Chemonics contract, the selection of HEIs that would participate in HELM was based on the following specifications: “Include at least five higher education institutions that directly influence the basic education system. Such teacher training institutions (Lembaga Pendidikan Tinggi Keguruan) provide teacher training and/or education administration and management programs, and thus support USAID’s prior and forthcoming investments in teacher training programs and the high priority that GOI gives to teacher training; Achieve a balanced and representative set of the types of higher education institutions; Achieve reasonable geographic distribution throughout the country, whereby no more than 10 of the institutions would be located on the island of Java; Be based on how well technical assistance provided at these institutions will yield widely applicable models and approaches; Be based on the ability of the selected institutions to serve as assets in disseminating approaches to large numbers of other higher education institutions and helping those institutions incorporate such approaches, for example by having large enrollments or by having a leadership relationship with several other institutions of higher education; Encourage improvements in the academic programs that GOI has identified as high priority for development, such as science, technology, engineering, mathematics, agriculture, health and water resource management.” 74 initially and then 25 HEIs in total beginning in year two. The MTE team was not able to locate a proposal review protocol or report to verify the selection process or the quality of the memoranda of understanding. Key Findings: The rectors and directors with whom the MTE team spoke noted that their respective institutions had been invited to submit a proposal. These individuals all said that their institutions had gone ahead with the proposal development and submission because HELM’s four core areas were relevant to the needs of the HEI. MTE participants from some of these original 25 HEIs did not mention the assessments and were not familiar with the special study related to these assessments. DIKTI personnel, Project Advisory Group (PAG) members, HELM technical experts and HELM documentation (including the Chemonics proposal) repeatedly brought up the need for an instrument to gauge and monitor institutional capacity development. Several administrators interviewed during the MTE asked whether an organizational development tracking tool could be introduced by HELM. In 2014, JBS proposed introducing the Organizational Capacity Assessment Instrument (OCAI) as an instrument that might enable HEIs to capture and track their progress against organizational development objectives. To date, HELM has not introduced the OCAI or another such instrument. As mentioned previously in this report, several senior and middle level administrators from the 17 HEI sites visited suggested the HELM introduce performance targets that the HEIs should be expected to achieve as a result of HELM assistance. There is no evidence that baselines have been documented for any of the core areas. USAID/Indonesia suggests that “institutional needs assessments should be conducted early on in order to provide baseline data for the development of goals and objectives for institutional development programs.” There is a lack of awareness of, understanding about and agreement on the HELM sustainability strategy. Aside from a few HELM personnel, MTE participants were not aware of a HELM sustainability strategy that provides a detailed plan for what will be sustained, how and by whom. According to a HELM technical expert, the universities that are part of the Indiana Alliance will continue their engagement with the four Indonesian HEIs involved in Postgraduate Degree in Higher Education Leadership initiative. The MTE team reviewed the memoranda of understanding between the US universities and the Indonesian HEIs and other HELM documentation but was unable to locate any specific references to or details about an ongoing relationship that transcends HELM closure. Another HELM staff person noted that the blended learning platform is the HELM sustainability strategy and a HELM advertising panel on the blended learning initiative supports this statement. “To promote high quality and globally recognized university programs in the 21st century and to reach the goals of Indonesia’s 2012 Higher Education Act, experimentation with blended and distance learning is an important and needed step. …The HELM blended learning model contributes to all four of HELM’s core areas by deepening the understanding and realization of key measurable outputs of the Performance Management Plan (PMP), including more effective strategic planning among HEI leaders, better financial performance on audit reports, improved and sustained accreditation ranking, and increased number of partnerships. Each of the four core areas will develop an interactive course with three modules and three sessions per modules (nine sessions per core area). The interactive webinar sessions will be visible to those who log on from Jakarta or across the country. The participants will receive intensive training with feedback and support from the HELM and UGM teams, and be able to engage other professionals at their higher education institutions to plan and learn together and to interact with their peers at other participating higher education institutions.” 75 Conclusions: The very important initial stage of laying the groundwork for engaging with an institution and incorporating dimensions known to be essential to the development of institutional capacity was rushed and counter-productive. The urgency has caused less than desirable repercussions that will have an impact on institutionalization of the benefits to be derived from engagement with HELM. For HELM, this stage should have ensured that targeted institutions could: (1) obtain the maximum benefit from the investment of donor resources; (2) function as incubators for effective and efficient practices; and (3) function of repositories for transformative change that could be used to expose other institutions to the possibilities and inspire replication. It should also have identified agreed performance expectations along with a set of agreed rewards for ongoing progress and performance56 and a set of actions to be taken should an HEI be found to be under-performing. Within the HELM results framework, improved capacity in the four core areas and the special initiatives results in increased management capacity in Indonesian HEIs that in turn results in improved education quality, which then results in students who are better prepared for learning and work.57 From an organizational development perspective, this results sequence jumps up the ladder of inference by omitting the need to integrate HELM components as a system within HELM and the HEIs. The extent to which institutionalization of increased management capacity will occur is dependent on the integration of the disconnected components into a system and on the extent of integration and cross fertilization both within and across implementing units of the HEIs during the next two years. To embed any change process in an institution of higher education requires attention to institutional systems and processes in addition to products. If HELM activities were intentionally integrated and cross-cutting elements were structured intentionally and according to acknowledged principles, HELM would have a greater likelihood of cutting through the sluggishness that affects most organizations and hastening institutional change. During the next two years, if constructive change is going to become more deep and pervasive in implementing units, alumni need to understand the importance of group learning as a catalyst for change and how to bring teams of people within their implementing units to a shared understanding of best practices for addressing the challenges and problems they face. As such, action research could become a powerful HELM strategy to coalesce and re-focus the learning of all alumni in each institution to solve a single problem of doable proportion as a means for utilizing and modeling effective group learning, cross-unit communication, and collaboration skills. Sustaining the positive changes that are now happening in HELM HEIs will depend upon the extent to which the individuals in each institution, regardless of their position, learn how to apply the skills of interpersonal and cross-unit communication, collaboration, and group learning in their daily work and then use these skills to interact effectively beyond the walls of their respective institution. Given the current HELM approach to institutional capacity development, it is unlikely that practices will be sustained, deepened and become pervasive (institutionalized) in order to stand the test of time. Three frameworks may be of benefit to HELM – the Innovation Network’s Transactions, Transformations and Translations matrix and the USAID Economic Growth and Trade (EGAT) Office visualization for HELM to orient all personnel and stakeholders to an institutionalization and sustainability vision and a set of possible performance indicators linked to HELM’s core areas. (See Annex VI – HICD Process Frameworks and Possible Performance Metrics.) 56 The Australian Agency for International Development (AusAID – now part of the Australian Department of Foreign Affairs and Trade or AusDFAT) calls this approach “capacity enhancement.” 57 Performance Management Report Version 4, p. 3. 76 Given the challenges HELM and the Cohort 1 HEIs will face in ensuring that a solid foundation for sustainability and institutionalization is secured, it is unrealistic to expect these Cohort 1 institutions to play the big brother/big sister to the Cohort 2 institutions. It is also not plausible to run the Cohort 2 HEIs through the gamut of core area workshops and then expect these institutions to be at a point where the practices and processes introduced can be sustained and institutionalized. Given the strength of the ARP, the limited time until the project enters the close-up stage and the intention that the blended learning platform becomes the vehicle for wide scale professional development, it does not make sense to replicate the Cohort 1 approach in the Cohort 2 HEIs. A risk matrix and a written sustainability strategy are fundamental pieces of sound project implementation. The lack of these elements has resulted in the realization of risks that could have been prevented or minimized and missed opportunities. 77 3. LESSONS LEARNED LESSON 1: A robust design and proposal can help a project withstand even the most extreme challenges including extreme distortion of the original design; particularly the increase from 25 to 50 higher education institutions across the whole of the archipelago and the truncation of the original time frame for carrying out key strategic and analytical tasks, a serious time gap between the release of the request for proposals and the mobilization of the HELM contractor, and flux in project personnel including those in the most senior positions. LESSON 2: A project that is advertised as building human and institutional development capacity for leadership and management must ensure that it reflects the principles of HICD and provides strategic and effective leadership and management. Higher education institutions internationally and certainly in Indonesia are notorious for their organizational rigidity and for working in silos. An implementing partner must take care to ensure that its organizational structures and technical units do not mirror the very organizational issues faced by its counterpart organizations. For example, the four core areas of HELM operate in isolation from each other and do not yet exploit the synergies and other benefits that could come from internal collaboration. If the project does not demonstrate competence in applying HICD principles and practices and forward looking strategic planning and management, the stakeholders may call the credibility of the project into question. This is counter-productive. LESSON 3: A lesson often cited in international development literature over the past two decades is that institutional development is a medium- to long-term endeavor and donor agencies should expect to invest in such an initiative at a respective institution for a minimum of 10 to 15 years in order to see results. HELM is a five-year project, so it is already compromised regarding the ideal implementation period. The challenges the project has faced with building the capacity of DIKTI and institutionalizing desired change within partner organizations without a clear strategy that is informed by a risk assessment further compound the situation. LESSON 4: Strategic communication for development purposes should be a serious endeavor. A lack of investment in this area will result in investment waste in other areas. For example while a series of very high quality literature reviews and research studies have been produced by acknowledged international and local experts, the insights contained in these documents have not communicated effectively to a range of audiences. This situation ultimately means that the very individuals the products are intended to benefit are not utilizing the knowledge products. LESSON 5: Shifts in socio-cultural norms open up spaces for groups that are excluded from participation. If a project is able to recognize and respond to these openings, it is often the case that a very small investment can yield significant dividends. The Women in Leadership initiative is a case in point: for a small project investment that covered the cost of around 50 – 60 women and men to attend a short workshop and then for a small investment on the part of their home institutions to conduct a short seminar for selected female staff, significant and unexpected dividends emerged. If HELM is able to exploit this obvious low-hanging fruit, the project will have helped bring about a substantive development results that will have far-reaching, and ultimately quantifiable, effects. However, if the project is unable to seize the day and the response to such an opening is timid or unfocused, the opportunity will be lost. LESSON 6: A good indicator can be hard to find. But if a project wants to tell the story of its real achievements as convincingly as possible, good indicators that conform to sound principles must be identified. These indicators perhaps sit beneath or alongside required USAID indicators and they tell the 78 story of achievements that goes beyond reporting on numbers on metrics of interest to the project. Up until now, HELM has not found the right indicators that enable its story to be told and that reveal, rather than obscure, the emerging development results. 79 4. RECOMMENDATIONS Given the fact that HELM has roughly 18 months remaining of solid implementation time before the processes for the closing of the project begin, the MTE team has included only those recommendations that are the most vital to ensuring that the project leaves a solid legacy for the higher education sub￾sector. Each of the recommendations included need to be acted on immediately and with urgency. RECOMMENDATION 1: Hold an intensive workshop to focus HELM implementation on consolidation of results from activities, sustainability and institutionalization for the final two years of the project. Without an all-out effort to regroup, develop and embed a strategic plan into HELM, the end results of the project and the overall development impact will suffer. The parameters of this workshop should include the following: Participants • All HELM technical personnel, including sub-contractor experts, for the full-time frame of the workshop. This strategy should be developed collaboratively with HELM’s high-level technical experts from JBS, the UKy and the Indiana Alliance to identify synergies and ways to consolidate returns on investment; • Key senior DIKTI stakeholders/counterparts who understand HELM in-depth for as much of the workshop timeframe as they can attend; and • Key senior BAN-PT stakeholders/counterparts for selected sessions. Workshop Facilitation • Bring in a skilled and experienced individual capable of facilitating and moderating the process. This individual should also understand HELM and the Indonesian higher education landscape well. Suggested Timeframe • Five days minimum. Workshop Products 1. Action plan: The workshop outputs should include an action plan with a timeline that addresses the 2013 Deliverable 11 (External Review) recommendations. These recommendations were relevant one year ago and they are still relevant and appropriate. The MTE team has made some minor adjustments to ensure that the recommendations reflect the current situation. For ease of reference, these recommendations are included in Annex XVI. 2. Prioritized set of activities: Given the limited time remaining for HELM implementation, the significant activities already solidly underway, and the extent to which HELM technical personnel are already stretched, it is prudent to prioritize the project’s activities. Those initiatives that are currently in a formative state and that show little potential for generating sustainable results over the next 18 months should be dropped, even if a significant investment of resources is directed toward these initiatives. Initiatives that are in an initial formative state and that have little time to come to fruition include the Women in Leadership initiative, the Google E￾University and IDEAL. The exception to this recommendation is the Women in Leadership initiative. Despite the fact that the investment of resources – human and financial – has been limited and the initiative has 80 not been located within a strategic approach to gender integration, nonetheless, the efforts to date have demonstrated the potential to generate significant dividends with respect to gender and development. It is highly likely that the initiative will become the showcase for successful transformation and significant social dividends within and beyond the 50 HELM HEIs, if expert strategic and technical advice is sought and the initiative is handled correctly over the next 18 months. The approach to date has been low-cost at the HELM implementation level and low or no cost at the HEI level. Accordingly, this initiative has the potential to be sustained beyond HELM. In addition to the book under development, it is recommended that a sequel be produced that captures the results that should continue to emerge from the initiative. 3. Sustainability and Institutionalization Strategy. The strategy must: a. Define what these two concepts mean; b. Revisit the original HELM theory of change and make explicit in text and visual form any modifications to the original theory of change; c. Capture and systematize the disparate ways and means HELM is seeking to address sustainability and institutionalization; and d. Include a risk matrix and a set of assumptions that underpin the strategy. 4. Strategic Communications for Development Strategy and Work Plan: Of special importance is the need to take urgent action on a strategic communications platform immediately so that the extensive repository of information and knowledge that has accrued under HELM is disseminated effectively upon platform design and development to different audiences. This strategy should: a. Include a Gallery Walk conference for all 50 HEIs at the end of 2015 where best practices and deep institutional change is shared. This initial effort should be expanded to a national conference, managed by a professional conference organizing company, that showcases the best of HELM results, has pre-conference workshops where individuals can participate in condensed versions of HELM workshops or undertake part of the executive/professional development courseware, is a marketing platform for the executive development courseware and the postgraduate degree programs, and stimulates networking. This conference should be open to any interested HEI personnel and specifically target the participation of associations and their members. High-level consultation with the GOI could result in this conference becoming an annual event. It is strongly recommended that HELM contact STTA with significant experience in strategic communications for development results, including relevant skills in social media and social network analysis; and b. Not include the development of any more video testimonials – unless they are used as a marketing tool for encouraging potential student to engage with the postgraduate program and/or the executive/professional development courses. 5. Scaled down set of indicators. Evaluate the 17 current indicators including some of the standard indicators. For the custom indicators, evaluate their merit against best principles in indicator design and eliminate those indicators that do not conform or restructure them so that they do conform, keeping in mind the challenges of collecting appropriate data. RECOMMENDATION 2: Act immediately to ensure the quality, relevance and appropriateness of the executive/professional development for higher education leadership and management materials. 81 HELM’s assumption is that the executive/professional development courseware for higher education leadership and management that is delivered via a blended learning platform will stand the test of time and sustain HELM investments. This assumption needs to be tested. HELM should engage an internationally recognized expert in cross-cultural instruction design for distance and open/blended learning.58 This individual will: 1. Provide technical input to improve the quality of the instructional materials and teaching methods utilized on the UGM web site. The expert should ensure that a recognized quality review instrument is used for the review of all existing materials. 2. Develop and oversee a rolling testing and revision plan for these materials that takes into account use of the materials in different regions of Indonesia; particularly locations where infrastructure concerns prohibit internet-based delivery platforms. 3. Design and conduct a training of trainers program for individuals from all 25 Cohort 1 HEIs so that they can become internal HEI and local technical experts on the HELM content for the core areas and special initiatives. As a possible part of this effort, HEI personnel could be grouped according to institution types/characteristics to bring a differentiated lens to the common core of HELM materials. HELM could use this engagement with Cohort 1 alumni to introduce performance metrics for the various core areas to consolidate and deepen the benefits of the project. The STTA should work with HELM technical specialists and targeted individuals from with DIKTI and from DIKTI’s stable of external technical trainers; these individuals will be the master trainers of the HELM alumni trainers. RECOMMENDATION 3: Bring the HELM databases up to an internationally standard. To do so, contract STTA with bona fide credentials in database design that can apply accepted principles and norms for database design, including the development of unique identification numbers. HELM monitoring and evaluation personnel will then need to clean and re-interrogate the existing HELM data. RECOMMENDATION 4: Prioritize the approach to the 25 new Cohort 2 institutions in the following ways: 1. Make the ARP initiative the cornerstone of HELM work with these institutions. This could be done by the international STTA working with the 25 Cohort 1 HEIs to institutionalize the ARP process. Establishing an action research unit within the QA unit with the intention that the ARP process becomes an institutional vehicle for change and change management could likewise do this. This would build the capacity of the ARP participants to serve as master mentors within their institution, to replicate the ARP process to others within their home institution and to colleagues within the 25 Cohort 2 HEIs; and 2. Roll out the executive/professional development courses in the 25 Cohort 2 HEIs using the blended learning platform. Involve the Cohort 1 alumni in the delivery of the courses at their home institutions and as mentors for Cohort 2 HEIs (See Recommendation 2). These courses complement the action research process and provide more detail about specific aspects of the various core areas. 58 During the in-country fieldwork period, the MTE team did review the document produced by the HELM expert on blended learning. The document focused on the delivery platforms. This work needs to be framed in an in￾depth review of the quality of the materials and teaching methods. 82 RECOMMENDATION 5: Produce a gender and social equity analysis and gender and social equity integration strategy in collaboration with the 50 HEIs and DIKTI and BAN-PT. An immediate action that could be carried out is to ensure that the Women in Leadership action plans are carried out at all 50 HEIs. The strategy should recognize that HELM has only two years remaining. While the gender and social equity analysis may not make a significant contribution to HELM implementation at this point in time, the output of the exercise may help inform future USAID investments in higher education. Beyond the self-efficacy indicator, ensure that the HEIs track and report on the number of women who move up the ladder as a result of HELM empowerment activities and change in men’s attitudes about and advocacy for female empowerment. ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK I. BACKGROUND Higher Education in Indonesia The higher education sector in Indonesia has expanded as the country’s economy has grown. Currently, the Indonesian higher education system has around 3,700 higher education institutions (HEI), which consists of universities, institutes, colleges, academies, polytechnics, and community colleges. These institutions serve over five million students. Unlike basic education, higher education in Indonesia is still centrally managed by the Ministry of Education and Culture (MOEC) through the Directorate General for Higher Education (DIKTI), which is responsible for managing public and private higher education institutions, and the Ministry of Religious Affairs (MORA), which is responsible for managing public and private faith-based higher education institutions. In addition, several other government ministries and agencies administer over 80 higher education institutes. There is also one Open University nominally managed by MOEC. Indonesia’s higher education landscape is characterized by a huge growth in service provision by private HEI institutions. Of the total number of higher education institutions, 96% are private. Although public institutions represent only 3.7% of the total, they account for 38% of enrollments. The Gross Enrollment Rate (GER) continues to grow, according to DIKTI, 30% of youth in the 19-24 age group were enrolled in higher education in Indonesia in 2012. To increase access, DIKTI provides scholarships for poor students called Bidikmisi whose recipients continue to grow every year. Also, there is cross-subsidy system for tuition fees called Uang Kuliah Tunggal (UKT) or single tuition fee, which is subject to university’s policy, ranging from IDR0 to 8 million (USD$666) per semester. Unfortunately, higher education in Indonesia lacks the quality and relevance needed to adequately address Indonesia’s social and economic challenges or fundamentally advance its social and economic development. Too many students are graduating without relevant knowledge and skills needed to secure employment. Too many faculty members lack the training, knowledge, resources or incentives needed to employ more effective pedagogy, increase the relevance of their courses, conduct cutting-edge research, or develop effective extension and community service programs. So long as this remains the case, the students who receive instruction from the faculty will be constrained in their ability to acquire the knowledge and skills they need to succeed in learning or work. Too many institutions lack the leadership and management practices needed to develop and deliver the incentive structures, resources, quality assurance, financial management, and collaboration with external stakeholders that are vital to improving the quality and relevance of higher education. If leadership and management practices are not significantly strengthened, it is difficult to see how institutions will incorporate and sustain improved practices, let alone best practices, in such areas as faculty development, instructional quality and student support. So long as leadership and management remains weak, higher education quality will be limited. As a result, students will remain under-prepared to succeed in work and under-equipped to contribute to Indonesia’s social and economic development The BAN-PT (Board for Higher Education Accreditation) confirms that the overall quality of many HEIs in Indonesia is limited. Covering all types of HEIs, there are more than 18,000 (2013) study programs offered to students throughout Indonesia, with public institutions leading the way in terms of quality (with the exception of a few well-established private universities). A cause for concern is that a significant proportion (80%) or 14,800 of the total number of study programs have not yet been accredited, due to the limited number of assessors and the large number of study programs. Accreditation59 by BAN-PT provides a picture of the differences in quality between public and private HEIs with public programs being rated significantly higher than private programs. Despite all those challenges, there are a small number of leading universities, such as Universitas Indonesia (UI), Institut Teknologi bandung (ITB), Institut Pertanian Bogor (IPB), Universitas Gajah Mada (UGM), and Universitas Airlangga (UNAIR), all located in Java, as well as Universitas Sumatera Utara (USU) in Sumatra and Universitas Hasanuddin (UNHAS) in Sulawesi, for which many management and leadership challenges may be less severe. Some of these universities have collaborative relationships with sister universities in the region for capacity building or joint research. In order to improve higher education in Indonesia, the Higher Education Law No. 12 of 2012 was passed. The law complements DIKTI’s existing policies and programs, such as KOPERTIS (Coordinator of Private Higher Education Institutions) established in several regions, scholarships for academic and non-academic staff, capacity building activities, and international collaborations. Under the law DIKTI functions such as student affairs, research, human resource development, and collaboration, will be decentralized and merged with KOPERTIS, so that DIKTI can function more efficiently and effectively. Unfortunately, implementation of the Law has been slow. For example, the law stipulates interest free student loans. However under the HELM project scenarios for student loans were developed and presented for student loans, but there has been no effort by DIKTI to act on these proposals. Another example is that while the law classifies community colleges as a type of HEI60, the regulation that provides clarity on format, curriculum, standards etc. is not yet in place. Despite the absence of community college regulation, HELM, has selected 3 community colleges to support in the 4 core areas. Being the largest member of ASEAN61, Indonesia also faces a new challenge from the ASEAN Economic Community (AEC)62, which will result in the free flow of skilled labor in 2015. According to the AEC Blueprint (2008), core competencies (concordance of skills and qualifications) for job/occupational skills required in all services sectors should be available by 59 Assessment criteria includes number of lecturers and academic qualifications, number of graduates, infrastructure (classrooms, library, equipment, and laboratories), teaching and learning process, financial management, external collaboration, and research program. 60 Higher Education Law No. 12 of 2012 stipulates that there are six types of HEIs in Indonesia: academies (akademi), community colleges (akademi komunitas), polytechnics (politeknik), advanced schools (sekolah tinggi), institutes (institut), and universities (universitas). 61 Association of South East Asian Nations, established in August 1967. 62 AEC key characteristics: (a) a single market and production base, (b) a highly competitive economic region, (c) a region of equitable economic development, and (d) a region fully integrated into the global economy. 2015. Unfortunately, the national standards on higher education are still under development, leaving Indonesian HEIs confused as to how produce qualified graduates who meet the AEC requirements. Finally, gender inequality increases as students move up through the various levels of higher education. However, there are variations between public and private institutions. Female students are more highly represented at the diploma level in most institutions, while at the highest level – PhD or S3 – women make up only between 16.6% (in faith-based public institutions) and 36.5% (in general public institutions) of total enrollments. HELM Project Overview The five-year USAID/Indonesia Higher Education Leadership and Management Project (HELM), contract AID-497-C-12-00001, was awarded to Chemonics International Inc. on November 28, 2011 to be completed on November 30, 2016. Chemonics International Inc. is the prime contractor for HELM and works with sub-contract consortium partners: JBS International Inc., University of Kentucky, and the Indiana University Alliance. HELM works in close collaboration with DIKTI and 50 HEI partners and under guidance from the U.S. Agency for International Development (USAID). The project targets increased capacity in four core management areas: 1. General administration and leadership 2. Financial management 3. Quality assurance 4. Collaboration with external stakeholders HELM is committed to programming that responds to needs identified by DIKTI and is similarly committed to advancing the development process among the 50 partner institutions. HELM goals will be achieved through a three-phase process: 1. The first phase (2011-12) consists of an intense, collaborative effort to assess the current context across the higher education sector; 2. The second, implementation phase (2013-15) focuses on capacity building within partner HEIs; 3. The final phase (2016) is considered the institutionalization phase. Institutionalization will be a focus throughout the program but in the final program year an intensified effort will sustain best practices and improve channels for dissemination of good practices. HELM coordinates with other donors and implementers working in the higher education sector and strives to learn from their experiences to build upon the successes of prior and existing projects, seeking to complement existing work and create synergies. Successes and lessons learned are to be shared widely and will remain in the public domain in an effort to disseminate best practices for systemic improvements and to build support for reform within DIKTI and at across the higher education sector as well as across a wide range of stakeholders. Recommendations are intended to link the initial assessment reports to future program implementation activities. The deliverables for the HELM program, as outlined in the contract, are organized under the following key components: A. Design technical assistance approaches to achieve effective implementation of key reforms across system, coordinating with DIKTI and maximizing opportunities to internalize best practice within HE system; B. Provide technical assistance to increase management capacity and improve performance at partner HEIs and disseminate best practices; C. Strengthen graduate-level programs in higher education leadership and management; D. Support special initiatives by providing assistance to advance reforms and innovation within management of HEIs. II. PURPOSE The purpose of this mid-term evaluation is to assess the progress and performance of the HELM project on those four key components mentioned above. More specifically, the evaluation should assess the progress of the HELM project in terms of: 1) Improving the knowledge and skills of higher education staff in the four core areas; 2) Supporting the partner HEIs in implementing their thematic action plans; and 3) Contributing to improved processes and systems at the institutional level. This evaluation should measure HELM’s progress in meeting these three goals met and the factors that have been responsible for, or detracted from, the achievement of these goals. The evaluators will be expected to identify design, programmatic, management, and/or financial obstacles and challenges affecting program implementation and recommend any changes in program or management strategies that would increase the efficiency and impact of the program. The evaluation will cover the HELM project performance up to the mid-point of the project (approximately 2.5 years of project implementation). In addition, the evaluation should inform management decisions regarding implementation for the remainder of the project, such as whether the project should be changed to improve effectiveness. Finally, this evaluation should provide an initial assessment of the sustainability of project’s achievements, the factors that have contributed to or detracted from the sustainability of project’s achievements, the lessons learned, and recommendations. III. EVALUATION QUESTIONS The contractor shall provide evaluation services sufficient to achieve the objectives set forth above. Specifically, the evaluation should address the questions that follow. 1. To what extent has HELM achieved its stated mid-point objectives and outcomes in a timely and effective manner? 2. Is HELM on track in terms of meeting its overall end-of-project goals? 3. What aspects of HELM are proving most and least effective in building capacity of individuals and institutions participating in HELM activities? 4. To what extent have indirect beneficiaries (HELM stakeholders) benefited from the project’s activities and what specific value has been added? 5. To what extent are the HELM project’s resources being implemented and managed efficiently and cost effectively? 6. What evidence is there to show that the HELM project’s activities and results are making progress towards sustainability and replication after the project is completed? 7. How effective has each component of the HELM project been in improving women participation? 8. Is the HELM project design, structure and approaches effective in achieving the desired results and, if not, how should the project be reorganized?” The contractor shall present evaluation findings to substantiate answers to these evaluation questions; findings that are based on evidence and data. Findings should be specific, concise, and supported by quantitative and qualitative information that is reliable, valid, and generalizable. Recommendations must be action-oriented, practical and specific. IV. METHODOLOGY The contractor shall propose an evaluation methodology or mix of methodologies for answering for the questions above that is effective and cost efficient. The contractor will examine both quantitative and qualitative approaches and a combination of secondary (existing) and primary (new) data during the course of the evaluation. Offerors are encouraged to include the following stakeholders in the evaluation: HELM Stakeholders: • USAID/Indonesia HELM team members (COP, DCOP, core specialists, instructional designer, M&E specialist, and graduate program adviser); • Indonesian service providers, such as Universitas Gajah Mada, Yogyakarta and Universitas Brawijaya, Malang; • MOEC/DIKTI, MORA and Kemenko Kesra officials of relevant units; • Rectors, vice rectors, directors, heads of units at partner HEIs (representing public and private universities, faith based universities, polytechnics, and community colleges); • Participants of in-country workshops and action research program; and • Students participating in the pilot student engagement survey conducted in 2013. The contractor must follow the evaluation policy guidance published in the January 2011 document entitled, “USAID Evaluation Policy.” In particular, the contractor should carefully review section 5 entitled, “Evaluation Requirements.” One example of the many points highlighted in this policy, the evaluation should use sex-disaggregated data and incorporate attention to gender relations in all relevant areas. V. COMPOSITION OF THE EVALUATION TEAM There are two key positions outlined in this RFP, the Evaluation Team Leader (1) and the Evaluation Expert (1). Offerors must provide CVs for these positions. Beyond these two key positions the offeror may propose additional Indonesian personnel, such as project evaluator(s), administrative/logistic officer(s), and translator(s). Evaluation Team Leader (expatriate): The Team Leader should possess graduate-level degree (Ph.D. or master’s degree) in education (preferably higher education), social sciences, or a related relevant field. The Team Leader should also have a minimum of five years of working experience with higher education evaluations, ten years of working experience with higher education activities and prior experience working in Indonesia. Evaluation Expert: The Evaluation Expert should possess graduate-level degree (Ph.D. or master’s degree) in education (preferably higher education) and should have a minimum of seven years of planning and evaluating education assistance projects. S/he must also have specific skills in evaluation methodology and planning, including demonstrated training and/or experience in developing evaluation methodologies and managing teams in primary data collection. Experience working in Indonesia is preferred. In addition, specific skills in gender analysis are an asset. The full composition of the evaluation team and the roles of the Evaluation Team Leader, the Evaluation Expert and other team members should be defined and delineated in the Technical Proposal. VI. LIST OF DOCUMENTS FOR EVALUATION TEAM TO REVIEW The following are a list of documents pertinent to HELM that will be forwarded to the Evaluation Team for review prior their arrival in Indonesia. • Quarterly reports and Annual reports • Annual Work Plans • Technical reports • Scope of Work including Amendments to the Scope of Work • Performance Management Plan (PMP) • External Assessment report • Modules, tools, and training materials • The Agency’s recent “USAID Evaluation Policy” report and ADS 203