EVALUATION FINAL PERFORMANCE EVALUATION OF THE HIGHER EDUCATION FOR ECONOMIC GROWTH ACTIVITY This report was produced at the request of the United States Agency for International Development. It was prepared independently by Megan Gavin, Team Leader, Cristina Accioly, and Oscar Picardo under the USAID/El Salvador Monitoring, Evaluation and Learning Initiative. FINAL PERFORMANCE EVALUATION OF THE HIGHER EDUCATION FOR ECONOMIC GROWTH ACTIVITY May 17, 2018 Prepared for the United States Agency for International Development Under AID-OAA-I-15-00024/TOAID-519-TO-16-00002 USAID/El Salvador Contact: Orlando Hidalgo ohidalgo@usaid.gov USAID/El Salvador Monitoring, Evaluation, and Learning Initiative Contact: Randal Thompson rthompson@engl.com Cover photo caption: Agronomic Engineering student, Carmen Esmeralda Hernández León, doing an internship for a research project on “Conservation and Management of Phylogenetic Resources of Robusta Coffee and Criollo Cocoa in Productive Agroecological Systems of El Salvador in the Face of Climate Change,” funded by the Higher Education Activity. Photo Credit: Communications Department, Research Triangle Institute. DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ABSTRACT The purpose of the final performance evaluation of the Higher Education for Economic Growth Activity (HEA) is to provide recommendations to: 1) the Activity for the final year; and 2) the United States Agency for International Development (USAID) for future programming. The evaluation focused on five evaluation questions (EQs), including: assessing results (EQ 1); the role of clusters (EQ 2); the government and higher education institutions (EQ 3); gender (EQ 4); and analyzing the extent to which results will be sustained (EQ 5). The evaluation used a mixed-methods approach, including document review; analysis of HEA’s monitoring and evaluation data and performance indicators; key informant interviews and surveys for faculty/staff; focus group discussions; and direct observation. Key illustrative findings include: 1) teachers consider diploma programs available, high quality, and well￾facilitated; 2) trust has been established via positive and significant effects from collaboration; 3) while there has been an established approach to construct a higher education policy, the approach needs to be broader in terms of participants (including the government); 4) there is improved access to science, technology, engineering, and math for women and girls; and 5) the Centers for Career Development, which promote student engagement with the private sector, are the most likely to sustain. The evaluation recommends that USAID continue to support the investigations, curriculum reform implementation, teacher training, and scholarships, as well initiate as the construction of an entity to strategically integrate and give continuity to the cluster model. CONTENTS EXECUTIVE SUMMARY.................................................................................................................................. i 1.0 EVALUATION PURPOSE AND QUESTIONS ................................................................................. 1 1.1 Evaluation Purpose ................................................................................................................ 1 1.2 Evaluation Questions............................................................................................................ 1 2.0 PROJECT BACKGROUND................................................................................................................... 2 3.0 EVALUATION METHODOLOGY ...................................................................................................... 3 3.1 Approach ................................................................................................................................. 3 3.2 Literature................................................................................................................................. 3 3.3 Interviews and FGDs............................................................................................................ 3 3.4 Site Visits................................................................................................................................. 4 3.5 Data Analysis.......................................................................................................................... 5 3.6 Limitations............................................................................................................................... 5 4.0 FINDINGS AND CONCLUSIONS...................................................................................................... 6 4.1 EQ 1: What have been the most significant intended and unintended results achieved to date?................................................................................................................... 6 4.1.1 What have been the main internal and external factors that have influenced the achievement or non-achievement of the Activity’s expected results as planned?.............................................................................................................. 11 4.2 EQ 2: To what extent has the demand-driven cluster model been an extensive space to build dialogue...................................................................................................... 12 4.2.1 To what extent has the cluster model been attractive to engage private sector and obtain their commitment through leverage?........................................... 12 4.3 EQ 3: What changes have been made in the GoeS and hEIs in terms of their institutional capacity........................................................................................................... 17 4.4 EQ 4: How were gender, female empowerment, and social inclusion integrated in implementation? ............................................................................................................. 20 4.4.1 To what extent is the Activity influencing changes in gender participation in higher education?........................................................................................................... 21 4.5 EQ 5: What approaches and results by the HEA have the potential to exist after USAID funding ends? which ones should continue to be supported?.................... 23 5.0 RECOMMENDATIONS....................................................................................................................... 26 5.1 Recommendation 1............................................................................................................ 26 5.2 Recommendation 2............................................................................................................ 27 5.3 Recommendation 3............................................................................................................ 28 5.4 Recommendation 4............................................................................................................ 28 5.5 Recommendation 5............................................................................................................ 29 ANNEXES ....................................................................................................................................................... 30 Annex 1: Statement of Work ......................................................................................................... 31 Annex 2: Team Composition and Conflict of Interest Cisclosures....................................... 43 Annex 3: Protocols/Instruments.................................................................................................... 49 Annex 4: Documents Consulted.................................................................................................... 56 Annex 5: Schedule of Key Informant Interviews........................................................................ 58 Annex 6: Supplemental Analysis for Question 3 ........................................................................ 60 Annex 7: Illustrative Examples for Gender Indicators.............................................................. 62 Annex 8: Evaluation Plan.................................................................................................................. 67 Annex 9: Statements of Difference of RTI and Response from ET........................................ 89 List of Tables Table 1: Evaluation Team Site Visits............................................................................................................. 4 Table 2: Principles of Adult Learning Scale Analysis................................................................................. 6 Table 3: Impact Condition of the Clusters’ Determinants................................................................... 15 Table 4: Aspects That Will Continue After the Activity If HEA Provides Support........................ 24 List of Figures Figure 1: KIIs Identified the Key Achievements......................................................................................... 9 Figure 2: Result Clusters................................................................................................................................. 9 Figure 3: Increased Communication............................................................................................................. 9 Figure 4: Conceptual Model to Analyze Clusters’ Development and Effectiveness....................... 12 Figure 5: KIIs That Reported on Importance of Gender Equity, Women in Leadership Roles, and Girls/Women in STEM Fields............................................................................................... 21 Figure 6: Student Ratings of Career Development Center Services................................................. 23 Figure 7: Additional Career Development Center Services................................................................ 24 ACRONYMS APP Annual Performance Plan APR Annual Performance Report ASER Spanish acronym for Asociación Salvadoreña de Energía Renovable (Salvadoran Association for Renewable Energy) ASI Spanish acronym for Asociación Salvadoreña de Industriales (Salvadoran Industrial Association) CAMAGRO Spanish acronym for Cámara Agropecuaria y Agroindustrial de El Salvador (Agricultural and Agro-industrial Chamber of El Salvador) CASATIC Spanish acronym for Cámara Salvadoreña de Tecnologías de la Información y las Comunicaciones (Salvadoran Chamber of Information and Communications Technologies) CDC Career Development Center CDCS Country Development Cooperation Strategy CEN U.S. Government Strategy for Engagement in Central America CONACYT Spanish acronym for Consejo Nacional de Ciencia y Tecnología (National Council of Science and Technology) COR Contracting Officer’s Representative CV Curriculum Vitae DNES Spanish acronym for Dirección Nacional de Educación Superior (National Directorate of Higher Education) DEC USAID’s Development Experience Clearinghouse DO Development Objective EQ Evaluation Question ET Evaluation Team FEDISAL Spanish acronym for Fundación para la Educación Integral Salvadoreña (Salvadoran Foundation for Integral Education) FGD Focus Group Discussion FY Fiscal Year GOES Government of El Salvador HEA Higher Education for Economic Growth Activity HEI Higher Education Institution HICD Human and Institutional Capacity Development ICT Information and Communications Technology IP Intellectual Property IR Intermediate Result KII Key Informant Interview LEED Leadership in Energy and Environmental Design M&E Monitoring and Evaluation MEL Monitoring, Evaluation, and Learning MINEC Ministry of Economy MINED Ministry of Education MOU Memorandum of Understanding N-CONACYT Spanish acronym for Nuevo Consejo Nacional de Ciencia y Tecnología (New National Council on Science and Technology) NGO Non-Governmental Organization PALS Principles of Adult Learning Scale R&D Research and Development RTI Research Triangle Institute SI Social Inclusion SOW Statement of Work STEM Science, Technology, Engineering, and Mathematics TA Technical Assistance TBD To Be Determined ToT Training of Trainers U.S. United States UCA Spanish acronym for Universidad Centroamericana “José Simeón Cañas” (Central American University “José Simeón Cañas”) UDB Spanish acronym for Universidad Don Bosco (Don Bosco University) UES Spanish acronym for Universidad de El Salvador (El Salvador University) UFG Spanish acronym for Universidad Francisco Gavidia (Francisco Gavidia University) UNICAES Spanish acronym for Universidad Católica de El Salvador (El Salvador Catholic University) UNIVO Spanish acronym for Universidad de Oriente (Oriental University) USAID United States Agency for International Development USG United States Government i EXECUTIVE SUMMARY The purpose of the final performance evaluation of the Higher Education for Economic Growth Activity (HEA) is two-fold: 1) provide an evaluation of and recommendations for the Activity in its final year; and 2) provide the United States Agency for International Development (USAID) with recommendations for future programming. HEA is a US$22 million project, implemented from 2014-2019 by Research Triangle Institute (RTI), with support from two subcontractors, World Learning and Rutgers University, and one grantee, the Salvadoran Foundation for Integral Education (FEDISAL). Within USAID’s Country Development Cooperation Strategy (CDCS) 2015-2017, the HEA contributes to Development Objective (DO) 2 “Economic Growth Opportunities in Tradables Expanded.” HEA builds partnerships between industry sectors and local higher education institutions (HEIs) to develop demand-driven educational programs and research. The purpose is to train highly-qualified professionals and contribute to industry growth and productivity, as well as help stimulate the economy and social development. The evaluation of HEA focused on the five evaluation questions (EQs) detailed below. They ranged from assessing results (EQ 1) to analyzing the extent to which they are sustainable (EQ 5). There were specific questions that examined the role of clusters (EQ 2), the government and HEIs (EQ 3), and gender (EQ 4). The evaluation used a mixed-methods approach that included key informant interviews (KIIs), focus group discussions (FGDs), direct observations, analysis of secondary data, and literature review. The evaluation team (ET) conducted over 25 KIIs as well as a total of seven FGDs with professors and students in seven universities. In addition, the ET consulted a range of stakeholders, including the implementers, RTI and its subcontractors; the client, USAID; the government; other implementing partners (IPs); and local non-governmental organizations (NGOs). A wide range of literature was reviewed, including project documentation and outside research, particularly of clusters. These sources afforded the ET the ability to triangulate findings and therefore build their reliability and validity. This process led to the conclusions, which clearly align with and support the recommendations being made. For a detailed table on the intended and unintended results, see EQ 1, and for clear guidance on areas where USAID could focus for a new design see the recommendations section. EQ 1: What have been the most significant intended and unintended results achieved to date? 1.1. What have been the main internal and external factors that have influenced the achievement or non-achievement of the Activity’s expected results as planned? According to participants, the most significant result from the project relates to completion of applied research that will be used by the relevant industry within a short period of time to improve commercial competitiveness. Twenty-six (26) studies were funded that allowed for applied research. Importantly, Rutgers also trained 100 researchers on how to complete applied research studies. KII participants stated that they had never received trainings on applied research before and felt that this training allowed them to better conduct applied research that was relevant to the economy of El Salvador. Furthermore, the training and completion of applied research allowed the academics to link directly with industry representatives on 26 studies, strengthening skill sets for academic-private sector research collaboration and setting a precedent for this sort of joint work. The most significant unintended result came about due to the project’s cross-cutting focus on gender and due to the fact that the project promoted communication, both within and between the universities. This joint focus resulted in the development of Girls Science Camps and the Network of Women Leaders, which were not initially planned to be part of the project but which have contributed ii significantly towards HEA goals and towards the DOs in the CDCS. There is more information on these in EQ 4. Evaluation Question 1 Findings Conclusions Teacher Formation Teachers reflect positively on the diplomas available [21st century, Training of Trainers (ToT) in English, publishing in English, etc.]. 1. The teacher training activities were considered high quality and well facilitated, although what participants learned can be shared more throughout the university. Few teachers have shared what they learned with other teachers in the university. In some cases, training opportunities for teachers were not adequately detailed or given sufficient time. 2. There were information deficits on the training programs and more people want more opportunities. Despite gains, equal gender participation has not yet been fully achieved. Teachers wanted to take more than one course but, in some universities, were restricted. Two hundred and fifty (250) men and 150 women took part in teacher training opportunities, indicating more men than women participated (although there are more males in the profession). Scholarships – Students and Teachers Students report they would not be able to afford all aspects of their studies without the scholarships, indicating a high demand and need. 3. Students and teachers greatly benefit from the scholarships, and there is demand for more—both in extended length of time and the number of student beneficiaries. 1 Teachers are enrolled in Master’s degree programs online/in person, and there have been five graduates to date. Universities seek opportunities to establish PhDs in order to strengthen academia. Curriculum Reform Programs were reformed to include competencies and experiential learning (28 approved by February 2018). 4. Despite delays, the new fields of study are currently being implemented in the universities, although more support is required, especially on competencies. There is a lack of implementation of teaching-learning based on competencies-based curriculum. There were delays in the approval on the part of the Government of El Salvador (GOES) of the new fields; however, this has improved within the 90 days from submitting the revised curriculum (see EQ 3). Applied Research Applied research was more systematic thanks to the Technical Assistance (TA) of the project. 5. Applied research is reportedly of a higher quality and based on industry needs. Now is the time to capitalize and share results with consumers. Applied research was based on the demands and needs of industry, whereas before there was not the same direct link with industry. There remains great demand on the part of universities for their research to be supported. 2 Although all applied research projects are supported by HEA in methodology development, publishing, and general 1 See body of report for gender disaggregation. 2 The evidence is that the project received far more proposals than it could fund, and these studies continue to need resources in order to be conducted. iii Evaluation Question 1 Findings Conclusions management, there is a gap in feedback loops and coaching is needed to improve research quality. HEA has not sufficiently disseminated research findings to date, especially within the industry. EQ 2: To what extent has the demand-driven model (clusters model) been an effective space to build dialogue within private sector, academia, and GOES in order to respond to private sector needs and priorities? 2.1 Are the four clusters already sustainable? 2.2 To what extent has the cluster model been attractive to engage the private sector and obtain its commitment through leverage? Evaluation Question 2 Findings Conclusions Collaboration There are positive and significant effects from collaboration for clusters. Industry and university find it mutually beneficial to collaborate. 6. There is a strong collaboration and trust between universities and industry. The consultative process for curriculum and program development strengthened trust and collaboration. Benefits and Barriers Clusters created collaborative research and development (R&D), joint projects, and a process for curriculum development. 7. There is solid collaboration in research, but the lack of timely implementation, support on publication writing and knowledge sharing could compromise the collaboration momentum. Clusters are behind schedule to produce results such as research products. There is limited cross-fertilization of knowledge and best practices within and across clusters. Sustainability Most interviewees were highly enthusiastic about continuing with the cluster model. 8. If well-implemented, the clusters could be a critical component of national innovation but creating a public policy and funding mechanism for research and innovation is strategically necessary to ensure sustainability. The lack of public policy to support cluster growth presents a barrier (funds to R&D projects, intellectual property regulation, and business incubators). There are limited policy incentives to consider the promotion of university-industry linkages as part of a broader science, technology, and innovation policy program. EQ 3: What changes have been made in HEIs in terms of their institutional capacity to continue introducing and/or supporting educational reforms and/or academic programs in the future? 3.1 What changes have been made in the GOES in terms of their institutional capacity to continue introducing and/or supporting educational reforms and/or academic programs in the future? Evaluation Question 3 Findings Conclusions The dialogue and closeness between sectors are important project achievements because they facilitated the development of investigations, curriculum, and the Career Development Centers (CDCs). 9. There needs to be leadership from a neutral entity. There has been an open and participatory process to construct the higher education policy and this process should continue. iv Evaluation Question 3 Findings Conclusions The GOES does not feel ownership of the project, which is crucial for sustainability. However, GOES does regard the project positively. There was little awareness of the bureaucratic aspects within the GOES in the design of new programs of study, especially on the part of the National Directorate of Higher Education (DNES). 10. The project was not fully conscious of the day to day challenges of GOES, which created unforeseen delays. 3 The written reforms are outlined in the higher education policy proposal, but they are not in operation/implementation (which requires commitment from government). 11. The policy needs to be assumed by the GOES and to do so they need to be included in the process (i.e., have ownership). Human institutional capacity development (HICD) assessment was instrumental to identifying the gaps in administration on the part of universities and providing a direction to improve capacity; however, there is no endline to date to assess the gains. 12. More data are required to assess the final impact of the HICD training. EQ 4: How were gender equality, female empowerment, and social inclusion integrated in the implementation of activity interventions? 4.1 To what extent is the activity influencing changes in gender participation in higher education? Evaluation Question 4 Findings Conclusions There has been an increase in access to science, technology, engineering, and mathematics (STEM) for women and girls. Most notable is the Girls Science Camp, with approximately 70 girl beneficiaries. 13. HEA has made substantial gains to improve access to higher education opportunities for women and girls. However, the project can be more expansive and include both genders to promote more involvement (including awareness campaigns). A Higher Education Network of Women Leaders in higher education is established and functioning, although there is no male involvement. Research teams include some women, but can include more; to date, 78 males and 35 females participated on research teams (although, significantly, these are young women with growth potential). The HEA has offered workshops to other IPs on how to integrate gender and female empowerment into programming. 14. HEA plays a leadership role in terms of exemplifying how to incorporate gender, and more research and data will continue to demonstrate achievements and make convincing arguments for society at large. 4 There is a lack of public messaging and convincing narratives on why girls should be educated, particularly about the benefits to industry and for their potential. There are no gender-specific indicators in the Monitoring, Evaluation, and Learning (MEL) plan. In terms of Social Inclusion (SI), universities are not accessible to students with disabilities. 15. Universities can be more inclusive to all students. 3 The HEA Activity disagrees with certain of the factual statements by the ET and their Statement of Difference is included in Annex 9 of this report. 4 While the project’s purpose was not to change universities’ gender policies, in the act of requiring equitable participation in trainings, scholarships, investigation teams, the project was a leader exemplifying to universities equitable gender practices. v EQ 5: What approaches utilized and results achieved by the HEA Activity have the potential to continue to exist after USAID’s funding ends? Which ones should continue to be supported?5 Evaluation Question 5 Findings Conclusions Students who use the CDC reflect positively on the workshops [especially entrepreneurship as well as those on curriculum vitae (CV), interview, and soft skills]. 16. Demand for the CDC is high on the part of all students and industry. CDC staff confirm that interest is high for the workshops and internships. CDC staff report increased participation6 from industry [i.e., memoranda of understanding (MOUs), participating in role plays, job posts] in the CDCs, with no gender differences (i.e., equal participation). CDCs are part of the budgeted plans, aesthetically pleasing, and aligned to curriculum (social service). 17. CDCs are not isolated but are integrated into the universities, which implies they are sustainable with little funding. Some CDCs are advanced while others are still being developed. 7 18. There is an opportunity to capitalize on the tour and share more between CDCs in El Salvador so that they learn from each other and collaborate effectively and in a timely manner. Participants in the CDC study tour to the United States (U.S.) find the experience was valuable. CDC staff are highly motivated and energetic vis-à-vis the goals of the CDCs. There is a lack of consolidated results of the CDCs that are ready to be shared and detail the benefits of the CDCs. The full report provides more detail on EQ 5, which cuts across all questions, e.g., the clusters have attributes that will continue to exist, gender advances have some attributes that will continue, etc. In these questions (e.g., EQs 2 and 4) the project’s aspects that will continue are discussed. The recommendations below are based on the conclusions outlined above. The body of the report provides more detail, especially delineating between future programming and the project’s actions. Recommendation 1: USAID should support the construction of an entity or contract a local NGO or an organization with strategic alliances to integrate the cluster model.8 This recommendation is based on conclusions 6, 8, 9, and 10. Recommendation 2: USAID should continue to support (in order of importance) research, curriculum reform, teacher training, and scholarship opportunities. This recommendation is based on conclusions 1, 2, 3, 4, and 5. Recommendation 3: USAID needs to continue to strategically incorporate gender equity and social inclusion into higher education programming. This recommendation is based on conclusions 13, 14, and 15. 5 Following the evaluation period, a sustainability plan has been developed for the Cluster model. The research units/labs have identified and further developed strategies for sustainability through their links with Industry. 6 Industry has participated in CDC trainings serving in a role play capacity with student interviews, job recruitment, and signing MOUs. 7 For example, in having staff to provide services, offering career development trainings, having a physical space adequate to meet with students, and including the CDC in university plans/budgets. 8 Since the end of the evaluation, the ET understands HEA has developed a sustainability plan for the clusters. vi Recommendation 4: USAID needs to include the government more in higher education policy development with USAID, and the process needs to be more open to include other government agencies, NGOs, etc. This is based on conclusions 10 and 119. Recommendation 5: The HEA should improve data management and knowledge sharing and learning throughout the system; a clearinghouse website [different from the Development Clearing House (DEC)] should be created on which information is made available to more stakeholders. This is based on conclusions 7, 12, 14, and 18. 9 Following the completion of the data collection and the scope of this evaluation, the call to participate according to one KII was broad: 75 percent out of the 41 Higher Education Institutions (HEIs) participated. The Strategic Committee also participated including staff from: Ministry of Education, Ministry of Economy, the Concejo de Educación Superior (CES), and the Comisión para la Acreditación de la Calidad en la Educación Superior (CDA). In all workshops, there was participation from the government, the private sector, the HEIs and students. 1 1.0 EVALUATION PURPOSE AND QUESTIONS 1.1 EVALUATION PURPOSE The purpose of the final performance evaluation of the Higher Education for Economic Growth Activity [also referred to as the Higher Education Activity (HEA)] is to provide inputs to improve the Activity and future United States Agency for International Development (USAID) programming in the higher education sector in El Salvador. Specifically, the evaluation will inform USAID of: a) the Activity’s achievements and challenges to date to meet or surpass contract objectives and assure sustainability; b) main approaches and interventions supported by the Activity that have been valuable to achieve HEA’s goal; and c) what worked well and what did not, based on lessons learned. In addition, the evaluation will provide recommendations that will serve as input for the design of any future USAID higher education activity. The evaluation’s results will be used for reporting purposes to stakeholders, such as the Government of El Salvador (GOES), private sector representatives, and academia. 1.2 EVALUATION QUESTIONS USAID identified five key evaluation questions (EQs), in priority order, for this evaluation. These are listed below, along with suggested sub-questions to clarify the intent of the key questions. The evaluation focused on the key questions, both in the design of instruments and analysis tools, and in the overall findings, conclusions, and recommendations of the evaluation report. The questions include: 1. What have been the most significant intended and unintended results achieved to date? 1.1 What have been the main internal and external factors that have influenced the achievement or non-achievement of the Activity’s expected results as planned? 2. To what extent has the demand-driven model (clusters model) been an effective space to build dialogue within private sector, academia, and GOES in order to respond to private sector needs and priorities? 2.1 Are the four clusters already sustainable? 2.2 To what extent has the cluster model been attractive to engage the private sector and obtain its commitment through leverage? 3. What changes have been made in higher education institutes (HEIs) in terms of their institutional capacity to continue introducing and/or supporting educational reforms and/or academic programs in the future? 3.1 What changes have been made in the GOES in terms of their institutional capacity to continue introducing and/or supporting educational reforms and/or academic programs in the future? 4. How were gender equality, female empowerment, and social inclusion integrated in the implementation of Activity interventions? 4.1 To what extent is the Activity influencing changes in gender participation in higher education? 5. What approaches utilized and results achieved by the HEA have the potential to continue to exist after USAID’s funding ends? Which ones should continue to be supported? 2 2.0 PROJECT BACKGROUND The HEA project (award # AID-519-C-14-00004, US$22,000,000) is implemented by Research Triangle Institute (RTI) with support from two subcontractors, World Learning and Rutgers University, and one grantee, the Salvadoran Foundation for Integral Education (FEDISAL). The projects runs from June 6, 2014 to June 5, 2019. Within USAID’s Country Development Cooperation Strategy (CDCS) 2015-2017, the HEA contributes to Development Objective (DO) 2 “Economic Growth Opportunities in Tradables Expanded.” Specifically, it contributes to Intermediate Result (IR) 2.2 “Productivity of Targeted Businesses Increased” and, jointly with the USAID Bridges to Employment Activity, Young Entrepreneurs Activity, and USAID Economic Competitiveness Activity, to Sub-IR 2.2.1 “Higher Education and Workforce Competencies Strengthened.” HEA contributes to the achievement of Objective 1 of the United States (U.S.) Government Strategy for Engagement in Central America (CEN Strategy) of “Prosperity and Regional Integration” (education and workforce development), and the second strategic line of action in the Plan of the Alliance for Prosperity in the Northern Triangle of “Developing Opportunities for Our People” (building human capital). The Activity contributes to Goal 2 of the USAID’s Education Strategy “Improved ability of tertiary and workforce development programs to generate workforce skills relevant to a country’s development goals,” and is aligned with the line of action in GOES’s Five-Year Plan of “Strengthening the coverage and quality of higher education, articulating it with other levels of the education system, and fostering research and knowledge generation.” The development hypothesis of the HEA states that “if El Salvador’s higher education system is strengthened and aligned with private productive priority sector needs, then competitiveness and productivity will improve, contributing to long-term, broad-based economic growth.” The development hypothesis assumes that if the higher education sector of El Salvador improves the capacity of faculty staff, aligns curricula and research with labor market demand, enables key educational institutional and systemic reforms, and improves collaboration among local and international higher education institutes, public and private sectors, and donors, then the relevance and quality of the higher education system’s response to priority sector needs will be increased, productivity improved, and long-term economic growth enhanced. HEA builds partnerships between industry sectors and local HEIs to develop demand-driven educational programs and research. The purpose is to train highly-qualified professionals and contribute to industry growth and productivity, as well as help stimulate the economy and social development. The Activity strengthens the higher education system to respond to the country’s productive sector needs by improving the relevance and quality of curricula and applied research, focusing on academic programs that respond to the needs of the labor market, and providing faculty and professionals with opportunities to advance their industry knowledge. HEA is implementing an industry demand-driven education model through four industry-higher education clusters in four sectors: • Information and Communications Technology (ICT); • Energy and energy efficiency; • Light manufacturing; and • Agro-industry and food processing. Each cluster includes the private sector, an anchor university, and associate HEIs. It also includes participation from the government. The collaboration between industry and HEIs helps create and update curricula that articulate skills requirements defined by industry for high-demand careers in growing sectors. Industry Advisory Boards are created to strengthen the link between educators and 3 business leaders, and to share information on the job market, curriculum planning, internships, and practical training. USAID/El Salvador also helps develop the capacity of HEIs to perform applied research, thus meeting the technical and technology-related challenges of Salvadoran industry. It offers scholarships to Salvadoran faculty and researchers to study innovative Professional Science Master’s degree programs. 3.0 EVALUATION METHODOLOGY The Evaluation Team (ET) used a mixed-methods approach to conduct the HEA evaluation. This included: a review of relevant documents; analysis of HEA’s performance data and performance indicators; key informant interviews (KIIs) and surveys for faculty/staff; focus group discussions (FGDs); and direct observation at Activity sites. Question protocols are included in Annex 3. With regard to the time and schedule, the evaluation was procured in January 2018 and the final report was completed in May 2018. The ET began fieldwork in San Salvador on February 13, 2018. The ET interviewed USAID/El Salvador, the HEA staff, HEA partners, GOES, and the private sector, and conducted site visits between February 13 and 28, 2018. Annex V identifies the date on which each KII was conducted while Table 1, below, identifies the date on which each site visit was conducted. 3.1 APPROACH The ET drew on utilization-focused methodologies to ensure that the information generated by the evaluation is useful to USAID. The team used the initial in-brief call to confirm USAID/El Salvador’s goals and objectives, as well as the type of information and insights of most use to USAID’s decision-making. On February 15, 2018, the ET also hosted an introductory breakfast with the implementer, RTI, and the universities (see Table 1) to understand their perspectives before delving into the field. Consistent with USAID’s Evaluation Policy, the ET applied a gender perspective given that results and/or outcomes can and might occur differently for people with different gender or gender identities. Starting with the background document review, the team examined data related to gender-based variances in outcomes, and reviewed Activity documents with a gender-sensitive lens in order to have a better understanding of gender dynamics in the implementation environment. Questions were also sensitively posed to participants. 3.2 LITERATURE During the initial phase of the evaluation, the ET reviewed background documents related to HEA and its implementing environment. These documents gave the team a deeper understanding of the Activity’s operations to date. They helped the ET finalize the evaluation design and data collection tools, and informed the overall Activity evaluation process, including developing findings, conclusions, and recommendations for future programming (see Annex 4 for a list of documents reviewed). In addition, each researcher on the team did his/her own supplemental review; for example, one researcher delved into clusters, while another examined the history of HEIs in El Salvador. 3.3 INTERVIEWS AND FGDS The ET conducted interviews with USAID/El Salvador, RTI staff and HEA partners, the GOES, the private sector, and participating HEIs, for a total of 24 KIIs (see Annex 5), of which 10 were women. Data collection protocols that address the EQs guided the interviews, but they were semi-structured (identifying probes to follow up on information related to key questions) to allow flexibility in the discussion and a natural flow to the conversation when the ET identified a point of particular interest to the research questions. FGDs were conducted during field visits with university faculty and staff. Each FGD included between 5-7 participants. The FGDs had equal participation of women and men and there 4 were no detectable differences in their remarks. Data was registered electronically and provided to the Monitoring, Evaluation, and Learning (M&E&L) Initiative. I In addition, staff who participated in FGDs completed a brief survey on teacher practices (see Annex 3). The survey was presented while participants congregated for the FGD. A total of 41 surveys were collected. The survey was taken from the HEA MELP and as such was reviewed by RTI and USAID under a previous circumstance. The survey was self-administered via paper and pencil/ or pen. The survey was anonymous. Data was transferred to excel and analyzed generating frequencies and percentages. 3.4 SITE VISITS An introductory breakfast was held with six universities on February 15, 2018. The purpose of the event was to introduce the ET as well as present the purpose of the study and the five EQs. Participants from universities presented their initial perspectives of the HEA. In addition, this breakfast served as a springboard to coordinate the site visits. It is important to note that these were the six anchor universities for the clusters. These six universities were the six universities outlined in the SoW produced for this evaluation under the guidance of USAID and the review of RTI. This proved to be valuable selection because it allowed all aspects of the intervention to be evaluated. In other words not all universities served by the HEA participate as cluster leaders nor do they all have CDCs. In addition to KIIs and FGDs, field notes were collected which allowed aspects captured from touring the facilities to be documented; this was especially the case for the CDCs. The ET, in consultation with USAID/El Salvador and RTI, selected relevant site visits based on a sampling plan that was developed for the evaluation. Six campuses of the beneficiary HEIs identified below were visited more than once on several occasions. The KIIs and FGDs mentioned under Section 3.3 were conducted at these sites. The ET recorded field notes and data from these events as inputs for the report. To capture the depth of intervention effectiveness within the given data collection timeframe and team size, the ET conducted interviews with directors of the clusters, directors of investigation and others involved in investigation, directors of curriculum and those involved in curriculum, and participants in the Career Development Centers (CDCs), including students (see Table 1). Table 1: Evaluation Team Site Visits Site Visits Date HEI Municipality February 19, 2018 Central American University (UCA) San Salvador February 27, 2018 El Salvador University (UES) San Salvador February 20, 2018 Francisco Gavidia University (UFG) San Salvador February 21, 2018 Don Bosco University (UDB) Soyapango (Metropolitan San Salvador) February 22, 2018 El Salvador Catholic University (UNICAES) Santa Ana (64 Kilometers northwest of San Salvador) February 23, 2018 Oriental University (UNIVO) San Miguel (138 Kilometers east of San Salvador) In the case of site visits, participants for FGDs/KIIs were chosen by a point of contact at the university. The ET established selection criteria (participation time in training, sex, and subject area). However, this method was still a limitation because of the tendency for representatives to choose participants who would speak in favor of the HEA. Nevertheless, the ET asked for information on challenges and areas for improvement, which provided more well-rounded data as a whole on the experience. 5 3.5 DATA ANALYSIS Quantitative Analysis of quantitative data from the brief surveys was done in Microsoft Excel. The data were categorical, scale, and/or numeric; therefore, they were easily assembled and analyzed using Excel. Specifically, this included generating descriptive statistics—for example sums, frequencies, and percentages. Given the small sample size, no regression analysis was conducted. Secondary data received from RTI was also examined and analyzed. This entailed cross checking the most recent data received with the quarterly and annual reports, which allowed for most recent gains to be quantified, verified and validated by an external source. All data was provided to the M&E&L Initiative in alignment with USAID Open Data Source policies. Qualitative The ET coded and analyzed the qualitative data gathered through the KIIs and FGDs using systematic qualitative methods. Specifically, this meant agreeing as a team on the codes to be applied, and defining the codes. Then each team member applied the codes. The team used Dedoose (an open source, free qualitative data coding software) to create a database of the qualitative data, codes, and application of the codes. To analyze the data, the ET counted the times the codes were applied and looked at the frequencies as a percentage of the total KIIs. This allowed for excerpts to be downloaded in Excel, and frequencies and percentages of codes to be generated for figures contained in this report. Access to Dedoose is available to the M&E&L Initiative and USAID. 3.6 LIMITATIONS One limitation of the evaluation was related to the inclusion of team member Oscar Picardo, who participated in HEA via a subgrant to UFG for ICTs. The ET mitigated any conflict of interest in appearance by not having Mr. Picardo conduct the site visit to UFG, filtering his reflections (by the other team members), documenting findings so they were evidence-based rather than opinions, and by having Mr. Picardo focus on EQ 3 and the role of the government rather than the universities. A second limitation was informing the universities in advance that the ET would be coming for field visits; this could have caused participants to change their behavior. The ET mitigated this limitation by avoiding sharing the instruments it was going to use to collect data. This ensured that the data were not contaminated. A third, related limitation is that with advance warning the university representatives chose participants in the FGDs with students and teachers. This had the potential to lead to selection bias and the possibility that selected participants would only speak highly of the Activity. This was mitigated in the way the ET framed the FGD questions—asking participants to reflect on what could be improved and ideas for the future. Lastly, it is important to note that sample sizes were small. This is especially the case with the KIIs, which were only 24 in number, and percentages thus are not generalizable. Nevertheless, their importance lies in the qualitative data they provide, adding depth to the findings. For example, the frequency with which some activity interventions were mentioned by beneficiaries as important, compared to others, is relevant. As mentioned above, all information provided in KIIs was coded, and qualitative analysis conducted. 6 4.0 FINDINGS AND CONCLUSIONS 4.1 EQ 1: WHAT HAVE BEEN THE MOST SIGNIFICANT INTENDED AND UNINTENDED RESULTS ACHIEVED TO DATE? Teacher Training Component 1 of the HEA focuses on improving human capital in El Salvador’s higher education sector to ensure that academic staff have the industry knowledge and teaching skills to prepare their students for the future. According to the annual report for Fiscal Year (FY) 2017, HEA trained 412 faculty and staff members in Fiscal Year 2017 (151 females and 261 males). The training includes high-quality training implemented by RTI and/or its subcontractors (World Learning and Rutgers University). First, there is an in-person training on content related to subject areas (e.g., mechanical engineering or agriculture) or on topics such as how to publish a journal article and how to prepare teachers teach English (1-2 weeks). Then there is self-study and study at-a-distance (monthly). This is followed by in-person meetings and discussions about what they studied remotely (over the course of six months). In the first quarter of 2018, the HEA selected the most qualified faculty members (who have effectively implemented the trainings themselves) to train an additional 25 new faculty members; more training events held by project trained faculty are required. In addition, the HEA offered 8 trainings in high￾demand fields (such as mechanical engineering, light manufacturing, ICT, and others), with a total of 81 men and 45 women participating. An exploratory analysis was conducted with Principles of Adult Learning Scale (PALS), in which 41 teachers who were selected by universities to participate in FGDs were asked to complete the survey. The survey assesses the key principles of adult learning and aligns to what was taught to faculty under the HEA. The results are presented in Table 2; it is important to note that, on average, for this group of participants, PALS scores (see instrument in Annex 2) are 78 percent, which implies that teaching in universities for this group of participants remains teacher-centered, despite training provided by HEA in the 21st century teaching methodologies. Table 2: Principles of Adult Learning Scale Analysis Factor Average per Factor Factor 1: Learner-Centered Activities 28.50 Factor 2: Personalizing Instruction 15.54 Factor 3: Relating to Experience 8.59 Factor 4: Assessing Student Needs 5.28 Factor 5: Climate Building 5.23 Factor 6: Participation in the Learning Process 7.44 Factor 7: Flexibility for Personal Development 7.59 Total (average) 78.17 In terms of roll out, teachers who participated in the trainings reflected that the project could have been implemented better by sharing detailed course descriptions via the universities to the teachers and allowing sufficient time in advance for participant teachers to plan and select the trainings in which they participated. During FGDs, participants reflected that they were provided very short notice about the trainings and, furthermore, that they did not know what the training would be about until they arrived. 7 In addition, they had to use their vacation time for the training. Linked to the fact that there was an information deficit, once the faculty learned about the trainings, there was increased demand. Therefore, more trainings are demanded by faculty. In addition, while 250 men participated in training, only 150 women participated. The reason for this is likely the scheduling of the trainings, which conflicted with other obligations (given the short notice). The project should consider the context in which to provide more opportunities for women (alternative hours, online, self-study, and possibly even childcare arrangements). Scholarships for Teachers A total of five faculty members (five men) have completed Master’s programs up to February 28, 2018. Teachers reflected positively on the Master’s scholarships and the ability to take their course of study online while still being able to fulfill professional and personal (family) obligations. Rutgers University was instrumental in supporting the students. The ET found that PhD programs are also in high demand. Scholarships for Students The HEA selected and reached an agreement with FEDISAL, a local NGO with more than 25 years of experience managing projects and, in particular, scholarships in El Salvador. In FY 2017, the HEA awarded 900 one-year scholarships to students from eight HEIs to enroll in the degree programs created by the project. In addition, over 90 scholarships were provided to UDB for the lead lab degree program. Box 1: Quotes on Master’s Degree Programs (KII) Rutgers contributed to selection of programs for Master’s degree programs. Candidates were of high quality and were enthusiastic about learning and advancing their knowledge base and careers. These faculty will successfully transfer their knowledge to the future scientists, engineers, and technocrats of El Salvador. The program has supported over 29 graduates in pursuing Master’s degree programs for both online and on campus/in-person programs. The Rutgers team has been contributing to monitoring and advising all of these candidates. Five faculty have already graduated and 24 are currently pursuing their degree programs. [Note: As of February 2018, there are 33 graduates; this value continues to grow.] Curriculum Reform Component 2 facilitates collaboration between the HEIs and industry by revising and improving degree programs to align with industry demands. In total, 16 degree programs were created or upgraded and approved in record time (i.e., within the 90 days) by the Ministry of Education (MINED) in FY 2017. In the first quarter of 2018, an additional four degree programs were approved. This, combined with the fact that eight programs had already been upgraded, indicates that the HEA created or upgraded 28 programs, surpassing its goal of 20. However, the bureaucratic process to review the curriculum was lengthy and delayed approval of the reformed curriculum. Legally, the approval is supposed to happen within 90 days. However, according to KIIs with RTI project staff, the government would send back a letter within the 90 days (often near the deadline) requesting more information or modifications. This therefore, extended the process. Applied Research Component 2 also includes supporting applied research to solve industry problems. Five seed grant projects were completed in FY 2017. In addition, there have been two rounds of calls for proposals. In the first round, five applied research projects were awarded. In the second round, a total of 10 research grants were awarded. The response rate or number of applicants was substantially higher in the second call for proposals compared to the first, demonstrating increased interest. As a result of the project, 8 more women are serving as leaders or principal investigators of the investigations. A total of 54 investigations had been approved and were ongoing at the time of the evaluation fieldwork (February 28, 2018). Box 2: Background on Applied Research In the area of applied research, one of the key components of the HEA was to enhance the capabilities of faculty and students at the HEIs of El Salvador to conduct applied research relevant to the economy of El Salvador and provide support to the local industries. The ET defined “applied research” as the research that will be used by the relevant industry within a short period of time to improve its commercial competitiveness. Involvement of industry during the identification and execution of research projects has been an essential part of the process. The Rutgers team conducted a series of workshops to train the faculty for writing research proposals. According to the Quarter 1 2018 report, Rutgers trained 100 researchers (68 men, 32 women) in applied research. These participants supported a total of 19 research studies funded by USAID under the HEA. These participants had not participated in a training such as the one on applied research before. During the field research, the ET learned that the HEA hosted a dissemination event on/about March 20, 2018, for industry, academics, and other stakeholders to learn the outcomes, results, and importance of the investigations. This is important to note because, while there is no evidence of these sharing events within the timeframe of the evaluation, it appears that they are being planned for in the future. Applied research projects were supported in three areas: methodology development, publishing, and general management. Rutgers University provided regular, in-country technical assistance every three months to each of the projects. Applied research projects are still ongoing. Therefore, some (not all) are at the point of disseminating findings and results. Unintended Results Two clear unintended results were the development of the Girls Science Camps and the Network of Women Leaders. These two initiatives emerged via the linkages created by the project within the universities and the actors within them. These are discussed further in EQ 4. In addition, as noted above, the clusters were an intended result. At the same time there were spillover results (i.e., unintended results); these included the strong communication developed between the private sector and university (to be discussed in EQ 2, below), and the trust that developed between these two groups. Key actors who participated in KIIs consider the different elements of EQ 1 mentioned above to be of varying importance for development of the higher education sector. As seen in Figure 1, KII participants considered tasks and support related to applied research as being the most critical—80 percent of KII participants identified applied research as a key achievement of the HEA project. 9 Figure 1: KIIs Identified the Key Achievements In addition, one key result was the introduction and development of cluster model (to be discussed in EQ 2 below). Most participants (72 percent) identified the benefits of the cluster model. This does not imply that 28 percent did not value them, rather that this subject was mentioned by nearly three￾quarters of the KII participants. Other mentioned topics included: collaboration (40 percent), trust (32 percent), and communication (48 percent) (see Figures 2 and 3). With regard to the importance of communication, this included communication between universities (24 percent), and communication between universities and industry (48 percent). The cluster model is discussed in-depth in EQ 2. Furthermore, the CDCs were also a noteworthy achievement and are discussed in more detail with regard to sustainability in EQ 5. Figure 2: Result Clusters Figure 3: Increased Communication CONSOLIDATED FINDINGS AND CONCLUSIONS: EQ 1 The table below consolidates the findings and conclusions of EQ 1. The findings on the left lead to the conclusions in the column on the right. They are divided by the four results discussed: teachers, scholarships, curriculum, and applied research. The next question details clusters in depth and presents a similar table with findings and conclusions. Evaluation Question 1 Findings Conclusions Teacher Formation 10 Evaluation Question 1 Findings Conclusions Teachers reflect positively10 on the diplomas available [21st century, Training of Trainers (ToT) in English, publishing in English, etc.]. 1. Although the teacher training activities were considered high quality and well facilitated, what participants learned can be Few teachers have shared what they learned with others. shared more throughout the university. 11 In some cases, training opportunities for teachers were not adequately detailed or given sufficient time. 2. There were information deficits on the training programs and more people want more opportunities. Despite gains, equal gender participation has not yet been fully achieved. Teachers wanted to take more than one course but, in some universities, were restricted. Two hundred and fifty (250) men and 150 women took part in teacher training opportunities, indicating that more men than women participated (although there are more males in the profession). Scholarships – Students and Teachers Students report they would not be able to afford all aspects of their studies without the scholarships, indicating a high demand and need. 3. Students and teachers greatly benefit from the scholarships and there is demand for more—both in extended length of time and the number of student beneficiaries. Teachers are enrolled in Master’s degrees online/in person, and there have been five graduates to date. Universities seek opportunities to establish PhDs in order to strengthen academia. Curriculum Reform Programs were reformed to include competencies and experiential learning (28 approved by February 2018). 4. Despite delays, the new fields of study are currently being implemented in the universities, although more support is required, especially on competencies. There is a lack of implementation of teaching-learning based on competencies-based curriculum. There were delays in the approval on the part of the GOES of the new fields; however, this has improved within the 90 days from submitting the revised curriculum (see EQ 3). Applied Research Applied research was more systematic thanks to the Technical Assistance (TA) of the project. 5. Applied research is reportedly of a higher quality and based on industry needs. Now is the time to capitalize and share results with consumers. Applied research was based on the demands and needs of industry, whereas before there was not the same direct link with industry. There remains great demand on the part of universities for their research to be supported. 12 Although all applied research projects are supported by HEA in methodology development, publishing, and general management, there is a gap in feedback loops and coaching is needed to improve research quality. 10 The teachers reflect positively on the usefulness and applicability of the trainings; they also note that they were high quality. 11 Barriers to sharing knowledge include that teachers already have heavy workloads and taking the time to plan and share what they learned requires additional time and work. The universities could incentivize sharing by modifying workloads so that those who have been trained have more responsibility to share what they have learned with other teachers. 12 The evidence is that the project received far more proposals than it could fund, and these studies continue to need resources in order to be conducted. 11 Evaluation Question 1 Findings Conclusions HEA has not sufficiently disseminated research findings to date, especially within the industry. 4.1.1 What have been the main internal and external factors that have influenced the achievement or non-achievement of the Activity’s expected results as planned? Internal and External Influencing Factors There are key internal and external factors which have facilitated the achievement of results. The external political economy is crucial to note as it relates to the achievement of results. While the scope of this evaluation was not to describe the political economy of El Salvador, a summary of the context can help frame why there were delays in the achievement of results. In addition, understanding the political economy helps frame why achieving trust and communication between universities and the private sector are remarkable achievements in and of themselves. For the purposes of this evaluation, what is especially important to note is the role of universities as key individuals within universities were revolutionary thinkers and leaders. In some cases, the state (i.e., GOES) took over the universities. In other cases, leaders fled one university to open another (including private universities). As such, the remnants of these individual moves affected the institutions themselves and remain part of the institutional “architecture” patterns of behavior and ways of thinking of the universities today. These universities are a part of the HEA, e.g., UCA, UES, and the private universities. In addition, within this external context not only were universities either in or out with the state (i.e., GOES) but they were also at odds with the private sector. Given this dynamic, it took consistent work on the part of the HEA to break down these barriers (which were constructed far before the project) and create an atmosphere of trust and communication on which to build other project achievements (i.e., the cluster model, the curriculum reform, etc.). Internal factors have included the staff and staff capacity within the universities and the implementing partner (IP). In the case of the IP, it appears that now there is strong leadership and, thanks to having individuals familiar with the universities involved in the project, the IP has been able to facilitate the communication in a shared language (namely the Chief of Party). While the project speaks the universities’ language and the private sector’s language, they, i.e., the project staff, have not communicated or involved the government adequately (to be discussed more in EQ 3). 12 4.2 EQ 2: TO WHAT EXTENT HAS THE DEMAND-DRIVEN CLUSTER MODEL BEEN AN EXTENSIVE SPACE TO BUILD DIALOGUE WITHIN PRIVATE SECTOR, ACADEMIA, AND GOES IN ORDER TO RESPOND TO PRIVATE SECTOR NEEDS? ARE THE FOUR CLUSTERS ALREADY SUSTAINABLE? 4.2.1 To what extent has the cluster model been attractive to engage private sector and obtain their commitment through leverage? To frame and develop the analyses, the ET, based on literature about success stories of cluster formation in the U.S. and the United Kingdom, identified seven specific determinants (factors) that, together, enable the cluster model to be effective and sustainable. For the purpose of this evaluation, the ET measures effectiveness as success towards reaching the results and the IRs. Sustainability entails being able to be maintained by the universities and/or other actors in El Salvador (NGOs, the government, private sector). The determinants include: 1. Close relationship with actors in cluster; 2. Intellectual property patent; 3. Research and development (R&D) activities; 4. Connection to market and commercialization; 5. Financial resources; 6. Culture of trust; and 7. Public/Economic policy (the political economy). These determinants can be categorized as micro and macro factors. More specifically, for the purposes of this study, micro factors are considered the elements in the universities’/private sector’s immediate area of operations that affect decision-making and performance, while macro factors are those elements which are outside of the universities’/private sector’s area of control. The conceptual model in Figure 4 below illustrates how these factors interact and influence actors and their collaborative relationships that contribute to or hinder the cluster’s development. Figure 4: Conceptual Model to Analyze Clusters’ Development and Effectiveness Close collaboration/relationship with actors in cluster. A collaborative relationship among system actors is one of the main critical success factors identified within the global literature search for 13 cluster development (Department of Trade and Industry, 2004). The ET found positive and significant effects of the role of collaboration for the clusters’ development and effectiveness (across all clusters, although most noteworthy was ICT). The KII findings suggest that, among all determinants investigated, the collaborative relationship between industry and university emerged as one of the main elements propelling the progress of the clusters. Beneficial documented outcomes from this collaboration include: a) collaborative R&D and joint projects; b) program/curriculum development; and c) industry-university collaborative and consultative process for curriculum development. The ET found that the motivations to collaborate within the clusters were to strengthen both university and industry. For the university, typical motivations to collaborate with industry included the: a) improvement of teaching, curriculum development, and research; b) reputation enhancement; and c) access to empirical data from industry. For industry, the motivations to collaborate with universities included: a) developing research to address industry needs; b) developing skilled workers in identified areas; and c) influencing the overall teaching and research agenda of universities. Without a baseline study or control design it is difficult to state with statistical significance the levels of advancement of the universities along the determinants. Nevertheless, the ET describes each determinant below and how it has changed/been strengthened via the HEA. Most importantly, the HEA had a positive effect on the process of understanding and building trust (i.e., determinant 6 in the list above) on both sides of the partnership. Prior to the HEA, there was distrust between industry and universities and between universities themselves. This also allowed for strong collaboration among actors. These two “key determinants” are discussed first, followed by the remaining determinants in order of importance. R&D Activities. While collaborative R&D and joint projects are one of the main accomplishments from the cluster model, knowledge spillover, which characterizes a sustaining cluster, is not realized. The interview data (especially with industry heads) show a lack of knowledge cross-fertilization within and across clusters. Furthermore, interview findings revealed that the outcomes associated with the research activities were delayed. Nevertheless, actors from industry expressed positive expectations regarding the applied research results. Intellectual Property and Patents. Intellectual property and patents are critical for managing innovation and protecting the knowledge of industry and the university in clusters. The KII findings indicate that industry and universities have low awareness of the importance of intellectual property and patents on their research products, as well as a lack of support from intermediaries to assist in the patenting process. Therefore, cluster actors require further awareness and knowledge of protecting their research findings and inventions, and more concerted practice mechanisms for enhancing effective management and use of the intellectual property in the clusters. The impact condition of the intellectual property in the clusters is considered low because industry and the university still lack knowledge of the benefits and processes for protecting their ideas. Public Policy. Public policy may influence the propensity of industry to collaborate with university and the scope of such collaborations in many different ways—through a direct role in providing funds to universities and R&D projects, as well as through a regulatory role, which influences the rule-sets of universities and shapes the intellectual property rights regime. The interviews with key informants found that GOES’s limited budget [0.1 percent of the national budget, Consejo Nacional de Ciencia y Tecnología (CONACYT), 2009] for higher education as well as competing priorities hinder its ability to influence and/or strengthen collaboration between industry and university. In addition, interview respondents stressed the lack of government involvement in the clusters’ development. Financial Resources. Other important elements for cluster development and sustainability are the financial support for business productivity and products that require sophisticated financial markets to make available capital for the private sector, institutions, and stakeholders in order for them to survive 14 and stay competitive in a cluster. From the KIIs, the ET found limited availability of financial support to clusters besides USAID support. It is important to note how USAID supported the clusters. They have kept the clusters moving forward by providing guidance and TA throughout the life of the project. Clusters leverage USAID investment to hire high-quality expert staff, host events, produce materials, etc. There have been little to no other resources put towards this (from other donors or industry). Thus, outside of USAID, the financial investment in the clusters is low. Culture of Trust. The development of trust among respondents is crucial for collaboration in clusters, and especially for acquiring tacit knowledge. From the KII findings, there is greater trust between industry and universities. The KII findings indicate that industry and university achieved mutual trust in the early stage of collaboration. The Role of Actors Within the Cluster. The success of clusters is associated not only with strong linkages among its actors but also with the complex roles played by the actors within the cluster. To analyze this dynamic relationship, the ET used the concept of the Triple Helix (Etzkowitz, 1997) to analyze the interactions between universities, industry, and the government.13 • The role of university. The cluster model demands more entrepreneurial and knowledge￾seeking activities from participating universities. This means universities are required to be more involved in national economic development by capitalizing their expertise and knowledge resources into something meaningful and lucrative. The interview findings indicated that the universities’ ability to shift their roles from traditional teaching and learning to knowledge providers is still in preliminary stage. It is important to note that some clusters (and in turn anchor universities) are more advanced, according to KIIs— namely Light Manufacturing (UCA), TIC, and Energy (UFG and UDB). Traditionally, universities in El Salvador have had limited responsibilities to produce and transfer local knowledge. Consequently, their research capacity is low. The HEA has provided capacity building to faculty and staff to increase their research skills, but it is still early to determine the impact of these interventions. The ET identified the lack of faculty members with experience in industry as another constraint. Although faculty received training in specific industry areas such as light manufacturing, experience in industry is relevant to increase teaching impact and use of experiential approach. Therefore, overcoming the cultural divide between academia and industry requires not only strong university leadership but also faculty members who understand industry and bridge that gap. • The role of industry. The role of industry is vital in generating economic growth in its cluster. Industry contributes to the knowledge transfer process, provides basic training to students, and shares knowledge resources to meet university and its own needs. Interviewees indicated that the industry actors were very comfortable having close links with participating universities. The reason for this is that universities were very active in collaborating with industry (e.g., on curriculum and program development and joint research projects) and responding to industry’s needs. Furthermore, from interviews conducted, the ET found that the involvement of industry in collaborative activities with universities increased the number of internships for university 13 The Triple Helix thesis is that the potential for innovation and economic development in a Knowledge Society lies in a more prominent role for the university and in the hybridization of elements from university, industry, and government to generate new institutional and social formats for the production, transfer, and application of knowledge. 15 students. Universities benefitted by enabling their students and graduates to get an early taste of employment and learn new skills, such as soft-skills and/or technical skills. The university “benefits” because their students get to learn about the world of work and have a space (both physical and mental) to link work and school. Also, universities benefit because they can market the higher numbers of graduates who get work. • The role of government. Government usually plays a major role in the innovation process of a country through the national system whereby it manages and stimulates innovation and learning at the regional and national levels. However, the ET found that the GOES’ role was the least dominant in providing a conducive innovative and economic environment, as well as connective tactics between university and industry (see more on the role of government in EQ 3). Table 3 summarizes the impact condition of the clusters’ determinants. Table 3: Impact Condition of the Clusters’ Determinants Determinants Clusters Impact Condition Collaboration with actors: Close collaboration with industry Close collaboration with university Close collaboration with government Very High Very High Low R&D and knowledge spillover Moderate & Low Connection to market and commercialization Low Financial support Low Intellectual property Low Government policy and regulations Low Culture and trust Very High Economic condition and environment Low A typical stimulating university-industry collaboration policy is to design R&D research grants, matching grants, and tax-incentives, with a requisite of a consortium of firms and universities for project eligibility. The interview data suggested that industry in El Salvador could be receptive to this policy because it is already collaborating with universities and is willing to match funds with internal resources. Likewise, universities are already aligning their research agendas with industry demands. Therefore, policy initiatives such as R&D incentives and grants could further strengthen collaboration between university and industry. Furthermore, the GOES/National Registry Center (the entity responsible for managing intellectual property) could stimulate collaboration between industry and universities through soft measures, such as providing support services to industry and universities in the dissemination of the advantages of intellectual property. However, as mentioned earlier, the ET found that industry and universities have low awareness of the importance of intellectual property and patents for their research products. The IP could proactively seek this support. The Role of Intermediary/System Integrator. The role of intermediaries is recognized as important and is the government’s main instrument for supporting cluster development. As the government has a limited role in cluster development, the role of intermediaries was found to be a weak broker between the universities and local industry. The most likely government counterpart to play this role is the National Direction of Higher Education (DNES by its acronym in Spanish). To address this gap, cluster directors and industry leaders indicated that the cluster needs a new actor to play the “system integrator” role. The cluster anchors could be considered for this role. The important point is that the leadership is autonomous and transparent in their interests. Ideally, a steering committee could (and should) be created with membership from all of the lead institutions already working together on the project. The entity would serve to work with universities and industry to 16 develop skill solutions, gather data, and identify and disseminate positive examples. Other responsibilities might include: 1) coordinating and integrating all cluster activity, from R&D to the implementation of solutions; 2) coordinating clusters and finding common areas to be supported and strengthened; 3) catalyzing stakeholder actions in priority areas; 4) helping with the process of market and commercialization; 5) assisting with IP and patent processing; and 6) monitoring and managing the quality of activity outcomes. CONSOLIDATED FINDINGS AND CONCLUSIONS: EQ 2 According to Porter (1998), the determinants of national competitive advantage are dynamic when all determinants interact with each other and the effect of one determinant depends on the other determinants. The ET found that the role of collaboration capability can enhance the process of innovation, such as the university-industry partnership. Interview findings suggest that the effect of collaboration capability (motives and barriers of collaboration) is dependent on the conditions of determinants in the clusters. As indicated in the beginning of this section, the motives of actors in the cluster for collaboration are related to economic and academic reasons. The development of trust among respondents is crucial for collaboration in clusters, and especially for acquiring tacit knowledge. From the KII findings, there is greater trust between industry and universities. The KII findings indicate that industry and university achieved mutual trust in the early stage of collaboration. The success of clusters is associated not only with strong linkages among its actors but also with the complex roles played by the actors within the cluster. To analyze this dynamic relationship, the ET used the concept of the Triple Helix (Etzkowitz, 1997) to analyze the interactions between universities, industry, and the government. Unfortunately, the potential for and effectiveness of collaboration activities in the cluster were found to be challenged by the barriers that exist as a result of weak conditions of the cluster determinants. As of the time of data collection (February 2018), the likelihood of sustainability of the cluster model is relatively low, with some clusters more likely to sustain than others (ICT, light manufacturing, energy). For example, some have fluid ongoing communication between the university (anchor) and the private sector, others do not. In other cases, the private sector trusts the quality of the cluster leadership (and in turn innovation). In other cases, this level of trust does not seem to exist in the cluster (nor the anchor). Interview data identified barriers, such as the lack of public policy, financial support, and R&D knowledge spillover, and the weak role of GOES and intermediaries, as threats to the clusters’ effectiveness. Evaluation Question 2 Findings Conclusions Collaboration There are positive and significant effects from collaboration for clusters. 6. There is strong collaboration and trust between universities and industry. Industry and university find it mutually beneficial to collaborate. The consultative process for curriculum and program development strengthened trust and collaboration. Benefits and Barriers Clusters created collaborative R&D, joint projects, and a process for curriculum development. 7. There is solid collaboration in research, but the lack of timely implementation, support of publication writing, and knowledge sharing could Clusters are behind schedule to produce results such as research products. 17 Evaluation Question 2 Findings Conclusions There is limited cross-fertilization of knowledge and best practices within and across clusters. compromise the collaboration momentum. Sustainability Most interviewees were highly enthusiastic about continuing with the cluster model. 8. If well-implemented, the clusters could be a critical component of national innovation but creating a public policy and funding mechanism for research and innovation is strategically necessary to ensure sustainability. The lack of public policy to support cluster growth presents a barrier (funds to R&D projects, intellectual property regulation, and business incubators). There are limited policy incentives to consider the promotion of university-industry linkages as part of a broader science, technology, and innovation policy program. 4.3 EQ 3: WHAT CHANGES HAVE BEEN MADE IN THE GOES AND HEIS IN TERMS OF THEIR INSTITUTIONAL CAPACITY TO CONTINUE INTRODUCING AND/OR SUPPORTING EDUCATIONAL REFORMS AND ACADEMIC PROGRAMS IN THE FUTURE? In order to answer EQ 3, it is important to provide contextual background on university reform in El Salvador. University reforms or transformations are very complex processes because of historical and ingrained patterns of behavior. The configuration of the tertiary system in El Salvador represents a diversity of institutions with specific circumstances and problems (public, confessional, private institutions, etc.). The 1995-1997 reform initiated a profound change (establishment of performance requirements, establishment of the Council of Higher Education and Subsystems of Information, requirement of evaluation and accreditation),14 and currently the HEA is being presented as a prelude to a “second generation of reforms.” Component 3 (and in turn EQ 3) of the HEA had the goal of strengthening the institutional capacity and effectiveness of HEIs for sustainability through linkages with U.S. and regional HEIs, public entities, and private sector companies to improve the quality, relevance, and access to tertiary education in key areas.15 Recent evaluations of the higher education system in El Salvador have consistently pointed to institutional weakness and widespread distrust among stakeholders. Likewise, El Salvador’s public investment in higher education is the lowest in Central America (Picardo, 2011). In this context, the third component of the HEA seeks to create a favorable environment where collaboration and mutual support dialogue will take place. Dialogues include efforts to obtain commitment from the interested parties, including HEI entities, the private sector, employers, students, and workers, for the necessary reforms. A specific objective of this activity was to provide: 1) technical assistance and institutional capacity building to local HEIs and to the main official entities of higher education in areas such as strategic planning, administrative and financial management, and collaboration with other interested parties to improve their effectiveness at the system level; and 2) develop links with the private productive sector and international HEIs and between local HEIs, without excluding public/private partnerships. 14 Cfr. Picardo, Oscar; La Reforma de Educación superior; MINED, San Salvador (1998); Colección en el camino de la trasformación educativa. 15 Higher Education Contract (06/06/2014), page 8. 18 It is important to note that Component 3 is intended to focus on a series of sub-components. These include strengthening the higher education system, increasing collaboration between local and international stakeholders, and improving capacity within the system. While the HEA is not responsible for the whole higher education system reforms in El Salvador, it is responsible for playing an instrumental role in this process through the provision of various forms of technical assistance. In addition, the advocacy for the importance of higher education and investment in it is an important aspect of HEA’s role. The HEA Quarterly Reports monitor the ongoing process and gains under HEA towards building HEI capacity. The Quarterly Report from October 1 to December 31, 2014 included a section on the “initial spirit of cooperation” and relevant dialogues with stakeholders. In the Quarterly Report from January 1 to March 31, 2015, the first rounds of dialogue with the leaders of HEIs and representatives of MINED stood out. The main theme of the dialogue—facilitated by Former Minister of Education, Darlyn Meza— was the need to modernize higher education in El Salvador. In the context of the “Challenges and Restrictions of Higher Education in El Salvador” conference, convened on March 19, 2015, the leaders of six HEIs held a new round of dialogue and agreed to sponsor the Third National Congress of Higher Education, entitled “Support for Research and Development” (April, 2015). The 2016 Annual Report documents the agreement between the MINED, USAID, and 13 institutions to draft the higher education policy proposal. The HEA outlines the process it used to work with the HEIs and the GOES to conduct the Human and Institutional Capacity Development (HICD) diagnosis tool. Important training programs were also registered to strengthen the teaching and research capacities, such as the professional development (diploma) courses in 21st Century Pedagogy, applied research, Learn English to publish papers, and other short-term programs for academic management skills. Box 3: Reflections from One KII on the State of Institutional Capacity It is difficult to ascertain the overall changes that have been made in the institutional capacity of the institutions; nonetheless, World Learning, for example, has provided technical input on the HICD grant reports since the start of FY 2018. Although these reports provide us with an overview of how implementation is occurring, from our observations, it appears that the HEIs are implementing the activities outlined in their capacity development plans, with each HEI having a set of major and/or minor issues that require attention and follow-up by their leadership. – Implementing Partner The 2017 Annual Report defined the content of the higher education policy in five key areas: 1) governance; 2) higher education functions; 3) quality; 4) access and retention; and 5) academic professional development. The HICD studies for the design of institutional strengthening—including the National Direction for Higher Education-MINED—were completed. FEDISAL was defined as the administrator of scholarships for students of new careers. Interviews with university officials, businessmen, cluster managers, and government authorities revealed the following: • The provosts (rectores) of the four anchor universities—in addition to the support of the National Higher Education Directorate—have created a policy proposal with support from the HEA. This public management tool is relevant (based on the recent experience with HEA and under the conditions of the current context in higher education in El Salvador) and represents a great effort, but it is unknown16 to the Vice Ministry of Science and Technology, to the New National Commission on Science and Technology (N-CONACYT), and to the rest of the universities in the system. 16 It is unknown to the GOES because the development has been behind closed doors with a small insular group of universities. The GOES has not been included in the process to date. Because of the recent elections (February/ March 2018), the HEA upon consultation with USAID decided not to share because of political sensitivity. 19 • The HEA—through its initiatives and tools—has created a favorable climate for the discussion of reforms. For example, there is a new, open-minded vision for higher education reform (i.e., the Vice Minister) and in turn for the redesign of curricula for academic programs. However, MINED unnecessarily asks HEIs for unnecessary pre-requisites, offers little flexibility for advancement and curricular mobility, has rigid course sequencing requirements, and has content demands, due to other laws of the Republic, which influence the implementation of new curricula. Basically, lack of curricular mobility is the inability to move from one year to the other in one’s studies. The problem is that there is such a dependence from one year to the next that if one does not pass every subject matter, she cannot advance. The challenge of curricular mobility demands reforms to the Law of Higher Education. • The formalized dialogue between the sectors (university, business, government) is one of the main achievements of the HEA. There is enough evidence of this process. At the governmental level—MINED, Ministry of Economy (MINEC)—there is less consensus between ministries due to their own interests as well as turnover among leadership, but significant progress has been made between the productive and academic sectors (with regard to trust building and communication). The most noteworthy indicator of progress is the development of the functioning clusters. • The most relevant changes found—which have not depended on legal or political reforms—are the: 1) design of career programs in coordination between academia and the private sector; 2) beginning of an important cultural change17 in the design of applied research seeking to solve industrial problems (54 initiatives have been submitted and 15 grants have been awarded, totaling an amount of $2,004,150.26); and 3) institutionalization of CDCs in the four universities heading each cluster—UCA, UDB, UFG, and UNICAES—plus one associate partner, UNIVO. The term institutionalization implies budgeting and allocating resources, including staffing and materials/supplies, and also a physical space; part of the annual plans. • The use of the evaluation tool for HICD—Thirteen (13) HEIs were administered the HICD. The majority of universities report valuing the HICD tool to diagnose their unique institutional capacity development needs, and to develop a plan to address those needs. Universities, for example UES, reflected positively on the HICD process. At the moment of this evaluation, the HEIs were conducting mid-term assessments along the dimensions of the HICD in order to assess gains made. In addition, HEIs will conduct endline assessments in the last year of the HEA. At the moment of this evaluation, two HICDs had been conducted with government (Vice Minister of Science and Technology and the DNES). • A structural weakness of the HEA is the scattered data. The HEA performance data are scattered and inconsistently found in various studies, documents, or reports. In some cases, it is expensive to integrate the figures due to the concepts that integrate or disintegrate categories of analysis. The HEA has achieved an important step for linking the sectors: to unite a block of relevant private universities with their respective private partner sectors [UDB-Salvadoran Association for Renewable Energy (ASER), UCA-Salvadoran Industrial Association (ASI), UFG-Salvadoran Chamber of Information and Communications Technologies (CASATIC), and UNICAES-Agricultural and Agro-industrial Chamber of El Salvador (CAMAGRO), more if smaller associations are included]. However, important actors have not been included in the process of creating the policy, namely: UES, the Accreditation 17 The cultural change described here is one in which the investigations respond to industry demands. In so doing making the research (and findings) marketable and useful for industry; or simply esoteric investigations which are not grounded in reality or real needs. 20 Commission, the N-CONACYT, the Higher Education Council,18 and the role that new Congress/Legislative Assembly Representatives (Senators/Deputies) may assume in their positions regarding reforms (political elections took place in March 2018). In any case, the pro-reform block is an achievement of the project and it is very significant. CONSOLIDATED FINDINGS AND CONCLUSIONS: EQ 3 The HEA has configured an important initial phase of reforms and will require support and leverage from the GOES to maintain the intensity of the dialogue and an active agenda in the field of applied research and design of new careers. If there is something important to highlight in the findings on Component 3, it is the establishment of trust and dialogue between the universities of the cluster and their respective private or productive sectors. In large measure there was minimal trust and communication, in other words there was distrust between the actors (as noted in the clusters section of this report). Continued dialogue with the GOES is important in the next phase. One key informant noted: “Initially there were interviews with the government and then when the award for the project was made there was little closeness with the ministry. If the project is to benefit the country then there needs to be better communication with the government, it cannot be managed separately…this is also for the sustainability and to date the GOES has not been taken into account sufficiently.” Evaluation Question 3 Findings Conclusions The dialogue and closeness between sectors are important project achievements because they facilitated the development of investigations, curriculum, and the CDCs. 9. Leadership from a neutral entity could help build more collaboration between universities and the GOES. There has been an open and participatory process to construct the higher education policy and this process should continue. The Vice Minister of Science and Technology does not feel ownership of the project, which is crucial for sustainability. However, he/she sees the project positively. There was little awareness of the bureaucratic aspects within the GOES in the design of new programs of study especially on the part of the DNES. 10. The project can be more grounded in the reality of the country/context and ways of operation within the GOES. The written reforms are outlined in the higher education policy proposal, but they are not in operation/implementation (which requires commitment from government). 11. The GOES could assist HEIs if they accept the policy and may do so if they are included in the process (i.e., have ownership). HICD assessment was instrumental to identifying the gaps in administration on the part of universities and to providing a direction to improve capacity; however, there is no endline. 12. The final impact of the HICD activities can be assessed after the gathering of more data. 4.4 EQ 4: HOW WERE GENDER, FEMALE EMPOWERMENT, AND SOCIAL INCLUSION INTEGRATED IN IMPLEMENTATION? 18 This was not necessarily intentionally planned, but more so by default; especially when the costs (including time) of coordinating and collaborating became too great and cumbersome. 21 4.4.1 To what extent is the Activity influencing changes in gender participation in higher education? Findings from the HEA gender analysis revealed that women are significantly underrepresented in key Science, Technology, Engineering, and Mathematics (STEM)-related degree programs [22 percent female enrollment compared to 54 percent overall female enrollment in HEIs (2017 data)]. The gender analysis also revealed that the culture of machismo (defined as strong masculine pride) continues to negatively influence women’s education and career opportunities and choices in El Salvador, steering them away from STEM-related fields—especially in technology and engineering. More than half of KII participants noted the importance of the HEA with regard to gender mainstreaming. Specifically, some (16 percent) articulated the role of leadership (for example with the HEIs Female Leaders in Higher Education Network). Others were specific in terms of the project’s gains with regard to getting girls and women into the fields of STEM (32 percent). It is important to note that not all participants in KIIs knew about the network of women, for example, and there is room for improvement (see Figure 5 below). Figure 5: KIIs That Reported on Importance of Gender Equity, Women in Leadership Roles, and Girls/Women in STEM Fields At the same time, background data and literature from the gender analysis suggest that there is a potential to capitalize on a changing mindset among the youth—both males and females—who believe that women and girls must be supported in STEM, specifically via scholarships as well as considering the times of days of coursework and, potentially, caregiving options for mothers. The gender analysis argues that, together with women leaders in university and industry, more women can be provided with opportunities in STEM (p. 2, USAID HEA Gender Analysis, 2015). This analysis provides mixed results in term of state of gender equity and social inclusion. According to the FY 2017 Annual Report, “Gender is being approached from the perspective that women must have equal rights and opportunities to access resources and benefits in higher education” (p.15). In large measure, the majority of KIIs identified the gains made with regard to gender equity (as noted in the figure above). The ET does not disagree with this statement, but the HEA should go beyond this and make the argument, with evidence, that women and girls are not only educated because of their rights, but also because of their potential. In the eyes of industry, the convincing argument—again through evidence in the context of El Salvador—must be made that girls and women have the potential to produce knowledge which, in turn, can lead to earning gains and profits for industry. To date, sufficient evidence at large has not been provided, including studies and investigations from women, which is the “evidence” required to continue to make the argument to industry for equal women’s participation. This is particularly the case because the HEA is for “economic growth” and falls below the respective DO within USAID/El Salvador’s CDCS. 22 The quotes below provide additional information on the importance of HEA’s gender efforts. As indicated in EQ 1, the establishment of the Higher Education Network of Women Leaders was an unintended result of the project. These leaders meet once a month and plan for activities related to gender equity in higher education (including supporting investigations and scholarships). The Science Girls Camps were promoted by UDB. They occur on the weekends and bring girls to the university to learn about science and technology. There is excess demand for the science camps. In addition, UDB works with parents and communities to challenge stereotypes that suggest boys should be the only ones learning about science and technology. UDB provides scholarships for the camps, including transportation and being taught by university staff. In the case of the Higher Education Network of Women Leaders, the project initially promoted connecting the women leaders in various spaces across the project; but now these women continue to communicate (for example via WhatsApp) and plan meetings and initiatives autonomously without the project. It is for this reason that it is likely the network will continue. “The Science Girls Camp is one of the greatest achievements. It is based on the needs of the Security Plan of El Salvador. It works with technology, computers, and other STEM-field related activities. It provides opportunity to vulnerable girls.” – University Staff “The women’s network was linked with the project. The women knew each other from the project and the project has been guiding each other, as a result of the link between the universities.” – University Staff “Given these initiatives [Science Girls Camp and the Network] it is important to continue these initiatives and to demonstrate the results. Women have not been a part of university with such a large presence in the past.” – University Staff CONSOLIDATED FINDINGS AND CONCLUSIONS: EQ 4 Evaluation Question 4 Findings Conclusions There has been an increase in access to STEM education for women and girls. Most notable is the Girls Science Camp, with approximately 70 girl beneficiaries. 13. HEA has made substantial gains to improve access to higher education opportunities for women and girls and there remains room for more expansion. A Higher Education Network of Women Leaders is established and functioning, although there is no male involvement. 19 Research teams include some women, and can include more; to date, 78 males and 35 females participated on research teams (although, significantly, these are young women—below 35 years old—with growth potential). The HEA has offered workshops to other IPs on how to integrate gender equality and female empowerment into USAID programming. 14. HEA plays a leadership role in terms of exemplifying how to incorporate gender, and more research and data will continue to demonstrate achievements and make convincing arguments for society at large. There is a lack of public messaging and convincing narratives on why girls should be educated, particularly about the benefits to industry and for their potential. In terms of social inclusion, most universities are not physically accessible to students with disabilities. 20 15. Universities can be more inclusive to all students. 19 This is because males are not invited to participate. 20 Although this was not an intent of the HEA, it is important to document for future USAID programming especially given USAID’s Social Inclusion and Gender Policies. 23 4.5 EQ 5: WHAT APPROACHES AND RESULTS BY THE HEA HAVE THE POTENTIAL TO EXIST AFTER USAID FUNDING ENDS? WHICH ONES SHOULD CONTINUE TO BE SUPPORTED? According to HEA’s internal evaluation (2017, RTI), the services offered by the CDCs are highly valued by the students because they are considered to be of a higher quality and better linked to industry than those previously provided. This finding also emerged during the ET’s data collection. The CDCs are operational and deliver critical students services. More than 2,000 students (50 percent women) participated in career-related workshops in FY 2017. These workshops included: how to create a curriculum vitae (CV); how to interview; self￾reflection; working with soft skills; and, importantly, entrepreneurship. These training opportunities cut across fields of study, an approach which is highly valued by students because the traditional university system does not provide this interdisciplinary experience for them. Image on the left shows a brochure from the CDC at UCA, which is used to advertise workshops. In the FGDs, student responses largely reflected their positive experiences with the CDCs and the workshops offered. Students were asked to rate on a scale of 1-5 (with 5 being very useful and 1 being not useful at all) the following workshops: CV development; interview practice; soft skills; self-reflection; and entrepreneurship. Across the board, they rated these courses positively. Students sharing these positive experiences via word of mouth will continue to expand interest in the CDCs. Figure 6 below shows the percentages of students who rated each course as “very useful.” The highest percentage of students found the interview practices workshop useful (82 percent). In some cases, industry even participated with students in the role play interviews for jobs. Figure 6: Student Ratings of Career Development Center Services During FGDs, students were consulted about additional services and benefits of the HEA and the CDCs. In general terms, the students who received the scholarships felt that they were highly valuable (100 percent). They articulated that the scholarships not only helped with fees but also with additional costs, especially in STEM areas of study. During the course of the guided FGDs, students revealed the value of the scholarships and the fact that given the additional costs associated with studying in university it would have been unlikely that they could have afforded to attend without the scholarships. The students 24 also specifically identified the role of the CDCs in providing the opportunity for internships (85 percent). In both cases, the question was only asked of students who participated in internships/scholarships. Figure 7: Additional Career Development Center Services In addition to the CDCs, there are a series of initiatives on the part of HEA that will likely continue when the activity ends. See Table 4 below. Table 4: Aspects That Will Continue After the Activity If HEA Provides Support Task Sustainable at Present Sustainability CDCs Yes The CDCs are critical to student success and have high student approval ratings. CDCs are integrated into the universities, which implies they are sustainable with little funding. Dialogue between cluster anchors Yes An evaluation of key conditions that impact sustainability shows that currently the activity has supported close collaboration with industry and academia, and strong trust, documented by collaborative R&D, joint curricula development, and new fields of study/program development as needed. This is the base for sustainability. Recommendations are made below, which would support sustainable collaboration of the universities and private sector. Trust between universities and industry Yes The use and management of grants for applied research Yes Researchers have rated this as the most important aspect of the project. One hundred (100) researchers have received first-time training on how to carry out these studies and on how to write research grants to fund them. Fifty-four (54) studies have been completed, including 19 by newly trained researchers, which set a precedent and ensured necessary practice and skills for ongoing completion of these studies. Processes for applied research (more training and supervision) Yes Girls Science Camp Yes A camp for young girls is sustainable because it is building their capacity and will lead to further opportunities for them in the future. Female Leaders in Higher Education Network Yes Female leaders have taken significant initiative to launch this platform. Now that this network of women has been built, there is opportunity for ongoing activities. Policy documents for higher education Yes There is a proposal completed by four universities to support a reform, in writing. But not all key stakeholders are aware of the proposal and support is not yet clear. However, the project has created a favorable environment for reform and if these skills are continued to be strengthened, reform may be possible. Reform in the Higher Education Law (Legislative Assembly) Partially (additional support needed) 25 Task Sustainable at Present Sustainability Dialogue between universities, industry, and government Partially (additional support needed) Dialogue has improved significantly due to the project. But there needs to be leadership from a neutral entity. Furthermore, the GOES does not feel ownership of these changes. Ownership of policies in other sectors (private universities, UES, Vice Ministry of Science, Higher Education Council) Partially (additional support needed) Evidence for collaboration and coordination is required and ongoing dialogue/convincing these actors to be involved is needed. Accreditation of majors by international agencies Partially (additional support needed) This is difficult because the process for accreditation has not been assumed by the universities (to date, with the exception of one, UDB); it is costly and a long process. Complementary scholarship funds (from universities and companies) Partially (additional support needed) Pending buy-in, based on the provision of evidence, from the private sector. Promote the hiring of more women in STEM, provide incentives for students in their last year Partially (additional support needed) In the eyes of industry, a convincing argument for training/hiring more females is that women have the potential to produce knowledge that can lead to profits. However, it is not clear how evidence in favor of this argument can be best produced. Furthermore, there are still barriers to full integration of women due to factors like machismo. Leadership in Energy and Environmental Design Lab Sustainability is achieved via the strengthening of the institutions and also via the capacity of individuals within them. For example, from the teachers trained and the students who have received scholarships, it is expected that they will continue to implement what they have learned beyond the life of the project. These are the local-level seeds of sustainability. In addition, it is important to note that there are ways that sustainability and continuity are reinforced by the project via the universities and the multiple activities they are engaged in (i.e., linking curriculum reform to investigations and to teacher training). By covering various areas within the higher education space, these initiatives reinforce each other. Therefore, another example of a university-driven initiative that is likely to continue is the Leadership in Energy and Environmental Design (LEED) Lab. The LEED Lab is an example of the way that the various aspects of the project crossed over from one HEA area to another: at first the lab was supported by applied research; now, it is part of the reformed curriculum. With these two interventions and forms of support (i.e., revised curriculum and the investigations, which included training teachers in applied investigations and equipment), it is certainly likely that the work of LEED will continue. The image above shows the 26 equipment in the LEED Lab. The important aspect is the training of professionals able to certify buildings as LEED certified. Table 4 (previous page) provides a view of the aspects discussed throughout the report, and specifically articulates those that will continue as of now and those that will likely continue if HEA gives support through the remainder of the year. CONSOLIDATED FINDINGS AND CONCLUSIONS: EQ 5 Evaluation Question 5 Findings Conclusions Students who used the CDCs reflect positively on the workshops (especially entrepreneurship as well as those on CV, interview, and soft skills). 16. Demand for the CDCs is high on the part of all students and industry. CDC staff confirm that student interest is high for the workshops and internships. CDC staff report increased participation [i.e., Memoranda of Understanding (MOUs), participating in role plays, job posts] from industry in the CDCs with no gender differences (i.e., equal participation). CDCs are part of the budgeted plans, aesthetically pleasing, and aligned to curriculum (social service). 17. CDCs are not isolated but are integrated into the universities, which implies they are sustainable with little funding. Some CDCs are advanced while others are still being developed. 18. There is an opportunity to capitalize on the tour and share more between CDCs in El Salvador so that they learn from each other and collaborate effectively. Participants in the CDC study tour to the U.S. report the experience was valuable. CDC staff are highly motivated and energetic vis-à-vis the goals of the CDCs. There is a lack of consolidated results21 of the CDCs that are ready to be shared and detail the benefits of the CDCs. 5.0 RECOMMENDATIONS The following recommendations are intended for both the current project and future programming. Below, each recommendation details are provided for the project and for USAID (separately). The recommendations for the project should be implemented by HEA before the end of the award, while others should be implemented by USAID in the long-term via future investment. 5.1 RECOMMENDATION 1 Recommendation 1: USAID should support the construction of an entity or contracting of a local entity to strategically integrate and give continuity to the cluster model. Supported by the 1. There is a strong collaboration and trust between universities and industry. 21 The lack of results measured is: 1) because of a time lag; and 2) because the HEA has not systematically asked CDCs to consolidate and report data. Furthermore, there are no M&E staff within the CDCs. 27 Recommendation 1: USAID should support the construction of an entity or contracting of a local entity to strategically integrate and give continuity to the cluster model. following conclusions 2. If well implemented, the clusters could be a critical component of national innovation. However, creating a public policy and funding mechanism for research and innovation is strategically necessary to ensure sustainability. 3. There needs to be guidance from a neutral entity. 4. There is a need to keep the project grounded in the reality of the country/context. There is an opportunity to capitalize on the collaboration and coordination between the clusters and within the clusters. Now is the time to pave the way for future sustainability. A legal entity must either be created, or a local NGO, think tank, or consortium of anchor universities, must be contracted, to continue to consolidate this process. The neutral entity could be an institution with a similar interest or alliance. That entity or NGO should be in charge of continuing communications between industry and universities, as well as between industries and between universities. This cross-fertilization will require financial management on the part of the entity and the management of data and information. The HEA can pave the way now for the remainder of the project. Then, new program funding can support the entity. 5.2 RECOMMENDATION 2 Recommendation 2: USAID should continue to support the investigations, curriculum reform implementation, teacher training, and scholarships. Supported by the following conclusions 1. Although the teacher formation activities were considered high quality and well facilitated, what the participants learned can be shared more throughout the university. 2. More information can be provided on the teacher training opportunities and more people are interested in participating in them. Equal gender participation should continue to be promoted. 3. Students and teachers greatly benefit from the scholarships and there is demand for more—both in extended length of time, and the number of student beneficiaries and PhDs. 4. Despite delays, the new fields of study are currently being implemented in the universities; however, more support is required. 5. The applied investigations are reported as higher quality and based on industry needs. Now is the time to capitalize and share results to consumers. More time and an incentivizing environment (i.e., acknowledging the workload associated with replicating training via financial or other) is required on the part of the universities to expand these trainings and share throughout the system. Advances have been made with the revised curricular programs, approved by the GOES. However, more time and support are required to implement the competency-based method as outlined in the curriculum (at a minimum of 16 hours up to 40 hours). Different modalities should be encouraged (including flexible timing, self-study, online, and childcare provision) to promote participation at large and equal participation of women. The evidence showed gains in the training of professors. These recommendations can be implemented in the short term by the HEA, especially with universities that do not have a foundation with competencies. Similarly, the investigations require continued coaching and mentoring, and the results must be shared (timing is opportune). It goes without saying that the scholarships need to be increased, especially for students in the short term (and universities and government can complement), but the process can be led by HEA (possibly via a no-cost extension). In the case of faculty scholarships, PhDs and stronger English programs should be considered in order to strengthen HEIs in future USAID programming. 28 5.3 RECOMMENDATION 3 Recommendation 3: USAID needs to continue to push the envelope with regard to gender equality and social inclusion. Supported by the following conclusions 13. HEA has made substantial gains in improving access to higher education opportunities for women and girls. However, the program can be more expansive and include both genders to promote cultural changes. 14. HEA plays a leadership role in terms of exemplifying how to incorporate gender, and more research and data will continue to demonstrate achievements and make convincing arguments for society at large. 15. Universities can be more inclusive to all students. Evidence suggests that great gains have been made in the HEA with regard to gender inclusion, especially in the area of STEM. Nevertheless, work should not stop. The Higher Education Female Leaders Network can be empowered to make this push sustainable. There is a powerful argument to be made for educating girls because of their potential, not only because it is right. This is the argument that will convince the private sector to invest in studies led by women and scholarships for girls, among other activities to promote gender equity. The only way to make this argument is to demonstrate results by women and girls; and the only way to do this is to track and monitor (by Missions). In the short term, data need to be consolidated and shared. HEA should also conduct rapid assessments every six months to ensure that trainings are effective and that participants are implementing what they learn. In future programming, the next project should include the new standard F indicators on gender and a Gender and Social Inclusion Specialist should be included as key personnel to push the endeavor. Persons with training and experience in the context of El Salvador in gender sensitivity analysis and social inclusion should be considered. In addition, more can be done with regard to social inclusion; specifically, future programming should include students with disabilities in universities. Lastly, men should be included in the endeavor(s) as the only way for true societal change is if all stakeholders are involved. USAID should mandate participation of men and women in activities that promote gender equity. 5.4 RECOMMENDATION 4 Recommendation 4: The HEA should include the government more in the policy reform process of higher education and the HEA should make the process more open22. Supported by the following conclusions 10. There is a need to keep the project grounded in the reality of the country/context. 11. The policy has the potential to be shared more openly with participants. Gains have been made in the practical sense beyond the mandates and detailed actions of the HEA, together with universities and the private sector. Government has been included to some extent and is pleased with the results. However, more inclusion of the array of actors is required, including the MINED. The policy is drafted. Government needs to be involved for the reform to take place. In the short term, this includes—upon completion of the presidential elections in February 2019—working more closely with the GOES. In the long term, USAID needs to write the Statement of Work (SOW) for a future contract with specific results and, in turn, measurable indicators, which indicate: capacity of 22 Following the completion of the data collection and the scope of this evaluation, the call to participate according to one KII was broad: 75 percent out of the 41 Higher Education Institutions (HEIs) participated. The Strategic Committee also participated including staff from: Ministry of Education, Ministry of Economy, the Concejo de Educación Superior (CES), and the Comisión para la Acreditación de la Calidad en la Educación Superior (CDA). In all workshops, there was participation from the government, the private sector, the HEIs and students. 29 the government strengthened to manage the quality of higher education and coordination with industry, with a result and clear IRs (indicators to guide the project will be created). 5.5 RECOMMENDATION 5 Recommendation 5: The HEA should improve data management, knowledge sharing, and learning throughout the system and create a knowledge clearinghouse or observatory. Supported by the following conclusions 7. There is solid collaboration in research, but the lack of timely implementation, support on publication writing, and knowledge sharing could compromise the collaboration momentum. 12. More data are required to assess the final impact of the HICD training. 14. HEA plays a leadership role in terms of exemplifying how to incorporate gender, and more research and data will continue to demonstrate achievements and make convincing arguments for society at large. 18. There is an opportunity to capitalize on the U.S. tour so that CDCS in El Salvador learn from each other and collaborate more effectively and in a timely manner23. As part of USAID’s new MEL policy (updated in 2016), more effort is being placed on sharing results and learning more broadly. Capturing and sharing data and information from, for example, the applied research studies (as intended on March 20-21, 2018) is essential to make the argument for industry investments. Furthermore, as noted above, sharing results by women is crucial for making the argument for their inclusion in industry and university. In addition, results from the CDCs should be captured via data and monitoring in the CDCs, which should filter-up to the HEA. In the short term, the project should work closely with the universities to consolidate all data and investigations for USAID’s Development Experience Clearinghouse (DEC). In future programming, a dashboard should be created with an observatory or clearinghouse for the investigations and reports generated by the investigators, professors, and others to share with industry and other universities. Lastly, USAID should consider a contractual modification that allows for a “no-cost extension.” Given the delays in implementation (due to staffing turnover and especially the delays in approval of the curriculum reforms, which in turn influenced the implementation of the scholarships), the dissemination of results from applied research could be shared broadly with stakeholders if the Activity continued for an additional six months. This would be facilitated by and at little cost to the Activity. It would ensure that the investment of the Activity in the research activities was capitalized on by sharing the results. Now, with universities, the private sector, and, to some extent, the GOES on board, is the time to leverage the momentum built and to share the knowledge generated through the HEA. There is a case for a no-cost extension to share more knowledge. 23 Following the evaluation period, a sustainability plan has been developed for the Cluster model. The research units/labs have identified and further developed strategies for sustainability through their links with Industry. 30 ANNEXES Annex 1: Statement of Work Annex 2: Team Conflict of Interest Disclosures Annex 3: Instruments/Protocols Annex 4: Documents Consulted Annex 5: Schedule of KIIs Annex 6: Question 3 Supplement Annex 7: Illustrative Gender Indicators Annex 8: Evaluation Plan Annex 9: Statement of Difference 31 ANNEX 1: STATEMENT OF WORK 32 STATEMENT OF WORK (SOW) USAID/El Salvador Final Performance Evaluation of the Higher Education for Economic Growth Project A. Purpose of the Evaluation The purpose of the Final Performance Evaluation of the USAID Higher Education for Economic Growth project is to inform USAID of: a) project achievements and challenges to date, to focus implementation to meet or surpass contract objectives and assure sustainability; b) main approaches and activities supported by the project that have been valuable to achieve the project goal; c) based on lessons learned, what worked well and what did not; and d) provide recommendations that will serve as input for the design of any future USAID Higher Education for Economic Growth project. The evaluation will provide empirical evidence to support learning and continuous improvement in USAID’s work in this activity and future ones. Evaluation results will be used for reporting purposes to stakeholders such as the Government of El Salvador (GOES), private sector representatives, and academia. The principal audience of this evaluation internally will be USAID, particularly the Economic Growth Office, the Regional Program Office, and the Front Office of the El Salvador Mission; and the USAID Bureau for Economic Growth, Education and Environment’s Office of Education in USAID Headquarters. The evaluation will be undertaken through a collaborative, utilization-focused approach.24 The main participants in the evaluation will be the prime implementing partner, RTI International; activity beneficiaries (academia, private sector associations, enterprises), Ministry of Education (MINED), National Directorate of Higher Education (DNES), Ministry of Economy of El Salvador (MINEC), students; and USAID. This Final Performance Evaluation is scheduled to occur at the two-thirds point of program implementation, starting during the first quarter of fiscal year 2018, with approximately one year and a half left of implementation (the activity ends in June 2019). The evaluation will cover the period June 6, 2014 through September 30, 2017. B. Background Information About the Higher Education for Economic Growth Project25 Activity Implementing Partner Award Number and Dates Funding Higher Education for Economic Growth RTI International AID-519-C-14-00004 June 6, 2014 – June 5, 2019 $22,000,000 Within USAID’s Country Development Cooperation Strategy (CDCS) 2015-2017, the Higher Education for Economic Growth project contributes to Development Objective 2 “Economic Growth Opportunities in Tradables Expanded.” Specifically, it contributes to Intermediate Result (IR) 2.2 “Productivity of Targeted Businesses Increased” and, jointly with the USAID Bridges to Employment activity, Young Entrepreneurs activity, and USAID Economic Competitiveness activity to Sub-IR 2.2.1 24 The evaluation should be planned and conducted in ways that enhance the likely utilization of both the findings and of the process itself to inform decisions and improve performance. 25 The Evaluation Team will have access to the contract and its amendments prior to departure for El Salvador and before the preparation of the Evaluation Plan. 33 “Higher Education and Workforce Competencies Strengthened.” The Higher Education for Economic Growth project contributes to the achievement of Objective 1 of the U.S. Government Strategy for Engagement in Central America (CEN strategy) of “Prosperity and Regional Integration” (Education and Workforce Development) and the second strategic line of action in the Plan of the Alliance for Prosperity in the Northern Triangle26 of “Developing Opportunities for Our People” (Building human capital). The activity contributes to the Goal 2 of the USAID Education Strategy “Improved ability of tertiary and workforce development programs to generate workforce skills relevant to a country’s development goals,” and is aligned with the line of action in GOES’ Five-Year Plan of “Strengthening the coverage and quality of higher education, articulating it with other levels of the education system and fostering research and knowledge generation.” The development hypothesis of the Higher Education for Economic Growth project states that “if El Salvador’s Higher Education system is strengthened and aligned with private productive priority sector needs, then competitiveness and productivity will improve, contributing to long-term, broad-based economic growth.” The development hypothesis assumes that if the higher education sector of El Salvador improves the capacity of faculty staff, aligns curricula and research with labor market demand, enables key educational institutional and systemic reforms, and improves collaboration among local and international higher education institutes, public and private sectors, and donors, then the relevance and quality of the higher education system’s response to priority sector needs will be increased, productivity improved, and long￾term economic growth enhanced. The Higher Education for Economic Growth project builds partnerships between industry sectors and local HEIs to develop demand-driven educational programs and research to form highly qualified professionals and contribute to industry growth and productivity to help stimulate the economy and social development. The project strengthens the higher education system to respond to the country’s productive sector needs by improving relevance and quality of curricula and applied research, focusing on academic programs that respond to the needs of the labor market and provide faculty and professionals with opportunities to advance their industry knowledge. The Higher Education and Economic Growth project is implementing an industry demand-driven education model through four industry-higher education clusters in four sectors: 1. Information and Communications Technology; 2. Energy and energy efficiency; 3. Light manufacturing; and 4. Agro-industry and food processing. Each cluster includes the private sector, an anchor university, and associate HEIs. It also includes participation from the government. The collaboration between industry and HEIs will help create and update curricula that articulate skills requirements defined by industry for high-demand careers in growing sectors, Industry Advisory Committees will be created to strengthen the link between educators and business leaders and to share information on the job market, curriculum planning, internships, and practical training. USAID/El Salvador also helps develop the capacity of HEIs to perform applied research, thus meeting the technical and technology-related challenges of Salvadoran industry. It offers scholarships to Salvadoran faculty and researchers for study at innovative Professional Science Master’s degree programs. 26 http://idbdocs.iadb.org/wsdocs/getdocument.aspx?docnum=39224238 34 C. Evaluation Questions The following evaluation questions, in priority order, have been identified by USAID/El Salvador. They should be answered by the Evaluation Team and clearly presented in the Final Report in terms of how they relate to the evaluation purpose. In addition, for each of the questions below, the Evaluation Team should incorporate, to the extent feasible, analysis of possible differences associated with gender or social groups, particularly historically excluded groups (youth and women/girls), and they should be reported separately for men and women. What have been the most significant intended and unintended results achieved to date? What have been the main internal and external factors that have influenced the achievement or non￾achievement of the activity’s expected results as planned? To what extent has the demand-driven model (clusters model) been an effective space to build dialogue within private sector, academia, and GOES in order to respond to private sector needs and priorities? Are the four clusters already sustainable? To what extent has the cluster model been attractive to engage private sector and obtain their commitment through leverage? What changes have been made in HEIs in terms of their institutional capacity to continue introducing and/or supporting educational reforms and/or academic programs in the future? What changes have been made in the GOES in terms of their institutional capacity to continue introducing and/or supporting educational reforms and/or academic programs in the future? How were gender equality, female empowerment, and social inclusion integrated in the implementation of activity interventions? To what extent is the activity influencing changes in gender participation in higher education? What approaches utilized and results achieved by the Higher Education for Economic Growth activity have the potential to continue to exist after USAID’s funding ends? Which ones should continue to be supported? D. Evaluation Methodology, Data Collection, and Analysis For the performance evaluation, a non-experimental mixed-methods design that combines a comprehensive, rigorous analysis of existing quantitative data with customized qualitative techniques designed to elicit primary data from a wide range of counterparts, partners, beneficiaries, and stakeholders is recommended. This approach allows for triangulation of complementary data to elucidate linkages between activity inputs, outputs, and outcomes. The Evaluation Team should consider a range of possible methods and approaches for collecting and analyzing the data to address the evaluation questions thoroughly. The use of participatory methods and activities that will enhance collaboration and dialogue among counterparts is required. Further, data collection and analysis methods should be sensitive to possible differences related to sex and/or social status and should follow applicable Institutional Review Board (IRB) guidance on data security to ensure safety and confidentiality of all individuals providing data or information for the purposes of the evaluation. In addition, these methods, to the extent feasible, should be capable of identifying both positive and negative unintended consequences for women or girls. The finalized evaluation method(s) and approaches, data collection plan, and analysis will be included in the Evaluation Plan submitted to USAID for revision and approval before field visits and data collection begin (see Deliverables section below). The method(s) proposed should comply with the USAID Evaluation Policy (http://www.usaid.gov/evaluation/policy). The data collection plan for this evaluation will include, at a minimum: a desk review of relevant 35 documents such as the contract and work plans; review of activity performance monitoring and context data; key informant interviews and/or focus group discussions promoting equal participation of women and men; and direct observation through site visits. USAID/El Salvador expects both qualitative and quantitative data to be collected; and the results will be coded, triangulated, and analyzed for content. The Evaluation Team is encouraged to propose additional/alternate data collection and analysis methods in the Evaluation Plan that they consider can yield stimulating, robust evidence in answering each of the evaluation’s questions. Data collection shall be systematic and data must comply with the five data quality standards of validity, integrity, precision, reliability, and timeliness.27 Specific interview, survey, and/or focus group protocols will be appended to the Evaluation Plan and finalized with approval from USAID; the questions should be used to answer each of the evaluation questions listed in this document and address the purpose of this evaluation. All data collected in response to the evaluation questions must have as much level of disaggregation as possible. As minimum, and per USAID Gender Equality and Female Empowerment Policy,28 all data must be disaggregated and analyzed by sex, as well as analyzed for any differences between effects on men and women or male and female participation. Desk review of relevant documents USAID will provide the Evaluation Team with all relevant strategy and activity specific documents, such as the contract between USAID/El Salvador and RTI International and its amendment(s) including the expected results, performance reports stating the results achieved, any prior assessments if applicable, etc. The Evaluation Team must review these documents and other existing literature provided by USAID/El Salvador and others in preparation for the initial team planning meetings and before meeting with local stakeholders for interviews. The Evaluation Team is expected to review these, make their own contextual literature research and review, and create a Review Matrix to be delivered to USAID as part of the final Evaluation Report, which indicates how key information extracted from reviewed documents and other methodologies used link to each evaluation question. As minimum, the Evaluation Team shall review the following document relevant to the Higher Education for Economic Growth Final Performance Evaluation: • Country Development Cooperation Strategy 2013-2017 • Higher Education for Economic Growth contract and amendments • Higher Education for Economic Growth Monitoring and Evaluation Plan • Higher Education for Economic Growth annual work plans • Higher Education for Economic Growth annual and quarterly progress reports • Higher Education for Economic Growth Participant Training Reports • Higher Education for Economic Growth Grants Manual • Higher Education for Economic Growth Gender Analysis • Higher Education for Economic Growth Clusters Profile Reports • U.S. Government Strategy for Engagement in Central America • Plan of Alliance for Prosperity in the Northern Triangle • USAID Education Strategy 2012 – 2016 • El Salvador Higher Education Law • Policy for Higher Education Proposal prepared with project support. 27 http://pdf.usaid.gov/pdf_docs/Pnadw118.pdf 28 USAID Gender Equality and Female Empowerment Policy (https://www.usaid.gov/sites/default/files/documents/1865/GenderEqualityPolicy_0.pdf) 36 USAID/El Salvador will provide the Evaluation Team access to these documents. Some of them may be publicly available on USAID’s Development Experience Clearinghouse (DEC): http://dec.usaid.gov Review of performance indicators Higher Education for Economic Growth has an activity-specific Monitoring and Evaluation Plan and has collected data on a number of standard and custom indicators during activity implementation. The monitoring data collected will provide one source of data on progress toward objectives and outcomes. The Evaluation Team may use monitoring data on performance indicators as part of the evaluation analysis and should report on it in the Final Report as much as it relates to the evaluation questions stated above and satisfies relevant data quality standards. Context data, such as the World Competitiveness Index’s subcomponent of Higher Education and Training, should be analyzed and included to the maximum extent possible when answering the evaluation questions. Key informant interviews, surveys, focus group discussions The Evaluation Team will interview stakeholders, through key informant interviews, group interviews, short surveys, and/or focus groups interviews. The Evaluation Team will include both men and women in the stakeholders’ consultation processes. USAID and/or its RTI International will provide key informant contact information once the evaluation begins. At minimum, the Evaluation Team will interview: • Key USAID/El Salvador staff: Contracting Officer’s Representative (COR), Bilateral Team Leader, Economic Growth Office Director, Regional Program Office representatives. • RTI International staff. • Higher Education for Economic Growth activity key partners: Rutgers University, World Learning, FEDISAL. • Government of El Salvador counterparts, including the Ministry of Education (including DNES) and the Ministry of Economy. • Private sector representatives from the activity’s clusters: the Salvadoran Industrial Associations (ASI), the Salvadoran Agricultural and Agro-Industrial Chamber (CAMAGRO), the Salvadoran Information and Communications Technology Chamber (CASATIC), and Asociacion Salvadoreña de Energía Renovable (ASER). • Participating higher education institutes. • Anchor universities: José Simeón Cañas Central American University (UCA), Catholic University of El Salvador (UNICAES), Don Bosco University (UDB), and Francisco Gavidia University (UFG). Other universities: such as the National University of El Salvador (UES), and Eastern University (UNIVO). The Evaluation Team is encouraged to interview other donors and key stakeholders as needed involved with higher education or workforce development-related cooperation programs and other actors that can provide an insight into USAID programmatic impacts. A sampling plan describing the selection process; such as purposeful, random, or a combination of approaches;29 for organizations and stakeholders for key informant interviews, surveys, and focus groups discussions (including sex disaggregation) is expected to be included in the Evaluation Plan and Final Report. 29 Some sampling approaches include quota sampling, proximity sampling, convenience sampling, theoretical sampling, typical case sampling, etc. 37 Site visits and direct observation The Evaluation Team, in consultation with USAID/El Salvador and RTI International, will select relevant site visits based on a sampling plan developed for the Evaluation Plan and included in the Final Report. At a minimum, the Evaluation Team should expect to visit the campus of the beneficiary higher education institutes: Higher Education Institute Municipality UCA San Salvador UES San Salvador UFG San Salvador UDB Soyapango (Metropolitan Area of San Salvador) UNICAES Santa Ana (64 Kilometers northwest of San Salvador) UNIVO San Miguel (138 Kilometers east of San Salvador) The higher education institutes are located in 4 out of the 50 municipalities prioritized by the GOES under its three-phase citizen security plan called “Plan El Salvador Seguro” (El Salvador Secure Plan). San Salvador, Soyapango, and Santa Ana are part of the Phase I of this plan; and San Miguel is part of the Phase II. All the facilities of the higher education institutes are located in urban areas and they are easily reachable by car departing from San Salvador. Some universities, such as UES and UNICAES, have a main campus but, also, they have branches in other municipalities. The Evaluation Team will conduct site visits only in the main campus of the universities. The Evaluation Team may attend events hosted or sponsored by the Higher Education for Economic Growth project during the field work period of the evaluation to conduct direct observation. The Evaluation Team can use these events to talk with stakeholders, conduct interviews, and collect additional data as evidence to answer the evaluation questions. USAID/El Salvador and RTI International will provide the Evaluation Team with a list of events once the evaluation begins. Team planning meetings An initial team-planning/kick off meeting will be held in El Salvador between USAID/El Salvador and the Evaluation Team before the submission of the Evaluation Plan so that USAID/El Salvador can clarify any questions from the Evaluation Team, expectations, and guidelines. The expected results of this meeting are to: • Clarify each team member's role and responsibilities; • Confirm the anticipated timeline and deliverables; • Discuss data collection tools and methodologies by evaluation question to be presented in the Evaluation Plan; and • Identify communications logistics and how the Evaluation Team, USAID/El Salvador, and RTI International will communicate with each other. Additional meetings may be held as deemed necessary by USAID/El Salvador and/or the Evaluation Team. E. Deliverables It is estimated that not more than 70 working days of services from the starting date of the evaluation will be required to complete a high-quality evaluation as required under this SOW. During that timeframe, the Evaluation Team shall submit the following deliverables: 38 An Evaluation Plan, in Word font Gill Sans30 size 12, to be completed by the Evaluation Team after the Team Planning Meetings, no later than 15 calendar days after the starting day of the evaluation. USAID/El Salvador will receive the Evaluation Plan via electronic mail and review it to provide comments no later than 5 working days after receiving the document. The Evaluation Plan will provide details of how the various deliverables, tasks, and activities will be undertaken. It must include at least: • Higher Education for Economic Growth description and logic (change theory/development hypothesis); • Evaluation design,31 and the explanation of why one design or mix of designs proposed is the most appropriate, its limitations, and how these limitations will be addressed; • A matrix summarizing the following information per each evaluation question: o Method(s) for data collection, data source, the explanation of why one method or mix of methods is the most appropriate, its limitations and the ways to address them; o Technique(s) for data analysis,32 the explanation of why one analysis technique or mix of techniques is the most appropriate, its limitations and the ways to address them. • Data Management Plan describing the capture of data (for example, interview notes or live recording), storage and transfer, and how all data will be handled in such a manner as to protect the identities of informants in any situations where there are comments could potentially have a negative impact on their employment or security; • Timeline and/or Milestone Plan, including tentative starting time for data collection and duration of each activity conducted under the evaluation; • Drafts of data collection protocols, such as questionnaires or focus group moderator guide(s), interview scripts, consent form,33 etc.; • Evaluation Team composition and roles; and • Location for the evaluation and Site visit plan. If the Evaluation Plan includes key informant interviews, surveys, and/or focus group discussions, the Evaluation Plan should include the following information: 1. How the interviews/surveys will help to answer the evaluation questions; 2. Who will conduct the interviews/surveys and why they are qualified to do so; 3. What the rationale and methods are for deciding the number, timing, and location of the interviews/surveys; 4. How the participants will be selected and recruited; 5. How the interviews/surveys will be recorded; and 6. How the interview/survey data will be analyzed and presented. The Evaluation Plan, particularly the data collection and analysis protocols, as well as interview and focus group guides must be approved by USAID/El Salvador prior to the start of data collection and the field work. All interview protocols must be submitted in English and Spanish. The Evaluation Team will have another five working days to make any changes. Once the Evaluation Plan is approved, the Evaluation Team will submit to USAID/El Salvador an electronic copy of it in PDF. Any subsequent change to the Evaluation Plan must be approved by USAID/El Salvador. The Evaluation Team shall provide USAID/El 30 If the Evaluation Team does not have Gill Sans font family available, they must use any other approved font as per the USAID Graphic Standards Manual and Partners Co-Branding Guide (https://www.usaid.gov/sites/default/files/documents/1869/USAID%20Graphics%20Standards%20Manual%20and%20Partner%20Co%20Branding% 20Guide%20February%202016.pdf) 31 Some examples of evaluation designs for performance evaluations include snapshot design, cross-sectorial design, before-and-after design, time series design, case study design, panel design, etc. 32 Some examples of data analysis techniques include parallel, conversion, sequential, multilevel, data synthesis, content analysis, contribution analysis, etc. 33 If underage persons (less than 18 years old) will participate in this performance evaluation, the Evaluation Team must make sure to comply with all national regulations related to Child Protection. 39 Salvador and RTI International with a preliminary briefing on the Evaluation Plan prior to the beginning of data collection. Brief weekly bullet reports of activities, submitted to the COR of this evaluation by electronic mail due every Monday by the close of business. A Draft of the Final Report in Word, font Gill Sans size 12, submitted for review due no later than 42 calendar days after the approval of the Evaluation Plan via electronic mail. USAID/El Salvador will be responsible for distributing it to the implementing partner and other stakeholders for comments. USAID/El Salvador will consolidate all comments and send the draft back to the Evaluation Team within 10 working days. At a minimum, and in accordance with the USAID Evaluation Policy and ADS 201, the Final Report and its draft versions must include the following sections: • An abstract of not more than 250 words briefly describing what was evaluated, evaluation questions, methods, and key findings or conclusions. The abstract should appear on its own page immediately after the evaluation report cover; • An Executive Summary 2-5 pages in length that summarizes key points (purpose and background, evaluation questions, findings, and conclusions). Any information provided in the executive summary appears in the full report; • Evaluation purpose, audience, and anticipated use(s) of the evaluation; • Description of the activity to be evaluated including (if available) award numbers, award dates, funding levels, and implementing partners; • Brief background information. This should include country and/or sector context; specific problem or opportunity the intervention addresses; and the development hypothesis, theory of change, or simply how the intervention addresses the problem; • Evaluation questions; • Through description of the evaluation design and any challenge/limitations,34 with emphasis on the timeliness and methods for data collection and data analysis; • Relevant data analysis tables; • Findings (facts), and conclusions drawn from the analysis of the findings related to the evaluation questions;35 • Action-oriented, practical, and specific recommendations drawn from the conclusions with defined responsibility assigned for the action; • A dissemination plan of findings, conclusions, and recommendations to intended users of the evaluation; and • Appendices: o Original SOW, annotated with any changes approved by USAID/El Salvador o Evaluation and data collection team composition (qualifications, experience, and roles), with conflict of interest disclosures for all real or perceived conflicts of interest o Data collection protocols and instruments including questionnaires and checklists o Review matrix of documents consulted o Meeting notes o Complete schedule of evaluation activities, meetings, and interviews o List of individuals and organizations contacted and sites visited o Tables, graphs, pictures taken during site visits, maps 34 The Evaluation Team must identify: a) steps taken to mitigate limitations; and b) how/whether the limitations affect any particular findings, conclusions, or recommendations. 35 The report clearly distinguishes findings, conclusions, and recommendations, and clearly differentiates them related to each evaluation question. The logical connections between findings, conclusions, and recommendations are clear to the reader. Each conclusion is based in specific findings, and each recommendation is clearly related to a conclusion. In moving from findings to conclusions, the analysis must be clear as to how findings are synthesized through different techniques such as divergence, convergence, and amalgamation; propensity; weighting; etc. 40 o Detailed and organized summary of findings from surveys, including summary statistics and an overview of respondents A Draft Report Briefing for USAID/El Salvador and other stakeholders that USAID consider necessary on the contents, findings, conclusions, and recommendations included in the Final Report. According to the audience, the Draft Report briefing may be conducted in English or Spanish. Only the Team Leader needs to be present for this briefing; however, local/regional Evaluation Team members may also attend. The Draft Report Briefing will be used by the Evaluation Team as a feedback exercise to adjust the Final Report. Final Report in PDF, font Gill Sans size 12, no longer than 40 pages in its body, excluding the cover page; Table of Contents; List of Acronyms; and Appendices. The approved Final Report must adhere to USAID’s Evaluation Policy and ADS 201 and must be submitted in English and Spanish and have incorporated USAID’s comments, as appropriate. The Final Report will be due to USAID 10 working days after the Evaluation Team receives comments on the draft and no later than 90 calendar days after the start of the evaluation. Five high-quality printed, bound copies in English and Spanish of the Final Report must be submitted to USAID/El Salvador within 10 calendar days of acceptance of the Final Report. A Two-Page summary of the evaluation purpose, findings, conclusions, and recommendations. The One-Page summary will be prepared in English and Spanish in PDF. Any raw data (qualitative or quantitative) collected in electronic form (DVD or flash drive, in original format of Word, Excel, etc.) is due no later than 100 calendar days after the starting date of the evaluation. As per ADS 540, the Evaluation Team must submit to the Development Data Library (DDL) at www.usaid.gov/data, in a machine-readable, non-proprietary format, a copy of any datasets that are used (or of sufficient quality) to produce an Intellectual Work. A Final Presentation with PowerPoint slides to USAID (and potentially to any other stakeholders that USAID considers relevant) in English and Spanish as the Final Report is being finalized no later than 100 calendar days after the starting date of the evaluation. Only the Team Leader needs to be present for the final presentation; however, local Evaluation Team members may also attend. The Evaluation Team will upload the final presentation to the DEC and submit an electronic copy of the final presentation to the COR of this evaluation. Other deliverables as identified during the Team Meeting and agreed to by USAID and the Evaluation Team. All reports and papers will be considered draft versions until they are approved by USAID. These draft documents must be labeled with the word “DRAFT” in watermark. Findings must be presented as analyzed facts, strong qualitative and quantitative evidence and data, and not based on anecdotes, hearsay or the compilation of people’s opinions. To ensure unbiased findings, there is no guarantee that findings will be modified based on USAID suggestions. The Evaluation Team will research, investigate, and corroborate as objective any suggestion before it is incorporated in the findings, and the change will be noted in the draft document so as to have a record of the change. All submitted reports and presentations must be thoughtful, well-researched, and well-organized documents, and objectively answer the evaluation questions. When writing the report, the Evaluation Team must remember the different audiences. The style of writing should be easy to understand and concise, while making sure to address the evaluation questions and issues with accurate and data-driven findings, justifiable conclusions, and practical recommendations.36 The Evaluation Team should clearly list 36 For additional information on the criteria to ensure the quality of Evaluation Reports, see Annex 1 of the USAID Evaluation Policy (https://www.usaid.gov/sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf) 41 any biases or limitations that exist during both data collection and analysis (selection bias, recall bias, unobservable differences between comparator groups, etc.). In addition, all real or possible conflicts of interest must be disclosed by each member of the Evaluation Team in writing. When quoting an individual in any report, the Evaluation Team must always give the context or circumstances of the quote. Correcting a grammatical error in the quote may be valid, but not rewording an entire phrase. When translating quotes from one language to another, the Evaluation Team should do so in an idiomatic way and care must be taken to ensure that the tone of the translation is equivalent to the tone of the original. Quotes should be presented in their original language in report texts. All reports must comply with the USAID Graphic Standards Manual and Partner Co-Branding Guide (https://www.usaid.gov/sites/default/files/documents/1869/USAID%20Graphics%20Standards%20Manual% 20and%20Partner%20Co%20Branding%20Guide%20February%202016.pdf) and the ADS Style and Format Guide (https://www.usaid.gov/sites/default/files/documents/1868/501mac.pdf).37 Once a Final Report has been approved by USAID, the Evaluation Team will make it compliant and submit it to the DEC.38 The Evaluation Team will send by electronic mail to USAID the DEC link where the evaluation reports are available. USAID will assess the quality of all evaluation reports using the Evaluation Report Checklist and Review Template (https://usaidlearninglab.org/sites/default/files/resource/files/template_- _evaluation_report_checklist_and_review_august_2017.pdf). If necessary, the final report may include a Statement of Differences as an annex if any significant unresolved differences of opinion on the part of USAID/El Salvador, RTI International, and/or members of the Evaluation Team remain in the final version. Evaluation Management Evaluation Team The Evaluation Team must have an appropriate mix of technical skills to conduct this performance evaluation which will use a combination of multidisciplinary international, regional, and local experts. The Evaluation Team must include at a minimum: Evaluation Team Leader Minimum qualifications Education: Master’s degree in development or related development fields, such as Economics, Political Science, Public Administration, Business Administration, or other disciplines related to development assistance is required. Ph.D. or Doctorate degree or professional with Doctoral candidacy is a plus. Formal training in monitoring and evaluation is preferred. Language Proficiency: American English Level IV and Spanish Level III Work Experience: At least 10 years of relevant prior experience conducting/leading rigorous evaluations using both quantitative and qualitative methods for development objectives and monitoring projects and programs, preferably in Latin America. Experience in Central America is a plus. At least eight years of project management experience. Experience with management of multidisciplinary teams is a plus. S/he must have familiarity with USAID’s objectives, approaches, operations, and policies, particularly as they relate to evaluations (USAID Evaluation Policy, USAID Gender Equality and Female Empowerment Policy, ADS 201). 37 Evaluation Team may use the Evaluation Report Template available in http://usaidlearninglab.org/library/evaluation-report-template 38 Per Automated Directives System 540, documents and development assistance projects materials produced or funded by USAID must be submitted for inclusion in the DEC: https://dec.usaid.gov/dec/content/Create.aspx?ctID=ODVhZjk4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy 42 Position Description: The Evaluation Team Leader will be responsible for overseeing and coordinating all activities related to this performance evaluation and for ensuring the production and completion of quality deliverables in a professional manner, in conformance with this SOW. Two (2) Higher Education Specialists Minimum qualifications Education: Master’s degree in Education, Academic Administration, Research, Business Administration, Economics, and other Economic Development fields. Ph.D. or Doctorate degree or professional with Doctoral candidacy is a plus. Formal training in monitoring and evaluation is preferred. Language Proficiency: Spanish Level IV and American English Level IV. Work Experience: At least eight years of progressively responsible, professional-level experience in assessment of any of the following fields: higher education curriculum design; labor market and private sector accreditation/articulation; Science, Technology and Innovation, and Applied Research; preferably in Central or Latin America. At least one of the Specialists must have experience or knowledge of El Salvador’s higher education issues and entities. Preferably with familiarity with USAID’s objectives, approaches, and operations, particularly as they relate to evaluations. Position Description: The Specialists will provide guidance to the Evaluation Team Leader on Higher Education Institutes Faculty’s Human Capital Development, Productive Sector Articulation, and Higher Education System Reform. The Evaluation Team should have considerable experience in designing, monitoring, and evaluating development assistance programs. They must have excellent written and oral presentation skills, strong interpersonal skills, ability to conceptualize and write clearly and concisely, and outstanding qualitative/quantitative research skills and ability to synthesize large amounts of disparate information. Understanding of the Latin American context is necessary, with a preference for personnel with work experience specifically in Central America. At least one key personnel member must have experience working in gender and social inclusion issues in a development context, transforming qualitative data, analyzing quantitative data, and producing data visualization in an easily digestible format. All Team members will be required to provide to USAID/El Salvador a signed statement indicating any conflict of interest. The Team Leader must be someone external to USAID. No key personnel shall have been directly involved in the implementation of the Higher Education for Economic Growth activity. Anyone who has worked directly with USAID/El Salvador or RTI International in the last five years must not be considered as part of the Evaluation Team. Logistics The Evaluator will be responsible for all logistics support under this SOW, including field office administration, all travel arrangements (with required USAID clearances), team planning facilitation and appointment scheduling, coordination with all partners and stakeholders involved, administrative services (computer support, printing and copying), report editing and dissemination, and for complying with provisions set forth in this SOW. USAID/El Salvador will provide limited support to the Evaluation Team. This support, if needed, includes assistance in arranging high-level meetings; access to the U.S. Embassy compound; recommendations on in-country lodging; and access to all reports, data, and other relevant documents created by RTI International under the Higher Education for Economic Growth activity. USAID/El Salvador representatives may accompany the Evaluation Team for some or all of the evaluation. The Evaluation Team is expected to consider this when making logistical arrangements. 43 ANNEX 2: TEAM ROLES< COMPOSITION AND CONFLICT OF INTEREST DISCLOSURES 44 Roles The Evaluation Team was composed of three individuals, Megan Gavin, Ana Cristina Accioly, and Oscar Picardo, supported by Logistics Coordinators Katia Zepeda and Cindy Abarca. The Team Leader, Megan Gavin, was responsible for coordinating the activities of the Evaluation Team, and ensuring the completion of quality deliverables in a timely way. She served as the main point of contact between USAID and the MEL Platform regarding this evaluation. The Team Leader drafted the final evaluation design, oversaw and supported the development of evaluation instruments, participated in data collection and analysis, integrate the findings of different team members, and coordinate preparation of the final report. The HE Specialist, Ana Cristina Accioly, supported Evaluation Question 2, concerning the demand￾driven cluster model. In addition, she supported data collection in particular. Ms. Accioly conceptualized evaluation instruments, sampling strategy, and data analysis methods per question for the evaluation design matrix, and supported overall analysis. The HE Specialist 2, Oscar Picardo, supported Evaluation Question 3, with an emphasis on GOES policy and capacity. Mr. Picardo supported data collection, conceptualized the evaluation instruments, sampling strategy, and data analysis methods per question for the evaluation design matrix, and supports overall analysis. He also supported analysis of quantitative data. The Logistics Coordinators worked with local partners to plan travel and schedule data collection, interviews, and assessment activities as required, and supported the Team Leader in all logistical and administrative aspects of the evaluation. This role was filled by Katia Zepeda and Cindy Abarca. Profile, experience, and conflict of interest Dr. Megan Gavin has 15 years of experience working in the international education sector. She conducted performance evaluations with USAID and has used both quantitative and qualitative techniques. She is well versed on USAID’s evaluation policy and USAID’s education policy. She has worked with multiple clients including the IDB, World Bank, USAID, UNESCO, and UNICEF and has worked in Belize, Dominican Republic, El Salvador, Egypt, Ethiopia, Honduras, Jamaica, Mexico, Nicaragua, Nigeria, Panama & Tajikistan & has supported projects in Ecuador, Ghana, & Liberia. Dr. Gavin holds a PhD in Education from the New School. There are no real or perceived conflicts of interest for Dr. Gavin in relation to this evaluation. Ana Cristina Accioly de Amorim is a senior education specialist with 17 years of international development experience. She has extensive experience managing and conducting USAID-funded program and evaluation to enhance educational opportunities for children and youth living in post￾conflict and crisis context. She has proven analytical experience designing and implementing research studies for rigorous evaluation using quantitative and qualitative methodologies. She recently completed a mid-term performance evaluation of education for Children and Youth (ECY) Program in El Salvador. Earlier she was Senior Education Specialist at the Cristo Rey Boston High School, implementing remedial programs and research to improve at-risk student’s academic performance. She served as Project Director for USAID’s Global Evaluation and Monitoring II IQC, providing management and quality assurance of education programs’ activities for USAID in the LAC region. She holds an EdM in International Education Policy from Harvard. There is no real or perceived conflict of interest for Ms. Accoily. Oscar Picardo Joao is a social science researcher, with significant experience in design, implementation and evaluation of education policies in Central America and the Dominican Republic. He was responsible for compiling the experiences of the Educational Reform in El Salvador 1994-2000, EDUCO and EDUCAME. He also led the technical portions of strategic planning and educational organizational analysis in the Dominican Republic and in Honduras. Additionally, he served as Academic Advisor of 45 Colegio García Flamenco, Advisor on Higher Education of the Ministry of Education and National Director of Higher Education from June to November of 2004, and as Assistant Rector of ISEADE￾FEPADE (2006-2012); Director of the Research Center on Sciences and Humanities at Universidad Dr. “José Matías Delgado” (2006-2011); and Director of the Science, Technology and Innovation Institute of Universidad Francisco Gavidia (2012-2014). Dr. Picardo holds a Masters of Education from the University of Louisville, and a post-degree in Distance Education and Digital Networks from the University of Murcia, Spain. Later he received a post-degree in Educational Finance from Harvard (2002) and a Master’s degree in Information and Knowledge Society at Universitat Oberta of Catalunya. He finished his PhD studies at Universitat Oberta of Catalunya, in “Information and the Knowledge Society.” Note: As noted in the Evaluation Report, special steps were taken to mitigate a potential perception of conflict of interest with regard to the inclusion on the team of Dr. Picardo, because he had participated in HEA via a subgrant to UFG (the Universidad Francisco Gavidia). These steps were as follows: Dr. Picardo did not conduct the site visit to UFG; his reflections were filtered (through the other team members); all findings were documented so they were based on evidence rather than opinions; and Dr. Picardo focused on EQ 3 and the role of the government rather than the universities. 46 47 48 49 ANNEX 3: PROTOCOLS/INSTRUMENTS 50 3.1 Principles of Adult Learning Scale (PALS) Developed by Gary J. Conti DIRECTIONS The following survey contains several things that a teacher of adults might do in a classroom. You may personally find some of them desirable and find others undesirable. For each item please respond to the way you most frequently practice the action described in the item. Your choices are Always, Almost Always, Often, Seldom, Almost Never, and Never. If the item does not apply to you, circle number 5 for never. Question/Item Response Category Value 1. I allow students to participate in developing the criteria for evaluating their performance in class. A AA O S AN N 2. I use disciplinary action when it is needed. A AA O S AN N 3. I allow older students more time to complete assignments when they need it. A AA O S AN N 4. I encourage students to adopt middle class values. A AA O S AN N 5. I help students diagnose the gaps between their goals and their present level of performance. A AA O S AN N 6. I provide knowledge rather than serve as a resource person. A AA O S AN N 7. I stick to the instructional objectives that I write at the beginning of a program. A AA O S AN N 8. I participate in the informal counseling of students. A AA O S AN N 9. I use lecturing as the best method for presenting my subject material to adult students. A AA O S AN N 10. I arrange the classroom so that it is easy for students to interact. A AA O S AN N 11. I determine the educational objectives for each of my students. A AA O S AN N 12. I plan units which differ widely as possible from my students’ socio-economic backgrounds. A AA O S AN N 13. I get a student to motivate himself/herself by confronting him/her in the presence of classmates during group discussions. A AA O S AN N 14. I plan learning episodes to take into account my students’ prior experiences. A AA O S AN N 15. I allow students to participate in making decisions about the topics that will be covered in class. A AA O S AN N 16. I use one basic teaching method because I have found that most adults have a similar style of learning. A AA O S AN N 17. I use different techniques depending on the students being taught. A AA O S AN N 18. I encourage dialogue among my students. A AA O S AN N 19. I use written tests to assess the degree of academic growth rather than to indicate new directions for learning. A AA O S AN N 20. I utilize the many competencies that most adults already possess to achieve educational objectives. A AA O S AN N 21. I use what history has proven that adults need to learn as my chief criteria for planning learning episodes. A AA O S AN N 22. I accept errors as a natural part of the learning process. A AA O S AN N Always Almost Always Often Seldom Almost Never Never A AA O S AN N 51 23. I have individual conferences to help students identify their educational needs. A AA O S AN N 24. I let each student work at his/her own rate regardless of the amount of time it takes him/her to learn a new concept. A AA O S AN N Question/Item Response Category Value 25. I help my students develop short-range as well as long-range objectives. A AA O S AN N 26. I maintain a well-disciplined classroom to reduce interference to learning. A AA O S AN N 27. I avoid discussion of controversial subjects that involve value judgments. A AA O S AN N 28. I allow my students to take periodic breaks during class. A AA O S AN N 29. I use methods that foster quiet, productive desk work. A AA O S AN N 30. I use tests as my chief method of evaluating students. A AA O S AN N 31. I plan activities that will encourage each student’s growth from dependence on A AA O S AN N others to greaterindependence. 32. I gear my instructional objectives to match the individual abilities and needs of the A AA O S AN N students. 33. I avoid issues that relate to the student’s concept of himself/herself. A AA O S AN N 34. I encourage my students to ask questions about the nature of their society. A AA O S AN N 35. I allow a student’s motives for participating in continuing education to be a A AA O S AN N major determinant in the planning of learning objectives. 36. I have my students identify their own problems that need to be solved. A AA O S AN N 37. I give all my students in my class the same assignment on a given topic. A AA O S AN N 38. I use materials that were originally designed for students in elementary and A AA O S AN N secondary schools. 39. I organize adult learning episodes according to the problems that my students A AA O S AN N encounter in everyday life. 40. I measure a student’s long-term educational growth by comparing his/her total A AA O S AN N achievement in class to his/her expected performance as measured by national norms from standardized tests. 41. I encourage competition among my students. A AA O S AN N 42. I use different materials with different students. A AA O S AN N 43. I help students relate new learning to their prior experiences. A AA O S AN N 44. I teach units about problems of everyday living. A AA O S AN N Always Almost Always Often Seldom Almost Never Never A AA O S AN N Scoring the Principles of Adult Learning Scale (PALS) Positive Questions Question numbers 1, 3, 5, 8, 10, 14, 15, 17, 18, 20, 22, 23, 24, 25, 28, 31, 32, 34, 35, 36, 39, 42, 43, and 44 are positive items. For positive questions, assign the following values: Always=5, Almost Always=4, Often=3, Seldom=2, Almost Never=1, and Never=0. Negative Questions Question numbers 2, 4, 6, 7, 9, 11, 12, 13, 16, 19, 21, 26, 27, 29, 30, 33, 37, 38, 40, and 41 are negative items. For negative questions, assign the following values: Always=0, Almost Always=1, 52 Often=2, Seldom=3, Almost Never=4, and Never=5. Missing Questions Omitted questions are assigned a neutral value of 2.5. Factor 1: Learner-Centered Activities Question # 2 4 11 12 13 16 19 21 29 30 38 40 Total Score Score Factor 2: Personalizing Instruction Question # 3 9 17 24 32 35 37 41 42 Total Score Score Factor 3: Relating to Experience Question # 14 31 34 39 43 44 Total Score Score Factor 4: Assessing Student Needs Question # 5 8 23 25 Total Score Score Factor 5: Climate Building Question # 18 20 22 28 Total Score Score Factor 6: Participation in the Learning Process Question # 1 10 15 36 Total Score Score Factor 7: Flexibility for Personal Development Question # 6 7 26 27 33 Total Score Score 53 Computing and Interpreting Scores Factor scores are calculated by summing the value of the responses for each item/question in the factor. Compare your factor score values to their respective means (see table below). If your score is equal to or greater than each respective mean, then this suggests that such factors are indicative of your teaching style. From such factors, you will then begin to identify what strategies you use to be consistent with your philosophy (from the Philosophy of Adult Education Inventory, PAEI). Those scores that are less than the mean indicate possible areas for improving a more learner-centered approach to teaching. An individual’s total score on the instrument is calculated by summing the value of each of the seven factors (see table below). Scores between 0-145 indicate your style is “teacher-centered.” Scores between 146-220 indicate your style as being “learner-centered.” For a complete description of PALS and each of the seven factors, see Conti, G.J. (1998). Identifying Your Teaching Style (Ch. 4). In M.W. Galbraith (Ed.), Adult Learning Methods (2nd ed., pp. 73-84). Malabar, FL: Krieger Publishing Company. Factor Mean Standard Deviation Your Score 1 38 8.3 2 31 6.8 3 21 4.9 4 14 3.6 5 16 3.0 6 13 3.5 7 13 3.9 TOTAL 146 20 54 3.2 Key Informant Interview—Stakeholders Name: Date: Role: Organization: Introduce self, purpose of study, and confidentiality. 1. Please reflect on the main results of the higher education (HE) Activity to date. Probe 1…Where these intended or unintended? Probe 2…What factors are associated with these results? Probe 3…Please reflect on the qualifications of faculty/staff. Probe 4…Please reflect on the curriculum and applied research. 2. Please reflect on the cluster model. Is it effective? Why or why not? Probe 1…Does it respond to the GOES needs? Probe 2…Does it respond to the private sector needs? Probe 3…How has the private sector been engaged? Probe 4…Is the model sustainable, i.e., will it continue beyond the life of the project? 3. What changes have been made in the institutional capacity of the institutions? Probe 1…What changes have been made in the GOES to support these institutions? Probe 2…How were things before and how are they now? Probe 3…Can you share more? 4. Please reflect on gender equity (do males and females have equal opportunity?). How have you seen it in this Activity? Probe 1…Please reflect specifically on female empowerment (do women have autonomy to make decisions?). Probe 2…Please describe how the HEA has influenced female participation in higher education. Probe 3…Please reflect on the situation before HEA. Probe 4…Please share how. 5. Please share your perspective as to what aspects of the HEA will continue to exist once there is no longer USAID investment? Why/why not? Probe 1…What aspects do you recommend should continue in the future? Probe 2…What aspects should be addressed in this current Activity now for future sustainability? Probe 3…What aspects of the activity are still pending and require more support? What aspects are on track on their own? Thank you. If there is more you would like to share, please feel free to do so. 55 3.3 Focus Group Discussion Guide Institution: Location: Date: Number of participants (males, females): Introduce self, purpose of study, group respect, and confidentiality 1. Please reflect on the main results of the HE Activity to date. Probe—Please share more. 2. Please reflect on the cluster model. Is it effective, does it work? Why or why not? Probe—Please share more. 3. What changes have been made in the institutional capacity of the institutions? How were things before? How are they now? Probe—Please share more. 4. Please reflect on gender equity. How have you seen it in this Activity? Do females and males have the same opportunities? Are they treated the same? Are they treated differently? Why and why not? Probe—Please share more. 5. Please share your perspective as to what aspects of the HEA will continue to exist once there is no longer USAID investment? Why/why not? Thank you. If there is more you would like to share, please feel free to do 56 ANNEX 4: DOCUMENTS CONSULTED 57 Country Development Cooperation Strategy 2013-2017 HEA contract and amendments HEA M&E Plan HEA annual work plans HEA annual and quarterly progress reports HEA Participant Training Reports HEA Grants Manual HEA Gender Analysis HEA Clusters Profile Reports U.S. Government Strategy for Engagement in Central America Plan of Alliance for Prosperity in the Northern Triangle USAID Education Strategy 2012 – 2016 El Salvador Higher Education Law Policy for Higher Education Proposal prepared with Activity support. Other documents related to higher education in El Salvador include: “Challenges, problems and perspectives of universities in El Salvador: Circumstantial opinions and pedagogic thinking (1997-2007) (Ed. Delgado 2008). “History of Universities in El Salvador” (Picardo, 2011). Department of Trade and Industry (2004). Practical Guide to Cluster Development, DTI Publication, London Etzkowitz, H. and Leydesdorff, L. (1997). Universities and the global knowledge economy: a triple helix of university-industry-government relations. Continuum: London; Guimón, Jose (2013). Promoting University-Industry Collaboration in Developing Countries. Policy Brief. The World Bank Publication, Washington, DC.; Mourshed, M., Farrell, D., and Barton, D. (2012). Education to employment: Designing a system that works. McKinsey & Company. 58 ANNEX 5: SCHEDULE OF KEY INFORMANT INTERVIEWS 59 Date Name Organization 02-20-2018 Sandra Duarte, COR; Carlos Arce, COR; Gabriela Vélez, M&E Specialist; Yolanda Martínez, Development Specialist USAID 02-20-2018 Julio Segovia, Karla Segovia, Annie Valencia: Program Team USAID 02-15-2018 Reina Durán, Chief of Party RTI 02-14-2018 Ernesto Martín Montero, Deputy Chief of Party RTI 02-14-2018 Edmundo Echegoyén, M&E Specialist RTI 02-15-2018 Gino Costa, Human Capital Specialist RTI 02-15-2018 Xiomara Hernández, HE Systems Specialist RTI 02-14-2018 Luis Rivera, Productive Sector Specialist RTI 02-26-2018 Jim Simon, Rodolfo Juliani Rutgers 02-26-2018 Neil O’Flaherty, Christopher Iverson World Learning 02-20-2018 Héctor Quiteño FEDISAL 02-21-2018 Erlinda Hándal, Vice Minister Technology MINED 02-23-2018 Francisco Marroquín, Director DNES 02-19-2018 Francisco Martínez, Vice Minister Ministry of Economy 02-16-2018 Mario Panameño ASI 02-27-2018 Agustín Martínez CAMAGRO 02-15-2018 Gracia Rossi CASATIC 02-20-2018 Carlos Saade ASER 60 ANNEX 6: SUPPLEMENTAL ANALYSIS FOR QUESTION 3 61 There Is No Hard Evidence The Evidence Is Weak There Is Evidence 3.1 Develop an information and monitoring system, including its sustainability plan, with critical indicators within the relevant organizations for the policy and decision-making of HE. 3.2 Develop the vision and strategic action plan of the Accreditation Commission (to include regional or international trends). 3.4 Facilitate collaboration and links between local and international HEIs, to include accreditation and English language institutes. 3.6 Develop the vision and the strategic action plan of the Higher Education Council, as evidenced in the HE evaluation. 3.8 Develop internship programs with organizations and companies that work in priority sectors, offering an experimental or practical approach to academic programs in priority key sectors. 3.10 Update the initial/original diagnosis and reference information and implement a plan for the collection of additional reference data. The participation and approval of USAID for new indicators will be required. 3.11 Create a Steering Committee, with the function of coordinating activities to improve HE, which would recommend or approve external activities for the support of the HE activity. The structure and composition of this committee will require the approval of USAID. 3.12 Address the issue of promoting the importance of HE with key actors, as a critical element in the Competitiveness Agenda of El Salvador. 3.3 Facilitate the establishment of associations and the promotion of collaboration among local HEIs since this is a key factor in the sustainability of the activity goal. 3.5 Carry out training for HEI staff involved in financial management, governance, and other issues relevant to the development of local capacities and sustainability. 3.7 Collaborate and establish partnerships between HEIs and the private sector to jointly address institutional or sectoral strengthening. 3.9 Facilitate the creation of career centers in participating HEIs and the training of staff to place students in internships and jobs, which requires the collaboration of private companies. 62 ANNEX 7: ILLUSTRATIVE EXAMPLES FOR GENDER INDICATORS 63 Indicator GNDR-2 Percentage of female participants in United States Government (USG)-assisted programs designed to increase access to productive economic resources (assets, credit, income, or employment)39 Definition Productive economic resources include: assets—land, housing, businesses, livestock, or financial assets such as savings; credit; wage or self-employment; and income. Programs include: • Micro, small, and medium enterprise programs; • Workforce development programs that have job placement activities; • Programs that build assets such as land redistribution or titling; housing titling; agricultural programs that provide assets such as livestock; or programs designed to help adolescent females and young women set up savings accounts. This indicator does NOT track access to services, such as business development services or stand-alone employment training (e.g., employment training that does not also include job placement following the training). The unit of measure will be a percentage expressed as a whole number. Numerator = Number of female program participants Denominator = Total number of male and female participants in the program The resulting percentage should be expressed as a whole number. For example, if the number of females in the program (the numerator) divided by the total number of participants in the program (the denominator) yields a value of .16, the number 16 should be the reported result for this indicator. Values for this indicator can range from 0 to 100. The numerator and denominator must also be reported as disaggregates. Primary SPS Linkage As a cross-cutting gender indicator, this indicator can be used to report on applicable activities under any of the Program Categories in the Standardized Program Structure and Definitions (SPSD). Linkage to Long￾Term Outcome or Impact The lack of access to productive economic resources is frequently cited as a major impediment to gender equality and women’s empowerment and is a particularly important factor in making women vulnerable to poverty. Ending extreme poverty, a goal outlined in the Sustainable Development Goals and USAID’s Vision to Ending Extreme Poverty, will only be achievable if women are economically empowered. Indicator Type Output Reporting Type Percent 39 This indicator is cross-cutting and linked to all results within the HEA (and or future USAID programming). 64 Use of Indicator Information generated by this indicator will be used to monitor and report on achievements linked to broader outcomes of gender equality and female empowerment and will be used for planning and reporting purposes by Agency-level, bureau-level, and in-country program managers. Specifically, this indicator will inform required annual reporting or reviews of the USAID Gender Equality and Female Empowerment Policy and the Joint Strategic Plan reporting in the Annual Performance Plan/Annual Performance Report (APP/APR), and Bureau or Office portfolio reviews. Additionally, the information will inform a wide range of gender-related public reporting and communications products and facilitate responses to gender-related inquiries from internal and external stakeholders such as the U.S. Congress, NGOs, and international organizations. Reporting Frequency Annual Reporting Data Source IPs Bureau Owner Agency: USAID Bureau/Office: PPL/P Point of Contact: Catherine Cozzarelli (712-1891, ccozzarelli@usaid.gov) Disaggregate(s) Numerator Denominator Indicator GNDR-4 Percentage of participants reporting increased agreement with the concept that males and females should have equal access to social, economic, and political resources and opportunities Definition This indicator will be used to gauge the effectiveness of USG efforts to promote gender equality by measuring changes in attitudes about whether men and women should have equal access to resources and opportunities in social, political, and economic spheres. Changes in attitudes are measured via the Equal Opportunity survey (see Data Source below for survey instructions) administered in conjunction with training or programs in any sector which include goals or objectives related to gender equality and women’s empowerment. Projects that aim to change participants’ broad attitudes about gender equality are particularly relevant. GNDR-4 is applicable to programs in multiple sectors that are designed to raise awareness of women’s human rights and/or to increase acceptance of gender equality among women and/or men (or girls/boys), including: programs that train journalists to report more responsibly on gender issues; education or social and behavior change programs designed to change gender norms and roles; programs designed to increase the political or economic participation of women; and health sector programs designed to drive changes in gender-based attitudes and behaviors, among others. Note that it is not necessary that programs be focused on the sectors reflected in the questions that comprise the indicator (i.e., political, economic) in order to report against GNDR-4. Any program that may feasibly alter attitudes about gender equality should report against this indicator. 65 The unit of measure will be a percentage expressed as a whole number. Numerator = the number of participants whose survey scores have improved over time Denominator = the total number of participants who participated in the relevant training/programming For example, if the number of participants whose scores improved over time (the numerator) divided by the total number of participants in the training/program (the denominator) yields a value of .40, the number 40 should be the reported result for this indicator. Values for this indictor can range from 0 to 100. The numerator and denominator must also be reported as disaggregates. This indicator must also be disaggregated by sex—see the disaggregates box below for details. Primary SPS Linkage As a cross-cutting gender indicator, this indicator can be used to report on applicable activities under any of the Program Categories in the SPSD. Linkage to Long￾Term Outcome or Impact This indicator measures changes in individual attitudes and norms about gender equality that may be a proxy for deeper structural changes in the social, political, and economic spheres. Indicator Type Outcome Reporting Type Percent Use of Indicator Information generated by this indicator will be used to monitor and report on achievements linked to broader outcomes of gender equality and female empowerment and will be used for planning and reporting purposes by Agency-level, bureau-level, and in-country program managers. Specifically, this indicator will inform required annual reporting or reviews of the USAID Gender Equality and Female Empowerment Policy as well as Joint Strategic Plan reporting in the APP/APR. Additionally, the information will inform a wide range of gender-related public reporting and communications products and facilitate responses to gender-related inquiries from internal and external stakeholders such as the U.S. Congress, NGOs, and international organizations. Reporting Frequency Annual reporting Data Source Data for this indicator will be collected by pre- and post-survey, once at the start of relevant USG-funded training/programming and a second time at the end of the training/programming. Results for GNDR-4 should therefore be reported at the end of the training/program when changes in attitudes can be calculated. The surveys should be administered to persons who can clearly be identified as program participants and should be translated into the language(s) spoken by participants, if necessary. The survey may be read to program beneficiaries who are illiterate. Each COR or Administrative Officer’s Representative (AOR) would be responsible for ensuring that implementers collect these data. Respondents will be asked: To what extent do you agree or disagree with the following statements? *Women should have equal rights with men and receive the same treatment as men do. *On the whole, men make better political leaders than women and should be elected rather than women. (r)*When jobs are scarce, men should have more right to a job than women. (r)Scale: Strongly Disagree, Disagree, Neither Agree nor Disagree, Agree, Strongly Agree. To score the opportunity measure, responses are coded 66 as follows: -2 = Strongly Disagree, -1 = Disagree, 0 = Neither Agree nor Disagree, +1 = Agree, +2 = Strongly Agree. The items with an (r) should be reverse-scored, i.e., those items followed by an “r” that have a score of -1 are recoded as +1. For example, for item 2 (“On the whole, men make better political leaders than women and should be elected rather than women” (r)), a response of ‘strongly agree’ is re-coded as ‘- 2’”). A higher score indicates greater agreement that men and women should have equal opportunities. Bureau Owner Agency: USAID Bureau/Office: PPL/P Point of Contact: Catherine Cozzarelli (712-1891, ccozzarelli@usaid.gov) Disaggregate(s) Numerator = the total number of participants whose survey scores have improved over time Denominator = the total number of participants Male (i.e., the percentage of male participants who showed increased agreement with gender equality concepts) and Female (i.e., the percentage of female participants who showed increased agreement with gender equality concepts) 67 ANNEX 8: EVALUATION PLAN 68 USAID/El Salvador Final Performance Evaluation of the Higher Education for Economic Growth Project Evaluation Plan January 30, 2018 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or of the United States government. This publication was produced for review by the United States Agency for International Development. It was prepared by Mendez England and Associates for the USAID Monitoring and Evaluation Activity, IDIQ No. AID-OAA-I-15-00024, Task Order No. AID-519-TO-16-00002. Recommended citation: USAID El Salvador Monitoring and Evaluation Services Task Order. 2018. Evaluation Plan Final Performance Evaluation of the Higher Education for Economic Growth project. Mendez England and Associates. Mendez England and Associates Contact: Randal Thompson Chief of Party Mendez England and Associates 70 CONTENTS EXECUTIVE SUMMARY................................................................................................................................................... i 1.0 Evaluation Purpose and Questions.................................................................................................. 1 1.1 Evaluation Purpose .............................................................................................................................. 1 1.2 Evaluation Questions .......................................................................................................................... 1 2.0 PROJECT BACKGROUND............................................................................................................. 2 3.0 EVALUATION METHODOLOGY................................................................................................ 3 3.1 Approach ............................................................................................................................................... 3 3.2 Literature............................................................................................................................................... 3 3.3 Interviews.............................................................................................................................................. 3 3.4 Site Visits................................................................................................................................................ 4 3.5 Analysis................................................................................................................................................... 5 3.6 Limitations............................................................................................................................................. 5 4.0 FINDINGS AND CONCLUSIONS.............................................................................................. 6 4.1 EQ 1: What have been the most significant intended and unintended results achieved to date?........................................................................................................................................................ 6 4.1.1........What have been the main internal and external factors that have influenced the achievement or non-achievement of the Activity’s expected results as planned?............... 6 4.2 EQ 2: To what extent has the demand-driven cluster model been an extensive space to build dialogue within private sector, academia, and GOES in order to respond to private sector needs? Are the four clusters already sustainable? ....................................................... 11 4.2.1..To what extent has the cluster model been attractive to engage private sector and obtain their commitment through leverage?............................................................................... 12 4.3 EQ 3: What changes have been made in the GoeS and hEIs in terms of their institutional capacity to continue introducing and/or supporting educational reforms and academic programs in the future? .................................................................................................................... 17 4.4 EQ 4: How were gender, female empowerment, and social inclusion integrated in implementation?................................................................................................................................. 20 4.4.1...To what extent is the Activity influencing changes in gender participation in higher education?............................................................................................................................................ 21 4.5 EQ 5: What approaches and results by the HEA have the potential to exist after USAID funding ends? which ones should continue to be supported?................................................. 23 5.0 RECOMMENDATIONS................................................................................................................... 26 5.1 Recommendation 1 ........................................................................................................................... 26 5.2 Recommendation 2 ........................................................................................................................... 27 5.3 Recommendation 3 ........................................................................................................................... 28 5.4 Recommendation 4 ........................................................................................................................... 28 3.5 Recommendation 5 ........................................................................................................................... 29 ANNEXES ......................................................................................................................................................................... 30 Annex 1: Statement of Work.......................................................................................................................... 31 Annex 2: Team Conflict of Interest Disclosures........................................................................................ 43 Annex 3: Protocols/Instruments..................................................................................................................... 49 Annex 4: Documents Consulted .................................................................................................................... 56 Annex 5: Schedule of Key Informant Interviews......................................................................................... 58 Annex 6: Supplemental Analysis for Question 3......................................................................................... 60 Annex 7: Illustrative Examples for Gender Indicators............................................................................... 62 Annex 8: Evaluation Plan................................................................................................................................... 67 72 ACRONYMS ASER Asociación Salvadoreña de Energía Renovable ASI Asociación Salvadoreña de Industriales CAMAGRO Cámara Agropecuaria y Agro-industrial de El Salvador CASATIC Cámara Salvadoreña de Tecnologías de la Información y las Comunicaciones CDCS Country Development Cooperation Strategy CEN U.S. Government Strategy for Engagement in Central America DNES Dirección Nacional de Educación Superior FEDISAL Fundación para la Educación Integral Salvadoreña FGDs Focus Group Discussions GOES Gobierno de El Salvador HEA Higher Education Activity IPs Implementing Partners IR Intermediate Result KIIs Key Informant Interviews MEL Monitoring, Evaluation and Learning RTI Research Triangle Institute SOW Scope of Work TBD To Be Determined UCA Universidad Centroamericana “José Simeón Cañas” UDB Universidad Don Bosco UES Universidad de El Salvador UFG Universidad Francisco Gavidia UNICAES Universidad Católica de El Salvador UNIVO Universidad de Oriente USAID United State Agency For International Development 73 EXECUTIVE SUMMARY The purpose of the evaluation is to have inputs to improve the Higher Education Activity (HEA) and also to have inputs for future USAID programming in the higher education sector. The evaluation will provide empirical evidence to support learning and continuous improvement in USAID’s work in this activity and future ones. Evaluation results will be used for reporting purposes to stakeholders such as the Government of El Salvador (GOES), private sector representatives, and academia. It consists of five research questions. 1. What have been the most significant intended and unintended results achieved to date? 1.1 What have been the main internal and external factors that have influenced the achievement or non-achievement of the activity’s expected results as planned? 2. To what extent has the demand-driven model (clusters model) been an effective space to build dialogue within private sector, academia, and GOES in order to respond to private sector needs and priorities? Are the four clusters already sustainable? 2.1 To what extent has the cluster model been attractive to engage private sector and obtain their commitment through leverage? 3. What changes have been made in HEIs in terms of their institutional capacity to continue introducing and/or supporting educational reforms and/or academic programs in the future? What changes have been made in the GOES in terms of their institutional capacity to continue introducing and/or supporting educational reforms and/or academic programs in the future? 4. How were gender equality, female empowerment, and social inclusion integrated in the implementation of activity interventions? 4.1 To what extent is the activity influencing changes in gender participation in higher education? 5. What approaches utilized and results achieved by the Higher Education for Economic Growth activity have the potential to continue to exist after USAID’s funding ends? Which ones should continue to be supported? The full plan details each research design, method, and analysis proposed. First, we will start with a thorough review of the literature and relevant documents. Then we will analyze secondary data and performance indicator data (provided by the Activity). In addition, we will conduct substantive fieldwork. This includes conducting KIIs with stakeholders, including USAID, the Higher Education Activity, Implementing Partners (IPs), the Government of El Salvador, and Universities. In the university field visits, we will also conduct FGDs with faculty and staff, as well as participate in events and directly observe/collect field notes. We will use electronic data collection methods to ensure rigor and rapid processing/cleaning to allow time for analysis. Software include Dedoose and CommCare. A debrief will be held with USAID prior to departure and there will be a draft produced within a month. The draft will be organized by research question and contain database findings, conclusions and actionable recommendations. Upon receipt of feedback and a possible virtual conversation, the team will submit the final draft on or about April 30, 2018. 74 BACKGROUND DESCRIPTION AND LOGIC Within USAID’s Country Development Cooperation Strategy (CDCS) 2015-2017, the Higher Education for Economic Growth project contributes to Development Objective 2 “Economic Growth Opportunities in Tradables Expanded.” Specifically, it contributes to Intermediate Result (IR) 2.2 “Productivity of Targeted Businesses Increased” and, jointly with the USAID Bridges to Employment activity, Young Entrepreneurs activity, and USAID Economic Competitiveness activity to Sub-IR 2.2.1 “Higher Education and Workforce Competencies Strengthened.” The Higher Education for Economic Growth project contributes to the achievement of Objective 1 of the U.S. Government Strategy for Engagement in Central America (CEN strategy) of “Prosperity and Regional Integration” (Education and Workforce Development) and the second strategic line of action in the Plan of the Alliance for Prosperity in the Northern Triangle40 of “Developing Opportunities for Our People” (Building human capital). The activity contributes to the Goal 2 of the USAID Education Strategy “Improved ability of tertiary and workforce development programs to generate workforce skills relevant to a country’s development goals,” and is aligned with the line of action in GOES’ Five-Year Plan of “Strengthening the coverage and quality of higher education, articulating it with other levels of the education system and fostering research and knowledge generation.” The development hypothesis of the Higher Education for Economic Growth project states that “If El Salvador’s Higher Education system is strengthened and aligned with private productive priority sector needs, then competitiveness and productivity will improve, contributing to long-term, broad-based economic growth.” The development hypothesis assumes that if the higher education sector of El Salvador improves the capacity of faculty staff, aligns curricula and research with labor market demand, enables key educational institutional and systemic reforms, and improves collaboration among local and international higher education institutes, public and private sectors and donors, then the relevance and quality of the higher education system’s response to priority sector needs will be increased, productivity improved and long￾term economic growth enhanced. The Higher Education for Economic Growth project builds partnerships between industry sectors and local HEIs to develop demand-driven educational programs and research to form highly qualified professionals and contribute to industry growth and productivity to help stimulate the economy and social development. The project strengthens the higher education system to respond to the country’s productive sector needs by improving relevance and quality of curricula and applied research, focusing on academic programs that respond to the needs of the labor market and provide faculty and professionals with opportunities to advance their industry knowledge. The Higher Education and Economic Growth project is implementing an industry demand-driven education model through four industry-higher education clusters in four sectors: 1. Information and Communications Technology; 2. Energy and energy efficiency; 3. Light manufacturing; and 4. Agro-industry and food processing. 40 http://idbdocs.iadb.org/wsdocs/getdocument.aspx?docnum=39224238 75 Each cluster includes the private sector, an anchor university, and associate HEIs. It also includes participation from the government. The collaboration between industry and HEIs will help create and update curricula that articulate skills requirements defined by industry for high-demand careers in growing sectors, Industry Advisory Committees will be created to strengthen the link between educators and business leaders and to share information on the job market, curriculum planning, internships and practical training. USAID/El Salvador also helps develop the capacity of HEIs perform applied research, thus meeting the technical and technology-related challenges of Salvadoran industry. It offers scholarships to Salvadoran faculty and researchers for study at innovative Professional Science Master’s degree programs. PURPOSE The purpose of the evaluation is to have inputs to improve the Higher Education Activity (HEA) and also to have inputs for future USAID programming in the higher education sector. Specifically, the purpose of the Final Performance Evaluation of the USAID Higher Education for Economic Growth project is to inform USAID of: a) project achievements and challenges to date, to focus implementation to meet or surpass contract objectives and assure sustainability; b) main approaches and activities supported by the project that have been valuable to achieve the project goal; c) based on lessons learned, what worked well and what did not; and d) provide recommendations that will serve as input for the design of any future USAID Higher Education for Economic Growth project. The evaluation will provide empirical evidence to support learning and continuous improvement in USAID’s work in this activity and future ones. Evaluation results will be used for reporting purposes to stakeholders such as the Government of El Salvador (GOES), private sector representatives, and academia. EVALUATION QUESTIONS USAID identified five key evaluation questions, in priority order. These are listed below, along with suggested sub-questions to clarify the intent of the key questions. The evaluation will focus on the key questions, both in the design of instruments and analysis tools, and in the overall findings, conclusions, and recommendations of the evaluation report. 1. What have been the most significant intended and unintended results achieved to date? 1.1 What have been the main internal and external factors that have influenced the achievement or non-achievement of the activity’s expected results as planned? 2. To what extent has the demand-driven model (clusters model) been an effective space to build dialogue within private sector, academia, and GOES in order to respond to private sector needs and priorities? Are the four clusters already sustainable? 2.1 To what extent has the cluster model been attractive to engage private sector and obtain their commitment through leverage? 3. What changes have been made in HEIs in terms of their institutional capacity to continue introducing and/or supporting educational reforms and/or academic programs in the future? What changes have been made in the GOES in terms of their institutional capacity to continue introducing and/or supporting educational reforms and/or academic programs in the future? 4. How were gender equality, female empowerment, and social inclusion integrated in the implementation of activity interventions? 4.1 To what extent is the activity influencing changes in gender participation in higher education? 76 5. What approaches utilized and results achieved by the Higher Education for Economic Growth activity have the potential to continue to exist after USAID’s funding ends? Which ones should continue to be supported? DESIGN, METHODS, ANALYSIS, AND ORGANIZATION The Evaluation Team will use a mixed-methods strategy to conduct the HE evaluation. This will include a review of relevant documents, analysis of HE’s M&E data and performance indicators, key informant interviews (KIIs) and surveys for faculty/staff, Focus Group Discussions, and direct observation in activity sites. Please see Annex I for an Evaluation Design Matrix. APPROACH The Evaluation Team will draw on utilization-focused methodologies to ensure that the information generated by the evaluation is useful to USAID. The team used the initial in-brief meeting call to confirm USAID/El Salvador’s goals and objectives and the type of information and insights that will be most useful to USAID’s decision-making. The Evaluation Team will also explore with USAID/El Salvador how the Mission will use the results of this evaluation in the current HE activity and in future programming. GENDER-SENSITIVE PERSPECTIVE Consistent with USAID’s evaluation policy and recognizing that effects of integration and the success of the activity might vary across gender, the Evaluation Team will apply a gender perspective to the entire evaluation process. Starting with the background document review, data related to gender-based variances in outcomes will be examined, and activity documents will be reviewed with a gender-sensitive lens, in order to inform a better understanding of gender dynamics in the implementation environment. The Evaluation Team will include both female and male researchers, and a gender balance among respondents will be sought during the recruitment phase, as respondents are available. During the data collection process, the Evaluation Team will endeavor to ensure that a female evaluator is present during interviews with female respondents, and some focus groups will be segregated by gender in order to encourage active participation by all respondents, and to limit the potential for one gender to dominate the conversation at the expense of another. During the data analysis phase, all participant-level data (both qualitative and quantitative) will be disaggregated and reported by sex. This analysis of gender aspects of the HE intervention and discussion of findings, and, where appropriate, findings and conclusions, will be included in the final evaluation reporting (both written and oral). DATA COLLECTION METHODS Literature Review During the initial phase of the evaluation, the team will conduct a thorough review of background documents related to higher education and its implementing environment. These documents will provide the team a deeper understanding of the activity’s operations to date. This will help the team finalize the evaluation design and data collection tools, and will inform the overall activity assessment process, including developing findings, conclusions, and recommendations for future programming. These documents will include the following: • Country Development Cooperation Strategy 2013-2017 • Higher Education for Economic Growth contract and amendments 77 • Higher Education for Economic Growth Monitoring and Evaluation Plan • Higher Education for Economic Growth annual work plans • Higher Education for Economic Growth annual and quarterly progress reports • Higher Education for Economic Growth Participant Training Reports • Higher Education for Economic Growth Grants Manual • Higher Education for Economic Growth Gender Analysis • Higher Education for Economic Growth Clusters Profile Reports • U.S. Government Strategy for Engagement in Central America • Plan of Alliance for Prosperity in the Northern Triangle • USAID Education Strategy 2012 – 2016 • El Salvador Higher Education Law • Policy for Higher Education Proposal prepared with project support. • Other documents related to higher education in El Salvador (internal evaluation, labor market study, cluster document, and sustainability document) Review of Performance Indicators and Data Higher Education for Economic Growth has an activity-specific Monitoring and Evaluation Plan and has collected data on a number of standard and custom indicators during activity implementation. The monitoring data collected will provide one source of data on progress toward objectives and outcomes. The Evaluation Team may use monitoring data on performance indicators as part of the evaluation analysis and should report on it in the Final Report as much as it relates to the evaluation questions stated above and satisfies relevant data quality standards. Context data, such as the World Competitiveness Index’s subcomponent of Higher Education and Training, should be analyzed and included to the maximum extent possible when answering the evaluation questions. KIIs and Group Interviews The Evaluation Team proposes to conduct approximately 25-30+ KIIs/group interviews with USAID/El Salvador, RTI International Staff and Higher Education for Economic Growth activity partners, the Government of El Salvador (GOES), the private sector, and participating higher education institutes. Data collection protocols that address the evaluation questions will guide the interviews but will be semi-structured (identifying probes to follow up on information related to key questions) to allow flexibility in the discussion and a natural flow to the conversation, should the Evaluation Team identify a point of particular interest to the research questions. In addition, whether it is a group interview or individual interview each participant will complete a brief survey in the case of faculty/staff. Group interviews will be conducted during field visits with university faculty and staff. Group interviews will include between 5-7 participants each. Data will be registered electronically. A list of participant categories is included below, along with the total sample sought from each stakeholder group. Number of Informants by Category Type of Informant Total # of KIIs targeted USAID/ El Salvador Contracting Officer’s Representative 1 Bilateral Team Leader 1 Economic Growth Office Director 1 Regional Program Office Representative 1+ Subtotal 4 RTI International Staff 78 Type of Informant Total # of KIIs targeted Chief of Party 1 Deputy Chief of Party 1 Monitoring and Evaluation Specialist 1 Human Capital Specialist 1 Productive Sector, Specialist 1 HE Systems Specialists 1 Subtotal 6 HE Partners Rutgers University 1 World Learning 1 FEDISAL 1 Subtotal 3 Government Ministry of Education 1 DNES 1 Ministry of Economy 1 Subtotal 4 Private Sector Salvadoran Industrial Associations (ASI) 1 Salvadoran Agriculture and Agro-industrial Chamber (CAMAGRO) 1 Salvadoran Information and Communications and Technology Chamber (CASATIC) 1 Salvadoran Association for Renewable Energy (SARE) 1 Subtotal 4 Universities Jose Simeon Canas Central American University (UCA) 2 Catholic University of El Salvador (UNICAES) 2 Don Bosco University (UDB) 2 Francisco Gavidia University (UFG) 2 National University of El Salvador (UES) 2 Eastern University (UNIVO) 2 Subtotal (KIIs) 6 Subtotal FGDs 6 Site Visits and Direct Observations The Evaluation Team, in consultation with USAID/El Salvador and RTI International, will select relevant site visits based on a sampling plan developed for the Evaluation Plan and included in the Final Report. At a minimum, the Evaluation Team should expect to visit the six campuses of the beneficiary higher education institutes identified below. The above mentioned KII and FGDs will be conducted in the sites. In addition, the Evaluation Team may attend events hosted or sponsored by the Higher Education for Economic Growth project during the field work period of the evaluation to conduct direct observation. The Evaluation Team will record field notes as data from these events as inputs for the report. SITE SELECTION To capture the depth of intervention effectiveness within the given data collection timeframe and team size, we will focus on visiting HE institutes in 4 of the 50 GOES priority municipalities. The Evaluation Team will visit six university activity sites in relevant areas to understand implementation trends in different locations. See table below. 79 HE Institute Municipality UCA San Salvador UES San Salvador UFG San Salvador UDB Soyapango (metropolitan area of San Salvador) UNICAES Santa Ana (64 kilometers northwest of San Salvador) UNIVO San Miguel (138 kilometers east of San Salvador) Total 6 Participants for FGDs/group interviews will be chosen by the point of contact for the university. This is a limitation because of the tendency for representatives to choose participants who will speak in favor of the HE Activity. Nevertheless, we will also ask for information on challenges and areas for improvement which will provide more well-rounded data as a whole on the experience. We will request an equal number of female/male participants. ANALYSIS Quantitative Data Analysis of quantitative data from the brief survey will be done in Excel. We will use tablets to collect survey data using a free open source software—CommCare. The data will be categorical, scale, and/or numeric therefore it will easily be downloaded into Excel and analyzed in Excel. We will report quantitative outcomes using tables, charts, and graphs in the final evaluation report. To the extent that performance indicator data is available and relevant, we will analyze it using quantitative techniques such as frequencies and percentages. Qualitative Data We will code and analyze the qualitative data gathered through the KIIs and FGDs using systematic qualitative methods. We will develop a coding system to identify and respond to information sought in the key evaluation questions and will orient each team member to the coding process to increase inter￾rater reliability. In addition, we will code for demographic information, such as gender, which we will also consider during the analysis process and include in the evaluation report. Qualitative data will be uploaded and coded in Dedoose (an open source free qualitative data coding software). This will also allow for excerpts to be downloaded in Excel and for frequencies and percentages of codes to be generated for figures. LIMITATIONS The Evaluation Team anticipates a series of limitations in undertaking this evaluation, as summarized below. Limitation Result Mitigation Strategy Team member Oscar Picardo has participated in the HEA via a sub grant (UFG for ICTs). This could influence objectivity. We have two other team members who will validate results and review Oscar Picardo’s work. Advising universities, we will come for field visits. Participants could change their behaviors given we are coming. Inform we are coming but not share instruments, use probing techniques to get at a more complete picture. University representatives choose participants for FGDs. Selection bias—representatives may choose participants who will speak highly of the activity. To mitigate this risk, the Evaluation Team will work closely with HE in respondent selection, using participant lists wherever possible. 80 TIMELINE The Evaluation Team will begin with a review of activity documents including key programming and implementation documents, progress reports, quantitative data, research studies, and surveys. The Evaluation Team will utilize the desk review and planning meeting period to refine data collection methods and tools and to clarify logistical and administrative procedures. The team participated in an initial team-planning/kick off meeting with USAID/El Salvador on January 16, 2018. FIELDWORK The Evaluation Team will begin fieldwork in San Salvador on February 12, 2018. The team will interview USAID/El Salvador, the HEA staff, HEA partners, GOES, and private sector between February 13-20, and site visits will be conducted between February 21-28. However, we will be flexible if there are university events which we should attend during our time in country. The Team Leader and Higher Education Specialist(s) will work independently. ILLUSTRATIVE FIELD WORK SCHEDULE Day Activity Location Mon. Feb. 12 International travel Tues. Feb. 13 Presentation event To be determined (TBD) Wed. Feb. 14 KIIs with stakeholders TBD Thurs. Feb. 15 KIIs with stakeholders TBD Fri. Feb 16. KIIs with stakeholders TBD Sat. Feb. 17 Cleaning/generating Findings Sun. Feb. 18 DAY OFF Mon. Feb. 19 KIIs with stakeholders TBD Tues. Feb. 20 KIIs with stakeholders TBD Wed. Feb. 21* Site visits *mid-point check in with CORs proposed TBD Thurs. Feb. 22 Site visits TBD Fri. Feb. 23 Site visits TBD Sat. Feb. 24 Cleaning/generating Findings Sun. Feb. 25 DAY OFF Mon. Feb. 26 Site visits TBD Tues. Feb. 27 Site visits TBD Wed. Feb. 28 Debrief USAID Thurs. Team meeting MEL Activity 81 ILLUSTRATIVE FIELD WORK SCHEDULE Day Activity Location March 1 Fri. March 2 Team meeting MEL Activity DEBRIEF AND REPORT SUBMISSION The team will present initial findings during a debrief at USAID/El Salvador, shortly after the data collection has been completed (Thursday, March 1, 2018). The Evaluation Team will submit the draft evaluation report to USAID/El Salvador, incorporating feedback provided during the debrief, on Monday April 2, 2018. The team will be available to discuss the draft virtually. The report will answer all of the evaluation questions and clarify how each was answered. We request that USAID provide comments on the draft report by Friday April 13, 2018. The Evaluation Team will then revise the draft report to fully reflect USAID comments and suggestions, submitting the final draft by Monday April 30, 2018. This includes the MEL Activity quality control and editing process built in. These dates are flexible and subject to change as well as confirmation per final approval of the evaluation plan. The outline of the report is: 1. Abstract (250 words) 2. Executive summary (2-5 pages) 3. Background 2. Purpose and questions 3. Methodology 4. Each research question and its respective evidence-based findings and conclusions 5. In question five, embed the actionable recommendations—directed towards the project and those directed for future programming with USAID 6. Annexes—SOW, team, protocols, matrix, sources, schedules, additional tables and graphs not embedded in the report TEAM COMPOSITION The Evaluation Team will be composed of four roles: • Team Leader. Responsible for coordinating the activities of the Evaluation Team; has the authority to make programmatic decisions regarding the evaluation. She will serve as the main point of contact between USAID and the MEL Platform regarding this evaluation. The Team Leader will draft the final evaluation design, oversee and support the development of evaluation instruments, participate in data collection and analysis, integrate the findings of different team members, and coordinate preparation of the final report. This role will be filled by Megan Gavin. • HE Specialist-1. Support as needed especially with data collection. In addition, will conceptualize the evaluation instruments, sampling strategy, and data analysis methods per question for the evaluation design matrix, and supports overall analysis. This role will be filled by Ana Cristina Accioly. • HE Specialist-2. Support question 3 with data collection. In addition, will conceptualize the evaluation instruments, sampling strategy, and data analysis methods per question for the evaluation design matrix, and supports overall analysis. He will support analysis of quantitative data. This role will be filled by Oscar Picardo. 82 • Logistics Coordinator. Work with local partners to plan travel and schedule data collection, interviews, and assessment activities as required. Will support the Team Leader in all logistical and administrative aspects of the evaluation. This role will be filled by Katia Zepeda and Cindy Abarca. . 83 ANNEX I: EVALUATION DESIGN MATRIX 84 Evaluation Questions Data Sources/Analysis Method 1. What have been the most significant intended and unintended results achieved to date? 1.1 What have been the main internal and external factors that have influenced the achievement or non￾achievement of the activity’s expected results as planned? Data Sources: • Document Review of HEA progress/quarterly progress reports, MEL plan, and baseline, a recent RTI internal evaluation • KIIs with stakeholders-especially USAID, HEA, IPs, Universities, and brief survey • FGDs with faculty and staff Analysis Method: • FGD and KII data will be entered an online qualitative data analysis software (Dedoose). • Codes will be determined in consensus with the team based on literature and data. • Codes will be applied to data. • Each code will be analyzed. • Brief survey results will be collected with ComCare and analyzed in Excel. • Analyze/write by component. 2. To what extent has the demand￾driven model (clusters model) been an effective space to build dialogue within private sector, academia, and GOES in order to respond to private sector needs and priorities? Are the four clusters already sustainable? 2.1 To what extent has the cluster model been attractive to engage private sector and obtain their commitment through leverage? Data Sources: • Document review, especially and including the recent document on the cluster model, and a recent document on sustainability • KIIs with stakeholders-especially GOES, universities, private sector Analysis Method: • FGD and KII data will be collected electronically. • Codes will be determined in consensus with the team based on literature and data. • Codes will be applied to data. • Each code will be analyzed. 3. What changes have been made in HEIs in terms of their institutional capacity to continue introducing and/or supporting educational reforms and/or academic programs in the future? 3.1 What changes have been made in the GOES in terms of their institutional capacity to continue introducing and/or supporting educational reforms and/or academic programs in the future? Data Sources: • Document Review of HEA progress/quarterly reports, baseline assessment and special studies, HICD tool (USAID guideline), and project reporting • KIIs with stakeholders-especially GOES • Brief survey • FGDs with staff/faculty • Direct observation at university events Analysis Method: • KII and FGD data will be entered electronically. • Codes will be determined in consensus with the team based on literature and data. Preliminary codes include capacity, literacy, and numeracy. • Codes will be applied to data. • Each code will be analyzed. • Direct observation field notes will be collected. 85 Evaluation Questions Data Sources/Analysis Method 4. How were gender equality, female empowerment, and social inclusion integrated in the implementation of activity interventions? 4.1 To what extent is the activity influencing changes in gender participation in higher education? Data Sources: • Document Review of HEA progress/quarterly reports, baseline assessment and special studies-including the gender analysis • KIIs with stakeholders-especially USAID, IPs, HEA • FGDs w staff/ faculty Analysis Method: • KII and FGD data will be entered electronically. • Codes will be determined in consensus with the team based on literature and data. Preliminary codes include capacity, literacy, and numeracy. • Codes will be applied to data. • Each code will be analyzed. 5. What approaches utilized and results achieved by the Higher Education for Economic Growth activity have the potential to continue to exist after USAID’s funding ends? Which ones should continue to be supported? Data Sources: • Document Review of government plans, and RTI labor market document • KIIs with stakeholders-especially GOES and private sector (perspectives from USAID and HEA) Analysis Method: • KII and data will be entered electronically. • Codes will be determined in consensus with the team based on literature and data. Preliminary codes include capacity, literacy, and numeracy. • Codes will be applied to data. • Each code will be analyzed. 86 ANNEX II: GANTT CHART AND FIELDWORK SCHEDULE 87 Fieldwork Schedule Feb 2018 Sunday Monday Tuesday Wednesday Thursday Friday Saturday 28 29 30 31 1 2 3 4 5 6 7 8 9 10 11 12 Travel 13 Team meeting 14 Presentation event 15 KIIs 16 KIIs 17 Cleaning 18 19 KIIs 20 KIIs 21 Site visits 22 Site visits 23 Site visits 24 Cleaning 25 26 Site visits 27 Site visits 28 Debrief 1 Team meeting 2 Team meeting 3 Subject to change depending on university events 88 Final Performance Evaluation of the Higher Education for Economic Growth Project Preliminary Plan Project Timeline - Proposed Start Date January 15, 2017 PROJUST Project W1 W2 W3 W5 W6 W7 W8 W9 W10 W11 W12 W13 W14 W15 W16 Desk review and submission of the Work Plan Editing process (HQ) (3 days) Review Inception Report by USAID Field work preparation Data collection Data Analysis, Report Writing and Submission of Draft Report Editing process (HQ) and Submission of Draft Report Review Draft Report (USAID) Finalization and Submission of Final Report Editing process (HQ) and Submission of Draft Report Final Report Presentation (TL) Note 1: In orange, editing days have been added. This increases the process from 13 weeks to 16 weeks. Note 2: Five days for field work preparation has been added. Needed to set up meetings and other arrangements. 89 ANNEX 9: STATEMENTS OF DIFFERENCE OF RTI AND RESPONSE FROM ET RTI’s statement of difference provides a number of project developments that describe some actions subsequent to those described in the evaluation, which suggest that they are already addressing some of the areas highlighted in the evaluation’s recommendations. Other RTI comments question findings that the ET reported that were based on interviews conducted during the evaluation. The ET believes that they are required to maintain these findings because these were valid based on information available during the evaluation period. These findings in no way disparage RTI. The evaluation by definition was undertaken at a certain point in time and reflected the findings based on interviews and documents reviewed at that time. Some examples are summarized below. The Evaluation Team acknowledges the gains made by RTI since the execution of the evaluation. The ET will stand by their recommendations which are just that, namely “recommendations.” They in no way compel USAID or RTI to implement them. With respect to Comment 1, the findings regarding delays were based on interviews with RTI staff during the evaluation. For example, project senior staff gave the impression that the delays in the scholarships was due to the fact that there were delays in approving the revised curricular programs. When the ET asked if the scholarships could have been provided for other programs of study, RTI staff explained that USAID preferred that the scholarships go towards the new revised and approved curricular programs. Regarding Comment 4, inclusion of government actors in the process has been part of the project from inception; nevertheless some key informant interviews stated very clearly that they thought Government should have been involved more closely. Specifically, a senior government official noted that they wished to have been more involved in the process. At the kick off evaluation event, participating anchor universities noted that they had not seen nor participated in the elaboration of the policy document. However, the ET understands that they now have been included. The ET would specifically like to acknowledge the gains made with regard to Comment 2 on clusters. The ET agrees that the clusters are not for research. Nevertheless the ET believes that it is commendable that the HEA has hosted events to share knowledge related to the positive work of the clusters. Furthermore, the ET congratulates the HEA on the events held in March of 2018, following the fieldwork for the evaluation which culminated in February 2018. In addition, the ET would like to acknowledge the gains with regard to Comment 3: gender indicators. The ET is pleased that now there is discussion regarding incorporating gender indicators into future USAID programing. 1. Comments related to delays in academic program approvals. Page ii: “There were delays in the approval on the part of the Government of El Salvador (GOES) of the new fields; however, this has improved within the 90 days from submitting the revised curriculum (see EQ 3).” 90 Page iv “10. The project was not fully conscious of the day to day challenges of GOES, which created unforeseen delays.” And related comments, “However, the bureaucratic process to review the curriculum was lengthy and delayed approval of the reformed curriculum. Legally, the approval is supposed to happen within 90 days. However, according to KIIs with RTI project staff, the government would send back a letter within the 90 days (often near the deadline) requesting more information or modifications. This therefore, extended the process.” Answer: The project does not register any delays in approvals by the Salvadoran National Superior Education Directorate (DNES) of the Government of El Salvador. As evidence, you can observe the design – submission / response – process for the Universidad Francisco Gavidia for the TIC academic programs. The Annexed I “Desarrollo Curricular de Tecnologías de la Información y Comunicación. Elaboración de planes de estudio y de implementación para ser aprobados por el MINED” highlights the coordinated work with the Ministry of Education and the priority given to a fast approval: Annexed I. Page 23. “La DNES/MINED se comprometió desde el mes de febrero del presente año, a reconocer la relevancia del esfuerzo curricular desarrollado en el marco del proyecto y buscar o aceptar opciones, para que, respetando la normativa, se garantizara el proceso de revisión de los planes de estudio y de implementación de manera expedita.” Actually, foreseeing the potential delays, the technical coordination of the project proposed to the DNES/MINED to temporarily contract a professional dedicated (Page 23) exclusively to check and validate the programs. This professional was selected by the DNES/MINED, and assigned to the HEP academic programs within the ministry. The Project created a special strategy to solve the bureaucratic and administrative potential delays. Moreover, you can observe the brief period of time in which the DNES/MINED responded to the first submission of the programs and then, the final approval. Page 23. “Las fechas de la primera entrega realizada a la DNES/MINED fueron el 22 y 25 de mayo del presente año. En la primera fecha se entregó el primer bloque de documentos, pertenecientes a UNICAES, ITCA/FEPADE y UNIVO y en la segunda fecha, se entregó el segundo bloque referidos a UFG, UGB y UTEC. Ver anexo 2. Cartas de primera entrega a la DNES/MINED, de los planes de estudio y de implementación. La devolución de los documentos con las respectivas observaciones por parte de la DNES/MINED, los recibió la coordinación técnica el 22 de junio y el segundo bloque el 4 de julio del presente año. Ver anexo 3. Observaciones de la DNES/MINED a los planes de estudio y de implementación.” First Submission to the DNES/MINED by the HEIs requesting approval of the new academic programs. 91 Higher Education Institution Academic Program Date Submission to DNES Date Response from the DNES No. of days UNICAES Ingeniería en Desarrollo de Software 22 May 2017 22 June 2017 24 días hábiles ITCA/FEPADE Ingeniería en Desarrollo de Software 22 May 2017 22 June 2017 24 días hábiles UNIVO Ingeniería en Desarrollo de Software 22 May 2017 22 June 2017 24 días hábiles UFG Ingeniería en Diseño y Desarrollo de Videojuegos 25 May 2017 22 June 2017 21 días hábiles UTEC Ingeniería en Gestión de Base de Datos 25 May 2017 11 July 2017 34 días hábiles UGB Ingeniería en Manejo y Gestión de Bases de Datos 25 May 2017 04 July 2017 29 días hábiles In the first response from the DNES/MINED, the HEIs received recommendations and adjustments they had to apply to the programs. The second submission included all the DNES recommendations in order to faster the final approval: Second Submission to the DNES/MINED by the HEIs requesting approval of the new academic programs. Higher Education Institution Program Date Submission to DNES Date Response from the DNES No. of days ITCA/FEPADE Ingeniería en Desarrollo de Software 30 June 2017 19 July 2017 13 días hábiles UFG Ingeniería en Diseño y Desarrollo de Videojuegos 30 June 2017 20 July 2017 19 días hábiles 92 UTEC Ingeniería en Gestión de Base de Datos 21 July 2017 27 July 2017 5 días hábiles UNIVO Ingeniería en Desarrollo de Software 30 June 2017 19 July 2017 14 días hábiles As an example, we observe that ITCA/FEPADE took 37 working days to approve Ingeniería en Desarrollo de Software, UNIVO 38 working days, UFG 40 working days, UTEC 39 working days. This strategy was highly efficient to avoid delays and distances with the MINED. At the same time, as a part of the Project, a Pilot has been implemented within the DNES to improve and better the checking and approval process of new academic programs. As you may see in the annexed II “Avance al 07 de Febrero 2018. Pilotaje Revision Electronica de Planes de Estudio” a big effort has been putted to improve and dynamize this type of process. The best results obtained in this Pilot are: The DEA (Departamento de Estudios Académicos) has achieved a level of delay in the checking process equal to 0. (Page 4). It has been employed permanent staff with the Project funds; It exists an excellent communication and coordination level among Higher Education Institutions and the DEA; and the resolution processes has been speeded. II. Comments related to Clusters and their responsibility with applied research products. Page iii. “Clusters are behind schedule to produce results such as research products but the lack of timely implementation, support on publication writing and knowledge sharing could compromise the collaboration momentum.” Answer: Clusters do not have any direct responsibility with the schedule and times of the research processes. As it can be seen in the annex III, page 4, Cluster´s goals related research are: Industry HEI Short term goals Identifying the immediate needs such as areas in which certification is needed, topics Identify areas of applied research Provide seminars to faculty and senior students regarding the latest development in their respective areas Establish functional working groups and bring together university-industry members Revise the curriculum to meet industrial needs and international standards (moving from five to four-year programs) Sponsor faculty for higher education in critically needed fields Create an atmosphere to encourage faculty to conduct applied research such as reduction of teaching load Find an avenue to train the graduates for successful completion of certificate programs Objectives Enhance economic competitiveness of El Develop a qualified workforce capable of 93 Salvador Support the development of applied research programs in collaboration with universities having an impact on the industry clusters Continually improve pedagogy, research, development and technology, and English skills of faculty Develop strong applied research, development and innovation base aligned with industry clusters Build HEI institutional capacity and facilitate system effectiveness to respond to the needs of faculty and the industry clusters Clusters are the platform to propose, share and validate information after the results are produced by the research team, in collaboration with their industry partner. Cross-fertilization within and across clusters began in March, with the dissemination of preliminary findings at the mini conferences, where posters were presented on each applied research project. In annex IV it can be found the Talking Points used to cross-fertilize and disseminate the preliminary findings of the applied researches. In March 22, it was done the presentation for the Light Manufacturing Cluster, in March 21 the Agro-Industry and Food Processing, in March 21 TIC and in March 20 the Energy and Energetic Efficiency. Another evidence of the knowledge sharing taking advantage of the collaboration momentum is the following list of tweeters used to disseminate the information: 1. https://twitter.com/Edusuperior_sv - It can be found the record of events related to disseminate HEPs results. Research Preliminary Findings Event: 2. https://twitter.com/Edusuperior_sv/status/976843944762970113 PET and short run molds. 3. https://twitter.com/unicaes_sv/status/976498405001383941 Ricotta Prototype. 4. https://twitter.com/unicaes_sv/status/976496253595979777 Agro-Industrial Reduction alternative. 5. https://twitter.com/unicaes_sv/status/976494438666121216 Panela diversification proposal. 6. https://twitter.com/unicaes_sv/status/976492784772317186 Recovery of biological material of local Cacao Species. 7. https://twitter.com/unicaes_sv/status/976487708255293440 Coffee and related Phyto-genetic material 8. improvements. 9. https://twitter.com/unicaes_sv/status/976486068110405635 Improvements in cattle feeding. 10. https://twitter.com/unicaes_sv/status/976483431919050758 The importance of applied research. 11. https://twitter.com/Edusuperior_sv/status/976482451030728706 Agro-Industry and Food Processing. 12. https://twitter.com/unicaes_sv/status/976478244055732224 General Presentation. III. Comments related to the absence of a specific gender indicator. Page iv. There are no gender-specific indicators in the Monitoring, Evaluation, and Learning (MEL) Plan 94 Answer: This is already being discussed with USAID in order to include a gender indicator. IV. Comments related to the low inclusion of the Government of El Salvador in the Higher Education Policy. Page v. “Recommendation 4: USAID needs to include the government more in higher education policy development with USAID, and the process needs to be more open to include other government agencies, NGOs, etc. This is based on conclusions 10 and 11. Answer: Government institutions related with Higher Education has been involved since the beginning and other institutions related. As you can see in the annex VI “Acuerdo Compromiso. Pacto Multisectorial para la Construcción de la Política Nacional de Educación Superior” signed on April 8, 2016, the Agreement stablishes: Page 1 “En el Marco del Proyecto de la Agencia de los Estados Unidos para el Desarrollo Internacional (USAID) de Educación Superior para el Crecimiento Económico, nosotros representantes de la Dirección Nacional de Educación Superior y del Viceministerio de Ciencia y Tecnología del Ministerio de Educación (MINED); el Viceministro de Industria y Comercio del Ministerio de Economía, la Universidad Francisco Gavidia, la Universidad Don Bosco, La Universidad Católica de El Salvador, la Universidad Centroamericana José Simeón Cañas, la Universidad de El Salvador, El Consejo de Educación Superior, el Consejo de Asociaciones Profesionales de El Salvador, la Comisión de Acreditación de la Calidad de la Educación Superior, el Consejo Centroamericano de Acreditación, el Instituto Salvadoreño de Formación Profesional y de Research Triangle Institute (RTI), implementador del Proyecto de USAID de Educación Superior para el Crecimiento Económico. CONSIDERAMOS: … V. Que las partes han expresado voluntad y están dispuestas a apoyar este esfuerzo con la representación de delgados de alto nivel definidos para conformar cada uno de los comités de trabajo. … PRIMERO: FIRMA DEL PRESENTE ACUERDO Firmar el presente acuerdo con el objetivo de concretar y detallar los términos bajo los cuales se decide participar en el Pacto Multisectorial para la Creación de la Política Nacional de Educación Superior y apoyar el Memorando de Reconocimiento firmado en el evento público para tal fin, realizado el 8 de abril de 2016 en la ciudad de San Salvador, El Salvador. … TERCERO: PRINCIPIOS QUE REGIRAN LA CREACION DE LA POLITICA. La política será realizada en cumplimiento de los siguientes principios: d. Consenso: la política será construida con el consenso entre los actores, el cual ha de lograrse de manera armoniosa y razonada. 95 e. Inclusiva-participativa: La elaboración de la Política tomará en cuenta a todos los actores claves identificados a este momento y durante el proceso que tengan un rol en la educación superior del país. … CUARTO: CARACTERÍSTICAS DE LA POLITICA. La política en su versión final será materializada en un documento único y de carácter oficial, que articule la visión en consenso de todas las partes y que regirá el qué hacer de la educación superior en el país… SEXTO: METODOLOGÍA PARA LA ELABORACION DE LA POLITICA. Las Partes convienen trabajar con una metodología basada en comités de trabajo, donde nombrarán a representantes según se requiera en los cuatro comités a formar. a. Comité Estratégico: Dirigir la creación de la política y supervisar su ejecución para la rendición de cuentas según corresponda. b. Comité Técnico Coordinador: Ejecutar la iniciativa incorporando las directrices del Comité Estratégico de alto nivel cumpliendo con las metas requeridas asegurando la calidad de los productos. c. Comité Consultivo: Participar activamente en la elaboración de la política de acuerdo con los requerimientos del comité técnico aportando según su especialidad. d. Petit Comité: Coordinación integral de la ejecución de la iniciativa. Este equipo se constituye a través de un delegado de la Dirección Nacional de Educación Superior y un delegado del Proyecto de USAID de Educación Superior para el Crecimiento Económico.” In terms of the activities related to assure a broad participation of a public in general, to construct an inclusive Higher Education Policy, we list some of the most important events used to include and promote participation and as it can be seen in the annex VII, broad participation form HEIs, Government, Educational Institutes, Ministries, Private Sector, Independent consultants, etc.: Date Activity 21/03/2017 Focus Group DNES- Consulta Presencial- Eje temático: Gobernanza. 22/03/2017 Taller de Gobernanza de la Educación Superior. 27/03/2017 Reunión Comité Técnico Coordinado – Consulta Internacional. Eje. 27/03/2017 Reunión Comité Estratégico – Taller sobre funciones de la educación superior. 28/03/2017 Taller Vinculación de la Docencia, Investigación y Proyección Social. 29/03/2017 Internacionalización de la Educación Superior 1/06/2017 Presentación de Avances Política Nacional de Educación Superior GOES, through its National Superior Education Directorate (DNES), has lead the process in coordination with HEP. GOES has been always a central actor in the construction of the National Policy for Higher Education. In the annex VII you may find the following references: “Component 3: Institutional Capacity and Effectiveness Heightened Jaime Valdez, Congressman and President of the National Commission for Education and Culture of El Salvador’s Congress joined the Higher-Level Policy Dialogue Commission - One of the key objectives of the high-level policy dialogue is to bring together all higher education stakeholders, government and industry, to address system-level challenges in higher education. The agreement signed on April 8, 2016, by USAID, MINED and thirteen other relevant stakeholders in higher education, represents the commitment to create a higher education policy and its roadmap, for the first time in the history of El Salvador. 96 The Higher-Level Policy Dialogue Commission identified the need to include other stakeholders to support the process. Jaime Valdez, Congressman and President of the National Commission for Education and Culture in the Salvadoran Congress accepted to join the commission.