EVALUATION OF LAC HIGHER EDUCATION SCHOLARSHIPS PROGRAM FINAL REPORT February 2013 This publication was produced for review by the United States Agency for International Development. It is authored by Dr. Ray Chesterfield and Dr. William Dant for JBS International, Inc. under Task Order AID￾OAA-BC-10-00001 of the Global Evaluation and Monitoring II Blanket Purchase Agreement, GEM II BPA EDH-E-00-08-00003-00. Evaluation of LAC Higher Education Scholarships Program FINAL REPORT DISCLAIMER The views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development. Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page i ACKNOWLEDGEMENTS This evaluation was undertaken from April through October 2012 by Dr. Ray Chesterfield and Dr. William P. Dant, as well as a team of interviewers who carried out the evaluation in three countries: Mexico, Honduras and Guatemala. Our sincere thanks go to the Field Interview Team Leaders in each country: Dr. Melba Castillo, who helped facilitate both the pilot in Nicaragua and the interview team training in the three study countries; Mr. Mario Rivas, who organized the process in El Salvador; Professor Isabel Garcés del Valle, who led the team of interviewers in Guatemala; and Dr. Francisco Millán and Dr. Elia Bautista, who coordinated teams of interviewers in Mexico. We would like to thank all of the interviewers working in the field especially for their persistence in getting in touch with program applicants. Thanks are also due to the recipients and non-recipients who so kindly permitted us to interview them. We would like to thank university staff and faculty of Kirkwood Community College and Northcentral Technical College, as well the Center for Intercultural Education and Development of Georgetown University and its staff in each study country for their time spent describing and explaining the programs under study. We acknowledge the support of JBS staff, including Mr. Felipe Millán’s endeavors during the field work period, Dr. Marcia Odell’s significant editorial contributions and Ms. Maggie Rasnake’s valuable efforts in coding the survey data, editing the report, and seeing the work through to completion. We also would like to thank the Education Team members at USAID/LAC, particularly Dr. Barbara Knox-Seith, Dr. Sara Harkness, and Mr. Michael Lisman, whose input has improved this final product. Over the course of the evaluation, the team met and worked with many other people who provided enthusiastic support to our efforts and we wish to thank them, even if we cannot name them all here. Any errors in the report are the sole responsibility of the evaluation team. ABOUT THE AUTHORS Dr. Ray Chesterfield is an anthropologist, linguist and program evaluator whose work has focused on assessment of education programs, cross-cultural communication, and human resource development. Much of his over 35 years of professional work has been devoted to the creation and evaluation of programs that promote the formation of human resources in marginalized communities of the U.S. and Latin America. Chesterfield is a specialist on the integration of qualitative and quantitative data sets gathered in multisite program evaluations and has written extensively on this subject. He received an M.A. in Latin American Studies, and a Ph.D. in Comparative and International Education from U.C.L.A. Dr. William P. Dant is principal of WPD Global Solutions, LLC in Washington, DC. He has more than 30 years’ experience in international education and exchange program design, implementation and evaluation, including management of Fulbright Academic Exchange programs for the Department of State and national leadership of the Hubert H. Humphrey Fellowships for the Institute of International Education. Dant holds a Ph.D. in International Education Policy from the University of Maryland and a Masters of Arts in Teaching from the SIT Graduate Institute. Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page iii TABLE OF CONTENTS Acknowledgements............................................................................................................................................................ i About the Authors............................................................................................................................................................. i Table of Contents............................................................................................................................................................. iii List of Tables and Figures................................................................................................................................................ v Acronyms........................................................................................................................................................................... vii Glossary ............................................................................................................................................................................ viii Executive Summary .......................................................................................................................................................... ix Introduction .................................................................................................................................................................. ix Methodology.................................................................................................................................................................. x Principal Findings.......................................................................................................................................................... xi Conclusions ................................................................................................................................................................. xiii Directions for Future Programming...................................................................................................................... xiv I. Introduction..................................................................................................................................................................... 1 II. Background..................................................................................................................................................................... 1 A. Program Overview.................................................................................................................................................. 1 B. Program Objectives................................................................................................................................................. 3 C. Previous Evaluations............................................................................................................................................... 3 D. Study Questions...................................................................................................................................................... 4 III. Design ............................................................................................................................................................................. 5 A. Variables.................................................................................................................................................................... 5 B. Sample......................................................................................................................................................................... 6 C. Instruments............................................................................................................................................................... 7 D. Piloting and Fieldwork ........................................................................................................................................... 7 E. Data Analysis............................................................................................................................................................. 9 F. Limitations .................................................................................................................................................................. 9 IV. Findings......................................................................................................................................................................... 10 A. Programmatic Characteristics and Costs........................................................................................................ 10 Program Overview................................................................................................................................................ 10 Program Funding and Recipient Costs............................................................................................................. 11 Program Components.......................................................................................................................................... 12 Program Recruitment........................................................................................................................................... 13 1. Two-year Program........................................................................................................................................ 14 2. Six- and Twelve-month Programs............................................................................................................. 17 Evaluation of LAC Higher Education Scholarships Program Page iv Evaluation of LAC Higher Education Scholarships Program U.S. Host Institutions............................................................................................................................................ 17 Pre-program Preparation..................................................................................................................................... 19 Program Implementation On-campus............................................................................................................... 19 Post-program Preparation................................................................................................................................... 23 Budgeting and Cost Factors................................................................................................................................ 25 Program Elements Compared with Other Similar Sponsored Programs................................................ 27 Key Points................................................................................................................................................................ 28 B. Recipients................................................................................................................................................................. 29 Introduction ............................................................................................................................................................ 29 Change within Recipient Group......................................................................................................................... 29 1. Where do past scholarship recipients live and how do they identify ethnically?...................... 29 2. Are recipients employed at a higher rate than before CASS/SEED?............................................ 30 3. Are the credentials of past recipients recognized in the home country?................................... 33 4. Do past recipients have increased leadership and participation in their communities? .......... 34 5. Have CASS/SEED recipients’ opinions of the U.S. changed?.......................................................... 35 Comparative Impact.............................................................................................................................................. 37 1. Are CASS/SEED recipients more likely to live in rural areas than non-recipients?.................. 37 2. Are CASS/SEED recipients more likely to be employed than non-recipients?.......................... 38 3. Do CASS/SEED recipients have higher rates of leadership in their workplaces? ..................... 41 4. Are CASS/SEED recipients more likely to participate in community organizations?............... 42 5. Are scholarship recipients more likely to have continued their education?.............................. 44 6. Are recipients more likely to have international involvement or contacts?............................... 44 7. Do recipients and non-recipients have different opinions of the U.S. and the CASS/SEED program?............................................................................................................................................................. 46 8. Do recipients have different aspirations for the future than non-recipients?............................ 46 9. What do recipients and non-recipients think of their scholarship/application experience?... 46 C. Programmatic Impact........................................................................................................................................... 48 1. Employment and Leadership Success....................................................................................................... 48 2. Pathways to Success..................................................................................................................................... 51 Summary .................................................................................................................................................................. 54 V. Conclusions and Suggestions for Future Programming..................................................................................... 55 Conclusions .................................................................................................................................................................. 55 Considerations for Future Programming.............................................................................................................. 58 Bibliography....................................................................................................................................................................... 60 Appendix 1: Summary Table of the LAC Higher Education Scholarships since Inception............................. 63 Appendix 2: Illustrative Examples of Other Sponsored Scholarships & Learning Options........................... 73 Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page v Appendix 3: Economically Active Population for Sample Countries................................................................... 77 Appendix 4: Male and Hispanic Mean Success Rates.............................................................................................. 78 Appendix 5: Scope of Work ......................................................................................................................................... 79 LIST OF TABLES AND FIGURES Figures Figure 1: Sequential Logic Model of the CASS/SEED Program............................................................................... x Figure 2: SEED’s Pyramid for Recruitment, Screening and Determination of Application Numbers per Scholarship Slot................................................................................................................................................................ 16 Figure 3: Employment Rates of Recipients at Time of Application to the Two-Year Program and Currently ........................................................................................................................................................................... 30 Figure 4: Change in Employment Rate for Two-year Program Recipients by Ethnicity ................................. 31 Figure 5: Time to Find Employment after Return for Two-year Recipients, by Gender............................... 33 Figure 6: Recipients' Opinion of the U.S. before and after Scholarship.............................................................. 36 Figure 7: Percentage of Two-year Recipients and Non-recipients Employed by Year of Application........ 39 Figure 8: Activities of Two-year Candidates while Looking for a Job, by Group............................................. 43 Tables Table 1: Recipient/Non-recipient Sample by Year (n = 452) .................................................................................. 7 Table 2: Recipient/Non-recipient Reasons for Non-response ............................................................................... 9 Table 3: Scholarship Recipients by Program Type over the Period of Study.................................................... 11 Table 4: Costs and Cost Structure of CASS/SEED Compared with Other International Sponsored Scholarship Programs ..................................................................................................................................................... 12 Table 5: CASS/SEED Candidate Recruitment Compared with Other Programs............................................. 17 Table 6: U.S. Institutional Networks of CASS/SEED Compared to Other Programs.................................... 19 Table 7: Housing and English Language Proficiency of CASS/SEED Recipients Compared to Those of Other Programs............................................................................................................................................................... 21 Table 8: Exposure to U.S. Culture and Networking for CASS/SEED Recipients Compared to Other Programs............................................................................................................................................................................ 22 Table 9: Technical and Academic Training Approaches Compared to Other Programs............................... 23 Table 10: Community-focused Projects Component of CASS/SEED Compared to Other Programs....... 25 Table 11: Costs per Recipient and Proportion of Cost-sharing over the Period of the Study .................... 26 Table 12: Recipient Change in Residence and Ethnicity at Entry and Currently.............................................. 29 Table 13: Percentage of Employed Recipients in Two-year Program by Year of Entry ................................. 30 Evaluation of LAC Higher Education Scholarships Program Page vi Evaluation of LAC Higher Education Scholarships Program Table 14: Employment of Recipients in Two-year Program by Job Type and Year of Entry ........................ 32 Table 15: Recipient Employment by Location and Group..................................................................................... 32 Table 16: Two-year Recipient Employment by Area and Country ..................................................................... 32 Table 17: Recognition of Credentials: All Academic Programs............................................................................ 34 Table 18: Leadership Roles in the Workplace of CASS and SEED Scholars – All Academic Programs..... 35 Table 19: Community Involvement of CASS and SEED Scholars......................................................................... 35 Table 20: Current Residence by Group and Academic Program ........................................................................ 37 Table 21: Location of Employment by Group and Academic Program.............................................................. 38 Table 22: Current Employment by Academic Program and Group.................................................................... 38 Table 23: Type of Employment by Academic Program and Group..................................................................... 39 Table 24: Salary Range by Academic Program and Group .................................................................................... 40 Table 25: Comparison of Job Advancement by Recipients and Non-recipients (n = 449)............................ 40 Table 26: Key Possessions of Recipients and Non-recipients............................................................................... 41 Table 27: Supervisory Position in the Workplace by Academic Program and Group.................................... 41 Table 28: Leadership Role in the Workplace by Academic Program and Group............................................ 42 Table 29: Leadership Responsibilities by Academic Program and Group.......................................................... 42 Table 30: Community Participation by Type of Organization and Group......................................................... 43 Table 31: Continuing Education after Scholarship Program Experience............................................................ 44 Table 32: Membership in Ex-student Associations by Academic Program and Group .................................. 45 Table 33: Communication Partners by Academic Program and Group............................................................. 45 Table 34: Opinion of the United States by Recipient and Non-recipient.......................................................... 46 Table 35: Employment Aspirations in Five Years by Group.................................................................................. 46 Table 36: Comments on the Program by Group..................................................................................................... 47 Table 37: Employment and Leadership Success by Academic Program and Group (n = 452)..................... 48 Table 38: Female Employment and Leadership Success by Academic Program and Group ......................... 49 Table 39: Indigenous Employment and Leadership Success by Academic Program and Group................... 50 Table 40: Recipient Employment and Leadership Scores by Year....................................................................... 50 Table 41: Correlations of Selection Score with Success Variables...................................................................... 51 Table 42: Elements Identified by Recipients and Non-recipients as Contributing to Leadership Success . 51 Table 43: Principal Elements Identified by Two-year Recipients and Reasons for Importance.................... 52 Table 44: Most Important Elements of the Scholarship Experience as Identified by One-year Recipients and Reasons for Importance ......................................................................................................................................... 54 Evaluation of LAC Higher Education Scholarships Program Page vii Evaluation of LAC Higher Education Scholarships Program ACRONYMS APSP Andean Peace Scholarship Program CASP Central America Scholarship Program CASS Cooperative Association of States for Scholarships CCI Community College Initiative CIED Center for Intercultural Education and Development (Georgetown University) CLASP Caribbean and Latin American Scholarship Program GEM II Global Evaluation and Monitoring II HELP Haitian Education and Leadership Program IFP Ford International Fellowship Program JJ/WBGS Joint Japan/World Bank Graduate Scholarship Program LAC Latin America and the Caribbean NICRA Negotiated Indirect Cost Rate Agreement PTIIC Presidential Training Initiative for the Islands of the Caribbean RSD Office of Regional Sustainable Development (of the LAC Bureau) SEED Scholarships for Education and Economic Development TOEFL Test of English as a Foreign Language USAID United States Agency for International Development Evaluation of LAC Higher Education Scholarships Program Page viii Evaluation of LAC Higher Education Scholarships Program GLOSSARY Change agent An individual who plans and manages the implementation of new structures, procedures, or methods in a social situation. Community Action Plan (CAP) A community development project identified and developed by each scholar over the course of the scholarship, to be implemented in the scholar’s home community upon return. The importance of CAPs has been increased under the revised scholarship known as SEED. Employment success Earning a wage or salary in a skilled technical or supervisory/management position, in which competence is recognized through promotion or salary increase. For the purposes of the study, operationalized as an affirmative response on holding a job, holding a skilled technical or management position, having received a raise, having received a promotion or being a member of a professional organization Non-recipient (or finalist) An individual who meets all of the requirements for a CASS/SEED scholarship up to the last review by the community colleges and did not receive an award, or if awarded a scholarship, did not undertake it. Leadership The provision of information, knowledge and/or methods to realize a goal or objective Leadership success Holding a recognized position in one’s occupation that includes supervising others, or involvement in community action through holding an official decision-making position in a community organization. For the purposes of the study, operationalized as an affirmative response on holding a supervisory position, having a leadership role in the workplace, or having a leadership position in a community organization Management position A wage or salaried employment that requires decision-making about the use of resources to meet organizational goals p ≤ .01or p ≤ .05 p value is the statistical probability that a given finding occurred by chance alone in comparison with the known distribution of possible findings, considering the kinds of data, the technique of analysis, and the number of observations. The p values in this report are noted as decimals: p ≤ .01 means that the likelihood that the phenomena tested occurred by chance is one percent or less; p ≤ .05 means that the likelihood of chance is five percent or less. The lower the p value, the less likely that the finding was by chance alone and the more significant the finding. Recipient A CASS or SEED scholar who received and completed the programs under study Skilled/Technical occupation A wage or salaried position requiring a wide range of related activities with a minimum of direction or supervision, secured through targeted education, apprenticeship or on-the-job training Supervisory position Wage or salaried employment that requires training and oversight of personnel Support/Low Skilled occupation A wage earning or salaried position that requires a narrow range of skills, often acquired on the job, with little opportunity for independent judgment Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page ix EXECUTIVE SUMMARY INTRODUCTION The LAC Higher Education Scholarships Program In 1984 the National Bipartisan Commission to Central America and the Caribbean (the Kissinger Commission) found that the United States was not sufficiently responsive to the political and civil turmoil in that region. Congress appropriated funding that directed USAID to address this need through participant training of youth, and designated a partner, Georgetown University, through its Center for Intercultural Education and Development (CIED), to implement the LAC Higher Education Scholarship Program. In 1985, the first Central American, Andean, and Caribbean participant training scholarships were launched. The program has provided two-year technical training for economically disadvantaged high school graduates from underserved areas of designated countries across Central America and the Caribbean. These individuals take English language training, which allows them to take standard coursework in the United States along with U.S. peers. They also live with U.S. host families as part of their curriculum. From 1989 through 2008, the program was known as CASS, the Cooperative Association of States for Scholars. Since 2008, it has been known as SEED, Scholarships for Education and Economic Development. The scholarship program later added professional development training to rural-based, mid-level professionals. It provides those groups with short-term specialization and skills upgrading of six to twelve months’ duration. Such recipients generally receive most of their training in Spanish or French, and live in their own housing in the host U.S. community. A total of 9,191 scholarships have been awarded through the above programs over the past 29 years. Funding and in-kind resources supporting them totaled more than $450,000,0000, of which $320,820,029 was provided by USAID through cooperative agreements. In brief, the program seeks to: 1. Create a cadre of change agents and future leaders; 2. Provide recipients with skills and knowledge to participate actively and responsibly in a democratic society; 3. Strengthen capacity in countries participating in free trade agreements in a range of targeted technical fields; 4. Create a cadre of potential leaders who have been immersed in American culture and are sympathetic to the United States. Central to the CASP, CASS and SEED programs has been the consistent focus on providing training opportunities to socially and economically disadvantaged rural populations across 12 countries in the region. As USAID has redefined its development priorities for the region, several countries have ‘graduated’ from the CASS/SEED program. Thus the CASS and SEED programs currently serve seven countries in the LAC region. Evaluation of LAC Higher Education Scholarships Program Page x Evaluation of LAC Higher Education Scholarships Program About this Report This report summarizes the findings of an impact evaluation of the above scholarship programs, conducted under a Task Order from the LAC/RSD through the GEM II support contract. The study dealt with program implementation and effects. Implementation was studied by examining service delivery and program costs in relation to other scholarship programs offering similar services. Effects were examined by comparing scholarship recipients to non-recipients who were judged to be outstanding candidates for a scholarship, but were not chosen in the final selection process. This document is organized in relation to the two principal evaluation questions: 1. To what extent did the program provide value for investment? 2. Did the scholarships enable participating individuals to become leading change agents in their respective professions and communities? Figure 1, below, illustrates the program components in sequence, illustrating on the upper tier the inputs and outcomes of the scholarships in a logic model. Parallel to this sequence is the “black box” of the experiences of the non-recipient applicants, who never underwent any evaluation or impact follow-up before this evaluation. Figure 1: Sequential Logic Model of the CASS/SEED Program METHODOLOGY The evaluation took place from May to October 2012 and focused on the last seven years of complete program implementation (2004-2010) in three countries – El Salvador, Guatemala and Mexico. The study was designed as a hybrid performance evaluation that includes elements of an impact evaluation. It Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page xi compared program characteristics with those of other scholarship programs, and also measured changes in a) scholarship recipients at different points of time in their post-program careers and b) similar individuals (non-recipients) who did not have scholarships. Open-ended interviews with 238 recipients, 214 non-recipients and 25 program administrators were the principal sources of the data collected. Recipient interviews were carried out by experienced local interviewers; program administrator interviews were made by a senior evaluator on the team. The field interview data were complemented by site visits to two of the twelve implementing colleges that included interviews with program administrators and students, and to the CIED headquarters and its field offices in the three study countries, featuring similar interviews with the in-country administrators of the program. For comparative purposes, evaluators also reviewed the programs’ operating policies, budgets and implementation documents, comparing them with those of other similar scholarship programs in terms of service delivery and costs. PRINCIPAL FINDINGS 1. Principal Findings - Value for Investment The examination of the value of USAID’s investment in the CASS and SEED programs looked at resource efficiencies and effectiveness of implementation of the studied programs as conceived. It also compared and contrasted these elements with other donor-sponsored international scholarships that were similar to CASS/SEED in their format, audiences or intended impacts. • The recruitment and selection of CASS/SEED scholars is one of the most thorough to be found across sponsored international scholarship programs. The process successfully identifies the populations targeted by USAID. Costs for this component of the program are comparable to or lower than those of other similar sponsored scholarship programs. • The U.S. components of the CASS/SEED programs are carefully integrated to help recipients meet their own objectives, as well as those of the program. Again, the costs are comparable or lower than those of other similar programs. • CASS/SEED cost-share requirements for subcontractors are higher than average. Financial expectations regarding the financial contributions of host institutions, however, are in some cases disincentives for continued participation by host colleges. • Compared to other sponsored scholarship programs, CASS/SEED as it is currently structured is weak in its follow-up and support of recipients after they return home. This may contribute to the somewhat low completion of Community Action Plans, and suggests that USAID’s considerable investment in the recipients and their future impact on communities is not being as fully realized as possible. 2. Principal Findings – Recipient Outcomes The examination of the role of scholarship recipients as change agents was organized around the objectives of the program in terms of employment, leadership development, community participation and international involvement. Employment • Employment rates were higher for recipients than non-recipients (91% vs. 81%, a statistically significant difference). This was a result of differences in the two-year program, as Evaluation of LAC Higher Education Scholarships Program Page xii Evaluation of LAC Higher Education Scholarships Program those in professional development programs were fully employed in both groups. Significantly higher percentages of recipients were employed and in skilled and management positions than in the non-recipient group (70% to 50%). English language competency often provides gateway access to first employment opportunities for the two-year program recipients. • Compared with non-recipients, scholarship recipients had higher employment levels in more demanding jobs, as reflected in fewer individuals at a low salary level; a higher average number of promotions and raises; and a greater likelihood of owning computers with internet access. • For both recipients and non-recipients, employment rates are much higher than those of the overall age group of 20-29 year-olds. In the latter group, only about 33 percent of the total age group is economically active across the three countries. This confirms that the program is effective in its targeting of the best-suited candidates. • Female recipients were employed (89% vs. 77%) and held management positions (66% vs. 43%) in significantly higher percentages than non-recipient females. Indigenous recipients were less likely to be employed at time of application than Hispanic recipients (26% vs. 52%). They were found to be employed in the same percentage at the time of the study. They also held jobs at a significantly higher percentage than indigenous non￾recipients (94% vs. 81%). • Professional development recipients’ participation in the scholarship program has encouraged these mid-career recipients, primarily teachers, to continue to live and work in rural areas in greater percentages than similar non-recipients. There was no significant difference between recipients and non-recipients of the professional development program at application in terms of residence. Currently, a significantly greater percentage of recipients of professional development scholarships than non-recipients live (82% vs. 66%) and work (91% vs. 63%) in rural areas. • Relatively few of the scholarship recipients in any program were able to have the credentials earned from their scholarship experience recognized. Although 54 percent of the recipients have continued studies on return, only 25 percent obtained equivalency for courses taken in the U.S. when they returned to local universities. Leadership • A significantly higher percentage of recipients hold leadership roles in their workplace than non-recipients (73% vs. 54%, significant statistically). Recipients in the two￾year program, as well as female and indigenous recipients were in leadership roles in significantly higher percentages than non-recipients. Community Participation • A majority of recipients (62%) have completed their Community Action Plans, but no significant differences were found in the percentage of recipients and non￾recipients who were members of community organizations or in the types of organizations to which they belonged. This may be changing with the increased emphasis on carrying out action plans under the Scholarships for Education and Economic Development (SEED) program, however, since a greater percentage of SEED scholars are members of community organizations than were Cooperative Association of States for Scholarships (CASS) scholars. Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page xiii International Involvement • A majority of recipients continued to communicate with classmates from other countries after completing their scholarship program. More than three-fourths of these individuals used electronic means such as email, twitter and Facebook for communicating. The most often mentioned reason for communicating was for professional issues. Non-recipients, on the other hand, communicated principally with local ex-classmates via the telephone. • The experience of living abroad provided by the scholarship program has increased positive perceptions of the United States. Only about half of scholarship recipients reported having a positive view of the U.S. at application. Over three-fourths of the recipients had a positive perception of the U.S. at the time of the study. Non-recipient Finalists • The non-recipient finalists for the program in the two-year group have also exceeded general age-group norms for employment in their respective countries, but not as much as have the recipients. Non-recipients in all groups remain largely favorable to the U.S. and the program despite not having received a scholarship. CONCLUSIONS The findings show the following about the LAC HE scholarships: • They have enabled recipients to become change agents principally through the use of new technologies and strategies in skill and management positions in the workplace. • The scholarships have given women and indigenous recipients increased access to employment and leadership opportunities and have contributed to teachers remaining in rural areas to teach. • They are contributing to USAID workforce development objectives by providing recipients with an experience that leads to greater employment opportunities which they would not have had without the scholarship. • They have had a positive impact on the ability of women and indigenous persons to find employment and hold skilled or management positions. • Participation in the scholarship program has encouraged its professional recipients, primarily teachers, to continue living and working in rural areas. • Credentialing is not generally part of the recipients’ pathways to success; only about one-quarter of recipients have had their credentials recognized. • Recipients are advancing their leadership development objectives, as higher percentages of all recipients, including female and indigenous recipients, held leadership roles in the workplace than non-recipients. • Participation in the program has had little effect on recipients’ community involvement. Recipients were not significantly different from non-recipients in their participation in community organizations. • The scholarship experience has given recipients an international perspective on knowledge acquisition and provided tools for international sharing of information. A Evaluation of LAC Higher Education Scholarships Program Page xiv Evaluation of LAC Higher Education Scholarships Program majority of recipients have continued to communicate with classmates across the targeted region as well as in the U.S. using electronic means. • The finalists for the program who did not receive scholarships make up a group of motivated, talented individuals who are largely positive about the U.S. They may be an important group that could be a worthy investment locally at reasonable cost, also furthering the leadership goals of the scholarship program. • Overall, the evaluation found that the program fulfills objectives of personal advancement for rural dwellers, women and indigenous people, and encourages leadership through use of new strategies and techniques, and, from this perspective and compared to other programs, offers good value. DIRECTIONS FOR FUTURE PROGRAMMING USAID should: • Consider ways to increase post-program support once recipients return home, in order to strengthen the completion of Community Action Plans and support the important young professional networks developed through the investment in their training. This could be accomplished at no significant cost increase by diverting a proportion of scholarship awards to this essential post-scholarship component. • Carry out an in-depth study to examine why there appears to be a relative lack of program impact on recipient community involvement. Include in the study the increased emphasis on community involvement and action planning under SEED. • Examine the elements of the teacher professional development program that have encouraged teachers to continue to work in rural areas after training. Such a study should determine the feasibility of incorporating elements of the program into in-country teacher professional development programs. • Continue and increase the emphasis on computer training and use, both for technical reasons and to facilitate international communication as a strategy to achieve global market participation. • Consider ways to address the high career potential of the non-selected finalists, who generally have all of the capacity of the scholarship recipients, but are left with no further support or input after the application process. • Consider strategies used by other scholarship programs and donors to broaden the reach of the program to non-recipient finalists, such as: o Educational support of family members in order to reinforce intact families and communities, as the Western Union Foundation has done with success to broaden impact; o Local micro-scholarships for study or training, as the State Department has implemented successfully for disadvantaged youth in Latin America and the Middle East; o Job skill and job-hunting support through local study/training centers and local scholarships, as done by the Haitian Education and Leadership Program (HELP). o Massive Open Online Coursework (MOOCs), the burgeoning low-cost distance higher education movement. MOOCs are still in their infancy, but as they grow and diversify in Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page xv content and diversify their instructional languages, could benefit past recipients and non￾awardees alike to enhance their knowledge and skills back home. Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 1 I. INTRODUCTION This document presents the findings for a performance evaluation of the Latin America and Caribbean (LAC) Higher Education Scholarships Program conducted under a Task Order from the LAC/RSD under the GEM II contract. The study dealt with program effects and implementation. Effects were examined by comparing scholarship recipients to non-recipients who were judged to be outstanding candidates for a scholarship, but were not chosen in the final selection. Implementation was studied by examining service delivery and program costs in relation to other scholarship programs offering similar services. The study took place from May to October 2012 and focused on the last seven years of complete program implementation (2004-2010). Three main aspects of the scholarship program were studied: 1) instructional delivery, including selection, orientation, skills training, enrichment activities designed to meet program objectives, and follow-up; 2) costs, including overall costs by program component, and costs in other scholarship programs with similar objectives; and 3) recipients, in terms of experience prior to training, and training experience, as well as outcomes related to post-training employment, leadership roles, community participation, continuing education, and international involvement. The initial sections of this report discuss the history and characteristics of the scholarship program and previous evaluation efforts. These discussions are followed by a description of the methodology used to gather and analyze data for the evaluation. Subsequent sections present the findings of the evaluation in terms of service delivery and relative costs, as well as impact of the program on recipients. The final section of the document presents conclusions drawn from the findings and offers suggestions for future programming based on the conclusions. II. BACKGROUND A. PROGRAM OVERVIEW In 1984 the National Bipartisan Commission to Central America and the Caribbean (the Kissinger Commission) found that the United States was not sufficiently responsive to the political and civil turmoil in the region. The commission urged that the United States Government provide immediate and direct support to the region’s non-elite populations through implementation of recipient training programs that would meet the civil society democratic and development needs of the region. In response Congress appropriated funding that directed USAID to address this need through recipient training of youth, and designated a partner, Georgetown University, through its Center for Intercultural Education and Development (CIED), to implement the LAC Higher Education Scholarship Program. A year later, Congress authorized the creation of the Caribbean and Latin American Scholarship Program (CLASP), to be implemented by USAID through Georgetown University. Training initiatives under CLASP I and CLASP II focused on Central America (CASP), the Andean region (APSP), and the Caribbean (PTIIC). In 1989, with the expansion of the CASP program into the Caribbean region, the program became known as the Cooperative Association of States for Scholarships (CASS), introducing a formalized cost￾sharing plan on the part of the community colleges and universities that hosted the scholarship recipients. At that time, the CASS program included only two-year recipient training exchanges for young high school graduates and its objectives focused strongly on technical training for employment, Evaluation of LAC Higher Education Scholarships Program Page 2 Evaluation of LAC Higher Education Scholarships Program leadership development and civil society diplomacy needs. Over time, the program has continued to focus on the most disadvantaged populations of the LAC region. In 1995, CASS first added professional development scholarships, which provided short-term specialization and skills upgrading to rural mid-level professionals to make them and their institutions more effective. As part of USAID’s Cooperative Agreement with Georgetown starting in 1998, this model was adopted, providing training to help support USAID’s commitment to improving basic education and fighting infectious diseases. Programs of one year, six months and three months were tailored for recipients to develop their professional and leadership skills and have been in fields of study supportive of participating missions’ Strategic Objectives. The program has provided those professional development scholarships in addition to the two-year scholarships for rural youth. Determination of field of study has been made in accordance with USAID Strategic Objectives and country development needs current at the time of scholarship programming. The program was openly competed by USAID in 2007, requesting bidders to design a program for the same populations as with CASS having more focus on leadership development and community engagement. The Scholarships for Education and Economic Development (SEED) program, awarded to Georgetown in 2008 on a contract running through 2014, added competitive sub-awards to colleges for the provision of training and a stronger focus on the above elements. Since these programs’ inception 29 years ago, 9,191 scholarships have been awarded overall at a total cost of more than $450,000,0001, of which $320,820,029 was provided by USAID through Cooperative Agreements to CIED. Central to the CASP, CASS and SEED programs has been the consistent focus on providing training opportunities to socially and economically disadvantaged rural populations across as many as 12 countries in the region. As USAID has redefined its development priorities for the region, several of these countries have “graduated” from the CASS/SEED program.2 Thus the CASS and SEED programs have served seven countries during the period considered under this study. The current SEED Program seeks to: 3 1. Create a cadre of change agents and future leaders with understanding, experience, and appreciation of democratic processes eight-year period under study. Provide recipients with skills and knowledge to participate actively and responsibly in a democratic society; 2. Provide participants with skills and knowledge to participate actively and responsibly in a democratic society; 3. Strengthen capacity in countries participating in free trade agreements in a range of technical fields that are seen as critical to help countries become more competitive and better realize the opportunities brought by free trade; and 1 Total amount estimated; records on specific cost-sharing amounts for CASP (prior to 1989) are not available. Support to CASS/NSPS from non-USG sources totaled $89,782,531 over 18 years; and to SEED, $9,472,013 to date (from 2008 through 2012). 2 Over time these programs have provided training to participants from a total of 17 countries: Antigua, Barbados, Barbuda, Belize, Costa Rica, The Dominican Republic, El Salvador, Guatemala, Haiti, Honduras, Jamaica, Mexico, Nicaragua, Panama, St. Kitts/Nevis, St. Lucia and St. Vincent. See Appendix 1 for detailed breakdowns by year, country and field of training. 3 USAID, 2008. “Request for Applications for Higher Education Scholarships for Training and Development”, p. 2. Washington, DC: Grants.gov. Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 3 4. Support U.S. public diplomacy by creating a cadre of potential leaders who have been immersed in American culture and are sympathetic to the United States. It is important to point out that while candidates for the CASS/SEED two-year program are largely from underserved populations in rural areas, they are unique in countries where very few individuals complete secondary school. Rural states of Mexico, which are targets for the program, have high school graduation rates from 64 percent (Chiapas) to 74 percent (Oaxaca), compared to the nearly 100 percent in the Federal District. In El Salvador, 2008 data showed that only 39 percent of all youth completed the nine-year basic education cycle. Of this group, 28 percent started upper secondary and only 22 percent completed high school studies.4 In Guatemala, 2005 data showed that 33 percent of the graduates of the six-year primary cycle entered the three-year lower secondary cycle, and just 20 percent completed lower secondary. Eighteen percent of the age group entered upper secondary and 10 percent completed their high school education.5 All applicants to the SEED program are among the limited number of educated youth in their communities by having completed upper secondary school. The table in Appendix 1 summarizes the LAC Higher Education programs’ overall history; more detail on the cohorts studied in this project is provided in Tables 1 and 2 in later sections of this report. B. PROGRAM OBJECTIVES The principle aim of LAC’s higher education activities is to “provide technical training to a significant number of participants in fields of study that address the development needs of the participating countries and help to achieve the USAID Mission and LAC Regional strategic goals and objectives.”6 In addition to providing technical training, the LAC/RSD higher education scholarship program is designed to support increased equality of opportunity and cultural understanding. The program provides opportunities for poor and disadvantaged youth in LAC with secondary school degrees who have the potential to be future leaders to gain technical skills in an array of fields deemed important to the social and economic development of the individual communities and countries, and to USAID goals for the region. C. PREVIOUS EVALUATIONS Evaluations of the CASS program were undertaken in 1994 and 2002 by Aguirre International. Those evaluations, along with quarterly and annual reporting records, have shown that program alumni regularly return to their home countries as well-trained change agents who exert influence on numerous others in their personal, professional and community networks. These evaluations conclude, though not empirically, that the program offers benefits and individual returns for participants and their professional sectors. It was noted that changes can often be immediate and observable (such as the introduction of new production techniques in local businesses or implementation of improved pedagogical methodology). The previous evaluations suggest, however, that alumni impact in leading social change may not be as immediately apparent and will require longer-term monitoring and evaluation to assess appropriately. These previous evaluations have relied upon beneficiary interviews and site visits, and provided useful summaries of the extent to which the CASS program met its originally stated goals. They did not 4 World Bank, 2011 5 Juárez and Associates, 2011 6 USAID, 2008. “Request for Applications for Higher Education Scholarships for Training and Development.” Evaluation of LAC Higher Education Scholarships Program Page 4 Evaluation of LAC Higher Education Scholarships Program address questions of cost-effectiveness or compared CASS to other scholarship programs, nor were any direct comparisons made with non-recipient peers. Thus, as the current SEED agreement nears its end – the final cohort of recipients is scheduled to begin study in 2013 – LAC/RSD requires information on aspects of the training program that have not previously been examined in order to inform possible future programming. D. STUDY QUESTIONS Through discussions with professionals in LAC/RSD, the evaluation team developed two principal evaluation questions. These questions guided the design, methodology, and reporting of the findings of the investigation: 1) To what extent can it be said the LAC/RSD higher education scholarships program provides value for the investment? This question aims to shed light upon the relative “worth” of the program, as in terms of the measurable results for the dollars invested, and compared to alternatives. It required a determination of absolute and relative costs of the program and the identification of programs with similar objectives and student populations. Within-program costs were also examined by component to identify innovative, potentially cost-saving strategies. (As it provides the reader with a more in-depth description and insights into the program, findings related to this question are discussed first, as a prelude to the question and findings on question 2 below.) 2) To what extent have LAC/RSD higher education scholarships enabled participating individuals to become leading change agents in their respective professional fields, communities, and/or countries? This question seeks to explore the connections between more measurable individual returns of the program and the core development objectives of the program over time. It required a determination of the outcomes of training and follow-up and the comparison of the results to individuals in the participating countries who had similar characteristics, but did not become scholarship recipients. Samples of candidates for scholarships who were in the finalist pool, but were not selected were used for this purpose. Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 5 III. DESIGN Although very basic demographic data exist on all recipients and non-recipients, systematic baseline and process data as they affect individual program recipients’ success in meeting program outcome objectives had not been previously collected. Also, the program’s structure and costs in relation to other programs with similar objectives had not been examined. Thus, this study was designed as a hybrid performance evaluation that includes elements of an impact evaluation. It compares program characteristics to those of other scholarship programs and also measures changes in similar groups of subjects who have and have not had scholarships on the same dimensions at different points of time in their post-program careers. To carry out the evaluation, a multi-method design consisting of checklist and group and focused individual interviews was employed. Retrospective reporting of key incidents in the scholarship recipients’ and non-recipients’ training, employment history, and community participation subsequent to being selected as program finalists was a principal technique used in the conduct of interviews. Field data from institutions providing training and interviews in the target countries were complemented by secondary data provided by program and training institution records, as well as review of documents and discussions with administrators of selected scholarship programs. Instruments were tested in a country not selected for intensive examination at the start of the evaluation. A. VARIABLES Study variables were of three principal types: those associated with program costs, those associated with instructional delivery, including orientation prior to arrival at the training institution, and those associated with the finalists for scholarships. Program cost variables included direct and indirect costs, component costs, in-kind contributions and other cost-saving strategies, home country contributions (financial or in-kind) and absolute recipient costs, as well as cost per successful completer. Instructional delivery variables were related to selection and orientation procedures prior to departure, placement at the training institution, academic training and support, enrichment activities, and job placement orientation and preparation. Finalist variables included those related to the recipient’s background, those related to selection and training, and those related to post-training employment history and leadership roles. Variable clusters are as follows: Program Costs: • Cost requirements for bidders; • Component direct and indirect costs (e.g. tuition, travel, sub-contract labor/administrative costs, and materials); • Allocation of costs for disbursement of funds to recipients; • Distribution of NICRA and/or administrative fees; • Overall cost per recipient; • Georgetown/CIED technical assistance for campuses on costing. Training Program Processes: • Public Relations, Outreach and Recruitment – criteria (sex, ethnicity, community location, education, job history/type, community participation, leadership, language skills), strategies for soliciting applicants; Evaluation of LAC Higher Education Scholarships Program Page 6 Evaluation of LAC Higher Education Scholarships Program • Pre-selection/Selection – procedures (interviews, tests, recommendations, match of skill set to available program, match of program desired to country/USAID development objectives), committee make-up, rating criteria; • Placement at Training Institution – orientation, language training, housing, social/cultural activities; • Course of Study (structure) – mix of academic and enrichment courses, mainstreaming, provision of tutors, availability of learning aids, degree or certification; • Job Search/Placement – program support in finding employment or returning to existing job, accreditation in home country, employment success in relation to national trends. Recipient/Non-recipient Experience: • Demographic Variables (to characterize change over time) – ethnic identification, marital status, place of residence, time in place of residence, employment (job type, job in rural area, promotions and salary), employment history – advancement to leadership, effect of work on institution and community; • Leadership/Community Participation – implementation of action plan, involvement in community organizations, leadership in community organizations, contribution of activities to community improvement; • Participation in International Community – continuing education, alumni associations, contacts with classmates, U.S. host campus and community contacts; • Pathways to Current Life Situation – identification of key elements of training experience that contributed to actual situation. B. SAMPLE The evaluation team drew the study sample from three countries: El Salvador, Guatemala, and Mexico. These countries were selected for intensive study because of the availability of data on non-recipient finalists, who could serve as a comparison group, and consistent yearly participation in different types of training through the period under evaluation. The sample included three CIED program administrators and three financial specialists in Washington, as well as the country coordinators and other available staff members in the three study countries. A total of 14 CIED personnel were interviewed. Two host colleges also were selected to gain information on program service delivery. A total of five college program administrators and six financial specialists were interviewed across the two colleges. A group interview with scholarship recipients who were enrolled at each of the colleges was held with 17 and 18 recipients. The field sample consisted of scholarship recipients and non-recipients from the years 2004 through 2010. A random sample stratified by sex, ethnicity, and year of entry was drawn from populations of 872 recipients and 1044 non-recipients in the three countries. The final sample size was set at approximately 250 recipients and 200 non-recipients. Owing to the difficulty in finding recipients and non-recipients during the pilot study, an oversampling strategy was used to create a sampling pool of 890 individuals. Of this pool, 238 recipients were located and interviewed. The recipient sample has a margin of error of 5.4 percent and a level of confidence of 93 percent. The comparison/counterfactual group of non￾recipients consisted of 214 individuals selected according to the same criteria as used for the recipient sample. This sample allowed for a 6 percent margin of error and a 90 percent level of confidence, which is acceptable since program outcomes are not being examined for this group. Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 7 Table 1 presents the overall sampling for the evaluation by cycle year. As can be seen, the percentages of recipients and non-recipients by year are very similar. In addition, percentages by both groups are fairly similar across all years except 2010. The increase in 2010 reflects the increase in total recipients in that year. Table 1: Recipient/Non-recipient Sample by Year Year/ Finalist Type 2004 2005 2006 2007 2008 2009 2010 Total Recipient 37 (14%) 30 (13%) 31 (13%) 32 (13%) 34 (14%) 41 (18%) 37 (15%) 238 (100%) Non-recipient 30 (14%) 26 (12%) 26 (12%) 27 (13%) 37 (17%) 36 (17%) 32 (15%) 214 (100%) n = 452 The sample reflects the overall recipient population in the target countries. It divides almost equally by sex, with females making up 48.7 percent compared to 49.3 percent of the entire population and males accounting for 51.3 percent, compared to 50.7 percent. Sixty-three percent of this population identifies as Hispanic/Mestizo, whereas 37 percent identifies as indigenous. This ratio compares to 59 percent Hispanic and 41 percent Indigenous for the overall recipient population. C. INSTRUMENTS In order to implement the multi-method design of the evaluation, several instruments were developed. These instruments included a variety of data retrieval forms, including checklists for certain financial data and secondary data previously collected on recipients and a guide for group interviews with current recipients on college campuses, as well as semi-structured interviews with program backstop personnel, implementation staff and scholarship recipients and non-recipients. Five semi-structured interview schedules, which included closed- and open-choice questions, were developed. The instruments were: • Program Administrator interview, for program staff and college implementing personnel; • Financial Administrator interview, for program and college staff managing the resource and financial aspects of the program; • Country Coordinator interview; • Recipient interview; and • Non-recipient interview. The various interviews explored the overall costs of the program and individual component costs; the procedures for selecting and fielding scholarship recipients; fielding and follow-up with recipients during training; implementation of training; and the experience of scholarship recipients and non-recipients with developing employable technical skills, leadership abilities, and a focus on community participation through program participation or by alternative means. D. PILOTING AND FIELDWORK Piloting of the instruments took place in Nicaragua, the country participating in the scholarship program that was most similar to the sample countries in its types of programs, number of students that had participated in the scholarship program, and available data on non-recipients. A primary concern of the pilot study was to determine the feasibility of tracking a random sample of non-recipients, who generally had no contact with the scholarship program after not receiving a scholarship. In order to test the feasibility of finding and contacting the such individuals, a sample of the Nicaraguan population of Evaluation of LAC Higher Education Scholarships Program Page 8 Evaluation of LAC Higher Education Scholarships Program recipients and non-recipients was drawn using the same criteria employed for the sample in target countries, applying a 10 percent and 25 percent oversample for recipients and non-recipients, respectively. A period of two weeks was allowed for updating contact information with the local SEED office, developing strategies for reaching potential sample members and determining the possibility of interviews. Contact strategies included phone calls, emails, and in a few cases, visits to explain the study and request an interview were used. Even with updated contact information, many of the phone numbers, especially for non-recipients, were found to be inactive or reassigned to other people. Of the 139 recipients in the pilot sample, 95, or 68 percent, were reached. With the non-recipient sample 34 of 131, or 26 percent, were successfully contacted. These results were used to adjust the sample by increasing the oversample for recipients from 10 percent to 25 percent and the oversample for non￾recipients from 25 percent to 100 percent. Instruments for recipients, non-recipients and the country coordinator were also piloted and adjustments were made in the phrasing of questions based on the interviews. The senior evaluation team members designed the training and field procedures for data collection teams numbering five to 12 in each targeted country. Guatemala and El Salvador each had a single country supervisor who was an experienced research manager. Because of the size of the country and dispersed nature of the sample, two country supervisors were used in Mexico. The data collection training took place in each country over a two-and-a-half-day period. Training across countries was staggered over a three-week period from mid-July to early August. Data collection team members, all local-hire nationals, had previous experience in fieldwork and were proximate to the geographic areas of sample members whenever possible. Training content included: introduction to the CASS/SEED program; introduction to the evaluation; role of the data collector; use of data retrieval forms; field interviewing techniques and use of the interview protocols; and simulation of fieldwork through role playing and practice interviews with recipients not in the sample. As a result of the pilot study, an extra training segment was included on phoning and other contact strategies for reaching sample members and ensuring their participation in the study. All training components were summarized in field manuals, which served as reference guides for field procedures for data collectors during the investigation. In addition to using field manuals to ensure consistency, other quality control procedures included weekly phone calls among supervisors to share strategies and results; instrument editing to ensure completeness and legibility; and re-interviewing of a small number of sample members, when necessary, to ensure complete information. Data collection took place over the months of July and August 2012. As anticipated, the greatest challenge of data collection was locating the sample members and obtaining agreement to be interviewed. The strategies of phoning and emailing had in some cases to be augmented with sending teams or individual data collectors to towns to identify the whereabouts of sample members through municipal records and the knowledge of local leaders, as well as sending telegrams and visiting the ministries of education and health to locate sample members employed by such institutions through employment records. Despite the above efforts, almost half of the potential sample members identified did not participate in the study. Table 2 summarizes the main reasons for lack of participation. Phone numbers had changed nationally in two study countries within the last five years; in other cases numbers in the data base were inactive, or people located at a given address had no information on the sample member. In a few cases, the people contacted at the address on record were family members, but did not have contact information on the sample member. A few of the potential interviewees had moved to an area of the country outside the study, or could not be reached because seasonal weather conditions made overland travel impossible. Only a few potential sample members were reported to be in the U.S. A significant Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 9 percentage of non-recipients refused to participate because of their unsuccessful experience in applying for the scholarship. Individuals, who scheduled an interview, but did not show up and could not be rescheduled, were also considered as refusals. Table 2: Recipient/Non-recipient Reasons for Non-response Reason/ Finalist Type Wrong Phone/ Address Out of Area/ Unreachable In USA Refused Recipient 90 (84.9%) 10 (9.4%) 2 (1.9%) 4 (3.8%) Non-recipient 248 (78.5%) 23 (7.3%) 14 (4.4%) 29 (9.1%) Percentages may not total 100% due to miscellaneous responses not included in column categories. E. DATA ANALYSIS Quantitative data analysis consisted of calculating the absolute and relative frequencies of each variable in order to examine costs and characterize the general trends in the program. Compound variables were created, where appropriate. These variables were used to examine the relationship of program experience to general program goals at the individual level. Analyses then identified and drew findings across relationships between individual and compound nominal and numeric variables and outcomes for recipient and non-recipient groups. Qualitative data analysis of the semi-structured interviews and focus group discussion transcripts was designed to identify and enable coding of key themes related to program value, cost-effectiveness, impact, leadership, change management and career pathways, in order to enrich and deepen trends found in the quantitative analyses. Data displays, presented in Part I of the Findings section of this report, were developed to examine program characteristics juxtaposed with those of similar programs. F. LIMITATIONS The principal limitation of the study was the lack of baseline data on the situation of the recipients and non-recipients in relation to the areas under evaluation. Application data retained were limited to sex, marital status, ethnicity, residence and employment. A second limitation was the lack of up-to-date contact information on former candidates for scholarships and scholarship recipients. Despite a number of strategies used to locate sample members, the total number was slightly less than would have been ideal. However, all findings were within the final margin of error. A further limitation was not having a comparison group that had been through an alternative scholarship program. Such a group would have allowed greater precision in determining program impact. To overcome this limitation, detailed information on non-recipient activities after applying for a scholarship was obtained. Finally, the sample is not representative of the entire scholarship program, but rather reflects those countries that maintained the greatest amount of data on non-recipients. Evaluation of LAC Higher Education Scholarships Program Page 10 Evaluation of LAC Higher Education Scholarships Program IV. FINDINGS A. PROGRAMMATIC CHARACTERISTICS AND COSTS This section describes the components of the CASS and SEED programs and discusses organization and implementation issues based on site visits to the programs’ central office, participating colleges, and country coordinating offices. Included in the discussion are the use of resources and perceived impact of the program by staff at Georgetown University, USAID’s implementing partner in Washington, DC. Georgetown University works with partners’ field staff in the three countries of the study, and administrative and instructional staff for the program at two sub-contractor host colleges in the United States. Comparisons and contrasts with other sponsored international scholarship programs sending students to the U.S. are presented where relevant. Program Overview As mentioned in Section II.A above, since inception Georgetown University, through its Center for Intercultural Education and Development, has been responsible for implementation of the CASS and SEED programs. CIED currently employs five staff full-time in its Washington, DC, headquarters and two to four individuals in each of the seven participating countries. Senior staff at CIED (notably the Center’s director and its director of finance and administrative services) also provide managerial and financial oversight to the program, on a donated services basis. The three senior staff members heading up the program at CIED headquarters have worked with the program in a variety of capacities since the 1980s. CIED country field staff also has generally been with the program at least 10 years. This staffing level is much reduced since CASP/CASS first started in the1980s, when as many as 20 staff supported a much larger program. While reduced program activity has lessened the staffing proportionately, CIED staff described this shrinkage of personnel as the outcome of a streamlined and flattened management approach, bolstered by more robust technology. It was also pointed out that this change has resulted in greater integration of program components by staff members, who must be aware of the status and interactions of program activities across separate components, whereas the previous structure sometimes encouraged smokestacking. The two-year program continues the original model for recent high school graduates with no further formal education, whereas the one-year and six-month programs are professional training models for career government employees or educators in the fields of education and health. The remaining scholarships and programming (following the same program models) have been leveraged through additional support provided by local USAID missions and home country governments (notably in Mexico). As will be discussed in detail below, scholarship numbers have been bolstered or at least maintained despite inflationary increases in cost due to increased cost-sharing and cost-saving by CIED and its host institutions. This evaluation deals with the period from 2004 to 2010. In that period, the program has been active in seven countries – the Dominican Republic, El Salvador, Haiti, Guatemala, Mexico, Honduras and Nicaragua. Available records show that it has had 3,141 finalists and served 1,960 recipients in that time. The majority of recipients (1064, or 54%) have been awarded two-year scholarships. A total of 439 recipients (22%) have participated in one-year programs, and 457 (23%) have participated in six-month programs. As can be seen from Table 3, below, the overall pattern of a majority of scholars entering two-year programs has held for each year under study, except 2008. There is generally an increase in the number of scholars beginning in 2007. The exception is 2009, when the program was transitioning from CASS to SEED. Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 11 Table 3: Scholarship Recipients by Program Type over the Period of Study Program type: Cycle Two-Year One-Year Six-Months CASS 2004 139 58 80 CASS 2005 138 58 56 CASS 2006 138 62 56 CASS 2007 174 60 90 CASS 2008 162 121 97 SEED 2009 149 21 18 SEED 2010 164 59 60 Total 1064 439 457 Source: CIED, Washington, DC Program Funding and Recipient Costs During the period under study, the program’s funding has been maintained at a steady annual amount— approximately $10 million per fiscal year. Costs for most components of the scholarship program have increased steadily over time due to general inflation. While the program’s size has remained fairly stable due to high institutional cost-shares, the national coordinators interviewed and staff at both U.S. host institutions visited all suggested that such contributions have at times eliminated valuable components or staff to the program. It was reported that some U.S. institutions might have withdrawn from further involvement due to expectations of cost￾share that exceeded their ability to comply. This issue is discussed in more detail in the program cost section below. The program most similar to SEED, funded by other sponsors, is the Community College Initiative (CCI) scholarship. This program has a per-year cost-per-recipient similar to that of the SEED youth program. Its overall administrative costs, however, were reported to run approximately 50 percent higher than SEED’s. The possible reasons for these differences are discussed in the comparative program section below. Table 4, and similar tables in later sections, compare and contrast some of the specific components and approaches of CASS/SEED with other programs reviewed in a benchmarking exercise involving nine international scholarship programs: • Community College Initiative (CCI) • Edmund Muskie Fellowship (EMF) • English Access Microscholarship Program (EAMP) • Ford International Fellowship Program (IFP) • Haitian Education and Leadership Program (HELP) • Joint Japan/World Bank Graduate Scholarship Program (JJWBG) • Organization of American States Academic Scholarship Program (OAS) • Programa de Crédito-Beca COLFUTURO (CF) • Western Union Foundation Family Scholarship (WU) (Full descriptions of these benchmarked programs can be found in Appendix 2.) Evaluation of LAC Higher Education Scholarships Program Page 12 Evaluation of LAC Higher Education Scholarships Program Table 4: Costs and Cost Structure of CASS/SEED Compared with Other International Sponsored Scholarship Programs Program elements CASS/SEED Other Programs Observations on CASS/SEED Costs Determined via capped target budget point established annually. Appropriations from sponsoring agency tend to follow inflation increases or change due to policy priorities (State Department-funded programs—FB, EMF, JJWBF). Many lines have not increased in years and some budget lines are no longer aligned with real costs. Cost structure Heavy emphasis on cost-sharing and partner donations. Emphasis on negotiated indirect rates (all other programs using Federal indirect cost agreements). Current cost sharing expectations may make keeping and attracting experienced partners difficult. Program Components The CASS/SEED programs, like most other sponsored scholarships, have a number of standard components. Broadly defined, they include: • Candidate recruitment and selection; • Program administration and management; • Participant components and support (orientations, travel, visa, insurance, stipends, housing); • University/college training, seminars and activities in US; and • Post-program support. The six- and twelve-month programs for professionals differ from the two-year student program not only by the recipients’ age range and level and fields of training, but also in their somewhat different mix of program components. Since the program does not require English language proficiency at application, English language training is a major component of the two-year program. It comprises approximately one third of the time given to coursework and the tuition associated with it. CASS/SEED understandably has higher proportional training costs for this component of the program than any similar sponsored scholarship program that sends participants to the United States that was reviewed. As for the professional recipients, comparing the six-month and twelve-month models, the per-recipient cost for 12 months is not quite double (some costs are fixed no matter what the program’s length: recruitment, screening and selection, international travel, passport and visa costs, recipient support, when compared on a monthly basis) by host institutions and CIED. Unlike the youth program, CASS/SEED’s professional development program content is not delivered in English; course content typically is presented in the recipients’ native language or interpretation is provided by local translation contractors. Thus, this program’s English language training is designed only to assist recipients with community integration, and this training component therefore is much less costly in both money and time than in the two-year program. Resources are generally shifted to the professional/internship components of these programs. Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 13 Figure 1 below illustrates the program components in sequence, illustrating the inputs and outcomes in a logic model. Parallel to this sequence is the “black box” of the experiences of the non-recipient applicants, who had never undergone any evaluation or impact follow-up before this evaluation. Figure 1: Sequential Logic Model of the CASS/SEED Program Housing costs are higher in the aggregate for the shorter programs, in that they do not have a live-in host family component, which costs significantly less than rental or dormitory housing. The host family facet is also unique compared to other programs reviewed, which do not place scholars with host families, or do so only for very limited periods. Other costs (travel, recruitment, insurances, support services, staff support, etc.) are non-differential components. Their costs are similar across all three models when program length is considered. Program Recruitment The recruitment of program recipients is handled in each participating country by the national coordinating office, following detailed standard recruiting guidelines produced by CIED in Washington. While recruitment outreach for the two-year youth program is handled by each national office using its own networks, recruitment for the professional programs (six-month and one-year programs) is initiated in collaboration with the participating country’s ministry or agency from which the recipients are drawn. In each country visited, the strategy for this collaborative recruitment was slightly different and was tailored to local realities. In Mexico, for example, teacher candidates are eligible through their success in a teacher-of-the-year competition; in Guatemala applicants can apply or be recommended internally through the ministry using an internal ministry website developed for the program. Evaluation of LAC Higher Education Scholarships Program Page 14 Evaluation of LAC Higher Education Scholarships Program While most other scholarship programs today rely heavily on electronic means to solicit, screen and review applications, CASS and SEED differ from that trend—they have used only traditional paper-based application approaches. It was argued that the access to computers and the ability to use computers could not be assumed for the rural and indigenous populations the program targets, thereby reducing the chances that good candidates would apply. Furthermore, the paper base was preferred because it allowed the applications to be specifically targeted to certain networks, communities and trusted contacts. This approach appears to trade off efficiencies of cost and time that other programs have gained through the online approach for ensuring that those without access to the Internet can apply. Finally, as indicated previously, a unique element of the CASS/SEED scholarship is that previous study or competence in the English language is neither required nor relevant to recruitment or selection. No other sponsored scholarship to the United States reviewed allows recipients to come (and obtain a student or exchange visitor visa) without demonstrated English-language capacity. Each of the two program types will now be discussed separately. 1. Two-year Program After more than a quarter-century of operation in all participating countries but Mexico, CIED administrators in Washington and in the two countries visited explained that outreach and recruitment of two-year CASS/SEED candidates is generally handled only through established networks and contacts. These programs have established strong networks of community- and rural-based organizations through which they reach out to identify the kind of candidate that they seek. This pre-identification strategy increases the likelihood that the candidate invited to apply will offer a high potential for community involvement and impact upon returning home. CIED program managers stated that more open promotion and wider distribution of information and applications for a program like SEED would generate a mountain of inquiries and applications, many of which would come from individuals without even the basic qualifications required. Such an approach would also risk being co-opted by education brokers who use such opportunities as a fee-for-service offering. (CIED works to eliminate the possibility of such practices through clearly stating on the application that it is free of charge and that the forms may not be copied and distributed to others.) The CIED office in Mexico, conversely, is more broad-based in its outreach, and uses a website and printed brochures in addition to distribution of paper applications through a support network. This is due to several reasons. The Mexico program is only eight years old and therefore less well known than it is in other countries. Additional factors include the communication challenge of the country’s large size and decentralization of the program’s outreach to state and regional levels (SEED is actively promoted in five states for the youth program and across 13 states for the professional offerings). Finally, current issues of safety and security in some regions of the country do not readily support a more direct outreach and presence by CIED-Mexico staff. Further, since Mexico program costs are co￾funded by several Mexican government and private-sector co-sponsors, the related networks and interest in a broader media presence by these Mexican partners also explain this more assertive outreach approach. Each National Coordinating Office uses a “recruitment pyramid” (see Figure 2 below), scaling its recruitment according to the probable number of awards that will be available in the coming year. The typical formula is to generate 12 applications for each projected slot, and to select four from that pool for further screening. This initial screening ensures that the basic criteria expected of applicants are met: age range; school completion and acceptable academic performance; match between the applicant’s background and interest, and the eligible field(s) of study; and initial evidence of community commitment. The second-tier selection (which includes only those applicants who have met the above first screening criteria on paper) then leads to interviews (three candidates per scholarship slot). Final Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 15 consideration of that shortlist is accomplished with the help of a national screening committee, made up of members of the program’s national network, as well as, if possible, educational and psychological experts and a representative of USAID. Coordinators in all countries visited reflected that members of the selection committees were conscientious about recusing themselves from selecting any candidate known to them, and that there was very little external pressure exerted on them about candidate selection. Individual selection criteria include7: • Very good to excellent academic performance in secondary school leading to a high school diploma; • A demonstrated commitment to community service; • Excellent references; • A strong performance at the individual interview; • Proof of a disadvantaged socio-economic background; and • Suitability for the area or areas of study offered in the given recruitment year. Collectively, the nominated finalist cohort in a given country must also reflect a gender balance (close to half men and half women), a predominance of candidates from rural settings (80%) in the participating country and—where applicable—a majority representation of indigenous nominees. Equally specific are a number of criteria that will disqualify a candidate from further consideration despite meeting the above positive criteria: • Current residence in the U.S. by any immediate family members (a parent, sibling or other close relative); • Post-secondary study since graduating from secondary school, even in the host country; or • Discovery of a less disadvantaged socio-economic background than was claimed in the application. Where deemed necessary, the last criterion above is verified through “drive-bys” and other fact￾checking strategies to ensure an applicant’s family resources have not been under-represented. The national committees’ deliberations reduce the finalist numbers to two per scholarship slot. These finalists’ applications are forwarded to CIED in Washington, which organizes the final selection of scholarship recipients and alternates, chosen by committees of host college representatives, convened by field of study. National coordinators also are on hand for this final selection meeting as resources and sounding boards, but they do not have a vote or final say in the host colleges’ selection of scholars. CIED believes that this approach ensures that a student is well matched with the resources of the host campus, as those college representatives know better than outsiders what is available to the student in the campus and community. Furthermore, it was argued that this “final say” on the colleges’ part strengthens the host institution’s buy-in and partnership in the program. 7 CIED, "SEED Country Coordinators Handbook", 2009. Evaluation of LAC Higher Education Scholarships Program Page 16 Evaluation of LAC Higher Education Scholarships Program Administrators interviewed suggested that beyond the official selection criteria outlined above, they were looking for certain elements in the application that suggested potential for success and impact. Each individual appeared to have his or her own set of points that were sought. They included: • Strong evidence of commitment to the home community; • A clear idea of specific impact that could be brought back home; • Consideration of the applicant’s family—their support, but also their needs (in some cases an applicant with a family in crisis is set aside because the crisis might worsen due to their absence); and • Evidence of a personal desire to participate in the program, rather than being pressured by family to do so. In one country this is measured by requiring applicants to write a short essay on the history and geography of their country—a means to measure their self-awareness of their own country and their commitment to its development. None of the above criteria appeared to be in conflict with the official selection criteria, but reflected the particular insights and experiences of an individual involved with the program over time. Source: CIED, Washington, DC Figure 2: SEED’s Pyramid for Recruitment, Screening and Determination of Application Numbers per Scholarship Slot Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 17 Once the final choices of finalists are made (along with a few waitlisted “alternates” who may fill slots that open up), CIED’s national coordinating offices notify all applicants directly of the outcome. At that point, those candidates not designated “finalist” or “alternate” are no longer engaged with the program and its support. That latter group of individuals was the source of the comparison group of this study. The above approach to candidate recruitment used by CIED is compared in Table 5 to that of other programs through the benchmarking exercise. The preference for paper-based application processes in order to reach disadvantaged populations is a primary difference when programs are compared. Table 5: CASS/SEED Candidate Recruitment Compared with Other Programs CASS/SEED Other Programs Observations Relies on established and known networks to refer, recommend and pre-screen candidates Uses highly controlled paper￾based application materials Most other programs broadly promote their opportunities and do pre-screening through an electronic application. Applications and screening generally are online with a possible option of paper-based applications if called for (all other programs reviewed). CASS and SEED are targeting specific populations with limited access to electronic media. This roadblock for applicants may be evolving in many places. 2. Six- and Twelve-month Programs The professional development programs target outstanding mid-career staff of cooperating ministries or agencies (depending on country, in the fields of education, health or both). Initial recruitment is coordinated with the concerned ministry or agency and may involve several different approaches, depending on the case. In some situations nominations are made by appropriate regional or subject￾matter experts within the ministry; in others, applications are received from individuals who self-identify as potential participants; a third strategy is providing access to those individuals who had been designated as finalists in a “teacher of the year” award in their region. Once the initial nomination process has been accomplished, however, the second and third selection phases are similar to that of the two-year student program, as described in Section I. above. U.S. Host Institutions Host colleges for all three program types are selected by CIED for participation on an annual basis, through the full and open competition it runs each year. Currently there are about 12 host institutions, each of which hosts a cohort of 18-20 recipients in a given field of study and program type. In some cases, institutions hosting the two-year program may have two cohorts present at the same time in overlapping cycles. Applicant institutions are required to demonstrate capability in8: • Academic training for the designated field of study; 8 CIED, “Proposal Guidelines – The USAID SEED Program, Cycle 2012”, 2012. Evaluation of LAC Higher Education Scholarships Program Page 18 Evaluation of LAC Higher Education Scholarships Program • English-language training (full-time training leading to academic mainstreaming into coursework for two-year students; basic conversational coursework for professional programs whose course content is presented in the native language); • Leadership training; • U.S. culture programming; • Arrival- and departure-orientation programming designed to ease the recipients’ challenges of settling-in and wrapping-up in the US; • Community action planning: development of a plan addressing an unmet development need in the home community that will be implemented once the recipient returns; • Housing (host family placement in year 1 for 2-year students; apartment-based housing in year 2 and for all professional recipients); • Disbursement of pocket money; • Academic and personal counseling and emergency assistance; and • Internships and professional linkages related to the field of study for recipients. CIED opens the annual college-hosting competitions late in each calendar year and reviews and decides on college participation during the succeeding months in time to invite selected colleges to the selection meetings the following March-April. Colleges then begin to host the recipients the following July or January, depending on the program type. The institutional application review is generally made using a 100-point rubric that stresses the importance of the program of study and project management (25 points each) along with institutional capacity/past performance and enrichment activities (20 points each). The remaining 10 points are awarded for the applicant’s plan for providing host families in the two-year program and student housing, when called for. In some program hosting competitions, the point system is adjusted slightly, based on the program’s particular focus. The successful applicant in the competitions receives a contract for one program cycle only—a one-year commitment for the shorter programs, a two-year commitment for the two-year program. There are no “renewals,” per se (i.e., a briefer re-application process or non-competitive extension); each application stands on its own merits and institutions compete each time, regardless of previous experience with the program. This very rigorous application and review process compares favorably with other programs’ approach. The expectations of the host institutions are equal to or often higher than those of other scholarship sponsors, who often expect recipients to locate their own housing, deal with insurance and tuition payments on their own, travel independently to and from the host campus to their homes, etc. CIED’s approach to ensuring cost-savings and institutional buy-in was recognized as effective by key staff working on the program in those institutions visited. Some administrators noted, however, that the short award cycle required considerable time annually for the contract application process, whereas some other program partnerships have multi-cycle contracts (with three or even five years before a fully competitive renewal is required). In one interview, for example, it was pointed out that the short award cycles can dissuade an institution from making longer-term staff or resource commitments to the program since there was no guarantee of continuation beyond the single cycle award. Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 19 When the CASS/SEED approach to networks is compared with other somewhat similar programs (see Table 6 below), its intake process is very strong, while follow-up and long-term engagement with campuses and past recipients may benefit from a more intentional approach. Table 6: U.S. Institutional Networks of CASS/SEED Compared to Other Programs US-based Community Host Colleges CASS/SEED Other Programs Observations Strong network of host institutions with commitments to the programs. Some have dropped out due to imposed budget pressures. Colleges have little information about past students’ activities and little opportunity to visit and interact with alumni once back home. Similar networks of institutions at the appropriate level. No other major program reviewed allows institutions to make the final candidate selection independently. Many other programs have more integrated feedback and involvement of host administrators and faculty in the sending countries and their alumni (FB, IFP, EMFP, CCI). This is an area that can be improved and is weak compared to other programs. USAID may want to look at other programs’ more integrated models for post￾program follow-up, support and engagement of host institutions and their surrounding communities in past recipients’ activities back home. Pre-program Preparation Selected CASS/SEED recipients9 undergo extensive pre-departure preparation in their home country. Three events are organized to prepare the departing recipients for the cultural, programmatic and logistical experiences they are about to encounter. CASS/SEED preparation of new recipients is thorough and well thought-through, based, as the administrators reported, on decades of trial-and￾error. In-country administrators reported an evolution in the applicants’ level of global awareness over the decades. Recipients of today come into the program better informed because of easier access to information through media, the Internet and the information provided by many past recipients. Thus, recipients’ interests, concerns and questions are more informed and pointed because of this. Similarly, the recipients’ initial expectations and aspirations are higher than in previous decades, since it is no longer implausible that a young person even from a disadvantaged background would ever complete secondary school, much less have access to higher education. Program Implementation On-campus Recipients are placed at an institution according to field of study or training, so that they spend their program with a cohort of individuals with similar academic and professional interests and objectives. In most cases, the cohort is a multi-national group of individuals who not only learn about their field of 9 Also included are candidates designated from the finalist pool as “alternates,” should a selectee withdraw or an extra space open up. Evaluation of LAC Higher Education Scholarships Program Page 20 Evaluation of LAC Higher Education Scholarships Program study and the U.S. culture together, but also learn about each other’s countries, cultures, professional challenges and language differences. This arrangement builds a strong base for international communication and potential collaboration, as is borne out by the findings concerning past recipients’ international interactions, reported in Section B.6 below. Two-year students’ first priority upon arrival is gaining English language proficiency sufficient to complete the coursework in their field of study successfully. Given the program’s lack of any English language requirement, language training constitutes the largest part of the first year of the two-year program, with learning reinforced by daily practice through interaction with the U.S. host family and campus life at large. This component is also designed to reinforce the recipients’ exposure to U.S. culture. By the second year, students are expected to be fully functional in order to be mainstreamed into coursework they do alongside their U.S. student counterparts. By all indications, almost all reach this point. Academic training for two-year SEED scholars has evolved away from the standard two-year university degree issued by community colleges (Associate of Arts or Associate of Science) that was often available in earlier years of the program, to a more tailored curriculum generating certificate-based completion documents. This evolution came for several reasons: a. Most U.S. states require general education coursework to earn the associate degree—liberal arts courses in sociology, English literature, etc.—that is of little direct application for most CASS/SEED recipients. Furthermore, such courses are generally not eligible for credit transfer should the recipient later decide to pursue university study back home. b. Coursework re-structuring to produce a series of specific and clearly understood professional certificates that could equip the returning scholars better as they seek employment requiring relevant training or skills. For example, a recipient in the business program now returns with a portfolio that includes individual certificates in leadership, marketing basics, entrepreneurship, small￾and medium-enterprise management, and service-learning. c. Savings in tuition costs in some cases, depending on the host college. For example, one college visited benefited from the state’s redefinition of eligibility for in-state tuition status, which included CASS/SEED recipients, who previously had been required to pay out-of-state rates. The CASS and SEED programs take a unique approach to integrating their students into U.S. culture and communication with language facility that very clearly meets these students’ needs. Table 7 compares these program elements with those of other programs reviewed. The current tendency of the two-year SEED program is to mainstream students into existing college coursework as much as possible. This approach was less prevalent in the past, when CASS students were offered more tailor-made courses. College administrators reported that the mainstreaming approach has reinforced students’ independence and increased their ability to use English and become more integrated into the student body at large. Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 21 Table 7: Housing and English Language Proficiency of CASS/SEED Recipients Compared to Those of Other Programs CASS/SEED Other Programs Observations US Housing Strong emphasis on US host community integration (homestays, friendship families, community activities). An integrated homestay feature is at best short-term in most other programs. Higher housing costs result and possibly fewer opportunities for cultural/social integration This is one of CASS/SEED’s strongest components—well beyond any other model reviewed. It is a cost-effective approach It is probably a strong contributing factor to positive perception of the United States and certainly contributes to mutual understanding between US citizens and those they host. English Language Training Used primarily as a tool for academic and community access, but upon return home, English competency often becomes an outcome that provides job access and advancement. All other programs reviewed feature an English language proficiency minimum and “top-up” training to meet admissions requirements based on Test of English as a Foreign Language (TOEFL) scores. CASS/SEED’s approach, while higher-cost in time and resources, is a unique approach to increasing capacity of the targeted population. On both campuses visited, faculty and administrators said that instructors view SEED students as bright, hard-working and often academically stronger than their U.S. classmates. Their active participation and critical thinking were cited as an inspiration in some cases to their U.S. counterparts, as evidenced in the vignette below. At Kirkwood Community College, which hosts SEED students in the field of agriculture, a class in pest and weed management was holding a debate about the best approach to weed management in corn crops. While the local students—many of whom grew up on local Iowa farms—argued for rapid clean-up using a weed management spray (“Round-Up”), a SEED student spoke up to remind classmates that “the cost of one application of that spray is higher than my family’s annual income—we have to be more resourceful than that in finding solutions.” This “teachable moment” for the U.S. classmates of this SEED student was cited as the kind of enrichment host colleges often seek—and gain—for their local students and communities. Some dedicated courses related to leadership and development of a Community Action Plan continue to be set up specifically for SEED students for their work after the return home, and sometimes there is coursework that targets the scholars’ specific profile and learning needs. For example, there may be computer courses given early on in students’ native languages, since many arrive with little to no computer literacy; there may be study-related English for special purposes (e.g., Agricultural English), designed to complement the academic coursework students are taking to reinforce specialized and technical vocabulary used in the field of study. Evaluation of LAC Higher Education Scholarships Program Page 22 Evaluation of LAC Higher Education Scholarships Program Table 8: Exposure to U.S. Culture and Networking for CASS/SEED Recipients Compared to Other Programs Exposure to US Culture & Networking CASS/SEED Other Programs Observations Informal exposure through homestay and community service activities is strong and highly appreciated Formal components (seminars and travel to Washington DC) were simplified or eliminated due to cost. The SEED program has recently hosted regional leadership workshops on an annual basis, in which 3-5 college groups come together at each site. Many recipients experience similar local cultural exposure, but it is often limited to the campus context and interaction with students and faculty. Several other programs provide formal seminars and study travel to Washington, DC and elsewhere (FB, MFP, JJWBFG, CCI). Beyond content delivered, this experience further bonds recipients across campuses for future networking and emphasizes program identity overall rather than just campus loyalty. Other programs that include large-group seminars also frequently mention the benefit of national gatherings that reinforce the U.S. cultural element. It appears particularly valuable for the creation of a scholarship’s “brand” and developing recipients’ loyalty to the program overall, and not just the institution hosting them. National gatherings also allow students to interact with others from their home country and to reinforce the development of ties begun during the pre-departure events. Another reality that many two-year students have experienced is that of trauma in their lives growing up at home—whether through violence in their communities or inside their own families, through disjointed or dysfunctional family systems, or through substance, sexual or child abuse. The CASS/SEED experience presents many recipients with a safer, more peaceful way of living in the safety of their U.S. communities and away from the challenges of their lives back home. This “takes the lid off” their previous, often tumultuous, living context, as some SEED administrators described the phenomenon. Over the years the importance of this has been increasingly recognized by campus administrators and CIED staff, and the program has made more allowances and effort to address the needs of the “whole person” that enters and goes through the program at a critical time of self-development as a young adult. It was argued that concern and support builds the recipient’s capacity to address those situations at home more effectively and to gain insights into the needs in their communities that will equip them for their role as change agent. Unlike U.S. four-year institutions, whose transcripts generally are recognized and understood worldwide, the community college concept is not broadly understood outside North America, and standard documents issued by them often have been misunderstood or found unacceptable. Therefore, CIED and its umbrella institution, Georgetown University, have over the years established a number of credit transfer protocols with educational authorities and/or specific universities in the recipients’ home Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 23 countries. These accords have been designed to facilitate the acceptance of CASS/SEED scholars’ U.S. coursework credits, and training for credit, towards continued undergraduate study at home. National coordinators reported that the protocols have often facilitated returning alumni’s access and advancement in further university study. Some protocols, however, have lapsed and have not been renewed. Table 9 below shows how these programs have accommodated the trainees’ job integration needs back home through an academic training approach that is in synch with the needs and expectations of the job market there. Table 9: Technical and Academic Training Approaches Compared to Other Programs Technical vs. Academic Training CASS/SEED Other Programs Observations Emphasis on academics has been reduced over time in favor of applicable technical skill-building. Degrees replaced with certificates. Most other programs based in higher education settings stress degree objectives (FB, EMFP, CCI, JJWBFG). Technical skill-building is sometimes an added component. The rationale for moving away from academic credential makes sense; employment and advancement rates of past recipients bears this out. Post-program Preparation All the personnel interviewed at CIED and the two colleges visited stressed how important the re-entry preparation and job-hunting process is for the two-year program recipients. “We spend Year 1 trying to make them comfortable here, but as soon as Year 2 begins, the focus is already on preparing to go home and get to work,” was the comment of one college coordinator. Likewise, all three in-country coordinating teams interviewed related how essential was their task of supporting, assisting and motivating returning two-year recipients in their job search and reintegration. Beyond the re-entry seminars organized shortly after all recipients return home, coordinators highlighted the considerable efforts they expend providing outreach to job contacts, making job referrals and giving references. All coordinators related that with the program’s success and reputation after many years, employers – some already employing CASS/SEED alumni, others themselves CASS/SEED alumni –now often contact them about potential recent returned recipients who might fill a given position. While both college administrators and national coordinators stressed the importance of the post￾program phase of the CASS/SEED scholarship to its ultimate success and impact, both groups reported that less is being done under SEED than in earlier years with CASS, and they found this to be a loss to the program’s impact. They also pointed out that there was little to no overall coordination or information-sharing about such efforts and accomplishments. College administrators and national coordinators indicated a desire to know and share more information about the post-program phase. This perceived lack runs counter to what was observed in comparable sponsored scholarship programs reviewed, whose investment in post-program activities and alumni support have grown over time. While CIED reported that there is a dedicated intranet highlighting program activities and updates, it needs broader usage and interactivity to allow for such connections to be made easily. Evaluation of LAC Higher Education Scholarships Program Page 24 Evaluation of LAC Higher Education Scholarships Program The recent emergence of social media networks, such as Facebook or Twitter, has allowed easier ongoing connections and updates between past students and the individuals they knew in their host communities, including host families, faculty and program administrators. Several individuals even reported that connections they had lost with past participants were revived after a long period, thanks to the increased ease of networking through Facebook. The Bureau of Educational Affairs of the State Department has opened its alumni web site to all past scholars to the U.S., including USAID-supported scholars, and some CASS/SEED alumni were reported to have joined that network as well. This type of alumni network, however, was only opened to USAID alumni during the past six months and is available only to those who choose to register for it and participate—CASS/SEED alumni appear to be doing so, as the number currently registered on the State Department’s site totals 412 in late 2012. Program administrators proposed a number of ways that could increase the mutual knowledge and information-sharing: • An official CASS/SEED alumni website; • Summary reports or success stories disseminated more broadly by CIED; • Officially organized events bringing alumni together with both national coordinating staff and U.S. campus representatives, etc. It was reported that USAID supported more efforts along these lines in the past, and a number of previous alumni support efforts and events were described. None seemed to be part of a broader, coordinated effort to address the need, however, even though it appeared that the desire to do so was unanimous.10 In comparison with other programs reviewed, (see Table 10), CASS/SEED’s alumni are less well supported after they return home than are the past participants of other programs. The pressures and complexities of returning home, seeking a job and possibly moving away from the home community diminish focus on the completion of the Community Action Plan for at least a third of the youth recipients in the programs. 10 USAID has recently included former exchange visitors to join the State Department virtual alumni community, and CASS recipients from Nicaragua recently were competitively awarded $25,000 to implement a community development project. Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 25 Table 10: Community-focused Projects Component of CASS/SEED Compared to Other Programs CASS/SEED Other Programs Observations Integration of Community￾focused Projects upon Return Home Strong framework for creation and implementation of community projects through the Community Action Plan component. Implementation and impact are dependent on external factors related to employment, politics, or resources. Some other programs provide more sustained support (human and financial resources) to the post-program component in￾country (EMFP, CCI). Community Action Plans are not fully implemented by at least one-third of the recipients upon return home. Budgeting and Cost Factors While the cost factors for the CASS/SEED programs are similar to other sponsored scholarship programs, there are some elements of costing that are unique when compared with those of other scholarships. This section considers those elements individually and in comparison to other programs’ approaches. International travel is purchased and coordinated centrally by CIED, which reports that it is able to negotiate lower group rates for recipients traveling together from each home country. This is not typical of many sponsored scholarship programs, whose scholars often travel individually to and from the U.S. SEED’s approach offers the possibility of cost savings, and also makes the travel process part of the program experience—recipients gather in the home country for their send-off and, likewise, go through the same process upon return. CIED has its U.S. host institutions appoint a representative to travel to Miami to meet recipients coming into the U.S. by country groups, where they are remixed into campus groups and accompanied by the campus representative to the host community. This is a unique model, creatively merging the program’s complex travel logistics with integrated host institution support from the start. Passport and visa coordination are handled by the National Coordinating office in the sending country, which not only assists recipients in obtaining their passports and completing visa application materials, but also coordinates visa interviews and coaches recipients about undergoing the now￾mandatory personal visa interview by the U.S. Consulate.11 Since SEED is the only known sponsored program that does not require prior English language competency for study in U.S. higher education, the national office also negotiates—where needed—the U.S. Consulate’s waiver of this usual requirement of language competency. The cost of the J-1 (Exchange Visitor) visa (currently $160 to apply for the visa) does not seem to be waived in SEED’s case, as they 11 Prior to 9/11/2001, such interviews were waived in some cases. Evaluation of LAC Higher Education Scholarships Program Page 26 Evaluation of LAC Higher Education Scholarships Program are for some State Department-funded programs (notably Fulbright), resulting in an additional cost per scholarship. Cost-sharing by host colleges currently must total at least 25 percent of the federally funded amount requested, according to published application guidelines. In CIED’s 2012 hosting competition, the announced contract caps per hosting institution totaled $604,800 for the two-year program; $330,600 for the 12-month program; and $174,000 for the six-month program. At 25 percent, minimal required contributions from the host institutions total $151,200, $82,650 and $43,500 respectively. The approach taken to meeting these high cost-share challenges tends to focus on the host institution waiving or reducing its Negotiated Indirect Cost Recovery Agreement (NICRA), donation of staff time, and obtaining discounts or cost waivers for various community-based services for recipients (housing, health care, shopping discounts, etc.). Neither campus visited indicated that it used the value of donated staff overtime, although both reported that the programs’ coordination regularly demanded more than 40 hours per week from key staff. Table 11 reflects the colleges’ actual contributions to the two-year program over the period studied. Table 11: Costs per Recipient and Proportion of Cost-sharing over the Period of the Study Program Cycle Per-Recipient Federal funding Per recipient Institutional Cost Share Total US Cost per Recipient % Cost-share from host institutions CASS 2004 $36,331 $22,069 $58,400 37.8% CASS 2005 $36,479 $20,075 $56,554 35.5% CASS 2006 $36,818 $18,696 $55,514 33.7% CASS 2007 $37,491 $21,916 $59,407 36.9% CASS 2008 $39,212 $19,370 $58,582 33.0% SEED 2009 $38,294 $20,905 $59,199 35.3% All costs cited above are average per-year costs for the 2-year program; donated staff overtime is not included in cost-share figures. SEED’s closest counterpart scholarship, CCI, currently requires only 15 percent cost-share, possibly making the opportunity to host disadvantaged students through CCI a more attractive option for some host institutions. At the colleges visited, SEED administrators reported that cost-sharing was not generally offered on tuition and fees, which is often the case with other sponsored scholarships in four-year and graduate￾level institutions. There, sponsored scholarship programs generally seek tuition reduction or outright tuition waivers, but do not necessarily expect reductions of negotiated indirect cost rates. There was a sense in both institutions visited that the pressure point for cost-sharing (sometimes up to 50% of legitimately billable costs when NICRA rates are taken into consideration) had been reached, and any additional pressures to waive or share costs for the program could well result in institutions withdrawing from future participation. Anecdotally, it was reported that some other institutions that had been viewed as particularly strong in the program had already come to such a point and no longer participate in hosting CASS/SEED recipients. It appears, on the other hand, that some student allowances are levied through the NICRA whereas such payments often are made by cooperating agencies such as CIED as “pass-throughs” (meaning no overhead is applied to them). Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 27 CIED’s own cost-sharing with USAID funding, likewise, focuses on reduction of its final NICRA rates. CIED’s home office in Washington, DC, qualifies for a significantly higher NICRA rate than its overseas coordinating offices, but CIED cost-shares this significant difference as part of its contribution to the USAID-required cost-share quota. Staff time for the full-time SEED staff, however, is fully charged against the contract’s budget, but, as mentioned previously, senior staff members donate part of their oversight time as a contribution to cost-share. CIED’s disbursement of funds and travel costs directly to participants, unlike the host colleges, is treated as “pass-through” and not levied with NICRA. Program Elements Compared with Other Similar Sponsored Programs Nine programs were closely examined from the several dozen that exist because they parallel CASS/SEED in some way: similar program objectives, Latin America focus, community college-based, or targeting disadvantaged/marginalized populations. (A full summary of each program and a comparison with CASS/SEED is in Appendix 2.) The CASS/SEED program models are unique in comparison with other sponsored higher education￾based participant training models. Many youth-oriented technical training offerings identified through the benchmarking exercise were found to target upper-level college students or college graduates, and place them in situations where they apply training already obtained somewhere else. These programs mostly involve well educated students from industrialized countries (e.g., the State Department’s Congress￾Bundestag Young Professional Exchange between the U.S. and Germany, and the Cultural Vistas program, offering a technical training experience in the U.S. on a fee-for-service basis). The program most similar to SEED today, as mentioned previously, is the Community College Initiative scholarship, first launched by the U.S. Department of State in 2007 in collaboration with the Community Colleges for International Development (CCID) Consortium. Its inspiration was a Summit of U.S. University Presidents, hosted jointly in 2006 by then-Secretary of State Rice and Secretary of Education Spellings, which focused on how to attract foreign students and scholars to the United States, the marketing of U.S. higher education programs abroad, and reaching out to underserved populations. The CCI scholarship, like SEED’s youth program, targets disadvantaged youth. As does SEED, CCI places its students in mixed-nationality cohorts in a U.S. community college for one year for the purpose of job skill development and exposure to U.S. culture. Recipients come from a broader range of countries than CASS/SEED—certain countries across Africa, the Middle East/North Africa and South Asia also participate along with certain Latin American countries. Since its inception in 2007, over 1,400 students have benefited from the CCI scholarship. The two programs themselves differ in significant ways: 1. CCI does not include a host family component; 2. CCI requires a fairly high level of English proficiency from applicants; CCI’s English proficiency requirement currently is 425 on the paper-based version of the Test of English as a Foreign Language (TOEFL); this score is below that generally required by U.S. universities for undergraduate admission for degree-seeking students12, but does reflect significant previous study of English; 3. CCI places more focus on coursework and less focus on community impact back home; 12 U.S. universities generally require a TOEFL score in the range of 500-600 for undergraduate admission. Evaluation of LAC Higher Education Scholarships Program Page 28 Evaluation of LAC Higher Education Scholarships Program 4. Recruitment of candidates is primarily done online and includes students who have had some previous university training, and in some cases, even a first university degree; 5. There are national seminars included in the CCI program that gather all recipients during their U.S. stay; 6. CCI offers a more integrated and better funded approach to alumni support back home; this is ensured through the more robust presence of staff in U.S. Embassy Cultural Affairs offices and/or Fulbright Commissions, who play this in-country role for CCI alumni. Despite these differences, like CASS/SEED, the emphasis in the selection process was described by its director as being on “the candidate’s potential, not just on background.” Key Points • USAID and implementing partner CIED have created a scholarship program that, in many ways, addresses the needs and potential of talented, but underserved, individuals from primarily rural communities. • Despite flat funding for CASS and SEED over the last decade, USAID’s implementing partner and its sub-contractor colleges have managed to maintain recipient numbers through a high level of efficiency and increasing cost-share contributions. • CIED has developed very effective mechanisms for identifying highly qualified candidates for the CASS/SEED scholarships. Those candidates who are “finalists,” but who do not ultimately benefit from the scholarship, are also high-potential individuals. • CIED’s college partners are highly committed to the program’s goals and outcomes, and have been willing to contribute generously in cost-shares, staff time and community involvement for the program’s success. Current economic realities in the U.S. may discourage them and other potential colleges from future participation, however, due to the current high cost-share burdens. • The CASS and SEED programs have outstanding approaches to recipient involvement in U.S. communities and the learning opportunities they gain from them. Follow-on once home is weaker than it might be in order for the recipients’ Community Action Plans to be as fully implemented as intended. • Post-program follow-up and integration of alumni within home countries, with U.S. counterparts and with alumni in other participating countries, are less robust than in other programs reviewed. Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 29 B. RECIPIENTS Introduction This part of the report covers the results of the field-based research of past CASS and SEED recipients and the post-facto proxy control group, the fully qualified selection finalists who were not awarded a fellowship or declined it if an award was offered. Changes and program impacts are first reported on the recipient group (alumni), followed by a discussion and comparison of findings between the recipient and finalist (non-awardee groups). Finally, general features of the impacts experienced by both groups are discussed along with career pathways they have experienced to date. Change within Recipient Group This section presents within-group changes in the scholarship recipients as a result of their training. Results are discussed in terms of the program assumptions and expected outcomes as stated in CIED documents, as summarized in Section A above, starting from time of application to August 2012, when the evaluation was conducted. The length of time since program completion for the recipients surveyed varied from less than one year to eight years. Recipients were grouped together and then stratified based on areas of analytical interest. Where appropriate, comparisons across years, among countries, or between groups are made. Such areas include: residence, ethnicity, employment, certification of training in the home country, year of completion of the program, completion of the action plan, leadership, and opinions of the United States. 1. Where do past scholarship recipients live and how do they identify ethnically? An objective of targeting rural dwellers for technical and professional training is that many of the CASS/SEED scholarship recipients will continue to live and work in rural areas, thereby contributing to economic development. A normal tendency in developing countries is for individuals with university training to gravitate to urban centers, where there is greater demand for their new skills and generally higher paying employment. The residence of the recipients, as a group, was found not to have changed. Seventy-one percent of the recipient sample lived in rural areas at the time of application and the same percentage of the sample continued to dwell in the countryside at the time of the study. When individual recipients were examined, 82 percent of those who had lived in rural areas at application continued to live in these areas. It is sometimes possible that individuals from an ethnic minority change their self-identification following higher education or experience abroad. This phenomenon has not occurred with the scholarship recipients, compared to their ethnic self-description at application; when asked to self-describe their ethnicity during the study interviews, only a few more identified themselves as Hispanic, but this difference is not significant, Table 12: Recipient Change in Residence and Ethnicity at Entry and Currently Area/ Time Residence Ethnicity Rural Urban Hispanic Indigenous Application 169 (71%) 61 (25.6%) 151 (63.4%) 87 (36.6%) Current 170 (71.4%) 66 (27.7%) 161 (67.2%) 78 (32.8%) Source: Databases - JBS, LAC HE Evaluation 2012. Percentages may not equal 100% owing to non-responses. Evaluation of LAC Higher Education Scholarships Program Page 30 Evaluation of LAC Higher Education Scholarships Program 2. Are recipients employed at a higher rate than before CASS/SEED? Since the professional development component of CASS/SEED is designed for individuals who are already employed, the examination of employment deals mainly with recipients in the two-year program. Figure 3 compares the number of recipients in this group employed at time of application with the number currently employed. Employment of the two-year recipients went from slightly less than half to over 80 percent, which represents a significant increase in employment. For the professional development scholars, all were employed at application and no significant changes in employment occurred. Figure 3: Employment Rates of Recipients at Time of Application to the Two-Year Program and Currently13 Since employment is often a result of time in the workforce, the two-year recipients’ employment rates were examined by year of entry into the program. As shown in Table 13, employment rates are relatively high in each year, ranging from 73 percent to 95 percent for recipients in the first six years included in the study. The exceptions are recipients who entered the program in 2010 and had only just returned to their countries at the time of the study. Even in this group, a majority (65%) had found employment by August of 2012, just weeks after returning home from the U.S. Table 13: Percentage of Employed Recipients in Two-year Program by Year of Entry Year/ Status 2004 2005 2006 2007 2008 2009 2010 Employed 13 (81%) 18 (95%) 15(88%) 11 (73%) 18 (95%) 17 (85%) 11 (65%) Unemployed 3 (19%) 1 (5%) 2 (12%) 4 (27%) 1 (5%) 3 (15%) 6 (15%) Source: Databases - JBS, LAC HE Evaluation 2012 13 After CASS/SEED, 103 recipients were employed which is a statistically significant increase. It is significant at p ≤ .01 49.1% 50.9% Time of Application 83.7% 16.2% Currently Employed Unemployed Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 31 When all recipients were examined, no differences were found by gender; rates of employment for both men and women increased significantly. Males had a slightly higher level of employment at application (51.9%, compared to 46.7% for females), but the difference was not significant. The change in employment rate is almost exactly the same, with an increase of 34.8 percent for women and 34.3 percent for men. Therefore, the difference has been maintained in terms of current employment status, with males at 86.2 percent employed and females at 81.5 percent. Changes in employment by ethnicity were significant for both Hispanic and indigenous scholars in the two-year program. However, as shown in Figure 4, indigenous scholars had greater gains in employment than did Hispanics. Recipients of indigenous background were significantly less likely to be employed at time of application than Hispanics. After completing their training, however, both groups were employed at the same rate. Employment for Hispanics increased by 32.1 percent, and employment for indigenous recipients increased 57 percent. Figure 4: Change in Employment Rate for Two-year Program Recipients by Ethnicity14 When the type of employment held was examined, a significant change was found from time of application to the present day. At application, the great majority of those employed (81.2%) were in support or low-level technical jobs, and only 17 percent in skilled technical employment, with slightly over 1 percent in management positions. Currently, 37.7 percent of those employed are in support and low-level technical jobs and 48.3 percent are in skilled technical or supervisory/management positions. Although promotion at one’s job may be associated with longevity, this is not entirely the case for two￾year recipients. As shown in Table 14, with the exception of 2008, a greater percentage of recipients found jobs in skilled technical or management areas than in lower level positions. 14 The number of employed indigenous recipients increased from 5 to 20. That change is statistically significant at p ≤ .01 51.7% 83.8% 26.3% 83.3% 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% Application Current Percentage of Recipients Employed Hispanic Indigenous Evaluation of LAC Higher Education Scholarships Program Page 32 Evaluation of LAC Higher Education Scholarships Program Table 14: Employment of Recipients in Two-year Program by Job Type and Year of Entry Year/ Type 2004 2005 2006 2007 2008 2009 2010 Support/Low technical 6 (37.5%) 7 (36.9%) 4 (23.5%) 5 (33.3%) 13 (68.5%) 6 (30%) 3 (18.3%) Skilled Technical/ Management 7 (43.9%) 11 (57%) 11 (64.7%) 7 (46.7%) 5 (26.4%) 10 (50%) 8 (50%) Source: Databases - JBS, LAC HE Evaluation 2012. Percentages may not equal 100% owing to non-responses. Past recipients continue to live in rural areas for the most part, but they could potentially commute to work in urban centers. As can be seen in Table 15, this has not generally occurred. The vast majority of professional development scholars, most of whom are in education, continue to work in rural areas. A majority (55.7%) of two-year program recipients work in rural areas rather than in urban centers, but that percentage is significantly lower than the percentage of the professional development scholars who work rurally (83.9%). When employment patterns were examined by country, this lower percentage was largely the result of 52.2 percent of the El Salvador recipients working in urban areas. In both Guatemala and Mexico, a majority worked in rural areas. Table 15: Recipient Employment by Location and Group Work Location/ Recipient Group Rural Urban Two-year 58 (55.7%) 46 (44.3%) Professional Development 94 (83.9%) 18 (16.1%) Source: Databases - JBS, LAC HE Evaluation 2012 Percentages may not equal 100% owing to non-responses. The areas of employment among two-year recipients differed somewhat by country, reflecting labor market demand. Table 16 shows that scholars returning to Guatemala were more likely to be employed in agriculture and service jobs than their classmates from other countries, whereas Salvadorans were employed in greater numbers in retail, service, teaching and health, and Mexicans in manufacturing. Table 16: Two-year Recipient Employment by Area and Country Country/ Area El Salvador Guatemala Mexico Retail 11 (32%) 1 (3%) 8 (19%) Public Sector 1 (3%) 3 (10%) 2 (5%) Agriculture 0 (0%) 6 (21%) 4 (10%) Service 6 (18%) 8 (27%) 7 (17%) Manufacturing 4 (12%) 2 (7%) 11 (27%) Teaching 6 (18%) 6 (21%) 5 (12%) Health 6 (18%) 2 (7%) 3 (7%) Trades 4 (12%) 1 (3%) 1 (2%) Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 33 The program aims for two-year recipients to find employment within six months of their return home. As shown in Figure 5, a majority of the respondents had found employment within six months. A slightly higher percentage of female recipients found a job in the target period than males, but the difference was not significant. Figure 5: Time to Find Employment after Return for Two-year Recipients, by Gender15 3. Are the credentials of past recipients recognized in the home country? Recognition of credentials earned abroad is often seen as an important step in professional and academic advancement on return to one’s country. This also has been a key objective of the SEED/CASS programs, as was discussed in Section A. However, Table 17 shows that relatively few of the scholarship recipients in any program were able to have the credentials earned from their scholarship experience recognized. The findings show that only 25 percent of all scholars received official recognition for their work. The majority of those who received credit were two-year recipients. This was a result of obtaining equivalency for courses taken in the U.S. when they returned to school. Twenty-five percent of those who received recognition stated that the ministry where they were employed gave them credit in terms of the official salary scale. The remaining scholars who received credit said that it was through professional recognition in the workplace. 15 Percentages may not equal 100% owing to non-responses. 60.0% 21.8% 3.6% 69.2% 15.4% 3.1% 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% Six Months or Less More than Six Months Hasn't Found Employment Number of Two-year Recipients Male Female Evaluation of LAC Higher Education Scholarships Program Page 34 Evaluation of LAC Higher Education Scholarships Program Table 17: Recognition of Credentials: All Academic Programs Received Recognition/ Academic Program Yes No Two-year 34 (27.6%) 80 (65%) One-year 10 (18.5%) 49 (74.1%) Six-months 5 (15.2%) 26 (78.8%) Total 49 (25.1%) 146 78.9%) Source: Databases - JBS, LAC HE Evaluation 2012. Percentages may not equal 100% owing to non-responses. When asked why no recognition was received for their credentials, a majority (61%) responded that no process existed for obtaining recognition in their workplace. Almost one-fourth said they did not try to secure recognition, and 14 percent said they were starting the process or would do so when they returned to school. The situation of Alicia, a two-year recipient from southern Mexico illustrates some of the problems encountered in trying to gain recognition for credentials from the scholarship program. Alicia was informed by the Mexican Secretary of Public Education (SEP) that her technical studies could not be recognized unless signed by the secretary of education in the state of the U.S. where she had studied. Through her host family she was able to obtain the signatures, but was told that the signed certificate could not be validated until she had her transcript translated at a cost of 2000 pesos and paid 1800 pesos to the SEP. She also had to pay a 1500 peso honorarium to the person that would do the validation at the SEP and pay an 800-peso tax to SEP. 4. Do past recipients have increased leadership and participation in their communities? Becoming a leader and change agent in both occupational and community settings has been an objective of the scholarship program since its inception. In the SEED program proposal, it was stated that these aspects of the scholarship experience would be stressed in the new programming. The two phases of the program were compared on the relative number of scholars who stated that they held supervisory and leadership roles in the organizations where they were employed, and on those who completed their action plans and were members of community organizations. Although SEED was funded in 2008, the first recipients under the program began the program in 2009. Therefore, only two years of data were available for this group. This study defined leadership as the provision of knowledge, information, and/or methods to realize a goal or objective and was examined in terms of supervisory positions and leadership responsibilities in the workplace. Table 18 shows very similar patterns for CASS and SEED scholars in terms of leadership. After the scholarship experience, more than 40 percent of each group held supervisory roles at their jobs, and over 70 percent of scholars in both groups stated that they held leadership positions within their organizations. A slightly lower percentage of SEED scholars held supervisory positions, which might be a result of the time needed to work up to a supervisory role within an organization, given that returned SEED scholars were newer to the workforce. As the overall data are somewhat skewed by the short-term program recipients, who were often selected because of their pre-existing leadership positions in their home communities, the results for the two-year and one-year programs were examined separately. In the two-year program, significantly more SEED recipients stated that they were in supervisory roles than those having gone through CASS (46% Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 35 for SEED, compared to 39% for CASS). For the one-year program, the difference was not significant between the number of CASS recipients and SEED recipients in supervisory roles (39% to 36%). In terms of leadership roles, there were no statistically significant differences between CASS and SEED recipients. Table 18: Leadership Roles in the Workplace of CASS and SEED Scholars – All Academic Programs Leadership/ Program Supervisory Role Leadership Responsibilities Yes No Yes No CASS 73 (45.3%) 81 (50.3%) 115 (73.2%) 31 (19.7%) SEED 32 (42.1%) 36 (47.7%) 58 (76.3%) 7 (9.2%) Source: Databases - JBS, LAC HE Evaluation 2012 Percentages may not equal 100% owing to non-responses. CASS recipients had completed their action plans at a significantly higher percentage than SEED scholars, as illustrated in Table 19 below. This appears to be a result of the short time that SEED recipients had been back in the country, since 46 percent of those who had not completed their action plan stated that it was in progress or that they were looking for resources to complete the plan. Despite not having completed their plan, scholarship recipients under the SEED program were significantly more likely to be members of a community organization than were CASS recipients. When academic programs were examined, differences in action planning were a result of the large percentage of one-year scholars who had completed their plan in CASS. The reverse was true for community membership, as two-year recipients accounted for much of the difference favoring SEED. Table 19: Community Involvement of CASS and SEED Scholars Leadership/ Program Completed Action Plan Member of Community Organization Yes No Yes No CASS 100 (62.5%)* 50 (31.3%) 72 (45.6%) 86 (54.4%) SEED 39 (50.6%) 37 (48.1%) 39 (50.6%)* 35 (45.5%) Source: Databases - JBS, LAC HE Evaluation 2012 Percentages may not equal 100% owing to non-responses. *significant at p ≤ .05 5. Have CASS/SEED recipients’ opinions of the U.S. changed? Yes.—Figure 6 shows that there was a change over time in recipients’ opinions of the United States. About half of the recipients had a positive view of the U.S. at the time of application for the scholarship. Many recalled being worried about discrimination, whereas over 20 percent stated that they had no strong opinion about the country they were preparing to visit. After completing the program, however, nearly 78 percent of recipients responded favorably when asked about the United States. Evaluation of LAC Higher Education Scholarships Program Page 36 Evaluation of LAC Higher Education Scholarships Program Figure 6: Recipients' Opinion of the U.S. before and after Scholarship The case of Eric, a dentist and HIV/AIDS educator, who had a six-month scholarship in 2006, illustrates the trend: When I went to the US, I thought the people were very rigid and discriminatory. After my stay, I realized that they are creative and not rigid at all. Summary • In general, recipients’ residence patterns were stable from before the program to the evaluation, particularly with respect to rural versus urban residence. There was a positive change in employment among two-year program recipients from application to the time of the study. Both men and women increased to an equal extent (about 30 percentage points). Gains were particularly strong for indigenous recipients, increasing by 47 percent. • A majority of two-year recipients found jobs within the target period of six months after returning from the U.S. Furthermore, there was a strong trend towards being employed in higher-level positions. • Over 80 percent of professional development scholars (mainly teachers) who participated in the professional development program continue to work in rural areas, although there are some differences among the countries reflecting local labor market demand. This is significantly higher than two-year recipients (56 percent). • Only 25 percent of recipients had their credentials recognized by institutions in their home countries. Recognition was mainly for credit when continuing their studies. According to the recipients, lack of recognition of their learning experiences was mainly due to lack of formal mechanisms for this in their home countries. • Community participation has increased significantly under SEED, which emphasizes such participation as part of the recipients’ scholarship programs. • Recipients’ positive views of the United States increased from 50 percent at the time of application to 78 percent at the time of the evaluation. 49.8% 19.7% 21.9% Pre-scholarship 77.7% 6.9% 12.4% Post-scholarship Positive Negative Neutral No response Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 37 Comparative Impact This section discusses differences between scholarship recipients and non-recipients in terms of outcomes targeted by the program, such as employment, leadership, community participation and participation in the international community. The identification of a non-recipient or comparison group enables an examination of the possibility that the types of individuals who qualified as finalists for scholarships could have arrived at similar employment, leadership, community participation and international communication opportunities without benefit of the scholarship. 1. Are CASS/SEED recipients more likely to live in rural areas than non-recipients? As mentioned previously, scholarship recipients, upon returning home, remained in rural areas in the same percentage as prior to study in the United States. Similarly, non-recipients maintained their residence in the same percentages as at the time of application. A significantly higher percentage of the one-year recipient group lives in rural areas than non-recipient applicants for that program, per Table 20. On the other hand, the difference between the residence of two-year recipients and non-recipients was not significant. The relatively higher mobility of the two-year group may be attributed to their younger age at the time of the CASS/SEED program experience. Table 20: Current Residence by Group and Academic Program Residence/ Contrast Group Rural Urban Two-year – Recipient 79 (64.2%) 44 (38.5%) Non-recipient 94 (68.1%) 48 (31.9%) One-year - Recipient 44 (81.5%)* 10 (18.5%) Non-recipient 25 (66%) 16 (39%) Source: Databases - JBS, LAC HE Evaluation 2012 Percentages may not equal 100% owing to non-responses. *significant at p ≤ .05 Location of employment follows a different pattern from residence. A significantly higher percentage of past recipients were employed in rural areas than were non-recipients (64% to 51%). As with residence, the difference is accounted for by those in the one-year group. As shown in Table 21, over 90 percent of those recipients are employed in rural areas, compared to 63.4 percent of non-recipients. As a large majority (93%) of both the one-year recipient group and the equivalent non-recipient group work in education, this suggests that a majority of recipients use their increased skills and knowledge in the target area of their program. This differs from a pattern often seen in Latin America where a primary goal for rural teachers is to be transferred to an urban area. No differences were found between the work locations of two-year recipients and the corresponding non-recipients. Evaluation of LAC Higher Education Scholarships Program Page 38 Evaluation of LAC Higher Education Scholarships Program Table 21: Location of Employment by Group and Academic Program Residence/ Contrast Group Rural Urban Two-year – Recipient 58 (47.52%) 46 (37.7%) Non-recipient 62 (47%) 41 (31.1%) One-year - Recipient 49 (90.7%)** 5 (9.3%) Non-recipient 26 (63.4%) 15 (36.6%) Source: Databases - JBS, LAC HE Evaluation 2012 Percentages may not equal 100% owing to non-responses. **significant at p ≤ .01 2. Are CASS/SEED recipients more likely to be employed than non-recipients? Yes. However, since all of the finalists in one-year and six-month academic programs were employed at the time of application, the focus of this discussion is on the two-year program. Table 22 shows that a significantly higher percentage of the overall recipient sample was employed than in the non-recipient sample. This is largely a result of the higher employment percentage of the two-year scholarship recipients, where a ten-percentage point difference was found in employment. One-year and six-month candidates were nearly all employed, whether past recipients or non-recipients. Overall, a significantly higher percentage of female (88.8% to 76.5%) and indigenous (93.6% to 80.7%) recipients were employed than female and indigenous non-recipients. Within the two-year group specifically, no differences by gender were found. However, indigenous recipients in the two-year group were employed at a significantly higher rate than indigenous non-recipients (83.3 to 59.3%). Table 22: Current Employment by Academic Program and Group Employment/ Contrast Group Employed Unemployed Overall – Recipient 216 (90.8%)** 22 (9.2%) Non-recipient 170 (80.6%) 40 (19.4%) Two-year - Recipient 103 (83.7%)** 20 (16.3%) Non-recipient 101 (73.7%) 35 (25.3%) Source: Databases - JBS, LAC HE Evaluation 2012 Percentages may not equal 100% owing to non-responses. **significant at p ≤ .01 While the programs have had a clear impact in boosting employment rates and levels, again, we also see in these comparisons the power of CASS/SEED’s recruitment process. Employment rates for both recipients and non-recipients in the two-year group are much higher than the 20-29 year-old age group of the general populations in all three targeted countries. Within this age group, the rates of economically active individuals in the three countries average only about 40 percent for men and 30 percent for women. (See Appendix 3.) Figure 7 below compares employment rates by year between two-year recipients and non-recipients. Employment is generally high for both recipients and non-recipients who have been participating in the workforce for the longest time. Both groups experienced a drop in employment rates in 2007, with the beginning of the global recession. The non-recipient employment rate was consistently lower than that Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 39 of recipients from 2007-2010. This indicates that fewer non-recipients for two-year programs found jobs during the downturn than recipients. Non-recipients in that group had an average employment rate of 61 percent during the four years compared to an average rate of 80 percent for two-year recipients. Figure 7: Percentage of Two-year Recipients and Non-recipients Employed at Time of Study by Year of Application When type of employment was examined (Table 23), scholarship recipients, as a group, held significantly more skilled technical positions than did non-recipients. The percentage of individuals in supervision and management positions was also higher for recipients. This overall tendency was also true for female recipients, but not for indigenous recipients. Two-year scholarship recipients accounted for much of the difference in supervision and management. No significant differences in job type were found among the one-year recipients and their non-recipient counterparts. The somewhat higher percentage of one-year non-recipients in management positions is a reflection of their residing in urban areas, where there are more opportunities for management positions, than in rural areas, where most recipients still resided. Table 23: Type of Employment by Academic Program and Group Employment Type/ Contrast Group Support/Low Level Technical Skilled Technical Supervision/ Management Overall - Recipient 53 (22.3%) 97 (40.9%)** 66 (28.7%) Non-recipient 70 (34%) 62 (30.1%) 41 (19.9%) Two-year - Recipient 46 (37.7%) 30 (24.6%) 29 (23.7%)* Non-recipient 64 (48.5%) 24 (18.2%) 15 (11.3%) One-year – Recipient 1 (2%) 36 (66.7%) 16 (29.7%) Non-recipient 1 (2.4%) 23 (56.1%) 17 (41.4%) Source: Databases - JBS, LAC HE Evaluation 2012 Percentages may not equal 100% owing to non-responses. *significant at p ≤ .05; **significant at p ≤ .01 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 2004 2005 2006 2007 2008 2009 2010 Percentage of Sample Employed Recipient Non-recipient Evaluation of LAC Higher Education Scholarships Program Page 40 Evaluation of LAC Higher Education Scholarships Program Several additional dimensions of employment were explored to determine the relationship of the scholarship experience with employment. Recipients and non-recipients were contrasted on salary ranges, which were established using public-sector data in each country for low level technical, skilled technical, and supervision or management. These ranges were adjusted, as necessary, so that each salary category was consistent for the three countries. Table 24 includes data for the entire sample and for the two-year program. As can be seen, there is a significantly higher percentage of non-recipients in the lowest salary range when compared to recipients. The two-year recipient group accounts for most of the overall difference. No significant differences were found among the academic programs focusing on professional development. Table 24: Salary Range by Academic Program and Group Salary ranges/ Contrast Group Low- level Technical Skilled Technical Supervision or Management Overall – Recipient 60 (25.2%) 95 (39.9%) 63 (26.5%) Non-recipient 72 (34.1%)* 71 (33.6%) 39 (18.5%) Two-year – Recipient 39 (31.7%) 47 (38.2%) 23 (18.7%) Non-recipient 59 (43.1%) 40 (27.2%) 15 (10.9%) Source: Databases - JBS, LAC HE Evaluation 2012 Percentages may not equal 100% owing to non-responses. *significant at p ≤ .05 Table 25 below presents the mean number of jobs held, promotions received, and raises in salary since participating in or applying to CASS/SEED scholarships for the sample as a whole. Respondents were asked to describe sequentially the jobs they have held since their experience with the scholarship program. Both recipients and non-recipients have held almost two jobs, on average. This average is consistent for the two-year and the one-year program. In the one-year program, jobs were generally described as parallel moves rather than advancement. Recipients received slightly less than one promotion on average, whereas promotions among the non-recipients were nearly non-existent. Similarly, recipients averaged almost one raise per person as a group, while very few non-recipients had received raises. For each variable, the same pattern is found for the two-year and one-year academic programs and in both cases recipients received significantly more promotions and raises. Table 25: Comparison of Job Advancement by Recipients and Non-recipients (n = 449) Contrast Variable Group Number Mean Jobs Recipient 238 1.8 Non-recipient 211 1.9 Promotions Recipient 238 .9** Non-recipient 211 .07 Raises Recipient 238 .8** Non-recipient 211 .02 Source: Databases - JBS, LAC HE Evaluation 2012 Percentages may not equal 100% owing to non-responses. **significant at p ≤ .01 Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 41 In addition to salaries and promotions, respondents were asked about several possessions that are proxy measures for wealth and which also provide an idea about use of technology. As can be seen in Table 26 illustrates how recipients were more likely than non-recipients to own a computer with an Internet connection, a cell phone with Internet capability, and a television with cable. Both ownership of a computer and a television are significant. Table 26: Key Possessions of Recipients and Non-recipients Possession/ Group Computer with Internet Connection Cellphone with Internet Capability Television with Cable Connection Yes No Yes No Yes No Recipient 141 (59.2%)** 96 (40.3%) 75 (25.4%) 166 (69.7%) 124 (52.9%)* 112 (47.1%) Non-recipient 88 (41.5%) 124 (58.5%) 46 (21.7%) 166 (78.3%) 89 (42%) 123 (58%) Source: Databases - JBS, LAC HE Evaluation 2012 Percentages may not equal 100% owing to non-responses. *significant at p ≤ .05; **significant at p ≤ .01 In examining employment, the time to obtain a first job for those in the two-year program was compared to non-recipients who sought employment after not being selected for the scholarship program. This is not a fair comparison, as non-recipients were high school graduates whereas those in the scholarship program had two years of college experience. Thus, as would be expected, recipients were significantly more likely to find a job in six months (65% to 26%), and were also more likely to obtain skilled jobs (35% to 17%) than non-recipients. 3. Do CASS/SEED recipients have higher rates of leadership in their workplaces? Using the definition of leadership as the provision of knowledge, information, and/or methods to realize a goal or objective, its frequency was examined in terms of supervisory positions and leadership responsibilities in the workplace. Less than 50 percent of the finalist candidates for scholarships in either group had supervisory positions at the time of the study, per Table 27 below. Those who received scholarships were ten percent more likely to be in such positions than non-recipients, but the difference was not significant in the overall group. The two-year program accounts for most of this difference since a significantly higher percentage of two-year recipients were supervisors than their non-recipient counterparts. In the one-year program, on the other hand, over 20 percent fewer recipients held supervisory responsibilities, but the difference was not statistically significant. No significant differences were found by gender or ethnicity. Table 27: Supervisory Position in the Workplace by Academic Program and Group Location/ Contrast Group Yes No Overall – Recipient 104 (43.9%) 118 (49.8%) Non-recipient 70 (33.8%) 119 (57.5%) Two-year - Recipient 50 (41%)* 58 (47.8%) Non-recipient 32 (24.1%) 83 (63.9%) Source: Databases - JBS, LAC HE Evaluation 2012 Percentages may not equal 100% owing to non-responses. *significant at p ≤ .05 Evaluation of LAC Higher Education Scholarships Program Page 42 Evaluation of LAC Higher Education Scholarships Program In terms of the number of individuals supervised, there were no significant differences between recipients and non-recipients. Recipients, as a group, reported supervising 11 people on average, and non-recipients reported an average of 9 supervisees. Recipients, as a group, were more likely to have leadership responsibilities than non-recipients. As can be seen in Table 28, the 73.4 percent of recipients who stated they held leadership roles was significantly higher than the percentage of non-recipients (54.4%). Again, this difference was attributable to the two-year recipients who held an advantage of 23.2 percent over their counterparts in terms of leadership roles. For the sample as a whole, a significantly higher percentage of female (75% to 47.8%) and indigenous (75.3% to 63.6%) recipients stated that they held leadership roles in the workplace, whether or not there were in a formal supervisory position. Table 28: Leadership Role in the Workplace by Academic Program and Group Leadership role/ Contrast Group Yes No Overall – Recipient 174 (73.4%)* 40 (16.9%) Non-recipient 112 (54.4%) 76 (36.9%) Two-year - Recipient 79 (64.9%)** 27 (22.1%) Non-recipient 55 (41.7%) 62 (47%) Source: Databases - JBS, LAC HE Evaluation 2012 Percentages may not equal 100% owing to non-responses. *significant at p ≤ .05; **significant at p ≤ .01 Types of leadership roles differed between recipients and non-recipients (Table 29). Both one-year and two-year recipients were significantly more likely to have an appointed position as a leader/supervisor than were non-recipients. Table 29: Leadership Responsibilities by Academic Program and Group Leadership Type/ Contrast Group Appointed Supervisor Directs Projects Demonstrates to Others In Work Committee Overall - Recipient 67 (28.8%)** 54 (23.2%) 33 (14.2%) 20 (8.6%) Non-recipient 32 (15.6%) 41 (20%) 30 (14.6%) 7 (3.4%) Two-year – Recipient 27 (22.5%)** 27 (22.5%) 18 (15%) 7 (5.8%) Non-recipient 13 (9.8%) 19 (14.4%) 19 (14.4%) 2 (1.5%) Source: Databases - JBS, LAC HE Evaluation 2012 Percentages may not equal 100% owing to non-responses. **significant at p ≤ .01 4. Are CASS/SEED recipients more likely to participate in community organizations? An examination of community participation included whether respondents were members of a community organization, as well as the type of organization. Scholarship recipients had a 9.5 percent advantage in membership in a community organization over non-recipients (47.9% compared to 38.4%). Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 43 This difference, however, was not significant at p ≤ .01. As shown in Table 30, participation in a community-based organization was the most common type of involvement. Participation in a church group was the second most common. There were no significant differences in type of organization between recipients and non-recipients. Table 30: Community Participation by Type of Organization and Group Organization Type/ Contrast Group Community Church School Municipal Political Recipient 76 (63.7%) 15 (12.6%) 14 (11.7%) 12 (10%) 1 (1%) Non-recipient 53 (66%) 19 (23.7%) 2 (2.5%) 4 (5%) 2 (2.5%) Source: Databases - JBS, LAC HE Evaluation 2012 Percentages may not equal 100% owing to non-responses. Community participation was also examined by asking two-year recipients about the activities they engaged in while looking for a job on their return. Similarly, those non-recipients that did not immediately enter the workforce or who began to study were asked about their activities. Figure 8 below reflects that 59 percent of recipients were either involved in carrying out their community action plans or doing other types of volunteer activities. Twenty-five percent said that they helped at home or did nothing. Over half of the non-recipients, on the other hand, stated that they helped at home or did nothing, and slightly more than a third were involved in volunteer activities. The completion of the action plan by recipients is another large difference between the activities of the recipient versus non￾recipient groups. Figure 8: Activities of Two-year Candidates while Looking for a Job, by Group16 16 Percentages may not equal 100 percent due to non-responses. 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% Action Plan Volunteer Activities Help at Home/Nothing Intern Percentage of Sample Recipient Non-recipient Evaluation of LAC Higher Education Scholarships Program Page 44 Evaluation of LAC Higher Education Scholarships Program Many of the recipients used their new technical skills to carry out their action plans. The case of José, a two-year recipient from rural Mexico, illustrates this trend. José had decided that on returning to his small town, his action plan would be to open a Cyber Café to help students with their studies. He wrote up a project that would provide several computers and printers that could be used by any student in the town for two hours a day at no cost. He presented the project to the local federal deputy and was provided sufficient resources to run the café for 18 months. 5. Are scholarship recipients more likely to have continued their education? Respondents were asked if they had continued their studies after their experience with the scholarship program. This question primarily sought to determine the number of non-recipients who had applied to the two-year program and who had then gone on to post-secondary education. As can be seen from Table 31, 60 percent of the recent high school graduates continued their education when denied the CASS/SEED scholarship. This finding suggests that the very restricted access to higher education in￾country for this marginalized group, as perceived in the early years of the CASS program, has opened up over the decades. This compares to the 100 percent of two-year recipients that received college training through the scholarship program. As shown in the table below, 44 percent of the two-year recipients have continued to enhance the education gained from the scholarship with additional studies. Almost three-fourths of both recipients and non-recipients in the one-year professional development group have continued studies since their experience with the scholarship program. In all, 54 percent of recipients stated that they had continued formal studies. Table 31: Continuing Education after Scholarship Program Experience Continue Studies/ Contrast Group Yes Two-year - Recipients 54 (44%) Non-recipients 82 (60%)* One-year – Recipients 40 (74%) Non-recipients 30 (73%) Source: Databases - JBS, LAC HE Evaluation 2012 *significant at p ≤ .05 6. Are recipients more likely to have international involvement or contacts? Yes. This section shows the effects of the scholarship experience on increasing recipients’ contact with a wider global society than that of their local community or country, as Table 32 below demonstrates. Membership in alumni associations, communication with classmates, and opinions of the United States are used to examine international involvement. Significantly more recipients were members of former-student associations than non-recipients, overall. This result was due to the two-year program where the difference was also statistically significant. No difference was found in the one-year program sample. It should be noted that despite these statistically significant differences, less than a fourth of any group belonged to any type of former student association. When asked about the type of associations they identified with, the majority of those who responded (59.5%) stated that they were in an organization of Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 45 scholarship recipients from their year or program. Twenty-one percent were in college alumni associations and 12 percent were in the Georgetown scholars alumni group. Table 32: Membership in Ex-student Associations by Academic Program and Group Ex-student Group Member/ Contrast Group Yes No Overall – Recipient 59 (24.1%)** 148 (62.9%) Non-recipient 22 (10.4%) 184 (87.2%) Two-year - Recipient 34 (27.9%)** 73 (59.8%) Non-recipient 14 (10.2%) 119 (86.9%) Source: Databases - JBS, LAC HE Evaluation 2012 Percentages may not equal 100% owing to non-responses. **significant at p ≤ .01 No significant difference was found overall in terms of communication with ex-classmates between recipients and non-recipients. A slightly greater percentage of recipients stated that they communicated with ex-classmates (79% to 68.2%). However, two-year recipients were significantly more likely to communicate with ex-classmates than their non-recipient counterparts (82.1% to 62%), whereas no difference was found in the one-year group. In examining the individuals with whom members of the sample communicated, significant differences were found in terms of international communication. As might be expected, significantly more recipients communicated with former classmates in other countries than did non-recipients (64% to 18.2%). As shown in Table 33, this difference was consistent across program types. A significantly higher percentage of two-year and one-year recipients communicated with classmates in other countries than did non-recipients. Recipients stated that they generally communicated about work and technical issues as they had realized the importance of learning from those in other countries during their scholarship experience. Non-recipients were significantly more likely to communicate with local classmates only. Table 33: Communication Partners by Academic Program and Group Communication Partner/ Contrast Group Classmate in Other Country Teacher Host Family Local Classmate Two-year – Recipient 84 (68.3%)** 7 (5.7%) 5 (4.1%) 6 (4.9%) Non-recipient 22 (16.3%) 2 (1.5%) 2 (1.5%) 59 (43.7%) One-year - Recipient 31 (59.6%)** 7 (13.5%) N/A 5 (9.6%) Non-recipient 8 (19.5%) 2 (4.9%) N/A 24 (58.5%) Source: Databases - JBS, LAC HE Evaluation 2012 Percentages may not equal 100% owing to non-responses. **significant at p ≤ .01 The channels used for communication also differed. Recipients of all academic groups were far more likely to use electronic and social media such as email, text-messaging, Facebook and Twitter, to communicate (76.7% to 26.9%) than were non-recipients. In each academic group, a significant difference of at least 40 percentage points favoring recipients was maintained. Non-recipients were more likely to use the telephone for communication (42.6% to 14.4%). Evaluation of LAC Higher Education Scholarships Program Page 46 Evaluation of LAC Higher Education Scholarships Program 7. Do recipients and non-recipients have different opinions of the U.S. and the CASS/SEED program? As previously mentioned, just under half of the recipients recalled having a positive view of the United States prior to their scholarship experience. Table 34 shows that this percentage contrasts significantly with the percentage of non-recipients who participated in the application process. Almost 83 percent of non-recipients said that they had a positive view of the U.S. at application. Currently, a majority of both groups see the U.S. in positive light, with over 50 percent of both groups characterizing the United States as technologically advanced or offering opportunities. As can be seen, the difference favoring the non-recipients in terms of a favorable perception of the U.S. at application, no longer exists. Non￾recipients are slightly less positive than at time of application to the program. Table 34: Opinion of the United States by Recipient and Non-recipient Opinion/ Contrast Group Positive Negative Neutral At application - Recipient 116 (49.8%) 46 (19.7%) 58 (21.9%) Non-recipient 173 (82.8%)** 5 (2.4%) 29 (13.9%) Current - Recipient 181 (77.7%) 16 (6.9%) 29 (12.4%) Non-recipient 154 (73.7%) 23 (11%) 29 (13.4%) Source: Databases - JBS, LAC HE Evaluation 2012 Percentages may not equal 100% owing to non-responses. *significant at p ≤ .05; **significant at p ≤ .01 8. Do recipients have different aspirations for the future than non-recipients? Respondents were asked about what they expected to be doing in five years. Table 35 shows that more recipients felt that they would eventually supervise or run a business than non-recipients. Only 15 percent of non-recipients indicated that they planned on gaining further training. Almost 30 percent of recipients indicated the same, or twice the percentage. Non-recipients, on the other hand, were more likely to aspire to working in a business in any capacity, rather than aspiring for leadership roles. Table 35: Employment Aspirations in Five Years by Group Aspirations/ Contrast Group Supervise/Run a Business Own a Business Teach or Train Adults Work in a Business Gain Further Training Two-year Recipient 47 (35%) 26 (21%) 10 (9%) 16 (13%) 27 (22%) Non-recipient 30 (23%) 18 (14%) 13 (10%) 48 (38%) 19 (15%) One-year Recipient 22 (42%) 1 (2%) 12 (23%) 6 (12%) 11 (21%) Non-recipient 17 (44%) 1 (2%) 3 (8%) 15 (38%) 3 (8%) Six-month Recipient 7 (23%) 0 11 (35%) 2 (6%) 11 (35%) Non-recipient 4 (36%) 0 2 (17%) 5 (45%) 0 Source: Databases - JBS, LAC HE Evaluation 2012 Percentages may not equal 100% owing to non-responses. 9. What do recipients and non-recipients think of their scholarship/application experience? Respondents were also invited to provide any other comments that they might have about the scholarship program to the interviewer. Slightly more than three-fourths provided comments. As can be Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 47 seen in Table 36, the majority of both recipients and non-recipients were positive about the program. When recommendations for improving the program were made, they were for greater follow-up by the program in terms of assisting communication among former candidates, simplifying the application process and a better process to inform non-successful candidates that they were not selected and the reasons for that decision. As might be expected, a small number of non-recipients also thought the selection process was unfair or distressing. Table 36: Comments on the Program by Group Opinions/ Contrast Group Good Program/ Expand More Follow-up Unfair/ Cruel Notify Non￾recipients Improve Application Process Recipients 155 (76.7%) 25 (12.3%) 0 11 (5.4%) 11 (5.4%) Non￾recipients 88 (61.52%) 7 (4.8%) 12 (8.4%) 17 (11.8%) 19 (13.2%) Source: Databases - JBS, LAC HE Evaluation 2012 Percentages may not equal 100% owing to non-responses. Summary • Over 90 percent of teachers who participated in the professional development program continue to work in rural areas. This is significantly higher than non-recipient teachers’ rate of rural location: 63percent at the time of the study, whereas all were in rural locations at the time of application. • At the time of the study, more recipients are employed and working than non-recipients. • A higher percentage of recipients are employed in management or skilled professions than non￾recipients. These differences are consistent for women, who also received more promotions and salary raises (indigenous non-recipients rarely received either). • Scholarship recipients have been less affected by the recent global economic downturn than have non-recipients. • A significantly higher percentage of recipients hold leadership roles in the workplace than non￾recipients. • There were no differences between recipients and non-recipients in their membership in community organizations or the types of organizations joined. • More of the non-recipients continued their education (60% after being denied the scholarship), compared to 44 percent of the recipients who pursued further education after their scholarships. • A majority of scholarship recipients communicate with former classmates from other countries by use of electronic media. In contrast, non-recipients’ communication networks are mostly local and non-electronically connected. • Recipients’ opinions of the United States became more positive whereas non-recipients, who initially had generally positive opinions of the U.S. at the time of application, became much less so. At the time of the study, about three-quarters of both groups had positive opinions. As noted above, however, some of the non-recipients refused to participate in the study due to Evaluation of LAC Higher Education Scholarships Program Page 48 Evaluation of LAC Higher Education Scholarships Program their negative opinions, and almost 10 percent of the non-recipients who did participate in the study commented that they found the selection process unfair or cruel. C. PROGRAMMATIC IMPACT This section deals with the impact of the scholarship experience on employment and leadership for respondents of different characteristics. Also discussed are the pathways to success in employment and leadership taken by the scholarship recipients, through an examination of the factors to which recipients and non-recipients attribute their current status as professionals and leaders. 1. Employment and Leadership Success In order to examine the overall employment and leadership experience of recipients and non-recipients, aggregate variables were created. Employment success was defined as earning a wage or salary in a skilled technical or supervisory/management position, in which competence was recognized through wage level, promotion and/or salary increase. Leadership success was defined as holding a recognized position in one’s occupation, supervising others and/or having an official decision-making capacity in a community organization or a professional organization. (See the Glossary at the beginning of the report for definitions of terms.) Using these definitions, a five-point “Employment Success” score, ranging from 5 (high) to 0 (low), was constructed. It was based on employment, job type, wage level, promotion and raises. Similarly, a four￾point “Leadership success” score was calculated based on supervisory role, official leadership position, community organization leadership, and professional organization leadership. As can be seen, recipients had, on average, significantly higher rates of success than did non-recipients for both kinds of outcomes. A similar difference was found for the two-year recipients over the non￾recipients on both employment and leadership. One-year recipients had significantly higher employment success than non-recipients. This is probably a result of the recognition of scholarship training on the salary scale of Mexican teachers after the program was founded by the Ministry of Education. It is important to note, however, that this group had a lower aggregate participation level, although the difference is not significant. Recipients in the six-month program did not differ from non-recipients in terms of employment success, since most individuals in both groups were already at a management level. However, recipients had a significantly higher rate of leadership success. Table 37: Employment and Leadership Success by Academic Program and Group (n = 452) Variable/ Group Employment Success Leadership Success Number Mean Number Mean Overall - Recipient 238 3.48** 238 2.26** Non-recipient 214 2.36 214 1.57 Two-year – Recipient 123 3.14** 123 1.87** Non-recipient 140 2.00 140 1.15 One-year – Recipient 54 3.57* 54 2.27 Non-recipient 41 2.92 41 2.70 6-month – Recipient 33 3.96 33 2.87* Non-recipient 11 3.63 11 1.81 Source: Databases - JBS, LAC HE Evaluation 2012 Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 49 Percentages may not equal 100% owing to non-responses. *significant at p ≤ .05; **significant at p ≤ .01 When gender differences were examined, somewhat similar patterns were found in the overall comparisons. As can be seen in Table 38, female recipients were significantly higher in terms of both employment and leadership success. Both two-year and six-month female recipients were significantly higher than non-recipients on both measures. No mean differences were found among recipients when males and females were compared. Table 38: Female Employment and Leadership Success by Academic Program and Group Variable/ Group Employment Success Leadership Success Number Mean Number Mean Overall - Recipient 116 3.45** 116 2.09** Non-recipient 117 2.24 117 1.27 Two-year – Recipient 65 3.06** 65 1.81** Non-recipient 83 1.87 83 1.09 One-year – Recipient 23 3.65 23 2.21 Non-recipient 14 3.50 14 2.07 6-month – Recipient 15 4.27* 15 2.60* Non-recipient 8 3.20 8 1.00 Source: Databases - JBS, LAC HE Evaluation 2012 Values provided are on a 0-5 scale, with 5 being the most successful Percentages may not equal 100% owing to non-responses. *significant at p ≤ .05; **significant at p ≤ .01 Male scholarship recipients were higher on employment than male non-recipients, but no differences were found in leadership success. This reflects the lower mean among the one-year recipients when compared to non-recipients. Male recipients were also significantly higher on both measures in the two￾year group, and one-year male recipients were significantly higher on employment success than non￾recipients (See Appendix 4). A comparison of indigenous recipients and non-recipients (See Table 39 below) shows that indigenous recipients had significantly more employment success, on average, than indigenous non-recipients. In contrast, no differences were found for leadership success among the indigenous individuals. Indigenous two-year recipients were significantly higher on both measures than non-recipients, whereas indigenous one-year professionals were significantly higher in employment, but significantly lower in leadership. This may be a result of the greater percentage of non-recipients in this group working in urban areas, where access to supervisory positions is more likely. There were insufficient indigenous professionals among the non-recipients to compare success in the six-month group. When Hispanic recipients and non-recipients were compared, recipients scored significantly higher on both measures. Hispanics in the two-year program were also significantly higher on both measures. No differences were found among Hispanics in the one-year group. Among Hispanics in the six-month Evaluation of LAC Higher Education Scholarships Program Page 50 Evaluation of LAC Higher Education Scholarships Program group, recipients in this group were significantly higher on the leadership success variable. (See Appendix 4.) Table 39: Indigenous Employment and Leadership Success by Academic Program & Group Variable/ Group Employment Success Leadership Success Number Mean Number Mean Overall - Recipient 78 3.47** 78 2.14 Non-recipient 57 2.30 57 2.21 Two-year – Recipient 24 3.33** 24 1.67 Non-recipient 27 1.96 27 1.29 One-year – Recipient 37 3.43** 37 2.14 Non-recipient 26 2.46 26 2.96* 6-month – Recipient 11 3.81 11 3.27 Non-recipient 1 3.06 1 1.00 Source: Databases - JBS, LAC HE Evaluation 2012 Percentages may not equal 100% owing to non-responses. *significant at p ≤ .05; **significant at p ≤ .01 As employment and leadership success may be a function of time on the job or increased experience, differences in recipients were examined by year. Table 40 supports the notion that time and experience contributes to success. As can be seen, those who were the earliest graduates of the program in the years under study have somewhat higher scores than those in later years. However, none of the differences are statistically significant, even when the earliest years are directly compared to the latest years. Table 40: Recipient Employment and Leadership Scores by Year Year/ Variable 2004 2005 2006 2007 2008 2009 2010 Employment Success 3.14 3.37 3.07 2.89 3.02 2.68 2.62 Leadership Success 2.15 2.16 2.07 1.88 1.55 1.81 1.84 Source: Databases - JBS, LAC HE Evaluation 2012 Percentages may not equal 100% owing to non-responses. *significant at p ≤ .05; **significant at p ≤ .01 Previous experience in leadership and community participation is among the criteria for scholarship candidate selection, along with academic records, recommendations and comments based on a personal interview. Based on these criteria and their own deliberations, screening committees in each sending country assign a selection ranking on a scale of one to four (1=high; 4=low) on all candidates whom they recommend for final consideration to CIED in Washington, DC and its host colleges. Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 51 In order to confirm that there was no selection bias contributing to differences between recipients and non-recipients in this evaluation, selection rankings were correlated with the success measures. Table 41 shows that there was almost no relationship between selection rank score and the success measures. As might be expected, there was, however, the two success measures were significantly correlated. Thus, the recipients were not more likely to have greater employment and leadership success, as defined in this study, at selection. Table 41: Correlations of Selection Score with Success Variables Selection Score Employment Success Leadership Success Selection Score 1 .035 .054 Employment Success .035 1 .375** Leadership Success .054 .375** 1 2. Pathways to Success All respondents were queried about which factors they saw as important to achieving success, and an overall comparison was made of their responses. This analysis revealed that the individual pathways of the different academic groups masked important trends. Table 42, for example, shows the elements of the scholarship experience attributed to being in a leadership position in one’s employment. Slightly over 20 percent of the recipient respondents attributed English language skills as the most important element in leadership. However, over 90 percent of the responses came from recipients in the two-year program who had learned English as part of their scholarship experience. Thus, the analysis was carried out by academic group for the two-year and one-year groups where there were significant numbers to identify robust trends. Table 42: Elements Identified by Recipients and Non-recipients as Contributing to Leadership Success Element/ Group English Skill Training Communication Skill Organization Skill Personal Attributes Job Search Recipient 40 (20.3%) 57 (30%) 21 (10.6%) 26 (13.2%) 12 (6%) 10 (5%) Non￾recipient 4 (3%) 8 (6%) 2 (1.5%) 6 (4.5%) 108 (81.2%) 4 (3%) It is interesting to note the general trends found in the table that were consistent throughout the analysis. Those skills associated with job search (contacts, résumé preparation, etc.) made up only a small percentage of the responses. Skill training was generally the most common response among recipients, and organizational skill, communication skill and English language skill were seen as part of skill training. When recipients were asked why a particular self-identified element was important, 85 percent responded that it was because of the new strategies and techniques they had learned. The other 15 percent responded that it was because of the confidence they had gained living abroad. Non-recipients’ responses differed greatly from recipients: over 81 percent of them attributed their success to personal characteristics such as a desire to improve themselves or personal beliefs. Evaluation of LAC Higher Education Scholarships Program Page 52 Evaluation of LAC Higher Education Scholarships Program Table 43 below illustrates elements of the scholarship experience that were found most effective in reaching the stated objectives of the academic program. In obtaining the first job upon return, English was mentioned by the largest percentage of recipients as the most important element in their scholarship experience. Formal training in English was seen as a saleable skill: the majority of two-year recipients stated that this element was important because it gave them new techniques and strategies that were important to employers. English was often used to obtain a short-term job while looking for something else or as a skill for obtaining a second job. Table 43: Principal Elements Identified by Two-year Recipients and Reasons for Importance Principal Element/ Area Element Why Important First Job English (40%)/Skills (27%) New Techniques (85%) Leadership English (41%)/Skills (27%) New Techniques (77%) Action Plan Skills (55%) Improve Country (52%) Community Participation Skills (49%) Improve Country (100%) Continuing Education Skills (43%) New Techniques (83%) International Involvement Learn from Others (44%) Learn from Other Cultures (81%) Skill training was the second most mentioned element. Recipients stated that since their training credentials were often not recognized; that being able to exhibit required skills was the best way to a job or advancement in a profession. Job search abilities, such as resume preparation, use of networking, and interview skills were mentioned by only about 5 percent of the respondents. In contrast, over 90 percent of the non-recipients stated that personal attributes were important in obtaining their first job after applying for a CASS/SEED scholarship. The case of Giomara, a 21-year-old Mexican graduate of the two-year program in quality control shows the importance of skill training and English. On her return, it took Giomara a month to find a job in her field as a low-level quality control assistant in an automobile plant. There she was told that her studies abroad didn’t make her anything more or less than anyone else and she would have to work her way up like everyone. She obtained a promotion after being the only person who could explain to the head auditor the exact steps in assembling a section of a vehicle. However, the new job did not include a salary raise and she left after three months. She found a job as a line supervisor in another factory, but she left after a month, as her supervisor was transferring much of his work to her. While looking for another position, she began to work part-time in a hotel that catered to foreign visitors to the industrial complex where she was seeking work. She said she obtained the job both for her knowledge of English and of factory operations. She continues to work part-time in the hotel. However, she found her current job at a tire plant in December of 2011. She is a quality control specialist who trains workers for three assembly lines and tracks scrap, or the disposal of damaged tires, through a computerized program that she had become familiar with in her training. Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 53 Among two-year recipients, English was also the most mentioned element for obtaining leadership positions. Again, it was seen as part of a set of technical skills, as more than three-fourths of the recipients mentioned new techniques and skills as why the element was important. Perhaps surprisingly, skill training was cited most often as the most useful element for completing the community action plan and for participating in community organizations, Respondents felt that their new skills contributed to the success of an organization in areas such as social rights, the environment, and education. Of two-year recipients who had continued their studies, 44 percent cited continuing to build on their new skills as the most important element in returning to school. Another significant percentage (20 percent) mentioned personal improvement as the motive for continuing their education. International involvement differed from other elements in that the possibility to learn from other cultures was the most cited element of the scholarship experience. The majority of respondents (81%) said learning from others was the reason this element was important. The remaining 20 percent stated that it was related to work they had done with classmates when in the U.S. The experience of Hortencia, a 24 year-old 2009 recipient in Agricultural Business Administration, illustrates many of the aspects of a two-year recipient’s path to successful employment and community involvement. Hortencia is a 24-year-old graduate from the 2009 program in administration of agricultural businesses. She lives in a small, rural town. Employment opportunities were limited on her return so she accepted a position teaching English in a preparatory school in the Fall of 2011. While working at that job she became active in a women’s organization; initially she organized recreational activities for indigenous rural married women, but currently she is providing workshops on women’s rights. She says that U.S. women, who take their rights as a given, were her inspiration for becoming involved in this area. She also mentioned that her involvement has not allowed her time to complete her action plan. She has recently obtained a job as an agricultural extension worker for a company engaged in amaranth production. She said she obtained the job based on her knowledge of agriculture and her ability to speak the local language. In her position, she organizes work groups and gives workshops on production techniques and use of natural fertilizers. When asked by the interviewer if she had any comments about her experience, she said to tell program organizers that: “I am meeting the objectives of the program: staying in my community and redistributing what I learned abroad.” (“Estoy cumpliendo con el objetivo del Programa: permanecer en su comunidad y retribuirles algo de lo que uno aprendió afuera”.) The most important elements of the scholarship experience identified by one-year recipients are summarized in Table 44 below. As might be expected, given that training was targeted at skills related to the recipients’ jobs, skills training was the most often cited element in all areas except international involvement. As recipients did not have English language training outside of basic survival skills, it was seldom mentioned. Communication skills were generally those that were mentioned second to technical skills. Generally, between 12 percent and 20 percent mentioned these skills. Again, the experience of living abroad and the necessity of “taking the bull by the horns” to communicate were seen as important. As with the two-year recipients, the principal international communication was with former classmates about technical and implementation concerns and experiences. Evaluation of LAC Higher Education Scholarships Program Page 54 Evaluation of LAC Higher Education Scholarships Program Table 44: Most Important Elements of the Scholarship Experience as Identified by One￾year Recipients and Reasons for Importance Most Important Element/ Area Element Why Important Leadership Skills (68%) New Techniques (93%) Action Plan Skills (55%) Improve Country (49%) Community Participation Skills (50%) Improve Country (100%) Continuing Education Skills (53%) New Techniques (72%) International Involvement Learn from Others (48%) Learn from Other Cultures (90%) Summary • Recipients, on average, score significantly higher than non-recipients on measures of employment and leadership success. These differences were consistent for female and indigenous scholars. • Recipients saw skill training as the most important element of the scholarship training in occupational advancement and leadership. In contrast, non-recipients attributed their success to personal qualities such as motivation. • Two-year recipients mentioned English language competency most frequently as an important skill, providing gateway access to first employment opportunities. Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 55 V. CONCLUSIONS AND SUGGESTIONS FOR FUTURE PROGRAMMING This section presents the conclusions drawn from the findings and synthesizes the conclusions from the two principal evaluation questions to provide suggestions for future programming. CONCLUSIONS Conclusions are presented in terms of the two principal evaluation questions under study. The first principal evaluation question is the following: Question 1: To what extent can it be said that LAC/RSD higher education scholarships program provides value for investment? The Findings in Section IV suggest a number of conclusions and possible ideas for future programming in the higher education scholarship area overall, and specific suggestions for the SEED program going forward, in particular. Recruitment The recruitment and selection of CASS/SEED scholars is one of the most thorough to be found across sponsored international scholarships. It successfully identifies the populations targeted by USAID. Costs for this component of the program are comparable or lower than other similar sponsored scholarship programs. The findings regarding the employment levels and career successes of finalist non-recipients, far outstripping that of the general population in their age group, reinforce the excellence of CASS/SEED’s approach in this area. In light of the above, there may be good reason to consider further engagement with such a high￾potential segment of the youth population in each country, by reinforcing their skill development while countering the somewhat negative attitude towards the U.S. created by the loss of the award in the final phase. The Microscholarship and HELP programs observed in the benchmarking exercise, described in Section IV and further outlined in Appendix 2, may offer some useful in-country models for such initiatives. More broadly to this point, further considerations are discussed in the final section of this report. Cost Components The cost components of the CASS/SEED programs compare favorably with other sponsored international scholarships at the higher education level. Nevertheless, CIED may be able to increase its cost-effectiveness in a few areas: • Consider making direct disbursement of student allowances centrally from CIED rather than relying on each host campus to perform this with the additional NICRA that is levied. This alternative approach may reduce some indirect costs overall. • Fees currently disbursed by CIED to U.S. Consulates for J-1 visa issuance could be waived, given published State Department fee policies on US Government-sponsored scholarships (described in Section IV). If obtained, such a waiver could result in total savings nearly equivalent to one scholarship each year. The components of the scholarship program are carefully integrated to help recipients meet the program objectives, as well as their own. This integration is particularly innovative in these areas: Evaluation of LAC Higher Education Scholarships Program Page 56 Evaluation of LAC Higher Education Scholarships Program • Travel and transfer to the U.S. campus, which was not only cost-effective but integrated the new arrivals into the multi-national cohort, the campus community and the host families receiving them. • Anticipation of home country re-entry and preparation for post-program activity—job entry/re￾entry and implementation of the Community Action Plan. CASS/SEED cost-share requirements for sub-contractors are higher than average compared to other publicly funded scholarships. However, the cost-sharing expectations levied on host institutions are in some cases disincentives for their continued participation, especially in light of the recent economic downturn in the U.S. Program Design and Strategy A focus on “keeping the numbers up” in terms of scholarships, despite flat funding and increasing inflation, may be detrimental to reaching some key program goals and outcomes. The subcontractor award duration of just one program cycle may be an area to review. It is possible that such short-term awards, with the amount of time and labor such repeat competitions impose on both CIED and its subcontractors, merits streamlining. They occupy a considerable amount of time to implement at CIED, and burden recipients with repeated paperwork. In some cases, such short-term award periods give subcontractors a disincentive for long-term resource commitments that could qualitatively improve the program on their campuses. Given the findings on non-recipients’ access to higher education in-country, it may be time to consider moving some training offerings in-country or in-region, thereby contributing to capacity-building and further integration of disadvantaged or marginalized youth into the labor market pools. Program Alumni and Post-program activity Compared to other sponsored scholarship programs, CASS/SEED, as it is currently structured, is weak in its follow-up and support of recipients after they return home. This may contribute to the somewhat low completion rate of community action plans. Some excellent models of more robust post-program support and outreach were seen in the benchmarking exercise. Details of some approaches that could be considered best practices in post￾program support are described further in Appendix 1. While access to the State Department-supported alumni website represents a useful new opportunity for global networking of all participant trainees or scholars who have studied in the U.S., it will likely require a more pro-active approach to CASS/SEED recipients’ registering on this network for it to become a significant channel of communication and collaboration among past CASS/SEED recipients. It may be more effective to set up a dedicated and universal alumni site for SEED/CASS, allowing USAID and CIED to monitor and update inclusion of all alumni more reliably and specifically meet their needs and interests. Overall, as summarized above, the evaluation found that the program is well organized, compares favorably with other programs in costs and commitments obtained from implementing partners, and (see Question 2) fulfills objectives of personal advancement for rural dwellers, women and indigenous people, and encourages leadership through use of new strategies and techniques. From this perspective and compared to other programs, the evaluation concludes that the program offers good value. Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 57 Question 2: To what extent have LAC/RSD higher education scholarships enabled participating individuals to become leading change agents in their respective professional fields, communities and countries? The examination of the role of scholarship recipients as change agents was organized around objectives of the program in terms of employment, leadership development, community participation and international involvement. The scholarships have enabled recipients to become change agents principally through the development of new skills such as English proficiency, technology expertise and management capacity. The scholarship experience has given women and indigenous recipients greater access to employment and leadership opportunities and contributed to teachers remaining in rural areas to teach. Employment The scholarship program is contributing to USAID workforce development objectives by providing recipients with an experience that leads to employment opportunities that they would not otherwise have had. This is especially true of the two-year recipients, who have had significantly more employment success overall (and particularly the group’s female and indigenous recipients) than highly qualified candidates for the program who did not receive scholarships. The most important elements of the scholarship experience are skill training and the confidence gained by having functioned in a foreign country. For two-year recipients, mastering the English language is often the pathway to a first job on return to one’s country. The scholarship experience has made scholars aware of the value of exchanging ideas with colleagues with different experiences and encouraged communication with classmates in other countries. The program has had a particular impact on the ability of females and indigenous persons to find employment. Female recipients were employed and held management positions in significantly higher percentages than did non-recipients. Indigenous recipients, who were less likely to be employed at application than Hispanic recipients, were employed in the same percentage at the time of the study. More of them held jobs than did indigenous non-recipients. Participation in the scholarship program has encouraged the program’s professional development recipients, primarily teachers, to live and work in rural areas. Although there was no significant difference between recipients and non-recipients of the professional development program at application, a significantly greater percentage of recipients of professional development scholarships lived and worked in rural areas at the time of evaluation. Recipients in the scholarship program are more likely to be employed and to hold skilled and managerial positions than similar individuals who did not receive scholarships. This conclusion only applies to the two-year program, as those in the professional development programs were by definition fully employed in both groups. Credentialing cannot generally explain the recipients’ pathways to success. Relatively few of the scholarship recipients in any program were able to have the credentials earned from their scholarship experience officially recognized. The few that received recognition were two-year scholars who obtained equivalency for courses taken in the U.S. when they returned to school. Leadership The emphasis of the scholarship program on leadership is reflected in the significantly higher percentage of recipients holding leadership roles in their workplace than non-recipients. Appointment to a leadership role was also more likely among recipients than non-recipients. Evaluation of LAC Higher Education Scholarships Program Page 58 Evaluation of LAC Higher Education Scholarships Program The program has had an impact on the ability of females and indigenous persons to assume leadership roles in the workplace, and this is especially true of the two-year recipient group. Female recipients took leadership roles in significantly higher percentages than did non-recipients. Indigenous recipients held leadership roles at a significantly higher percentage than indigenous non-recipients. Community Development The scholarship program has had little effect on community involvement. About two-thirds of the recipients completed their Community Action Plans and no significant differences were found in the percentage of recipients and of non-recipients who were members of community organizations or in the types of organizations to which they belonged. However, this may be changing with the increased emphasis on developing and carrying out Community Action Plans under the SEED program, as a greater percentage of SEED scholars were members of community organizations than CASS scholars. Given the recent implementation of SEED and the relatively few SEED scholars observed in this study, the effects of the increased emphasis cannot be determined at this time. International Involvement The scholarship experience has given recipients an international perspective in knowledge acquisition and has provided tools for international sharing of information. A majority of recipients have continued to communicate with classmates from other countries after completing their scholarship program. More than three-fourths of these individuals use electronic means such as email, twitter and Facebook for communicating. The most often mentioned reason for communicating was for professional reasons. Non-recipients, on the other hand, communicated principally with local ex-classmates via the telephone. The experience of living abroad provided by the scholarship program has increased positive perceptions of the United States. Only about half of scholarship recipients reported having a positive view or the U.S. at application. Over three-fourths of the recipients had a positive perception of the U.S. at the time of the study. Non-recipients The finalists for the program who do not receive scholarships make up a group of motivated, talented individuals, who among the two-year group have exceeded age-group norms for employment. They remain largely favorable to the U.S. and the program despite not receiving a scholarship. They may well represent an opportunity for additional impact on important rural-based populations who could benefit from investment in a modest local support, generating more impact on the economic and development needs of these communities, which could further boost the impact of the scholarship program overall. CONSIDERATIONS FOR FUTURE PROGRAMMING While the above suggestions focus most specifically on the potential for improving the existing program strategy and design going forward, the considerations below seek to address a longer-term considerations for USAID’s strategy for educational and labor-market support to disadvantaged and rural-based young and mid-career professionals in the LAC Region. USAID should: • Carry out an in-depth study to examine why there appears to be a relative lack of program impact on recipient community involvement. Include in the study the increased emphasis on community involvement and action planning under SEED through the Community Action Plans. • Examine the elements of the teacher professional development program that have encouraged teachers to continue to work in rural areas after training. Such a study should determine the Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 59 feasibility of incorporating elements of the program into in-country teacher professional development programs. • Continue and increase the emphasis on computer training and use, which can facilitate international communication as a strategy to achieve global market participation. • USAID and SEED should consider addressing the potential of the non-recipient finalists, who generally have all of the capacity that the scholars have, but are left behind with no further support or input. A modest level of support in-country might be a way to continue to build the potential of such highly qualified individuals who face the same limitations of access and resources as the selected scholars. A low-cost program involving a “buddy system,” in which a non-recipient works with a recipient on implementing an action plan or a class project, might be a low-cost option to be tested on a trial basis. • Strategies used by other scholarship programs to broaden impact might be considered, given the evolution of resources in the region. In-country alternatives such as educational support of family members, to reinforce intact families and communities, as the Western Union Foundation has done with success, might be considered. Another approach might be to adapt the State Department’s Microscholarship Program, focusing on job-related skill development or undergraduate coursework relevant to economic development and youth employment potentials. • Consistent with USAID’s education policy, consider whether investing in secondary school completion for disadvantaged youth similar to those who had access to CASS/SEED as secondary school graduates may hold merit, given the above-average accomplishments observed in non-recipient secondary school graduates. Evaluation of LAC Higher Education Scholarships Program Page 60 Evaluation of LAC Higher Education Scholarships Program BIBLIOGRAPHY Aguirre Division, JBS International, 2007. “Evaluation of the English Access Microscholarship Program: Final Report.” Washington, DC: U.S. Department of State. Accessed online on 10/20/2012 at http://exchanges.state.gov/englishteaching/eam.html. Aguirre International, 2004. “An Evaluation of the Cooperative Association of States for Scholarships Program (1994-2001); Técnicos en Potenciar: Global Processes and Local Choices in Technical Training.” Bethesda, MD: Aguirre International, Inc. Accessed online on 11/10/2012 at http://pdf.usaid.gov/pdf_docs/PDABZ325.pdf. __________, 1991. “Caribbean and Latin American Scholarships Program--Sixth Annual Report.” Rosslyn, VA: Aguirre International, Inc. Accessed online on 1110/2012 at http://pdf.usaid.gov/pdf_docs/PDABD838.pdf. American Council on Education, Center for Internationalization and Global Engagement, 2013. “MOOCs and International Education.” (Unpublished panel presentation to the Washington International Education Group, Washington, DC, Jan. 15, 2013). Clinton Global Initiative, 2012. “Clinton Global Initiative University, 2012.” (web site). New York, NY: Clinton Global Initiative. Accessed online on 06/28/2012 at http://www.cgiu.org. Georgetown University, 1992. “Quarterly Progress Report, CASP, CASS and Nicaraguan Peace Scholarship Program (NPSP), Vols. 30 & 31.” Washington DC: Georgetown University. __________, 2012. “Center for Intercultural Education & Development Web Page.” Accessed online on 06/01/12 at http://cied.georgetown.edu/seed/index.cfm. ________, 2012. “College Status Reports.” (Seven sample reports from SEED host campuses). Washington, DC: CIED, Georgetown University. _________, 2008. “Higher Education Scholarships and Training for Development — Volume 1: Technical Proposal.” Washington, DC: CIED, Georgetown University. ________, 2012. “Proposal Guidelines – The USAID SEED Program, Cycle 2012.” Washington, DC: CIED, Georgetown University. ________, 2009. “Scholarships for Education and Economic Development (SEED) – Orientation I-II-III Country Coordinators Handbook.” Washington, DC: CIED, Georgetown University. ________, 2009. “Scholarships for Education and Economic Development (SEED) – Administration – College Coordinators Handbook.” Washington, DC: CIED, Georgetown University. ________, 2009. “Scholarships for Education and Economic Development (SEED) – Community Action Plan Guidelines.” Washington, DC: CIED, Georgetown University. ________, 2009. “Scholarships for Education and Economic Development (SEED) – Administration – Recruiting, Screening, and Interviewing Scholarship Candidates – Country Coordinators Handbook.” Washington, DC: CIED, Georgetown University. ________, 2012. “SEED Participant Online Survey Report and Compilation—Northcentral Technical College – Cycle 2010.” Washington, DC: CIED, Georgetown University. Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 61 ________, 2009. “Scholarships for Education and Economic Development (SEED) – Orientation I-II-III Country Coordinators Handbook.” Washington, DC: CIED, Georgetown University. ________, no date. “SEED Leadership Skills Training Materials”. Washington, DC: CIED, Georgetown University. ________, no date. “SEED Participant Acceptance Packet and Welcome Packet Guidelines”. Washington, DC: CIED, Georgetown University. ________, no date. “SEED Participant Evaluation Tools.” Washington, DC: CIED, Georgetown University. ________, 2009. “SEED Two-year Participant Credential File Sample.” Washington, DC: CIED, Georgetown University. ________, 2009. “SEED Participant Re-entry Guide.” Washington, DC: CIED, Georgetown University. ________, no date. “Survey Tools for CASS Alumni Employment and Impact.” Washington, DC: CIED, Georgetown University. Interagency Working Group (IAWG), 2012. FY 2011 Annual Report (includes FY 10 Inventory of Programs). Washington, DC: Interagency Working Group. Accessed online on 10/02/12 at www.iawg.gov. Juárez and Associates, 2011. “Estado Educativo del país: Progreso y retos.” PowerPoint Presentation, developed for USAID/Guatemala. Kirkwood Community College, 2012. “Agricultural Sciences - SEED International.” (web page). Cedar Rapids, IA: Kirkwood Community College. Accessed 06/15/12 at http://www.kirkwood.edu/site/index.php?p=25354. Kissinger, Henry A., 1984. Report of the National Bipartisan Commission on Central America. Washington, DC: U.S. Government Printing Office. No name, 2012. “CASP CASS Seed Ex-becarios Centroamericanos y del Caribe” (social media page). Accessed online on 07/01/12 at https://www.facebook.com/caspcass. Mexico, 2006/2007. “ Resultados sacados de la tasa de aprobacion entidad federativa y nivel educativo segun tipo de servicio” at http://www.inee.edu.mx/images/stories/Publicaciones/Panorama_educativo/2008/Partes/8_acceso_y_tray ectoria.pdf Northcentral Technical College, 2012. “International Programs” (web page). Wausau, WI: Northcentral Technical College. Accessed online on 06/15/12 at http://www.ntc.edu/international￾education/international-programs. US Agency for International Development , 2012. “Programa de Beca Semilla El Salvador” (web site). San Salvador, El Salvador: USAID El Salvador. Accessed online on 07/01/12 at http://www.becasemilla.org ________, 2012. “Scholarships for Youth and Teachers” (web site). Mexico City, Mexico: USAID Mexico. Accessed online on 07/01/2012 at http://transition.usaid.gov/mx/scholarshipseng.html. ________, 2008. “Request for Applications (RFA) No: USAID-M-OAA-GRO-LMA-08-2277—Higher Education Scholarships for Training and Development”. Washington, DC: Grants.gov. Accessed 6/20/2012 at Evaluation of LAC Higher Education Scholarships Program Page 62 Evaluation of LAC Higher Education Scholarships Program http://www.grants.gov/search/announce.dojsessionid=TKlXQGlM2bfzdvh22GgXtvrhQxYhB1vZ2GJylmh xzf6plVHpT4x1!2071562142. U.S. Department of State, 2011. Congressional Budget Justification: Department of State Operations, Vol. 1. Washington, DC: U.S. Department of State. _________, 2012. “Fees for Visa Services” (web page). Washington, DC: U.S. Department of State, Bureau of Consular Affairs. Accessed online on 10/01/2012 at http://travel.state.gov/visa/temp/types/types_ 1263.html. ________, 2012. “State Alumni—Your Global Community” (web site). Washington, DC: U.S. Department of State, Bureau of Educational and Cultural Affairs. Accessed 08/02/2012 at https://alumni.state.gov/landing-page. ________, 2012. “Alumni Engagement Innovation Fund” (web site). Washington, DC: U.S. Department of State, Bureau of Educational and Cultural Affairs. Accessed 08/02/2012 at http://exchanges.state.gov/alumni/aeif.html. World Bank, 2011. “Educational Quality Improvement Project Appraisal Document,” 65034, World Bank. Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 63 APPENDIX 1: SUMMARY TABLE OF THE LAC HIGHER EDUCATION SCHOLARSHIPS SINCE INCEPTION. (Anticipated Funding Completion Date: February 1, 2013) Funding Year Program Participating Countries No. of Scholarships Awarded No, of Host Institutions Fields of Study 1985 CASP Belize, Costa Rica, El Salvador, Guatemala, Honduras, Panama 76 4 Electronics Technology, Industrial Sewing Technology, Agricultural Technology, Electrical Engineering, Food Processing, Machine Tool Technology 1986 CASP Belize, Costa Rica, El Salvador, Guatemala, Honduras, Panama 272 19 Accounting, Agricultural Technology, Biology, Chemistry, Clothing Manufacturing Merchandising, Computer Science, Electrical Engineering, Electronics Technology, Environmental Health, Food Processing, Hospitality Management, International Studies, Machine Tool Technology, Math, Nursing, Quality Control Technology 1987 CASP Belize, Costa Rica, El Salvador, Guatemala, Honduras, Panama 200 16 Agribusiness, Appliance Servicing, Banking & Finance, Biology, Business Administration, Business Management, Chemistry, Clothing Merchandising, Commercial Food Preparation, Community Health, Computer Science, Economics, Education Teacher Training, Electronics Technology, English Literature, Finance, Hospitality Management, Hotel & Restaurant Management, Human Services, Industrial Diesel Mechanics, Math, Occupational Therapy, Offset Printing, Physical Education, Psychology, Quality Control, Visual Arts 1989 CASP Belize, Costa Rica, El Salvador, Guatemala, Honduras 240 25 Accounting, Biology, Business Administration, Clothing Manufacturing Merchandising, Computer Science, Economics & Political Science, Education Teacher Training, Electronics Technology, English Literature, Food Technology, Evaluation of LAC Higher Education Scholarships Program Page 64 Evaluation of LAC Higher Education Scholarships Program Funding Year Program Participating Countries No. of Scholarships Awarded No, of Host Institutions Fields of Study Management Training, Marketing, Nursing, Psychology, Small Business Management 1990 CASP Belize, Panama 25 16 Accounting, Automotive Technology, Biology, Biology/Chemistry, Business Administration, Business Information, Civil Engineering, Computer Programming, Computer Science, Dairy Science, Electronics Technology, English Literature, Environmental Science, Graphic Design, Legal Secretary/Justice, Marketing, Math, Medical Lab Technology, Office Management, Surveying CASP Total Scholarships 1145 CASP Total Funding: $32,961,921 1989 CASS Antigua and Barbuda, Belize, Costa Rica, Dominica, Dominican Republic, El Salvador, Grenada, Guatemala, Haiti, Honduras, Jamaica, St. Lucia, St. Vincent and the Grenadines 332 18 Business Management, Clothing Manufacturing and Merchandising, Electronics Technology, Food Science Technology, Hotel/Restaurant Management, Quality Control Technology 1990 CASS Antigua and Barbuda, Belize, Costa Rica, Dominica, Dominican Republic, El Salvador, Grenada, Guatemala, Haiti, Honduras, Jamaica, St. Kitts and Nevis, St. Lucia, St. Vincent and the Grenadines 551 34 Agricultural Technology, Business Management, Clothing Manufacturing and Merchandising, Computer Business Applications, Computer Science, Education Teacher Training, Electronics Technology, Engineering, Food Science Technology, Hotel/Restaurant Management, Industrial Machine Maintenance, Medical Equipment Repair, Quality Control Technology Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 65 Funding Year Program Participating Countries No. of Scholarships Awarded No, of Host Institutions Fields of Study 1991 CASS Antigua and Barbuda, Barbados, Belize, Costa Rica, Dominica, Dominican Republic, El Salvador, Grenada, Guatemala, Haiti, Honduras, Jamaica, Panama, St. Kitts and Nevis, St. Lucia, St. Vincent and the Grenadines 200 23 Agricultural Technology, Business Management, Computer Science, Education Teacher Training, Electronics Technology, Engineering, Food Science Technology, Hotel/Restaurant Management, Industrial Machine Maintenance, Quality Control Technology, Science, Sign Language Interpreter Training 1992 CASS Antigua and Barbuda, Barbados, Belize, Costa Rica, Dominica, Dominican Republic, El Salvador, Grenada, Guatemala, Honduras, Jamaica, Nicaragua, Panama, St. Kitts and Nevis, St. Lucia, St. Vincent and the Grenadines 325 29 Business Management, Clothing Manufacturing and Merchandising, Computer Science, Education Teacher Training, Electronics Technology, Engineering, Environmental/Natural Resources Science, Food Science Technology, Hotel/Restaurant Management, Industrial Machine Maintenance, Industrial Printing Technology, Medical Equipment Repair, Natural Resources Management, Quality Control Technology, Science 1993 CASS Antigua and Barbuda, Barbados, Belize, Costa Rica, Dominica, Dominican Republic, El Salvador, Grenada, Guatemala, Haiti, Honduras, Jamaica, Nicaragua, Panama, St. Kitts and Nevis, St. Lucia, St. Vincent and the Grenadines 327 30 Agricultural Technology, Business Management, Computer Science, Education Teacher Training, Electronics Technology, Engineering, Environmental/Natural Resources Science, Food Science Technology, Hotel/Restaurant Management, Industrial Machine Maintenance, Medical Equipment Repair, Natural Resources Management, Quality Control Technology, Science, Sign Language Interpreter Training Evaluation of LAC Higher Education Scholarships Program Page 66 Evaluation of LAC Higher Education Scholarships Program Funding Year Program Participating Countries No. of Scholarships Awarded No, of Host Institutions Fields of Study 1994 CASS Antigua and Barbuda, Barbados, Belize, Costa Rica, Dominica, Dominican Republic, El Salvador, Grenada, Guatemala, Haiti, Honduras, Jamaica, Nicaragua, Panama, St. Kitts and Nevis, St. Lucia, St. Vincent and the Grenadines 261 25 Agricultural Technology, Business Management, Computer Science, Environmental/Natural Resources Science, Food Science Technology, Health Administration, Hotel/Restaurant Management, Industrial Machine Maintenance, Medical Equipment Repair, Natural Resources Management, Science, Small and Medium Enterprise Management 1995 CASS Dominican Republic, El Salvador, Guatemala, Haiti, Honduras, Jamaica, Nicaragua, Panama 297 17 Agricultural Technology, Business Management, Education Teacher Training, Electronics Technology, Environmental/Natural Resources Science, Food Science Technology, Hotel/Restaurant Management, Industrial Machine Maintenance, Natural Resources Management, Quality Control Technology, Rural Health Management 1996 CASS Costa Rica, Dominican Republic, El Salvador, Guatemala, Haiti, Honduras, Jamaica, Nicaragua, Panama 285 17 Agricultural Technology, Business Management, Computer Business Applications, Education Teacher Training, Electronics Technology, Environmental/Natural Resources Science, Food Science Technology, Industrial Machine Maintenance, Quality Control Technology, Rural Health Management, Sign Language Interpreter Training, Small and Medium Enterprise Management 1997 CASS Dominican Republic, El Salvador, Guatemala, Haiti, Honduras, Jamaica, Nicaragua, Panama 349 18 Agricultural Technology, Business Management, Computer Business Applications, Computer Sciences, Construction Management, Education Teacher Training, Electronics Technology, Engineering, Environmental/Natural Resources Science, Food Science Technology, Industrial Machine Maintenance, Quality Control Technology, Rural Health Management, Sign Language Interpreter Training Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 67 Funding Year Program Participating Countries No. of Scholarships Awarded No, of Host Institutions Fields of Study 1998 CASS Dominican Republic, El Salvador, Guatemala, Haiti, Honduras, Jamaica, Nicaragua, Panama 281 18 Agribusiness, Agricultural Technology, Business Management, 5) Computer Business Applications, Education Teacher Training, Electronics Technology, Environmental/Natural Resources Science, Food Science Technology, Industrial Machine Maintenance, Natural Resources Management, Quality Control Technology, Rural Health Management, Sign Language Interpreter Training, Small and Medium Enterprise Management, X-Ray Technology 1999 CASS Dominican Republic, El Salvador, Guatemala, Haiti, Honduras, Jamaica, Nicaragua, Panama 251 12 Agricultural Technology, Business Management, Education Teacher Training, Environmental/Natural Resources Science, Food Science Technology, Medical Equipment Repair, Medical Laboratory Technology, Natural Resources Management, Public Health/Vector Control, Quality Control Technology, Rural Health Management, Small and Medium Enterprise Management 2000 CASS Dominican Republic, El Salvador, Guatemala, Haiti, Honduras, Jamaica, Nicaragua, Panama 262 13 Agribusiness, Agricultural Technology, Computer Business Applications, Construction Management, Education Teacher Training, Industrial Machine Maintenance, Natural Resources Management, Quality Control Technology, Rural Health Management, Small Business Management, X-Ray Technology 2001 CASS Dominican Republic, El Salvador, Guatemala, Haiti, Honduras, Jamaica, Nicaragua, Panama 230 12 Agricultural Technology, Aquaculture Technology, Education Teacher Training, Electronics Technology, Environmental/Natural Resources Science, Industrial Engineering Technology, Medical Equipment Repair, Nursing, Public Health/Vector Control, Quality Control Technology, Rural Health Management Evaluation of LAC Higher Education Scholarships Program Page 68 Evaluation of LAC Higher Education Scholarships Program Funding Year Program Participating Countries No. of Scholarships Awarded No, of Host Institutions Fields of Study 2002 CASS Dominican Republic, El Salvador, Guatemala, Haiti, Honduras, Jamaica, Nicaragua, Panama 300 14 Agribusiness, Agricultural Technology, Computer Business Applications, Construction Management, Education Teacher Training, Medical Equipment Repair, Natural Resources Management, Nursing, Quality Control Technology, Rural Health Education, Rural Health Management, Sign Language Interpreter Training, X-Ray Technology 2003 CASS Dominican Republic, El Salvador, Guatemala, Haiti, Honduras, Mexico, Nicaragua, Panama 481 14 Agribusiness, Agribusiness/Agricultural Business Operations, Agricultural and Food Products Processing, Aquaculture Technology, Computer Science/Management Info Systems, Computer Sciences/Business Applications, Education Teacher Training, Electromechanical Technology/Electromechanical Engineering Technology, Electrical, Electronic, and Communications Engineering Technology/Technician, Electronics Technology, Electronics Technology for Manufacturing, Food Science Technology, Manufacturing Technology, Natural Resources Management, Quality Control Technology, Rural Health Education, Rural Health Management, Strengthening Rural Cooperatives 2004 CASS Dominican Republic, El Salvador, Guatemala, Haiti, Honduras, Mexico, Nicaragua 487 13 Agribusiness, Agribusiness/Agricultural Business Operations, Agricultural Technology, Business Management, Computer Information Science, Computer Business Applications, Computer Integrated Manufacturing, Education Teacher Training, HIV Prevention and Outreach, Manufacturing Technology, Natural Resources Management, Quality Control Technology, Rural Health Management, Small and Medium Enterprise Management, Small Business Management, Strengthening Rural Cooperatives Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 69 Funding Year Program Participating Countries No. of Scholarships Awarded No, of Host Institutions Fields of Study 2005 CASS Dominican Republic, El Salvador, Guatemala, Haiti, Honduras, Mexico, Nicaragua 268 13 Agribusiness, Agricultural Technology, Agroproduction for Microfinance Managers, Education Teacher Training, Electronics Technology for Manufacturing, Natural Resources Management, Quality Control Technology, Rural Health Management, Small and Medium Enterprise Management 2006 CASS Dominican Republic, El Salvador, Guatemala, Haiti, Honduras, Mexico, Nicaragua 237 12 Agribusiness, Education Teacher Training, Electrical, Electronic and Communications Engineering Technology/Technician, Natural Resources Management, Quality Control Technology, Rural Health Management, Small and Medium Enterprise Management, Women's Leadership Studies 2007 CASS Dominican Republic, El Salvador, Guatemala, Haiti, Honduras, Mexico, Nicaragua 309 14 Agribusiness, Agroproduction for Microfinance Managers, Computer Sciences/Business Applications, Education Teacher Training, Electronics Technology for Manufacturing, Industrial Engineering Technology, Natural Resources Management, Quality Control Technology, Rural Health Management, Small and Medium Enterprise Management, Youth Services/Administration 2008 CASS Dominican Republic, El Salvador, Guatemala, Haiti, Honduras, Mexico, Nicaragua 311 14 Agribusiness, Education Teacher Training, Electronics Technology for Manufacturing, Industrial Engineering Technology, Manufacturing Technology, Natural Resources Management, Quality Control Technology, Small and Medium Enterprise Management, Small Business Management, Water, Wetlands, and Marine Resources Management, Youth Services/Administration 2009 CASS Dominican Republic, El Salvador, Guatemala, Honduras, Nicaragua 59 3 Education Teacher Training 2010 CASS Dominican Republic, El Salvador, Guatemala, Honduras, Nicaragua 103 2 Education Teacher Training Evaluation of LAC Higher Education Scholarships Program Page 70 Evaluation of LAC Higher Education Scholarships Program Funding Year Program Participating Countries No. of Scholarships Awarded No, of Host Institutions Fields of Study 2011 CASS 0 0 0 0 2012 CASS Dominican Republic, El Salvador, Guatemala, Honduras 23 1 Education Teacher Training (*No Cost Extension) Total CASS Scholarships 6529 CASS Total Funding: $233,916,863.00 1992 NPSP Nicaragua 90 6 Automotive Technology, Clothing Manufacture Technology, Computer Maintenance Technology, Computer Processing Technology, Small Business Management, Business Solutions Technology (TSE) 1993 NPSP Nicaragua 65 5 Computer Maintenance Technology, Automotive Technology, Electronics Technology, Construction Management, Business Solutions Technology (TSE) 1994 NPSP Nicaragua 57 4 Small Business Management, Hotel and Restaurant Management, Quality Control Technology, Business Solutions Technology (TSE) 1995 NPSP Nicaragua 45 3 Small Business Management, Computer Programming Technology, Food Science Technology 1996 NPSP Nicaragua 68 4 X-Ray Technology, Food Science Technology, Solid Waste Management Technology, Business Solutions Technology (TSE) 1997 NPSP Nicaragua 50 3 Electronic Communication Technology, Manufacturing Management Technology, Rural Public Health Technology 1998 NPSP Nicaragua 55 3 Electronic Communication Technology, Manufacturing Management Technology, Rural Public Health Technology NPSP Total Scholarships 430 NPSP Total Funding: $13,361,613 Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 71 Funding Year Program Participating Countries No. of Scholarships Awarded No, of Host Institutions Fields of Study 2009 SEED Dominican Republic, El Salvador, Guatemala, Haiti, Honduras, Mexico, Nicaragua 170 9 Agribusiness, Education Teacher Training, Quality Control Technology, Small Business Management 2010 SEED Dominican Republic, El Salvador, Guatemala, Haiti, Honduras, Mexico, Nicaragua 216 11 Agribusiness, Small Business Management, Computer Science/Business Applications, Education Teacher Training, Natural Resources Management, Quality Control Technology 2011 SEED Dominican Republic, El Salvador, Guatemala, Haiti, Honduras, Mexico, Nicaragua 349 13 Agribusiness, Education Teacher Training, Natural Resources/Management, Quality Control Technology, Small Business Management, Transmissible Disease Vector Control 2012 SEED Dominican Republic, El Salvador, Guatemala, Haiti, Honduras, Mexico, Nicaragua 352 13 Agribusiness, Education Teacher Training, Natural Resources Management, Quality Control Technology, Small Business Management, Transmissible Disease Vector Control SEED Total Scholarships 1087 SEED Total Funding: $40,561,632 Total number of overall scholarships: 9191 Total Overall Funding: $320,820,029 Central American Scholarship Program (CASP): Program provided technical training, leadership, democracy skill building and “Experience America” experiences to impoverished, rural high school graduates from Central America. St. John’s Scholarship Program (SJP): USAID/LAC Education Chief Joseph Carney requested that Georgetown University launch the St. John’s Scholarship Program. Program provided top graduates of St. John’s College in Belize with tuition-free scholarships at Jesuit universities throughout the United States to complete undergraduate degrees. Training programs were individualized and students were placed in individualized academic programs. Evaluation of LAC Higher Education Scholarships Program Page 72 Evaluation of LAC Higher Education Scholarships Program Cooperative Association of States for Scholarships (CASS): Program provided technical training, leadership, and community development skills to impoverished, rural high school graduates, and professional skills upgrading to rural primary education teachers, health workers, cooperative and microfinance managers, and anti-seismic construction foremen from Central America, Mexico and the Caribbean. Nicaragua Peace Scholarship Program (NPSP): Program provided Contra and Sandinista youth ex-combatants with academic skills upgrading, technical training, reconciliation and democracy-building skills. Scholarships for Education and Economic Development (SEED): SEED is the successor program to the CASS program and provides technical training, leadership and community development skills to impoverished, rural high school graduates, and professional skills upgrading to rural primary education teachers, health workers, cooperative and microfinance managers, and anti-seismic construction foremen from Central America, Mexico and the Caribbean. *Note: The countries of Belize, Jamaica, Panama, Costa Rica and seven countries of the Eastern Caribbean were phased out of the scholarship program due to USAID Mission program phase-outs. Mexico was added with the advent of the Partnership for Prosperity agreement between the USA and Mexico. Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 73 APPENDIX 2: ILLUSTRATIVE EXAMPLES OF OTHER SPONSORED SCHOLARSHIPS & LEARNING OPTIONS The Organization of American States Academic Scholarship Program provides grants to individuals from OAS member countries to pursue undergraduate or graduate studies. Grants are awarded by the Department of Human Development, Education and Culture (DHDEC) of the OAS for either on-site or distance education in a university or higher learning institution in an OAS member country. OAS Scholarships, awarded by the Organization of American States (OAS) to applicants from its member states, provide graduate-level study opportunities to all members. Undergraduate-level scholarships also are awarded for the last two years of undergraduate study only, for applicants from English- and Dutch-speaking member states of the Caribbean. Full and partial scholarships are awarded, along with interest-free study loans and additional awards for online coursework in targeted areas. The program is administered by through LASPAU, affiliated with Harvard University. It does not include special programming or activities; students are self-placed and apply for support on the basis of an offer of admission. www.oas.org/en/scholarships/ www.laspau.harvard.edu/current-programs/organization-american-states-academic-scholarship-program Joint Japan/World Bank Graduate Scholarship Program (JJ/WBGS) –The World Bank sponsors academic scholarship programs targeted at capacity building for member countries, with a special focus on degree-objective graduate study in the economic development sector for highly talented individuals. Admission to the program is based on academic achievement and not means. Through partnerships with 16 universities in Africa, Japan, Latin America and the United States, the program features access to best-suited academic resources worldwide, and in some cases takes the cohort approach similar to World Bank scholarship is funded 50 percent by the government of Japan, but does not require residence in Japan for study. JJ/WBGS has made significant investment over its existence in Tracer Studies of all former participants since the program began in 1987. The eight studies published to date report on inputs, outcomes and impacts, and regularly updated alumni contact information in the process. Such studies may provide a useful model for ongoing tracking of USAID’s future participant trainees. www.oas.org/en/scholarships/ Ford International Fellowship Program (IFP) – This program was a ten-year investment (2003- 2013) by the Ford Foundation to create young social-justice leaders in 22 countries of the developing world. Like CASS/SEED, the IFP places a special emphasis on indigenous and disadvantaged participants. The program model also has targeted strengthening of local NGOs by relying on an appropriately suited local NGO in each country to support its program there (recruitment, preparation, post-program follow-up), with the intention of strengthening a local organization that could expand to support further educational support for the targeted population. For example, in Guatemala the Centro de Investigaciones Regionales de Mesoamérica (CIRMA) is Ford’s partner for recruitment, screening and post-program support; in Mexico the Centre de Investigaciones y Estudios Superiores en Antropología (CIESAS) plays a similar role. Evaluation of LAC Higher Education Scholarships Program Page 74 Evaluation of LAC Higher Education Scholarships Program Academic placement of IFP scholars is worldwide and includes non-English-medium universities (French￾speaking in Africa; Spanish-speaking in Latin America). Program impact is strengthened through a series of regional seminars and workshops building community-based leadership skills and strengthening the participants’ future collaboration as a regional and global network. www.fordifp.org Haitian Education and Leadership Program (HELP) is a new in-country scholarship model providing scholarships to young, disadvantaged Haitians. Started in 2004 with private-sector contributions, its mission is “to create, through merit- and means-based scholarships, a community of young professionals and leaders promoting a more equitable society in Haiti.” All awardees must commit to repay 15 percent of their scholarship award over a nine-year period following the end of their studies. Beyond scholarship support, HELP has a study center, equipped with computers and a library that provides additional support and guidance to its participants. www.uhelp.net Massive Open Online Courses (MOOCs) – This is a burgeoning movement in higher education to share, at little or no cost to the recipient, coursework and skill development opportunities with the world at large. MOOCs currently are growing in their presence across major universities worldwide and diversifying quickly in their content offerings and languages of instruction. Taking a MOOC presumes the participant has access to the computer hardware, software and Internet connectivity that this interactive coursework requires. It further expects at least passive language fluency and a participatory approach to learning to ensure successful completion. At present, participation in such courses is massive but course completion is currently estimated by the American Council on Education at only about 10% overall. www.coursera.org/about www.edx.org www.udacity.com Muskie Fellowships – The Edmund Muskie Fellowship, started in 1992, provides degree and professional training scholarships in U.S. universities to young university graduates from Eurasia. Beyond relevant coursework (and sometimes a graduate degree), Muskie Fellows must perform volunteer community service and pursue an internship following their academic training component. Upon return home Muskie Fellows benefit from a well-developed alumni network, alumni websites and blogs, post-program seminars and may qualify for small grants to conduct community service projects, organize their own conferences or trainings, or release publications of scholarly work developed in their home countries. The program is administered by IREX (International Research and Exchanges Board) under a cooperative agreement with the Department of State. exchanges.state.gov/academicexchanges/muskie.html www.irex.org/project/edmund-s-muskie-graduate-fellowship-program Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 75 The Community College Initiative (CCI) is an initiative of the U.S. Department of State, Bureau of Educational and Cultural Affairs (ECA), which started in 2006 in collaboration with the Community Colleges for International Development (CCID) consortium. Probably the closest program model to SEED, and was in some ways modeled after the earlier CASS program. Like CASS/SEED, CCI targets “underserved, non-elite international young adults,” providing them with a year of technical training and a first-hand understanding of American society. The CCI program does require English-language proficiency and is only one year in duration. Participants receive certificates relating to their field of study. CCI’s administration overseas is managed by staff of the U.S. Embassy Cultural Affairs Section, or where they exist, by the Fulbright Commission staff. As such, the staff costs for recruitment and post-program support are covered by the Department of State and not attributed directly against the program’s appropriated funds. This results in significantly lower per-participant recruitment and alumni support costs. Much like CASS/SEED, the U.S. side of the program is implemented by the CCID, under a cooperative agreement with State/ECA, and its member institutions. exchanges.state.gov/globalexchanges/community-colleges-initiative.html programs.ccid.cc/cci/node/752 COLFUTURO is a joint public and private sector initiative in Colombia, aimed at facilitating the access of Colombian professionals to further education abroad. Created as a non-profit organization in 1991, its mission is to “contribute to the entrance of Colombia into a world undergoing globalization by offering information about international education and supporting students to have a successful study experience overseas.” Since its inception it has awarded over 2,000 scholarship loans for study in several dozen countries (the most frequent study destinations in descending order: the United States, the United Kingdom, France, Germany, Spain and Italy). Colfuturo’s Scholarship Loan Program finances the studies of Colombian graduate students abroad (up to US$50,000 is available over a two-year period) thanks to an initial endowment of US$13 million. Depending on field of study, the program forgives 25-50 percent of the loan given to the student upon successful completion and return to Colombia, with an additional 10 percent forgiven if the returning graduate works in the public, non-profit or education sector. The recipient repays the balance loaned over time. Colfuturo also has a Work Placement Program to assist students it financed in obtaining suitable employment on their return to Colombia with degrees from overseas universities. www.colfuturo.org The Western Union Foundation Family Scholarship offers scholarships in the United States to recent immigrants from developing countries with the family unit in mind. Applicants for college-level scholarships must apply with another family member who also is motivated to increase their educational level. This typically involves a young adult who applies for undergraduate study along with a parent or sibling seeking a lower-level educational objective—English language skills, literacy, numeracy, basic business training, etc. Western Union is committed to having family members “move along” with their child or more advanced sibling, and thereby solidifying the success potential of the whole family unit. foundation.westernunion.com/education_programs.html www.iie.org/en/Programs/Western-Union-Foundation-Family-Scholarship Evaluation of LAC Higher Education Scholarships Program Page 76 Evaluation of LAC Higher Education Scholarships Program The State Department’s English Access Microscholarship Program has provided non-elite 14 to 18 year old students in 44 countries with significant Muslim populations English language instruction through a U.S.-style classroom experience since 2004. In recent years, the model has also been applied to some countries in the LAC region. The overarching goals of the Program are for students to gain an appreciation for American culture and democratic values, acquire sufficient English language skills to increase their ability to successfully participate in the socio-economic development of their countries, and improve their chances of participation in future U.S. educational and exchange programs. Its intended outcomes are to: 1) Provide a significant English language learning experience to students in a cost-effective manner (approximately $1,000 per student) to maximize the number of students reached; 2) Provide direct English teaching to non-elite students living in underserved neighborhoods; 3) Engage in the design and implementation of a transparent Access student selection process; and 4) Recognize student achievement through award certificates signed by the U.S. Ambassador http://exchanges.state.gov/non-us/program/english-access-microscholarship-program Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 77 APPENDIX 3: ECONOMICALLY ACTIVE POPULATION FOR SAMPLE COUNTRIES (Latest Year Available) El Salvador: Active Population 20-29 by Sex – 2007 (in thousands) Age Group Total Population Active Men Percentage of Population Active Women Percentage of Population 20-24 506.9 198.8 39.2% 119.5 23.5% 25-29 463.4 192.8 41.6% 142.4 30.7% Total 970.3 391.6 40.3% 261.9 26.9% Source: Labor Force Survey. www.laborsta.ilo.org Guatemala: Active Population 20-29 by Sex – 2006 (in thousands) Age Group Total Population Active Men Percentage of Population Active Women Percentage of Population 20-24 1101.9 452.3 41% 300.4 27.2% 25-29 914 387.9 42.4% 262.3 28.6% Total 2015.9 840.2 41.6% 562.7 27.9% Source: Labor Force Survey. www.laborsta.ilo.org Mexico: Active Population 20-29 by Sex – 2008 (in thousands) Age Group Total Population Active Men Percentage of Population Active Women Percentage of Population 20-24 8895.4 3343.2 37.6% 2166.2 24.3% 25-29 7802.6 3408.7 43.7% 2177.5 27.9% Total 16698 6751.9 40.4% 4343.7 26% Source: Labor Force Survey. www.laborsta.ilo.org Evaluation of LAC Higher Education Scholarships Program Page 78 Evaluation of LAC Higher Education Scholarships Program APPENDIX 4: MALE AND HISPANIC MEAN SUCCESS RATES Hispanic Employment and Leadership Success by Academic Program and Group Variable/ Group Employment Success Leadership Success Number Mean Number Mean Overall - Recipient 149 3.53** 149 2.29** Non-recipient 115 2.40 115 1.38 Two-year – Recipient 93 3.16** 93 1.99** Non-recipient 80 2.03 80 1.16 One-year – Recipient 16 3.87 16 2.50 Non-recipient 12 3.50 12 2.25 6-month – Recipient 21 4.14 21 2.80* Non-recipient 7 3.70 7 1.57 Source: Databases - JBS, LAC HE Evaluation 2012 Percentages may not equal 100% owing to non-responses. *significant at p ≤ .05; **significant at p ≤ .01 Male Employment and Leadership Success by Academic Program and Group Variable/ Group Employment Success Leadership Success Number Mean Number Mean Overall - Recipient 122 3.52** 122 2.32 Non-recipient 96 2.52 96 1.93 Two-year – Recipient 58 3.24** 58 1.93** Non-recipient 57 2.19 57 1.23 One-year – Recipient 31 3.52* 31 1.19 Non-recipient 27 2.63 27 1.53 6-month – Recipient 18 3.72 18 3.11 Non-recipient 6 4.00 6 2.50 Source: Databases - JBS, LAC HE Evaluation 2012 Percentages may not equal 100% owing to non-responses. *significant at p ≤ .05; **significant at p ≤ .01 Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 79 APPENDIX 5: SCOPE OF WORK Terms of Reference Evaluation of LAC Higher Education Scholarships Program I. Purpose of the Evaluation The purpose of this evaluation will be to assess the merit and value of USAID’s investment in higher education scholarships in the Latin America and the Caribbean (LAC) region. USAID’s Bureau for Latin America and the Caribbean Office of Regional Sustainable Development (LAC/RSD) has implemented a higher education scholarships program for over 25 years. As the current agreement nears completion— the final cohort under the current agreement is scheduled to begin studies in FY 2013—it is critical that USAID examine the results of its long term investment. The evaluation will also inform future LAC higher education program design. This evaluation will be one of several that LAC is conducting in 2012, and to the degree possible, should be designed to comply with the USAID’s Evaluation Policy and its protocol for evaluative rigor. Because of the severe data limitations, the evaluation (ex-post by design) will employ a series of mixed-methods approaches. The final evaluation report should include a section on what the gold standard for impact evaluation would be for the program in question, as well as what key steps could be taken to facilitate such an evaluation in the future. The final evaluation will be submitted to the Bureau for Policy and Planning (PPL). A final report will be due by October 31, 2012. II. Background II.A. LAC Higher Education Scholarship Programming To help countries meet their needs for skilled manpower to advance development priorities, USAID has supported higher education scholarships and training programs for Latin America and the Caribbean for over a quarter of a century. Higher education scholarships currently are implemented through Scholarships for Education and Economic Development (SEED). Earlier iterations of the program are the Central American Scholarship program (CASP) and the Cooperative Association of States for Scholarships (CASS). All have been implemented by the Georgetown University Center for Intercultural Education and Development (CIED). Socio-economically disadvantaged youth and professionals from LAC are recruited for two-year technical training as well as one-year, and six-month professional development programs. The training takes place at community colleges and universities in the United States. The specific countries supported have changed over the years with the program currently implemented in seven countries – Mexico, Haiti, Dominican Republic, El Salvador, Guatemala, Honduras, and Nicaragua. Two evaluations of the higher education scholarships programming have been conducted to date. The first, Training Impact and Development: An Evaluation of the Impact of the Cooperative States for Scholarships (CASS) Program, was completed in November 1994, and is available online at: http://pdf.usaid.gov/pdf_docs/pdabk526.pdf. The second evaluation, An Evaluation of the Cooperative States for Scholarships Program (1994-2001), was completed March 2002, and is available online at: http://pdf.usaid.gov/pdf_docs/pdabz325.pdf II.B. Program History In 1985, USAID initiated Central American, Andean, and Caribbean participant training scholarship programs, after the National Bipartisan Commission to Central America and the Caribbean (the Kissinger Commission) found that the United States was not sufficiently responsive to the political and civil turmoil in the region. The commission urged that the United States government provide immediate Evaluation of LAC Higher Education Scholarships Program Page 80 Evaluation of LAC Higher Education Scholarships Program and direct support to the region’s non-elite populations through implementation of participant training programs which would meet both the civil society democratic and development needs of the region. In response, Congress authorized the creation of the Caribbean and Latin American Scholarship Program (CLASP), to be implemented by USAID. Training initiatives under CLASP I and CLASP II focused on Central America (CASP), the Andean region (APSP), and the Caribbean (PTIIC). CASS came into being in 1989 with the expansion of the CASP program into the Caribbean region, and the formalization of a cost-sharing plan on the part of the community colleges and universities that hosted the scholarship recipients. Central to the CASP, CASS and SEED programs has been the consistent focus on providing training opportunities to socially and economically disadvantaged rural populations. At its inception, the CASS program included only two-year participant training exchanges for youth, and its objectives focused strongly on technical training for employment, leadership development and civil society diplomacy needs. Over time, the program has continued to focus on the most disadvantaged populations of the LAC region. In 1989, with the development of the Caribbean Basin Initiative (CBI), additional countries were added to the CLASP initiative and the CASS program. These included Jamaica, Belize, Panama, and seven countries of the Eastern Caribbean. In 1991, Nicaragua joined CASS and in 2003 Mexico joined with the launching of the TIES initiative. As USAID has redefined its development priorities for the region, countries have graduated leaving the CASS and SEED programs with seven countries. Field of study determination has been made in accordance with the USAID Strategic Objective and development needs current at the time. In 1995, CASS added professional development scholarships, which provided short-term specialization and skills upgrading to rural mid-level professionals to make them and their institutions more effective. As part of CASS’s Cooperative Agreement in 1998, this model was adapted to provide training to help support USAID’s commitment to improving basic education and fighting infectious diseases. Programs of one year, six months and three months were tailored for participants to develop their professional and leadership skills and have been in fields of study supportive of Mission Strategic Objectives. The program has provided six- and twelve-month professional development scholarships in addition to the two-year scholarships for rural youth. The SEED agreement, awarded in 2008, added competed sub awards to colleges for training provision. SEED helps to meet a presidential commitment announced at the 2009 Summit of the Americas. The commitment stated that, through SEED, 1300 students from the region would receive scholarships over the next five years to study in the United States. A comparative look at the different iterations of the Georgetown-managed scholarships programs suggests that each can be characterized by at least the following three constants: 1. Targeting of socially and economically disadvantaged people from rural areas and indigenous populations 2. Recruitment, orientation, and follow-on in country to build capacity 3. US Community colleges have always been partner institutions (though the number has grown) II.C. Overall Goals and Objectives of HE program The principle aim of LAC’s higher education activities is to help countries meet their needs for skilled manpower in order to meet their most important development needs. The LAC/RSD higher education scholarships program has supported U.S. Government (USG) economic and governance goals for the region by increasing human and institutional capacity in ways that increase Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 81 equity, experience, and cultural understanding. The program provides opportunities for poor and disadvantaged youth in LAC who are likely future leaders in their countries to gain technical skills in an array of fields deemed important to the social and economic development of the individual communities, countries and to USAID goals for the region. The youth targeted by the program have extremely limited opportunity to attend institutions of higher education, particularly as their communities and countries lack resources for student loans and scholarships. Through improved access to higher education, the program seeks to: 1. Create a cadre of change agents and future leaders with an understanding, experience, and appreciation of democratic processes and values; 2. Provide participants with skills and knowledge to participate actively and responsibly in a democratic society; 3. Strengthen capacity in countries participating in free trade agreements in a range of technical fields that are critically-needed to support implementation and participation in free trade and help countries become more competitive and better realize the opportunities brought by free trade; 4. Support U.S. public diplomacy by creating a cadre of future leaders who have been immersed in American culture and are sympathetic to the United States. In order to meet these top line goals, the program’s current incarnation (SEED) aims to contribute to the following intermediate results: 1. Reach full employment of all recipients. 2. Apply technical skills, including ESL, by alumni in their place of employment 3. Strengthen businesses and NGOs through the application of newly acquired skills by program alumni 4. Increase leadership and civic participation of alumni in their home countries 5. Increase involvement by alumni in community development and volunteer actions 6. Strengthen citizen diplomatic as well as business ties between the United States and program alumni 7. Improve quality of teacher training for rural schools teachers resulting in higher student retention, increased test scores and greater parent and community participation in at-risk low performance schools 8. Increase capacity for local health units and the Ministry of Health to identify transmissible disease threats and to develop health care responses to these threats. 9. Strengthen and more profitable rural cooperatives. II.D. Two Types of Scholarships The LAC/RSD higher education program offers two types of scholarships: two-year technical scholarships for youth, and shorter-term (six- and twelve-month) scholarships for professionals. Both types of training seek to develop the capacity of a critical mass of individuals to improve the development of their countries. a) Two-years scholarships for youth Evaluation of LAC Higher Education Scholarships Program Page 82 Evaluation of LAC Higher Education Scholarships Program The two year program provides targeted technical training along with leadership and community development training to rural youth from 18 to 25 years of age who do not have the ability to access higher education opportunities but demonstrate strong leadership potential and qualities. Youth recipients are from economically disadvantaged families. The program targets the most rural areas of each country, prioritizes recruitment of indigenous and other ethnic minorities, persons with disabilities, and seeks to provide 50% of all scholarships to women. Youth recipients receive intensive ESL training in the US, in addition to technical and academic training. All two-year youth recipients participate in on￾the-job internships in the United States, develop Community Action Plans (CAPS) to be implemented in their home countries and receive guidance and support to secure employment once they return home. b) Six- and twelve-month scholarships for professionals The programs for rural professionals focus on developing the skills of rural primary education teachers, rural health workers and finance managers of rural cooperatives. Recipients are between 25 to 45 years of age, are active in their communities, have at least three years of professional experience and must receive the support from their employers (Ministry of Education, Ministry of Health, etc.) through paid leaves of absences for the duration of their program. Similar to the two-year scholarship program, priority is given to applicants from the most remote rural regions, persons from indigenous and other ethnic minorities, and seeks to ensure full participation of women. II.D Previous Evaluations Evaluations of the CASS program were undertaken in 1994 and 2002. Those evaluations, along with quarterly and annual reporting records have shown that program alumni regularly return to their home countries as well-trained change agents who exert influence on numerous others in their personal, professional and community networks. They conclude, though not empirically, that the program offers benefits and individual returns for recipients and their professional sectors. It has been noted that this change can often be immediate and observable (such as the introduction of new production techniques in local businesses and implementation of improved pedagogical methodology). It has also been noted that other alumni impact may not be as immediately apparent and will require longer-term monitoring and evaluation to appropriately assess. These evaluations have relied upon beneficiary interviews and site visits, and provide useful summaries of the extent to which the CASS program met its originally stated goals. They do not address questions of cost-effectiveness or comparison to other comparable or alternative programs (the 2002 evaluation considered this aspect “outside of its scope,” and deemed the CASS model to be too dissimilar to other investments with similar outcome goals). III. Evaluation Questions This evaluation is meant to help shed light on the overall question of what the development impact of the higher education scholarships has been. While attributing the individual scholarships and their outcomes to development results may not be possible, the following two research questions should guide the design, methodology, and outputs of the evaluation, and help the evaluators to extrapolate on the overarching question of interest. 3) To what extent have LAC/RSD higher education scholarships enabled participating individuals to become leading change agents in their respective professional fields, communities, and/or countries? a. How likely is it that these outcomes would have been generated without provision of these scholarships? Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 83 4) To what extent can it be said the LAC/RSD higher education scholarships program provides value for the investment? a. How does the LAC/RSD higher education scholarships program compare to other programs with similar aims in terms of impact and costs? Below, each principal research question is discussed in further detail in order to flesh out the intention and context for each. 1) To what extent have LAC/RSD higher education scholarships enabled participating individuals to become leading change agents in their respective professional fields, communities, and/or countries? a. How likely is it that these individual outcomes would have been generated without the provision of these scholarships? This question is meant to assess the overall impact of the program by assessing post-scholarship experiences of recipients in key areas. Research performed towards answering the counterfactual (sub￾question A) should utilize rigorous empirical methods to the fullest extent possible, including those that would align with the best practices outlined in the USAID Evaluation Policy (2011). The outputs and outcomes to be analyzed in the evaluation will be linked back to the broader developmental goals of the program. This will allow the evaluators to address the effectiveness of the program in contributing to broader developmental impact, not just the individual returns for recipients. Some outcomes of interest and for which data may be available include: individual education levels, individual income, and a “leadership index” by which recipients can be compared to a control group. Evaluators should discuss these options with USAID after carefully ascertaining data availability and methodological options / constraints. Preliminary discussions on methodology have focused on the use of a tracer study, though other options should be considered. Recipients from the 6 month, 1 year, and 2 year scholarship programs should be analyzed separately, or in “cohorts” determined by the evaluators with feedback from USAID. In preparing the analysis, it will be important to draw an appropriate and statistically significant sample from the universe of all recipients in the given reporting timeframe. Likewise, constructing a valid counterfactual will be of paramount importance. Site visits, key informant interviews, and online surveys may be appropriate data collection methods for researching this question. To the extent possible, researchers should construct valid counterfactuals to adequately address the issue of outcome attribution, and the likelihood of whether or not program recipients could have found alternative means to achieving the same or similar outcomes. Thorough definition of the key term “leading change agent” should be constructed in consultation with USAID, and be measured by a custom index or a series or proxy measures which should take into account at least the following four factors, or approximations thereof: 1. Positions of leadership in community (aka the leadership index alluded to below) 2. Degree of involvement in community (hours per month) 3. Chosen career in development (yes or no) 4. Income (within a employment category, say local NGOs, if recipients are paid more, it could imply they are more valuable to the organization) 2) To what extent can it be said the LAC/RSD higher education scholarships program provides value for the investment? Evaluation of LAC Higher Education Scholarships Program Page 84 Evaluation of LAC Higher Education Scholarships Program a. How does the LAC/RSD higher education scholarships program compare to other programs with similar aims in terms of impact and costs? This question is meant to inform consideration of alternatives to the LAC/RSD higher education scholarships program, and may be used to inform how USAID programs future funding in this area. Given appropriate and available comparison points, this question should help guide evaluators in constructing a matrix of the relative strengths and weaknesses, or advantages and disadvantages, to the overall approach employed by the SEED / CASS program. In constructing this analysis, evaluators should carefully identify and rigorously consider the advantages and disadvantages of the constituent components of the current SEED program’s approach (i.e. costs and cost structure, administration, U.S. residencies, use of US-based community colleges, incorporation of English instruction and immersion, integration of community-focused projects upon return to home countries, etc.). Conducting rigorous cost analysis (such as cost benefit analysis or cost effectiveness analysis) should be considered where possible. While it is unlikely that adequate data or comparisons exist to enable this, a careful explanation in the evaluation report should be provided on data limitations and the analysis conducted on which method was ultimately chosen. It is USAID’s understanding that no other scholarship programs combine the same approach as SEED (1-target underserved communities, 2-focus on technical training, and 3-utilize community colleges and universities in the USA). On the other hand, it is important to compare this approach to similar approaches with at least some of the same aspects. The following examples are meant only as illustrative examples and may contain aspects that do not make them suitable for comparison. • Illustrative example: Other scholarship programs that utilize U.S. higher education institutions and target underserved communities in LAC (i.e. Indigenous and Afro Latino Scholarship Program (IALS) of the Inter-American Development Bank (IADB), or other State/WHA-sponsored scholarship programs. • Illustrative example: South-South or other international scholarship programs (i.e. the OAS-funded program that sponsors Caribbean students to study in U.S. undergraduate institutions, administered via LASPAU, the Ford Foundation Fellows program, or other programs funded from within the region). • Illustrative example: Intra-national scholarship programs designed with similar aims (i.e. the Haitian Education Leadership Program, or HELP, or other programs that offer scholarships to nationals of the donor country for technical training within the confines of the donor country, with or without international funders). While it is highly unlikely that other scholarship programs or models will include all of the defining features of SEED / CASS or that comprehensive data sets are readily available for comparison, non￾experimental research on this question should make an effort to winnow down the cost and benefits of alternative approaches in order to be able to make useful if not definitive statements regarding their costs and relative advantages / disadvantages in comparison to CASS/SEED. Such analysis therefore should make explicit any assumptions or data gaps present therein. Research on this question should culminate in data-driven, objective recommendations on how to optimize the LAC/RSD higher education scholarships program’s impact while minimizing the costs, and findings on whether or not the current approach does either or both. Recommendations should be qualified wherever possible given the data limitations and the obstacles they present in making definitive data-driven recommendations. IV. Evaluation methodology Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 85 A performance evaluation of higher education scholarship programming in the LAC region will require access to data from multiple sources. A mixed methods approach, i.e., an evaluation methodology that employs quantitative and qualitative methods, should be considered in order to collect the most relevant and useful data. Because of the variety of methodological options and data limitations (some of which are summarized below), the design of the methodology should be closely coordinated with USAID in order to ensure that the design represents an optimal research approach. In designing the methodology and in preparing the expected outputs of the research, close consideration should be given to the role of gender, attempts to disaggregate results by gender as well as other relevant demographic factors should be made wherever possible. Recommendations from the LAC Gender Analysis (see Section X.I.1) should be reviewed by the research team in order to inform the research design and final products. V. A. Limitations USAID is aware of several key design limitations of its higher education scholarship programs that may affect the evaluation methodology for this performance evaluation: 1. The higher education scholarship programs to be studied, e.g., CASP, SEED, CASS, were not designed as experimental designs, and thus did not identify a control group throughout the program cycle. 2. The higher education scholarship programs to be evaluated were implemented across various countries with disparate labor and economic conditions. 3. The programs had relatively low numbers of observations per year as needed to compute statistically significant differences by country and year. With these limitations in mind, the researcher team should consider innovative techniques and a mixed￾methods approach to be able to identify meaningful results given the evaluation questions. For example, researchers may be able to construct a natural comparison group from students who were eligible, yet not chosen, for the programs. VI. B. Study Sample As noted, recipients that have completed USAID higher education scholarships can be categorized into three groups based on their length of study in the United States: two years, one year, and six months. As the length of study has a significant effect on the study experience, these populations should be treated separately in the evaluation methodology. If it is deemed possible, the one year and six month recipients may be combined into one sample cohort; however, this should only be done if no statistical differences can be found in the personal outcomes of these recipients by length of training. The parameters of the study sample will be discussed and approved by USAID once a thorough data review of contact information is completed in the first month of the study. This review should include an overview of the numbers of recipients (and comparison group, if available) for each cycle year, disaggregated by country and length of training. It will be necessary to identify the recipients for whom current contact information is available. Sample recipients should include those students who have returned to their home country for at least one year, in order to provide sufficient time for students to re-adjust and look for employment. According to studies done of graduate training programs, the optimal time to follow up with students is Evaluation of LAC Higher Education Scholarships Program Page 86 Evaluation of LAC Higher Education Scholarships Program between one to five years, in order to capture the transition process into professional activities after the scholarship experience.17 VII. C. Data Collection Methods Quantitative and qualitative methods should be considered in the evaluation methodology. The first step in an effective and efficient methodology for a performance evaluation should be a synthesis and analysis of program monitoring data from the implementing partner. From these data, a quantitative framework can be built, thus ensuring that data already collected does not have to be recreated. Thereafter, additional quantitative data will enable an analysis of program impact over a span of time, and in comparison to a counterfactual. Where available, qualitative data should also be collected to allow triangulation of approaches to provide solid understanding of the different perspectives on implementation and outcomes. Because interventions to enhance recipients’ education, training, and professional outcomes are a result of multiple combined efforts, both objective and subjective data are required to assess them. Through the collection of various kinds of data, an accurate description of the dynamics involved and interpretation of observed patterns becomes possible. Possible data collection methods are included in the following table Suggested Data Collection Methods Quantitative • Online, phone, and in-person surveys of former recipients • Online, phone, and in-person surveys of comparison group Qualitative • Site visits to U.S. sponsoring institutions • In-depth interviews with key informants, including employers, former employers, faculty, other program staff • Focus groups with former beneficiaries and/or beneficiary communities in-country VI. General Evaluation Parameters • Planning and implementation of the evaluation study will be closely coordinated with LAC/RSD, USAID Missions, the implementing partner and its local subsidiaries in the region. • Because of the limited amount of time for and specialized nature of this evaluation, it is of particular importance that the right evaluator(s) be selected. Selection of the evaluator(s) should be done in close collaboration with USAID, and according to the below standards. o The evaluation team must have an appropriate mix of technical skills to conduct the evaluation. Experts with experience with ex-ante impact evaluations with imperfect data sets; an advanced degree in education, economics, labor markets, rural / community development, or other similar field strongly preferred. 17 Schomburg, H. (2003), Handbook for Graduate Tracer Studies. Kassel, Germany: Centre for Research on Higher Education and Work, University of Kessel. Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 87 o The team leader must have:  Fluent / native English and professional Spanish language fluency.  Excellent writing skills (English).  Outstanding research skills and ability to synthesize large amounts of disparate information. o A wealth of local expertise exists in the countries where the evaluation may take place. The evaluation team should make real efforts to involve evaluation experts from partner countries, but not involved in project implementation, for analysis as well as data collection. • As pertinent, the evaluators should draw upon earlier CASS evaluations in the final report. VII. Services, Deliverables, and Performance Requirements and Standards Result 1: Evaluation/Assessment Addressing Key Questions in Manner of Utility to USAID Completed Requirement 1.1: Develop evaluation plan, including data collection and analysis plan and instruments Standards: • Evaluation plan provided within 2 weeks • Plan for evaluation includes data collection methodology, an analysis plan, and instruments to address the key questions listed in section III. • Plan identifies data sources that will be used for each question, including sources of data that are already available, such as monitoring reports and prior evaluations. • Plan includes consideration of how prior USAID evaluations will be used to inform analysis and conclusions. • Plan does not duplicate past evaluations. • Evaluation plan and instruments are informed by discussion with USAID. Data collection instruments, including any survey, its questions and recipient list, would be designed in close collaboration with USAID. Illustrative data collection methodology for discussion: One illustrative methodology to be discussed is the use of survey of beneficiaries and/or beneficiary communities. Requirement 1.2: Collect and analyze evaluation data Standards: • Site visits to at least these three countries, which are to include Mexico and up to two other countries, depending on data availability. Countries will be determined between JBS and USAID. , • Key informants identified • Number of individuals surveyed or interviewed and/or focus groups held is sufficient to provide meaningful representation and draw meaningful conclusions. This includes sufficient representation by each relevant stakeholder group, each country, and each modality. • Any conclusions developed are based on analysis of findings. • Any recommendations are based on conclusions from analysis of findings. Evaluation of LAC Higher Education Scholarships Program Page 88 Evaluation of LAC Higher Education Scholarships Program Requirement 1.3: Produce a high-quality evaluation report that meets the criteria the USAID Evaluation policy (enumerated on page 11, Annex I) in addition to the following standards: Standards: • USAID input incorporated to report outline. • Draft report submitted within 30 days of completion of analysis. • Final report incorporates USAID input. • Report includes a 3-5 page executive summary summarizes significant points from the full report, including key findings and recommendations. Any information provided in the executive summary appears in the full report.Report includes a section providing definitions and framework, including background of SEED. • The report includes the main evaluation questions and methods. • Report includes a section clearly delineating and explaining relevant limitations to the evaluation’s findings, and an overview of efforts to overcome limitations and mitigate data￾related or methodological challenges. • Report clearly distinguishes findings (facts), conclusions, and recommendations. • The logical connections between findings, conclusions, and recommendations are clear to the reader. Each conclusion is based in specific findings, and each recommendation is clearly related to a conclusion and assigns a responsible party. • Report clearly differentiates analysis, conclusions, recommendations, etc., related to the two evaluation questions. • Report includes financial data that permits computation of unit costs and analysis of cost structure • For any survey data, the report includes an annex presenting a detailed and organized summary of findings from the survey, including summary statistics and an overview of respondents. • If applicable, the report will include statements regarding any significant unresolved differences of opinion on the part of USAID, implementers, and/or members of the evaluation team. • A glossary of terms used is included in the report. • Grammatically correct and no spelling or punctuation errors. Requirement 1.4: Present findings of analysis Standards: • Presentation includes all key findings. • Presentation includes a PowerPoint that summarizes findings. Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 89 VIII. Budget Date: Action / deliverable: Comments: Phase One: Concept and Methodology April 2012 Planning meeting(s) with JBS and USAID to define and specify evaluation objectives Selection of Evaluation consultants (2) Initial review of data received from CIED to specify evaluation design and methodology Evaluation design finalized by JBS and Team Leader Evaluation design for review by USAID Workplan for review by USAID Phase Two: Instruments and Preparation for Data Collection May 2012 Instrument development and review by JBS Draft instruments for review by USAID Adjustment and final instruments approved May – June 2012 Initial contact with in-country data collection teams to specify scope of data collection Work with in-country data collection teams to prepare final contact lists, procuring addresses and other contact information, size of needed sample Work with in-country data collection teams to set up fieldwork schedules for evaluation team Phase Three A: Data Collection (In-country Teams) July 2012 Training of in-country teams on proper data collection and data entry July – September 2012 In-country data collection teams collect surveys from recipients and comparison groups Development of the codebook and compilation of data from in-country data collection teams Data cleaning and editing (quality control) Phase Three B: Data Collection (Evaluation Team) July 2012 Pre-departure meeting with evaluation team July – August 2012 Data collection by evaluation team in three countries Evaluation of LAC Higher Education Scholarships Program Page 90 Evaluation of LAC Higher Education Scholarships Program The evaluation budget will be estimated in conjunction with JBS based on project timing, staffing, and associated costs. IX. Key Personnel Key personnel must include a Team Leader (preferably a development economist with impact evaluation experience and ideally with experience in higher education for development evaluations); an evaluation methods specialist with focus on qualitative methods, and an evaluation methods specialist with a focus on applied cost analyses. This required expertise may be found in a team of two or three, and augmented with evaluation advisors as helpful and agreed upon between JBS and USAID. The team leader and senior advisors should be prepared to review the TOR and make comments to the team about feasibility, data questions, and methods for improving the rigor and significance of the evaluation results. Evaluation team members will provide a signed statement attesting to a lack of conflict of interest, or describing an existing conflict of interest relative to the project being evaluated. X. Illustrative Timeline Timeline, including specific deliverables, is to be proposed and agreed upon with JBS. XI.1 Mandatory References The following documents are mandatory background reading for the evaluation team and will be provided prior to initial meetings to discuss the evaluation plans with USAID: 1) 1994 CASS Evaluation Report (USAID) 2) 2002 CASS Evaluation Report (USAID) 3) 2005 CASS Final Performance Report to USAID from Georgetown University (USAID) 4) 2009 SEED Cooperative Agreement and Technical Proposal (USAID) 5) 2011 Evaluation Policy (USAID) 6) 2012 LAC Bureau Gender Analysis (DevTech Systems) Phase Four: Data Analysis and Reporting September – October 2012 Data coding and analysis (frequencies, statistical analysis) Report writing November 2012 Draft evaluation report for review by USAID December 2012 Briefing on evaluation findings & recommendations to USAID Final report Evaluation of LAC Higher Education Scholarships Program Evaluation of LAC Higher Education Scholarships Program Page 91 XI.2 Suggested References: The following documents are suggested background reading for the evaluation team and should be obtained and reviewed as pertinent to the evaluation, or for bibliographical guidance in finding other relevant references. 1) 2011 USAID Education Strategy: Opportunity Through Learning (USAID) 2) “Examining development evaluation in higher education interventions: a preliminary study” (LICD, 2012) 3) “Other Donors’ Scholarship Programs: What other donors are doing with development scholarships around the world.” (Gosling, M. 2008).