USAID Pakistan March 14, 2011 This report was produced for review by the United States Agency for International Development by International Business & Technical Consultants, Inc. (IBTCI) under the Evaluation Services IQC, AID-RAN-I-00-09-00016, REQ￾LAC-10-000019. Its authors are Frank Dall, Jeffrey Tines, Ana Cristina Accioly de Amorim, Sheere Kadez Brooks, Thomas D. Tilson, and Luis Enrique Medrano. EVALUATION Evaluation of the LAC/RSD Regional Education Program Volume I. Main Report March 14, 2011 This report was produced for review by the United States Agency for International Development by International Business & Technical Consultants, Inc. (IBTCI) under the Evaluation Services IQC, AID-RAN-I-00- 09-00016, REQ-LAC-10-000019. Its authors are Frank Dall, Jeffrey Tines, Ana Cristina Accioly de Amorim, Sheere Kadez Brooks, Thomas D. Tilson, and Luis Enrique Medrano. Evaluation of the LAC/RSD Regional Education Program Volume I. Main Report Evaluation of the LAC/RSD Regional Education Program Volume I. Main Report March 14, 2011 Prepared by: Frank Dall (Team Leader) Jeffrey Tines Ana Cristina Accioly de Amorim Sheere Kadez Brooks Thomas D. Tilson Luis Enrique Medrano The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Evaluation of LAC/RSD Regional Education Program: Volume I International Business & Technical Consultants, Inc. Table of Contents List of Acronyms ................................................................................................................................................................. i EXECUTIVE SUMMARY ................................................................................................................................................. iii Objectives and Rationale ............................................................................................................................................ iv Methodology .................................................................................................................................................................. v Analysis of Findings ....................................................................................................................................................... v The Benefits and Challenges of the Regional Programs ................................................................................. vi The Broader Impacts of the Regional Education Programs ......................................................................... vii The Value-Added Of Technical Assistance ....................................................................................................... ix Lessons Learned and Recommendations ............................................................................................................... ix Lessons Learned ...................................................................................................................................................... ix Recommendations ................................................................................................................................................... xi I. INTRODUCTION ........................................................................................................................................................ 1 Summary: USAID/LAC’s Regional Education Program ........................................................................................ 1 II. EVALUATION OBJECTIVES AND RATIONALE ............................................................................................... 3 A. Evaluation Purpose and Objectives .................................................................................................................... 3 B. Rationale ..................................................................................................................................................................... 3 III. METHODOLOGY ..................................................................................................................................................... 6 A. The Country Sample Selection ............................................................................................................................ 6 B. Data-Collection Instruments ................................................................................................................................ 6 1. Document Review ........................................................................................................................................... 6 2. In-depth Key Informant Interviews (KII) .................................................................................................... 6 3. Focus Group Discussions (FGD) ................................................................................................................. 9 4. The Online-Survey Questionnaire ............................................................................................................ 10 C. Analysis of the Data from All Sources .......................................................................................................... 12 D. Key Evaluation Considerations ....................................................................................................................... 13 E. Challenges Encountered ................................................................................................................................... 13 IV. ANALYSIS OF FINDINGS ..................................................................................................................................... 14 A. The Benefits and Challenges of the Regional Programs .............................................................................. 14 1. Summary of the Regional Programs’ Challenges ....................................................................................... 14 Evaluation of LAC/RSD Regional Education Program: Volume I International Business & Technical Consultants, Inc. 2. Caribbean-CETT ............................................................................................................................................... 15 3. CARD-CETT ..................................................................................................................................................... 16 4. Andean-CETT .................................................................................................................................................... 17 5. CERCA ................................................................................................................................................................ 18 6. PREAL .................................................................................................................................................................. 19 B. Coordination and Synergy among the Regional and Bilateral Programs ................................................. 21 1. Coordination and Synergy across Regional Programs ............................................................................. 21 2. Coordination and Synergy between Regional and Bilateral Programs ................................................. 22 C. The Broader Impacts of the Regional Education Programs ....................................................................... 24 1. Overall Observations ....................................................................................................................................... 24 2. Impacts on Attitudes, Perceptions, and Policy .......................................................................................... 25 3. The Regional Models’ Capacity and Sustainability .................................................................................... 27 D. The Value-Added of Technical Assistance ..................................................................................................... 30 V. LESSONS LEARNED AND RECOMMENDATIONS ........................................................................................ 32 A. Lessons Learned ................................................................................................................................................... 32 B. Recommendations ................................................................................................................................................ 34 1. The Benefits and Challenges of Regional Education Program Models .................................................. 34 2. Coordination and Synergy among Regional Programs ............................................................................ 36 3. Coordination and Synergy among Regional and Bilateral Education Programs ................................. 37 4. The Broader Impacts of the Regional Education Programs .................................................................... 38 5. Attitudes, Perceptions, and Policy ................................................................................................................ 38 6. Capacity and Sustainability .............................................................................................................................. 38 Evaluation of LAC/RSD Regional Education Program: Volume I i International Business & Technical Consultants, Inc. List of Acronyms AED Academy for Educational Development AOTR Agreement Officer’s Technical Representative CARD Central America - Dominican Republic CARICOM Caribbean Community and Common Market CDB Caribbean Development Bank CERCA Civic Engagement for Education Reform in Central America C-CETT Caribbean Centers for Excellence in Teacher Training CETT Centers for Excellence in Teacher Training CFA Cost-effectiveness Analysis CINDE Research Corporation for Development (Corporación de Investigaciones para el Desarrollo) CPAC Country Project Advisory Committees CSRM Corporate Social Responsibility Movement CTO Cognizant Technical Officer EMIS Education Management Information System EU European Union FALCONDO Falconbridge Foundation FEREMA Fundacion Para La Educación Ricardo Ernesto Maduro Andreu FEPADE Business Foundation for Education Development (Fundación Empresarial para el Desarrollo Educativo) FGD Focus Group Discussions IBTCI International Business & Technical Consultants Incorporated ICT Information and Communications Technology IDI In-depth Interview ILCE Latin American Institute for Education Communication (Instituto Latinoamericano para la Comunicación Educativa) IP Implementing Partner JBTE Joint Board of Teacher Education KII Key Informant Interview LAC Latin America and the Caribbean LOE Level of Effort LOP Life of project/program MoE Ministry of Education MoU Memorandum of Understanding NGO Non-governmental Organization OECS Organization of Eastern Caribbean States Evaluation of LAC/RSD Regional Education Program: Volume I ii International Business & Technical Consultants, Inc. OEI Organización de Estados Iberoamericanos PIU Program Implementation Unit PPP Public- Private Partnership PREAL Partnership for Educational Revitalization in the Americas (Programa de Promoción de la Reforma Educativa en América Latina y el Caribe) PUCMM Pontifica Universidad Católica Madre y Maestra (Dominican Republic) RICETT Red Inter-institucional del CETT (CETT CA-RD Inter-institutional Network) RSD Regional Sustainable Development SALCC Sir Arthur Lewis Community College SRC School-based Report Card TTI Teacher Training Institutions UASB Universidad Andina Simon Bolivar (Ecuador) UPCH Universidad Peruana Cayetano Heredia (Peru) UPN Universidad Pedagógica Nacional Francisco Morazán (Honduras) USAID United States Agency for International Development USAID/W United States Agency for International Development – Washington, D.C. USG United States Government UVG Universidad del Valle de Guatemala UWI University of the West Indies WAN Wide Area Network WB World Bank WDC Washington, D.C. Evaluation of LAC/RSD Regional Education Program: Volume I iii International Business & Technical Consultants, Inc. EXECUTIVE SUMMARY In the course of the last two decades, the Office of Regional Sustainable Development (RSD), an integral part of the USAID/LAC Bureau, has been implementing several regional educational programs. The objectives of these programs were and, in several cases still are, diverse as they respond to different real and perceived needs within the LAC Region as a whole, and the individual countries within the region. The programs’ objectives addressed efforts designed to directly benefit teachers’ skills and, more importantly, to benefit students in the early, formative years of their education. The objectives also included supporting organizations whose focus was/is to influence educational policy at the national level, which in turn would contribute to the adoption of more effective education policies, the professional development of more able policy-makers, and the strengthening of private sector and other non-governmental support to education. For this evaluation, USAID/LAC/RSD selected International Business & Technical Consultants, Inc. (IBTCI) to address how RSD could design and implement more effective regional education programs in the future and better support bilateral programs and projects. The evaluation therefore included research on the positive actions and results for LAC/RSD to consider in developing future projects and the aspects that “did not work.” In order to meet these requirements the Scope of Work for the evaluation included the review of three specific regional projects.1 These three include, first chronologically, the 15-year-old Partnership for Educational Revitalization in the Americas (PREAL), a program that was developed and launched to address education needs in the LAC region by promoting improvements in the quality and relevance of policy dialogue in education research and reform as well as to strengthen regional policy dialogue around important issues such as educational accountability, better education planning and the dissemination of reports on the quality and performance of national education systems, e.g., through National Education Report Cards. Overall, PREAL is a network of civil society organizations, research groups, and private sector entities in 17 countries in Latin America, not all of which receive USAID assistance for education. Second was an approach focused on improving education quality by creating effective change models through the Centers of Excellence for Teacher Training (CETT) program launched in 2001 as a Presidential initiative. CETT began with 13 countries in Latin America and the Caribbean and saw further expansion, some of it funded primarily by the participating countries, to further countries in the English-speaking Caribbean. In practice, this program consisted of the implementation of three essentially autonomous sub-regional CETT programs established to provide an effective model to improve teaching literacy in the early primary grades but tailored to the needs of each sub-region. The third regional effort was the Civic Engagement for Educational Reform to Central America (CERCA) program (2002 – 2006), which sought to facilitate and encourage the mobilization of broad constituencies to improve school quality in five countries in the CARD region – Guatemala, Honduras, El Salvador, Nicaragua, and the Dominican Republic. Combined, USAID support for the three programs served 21 of the countries in the LAC region. This evaluation presents the observations and findings of the six-person team as to issues associated with implementation of the three programs, their apparent impacts, and their relationships with each other and with other USAID bilateral education activities. In brief: 1 Although in the “big picture” view there were three programs: PREAL, CETT, and CERCA, as noted below pragmatically CETT functioned as three separate projects, so, depending on the context, there were in total either three programs or five programs. The LAC region includes several sub-regions; of relevance to this evaluation are South America, specifically the Andean countries, Central America and the Dominican Republic, and the English￾speaking Caribbean. The terms “sub-regional” and “regional,” as used in this evaluation, are generally synonymous, with context making the geographic scope clear. Evaluation of LAC/RSD Regional Education Program: Volume I iv International Business & Technical Consultants, Inc. • PREAL and CETT have both been generally effective in giving beneficiaries valuable opportunities to learn from the experiences of peers in other countries, especially those in the same sub-region and to apply this learning to their own circumstances. PREAL, which has USAID funding until 2012, has been successful in promoting positive educational change in various countries through its applied comparative research and its ability to mobilize key public-sector and private-sector education stakeholders. The CETT model has proven effective in improving teachers’ pedagogical skills and, even more importantly, children’s learning; however, Ministries of Education (MoEs) believe that the model is, relatively, too expensive for widespread implementation, although various colleges of education are incorporating the approaches in their in-service training. Unfortunately, although there were some obvious areas where synergy could have been developed (e.g., the use of reading standards), there was negligible interaction between the two programs. • CERCA, which was truncated, had little impact other than introducing the use of report cards, an important aid to accountability. • Having an AOTR present in the sub-region for each of the CETT cooperative agreements proved very helpful in promoting intra-regional cooperation and learning. • There were some noticeable inconsistencies between the pedagogical approaches being recommended by the sub-regional CETT Andino program and the bilateral Peru AprenDes project that could have been eased through greater coordination between LAC and USAID/Peru. There was also a perception by some USAID education officers and by some MoE officials that earlier consultation between LAC and/or the respective CETT AOTRs, on the one hand, and Mission education officers and/or MoE officials on the other would probably have resulted in greater local buy-in. Among the chief recommendations are: • The regional programs did contribute greatly to improving their objectives of promoting positive policy reform (in the case of PREAL) and in strengthening teacher training (in the case of CETT) and should be continued, preferably with USAID management following the CETT model of having an AOTR in each sub-region. • Both on a regional level and on a national level, programs for policy reform and programs for pedagogical reform need to “talk to one another” much more. Very often, as in the case of the PREAL work on standards for reading and CETT’s efforts to improve the teaching of reading, each has valuable information that would be beneficial for both to know. • Greater and earlier coordination between USAID managers of regional programs and mission staff, who are the people primarily responsible for interaction with local partners and local education officials, would promote more effective implementation of activities and could improve the probabilities of institutionalizing pedagogical reforms. • USAID should be aware that while they may have had experience in implementing international activities, some implementing partners for regional programs may need organizational strengthening so that they can be effective implementers in managing multi-national activities. Objectives and Rationale The objectives of this evaluation were (i) to evaluate the benefits and challenges as well as the advantages and disadvantages of different models for regional program implementation; (ii) to identify the broader impacts of regional programs and the key contributions that these make to improve the quality of education, both regionally as well as in individual countries; (iii) to determine any benefits and/or value added to implementing partners of technical assistance provided through centrally-funded contracts and cooperative agreements; and (iv) to make recommendations for the design and implementation of future LAC/RSD regional education programs. USAID intends to use this information to make potential regional programs more effective in general, not only as programs that serve the Evaluation of LAC/RSD Regional Education Program: Volume I v International Business & Technical Consultants, Inc. education needs of more than one country but also as programs that more effectively complement bilateral education projects and reform efforts in individual countries. Methodology The period of performance for this evaluation was September 30, 2010-March 15, 2011. Field work for the evaluation took place in November and December 2010 in eight LAC countries selected by USAID as believed to be representative of the countries participating in one or more of the three programs being evaluated. These countries were the Spanish-speaking Dominican Republic (DR), Honduras, Ecuador and Peru and the English-speaking Jamaica, Grenada, St. Lucia, and Antigua. All team members travelled to the DR; one sub-team then travelled to Honduras, Ecuador, and Peru, and the other travelled to Jamaica, St. Lucia, Grenada, and Antigua. The primary sources of data were Key Informant Interviews (KII) and Focus Group Discussions (FGD), informed by document review and an on-line survey of a purposive sample of individuals involved in the programs being evaluated in countries not visited plus some other individuals whom the Team was not able to speak with directly. The initial list of Key Informants to be interviewed was provided by USAID and augmented by other individuals identified as the evaluation progressed. They included current and past actors with key roles in the management and organization of each of the three programs and/or representatives of other major stakeholders: senior MoE personnel; PREAL, CERCA and CETT coordinators and managers; senior-level USAID staff; and other important stakeholders. The IBTCI Team conducted 123 KIIs in the eight countries. Additional interviews were conducted in Washington, D.C. and via telephone with USAID staff and others involved with the programs. The team met with 12 USAID staff at missions and met or spoke with eight other current or former USAID staff in Washington or elsewhere, including three program officers now on other international assignments. The teams also conducted a total of eleven focus group discussions (FGD) in the countries visited. Seven groups were held for representatives of the public sector and four for the private sector (in the Spanish-speaking countries – C-CETT had negligible private sector involvement). Most had attendance varying from five to 15 participants. In addition to the KII interviews and focus groups, IBTCI constructed an on-line survey questionnaire, both in Spanish and English, and e-mailed instructions to a purposive sample of some 200 particularly relevant informants (e.g., administrators, senior professionals, trainers, and ministry staff), who, for the most part, were in countries where the IBTCI teams did not visit. The team received responses from 76 individuals in twelve countries. This response rate was sufficient to indicate that the feedback would probably reflect the purposive sample as a whole. However, in large part because only three teachers and seven teacher-trainers responded, the team opted to consider the survey results as being illustrative and confirmatory rather than conclusive. The team also reviewed project documents, which provided highly valuable information as to the implementation process of the projects, their activities over time, etc., providing perspectives on points that the team should explore with the interviews and focus groups. Of particular value was the 2010 evaluation that The Mitchell Group conducted for USAID/Peru on AprenDes and CETT Andino and the draft 2010 White Papers that the Aguirre Division prepared on CETT. Analysis of Findings The team found that both PREAL and CETT were effective as regional programs in supporting the education activities being assisted by local USAID missions, giving beneficiaries the opportunity to learn from the experiences of colleagues in other countries that could not have been provided through Evaluation of LAC/RSD Regional Education Program: Volume I vi International Business & Technical Consultants, Inc. bilateral programs. However, this work was hampered due largely to inadequate coordination between the centrally-funded projects and individual missions and due to negligible interaction between PREAL and CETT. For CETT, coordination and cooperation improved greatly when each of the sub-regional CETT programs was assigned an AOTR who was based at a mission within the region. A definite factor contributing to the success of CETT was that each of the sub-regional programs had significant flexibility to address the specific needs and circumstances of the respective sub-regions. PREAL, which also includes countries that do not receive USAID assistance, was able to provide education stakeholders in individual countries a comparative research base that allowed them to bring data drawn from other countries in Latin America to help influence positive education reform in their own countries. For practical purposes, CERCA functioned as a set of bilateral programs with some regional components; although it did have successes, notably the report cards, its effectiveness was limited because it did not have either the time or the money that was originally authorized. The Benefits and Challenges of the Regional Programs Benefits. Informants involved with PREAL and all implementations of CETT reported there were clear benefits obtained from working with such regional activities. In addition to support for their own direct work, the majority of informants indicated that, as a result of their participation, these programs challenged them and let them look beyond their own borders at other educational system and to see how some counterparts already had tackled many of the educational problems they faced themselves. This opportunity to learn from the experiences of educators in other countries, particularly if the other countries had progressed farther than their own with respect to particular matters, was something that could not have taken place with bilateral programs. Vignettes of representative majority responses follow. C-CETT respondents listed as benefits: access to regional experts in reading and literacy, and benefitting from regional “lessons learned” and sharing experiences in the sub-region; and stimulation from high-level discussion on literacy at the national levels that produced innovative methodological and instructional materials. Respondents said that C-CETT brought a new approach to teaching reading whose benefits included using data for decision-making at the policy level, promoting accountability within ministries, improving school leadership, and having some influence on policies and teacher￾training initiatives. The use of diagnostic tests for measuring progress and influencing teaching strategies was a strong, positive impact on approaches for teaching reading, and ultimately on reading achievements. University-based CARD-CETT respondents indicated that this sub-regional program definitely made a positive contribution toward the expansion of institutional and national capacities in the areas of research, developing educational materials, education, and assessment. The institutional-level benefits reported included learning how to solve problems collectively and across national borders; analyzing data to identify educational issues of mutual concern; working together to develop and produce educational materials and publications that met international quality standards; and the opportunity to develop and expand regional professional networks. As institution-wide organizational development benefits, in addition to those primarily related to the education sector, these staff also reported that their knowledge of project and financial management increased from working with CARD-CETT and that their institutions learned how to write proposals for external funding and to develop outreach programs. Many informants stated that most of these benefits would not have occurred without an adequately funded regional program such as CARD-CETT. Andean-CETT respondents provided perspectives similar to those of C-CETT and CARD CETT. Among specifics they noted were: exposure to international ideas on differentiated instruction that helped teachers to explore management techniques for classrooms with students at different cognitive Evaluation of LAC/RSD Regional Education Program: Volume I vii International Business & Technical Consultants, Inc. levels, benefitting both higher- and lower-capacity students; stimulation of a technical discussion in Ecuador about literacy, which resulted in a study that identified gaps in performance in literacy and the lack of in-service training programs to address them and, as Peru’s Technical Vice-Ministers of Education reported, help in consolidating the MoE’s reading and comprehension methodology and its evaluation process; and use of a feedback process, which encouraged teachers and trainers to give their opinions about the program’s positive and negative aspects and which incorporated their input, which was used to improve the Andean-CETT program. PREAL designed and promoted reforms to rectify various problems in Latin America’s educational systems, including setting learning standards and evaluating progress toward educational goals; developing tools to monitor progress on education reform to improve accountability at the regional and national levels; and strengthening the capacity of national public sector and private sector partners to promote education reform. Among its most successful activities have been the Working Group on Learning Standards and the use of report cards to spur reform. CERCA’s major contribution was the development and implementation of school-based report cards. Challenges. By far the major challenge – and it was a significant challenge – involving the regional programs, particularly CETT, which had direct involvement within schools, was inadequate coordination between the regional programs and individual missions (and Ministries of Education), especially at the early stages of CETT. Missions and MoE did not feel adequately involved in planning the CETT sub￾regional programs and determining how CETT approaches could be integrated with other ongoing activities. This greatly hampered the ability of MoEs and colleges of education to institutionalize CETT approaches. Also, in part because there was no planning to create linkages between the two projects, there was negligible interaction between PREAL and CETT, even though each had much to offer the other. The Broader Impacts of the Regional Education Programs The study revealed important and significant outcomes and impacts attributable to CETT and PREAL. Among the most important for CETT, the study found to be 1) the measurable improvement in teachers’ pedagogical skills and in students’ reading and writing skills; 2) introducing innovative methods for teaching reading and writing; 3) introducing a practical and appropriate teacher-training model; 4) introducing a relevant learning-assessment approach to measure the effectiveness of teaching reading and writing; 5) introducing a school-level evidence-based M&E procedure; and 6) making a positive change in the way partner institutions organized and managed themselves. PREAL’s major impacts to date include: 1) systematic applied comparative research and associated dissemination of relevant data, “best practices” and “lessons learned”; 2) assessment instruments, including report cards; 3) the capacity-building program; 4) the Network and Policy dialogue; and 5) setting up of cross-national Working Groups on educational quality standards. Impacts on Attitudes, Perceptions and Policy The extent to which regional programs changed attitudes was most evident and significant at the school level, among teachers and school principals, where educators became strong advocates of the CETT model. At the ministerial level, the regional programs increased awareness of the benefits of reading and writing programs as well as the need to train teachers in specific pedagogical methods. However, the regional CETT programs could have done a better job of informing senior government officials of the positive impact that the CETT model made on improving the reading and writing skills of students in Evaluation of LAC/RSD Regional Education Program: Volume I viii International Business & Technical Consultants, Inc. poor and marginalized schools, and better integration with PREAL would very likely have led to greater success. The influence of the regional programs on national educational policies was uneven and difficult to assess. PREAL had a significant influence on attitudes and policy-making by introducing issues such as accountability, a relatively new concept in Latin America, into the policy dialogue through report cards, conferences and publications and by the development of learning standards. PREAL also developed approaches to engage business leaders in efforts to strengthen tangible and intangible support for public education. With respect to CETT, the majority of MoE informants interviewed stated that while CETT teacher-training and pedagogical interventions were generally recognized as having a positive impact on teaching reading and writing, CETT’s activities have had less of an impact on national education policies than could have been expected, in part because the ministries were not fully involved in the CETT program from the beginning, which limited the ability of national education decision-makers to incorporate CETT approaches into ongoing activities, and in large part because of perceptions that some aspects of CETT activities are too expensive to be implemented on a broad scale. Colleges of education will be able to incorporate some aspects of the pedagogical approaches in their courses, but this will be limited given that the implementation of CETT approaches into the schools will generally be limited. Because CERCA was a short-lived and truncated pilot program, its only real impact has been through the use of school-based report cards. The Regional Models’ Capacity and Sustainability Capacity and Sustainability are separate issues. To varying degrees, CETT developed capacity in the MoEs in Nicaragua, Honduras, the DR, Guatemala, Peru, Ecuador and Jamaica by introducing the model and its components, and by training specialized staff. The latter included Reading Specialists, school supervisors and trainers of teacher-trainers who were trained how to implement the CETT pedagogical approach in primary schools. But further capacity-building is necessary: the extent of attrition among CETT-trained personnel at MoEs will be a critical factor inhibiting the replication and sustainability of the model. In the Andean region, CETT has encouraged the organization and training of teachers to promote CETT approaches to the teaching of reading. The extent to which the teacher-training institutions are capable of and interested in teaching CETT’s technical components is limited. CETT strengthened the capacity of teacher-training institutions in Jamaica, Nicaragua, the DR, Honduras, Guatemala, Ecuador, Peru and Bolivia, and the teacher-training institutions that were more engaged in the CETT were more likely to have integrated CETT’s components and principles into their teacher-training courses and curricula. However, in general, the model was not well integrated into national college programs and the national teacher-training curriculum. This has been a major factor in the teacher-training institutions’ limited capacity and interest in continuing to teach the CETT model’s technical components in their curricula. In the Caribbean, for example, where the MoEs did not officially adopt the model and, therefore, the model has not fully incorporated into the teacher training curricula, too few teachers will be CETT-trained to ensure the program’s sustainability. PREAL has helped its partners to develop their capacity in applied research, policy development, and policy advocacy. USAID funding is scheduled to end in 2012; however, PREAL does have other funders (although the team does not know what plans, if any, there are for any of them to continue their support after 2012), and it is our belief that a number of PREAL’s partners have institutionalized the approaches developed by PREAL. Evaluation of LAC/RSD Regional Education Program: Volume I ix International Business & Technical Consultants, Inc. The Value-Added of Technical Assistance Technical assistance was provided both by U.S.-based organizations such as AED and the Aguirre Division of JBS International and by peer program partners, such as Universidad del Valle de Guatemala (UVG), which developed a useful school- and classroom-based M&E system that later was transferred through training workshops to schools and classrooms in Guatemala, Honduras, the DR and Nicaragua. Respondents found that the technical assistance on topics such as Differentiated Instruction, active learning, and a broad range of M&E-related activities was highly effective in improving the effectiveness of CETT. They reported also that the most useful types of technical assistance in the region were the inter- and intra-regional meetings and workshops in which people from different—or even the same – areas came together to: (a) share experiences related to implementing their respective CETT models, or (b) learn about specific pedagogical approaches that enhanced teaching literacy or the learning processes related to reading. Lessons Learned and Recommendations The recommendations and “lessons learned” presented here are drawn from the analysis of the information gathered through the review of project documents and, particularly, through the observations during the field work to the eight countries in Latin America and the Caribbean whose educators participated in at least one of the programs, as further informed by the on-line survey and telephone interviews. While the recommendations and Lessons Learned come from the information that was gathered by evaluating three regional projects, the recommendations that are offered here are being submitted as points that should be considered and adopted in the planning of future regional education programs as well as for further PREAL and potential CARD CETT institutional network (RICETT) activities. Lessons Learned 1. A program should have active links with national and international projects and programs to be relevant and have impact. Based on interviews with senior ministry and CETT staff in visited CARD and Andean CETT countries, findings revealed that opportunities had been missed to better integrate the best practices and policies being developed by both PREAL and CETT into quality improvement strategies for USAID bilateral projects. • Exploiting the synergies between CETT’s center-based program-implementation strengths and PREAL’s strong policy and public-private partnership (PPP)-based approach probably would have increased both programs’ relevance, acceptability, and effectiveness. • Exploiting potential synergies among the regional programs in key areas of education reform (e.g., combining CERCA’s successful school-based report card (SRC) project with PREAL’s National Report Card initiative to improve accountability between student assessments and developing relevant national education learning indicators, or substantial sharing of the PREAL standards for reading with CETT activities to improve the teaching of reading) could have resulted in a greater impact on education quality improvement reforms throughout the LAC region. 2. A business leadership model like that created within PREAL can contribute to promoting entrepreneurship in education. This type of model also encourages community leadership by accelerating the dissemination of information about important educational issues to the public. This observation is the outcome of discussions and interviews carried out with PREAL coordinators and senior policy makers in ministries of education and in CETT partner institutions in the CARD, Andean, and Caribbean CETT countries visited. However: Evaluation of LAC/RSD Regional Education Program: Volume I x International Business & Technical Consultants, Inc. • While private-sector support can make important contributions to a program, maintaining this support needs special, continuous attention from USAID and the Implementing Partners. This is particularly the case for countries which have had only limited experience with formalized public-private partnerships. 3. The CETT applied-research and in-school evaluation model should be widely disseminated for potential adaption/adoption, owing to partner institution staff in the DR, Honduras and Ecuador drawing attention to the importance of the work that had been carried out by the UVG in Guatemala in the development of a practical and innovative monitoring and evaluation process, for use in schools. 4. Private-sector schools and key national institutions like MoEs should be fully involved in regional programs like CETT so that program ownership is transferred to them to ensure sustainability. Andean CETT partner institutions’ commercialization of CETT-generated information, materials and procedures was a disincentive to the Latin American MoEs’ acceptance of CETT. This observation was shared by CETT managers in Ecuador, Honduras, and to a lesser extent in the DR. A majority of MoEs stated that because they had not been fully involved with CETT from the beginning, CETT’s influence on national education practices was not as great as it could have been. However, it also should be noted that other factors also contributed to the ongoing challenge of engaging MoEs in CETT, such as the high turnover of many ministers. 5. USAID/Jamaica and C-CETT’s relationship shows that a Mission responsible for managing a regional education program must be able to fully integrate other local Missions and countries into working with the program. While the C-CETT’s AOTR’s location within USAID/Jamaica proved effective for that Mission’s bilateral education program, the USAID/Barbados Mission felt isolated by the implementation of a regional educational program in a geographical area for which it was solely responsible, i.e., the Eastern Caribbean. USAID/Barbados personnel felt that they lacked involvement in C-CETT, which they also believed impacted negatively on the sustainability of the C￾CETT Model in the Eastern Caribbean. In CARD-CETT, all five countries had signed an MOU to commit resources and effort into supporting CETT activities within their countries, but only three countries seemed to have had committed the LOE and resources needed to make this a reality. 6. USAID missions should provide support and consistent management to a teacher-training model like CETT’s in order for the model to achieve its full potential. Sustainability of CETT was seriously questioned, especially in countries like Honduras, Ecuador, and Peru, where financial and technical support for CETT from USAID missions and ministries of education was weak or absent. 7. Teacher training and other models developed by local educators can benefit greatly from the formal and informal knowledge and skills sets derived from collaborating with similarly situated peers in other countries, in addition to technical assistance. 8. Regular national and/or regional meetings promote program progress and achievements. Throughout our field work respondents raised the importance of regional conferences, seminars and other events. For many individuals interviewed, travel and cross-border exchanges were the most important and memorable aspect of the regional program experience. 9. USAID should manage community-based programs like CERCA through bilateral agreements between missions and competent local institutions or partners, rather than centrally or by U.S.- based contractors. 10. Washington-based USAID programs can have an important positive impact on the quality of education in the LAC region. Overall, this evaluation found that all three regional programs did Evaluation of LAC/RSD Regional Education Program: Volume I xi International Business & Technical Consultants, Inc. manage to raise awareness of the problems each attempted to tackle. In the CETT and PREAL programs, this awareness was translated into action and in some instances actions that have left their mark on national practice. For example, the CETT teacher training model had a positive impact on teacher training practices in most CETT countries. 11. Washington-based education programs can address common education issues across a region and work in countries that bilateral projects cannot reach. However: • Local Missions should have a major role in designing and planning regional programs so that they are better able to fully support program implementation. • Field missions should be given major program implementation responsibilities because of their knowledge of, and proximity to, the field. CETT worked much better after AOTR’s for each cooperative agreement started working from missions in each sub-region. 12. Regional programs like PREAL, CERCA and CETT have a better chance of succeeding if the USAID missions involved have active bilateral education programs supported by an education officer. In countries without USAID bilateral programs, which was the case in several Caribbean countries visited, or where USAID had a bilateral program but didn’t have mission level education support, regional programs were less likely to succeed because they were relatively unknown, lacked sufficient financial and technical support, and were poorly supervised. 13. ICTs can make important contributions to project success if well planned and implemented, especially where regular communication is critical for project implementation. 14. USAID must ensure that the hub-partner institutions responsible for managing/disbursing program funds have good project management skills. A more careful selection of lead partner institutions is called for if CETT–type regional program approaches are to be adopted in future. Respondents in C-CETT, CARD-CETT and Andean-CETT all complained about weak leadership and poor management skills on the part of their respective lead institutions. 15. In-depth understanding of the local public- and private-sector cultures is imperative before transferring or planning to transfer regional education programs like CETT to these sectors for onward implementation and funding. Recommendations The recommendations below are linked to the previous analysis, findings and conclusions, which are derived from the Team’s comprehensive analysis of data gathered from both secondary documentary sources and field data collected from interviews, focus groups, informal discussions and field observations. This data was complemented by information collected through an online survey, which was distributed to key informants who could not be reached face-to-face. Data gathered using all these methods was aggregated into emerging themes and then carefully entered into five matrices that answer research questions listed in Annex VII of Volume II. A further synthesis of the data entered provided the analysis and findings sections above. The Benefits and Challenges of Regional Education Program Models 1. Designate Mission–Based USAID Staff to Manage Regional-Bilateral Program Relations: The Team found that the two major strengths in the PREAL program - its flexibility and responsiveness – were undermined by the program’s difficulty in engaging with local USAID missions and their bilateral programs. To compensate for this and maximize PREAL’s benefits, the Team recommends that the AOTR/COTR of a Washington-based program have support (possibly an alternate AOTR/COTR Evaluation of LAC/RSD Regional Education Program: Volume I xii International Business & Technical Consultants, Inc. who works with or for the AOTR/COTR specifically) to serve as a project coordinator and communication facilitator. This would help future regional initiatives to maintain (or improve) communication with local USAID Missions, improve coordination and synergy, fill any other gaps that might exist in the program’s management structure, and ensure strong relations with USAID bilateral programs. 2. Engage host country decision-makers early in the program planning process: Host-country education decision-makers in the DR, El Salvador, Ecuador, Peru, and Bolivia, for example, resisted CETT as a sub-regional program because of inadequate pre-program consultations with USAID. A more localized approach to encourage greater integration and participation and frequent ongoing dialogue, would be helpful. If not already in place, we recommend establishing Technical Working Groups (TWG) comprised of senior MoE officials, project implementers and managers, and USAID education-office staff. TWGs should meet quarterly to review project activities and gain a better perspective on potential interventions – before the program begins. 3. Plan Ahead for Distance-Learning Activities: The strategies that the CETTs proposed to introduce ICTs and education technologies into the teacher-training processes were challenging, or unworkable for most CETT countries. To better inform any future program that may try to use similar approaches, USAID/W should first require a review of the proposed ICT technologies and confirm that the implementers and intended beneficiaries in a country to ensure that the local Mission/s have – or can be provided – the technology and expertise necessary to support distance learning. USAID/W also should review and evaluate the ICT approaches that C-CETT, CARD￾CETT and the Andean CETT developed, as well as assess the roles that local providers and Mexico’s Latin American Institute for Education Communication played in supporting ICT services for USAID’s regional programs. 4. Maintain/Expand Effective Program Components: The Team concluded that several interventions and project activities merited special attention. Successful regional program components require sufficient levels of technical support, including coaching, monitoring and evaluation, and sufficient funding to ensure their implementation and their expansion. Effective regional activities, such as PREAL’s Strategic Partnership Project in Central America, provide a useful model for local capacity￾building that merits support. a. The DR, Honduras, Nicaragua and El Salvador decided to include CETT components in their national education reform plans. Several of the institutions participating in CARD-CETT agreed to continue CARD-CETT activities through Red Inter-institucional del CETT (RICETT). PREAL plans to extend activities at least through 2012. It would be advantageous for USAID to consider providing some degree of support to RICETT through the remaining funding period for PREAL, either separately or in conjunction with PREAL. This may help to generate some collaboration between the programs that has been lacking to date in this regard. If it is not feasible or possible to develop a formal follow-on equivalent, this support could take the form of support for technical assistance or travel to be undertaken in conjunction with regional/sub-regional activities being funded by MoEs or other donors, conferences, etc. b. Successful regional program elements like PREAL’s Strategic Partnership project in Central America need encouragement by providing adequate funding and technical support such as coaching and monitoring and evaluation, to ensure proper implementation. c. Successful elements like the CERCA SRCs and PREAL Report Cards (RCs), whose expansion was hampered due to lack of funding, should be more widely adopted and incorporated into USAID basic education programs because of their positive impact on accountability. Report Cards can be used from the single-school level to the national-system level, and experience Evaluation of LAC/RSD Regional Education Program: Volume I xiii International Business & Technical Consultants, Inc. shows that they have been highly effective in Latin America as well as in Eastern Europe and in some African countries. According to Ecuadorian respondents, Ecuador’s Ministry of Finance threatened to reduce the Ministry of Education’s budget because, according to the 2008 RC, the MoE had not been spending its budget efficiently. d. Bilateral programs should provide bridging funds to support the work already going on in local hubs of excellence at centers like “The Center for Language Teaching Excellence” at the Pontifica Universidad Católica Madre y Maestra in the DR, the “CETT Technical Support Center” at the Universidad Pedagógica Nacional Francisco Morazán in Honduras, or the CETT Service Unit at the Universidad Andina Simon Bolivar in Ecuador. All are CETT products and have a good chance of continuing their good work if they receive enough funding to support them until they become more self-supporting. 5 Adopt and Adapt Effective CERCA Activities: CERCA program elements that stakeholders reported were effective included the Report Cards, teacher-parent collaboration, school-based action research and community-involvement activities. USAID should consider adopting and adapting these elements into local bilateral programs if community-based regional programs similar to CERCA are implemented in future. Coordination and Synergy among Regional Programs 1. Give Partners Early Training in Building PPP Alliances: The Evaluation Team recommends that USAID modify its approach toward implementing regional programs like CETT by enabling partner institutions and countries participating to become better prepared. Sub-regional partner￾coordinating institutions need to be trained how to build stronger working alliances with important program partners in both the public and private sectors before programs are implemented. The PPP approach was not effectively implemented in the majority of CETT countries and program managers did little to encourage MoEs and teacher-training institutions to become owners and partners of the three sub-regional programs. Sharing the lessons learned from PREAL’s previous experience in this area, would have been helpful. 2. Promote Even Regional Distribution of Working Groups: Assuming that resources are available, the Team recommends that PREAL establish working groups (WG) in the Caribbean and Central American sub-regions, in addition to those already operating in Peru and Uruguay. Creating more WGs will broaden knowledge-sharing across a wider area, as well as increase ownership of the knowledge-creation process. New WGs should follow the example of groups in Peru, where a national discussion on learning standards has also worked to strengthen existing mechanisms for monitoring and evaluating learning progress at the regional level. 3. Promote PPPs through Additional Working Groups: Creating more WGs to improve education within countries may enable “The Strategic Partnership Program” to play a more supportive role in increasing local capacity and bridge-building between the public and private sectors in the context of CETT-type regional programs. 4. Share CETT’s Lessons and Achievements with PREAL: In principle, the LAC Bureau’s decision to complement the PREAL program with CETT was well-founded. In practice, however, there was little interaction between the two projects, either regionally or nationally. As a result, the adoption and institutionalization of both PREAL and CETT policy recommendations was slow and less successful than had been expected. The work of PREAL is continuing in a number of countries. PREAL partner institutions have been successful in using both comparative and national research to promote national and sub-national education reforms, and it is anticipated that the program will continue advocating education reform to MoEs and other public and private education-sector stakeholders. PREAL also has voiced Evaluation of LAC/RSD Regional Education Program: Volume I xiv International Business & Technical Consultants, Inc. interest in utilizing CETT’s successful examples to further these goals. As a result, the team offers the following recommendations: a. Literacy and reading are both fields in which the quality of teaching and learning would have benefitted from greater interaction between the two programs (i.e., CETT schools would have gained from the work PREAL did in developing national educational standards, and PREAL would have benefitted from learning how useful CETT’s standards were in helping children from highly disadvantaged backgrounds to learn). USAID can use an existing forum to provide for knowledge-sharing between the two programs by involving qualified staff from participating CETT academic and teacher-training institutions into PREAL’s working groups. This would add experiences taken from classroom teaching situations to the recommendations that national PREAL partners make to their MoEs and other stakeholders. Conversely, national PREAL partners can draw attention to potential constraints that may have been overlooked in advocating the adoption of CETT’s teacher-training approaches at the national level. b. Most CETT partner institutions represented teacher-training institutions and the few schools that had implemented elements of CETT. By working more closely, CETT staff collaborating with PREAL WGs could benefit from knowledge-sharing about alternative/additional pedagogical approaches already in use. To bolster such sharing, the team recommends that the two programs’ collaboration be more systematic than merely attending workshops or doing panel presentations at conferences; rather, such interaction should be frequent and rigorous enough to promote increased opportunities for robust applied research. More frequent collaboration may also create greater opportunities for networking with education sector decision-makers, both at the national and regional levels. c. A closer collaboration would give CETT and PREAL an opportunity to re-engage MoEs in a dialogue that could lead to a more comprehensive and, preferably, more cohesive evidence￾based approach for effecting positive educational change. For example, a major objection to adopting CETT approaches voiced by MoEs was that while they appreciated the CETT teacher training methodology, it was unaffordable, given their current budget constraints. Here, perhaps, joint studies undertaken by CETT and PREAL partners in participating countries could document what improvements, if any, in educational efficiency resulted from adopting CETT components. Developing a hypothetical analysis was well beyond the evaluation’s SOW. 5. CETT’s Sustainability: Although the CETT approach has proven to be both effective and popular with teachers, with the end of USAID funding and with the attrition of CETT trained personnel, the impact of CETT probably may dissipate fairly rapidly. To make up for the end of CETT, Missions with bilateral basic-education programs should confer with implementing partners to determine how the CETT experience can be incorporated into ongoing activities, either broadly or at the district/region levels. Interested Missions should also discuss with official stakeholders about the desirability and feasibility of incorporating appropriate aspects of CETT into their national pre-service and/or in-service teacher and administrator training, and professional development. Coordination and Synergy among Regional and Bilateral Education Programs 1. Consider Alternative Contracting Models: CERCA’s centralized LAC Bureau management and funding structure required the program to operate regionally and through a cooperative agreement with a U.S.-based technical assistance provider such as AED. This arrangement proved less than productive in CERCA’s case, which functioned essentially as a collection of bilateral programs; comparable programs in the future could benefit from a more direct transfer of knowledge and skills Evaluation of LAC/RSD Regional Education Program: Volume I xv International Business & Technical Consultants, Inc. to local institutions. Local technical assistance can improve local ownership and contribute to the sustainability of successful program components after the assistance ends. 2. Bilateral Involvement in Regional Programs: All USAID missions in the LAC Region should designate activity managers to supervise the implementation of regional programs in the countries for which they are responsible. This would help to encourage the sustainability of the activities that regional programs like CETT and PREAL are implementing. The Broader Impacts of the Regional Education Programs 1. Long-Term Perspective: For regional programs to have a significant and lasting effect on national and regional education polices, they need to operate for sufficient time beyond the pilot phase to allow innovations to be absorbed into local educational practices. This can only happen if USAID provides the funding required to encourage MoEs to play a more active supporting role, to sustain program implementation over a period of at least eight to ten years. As an overarching strategy – rather than a specific recommendation – the Team therefore suggests that the personnel supervising and implementing regional programs have this long-term perspective in mind. For example, PREAL required 15 years of focus on education policy dialogue within and between ongoing national education reforms in LAC to ensure an active role for itself in education policy-making at the national and regional levels. For programs like CETT and CERCA, which were shorter-term in nature and focused on changing specific aspects of education practice, the policy￾influencing role was less certain. It should be noted that both CERCA and CETT aimed at influencing national education policy indirectly, by developing and implementing regional solutions to common education problems. Attitudes, Perceptions, and Policy 1. Increase MoE Involvement: In case future CETT-type regional programs are established, it is imperative, in cultural contexts like LAC, to include MoEs as equal partners in the program planning and implementation phases from the outset. Not doing so will threaten the sustainability of policy￾sensitive reforms like those that CETT proposed in the LAC region. CETT evidently has had some influence on changing education teacher-training practices in some countries – mainly where the CETT model was perceived as the outcome of a shared process between MoE officials and program implementers. Creating a regional MoE Consultative Committee would help to improve the sharing and coordination of partner country perspectives on education reform. 2. Engaging the Private-Sector: PREAL and its partners need to provide more opportunities for systematic engagement with the private sector and the business community at the national level. This is needed because the Team found that, with the exception of Honduras’s Chamber of Commerce (Camara de Comercio), respondents from four of the seven countries visited knew little of PREAL’s knowledge-sharing business approach. Creating opportunities for private-sector engagement in education policy reform is important because most PPP initiatives in the region are focused mainly on school infrastructure and construction rather than improving education quality. Capacity and Sustainability 1. Support Local Education Service Providers: Providing printed and published materials is essential in a program that promotes reading and writing, and should be shared equally among private-sector partners. Each country should have the chance to compete for contracts to produce and print essential materials through local service providers. This will make the materials better fit the young students’ educational settings in the different countries. Local production also will reduce potential resistance to using materials developed with a regional, rather than a national, student body in mind, it will create jobs and it will be an incentive for the local publishing and printing industries to Evaluation of LAC/RSD Regional Education Program: Volume I xvi International Business & Technical Consultants, Inc. improve the quality of their services. A centrally-contracted international provider may appear to be efficient but it may hinder the building of sustainable local capacity in this key education service area. 2. Allow Flexibility for Varying Private-Sector Roles: In countries without a tradition of private-sector support for social development, such as in most Central American countries and in the Caribbean, the Team recommends that USAID not require developing PPPs as a prerequisite for funding and support to regional programs, since this may be perceived as inappropriate by partners. Where private-sector engagement is already a social norm, developing strong private-sector support for regional education programs like PREAL and CETT is feasible only if public-sector institutions are comfortable sharing donor funding and support with the private sector. 3. Leverage Corporate Social Responsibility Efforts: Future regional initiatives should promote PREAL’s approach especially as regards how the “Business Leadership and Education Program” might play a key supportive role in countries where the CSRM already has made an impact. In Peru, where the CSRM has a full agenda, respondents reported having very limited knowledge of PREAL’s PPP knowledge-sharing activities. Some CSRM respondents spoke of difficulties in identifying best practices to help the CSRM improve the promotion and sharing of available business resources like knowledge, financial resources, and management skills; others were not sure if CSRM’s resources for education programs were being used to support the best and most appropriate national education initiatives. 4. Improve Implementers’ Management Capacity: Implementer’s lack of organizational “know-how” about how to technically and operationally manage a sub-regional project was a major constraint in initiating CETT. The staff of regional coordinating institutions that are new to this type of activity could benefit from being mentored by more experienced institutions such as CINDE or UWI, or informal guidance from peer institutions participating in, for example, the USAID-assisted Higher Education for Development activity. 5. Public-Private Partnerships: There was very significant variability in the establishment, let alone the maintenance, of PPP in the countries visited as a whole and within the sub-regions. Due to time and the nature of the SOW for this assignment, it was not possible for the team to do more than touch on apparent effectiveness, causes, and outcomes/impacts of PPP for the five programs. We recommend that LAC commission assessment of the role and effectiveness of PPPs (a) in education in the LAC region and/or (b) in a sub-region within the LAC region, to provide region-specific complementarity to the work being done by USAID's Global Development Alliances. Evaluation of LAC/RSD Regional Education Program: Volume I 1 International Business & Technical Consultants, Inc. I. INTRODUCTION Summary: USAID/LAC’s Regional Education Program Over the last 20 years, the United States Agency for International Development (USAID) Latin America and the Caribbean (LAC) Bureau implemented a series of ambitious regional educational programs. Effective from 2001, activities under the LAC Regional Education SO focused on improving the equity and quality of education through three Intermediate Results (IR): (1) enhancing the environmental for education reform, (2) improving the skills of teachers and administrators, and (3) improving the relevance and skills of the workforce. Since 2009, the LAC Bureau has taken a two-pronged regional approach to address challenges in the education sector: improving the quality of education by creating effective models for change, and improving the quality and relevance of policy dialogue around education reform, both as a means of strengthening regional capacity to implement reforms, as well as a way to build political support for improving educational quality. To help the LAC Education team better target USAID development resources and better support LAC field missions, it has contracted with IBTCI to conduct an evaluation of three regional education programs which addressed the first two Intermediate Results listed above. Each program was designed to address a specific aspect of the region’s education problems. The LAC Bureau addressed education needs in the region using multiple programs. Chronologically, the Partnership for Educational Revitalization in the Americas (PREAL) is the first of the three regional programs that are the subject of this evaluation; it began in 1995 and USAID support is scheduled to end in 2012. It consists of a network of civil-society organizations, research groups, and private-sector entities in 17 countries in Latin America, not all of which currently receive USAID assistance for education. The second initiative, the Centers of Excellence for Teacher Training (CETT) program, was the only one of the three programs to work in the English-speaking Caribbean as well as in Spanish-speaking countries and was implemented essentially as three coordinated sub-regional programs. It began in 13 countries in LAC and expanded further, partly through “self-funding,” to additional English-speaking countries in the Caribbean. CETT began in 2002 and ended in 2009, with some related activities continuing into 2010. Finally, CERCA, the Civic Engagement for Educational Reform to Central America program, operated in Guatemala, Honduras, El Salvador, Nicaragua, and the Dominican Republic (DR) from 2002 to 2006. In practice, therefore, the activities described in this report are as much sub-regional (i.e., English￾speaking Caribbean, Central America and the Dominican Republic, and Andes) as they are regional (i.e., Latin America and the Caribbean as a whole). Combined, USAID support for the three programs served 21 of the countries in the LAC region. The PREAL initiative has addressed education needs in the LAC region by promoting improvements in the quality and relevance of policy dialogue in education research and reform; USAID has active bilateral education programs in a number of PREAL countries. A key aspect of the PREAL program, which has been part of USAID/LAC education policy reform efforts for almost 14 years, has been the effort to strengthen regional policy dialogue on important issues such as educational accountability, better education planning and the dissemination of reports on the quality and performance of national education systems, through National Education Report Cards. PREAL’s goal was to produce and use knowledge to improve the quality, efficiency and equity of education in LAC. The program was designed to engage policy makers, researchers and academics in more precisely defining the education problems in the LAC region in order to initiate a higher-level policy dialogue that would lead to education reforms. PREAL implements more activities in Central America than in LAC’s other sub-regions (defined as Central America, the Caribbean and the Andes in this report), although its Working Groups (WG) are based mainly in South America. This composition of WGs may be partially a product of the fact that Research Corporation for Development (CINDE), its key implementer, is based in Chile; however, Central American educators felt somewhat left out. PREAL has considerable engagement in Evaluation of LAC/RSD Regional Education Program: Volume I 2 International Business & Technical Consultants, Inc. countries that produce national education report cards and that establish national and business partnerships for education. The second of USAID/LAC’s regional education programs focused on improving education quality by creating effective change models through the CETT. The three regional LAC CETTs were established to provide an effective model to improve teaching literacy in the first three primary grades, with a special focus on particularly disadvantaged children. The program was designed to be a Public Private Partnership that worked to create collaboration among USAID, private-sector companies and international donors. As of 2009-2010, the three CETTs had reached over 800,000 children and trained over 35,000 teachers and administrators. CERCA was the Bureau’s third approach, focusing on the mobilization of broad constituencies to improve school quality in five countries in the CA-RD region: Guatemala, Honduras, El Salvador, Nicaragua and the Dominican Republic. The project’s timeframe was 2002-2006 but in mid-2005, due to severe budget cuts, USAID/LAC advised CERCA that no new obligations would be made, which reduced CERCA’s funding by one-fifth to $2.7 million. CERCA’s program activities were revised and it was unable to achieve all of its goals in the five countries. Evaluation of LAC/RSD Regional Education Program: Volume I 3 International Business & Technical Consultants, Inc. II. EVALUATION OBJECTIVES AND RATIONALE A. Evaluation Purpose and Objectives The overall purpose of the evaluation was to assess the comparative advantages of LAC’s regional education programming in order to identify best practices and to inform the design of its new regional education interventions. The specific objectives of the evaluation were to: • Evaluate the benefits and challenges of supporting different models for regional programming, including the advantages and disadvantages of working regionally versus bilaterally to address education development challenges; • Contrast the different models for program implementation at the USAID- and implementing-partner level; • Identify the broader impacts of the regional programs; and consider the value-added of USAID/Washington-based contracts and cooperative agreements to provide technical assistance to implementing partners (IP); and • Provide recommendations for the design of future regional assistance programs in the education sector. B. Rationale The evaluation was intended to provide guidance in the design of potential future LAC regional education interventions, as well as a broader collective assessment of the three initiatives rather than individual studies of each program, several of which have already been conducted. Essential to the IBTCI Team’s approach was the effort to better understand the impact of the changes that the three regional programs produced with respect to ongoing national education reforms and to USAID’s own bilateral programs. It should be noted that the short time for work in each country did not always provide sufficient data to determine this impact on the issues of reform and social change. Nevertheless, the Team developed a methodology that prioritized collecting qualitative data through key informant interviews and FGDs in order to capture to the greatest extent possible the political, cultural and institutional changes that are likely to have resulted from the impact of any or all of the three regional programs. Tables 1 and 2 on the following pages summarize USAID/LAC’s life-of-project (LOP) dates, lead organizations, implementation structures, USAID’s role, and other key aspects of these programs, by program. Evaluation of LAC/RSD Regional Education Program: Volume I 4 International Business & Technical Consultants, Inc. Table 1. USAID LATIN AMERICA and CARIBBEAN (LAC) REGIONAL EDUCATION PROJECTS - Overview Caribbean CETT CARD CETT Andean CETT PREAL CERCA Dates 2002 - 2009 (except as noted below) 2002 - 2009 2002 -2009 1995 - 2012 2002 - 2006 Lead Organization Joint Board of Teacher Education (Jamaica). Lead office UWI Mona in Kingston with support from UWI Cave Hill in Barbados. Univerdidad Pedagogica Nacional (UPN), Honduras Universidad Peruana Cayetano Heredia, Perú Jointly managed by the Inter￾American Dialogue, Washington, DC and the Corporación de Investigaciones para el Desarrollo (CINDE), Chile AED with MSI and managed out of USAID LAC in Washington, DC. Implementation Structure Implemented through a PIU. Worked thru TTIs (with reading specialists) to clusters of 6-8 schools. There was no country-level management outside of PIU. MOEs identified poorest schools. Support from Universidad de Valle-UVG (Guatemala), Pontificia Universidad Catolica Madre y Maestra-PUCMM (DR), Fundación Empresarial para el Desarrollo Educativa￾FEPADE (El Salvador) and Escuela Normal Ricardo Morales Aviles and MOE (Nicaragua). ILCE in Mexico provided advice and technical assistance in the areas of ICT and e education. Executive Committee principal policy-making body. Technical Committee was principal implementing body. National Coordinators implemented in each country. Support from Universidad Andina Simon Bolivar (Ecuador) and Universidad Nur (Bolivia). Executive Committee principal decision-making body. Components developed by regional technical teams. MOEs coordinated work in schools. Piloted activities in five countries to explore the relevance of engaging communities and parents in improving the quality of local primary school education using civic action and community mobilization methods applied to schools in Africa. The program emphasized the development of school report cards though action research. Teacher initiative groups were used to gain approval from teachers opposed to parental involvement. USAID Role Conceived and developed by USAID/Washington. Funding provided to CETTs through the Office of Education and Human Resources in LAC. Each of the three principal implementing institutions had a local CTO (now AOTR) by Year 2. USAID missions in other countries were not much involved. Centrally funded from the LAC Bureau Centrally funded from the LAC Bureau Evaluation of LAC/RSD Regional Education Program: Volume I 5 International Business & Technical Consultants, Inc. Table 2. USAID LATIN AMERICA and CARIIBBEAN (LAC) REGIONAL EDUCATION PROJECTS, Country Break-Down Caribbean CETT CARD CETT Andean CETT PREAL CERCA Countries (only countries with USAID bilateral programs are included in this list ) (only Central American countries selected for pilot activities) Guatemala Research & evaluation x x Honduras Teacher training methods x x El Salvador Partnership, fund raising & sustainability x x Nicaragua Validation of materials and components x x DR Materials development x x Peru Teacher training and pedagogical framework x Bolivia Materials development Ecuador Evaluation and Research Jamaica x St. Lucia x St. Vincent & Grenadines x Belize x Guyana x Dominica x Trinidad & Tobago (*) x Grenada (**) 2005 - St. Kitts & Nevis(*) 2009 - Antigua & Barbuda (*) 2009 - Anguilla (*) 2009 - Montserrat (*) 2009 - British Virgin Islands (*) 2009 - KEY (*) Self financed with USAID technical assistance (**) Self financed plus minimal USAID financial support Countries visited by Team 1 Countries visited by Team 2 (Both teams visited the Dominican Republic.) Evaluation of LAC/RSD Regional Education Program: Volume I 6 International Business & Technical Consultants, Inc. III. METHODOLOGY The IBTCI Team’s evaluation methodology focused on gathering qualitative and quantitative data, both from the field and from sources such as existing evaluation studies, reports and other documents that USAID provided. The evaluation methodology consisted of four kinds of data collection: (1) a document review; (2) in-depth, key-informant interviews (KII); (3) focus group discussions (FGD); and (4) an on￾line survey, followed by analysis of the data and synthesis of findings. A. The Country Sample Selection During November and December 2010, the IBTCI Team traveled to eight countries that USAID/LAC identified as representative of a cross-section of program activities and issues for evaluation. The Team split into two groups and worked in the following countries: the DR (all Team members); Honduras, Peru and Ecuador (Team 1); Jamaica and the Eastern Caribbean (St. Lucia, Grenada, and Antigua) (Team 2). B. Data-Collection Instruments 1. Document Review This evaluation began with a comprehensive review of documents, evaluation reports and other written information and data related to the three programs from both local and international sources. Besides providing key background and contextual information, the review was particularly helpful in filling in information about CERCA, which ended in 2006, and the earlier stages of PREAL, which was launched 14 years ago, because a number of the key informants and principal actors were no longer available. The Team also reviewed programs’ and partners’ annual and performance reports, country program and regional activity descriptions, national education report cards, and white papers. The document review provided context and informed the Team’s development of the instruments for KIIs, FGDs and the online survey that generated primary information on the current status of key topics including the projects’ outcomes and impacts. Because the primary purpose of this evaluation is to provide guidance to the LAC bureau as to how it can more effectively support bilateral and regional programs rather to serve as a performance evaluation, the team gave greater weight to the input from the KIIs, focus groups, and the survey than it did to the historical records, which primarily provided information as to start-up and implementation issues.2 2. In-depth Key Informant Interviews (KII) a. Research Objective The IBTCI Team conducted KIIs with 5-19 principal actors in each country. The objective of these interviews was to determine the benefits and challenges that the interviewees had encountered during their work with the three regional programs. The evaluators utilized the interviewees’ knowledge to document the wide-ranging effects of the regional education programs, such as the degree to which they influenced educational policies. b. Key Informant Interview Methodology Key-informant interviews were conducted in all eight countries, with interviewees drawn from the names provided on USAID’s participant list and from additional names found in documents and through referrals. The Team contacted potential interviewees before arrival in-country through e-mail, phone 2 During 2010, the Aguirre Division of JBS International prepared a series of White Papers reviewing its findings of its life-of-project M&E of CETT for the LAC bureau and The Mitchell Group prepared an evaluation of the bilateral AprenDes and regional CETT projects for USAID/Peru, and these provided particularly valuable information to the Team. Evaluation of LAC/RSD Regional Education Program: Volume I 7 International Business & Technical Consultants, Inc. and when necessary, with the help of regional or local field assistants contracted by IBTCI. USAID Missions also assisted with contacting key informants when necessary. Two of the KII draft instruments were pre-tested in the DR on four university-based people. The instruments’ content and length were adjusted to reflect respondents’ understanding of the questions, their ability to provide appropriate answers, and the time required to complete the interviews. The interviewers used an interview guide that provided a flexible framework for the interviews. The Team developed a total of five KII interview guides, in Spanish and English, which were used for interviews with USAID, MoEs, PREAL, CERCA and CETT. The questions on the different instruments were standardized in order to ensure that the evaluators asked the same questions, or a subset, to interview the different respondent groups. This facilitated comparing responses and analyzing the information across different groups. When possible, two IBTCI Team members attended each KII to ensure accuracy in recording the interviewee’s responses. The KII guide also was used for telephone/Skype interviews conducted in the United States and the LAC region. The Team designed phone-interview versions of the KIIs to use when in-person interviews were not possible. Phone interviews were recorded when possible, or carefully noted and then written up as field interviews. The write-up of all the phone interviews is archived with the rest of the interviews conducted in the field. c. Key Informant Interview Sample Profile The KIIs included present and past principal actors who have or had key roles in the management and organization of the three programs, and other relevant stakeholders. These informants included present and past senior-level USAID staff; senior MoE personnel; PREAL, CERCA and CETT coordinators and managers; and other principal stakeholders. The first stage in the sampling process was to review CETT, CERCA and PREAL reports and studies to identify potential key informants. These informants also were identified by the USAID/LAC Bureau, the Inter-American Dialogue, AED, and retired senior USAID managers who had had key roles in all three regional programs. In addition, the IBTCI Team asked the key respondents to identify other potential interviewees. The list of key informants was enhanced by the Team’s early contact with the USAID Mission in each country.3 Based on the evaluation’s objectives, the sample of KIIs also included private-sector actors, researchers, academics, representatives from the MoEs and non-governmental organizations’ (NGO) personnel. The Team contacted the individuals by telephone and email to explain the evaluation and request their participation. A possible source of bias is that the evaluators identified interviewees with the USAID country missions’ help. But this bias was mitigated by the Team’s meetings with MoEs and with CETT staff in each country to obtain additional names. Key informants were generally receptive in all eight countries, with individuals offering to make time for an interview at very short notice and in some cases on local public holidays. The process of interviewing many individuals was facilitated by clustering the interviews in one location, such as at the MoE or at a Teacher Training Institute. This was time-efficient because the Team was able to conduct several interviews without losing time to travel. 3 This was not the case in Grenada, St. Lucia and Antigua, where there is no local USAID Mission. The USAID Mission in Barbados oversees these countries. The Caribbean Team had to rely on the PIU/JBTE in Kingston and use the snow-ball method in other countries to identify potential key informants. Evaluation of LAC/RSD Regional Education Program: Volume I 8 International Business & Technical Consultants, Inc. The IBTCI Team conducted 123 KIIs in the eight countries plus additional KIIs in Washington and elsewhere with USAID staff and with others involved with implementation of the programs. Fourteen KIIs included USAID education and other program staff. Three USAID program officers who had been transferred overseas participated in telephone interviews. The total number of key informant interviews (from both in-person and telephone interviews) conducted in each country and for each LAC program is shown in Table 3. d. Organization and Analysis of the Key Informant Data The Team developed a consistent approach for the data collection and analysis during the first week in the field, i.e., in the DR. The 10 pre-test interviews showed clear cross-cutting themes, which indicated that a thematic approach would best capture informants’ responses and facilitate answering the SOW’s questions. The Team concurred that a thematic approach also would facilitate coordinating the research as its members would separate for work in seven additional countries. The Team’s first step in organizing the key￾informant data was to make five matrices with the main SOW questions along the vertical axis and the types of informants along the horizontal axis. As illustrated in the chart below, extracted from the matrix for CARD-CETT, each question in the SOW had the detailed responses from different types of informants laid out beside it. This made it relatively easy (1) to compare responses to individual SOW questions from, e.g., USAID staff, implementers, local officials, focus group participants, and respondents to the online survey, who came primarily from countries that were not visited by the teams and (2) to note how participants from the different types of stakeholder group saw the programs in total as well as with respect to particular topics. Chart. Extract from the CARD-CETT Matrix (Vol. II, Annex VI) SOW Question Project Leadership Ministry of Education and Other Public Sector Private Sector Focus Group Online Survey What are the key contributions of the regional education program to improving education quality in the region as well as in individual countries? ●CARD-CETT’s main contribution, according to informants, was to begin a professional dialogue between neighboring CARD institutions for the first time. ●A clearly defined ●In the DR senior MoE officials praised CETT’s contribution to the regional education dialogue, and thought that regional workshops, conferences and meetings funded by USAID, through CETT & PREAL, had ●In Honduras, CARE International claimed they had adopted the CETT model regionally and were using CETT materials to train teachers in several LAC countries including Nicaragua, El Salvador, Belize and Ecuador, with ●The regional contribution participants in the FGs lauded was the opportunity to visit and learn from other countries how to tackle common problems. This had never happened before. Tangible improvements in reading and writing from students in Grades 1 to 4. Table 3. Number of Key Informant Interviews Conducted, by LAC Program & Country Country Type of LAC/RSD Program Number of KIIs Done Antigua C-CETT 4 Dominican Republic CERCA 1 CETT 10 PREAL 7 Ecuador CETT 7 PREAL 2 Grenada C-CETT 15 Honduras CERCA 5 CETT 14 PREAL 11 Jamaica C-CETT 19 Peru CETT 12 PREAL 3 St. Lucia C-CETT 13 Total KIIs 123 Evaluation of LAC/RSD Regional Education Program: Volume I 9 International Business & Technical Consultants, Inc. SOW Question Project Leadership Ministry of Education and Other Public Sector Private Sector Focus Group Online Survey set of common education needs in the field of reading, writing and teacher training were identified. made an important contribution both to the internal as well as the regional policy dialogue. positive results. All the interview data were transferred into these matrices, which organized the data and facilitated its analysis. There was one matrix each for PREAL and CERCA, and three matrices for the CETT data from the three sub-regions. The evaluators entered the information from the KIIs and FGDs in bulleted form in the matrices at the end of each week’s field work. The Team’s findings and conclusions were based on the data which is incorporated in the matrices. The matrices, which represent first-order syntheses of the data provided by informants, appear in Annex VI of Volume II of this report. The team leaders used Skype and email for frequent contact throughout the field work. This enabled the Team to confirm and, when necessary, modify the common thematic framework emerging from the different respondent groups in the different countries. 3. Focus Group Discussions (FGD) a. Methodology Focus Group discussion (FGD) guides were designed to harmonize the discussions in the different countries. The participants worked in groups on a common set of topics. Group responses were recorded on flip-charts and discussed during the final plenary sessions, and the Team members recorded the responses. Teleconferences were held with FGD participants who were unable to attend the group discussions. The methodology for these “virtual” groups was the same as that for “live” groups. b. Sample Profile The USAID/LAC Bureau requested that the evaluators hold two FGDs in each country, with separate groups for representatives from the private and the public sectors. Except for Antigua, where no focus group discussions were held because C-CETT had not yet been fully implemented, public-sector focus group discussions were held in each country. Private-sector FGDs were held only in the CARD-CETT and Andean-CETT countries. Private-sector FGDs were not held in the Caribbean countries because the private-sector members were not available to participate and because of the sector’s limited involvement in C-CETT. Instead, the Team interviewed individual members of the private sector. Most of the eleven FGDs consisted of 5-15 participants, although because of problems contacting some potential participants and the proximity to the Christmas holidays, some focus groups were quite small. Table 4 below shows the number of participants in each focus group by country. Questions asked during these discussions focused on the benefits and challenges that participants had experienced with the regional programs. The participants in the public-sector FGDs included workers from MoEs and from NGOs that are/were actively involved in the regional LAC programs. These participants provided the names of additional potential participants for the FGDs. The Team selected FGD participants from this list based on their ability to provide information about the LAC programs. Evaluation of LAC/RSD Regional Education Program: Volume I 10 International Business & Technical Consultants, Inc. Table 4. Number of Individuals Participating in Focus Group Discussions by Public/Private Sector and Country Country Private-Sector Focus Groups Public- Sector Focus Groups Antigua Not held Not held Dominican Republic 2 5 Ecuador 7 4 Grenada Not held 15 Honduras 6 4 Jamaica Not held 13 Peru 5 4 Saint Lucia Not held 17 4. The Online-Survey Questionnaire a. Methodology The IBTCI staff used “Survey Monkey”4 to prepare an online questionnaire in Spanish and in English. The questionnaire was designed to collect information to assess each program’s performance in terms of its management, leadership, research, and other donor-coordination aspects. It was structured and consisted of open-ended questions in the following areas: • Program information; • Management; • Technical assistance; and • Recommendations The questionnaire, with the link to the Survey Monkey site, was e-mailed to a list of some 200 key informants, primarily drawn from people in countries that the Team did not visit. This list, provided by USAID/W, was a purposive sample of senior management and professionals familiar with PREAL, CERCA and/or CETT activities. Survey Monkey was useful because it was cost-effective (e.g., it reached some people outside the eight countries that the Team visited), controlled for interview fatigue that can occur during in-person interviews, and allowed respondents to complete the questionnaire at their convenience. IBTCI staff conducted the process of putting the questionnaires online, notifying the informants, monitoring the completion rate, and analyzing the data. Seventy-six individuals from twelve countries (Jamaica, the DR, El Salvador, Guatemala, Honduras, Nicaragua, Panama, Peru, Bolivia, Ecuador, Brazil, and Mexico) responded to the on-line survey. While generally substantiating the findings developed through the face-to-face and telephone interviews and discussions, and while the response rate was sufficiently high to give the team confidence that the survey results were reliable, the team opted not to generalize the survey findings to the larger population of people involved with CETT, CERCA and PREAL because the sampling was purposive, because the 4 “Survey Monkey”™ is an “off-the-shelf” web-based software program that is very commonly used to conduct self-administered quantitative and qualitative surveys and to analyze the findings along user-defined dimensions. Evaluation of LAC/RSD Regional Education Program: Volume I 11 International Business & Technical Consultants, Inc. respondents were self-selected, and because the team felt that there were too few respondents for some major categories (e.g., teachers and teacher trainers). b. Sample Profile Table 5 below summarizes the distribution of the online survey respondents by program and occupation. People affiliated with CETT constitute the majority (72%) of all the online respondents; 18% of the respondents work/worked with PREAL, and a very small proportion (9%) are/were affiliated with CERCA. The online survey thus provided more information on CETT than on the other two programs. Table 5 also shows that the majority of all the online respondents had advisory and management experience with the regional programs. Two-thirds of all the respondents were education specialists or advisors; managers of education projects or programs; directors of education programs; and presidents or vice-presidents of education programs. The majority of respondents therefore were most likely to be knowledgeable about adapting and managing the regional programs. Few teachers (only 3/76) and teacher trainers (7/76) responded to the survey, so the team considered these responses to be essentially anecdotal. (Overall, there were enough in-person interviews with people in these categories to more than make up for their limited response.) Table 5. Distribution of Online Survey Respondents Number of Respondents Percent of Respondents Distribution of Respondents by Program Affiliation CETT 55 72% PREAL 14 18% CERCA 7 9% Total 76 100% Distribution of Respondents by Current Occupation Education Specialists, Advisors 22 29% Management of Education Projects or Programs 14 18% Director for Education Programs 11 14% Teacher Trainer 7 9% Academia 3 4% President or Vice-President of Education Programs 3 4% Teachers 3 4% Independent Consultants 2 3% M&E Specialists 2 3% Other Unclassified 9 12% Total 76 100% c. Data Collection and Analysis The online survey data were stored in Microsoft Excel and analyzed using MaxQDA, a software application designed to analyze qualitative data. IBTCI staff entered the online survey data into the Evaluation of LAC/RSD Regional Education Program: Volume I 12 International Business & Technical Consultants, Inc. evaluation matrices and analyzed the data. The analysis consisted of descriptive statistics, correlations and cross tabulations. C. Analysis of the Data from All Sources Table 6 below shows the final sample of all the evaluation’s informants other than those located in the Washington area or interviewed by telephone. The two field teams and their leaders harmonized and aggregated the field data primarily through frequent e-mail and Skype contact since time and budget precluded physical meetings. However, prior to finalizing data entry into the matrices, three Team members (Frank Dall, Jeffrey Tines, and Cristina Accioly de Amorim) met in IBTCI’s home office to consolidate the field-data findings and coordinate the report writing. Table 6. Number of Key Informant Interviews and Focus Groups, by Program & Country The responses from the online survey were uneven: 71% of all responses were from the Andean and CARD-CETT sub-regions (n=54); there was only one response from the C-CETT region. CETT’s respondents accounted for 72% of the total, PREAL respondents comprised 18% of all responses (n=14) and CERCA respondents only 10% of the total (n=7). Six USAID and other senior respondents involved Country Key-informant Interviews Focus Groups Online Surveys Type of LAC/RSD Program Number of KIIs Done Private Sector Public Sector Antigua C-CETT 4 0 0 CERCA: 7 Dominican Republic CERCA 1 CETT 10 2 5 PREAL 7 Ecuador CETT 7 7 4 PREAL 2 Andean CETT: 20 Grenada C-CETT 15 0 15 Honduras CERCA 5 CETT 14 6 4 CARD CETT: 34 PREAL 11 Jamaica C-CETT 19 0 13 C-CETT: 1 Peru CETT 12 5 4 PREAL: 14 PREAL 3 Saint Lucia C-CETT 13 0 17 Totals 123 20 63 76 Evaluation of LAC/RSD Regional Education Program: Volume I 13 International Business & Technical Consultants, Inc. with at least one of the three CETT programs completed the online questionnaire, as did 25 managers and program participants. D. Key Evaluation Considerations 1. Confidentiality Given the regional and national sensitivities implicit in the implementation of five multi-national programs, the Team used several techniques to ensure the confidentiality of respondents’ information, including anonymous surveys and one-on-one unattributed interviews. 2. Cultural Sensitivity All members of the Evaluation Team have lived and worked in each of the evaluation areas, and therefore were well aware of the cultural differences between the areas. The Team took these differences into account while conducting the FGDs, interviews and meetings. For example, FGD participants in Ecuador and Peru received refreshments and a modest lunch to compensate for their time during the Christmas festivities. E. Challenges Encountered There were four major challenges encountered during the course of this evaluation. Challenge 1: The time allocated for this evaluation under-estimated the level of effort (LOE) required to complete a multi-country field research study using mainly qualitative data-collection methods. Impact: The short time for the field-research component in each country may have prevented researchers from gaining an optimal level of understanding and analysis. Challenge 2: The lists of key informants’ and key field personnel’s e-mail addresses and telephone numbers involved with CETT, PREAL and CERCA that the client and other sources provided were not accurate. Impact: More time was spent tracking down key informants than was anticipated. Challenge 3: Fieldwork coincided with the Christmas season. Impact: Despite help from the field assistants hired to invite potential FGD participants before the Team’s arrival, FGDs were sometimes poorly attended. This was the case with the private sector’s participation in the Caribbean and in some Latin American countries. Challenge 4: Telephone interviews were difficult to set-up and complete because informants did not have time, or because telephone communications in many C-CETT, CARD and Andean countries were cumbersome. The online survey questionnaire was sent to respondents who could not be reached by phone. Impact: Many phone interviews yielded less than optimal results, especially with key informants in Guatemala, Nicaragua and El Salvador. Evaluation of LAC/RSD Regional Education Program: Volume I 14 International Business & Technical Consultants, Inc. IV. ANALYSIS OF FINDINGS A. The Benefits and Challenges of the Regional Programs Beneficiaries and partner institutions reported that working with CETT and PREAL had clear benefits. The benefits included the opportunity to learn about educational problems and solutions in neighboring countries; to work as a team to achieve defined educational goals; to combine resources, knowledge and skills to tackle common regional problems; and to work together to produce well-researched and well￾designed educational materials that met international standards. The majority of key informants interviewed in the CARD CETT and Andean CETT sub-regions reported that, for the first time, these regional programs challenged them to look beyond their own borders at other educational systems, which in some cases were better than their own and already had tackled many of the educational problems that they still faced. The CARD CETT and Andean CETT partner institutions also noted a broader benefit of USAID regional programs: since they were managed by AOTRs at the missions but funded by USAID/W, funding and technical support were received quickly and in amounts that met the needs of program activities planned and implemented in the field. It is important to note that of the three regional programs, only CETT had on-the-ground activities in both Latin America and the English-speaking Caribbean. PREAL functions in Latin America, and CERCA, which ended in 2006, was implemented only in Central America and the DR. Given this, the Evaluation Team believes that it is appropriate to discuss the three regional programs primarily in terms of the Latin American sub-regions in which they work. The sub-regional framework is particularly appropriate for CETT, which evolved into separate approaches that corresponded to the sub-region’s different educational needs and types of social organization. The C-CETT model became a highly center-periphery model that fit the management and leadership style of its hub institution, the University of the West Indies in Kingston, Jamaica. The Andean-CETT became a separate and loosely administered technical partnership model that worked through three autonomous, private institutions in three neighboring Andean countries. The CARD￾CETT model evolved from the need to bring together different national organizational styles and very diverse national educational cultures under one cooperative umbrella. In order to function in Central America, CETT needed strong leadership, well-coordinated action plans, and the allocation of clear and complementary technical roles to all countries’ lead institutions. 1. Summary of the Regional Programs’ Challenges The Team found that the most significant internal challenge that the programs faced was difficulty in developing an effective regional communication strategy for coordinating the participation of key partners and stakeholders. This challenge stemmed from the internal diversity of each region, whether it was distance between the Caribbean islands or different national socio-cultural contexts. The resulting lack of a successful communication and coordination strategy had a negative impact on the development of: • Successful and durable public-private partnerships; • An effective distance training and learning approach, including communication infrastructure and hardware; and, most importantly, • A sense that the local USAID missions “owned” the CETT programs, e.g., as was the case in the MoEs and national teacher-training institutions in Barbados and other Caribbean island countries. Evaluation of LAC/RSD Regional Education Program: Volume I 15 International Business & Technical Consultants, Inc. Additionally, the Team’s analysis of the programs’ findings revealed a number of other major challenges in supporting regional education programs: • Although PREAL has had tangible results from its knowledge-sharing projects, the program has not systematically integrated these results into other national and international organizations’ education programs. • With few exceptions, senior CETT managers were critical of the lack of coordination between CETT and PREAL program activities. People interviewed in Honduras, the DR, and Ecuador suggested that the opportunity had been missed to link PREAL’s policy emphasis with CETT’s practical approach. • Respondents in Central America reported a feeling that the PREAL WGs were lopsided in distribution, with both based in South America (one in Peru and one in Uruguay). Better regional distribution of the WGs would have enhanced knowledge-sharing and ownership of the knowledge￾production process. This in turn would have encouraged dissemination of successful practices, such as those of Peru’s WG, which was working to strengthen existing mechanisms for monitoring and evaluating learning progress, at the national and regional levels. • Even though USAID is a major funder of PREAL, there was limited engagement between PREAL and local USAID missions. However, a U.S.-based coordinator was brought on board in 2003. This individual is perceived as not being primarily associated with a single country, and it is our understanding that this has greatly improved coordination. • The successful implementation of the regional CETT program depended upon the leadership capacity of the institution and person with overall responsibility for the program. The National Teaching University in Honduras (UNP), which was selected to lead the CARD-CETT program in Central America and the DR, lacked international-program experience, which led to delays in implementation. The program’s activities began to produce tangible results in its second year, but only after the program’s leadership had changed twice and an experienced Honduran educator with ample USAID and university experience took the lead role. 2. Caribbean-CETT a. Benefits There was a consensus among the Caribbean respondents that CETT’s sub-regional approach had benefits that a bi-lateral approach would not have had. The C-CETT countries benefitted from the program’s flexibility as they tailored it to their respective national needs and built ownership during its implementation. Based on its interviews, the Team found that the benefits of managing a regional project at the sub-regional level – the way that the AOTR based at USAID/Jamaica managed C-CETT, beginning in its second year – are that the management staff is familiar with the sub-regional context, has easy access to sub-regional stakeholders, and can communicate effectively with the sub-regional IPs. C￾CETT’s first phase and the effectiveness with which it introduced its technical component demonstrate the benefits of the sub-regional approach to managing a regional program. All the stakeholder groups highlighted a number of the same C-CETT benefits. The program, for example, gave them access to regional experts in reading and literacy, and a forum for sharing lessons learned and experiences in the sub-region. The program stimulated high-level discussion on literacy at the national levels, which in turn produced innovative methodological and instructional materials. Representatives from the private sector stated that the sub-regional PPPs were more effective and efficient than national ones. Informants also reported that the regional workshops were very beneficial Evaluation of LAC/RSD Regional Education Program: Volume I 16 International Business & Technical Consultants, Inc. because they gave educational personnel the opportunity to share experiences, exchange ideas and learn about sub-regional best practices. More specifically, respondents said that C-CETT brought a new approach to teaching reading whose benefits included using data for decision-making at the policy level, promoting accountability within ministries, improving school leadership, and having some influence on policies and teacher-training initiatives. The benefit of C-CETT’s use of diagnostic tests for measuring progress and influencing teaching strategies was a strong, positive impact on approaches for teaching reading, and ultimately on reading achievements. The program also increased technical capacity among the universities involved in its implementation. b. Challenges C-CETT and all the other education programs faced the same major challenge: the implementing agencies’ difficulty in developing an effective regional communication strategy for coordinating key partners’ and stakeholders’ participation. This was due to each region’s internal diversity, such as the distance between the islands for C-CETT. Additional challenges facing C-CETT include the following: • In terms of the program’s expansion and sustainability, the diagnostic tests used to measure progress and guide teaching strategies could have been a long-term liability for three reasons: difficulties in developing computer skills at the school level; UWI’s delays in returning some locations’ analyses; and the extra and complex work the tests produced for teachers, who had to design differentiated reading instruction. • The technological aspect of the project never was implemented as it had been conceptualized, thereby undermining the efficient use and evaluation of the diagnostic tests. • The countries that joined C-CETT during its second phase never received the technical and administrative support they needed to develop their capacity to implement the program well. • The USAID/Barbados mission staff believed that they were insufficiently involved in C-CETT’s management, which they felt had a negative impact on the program’s sustainability. • JBTE’s and USAID/Jamaica’s joint management of the program was less than optimal due to changes in key personnel in both organizations. The challenges arising from this were related to ownership, sustainability and PPPs, which the two organizations’ personnel apparently did not have the capacity to manage. • The relationship between the MoEs and the teacher-training institutions was not managed effectively and efficiently enough to ensure their collaborative participation and their ownership of C-CETT. Also a major challenge to C-CETT’s sustainability was, in general, the MoEs’ and/or the teacher￾training institutions’ lack of perceived ownership of the program. • There was a lack of communication between the public and the private institutions. Some private￾sector actors reported that they were not adequately informed about the target population’s benefits from their contribution, or the program’s progress. As a result, many felt that there was little reason to continue with their support. 3. CARD-CETT a. Benefits Compared to C-CETT, CARD-CETT comprised a much more diverse group of countries that were not accustomed to working together on common education problems. They benefitted from CETT’s sub￾regional approach, centralized, well-managed international funding, and timely technical assistance. Evaluation of LAC/RSD Regional Education Program: Volume I 17 International Business & Technical Consultants, Inc. These factors allowed them to address their education problems better than they would have if they had been working alone. The majority response about CARD-CETT, based on in-person interviews in the DR and Honduras, telephone interviews and the online survey, was that the sub-regional program’s benefits were greater than its disadvantages. CARD-CETT definitely made a positive contribution to the expansion of institutional and national capacities in the areas of research, development of educational materials, and assessment, according to staff at PUCMM and UPN. These staff also reported that their knowledge of project and financial management increased from working with CARD-CETT and that their institutions learned how to write proposals for external funding and to develop outreach programs. The institutional-level benefits reported by informants included the strengthening of their capacity to: solve problems collectively and across national borders; analyze data to identify educational issues of mutual concern; work together to develop and produce educational materials and publications that met international quality standards; and develop and expand regional professional networks. CARD-CETT enabled these institutions to expand their research and share their results across the sub-region, which ultimately improved their research capacity and their professional networks. These factors in turn opened up the academic research dialogue to other regional researchers and education practitioners, enabling them to work together for the first time toward common goals. Many informants stated that most of these benefits would not have occurred without an adequately funded regional program such as CARD-CETT. b. Challenges According to informants in the CARD countries, one factor in CARD-CETT’s weak start was a USAID management issue: the degree to which the participating USAID missions which had signed the official CETT Support Agreement provided actual support. The Missions’ support of CARD-CETT was uneven and in some cases not perceptible. Informants’ general opinion was that even after they started supporting CARD-CETT, the Missions were not the driving force behind its success: USAID/W was, as it had the authority to manage, fund, and supervise the program. Another challenge in program management was the choice of Honduras’s UPN as the program’s lead partner. UPN’s poor record of regional leadership in education, its lack of experience in managing international programs like CARD￾CETT, and the university’s lack of strong leadership contributed to what informants described as an initial two-year period of confusion and lost opportunity. CARD-CETT faces two challenges to its sustainability: First, the program’s policy achievements are fragile, which means that it is not solidly integrated into national education policies. Second, the MoEs in most CARD countries are seeking financial support from international donors to maintain the program, and are hesitant to move forward without it. The evaluators’ research showed that in Honduras, for example, while the MoE knows about CARD-CETT’s achievements and is supportive, it also is uncertain if the Ministry can implement the program’s methods nationally without financial support from international sources. 4. Andean-CETT a. Benefits The Andean-CETT brought three Andean countries (Bolivia, Ecuador, Peru) together around a central hub led by Universidad Peruana Cayetano Heredia (UPCH) in Lima, Peru. These countries benefitted from this sub-regional CETT, because while there were national differences, particularly for Ecuador, they focused on one common, shared educational issue – how to improve the quality of students’ reading and writing, from both the teacher’s and the students’ perspectives – and developed a common training model. CETT’s technical capacity in teaching methods, M&E and program management were Evaluation of LAC/RSD Regional Education Program: Volume I 18 International Business & Technical Consultants, Inc. some of the program’s key benefits, and helped to establish its high-quality technical status in Ecuador and Peru, and to a lesser degree in Bolivia, according to key informants interviewed at UASB in Quito and UPCH in Lima. A major benefit for the countries involved with the Andean-CETT was that it had the flexibility to adapt its pedagogical model, educational materials and assessment tools to each country’s needs. In Ecuador, the program stimulated a technical discussion about literacy, which resulted in a study that identified gaps in performance in literacy and the lack of in-service training programs to address them. In Peru, the Technical Vice-Minister of Education reported that the program helped to consolidate the MoE’s reading and comprehension methodology and its evaluation process. Respondents from UASB in Ecuador reported multiple benefits from the program, including that: (a) it instilled the need for further research on themes related to educational quality; (b) the pedagogical approach was successful and decreased teachers’ mistreatment of students; (c) coaching and monitoring helped to measure learning based on standards of quality; (d) class simulation was an activity that teachers appreciated very much; and (e) the MoE recognized CETT’s graduate teachers by promoting them in its career hierarchy. The three Andean-CETT countries benefitted from improved M&E of both student and teacher learning from: • The creation of a series of M&E feedback loops that generated a new work dynamic and helped focus partners on the current education issues in hand; • The ongoing monitoring process, which allowed beneficiaries to learn from experience and apply their lessons learned to improve training outcomes; • The flexibility to define the content and purpose of each teacher-training program; and • Its feedback process, which encouraged teachers and trainers to give their opinions about the program’s positive and negative aspects and incorporated their input, which was used to improve the Andean-CETT program. In Ecuador, Peru and Bolivia, the technical assistance on Differentiated Instruction (DI) in classroom management was critical. The findings from Ecuador were that exposure to international ideas helped teachers to explore management techniques for classrooms with students at different cognitive levels. Training teachers in DI benefitted both the higher- and the lower-capacity students. Another benefit was that the Andean-CETT program brought all levels of education stakeholders together to collaborate. Book fairs and other program activities enhanced participation and ownership in the learning processes that CETT promoted. Peru created a volunteer movement (Movimiento Pedagogico por la Lectura y Escritura, the Teaching Movement for Reading and Writing) in which over 800 teachers and teacher-trainers – mainly CETT graduates – committed to training other teachers in the CETT methods. The program now has over 280 teacher-trainers who can implement new training programs. 5. CERCA a. Benefits Although intended to be a regional project, in practice CERCA functioned as several bilateral projects for the most part. Although modest and difficult to assess, CERCA’s impact and benefits were tangible, particularly given the program’s short life-time and limited resources. Informants reported that CERCA’s most important contribution was the introduction and development of school-based report cards (SRCs). The SRC pilot project was implemented through a process that mobilized, raised Evaluation of LAC/RSD Regional Education Program: Volume I 19 International Business & Technical Consultants, Inc. awareness and empowered communities and parents to get involved in the decision-making processes of their local schools. In Guatemala and El Salvador, CERCA SRCs were quickly adapted and adopted by both ministries. In Nicaragua, although progress was slower, SRCs eventually played a role there too. CERCA informants also cited AED’s support and training in how to mobilize communities and parents, conduct case studies, establish and convene teacher-initiative groups, and carry out monitoring and evaluation activities as benefits. CERCA thus had some limited, sub-regional benefits. b. Challenges CERCA did not develop into a fully-functioning regional program. The Evaluation Team interviewed only six informants in Central America who remembered CERCA. They agreed that CERCA had been underfunded, poorly supported and ended prematurely. Key informants in two NGOs in Honduras and the Dominican Republic were quick to point out that if the program had been allowed to complete what it had started in their respective countries, CERCA would have left a helpful procedures for mobilizing community and parental support for education through its successful school report cards, action research and teacher involvement activities. They supported the few regional conferences, training workshops and visits to neighboring countries, as positive CERCA contributions, but they thought these were too few to make a lasting impact. 6. PREAL a. Benefits After more than a decade of activity PREAL has become one of Latin America’s leading private organizations dedicated to changing the way public- and private-sector leaders in Latin America conceptualize schools and education. The Working Group on Learning Standards is one of the most successful programs that PREAL created: the group promoted national discussion about setting learning standards and concomitantly worked to strengthen existing mechanisms for monitoring progress toward learning goals at the regional level. PREAL has successfully mobilized key civil-society actors to influence education policy reforms. The program’s experience clearly shows that politicians, decision-makers, and opinion-shapers – and the policies and initiatives they support – can be influenced by good ideas, information, and recommendations from reliable sources, not only by pressure. The program has shown that influencing policy is effective when regional networks work together, sharing information and providing mutual support. Regional networks also help create a critical mass of people dedicated to improving education at both the national and regional levels. This is particularly important because translating good ideas into practice generally requires sustained and consistent messages over several years. There are many benefits of the PREAL program, which designed and promoted reforms to rectify various problems in Latin America’s educational systems. These problems included challenges in setting learning standards and evaluating progress toward educational goals; developing tools to monitor progress on education reform to improve accountability at the regional and national levels; and strengthening the capacity of national partners to promote education reform. Respondents reported that PREAL had numerous specific benefits: • PREAL created an important permanent space for education policy dialogue at the national and regional levels, and supported a national network of people involved in education. • One of PREAL’s most important contributions was the promotion of common educational standards and assessments. Evaluation of LAC/RSD Regional Education Program: Volume I 20 International Business & Technical Consultants, Inc. • Most of the program’s publications, conferences, electronic communications and information are designed to improve information-sharing and collaboration in the region, and to be relevant at both the regional and national levels; • PREAL’s Central American and South American NRCs provide country-specific data and prioritize comparative analyses at the sub-regional and regional levels. The NRCs provide insights into national education policy topics and generate comparable regional and international data to show trends, common problems, best practices and shared solutions. • PREAL’s Best Practices Program, its research program and WGs, and its Business-Education programs are designed to foster collaborative regional cooperation to solve common education problems. • Countries in Latin America can learn about their neighbors’ education sectors through PREAL. • The program provides a knowledge center that shares research and publications on important educational themes. • PREAL promoted strong civil-society partnerships that provided continuity in the education dialogue to each new administration. These partnerships were both a stimulus for change, as they pressed governments for reform, and were political allies in the efforts for improvement in the education sector. • PREAL supported knowledge sharing on key education issues based on high-quality research and on publications focused on improving educational quality and equity. PREAL’s regional publications helped bridge the gap between researchers and education practitioners by providing user-friendly information on common problems, progress, best practices and innovations. • The program disseminated information on education issues to decision-makers, education specialists and academics. Challenges No respondents reported having had a negative experience with PREAL. However, several informants in different countries agreed that PREAL needs to increase the dialogue among participants in the Associaciones Básicas program. There was more opportunity for partners to exchange experiences and learn from one another when the program began. The program has now suspended all regional Committee meetings. This has created a number of challenges: • Working Groups are now concentrated in South America; there are none in Central America. • The funding for PREAL’s activities is insufficient to fully implement its work. • Study tours are now limited and travel experiences are seldom integrated into the country program. • The number of publications produced is insufficient. • Poor communication due to heavy workloads is a constraint on coordinating and interacting with USAID missions. • Problems in program planning occurred due to delays in USAID funds, which made it difficult for PREAL to renew agreements with partners/WGs on time. Evaluation of LAC/RSD Regional Education Program: Volume I 21 International Business & Technical Consultants, Inc. B. Coordination and Synergy among the Regional and Bilateral Programs 1. Coordination and Synergy across Regional Programs In practice, CERCA worked primarily as a set of bilateral activities with some regional components, rather than as a regional program. The Education Staff in the DR and Honduras, and the AED/W managers, said that before CERCA ended, USAID tried to transfer its School Report Card initiative to PREAL. PREAL has continued work on report cards, and we know that the principle of report cards has been replicated in the region, but we do not have enough information to determine the extent, if any, on which PREAL’s work has built on the work done by CERCA. While the Team did find that the PREAL and CETT programs did support and/or complement the bilateral programs in the LAC region, overall they did not do so to the extent that they could have. While both PREAL and CETT were effective programs, the Team did not find any clear evidence of any coordination or synergies between CETT and PREAL, or between those two programs and USAID’s bilateral programs, in any of the eight countries that the Team visited. There was neither direct coordination nor any apparent synergies between the CETT and PREAL programs in any of the eight LAC countries which the Team visited. Informants in USAID missions, MoEs and NGOs in the DR, Honduras, Nicaragua and Guatemala stated that they were not aware of any synergies between CETT and PREAL in their countries. The relationship between PREAL and CETT AOTRs and staff seemed to consist of receiving publications and attending occasional PREAL seminars, meetings and conferences in their sub-regions. Five Missions, three in the CARD-CETT and two in the Andean-CETT sub-regions, reported that they received information and publications regularly from PREAL's coordinating office in Washington. Some key informants in senior university and MoE management positions in the DR, Honduras, Ecuador and Peru had connections with PREAL only from receiving program publications and information. CETT senior managers in the DR, Honduras and Ecuador said that their connection with PREAL was through attendance at its national seminars and/or international conferences. The major constraint on coordination and synergy was that the regional programs were not specifically designed to integrate with each other; this obviously had an effect on potential coordination and synergies. Other constraints that informants reported included the frequent lack of agenda during discussions and coordination meetings between the local Missions and USAID/W; frequent staff turnover in USAID/W that changed management styles and priorities; and USAID/W’s poor monitoring and information-sharing. PREAL’s program leadership in USAID/W provided answers to most of the questions about coordination and synergy between PREAL and the other regional education programs. PREAL has made a concerted effort to coordinate with national partners and USAID mission staff in areas of mutual interest. Coordination among the PREAL programs has been one of its stronger points. This is a product of jointly developing strong conceptual frameworks, program models, collaborative activities, and multiple perspectives on complex education policy issues. The program continues to coordinate with a variety of organizations (bilateral, multilateral and private) by sharing its information, expertise and publications regularly with them, and seeks synergy with them on topics of mutual interest. PREAL’s leadership is in regular communication with that of the Organization of Ibero-American States (OEI), Organization of American States, World Bank, Inter￾American Development Bank, Central American Coordination for Culture and Education (CECC), and others. In the near future PREAL will sign agreements with UNESCO and the CECC to begin joint activities. PREAL also has been successful in focusing mainly on influencing opinion leaders, including those in the business sector. In addition, the following contribute to PREAL’s regional coordination and synergy: Evaluation of LAC/RSD Regional Education Program: Volume I 22 International Business & Technical Consultants, Inc. • A Central American Coordinator provides a bridge between PREAL staff, partners and USAID missions by visiting the participating countries frequently. PREAL’s national partners cooperate with Mission staff in areas of mutual interest, from jointly organizing events to carrying out special briefings for Mission staff. • PREAL’s Business Education Alliance program has successfully coordinated national business associations’ work by providing technical information, sharing best practices, conducting study tours and private consultancies, and planning conferences. • Working Group Coordinators have played a pivotal role in connecting regional experts around specific topics. PREAL also has brought together groups of national partners – Report Card team leaders, strategic partners, the Central American Coordinating Committee – to discuss common education problems and strategies. This is how PREAL’s co-directors maintain frequent contact with the representatives and leaders of international organizations and governments. PREAL also has areas in which its coordination has been weak. Before establishing the Central American Coordinator’s position, PREAL did not have a system in place that provided sufficient opportunity for interaction with USAID missions. There are still some instances when poor communication due to heavy workloads impedes coordination. The program has made considerable effort but has not yet found a successful formula for working with teachers’ unions. Nor has PREAL made a systematic effort to fully capitalize on creating synergies with international organizations in order to implement parallel education activities. The program has been less successful in connecting with parents or private citizens, although it should be noted that this connection is not a primary PREAL objective. 2. Coordination and Synergy between Regional and Bilateral Programs The evaluation fieldwork identified several constraints on coordination and synergy among the regional and bilateral education programs. Overall, respondents felt that Washington-based management of CERCA and the three CETT sub-regional programs did not work. Initially the LAC Bureau supervised the three CETT sub-regional programs, but they gained credibility and viability when sub-regional AOTRs took over their supervision and management, as discussed below. Ultimately, however, the three sub-regional CETT models’ success was linked to their partner institutions’ management and leadership capacities, and their ability to implement the CETT teacher-training model in coordination with the MoEs and national education reforms. For CETT, the centralized approach did not work at the regional level due to USAID/W’s geographic and administrative distance from the programs and their implementers. The results of this distance were that USAID/W did not understand regional educational needs well and was unable to communicate effectively with the regional implementing partners. The Team found that effective coordination of the regional education programs was based on management by a local USAID Mission and locally-based AOTRs who were familiar with the region’s context and needs and who could communicate frequently and directly with the IPs. This was a more effective and appreciated management approach than when USAID/W managed the program remotely. USAID/W placed AOTRs in Tegucigalpa, Kingston and Lima, which was appropriate, but the effectiveness of a particular CETT AOTR apparently depended on the extent to which different missions allowed him/her sufficient authority to supervise the CETT program activities and interact with other activities in their missions and on the extent to which individual mission management encouraged or discouraged coordination between the regional program(s) and its own bilateral programs. The extent to which there was cooperation at the mission level with the regional programs varied. From the interviews, it appears that Evaluation of LAC/RSD Regional Education Program: Volume I 23 International Business & Technical Consultants, Inc. to the extent that this was an issue, it was a constraint primarily during the early implementation phase of the program.5 CETT and PREAL could have better coordinated with bilateral education programs if USAID/W and the local Missions had established stronger management procedures. That would have led to more frequent coordination and joint planning, and to implementing regional program activities that were more closely aligned to bilateral and national educational needs. In Ecuador, Peru, Honduras, and the DR, USAID country program managers reported that USAID/W’s regional programs were not always aligned with local Missions’ bilateral program priorities. The AOTR in Peru found it challenging to harmonize the CETT program and the Mission’s bilateral AprenDes education program largely because the two programs had rather different foci. (As implemented in Peru, CETT was a two-year teacher- training program that worked with grades 1-3, mainly in peri-urban schools; AprenDes was a four-year intervention that worked with all primary-school grades and only in rural areas). The fact that some managers in USAID/Peru preferred that USAID promote only one teacher-training model exemplified the lack of coordination between the Mission’s bilateral program and the regional programs. There were also some site-specific factors, such as USAID/Ecuador’s lack of an education specialist or a bilateral education program that served as constraints on coordination with the regional programs. Among other constraints on coordination between regional and bilateral or Mission-based activities were: • Regional program activities were difficult to accommodate in bilateral programs when the local Missions had limited funding and sometimes struggled to implement their own education programs. • Local USAID missions and USAID/W did not create sufficient awareness about the regional programs, did not develop adequate capacity for coordination among them, and/or did not more actively promote coordination and synergy between the regional and the bilateral programs. • The observation from interviewees that USAID/W signed Memoranda of Understanding (MOU) for C-CETT and, to a lesser extent, CARD-CETT with many LAC Missions, but USAID/W’s regional program managers did not necessarily follow up after this and give the Washington-managed regional programs sufficient support. • The statements from some staff at local Missions that the Washington-based programs were imposed without sufficient Mission participation in their conceptualization and design, and without sufficient discussion about them. Apart from having management of the CETT cooperative agreements shifted to AOTRs in sub-regional hubs, several factors improved coordination between the bilateral and the CETT programs. These factors included: 1) regional education conferences for USAID staff held in Washington, D.C. and CETT workshops held in the region that enabled people to work together, share results and build professional networks; and 2) technical assistance from the Aguirre Division of JBS International that encouraged in￾depth technical discussions of common issues and trends. On the other hand, PREAL, another centrally funded program, but one which is focused on policy dialogue and research, has benefited from strong Washington based management, oversight and funding support. As noted above, in practice CERCA was not a regional program. Interviews with the few USAID and national informants who had experience with CERCA revealed their strong opinions that managing 5 This information was garnered through interviews with AOTRs and with other education staff at missions in each of the CETT sub-regions. To protect informant confidentiality, we are not specifying “Who said what.” Evaluation of LAC/RSD Regional Education Program: Volume I 24 International Business & Technical Consultants, Inc. CERCA from USAID/W and with contractors was inappropriate. The informants believed that as a culturally sensitive social-mobilization program, CERCA would have been more successfully addressed as bilateral programs managed by USAID missions working through locally-identified national institutions. Only the USAID/Nicaragua program officer reported coordination with CERCA, mainly because a CERCA pilot project was implemented in Nicaragua when the MoE was seeking assistance for a national education-decentralization project. C. The Broader Impacts of the Regional Education Programs 1. Overall Observations In all eight countries, KIIs and meetings with partner institutions and MoEs revealed broad positive outputs of the CETT and PREAL programs on beneficiaries. For CETT a key impact has been the measurable improvement in teachers’ pedagogical skills and in students’ reading and writing skills, as evidenced in participating schools in Honduras, the DR, Guatemala, Nicaragua, El Salvador, Ecuador, Jamaica, Grenada, Antigua, and Peru. Key in reaching these impacts were CETT’s outputs, the most significant of which were found to be: • Introducing innovative methods for teaching reading and writing. This has happened formally, at the national level, in Honduras, Nicaragua, Jamaica and the DR. • Introducing a practical and appropriate teacher-training model. The CETT teacher training approach has been incorporated into teacher training methodology courses at the PUCCM in the DR, the UPN in Honduras, the Escuela Normal in Nicaragua, the UASB in Ecuador, the UPCH in Peru, and the UVG in Guatemala. • Introducing a relevant learning-assessment approach to measure the effectiveness of teaching reading and writing. These approaches have been incorporated and are being implemented in selected school in the DR through a USAID-supported Effective Schools project. In Honduras international NGOs like CARE are actively incorporating CETT methods in a pilot program. In Ecuador CETT methods and assessment approaches have been incorporated through the Reading Schools project, and in Guatemala the UVG mobilized parental support to introduce new reading and writing learning assessment approaches through their Creative Classrooms project. • Introducing a school-level evidence-based M&E procedure. With the help of a classroom based evaluation procedure developed with JBS/Aguirre’s help, the UVG in Guatemala implemented an in￾country training program which was carried out at the request of participating CARD-CETT partners, for their CETT evaluation staff. • Making a positive change in the way partner institutions organized and managed themselves, tracked their funds, and coordinated their research and training activities with other national and international institutions. Senior staff at the PUCCM, the UPN, the UASB and the UPCH attested to having learned how to improve their in-house financial management, planning, outreach and other administrative procedures, as a result of their exposure to USAID management protocols, through CETT. In the case of PREAL, its four main knowledge-sharing programs were designed to support reforms needed to overcome some shortcomings of Latin America’s educational systems. The four were: 1) the dissemination of relevant data, best practices and lessons learned; 2) Assessment instruments; 3) the capacity-building program; and 4) the Network and Policy dialogue). To disseminate knowledge, PREAL created the WGs program to support educational improvement at the national and regional levels. The Evaluation of LAC/RSD Regional Education Program: Volume I 25 International Business & Technical Consultants, Inc. WGs currently are focused on the teaching profession, national-education standards and assessments, and school management issues. A key impact of PREAL has been the ability of its WG on standards and evaluation to help move standards from relative obscurity to a rigorous debate, and in Peru, Guatemala and Honduras, to implementation. The WG on the teaching profession worked extensively with programs in several countries to recognize teaching excellence and to foster informed discussion throughout the region on effective teaching. Awards, sponsored by business and NGOs with support from MoEs, have created incentives for improved teacher performance, a vehicle for consolidating and discussing better quality teacher training, and have led to creation of a databank of best teaching practices. PREAL focused on improving quality and equity in education while simultaneously helping to bridge the gap between researchers and educators with user-friendly information on problems, progress, best practices and new ideas in education. Regional publications were another knowledge-sharing instrument that PREAL developed to help create demand for sound, comparative, easily-comprehensible information. The “best-practices” program provided a regional (and inter-regional) perspective and contacts that facilitated the communication of best policies and practices, and the development of issue￾specific networks. After policy discussions, conferences and publications on the theme of accountability in the region, PREAL developed the Report Card (RC). A key impact of the RC has been its adoption and use as a powerful instrument for national accountability, and for promoting policy dialogue and decision-making among education stakeholders. For instance, respondents in Ecuador reported that after analyzing the 2008 RC, the Ministry of Finance threatened to reduce the MoE’s budget because, based on the RC, the latter was not spending its budget efficiently. For CERCA, key informants in Nicaragua, the DR and Honduras believed that if CERCA had continued and had had sufficient support and funding, its SRCs, teacher-parent collaboration, school accountability and community involvement activities would have made an important contribution to the decentralization efforts that already had begun in Nicaragua and were being adopted in the DR and Honduras. Respondents agreed that the SRCs have helped to hold schools more accountable for the quality of their teaching and learning, and as noted above, the education staff in the DR and Honduras, and the AED/W managers, said that before CERCA ended, USAID tried to transfer its School Report Card initiative to PREAL. 2. Impacts on Attitudes, Perceptions, and Policy The extent to which CETT programs changed attitudes was most significant at the school level, among teachers and school principals, where educators became strong advocates of the CETT model. At the ministerial level, the regional programs increased awareness of the benefits of reading and writing programs as well as the need to train teachers in specific pedagogical methods like those the CETT model promoted. However, the regional CETT programs could have done a better job of informing senior government officials of the positive impact that the CETT model made on improving the reading and writing skills of students in poor and marginalized schools. This communication gap may have been because CETT gave academic institutions with little or no experience with disseminating and sharing innovations full responsibility for promoting CETT outcomes and products. This deficit was partly the result of poorly-implemented outreach and dissemination strategies that the IPs employed. The influence of the regional programs on national educational policies was uneven and difficult to assess. The Caribbean CETT model had a positive influence on the MoE in Jamaica, in terms of new approaches for teaching reading and writing. In Ecuador, the national government adopted some aspects of the CETT pedagogical approach in its policy for reforming how reading and writing were taught in Evaluation of LAC/RSD Regional Education Program: Volume I 26 International Business & Technical Consultants, Inc. elementary schools. In Honduras, with help from the National Teaching University (UPN), the national government training center has begun to instruct teacher-trainers in order to incorporate aspects of CETT’s teacher-training methodology into the college education curricula. In the DR, where the current Deputy Minister of Education and the Director of National Testing and Assessment were both responsible for initiating CETT activities in their country, the CETT teacher-training model is being incorporated into the national education reform process through the USAID-funded “Reading Schools (Escuelas Lectoras)” project. However, aside from these limited examples, the majority of MoE informants interviewed stated that because the ministries were not fully involved in the CETT program from the beginning, CETT’s activities had had less of an impact on national education policies than had been expected. The extent to which MoEs incorporated components of the CETT model into national educational policy or practice was limited, even though there were isolated cases in which CETT teacher-training and pedagogical interventions were recognized as having a positive impact on teaching reading and writing. The perceived high cost of implementing the CETT model was a significant factor that inhibited the incorporation of the CETT model’s components into national teacher-training policies and practices. In St. Lucia, an informal cost-benefit analysis found that the costs associated with implementing the CETT model were unaffordable when compared with CETT’s overall benefits. As a result, the MoE did not adopt the CETT model. Unofficially, however, many components of the CETT model are still being implemented by school principals and teachers, despite the MoE’s decision. This high cost of implementation was the primary factor that inhibited MoEs’ adoption of components of the CETT program in the three regions. In Honduras, although supportive of CETT’s high-quality, technical elements, the Deputy MoE was adamant that the program could not be fully adopted during Honduras’s current economic situation because it was too costly. MoE officials in the DR said that implementing CETT at the national level was impossible until the government increased the budget for education. At the same time, the IPs in the region did not communicate well with their respective MoEs about the CETT model’s positive impact on teachers’ and students’ reading and writing performance in schools that were applying CETT methods. This inability of partner institutions and CETT beneficiaries to communicate the results of limited but successful applications of the CETT model contributed to inhibiting national support for CETT in countries such as Ecuador, Peru and Bolivia. In contrast, in Nicaragua and the DR, where MoEs had been brought into the CETT development process from the outset, commitment to the CETT model and methods currently is reflected in national educational policy and practice. In Nicaragua, the CETT method is already officially part of the National Teacher-training Curriculum for all elementary teachers trained at the National Teacher Training School (Escuela Nacional Normal). PREAL had a significant influence on attitudes and policy-making by introducing issues such as accountability, a relatively new concept in Latin America five years ago, into the policy dialogue through conferences and publications. The results were PREAL’s hallmark Report Card assessment instruments, now widely used in the region. By gathering national and regional education supporters into WGs, PREAL fostered informed discussion on effective teaching throughout the region. PREAL developed a network of business leaders concerned with promoting education reform, established business-education initiatives in five countries (Guatemala, Panama, El Salvador, Honduras and the DR) and regularly provided advice and assistance to groups in Nicaragua by encouraging business leaders to promote education reform. PREAL has helped these groups develop strong programs, connecting them with similar groups in other countries and working to put education on the Corporate Social Responsibility movement’s agenda in Latin America. Evaluation of LAC/RSD Regional Education Program: Volume I 27 International Business & Technical Consultants, Inc. CERCA’s status as a short-lived pilot program, with some local USAID Mission and MoE support in Nicaragua, El Salvador and Guatemala meant that it never attained its potential to influence national policies or change national education attitudes. If CERCA had an effect, it was at the local community and municipality level, but as its involvement was limited to only a few districts and schools, its visible effects were negligible. The exception to this was in Nicaragua, where national policies already existed to promote decentralization, where CERCA’s school-based action research, SRCs, parent-teacher coordination groups, community involvement activities, and training helped to institutionalize the country’s national decentralization policies. As a result of the SRC process and the World Bank’s parallel efforts to introduce SRCs in some districts, Nicaragua’s government absorbed SRCs into its district-level school accountability and quality-improvement policies. CERCA’s pilot status and short life span in each country most likely impeded its effecting educational change at the national level. A lack of public funding and USAID’s lack of sustained financial support for promising CERCA activities was another disincentive to adoption of this model. Finally, local Missions’ lukewarm support for a program that many felt USAID/W had imposed on them added another obstacle to promoting MoEs to acceptance of CERCA. 3. The Regional Models’ Capacity and Sustainability a. CETT To varying degrees, CETT developed capacity in the MoEs in Nicaragua, Honduras, the DR, Guatemala, Peru, Ecuador and Jamaica by introducing the model and its components, and by training specialized staff, with the impetus for having MoE personnel trained generally coming from partner institutions like the UPN in Honduras, the PUCMM in the DR, the Teaching School in Nicaragua and UVG in Guatemala. The specialized staff included Reading Specialists, school supervisors and trainers of teacher-trainers who were trained how to implement the CETT pedagogical approach in primary schools. Based on somewhat limited data, there is some evidence that ministry personnel have been adequately trained in the implementation of some CETT model elements. But further capacity-building is necessary: the extent of attrition among CETT-trained personnel at MoEs will be a critical factor inhibiting the replication and sustainability of the model. In the Andean region, CETT has encouraged the organization and training of teachers to promote CETT approaches to the teaching of reading. The extent to which the teacher-training institutions are capable of and interested in teaching CETT’s technical components is limited. CETT strengthened the capacity of teacher-training institutions in Jamaica, Nicaragua, the DR, Honduras, Guatemala, Ecuador, Peru and Bolivia, and the teacher-training institutions that were more engaged in the CETT were more likely to have integrated CETT’s components and principles into their teacher-training courses and curricula. However, in general, the model was not well integrated into national college programs and the national teacher-training curriculum. This has been a major factor in the teacher-training institutions’ limited capacity and interest in continuing to teach the CETT model’s technical components in their curricula. In the Caribbean, for example, where the MoEs did not officially adopt the model, too few teachers will be CETT-trained to ensure the program’s sustainability. Unfortunately, few MoEs in the region have shown any willingness to make a long-term commitment to continuing with the CETT program, for the reasons already stated. MoE informants reported that the extent to which the MoEs are capable of and interested in using the CETT model is limited due to factors related to finance, motivation, and political change. In general, the MoEs have developed some capacity to implement the CETT model, but further capacity-building is necessary. While the CETT program’s implementing agency has offered further technical assistance in this area, the MoEs that most need further capacity-building lack the financial resources and/or the will to take advantage of this technical assistance. Evaluation of LAC/RSD Regional Education Program: Volume I 28 International Business & Technical Consultants, Inc. The extent to which the teacher-training institutions are capable of and interested in teaching CETT’s technical components is limited. Although the CETT program was implemented on a limited scale through teacher-training institutions in most participating countries, the model was not well integrated into national college programs and the national teacher-training curriculum. This has been a major factor in the teacher-training institutions’ limited capacity and interest in continuing to teach the CETT model’s technical components in their curricula. This was observed at the Sir Arthur Lewis Community College in St. Lucia. Initially, the MoE strongly endorsed the CETT model, and it was introduced into the teacher-training program at Sir Arthur Lewis Community College. The JBTE in St. Lucia determines the content of teacher-training programs at a national level and unfortunately never revised the teacher￾training curriculum to include the CETT model. As a result, while some of CETT’s technical components have been integrated into some teacher-training courses, the teacher-training curriculum at the national level has not adopted the CETT model as a national standard. Information from senior MoE informants in technical positions indicated that in the eight countries visited, the extent to which CETT built sufficient capacity in partner institutions and/or MoEs to sustain and/or replicate the model is questionable. The primary reason for this is that the partner institutions and MoEs never committed themselves to the model sufficiently to enable them to obtain CETT’s assistance to develop their staffs’ capacities, which was necessary to adopt and implement a model of CETT’s complexity. Furthermore, many of the experienced and trained CETT trainers who worked in partner institutions and some MoEs left their CETT-related positions and found employment outside these institutions, often in the private sector. The extent of attrition among CETT-trained personnel has been a critical factor inhibiting the replication and sustainability of the model. Stakeholders in the DR, Honduras, Jamaica, Guatemala and El Salvador pointed out that the ad hoc movement and promotion of MoE personnel between departments and across professional boundaries was a negative factor in CETT’s replication and sustainability. In all eight countries, the newly-trained classroom teachers, reading specialists, supervisors and MoE technical staff that the CETT program had trained and supported used their new status to seek promotion or move to new positions. Some trainees moved into the private sector for financial reasons. Senior-level ministry personnel in Jamaica, the DR, Honduras and Guatemala who completed their CETT training left their ministry positions and have been replaced by new staff who were neither trained nor familiar with the CETT program. A reading specialist in Jamaica stated that there was a 40% turnover in CETT-trained classroom teachers in one school alone during a three-year period. This sort of high turnover rate obviously will have a negative impact on ministries’ abilities to sustain the current level of CETT program activities in the future. The Andean CETT program strengthened capacity in partner institutions by providing technical assistance to implement the regional program and by encouraging the creation of graduate courses at partner institutions. For instance, the UASB in Ecuador created a quality reading and writing course for the general public and the attendance exceeded all expectations. In Peru, the UPCH developed a graduate degree in reading and writing with progressive certification: Diploma, followed by a Master’s Degree and finally a Post-Graduate Specialization. UPCH’s program trained a small proportion of technical staff from the MoE and partner institutions. The UPCH program’s success in creating regional graduate programs in literacy skills, in all the partner institutions visited, indicates remarkable interest and ability in this sub-region in implementing CETT’s technical components. Senior informants in the DR and Honduras advised the team that institutions participating in CARD CETT have developed plans to continue collaboration through a new organization, RICETT, (Red Inter￾institucional del CETT, CETT CA-RD Inter-institutional Network) and hope to find sufficient support during the next three or four years to give the partner institutions and ministries interested in continuing CETT activities the time to consolidate existing gains and expand them while they seek Evaluation of LAC/RSD Regional Education Program: Volume I 29 International Business & Technical Consultants, Inc. longer-term financial support. However, per Barbara Knox-Seith, while it may be possible to provide some support for some facilitation, USAID is not contemplating anything more substantial at this time. In the Andean CETT, although there was no formal interest in continuing CETT’s technical components, coaching teachers in the classroom contributed to establishing the nationwide program PELA, the Strategic Program of Success in Learning (Programa Estratégico de Logros de Aprendizaje). According to Peru’s Technical Vice-Minister of Education, the model’s “value added” was that it helped the MoE to consolidate education’s reading, comprehension, and evaluation components. Notwithstanding Andean CETT’s accomplishments in building technical capacity in partner institutions, it failed to promote or develop partnerships with the MoEs or to design a workable public-private partnership model. With regard to partnerships with the private sector, these efforts soon failed, either because the local business or private sector groups did not have a tradition of providing significant support to public-sector programs, as with C-CETT and CARD-CETT, or because the university partner institutions tasked with developing PPPs did not have the necessary motivation or experience to do so. These factors may make the program difficult to replicate and sustain in the Andean sub-region, but it also should be noted that the current bilateral program in Peru, USAID/Peru/SUMA, is promoting the CETT model as well (though this has not yet yielded public-private partnerships), and its local implementing partner, UPCH, has continued promoting the CETT model among private companies. Additionally, while private commercial businesses never became supporting actors, CETT did form partnerships with other groups. The Andean Center for Ecuador, part of the Social Contract for Education, enabled CETT to connect with all the national and international NGOs working in education in the country. CETT also made an agreement with the Municipality of Otavalo in Ecuador to train 300 teachers. Other partners who helped finance the CETT in Ecuador were the Ecuadorian Fund for Progressive People (Fondo Ecuatoriano Populorum Progressio) (FEPP) and the Spanish International Development Agency (AECI). In Peru, CETT established partnerships with local companies like Pluspetrol, Natura, and Backus. CETT also developed partnerships with NGOs such as Entrepreneurs for Education (Empresarios por la Educación) and Plan International. Respondents reported that CETT products were very expensive: UPCH has begun to commercialize CETT training and products, and currently charges about US $80 to train each student and US $1,725 to train each teacher. 6 b. PREAL PREAL, after more than a decade of activity, has achieved a number of notable successes. It has become one of the region’s leading organizations dedicated to changing the way public and private leaders in Latin America think about schools and education. It has become a credible source of education information, analysis and policy recommendations for governments and civic groups across the political spectrum. Examples of its influence on policy include the following: • Peru’s Vice-Minister of Education, a longtime skeptic of education standard-setting, decided to initiate a program to establish national learning standards after repeated exposure to the arguments of PREAL’s working group on education standards and assessment. • With PREAL’s support, the Honduran national private-sector partner FEREMA has become a major source of information and influence on national education policies, including on the use of the first two national RCs to impact national education strategies, as well as on Education For All (EFA) planning and support to municipalities through a PREAL-FEREMA Strategic Partnership. 6 Please refer to The Mitchell Group’s Evaluation of USAID/Peru’s Education Program: Aprendes and CETT‐Andino, prepared for USAID/Peru, April 2010, which contains comparative cost information for Peru, with caveats due to data quality for cost comparisons of these two activities, and the Aguirre draft CETT White Papers by Ignacio Enrique Zardini and Pablo Javier Zardini on Cost Effectiveness and Sustainability for cross-national discussion. Evaluation of LAC/RSD Regional Education Program: Volume I 30 International Business & Technical Consultants, Inc. • After a PREAL-sponsored visit to successful business-education initiatives in Colombia, three private-sector leaders in El Salvador persuaded the MoE to establish a Business Advisory Council to channel business ideas directly to the ministry, modeled on a similar institution operating in Colombia. • After PREAL helped produce two national- and six state-level education report cards in Colombia, national organizations expanded the state-level report card initiative to include 16 state reports, from which local funding periodically produces a national report • PREAL worked during nearly two years to develop and encourage Central American ministers of education to jointly adopt a set of modern education-reform measures. In 2007, the ministers approved the Education Decalogue 2021, which contained all of PREAL’s key recommendations PREAL supported PPPs through its Business Leadership and Education Program. These activities aimed at mobilizing business leaders in specific countries to press for changes in educational policy and practice, and to invest in innovation and reform at the community- and local-school levels through business-education “partnerships.” To date, PREAL has developed a network of business leaders concerned with promoting education reform, and business-education initiatives in five countries: Guatemala, Panama, El Salvador, Honduras and the DR. These business leaders also regularly provide advice and assistance to groups in Nicaragua. PREAL has helped these business leaders to develop strong programs and connected them with similar groups in other countries. It has also worked to place education on the Corporate Social Responsibility Movement’s agenda in Latin America, but with mixed results. While PPPs have and are playing an important supportive role in education innovation and change In Latin America, compared to the Caribbean, PREAL could be doing more to take advantage of the movement’s involvement in the region’s covered countries. In Peru, for example, where the movement was very active, respondents reported limited knowledge of PREAL and/or its role in sharing knowledge in private-public partnerships. c. CERCA There is some evidence to show that CERCA has built some national capacity in some contexts. In Nicaragua where the CERCA program coincided with the country’s need to find new ways of implementing the decentralization program that the MoE was promoting CERCA methods were adopted to support this process. In Haiti, the USAID country program has adapted the CERCA SRC process for monitoring the quality of existing elementary schools. Two NGOs in the DR and Honduras, the Falconbridge Foundation (FALCONDO) and the Ricardo Maduro Education Foundation (FEREMA), reported that they had learned how to conduct action research, work with parents and communities, and implement the SRC process as a result of training from CERCA via AED, its implementer. The CERCA program pursued PPP elements largely by encouraging local NGO participation as a substitute for the private sector in countries where private-sector involvement and support for public services was never a strong tradition, such as the DR, Honduras, Guatemala and Nicaragua. In fact, in nearly every country where CERCA initiated activities, local and international NGOs were instrumental in working with AED and helping local municipalities adapt and adopt CERCA’s SRCs and other activities for marginalized local schools. D. The Value-Added of Technical Assistance AED provided technical assistance which made a significant positive contribution to the few gains that CERCA achieved in all five participating CARD countries during the short life of the program. Informants especially praised the help of AED’s field-coordination team and trainers who were Evaluation of LAC/RSD Regional Education Program: Volume I 31 International Business & Technical Consultants, Inc. responsible for guiding participating NGOs and communities through an intensive program of training workshops, seminars, and conferences. Informants also expressed special praise for the training they received to carry out SRCs, teacher-involvement groups, and school-based action research. Via an existing GEM I task order, and subsequently using a CETT TA MOBIS mechanism, USAID/W was able to provide flexible, timely and much appreciated technical assistance led by the Aguirre Division of JBS International, a firm that specializes in evaluation and assessment. Aguirre provided need-based, planning, training, ICT and evaluation services to all three CETTs in coordination with partner organizations and CETT AOTRs. Key informants, especially senior managers of the CETT program in partner institutions in the DR, Honduras, Jamaica, Ecuador and Peru, expressed appreciation for the inter-regional meetings that brought people together from the region who were involved in implementing the CETT models. Meeting participants reported that sharing their learning and problem-solving experiences across countries were highly valued benefits of the program. The Caribbean and Central America were the two sub-regions that evidently benefitted the most from the technical-assistance interventions that the implementing agency provided. These are also were the sub-regions where aspects of the CETT model were appreciated, adopted and partially implemented. Informants reported that the most useful types of technical assistance in the region were the inter- and intra-regional meetings and workshops in which people from different – or even the same – areas came together to: (a) share experiences related to implementing their respective CETT models, or (b) learn about specific pedagogical approaches that enhanced teaching literacy or the learning processes related to reading. However, respondents in the Caribbean also noted that the technical assistance provided on the pedagogical approach that the C-CETT model introduced and the emphasis on “active” learning tremendously enhanced the learning process in primary schools. Training Reading Specialists who then trained school-based practitioners to apply the CETT model was very effective. The most important outcome of the CETT program in CARD was the development and dissemination of a school-based M&E model that helped to bring accountability to primary schools where none had existed before. Technical assistance to CARD-CETT from the Language Learning Center at the PUCMM played a pivotal role in bringing partner institutions in line with the latest education research and thinking about teaching and learning language. With technical assistance from Aguirre, Universidad del Valle de Guatemala (UVG) developed a useful school- and classroom-based M&E system that later was transferred through training workshops to schools and classrooms in Guatemala, Honduras, the DR and Nicaragua. The PUCCM’s strong case for adopting their language-learning model at CETT’s initial stages was successful, bringing some consensus to the issue of which language theories to use, as well as which teaching methods and learning materials to develop and incorporate into the CETT model. In terms of the Andean CETT, the AOTR in Peru received technical assistance from Washington, D.C.- based contractors who came to the sub-region. Aguirre provided a broad overview of current issues and trends in education, not only in Peru but also throughout the LAC region. The AOTR stated that the CETT program was a good opportunity to encourage collaboration among Andean countries with similar education issues. Additional benefits of CETT’s technical assistance included building or strengthening technical capacity in key teaching skills and methodologies, areas of expertise that still are under-supported in many Latin American countries. Evaluation of LAC/RSD Regional Education Program: Volume I 32 International Business & Technical Consultants, Inc. V. LESSONS LEARNED AND RECOMMENDATIONS A. Lessons Learned 1. A program should have active links with national and international projects and programs to be relevant and have impact. Based on interviews with senior ministry and CETT staff in visited CARD and Andean CETT countries, findings revealed that opportunities had been missed to better integrate the best practices and policies being developed by both PREAL and CETT into quality improvement strategies for USAID bilateral projects. • Exploiting the synergies between CETT’s center-based program-implementation strengths and PREAL’s strong policy and PPP-based approach probably would have increased both programs’ relevance, acceptability, and effectiveness. • Exploiting potential synergies among the regional programs in key areas of education reform (e.g., combining CERCA’s successful SRC project with PREAL’s National Report Card initiative to improve accountability between student assessments and developing relevant national education learning indicators, or substantial sharing of the PREAL standards for reading with CETT activities to improve the teaching of reading) could have resulted in a greater impact on education quality improvement reforms throughout the LAC region. 2. A business leadership model like that created within PREAL can contribute to promoting entrepreneurship in education. This type of model also encourages community leadership by accelerating the dissemination of information about important educational issues to the public. This observation is the outcome of discussions and interviews carried out with PREAL coordinators and senior policy makers in ministries of education and in CETT partner institutions in the CARD, Andean, and Caribbean CETT countries visited. However: • While private-sector support can make important contributions to a program, maintaining this support needs special, continuous attention from USAID and the Implementing Partners. This is particularly the case for countries which have had only limited experience with formalized public-private partnerships. 3. The CETT applied-research and in-school evaluation model should be widely disseminated for potential adaption/adoption, owing to partner institution staff in the DR, Honduras and Ecuador drawing attention to the importance of the work that had been carried out by the UVG in Guatemala in the development of a practical and innovative monitoring and evaluation process, for use in schools. 4. Private-sector schools and key national institutions like MoEs should be fully involved in regional programs like CETT so that program ownership is transferred to them to ensure sustainability. Andean CETT partner institutions’ commercialization of CETT-generated information, materials and procedures was a disincentive to the Latin American MoEs’ acceptance of CETT. This observation was shared by CETT managers in Ecuador, Honduras, and to a lesser extent in the DR. A majority of MoEs stated that because they had not been fully involved with CETT from the beginning, CETT’s influence on national education practices was not as great as it could have been. However, it also should be noted that other factors also contributed to the ongoing challenge of engaging MoEs in CETT, such as the high turnover of many ministers. 5. USAID/Jamaica and C-CETT’s relationship shows that a Mission responsible for managing a regional education program must be able to fully integrate other local Missions and countries into working with the program. While the C-CETT’s AOTR’s location within USAID/Jamaica proved effective for that Mission’s bilateral education program, the USAID/Barbados Mission felt isolated by the Evaluation of LAC/RSD Regional Education Program: Volume I 33 International Business & Technical Consultants, Inc. implementation of a regional educational program in a geographical area for which it was solely responsible, the Eastern Caribbean. USAID/Barbados personnel felt that they lacked involvement in C-CETT, which they also believed impacted negatively on the sustainability of the C-CETT Model in the Eastern Caribbean. In CARD-CETT, all five countries had signed an MOU to commit resources and effort into supporting CETT activities within their countries, but only three countries seemed to have had committed the LOE and resources needed to make this a reality. 6. USAID missions should provide support and consistent management to a teacher-training model like CETT’s in order for the model to achieve its full potential. Sustainability of CETT was seriously questioned, especially in countries like Honduras, Ecuador, and Peru, where financial and technical support for CETT from USAID missions and ministries of education was weak or absent. 7. Teacher training and other models developed by local educators can benefit greatly from the formal and informal knowledge and skills sets derived from collaborating with similarly situated peers in other countries, in addition to technical assistance. 8. Regular national and/or regional meetings promote program progress and achievements. Throughout our field work respondents raised the importance of regional conferences, seminars and other events. For many individuals interviewed, travel and cross-border exchanges were the most important and memorable aspect of the regional program experience. 9. Based on statements from respondents involved with CERCA, USAID should manage community￾based programs like CERCA through bilateral agreements between missions and competent local institutions or partners, rather than centrally or by U.S.-based contractors. 10. Washington-based USAID programs can have an important positive impact on the quality of education in the LAC region. Overall, this evaluation found that all three regional programs did manage to raise awareness of the problems each attempted to tackle. In the CETT and PREAL programs, this awareness was translated into action and in some instances actions that have left their mark on national practice. For example, the CETT teacher training model had a positive impact on teacher training practices in most CETT countries. 11. Washington-based education programs can address common education issues across a region and work in countries that bilateral projects cannot reach. However: • Local Missions should have a major role in designing and planning regional programs so that they are better able to fully support program implementation. • Field missions should be given major program implementation responsibilities because of their knowledge of, and proximity to, the field. CETT worked much better after AOTR’s for each cooperative agreement started working from missions in each sub-region. 12. Regional programs like PREAL, CERCA and CETT have a better chance of succeeding if the USAID missions involved have active bilateral education programs supported by an education officer. In countries without USAID bilateral programs, which was the case in several Caribbean countries visited, or where USAID had a bilateral program but didn’t have mission level education support, regional programs were less likely to succeed because they were relatively unknown, lacked sufficient financial and technical support, and were poorly supervised. 13. ICTs can make important contributions to project success if well planned and implemented, especially where regular communication is critical for project implementation. 14. USAID must ensure that the hub-partner institutions responsible for managing/disbursing program funds have good financial-management skills. A more careful selection of lead partner institutions is Evaluation of LAC/RSD Regional Education Program: Volume I 34 International Business & Technical Consultants, Inc. called for if CETT–type regional program approaches are to be adopted in future. Respondents in C-CETT, CARD-CETT and Andean-CETT all complained about weak leadership and poor management skills on the part of their respective lead institutions. 15. In-depth understanding of the local public- and private-sector cultures is imperative before transferring or planning to transfer regional education programs like CETT to these sectors for onward implementation and funding. B. Recommendations The recommendations below are linked to the previous analysis, findings and conclusions, which are derived from the Team’s comprehensive analysis of data gathered from both secondary documentary sources and field data collected from interviews, focus groups, informal discussions and field observations. This data was complemented by information collected through an online survey, which was distributed to key informants who could not be reached face-to-face. Data gathered using all these methods was aggregated into emerging themes and then carefully entered into five matrices that answer research questions listed in Annex VII of Volume II. A further synthesis of the data entered provided the analysis and findings sections above. The Benefits and Challenges of Regional Education Program Models7 1. Designate Mission–Based USAID Staff to Manage Regional-Bilateral Program Relations: The Team found that the two major strengths in the PREAL program - its flexibility and responsiveness – were undermined by the program’s difficulty in engaging with local USAID missions and their bilateral programs. To compensate for this and maximize PREAL’s benefits, the Team recommends that the AOTR/COTR of a Washington-based program have support (possibly an alternate AOTR/COTR who works with or for the AOTR/COTR specifically) to serve as a project coordinator and communication facilitator. This would help future regional initiatives to maintain (or improve) communication with local USAID Missions, improve coordination and synergy, fill any other gaps that might exist in the program’s management structure, and ensure strong relations with USAID bilateral programs. 2. Engage host country decision-makers early in the program planning process: Host-country education decision-makers in the DR, El Salvador, Ecuador, Peru, and Bolivia, for example, resisted CETT as a sub-regional program because of inadequate pre-program consultations with USAID. A more localized approach to encourage greater integration and participation and frequent ongoing dialogue, would be helpful. If not already in place, we recommend establishing Technical Working Groups (TWG) comprised of senior MoE officials, project implementers and managers, and USAID education-office staff. TWGs should meet quarterly to review project activities and gain a better perspective on potential interventions – before the program begins. 3. Plan Ahead for Distance-Learning Activities: The strategies that the CETTs proposed to introduce ICTs and education technologies into the teacher-training processes were challenging, or unworkable for most CETT countries. To better inform any future program that may try to use similar approaches, USAID/W should first require a review of the proposed ICT technologies and confirm that the implementers and intended beneficiaries in a country to ensure that the local Mission/s have – or can be provided – the technology and expertise necessary to support distance learning. USAID/W also should review and evaluate the ICT approaches that C-CETT, CARD- 7 While many of these recommendations are viewed as cross-cutting (i.e., are not seen as belonging strictly to one evaluation question/focus area versus another), we have chosen to categorize them within the three main focus areas in order to facilitate the reader’s linkage of the recommendations to the appropriate findings/conclusions within Section IV. Evaluation of LAC/RSD Regional Education Program: Volume I 35 International Business & Technical Consultants, Inc. CETT and the Andean CETT developed, as well as assess the roles that local providers and Mexico’s Latin American Institute for Education Communication played in supporting ICT services for USAID’s regional programs. 4. Maintain/Expand Effective Program Components: The Team concluded that several interventions and project activities merited special attention. Successful regional program components require sufficient levels of technical support, including coaching, monitoring and evaluation, and sufficient funding to ensure their implementation and their expansion. Effective regional activities, such as PREAL’s Strategic Partnership Project in Central America, provide a useful model for local capacity￾building that merits support. a. The DR, Honduras, Nicaragua and El Salvador decided to include CETT components in their national education reform plans. Several of the institutions participating in CARD-CETT agreed to continue CARD-CETT activities through Red Inter-institucional del CETT (RICETT). PREAL plans to extend activities at least through 2012. It would be advantageous for USAID to consider providing some degree of support to RICETT through the remaining funding period for PREAL, either separately or in conjunction with PREAL. This may help to generate some collaboration between the programs that has been lacking to date in this regard. If it is not feasible or possible to develop a formal follow-on equivalent, this support could take the form of support for technical assistance or travel to be undertaken in conjunction with regional/sub-regional activities being funded by MoEs or other donors, conferences, etc. b. Successful regional program elements like PREAL’s Strategic Partnership project in Central America need encouragement by providing adequate funding and technical support such as coaching and monitoring and evaluation, to ensure proper implementation. c. Successful elements like the CERCA SRCs and PREAL Report Cards (RCs), whose expansion was hampered due to lack of funding, should be more widely adopted and incorporated into USAID basic education programs because of their positive impact on accountability. Report Cards can be used from the single-school level to the national-system level, and experience shows that they have been highly effective in Latin America as well as in Eastern Europe and in some African countries. According to Ecuadorian respondents, Ecuador’s Ministry of Finance threatened to reduce the Ministry of Education’s budget because, according to the 2008 RC, the MoE had not been spending its budget efficiently. d. Bilateral programs should provide bridging funds to support the work already going on in local hubs of excellence at centers like “The Center for Language Teaching Excellence” at the Pontifica Universidad Católica Madre y Maestra in the Dominican Republic, the “CETT Technical Support Center” at the Universidad Pedagógica Nacional Francisco Morazán in Honduras, or the CETT Service Unit at the Universidad Andina Simon Bolivar in Ecuador. All are CETT products and have a good chance of continuing their good work if they receive enough funding to support them until they become more self-supporting. 5 Adopt and Adapt Effective CERCA Activities: CERCA program elements that stakeholders reported were effective included the RCs, teacher-parent collaboration, school-based action research and community-involvement activities. USAID should consider adopting and adapting these elements into local bilateral programs if community-based regional programs similar to CERCA, are implemented in future. Evaluation of LAC/RSD Regional Education Program: Volume I 36 International Business & Technical Consultants, Inc. Coordination and Synergy among Regional Programs 1. Give Partners Early Training in Building PPP Alliances: The Evaluation Team recommends that USAID modify its approach toward implementing regional programs like CETT by enabling partner institutions and countries participating to become better prepared. Sub-regional partner￾coordinating institutions need to be trained how to build stronger working alliances with important program partners in both the public and private sectors before programs are implemented. The PPP approach was not effectively implemented in the majority of CETT countries and program managers did little to encourage MoEs and teacher-training institutions to become owners and partners of the three sub-regional programs. Sharing the lessons learned from PREAL’s previous experience in this area, would have been helpful. 2. Promote Even Regional Distribution of Working Groups: Assuming that resources are available, the Team recommends that PREAL establish working groups (WG) in the Caribbean and Central American sub-regions, in addition to those already operating in Peru and Uruguay. Creating more WGs will broaden knowledge-sharing across a wider area, as well as increase ownership of the knowledge-creation process. New WGs should follow the example of groups in Peru, where a national discussion on learning standards has also worked to strengthen existing mechanisms for monitoring and evaluating learning progress at the regional level. 3. Promote PPPs through Additional Working Groups: Creating more WGs to improve education within countries may enable “The Strategic Partnership Program” to play a more supportive role in increasing local capacity and bridge-building between the public and private sectors in the context of CETT-type regional programs. 4. Share CETT’s Lessons and Achievements with PREAL: In principle, the LAC Bureau’s decision to complement the PREAL program with CETT was well-founded. In practice, however, there was little interaction between the two projects, either regionally or nationally. As a result, the adoption and institutionalization of both PREAL and CETT policy recommendations was slow and less successful than had been expected. The work of PREAL is continuing in a number of countries. PREAL partner institutions have been successful in using both comparative and national research to promote national and sub-national education reforms, and it is anticipated that the program will continue advocating education reform to MoEs and other public and private education-sector stakeholders. PREAL also has voiced interest in utilizing CETT’s successful examples to further these goals. As a result, the team offers the following recommendations: a. Literacy and reading are both fields in which the quality of teaching and learning would have benefitted from greater interaction between the two programs (i.e., CETT schools would have gained from the work PREAL did in developing national educational standards, and PREAL would have benefitted from learning how useful CETT’s standards were in helping children from highly disadvantaged backgrounds to learn). USAID can use an existing forum to provide for knowledge-sharing between the two programs by involving qualified staff from participating CETT academic and teacher- training institutions into PREAL’s working groups. This would add experiences taken from classroom teaching situations to the recommendations that national PREAL partners make to their MoEs and other stakeholders. Conversely, national PREAL partners can draw attention to potential constraints that may have been overlooked in advocating the adoption of CETT’s teacher-training approaches at the national level. b. Most CETT partner institutions represented teacher-training institutions and the few schools that had implemented elements of CETT. By working more closely, CETT staff collaborating Evaluation of LAC/RSD Regional Education Program: Volume I 37 International Business & Technical Consultants, Inc. with PREAL WGs could benefit from knowledge-sharing about alternative/additional pedagogical approaches already in use. To bolster such sharing, the team recommends that the two programs’ collaboration be more systematic than merely attending workshops or doing panel presentations at conferences; rather, such interaction should be frequent and rigorous enough to promote increased opportunities for robust applied research. More frequent collaboration may also create greater opportunities for networking with education sector decision-makers, both at the national and regional levels. c. A closer collaboration would give CETT and PREAL an opportunity to re-engage MoEs in a dialogue that could lead to a more comprehensive and, preferably, more cohesive evidence￾based approach for effecting positive educational change. For example, a major objection to adopting CETT approaches voiced by MoEs was that while they appreciated the CETT teacher training methodology, it was unaffordable, given their current budget constraints. Here, perhaps, joint studies undertaken by CETT and PREAL partners in participating countries could document whether savings, or losses, in educational efficiency resulted from adopting CETT components. Developing a hypothetical analysis was well beyond the evaluation’s SOW. 5. CETT’s Sustainability: Although the CETT approach has proven to be both effective and popular with teachers, with the end of USAID funding and with the attrition of CETT trained personnel, the impact of CETT probably may dissipate fairly rapidly. To make up for the end of CETT, Missions with bilateral basic-education programs should confer with implementing partners to determine how the CETT experience can be incorporated into ongoing activities, either broadly or at the district/region levels. Interested Missions should also discuss with official stakeholders about the desirability and feasibility of incorporating appropriate aspects of CETT into their national pre-service and/or in-service teacher and administrator training, and professional development. Host-country education decision-makers in the DR, El Salvador, Ecuador, Peru, and Bolivia, for example resisted CETT as a sub-regional program because of inadequate pre-program consultations with USAID. A more localized approach to encourage greater integration and participation and frequent ongoing dialogue, would be helpful. If not already in place, we recommend establishing Technical Working Groups (TWG) comprised of senior MoE officials, project implementers and managers, and USAID education-office staff. TWGs should meet quarterly to review project activities and gain a better perspective on potential interventions – before the program begins. Coordination and Synergy among Regional and Bilateral Education Programs 1. Consider Alternative Contracting Models: CERCA’s centralized LAC Bureau management and funding structure required the program to operate regionally and through a cooperative agreement with a U.S.-based technical assistance provider such as AED. This arrangement proved less than productive in CERCA’s case; comparable programs in the future could benefit from a more direct transfer of knowledge and skills to local institutions. Local technical assistance can improve local ownership and contribute to the sustainability of successful program components after the assistance ends. 2. Bilateral Involvement in Regional Programs: All USAID missions in the LAC Region should designate activity managers to supervise the implementation of regional programs in the countries for which they are responsible. This would help to encourage the sustainability of the activities that regional programs like CETT and PREAL are implementing. Evaluation of LAC/RSD Regional Education Program: Volume I 38 International Business & Technical Consultants, Inc. The Broader Impacts of the Regional Education Programs 1. Long-Term Perspective: For regional programs to have a significant and lasting effect on national and regional education polices, they need to operate for sufficient time beyond the pilot phase to allow innovations to be absorbed into local educational practices. This can happen only if USAID provides the funding required to encourage MoEs to play a more active supporting role, to sustain program implementation over a period of at least eight to ten years. As an overarching strategy – rather than a specific recommendation – the Team therefore suggests that the personnel supervising and implementing regional programs have this long-term perspective in mind. For example, PREAL required 15 years of focus on education policy dialogue within and between ongoing national education reforms in LAC to ensure an active role for itself in education policy-making at the national and regional levels. For programs like CETT and CERCA, which were shorter-term in nature and focused on changing specific aspects of education practice, the policy￾influencing role was less certain. It should be noted that both CERCA and CETT aimed at influencing national education policy indirectly, by developing and implementing regional solutions to common education problems. Attitudes, Perceptions, and Policy 1. Increase MoE Involvement: In case future CETT-type regional programs are established, it is imperative, in cultural contexts like LAC, to include MoEs as equal partners in the program planning and implementation phases from the outset. Not doing so will threaten the sustainability of policy￾sensitive reforms like those that CETT proposed in the LAC region. CETT evidently has had some influence on changing education teacher-training practices in some countries – mainly where the CETT model was perceived as the outcome of a shared process between MoE officials and program implementers. Creating a regional MoE Consultative Committee would help to improve the sharing and coordination of partner country perspectives on education reform. 2. Engaging the Private-Sector: PREAL and its partners need to provide more opportunities for systematic engagement with the private sector and the business community at the national level. This is needed because the Team found that, with the exception of Honduras’s Chamber of Commerce (Camara de Comercio), respondents from four of the seven countries visited knew little of PREAL’s knowledge-sharing business approach. Creating opportunities for private-sector engagement in education policy reform is important because most PPP initiatives in the region are focused mainly on school infrastructure and construction rather than improving education quality. Capacity and Sustainability 1. Support Local Education Service Providers: Providing printed and published materials is essential in a program that promotes reading and writing, and should be shared equally among private-sector partners. Each country should have the chance to compete for contracts to produce and print essential materials through local service providers. This will make the materials better fit the young students’ educational settings in the different countries. Local production also will reduce potential resistance to using materials developed with a regional, rather than a national, student body in mind, it will create jobs and it will be an incentive for the local publishing and printing industries to improve the quality of their services. A centrally-contracted, international provider may appear to be efficient but it may hinder the building of sustainable local capacity in this key education service area. 2. Allow Flexibility for Varying Private-Sector Roles: In countries without a tradition of private-sector support for social development, such as in most Central American countries and in the Caribbean, the Team recommends that USAID not require developing PPPs as a prerequisite for funding and support to regional programs, since this may be perceived as inappropriate by partners. Where Evaluation of LAC/RSD Regional Education Program: Volume I 39 International Business & Technical Consultants, Inc. private-sector engagement is already a social norm, developing strong private-sector support for regional education programs like PREAL and CETT is feasible only if public-sector institutions are comfortable sharing donor funding and support with the private sector. 3. Leverage Corporate Social Responsibility Efforts: Future regional initiatives should promote PREAL’s approach especially as regards how the “Business Leadership and Education Program” might play a key supportive role in countries where the CSRM already has made an impact. In Peru, where the CSRM has a full agenda, respondents reported having very limited knowledge of PREAL’s PPP knowledge-sharing activities. Some CSRM respondents spoke of difficulties in identifying best practices to help the CSRM improve the promotion and sharing of available business resources like knowledge, financial resources, and management skills; others were not sure if CSRM’s resources for education programs were being used to support the best and most appropriate national education initiatives. 4. Improve Implementers’ Management Capacity: Implementer’s lack of organizational “know-how” about how to technically and operationally manage a sub-regional project was a major constraint in initiating CETT. The staff of regional coordinating institutions that are new to this type of activity could benefit from being mentored by more experienced institutions such as CINDE or UWI, or informal guidance from peer institutions participating in, for example, the USAID-assisted Higher Education for Development activity. 5. Public-Private Partnerships: There was very significant variability in the establishment, let alone the maintenance, of PPPs in the countries visited as a whole and within the sub-regions. Due to time and the nature of the SOW for this assignment, it was not possible for the team to do more than touch on apparent effectiveness, causes, and outcomes/impacts of PPP for the five programs. We recommend that LAC commission assessment of the role and effectiveness of PPPs (a) in education in the LAC region and/or (b) in a sub-region within the LAC region, to provide region-specific complementarity to the work being done by USAID's Global Development Alliances. U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Tel: (202) 712-0000 Fax: (202) 216-3524 www.usaid.gov It